[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[EP-tech] Re: load throttling strategy



I bet he's suffering the problem I had: Google loads a 'view' page, 
which has hit it's "out of date" time so rebuilds.
In the mean time, another httpd catches the next google page load, which 
has also hit its "out of date" time, so spawns ANOTHER page rebuild.

It very quickly ramps up, and WILL swamp the server... and WILL crash 
the server.

On 15/12/14 15:21, sf2 wrote:
> Which issue are you trying to solve by introducing such a delay?
>
> Seb
>
> On 15.12.2014 15:17, Gooding R.W. wrote:
>
>> Hi,
>>
>> does anyone have an easy way to throttle eprints website crawlers ?
>>
>> I?m wondering is something as a short delay for each ePrint accessed -
>> the delay being proportional to the current load ?  That?s perhaps a
>> little blunt but might it be very easy to implement ?


-- 

Ian Stuart.
Developer: ORI, RJ-Broker, and OpenDepot.org
Bibliographics and Multimedia Service Delivery team,
EDINA,
The University of Edinburgh.

http://edina.ac.uk/

This email was sent via the University of Edinburgh.

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.