EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #10180

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] DDoS on simple and advanced search

Hi all,

A further follow up on mod_security.  On the topic discussed below, I have rewritten the configuration to put all processor-intensive pages (for a vanilla EPrints installation) that are not cached* into a single location match:

ErrorDocument 429 /rate_limited.html

<LocationMatch "^/(cgi/search/(archive/)?(advanced|simple)|cgi/export(view)?/)">
  SecAction id:210001,initcol:ip=0.0.0.0,pass,nolog
  SecRule REQUEST_URI "^/" "id:210002,phase:2,nolog,setvar:ip.counter=+1,deprecatevar:ip.counter=60/12,expirevar:ip.counter=900"
  SecRule IP:COUNTER "@gt 300" "phase:2,id:210003,deny,status:429,setenv:RATELIMITED=1,skip:1,log"
</LocationMatch>

<LocationMatch "^/rate_limited.html">
    Header always set Retry-After "12" env=REDIRECT_RATELIMITED
</LocationMatch>

The main changes are the more complex LocationMatch regular _expression_ so only one rather than three blocks are needed hopefully mitigating the issue described at:

https://www.loadbalancer.org/blog/modsecurity-and-the-case-of-the-never-decreasing-variables/#:~:text=The%20act%20of%20deprecating%20one%20variable%20changes%20the%20timestamp%20that%20all%20variables%20in%20the%20collection%20rely%20on%20for%20the%20purposes%20of%20calculating%20their%20deprecation!

As I said in the earlier email it does mean all processor-intensive pages are lumped in together.  However, if you are under DDoS there is a fair chance that all of these pages may be being hit so separating them out is wasted effort. It does mean that setting suitable limit, decrementing and expiring is more important, otherwise you could lockdown a fair chunk of your EPrints repository for an extended period, potentially unnecessarily.  This why I have tweaked some of the settings, tripling both the limit and the amount the counter is decremented every 12 seconds.  I am still undecided whether it is better to have a bigger decrement less often of a smaller one more often.  I can see how both may have benefits but I expect real-world observation is probably necessary to determine, which way is better.

Another note I should make to people who are installation mod_security is that one of its default rules can affect you from being able to upload files 10MiB or larger and potentially even smaller than that.  I have therefore disabling (i.e. commenting out) the following rule in /etc/httpd/conf.d/mod_security.conf (will likely be a different location for non RHEL-based Linux OS):

    SecRule MULTIPART_UNMATCHED_BOUNDARY "!@eq 0" \
    "id:'200003',phase:2,t:none,log,deny,status:44,msg:'Multipart parser detected a possible unmatched boundary.'"

mod_security is possibly not the most "enterprise" solution to dealing with being eaten by AI bots but it is a solution that can be achieved without paid for software and should be no more complicated than installing a package (with DNF, APT or similar) and copying and pasting appropriate configuration into a suitable file.  Obviously determining what is "appropriate configuration" has so far been quite challenging.

Regards

David Newman

*Simple and advanced search, metadata export for individual eprint records and browse view lists of eprint records.  This does not include IRStats2, which can also be processor intensive.  Also, depending on whether you have a cron job to regenerate browse views overnight, it maybe worth considering adding /view/ to this LocationMatch regexp.

On 24/07/2025 14:37, David R Newman wrote:

Hi Yuri,

I had been reading that link as well.  I think the main point being you cannot trust deprecatevar to decrement the counters in quite the way you expect if you have multiple counters.  I think that means that my configuration below is best rewritten as a single albeit rather complex LocationMatch for all the pages that can be highly susceptible to DDoS or use a different variable collection (e.g. ip, tx, etc.) for each counter.  I am just about to test that.  In theory, it would be a much more condensed piece of configuration but with the inflexibility of blocking all configured pages or none.  However, when a DDoS is happening you probably want to go into hunker down mode.

Regards

David Newman

On 24/07/2025 13:39, Yuri wrote:
CAUTION: This e-mail originated outside the University of Southampton.
CAUTION: This e-mail originated outside the University of Southampton.

https://www.loadbalancer.org/blog/modsecurity-and-the-case-of-the-never-decreasing-variables/

This is a very interesting article, starting to show why variable can decrement slower than expected to script to calculate the decrement and the TX collection.

Best quote: "The same logic that requires a table of constants and dozens of SecRules to approximate can be accomplished precisely with a single line of Lua code:"

Il 24/07/25 13:48, David R Newman ha scritto:

Hi all,

An update on the mod_security configuration I sent round yesterday.  Both through results from real-world deployment and further testing I have picked up a few issues, which I have addressed with the following updated configuration:

ErrorDocument 429 /rate_limited.html

<LocationMatch "^/cgi/(search|facet)/(archive/)?(advanced|simple|simple2)">
  SecAction id:210001,initcol:ip=0.0.0.0,pass,nolog
  SecRule REQUEST_URI "^/cgi/" "id:210002,phase:2,nolog,setvar:ip.searchcntr=+1,deprecatevar:ip.searchcntr=20/12,expirevar:ip.searchcntr=900"
  SecRule IP:SEARCHCNTR "@gt 100" "phase:2,id:210003,deny,status:429,setenv:RATELIMITED=1,skip:1,log"
</LocationMatch>

<LocationMatch "^/cgi/export/">
  SecAction id:210011,initcol:ip=0.0.0.0,pass,nolog
  SecRule REQUEST_URI "^/cgi/" "id:210012,phase:2,nolog,setvar:ip.expcntr=+1,deprecatevar:ip.expcntr=20/12,expirevar:ip.expcntr=900"
  SecRule IP:EXPCNTR "@gt 100" "phase:2,id:210013,deny,status:429,setenv:RATELIMITED=1,skip:1,log"
</LocationMatch>

<LocationMatch "^/cgi/exportview/">
  SecAction id:210001,initcol:ip=0.0.0.0,pass,nolog
  SecRule REQUEST_URI "^/cgi/" "id:210002,phase:2,nolog,setvar:ip.expvcntr=+1,deprecatevar:ip.expvcntr=20/12,expirevar:ip.expvcntr=900"
  SecRule IP:EXPVCNTR "@gt 100" "phase:2,id:210003,deny,status:429,setenv:RATELIMITED=1,skip:1,log"
</LocationMatch>

<LocationMatch "^/rate_limited.html">
    Header always set Retry-After "12" env=REDIRECT_RATELIMITED
</LocationMatch>

As I mentioned in my earlier email you can create a rate_limited.xpage under your archive's cfg/lang/en/static/ directory to explain when a service has been DDoS-ed so real users understand why they have been able to access the page they expected.  This is then tied (in its rate_limited.html form) to responses that generate the 429 HTTP error code generated by the mod_security configuration above. 

However, I found that the Retry-After header was not getting set when requests were getting 429-ed.  This was because this technically is a redirect so the environment variable get prefixed with "REDIRECT_".  This needs to go in its own LocationMatch for the RateLimited page and means it need not be in all the other LocationMatch blocks.  For sanity, I also decided it best to set a value (i.e. 1) for the environment variable RATELIMITED.  I am not sure if this is necessary but it does not do any harm.

I noted from real-world analysis, that I was getting bursts of requests that would drop off after 10 minutes or so but that I was still getting requests that were being 429-ed, sometimes hours later.  I double-checked that deprecatevar:ip.searchcntr=20/12 definitely means decrement the counter by 20 every 12 seconds.  I have to reckon that this only works when a request is made, so if you had loads of requests to virtually no requests, this could lead to 429s being sent such a long time later.  Therefore, it seems sensible to have something that zeroes the counter after a certain amount of time.  I have gone for expirevar:ip.<COUNTER_NAME>=900 (seconds = 15 minutes).  If you were under persistent attack for hours this might keep re-opening your server to being DDoS-ed.  However, you will still only have up to 100 further requests, after hopefully a suitable amount of time since the previous tranche of requests that were processed, which should have allowed the server to recover.

Regards

David Newman

On 23/07/2025 17:59, David R Newman wrote:

Hi John,

I am not sure how easy it is to augment the mod_security configuration but maybe just encapsulating it in something like the following would do the job:

<If "%{REMOTE_ADDR} !~ /^(192.0.2.|198.51.100.|203.0.113.)/">
    [MOD_SECURITY CONFIG]
</If>

Regards

David Newman


On 23/07/2025 17:48, John Salter wrote:
CAUTION: This e-mail originated outside the University of Southampton.
CAUTION: This e-mail originated outside the University of Southampton.

Hi David,
This looks like some good stuff!

One thing that might be worth considering is a whitelist of IP addresses e.g. a campus-network – so ‘internal’ users aren’t bitten by this in their line of work.


I’ve started putting a ‘Bots’ wiki page together – are you happy I include the approach below on there?

I was also going to include some of the other contributions to this list – how to analyse web logs etc. – with attribution to the list – if people are happy with that?

 

Cheers,

John

 

 

From: eprints-tech-request@ecs.soton.ac.uk <eprints-tech-request@ecs.soton.ac.uk> On Behalf Of David R Newman
Sent: 23 July 2025 15:13
To: eprints-tech@ecs.soton.ac.uk
Subject: Re: [EP-tech] DDoS on simple and advanced search

 

CAUTION: External Message. Use caution opening links and attachments.


Hi John, Tomasz and others,

I have been reviewing how to get mod_security to do what I want, which is to start 429-ing (Too Many Requests) any service that is getting hammered by bots.  As we know, this is a DDoS, so the rules need to cover all rather than individual IP addresses.  This is not ideal, as it could block genuine requests. So it is important to get the depreciation on the counters right, to only 429 when not doing so would take the repository offline (or at best make it very unresponsive), whilst not leaving services (e.g. search, export and exportview) perpetually unavailable. 

I believe the following configuration (to /etc/httpd/modsecurity.d/local_rules/modsecurity_localrules.conf on a RHEL 8/9 or similar Linux OS) is hopefully following that middle ground but depending on the resources your EPrints repository server has you may want to tweak some of these parameters:

<LocationMatch "^/cgi/search/(archive/)?(advanced|simple)">
  SecAction id:210001,initcol:ip=0.0.0.0,pass,nolog
  SecRule REQUEST_URI "^/cgi/" "id:210002,phase:2,nolog,setvar:ip.searchcntr=+1,deprecatevar:ip.searchcntr=20/12"
  SecRule IP:SEARCHCNTR "@gt 100" "phase:2,id:210003,deny,status:429,setenv:RATELIMITED,skip:1,log"
  Header always set Retry-After "12" env=RATELIMITED
</LocationMatch>

<LocationMatch "^/cgi/export/">
  SecAction id:210011,initcol:ip=0.0.0.0,pass,nolog
  SecRule REQUEST_URI "^/cgi/" "id:210012,phase:2,nolog,setvar:ip.expcntr=+1,deprecatevar:ip.expcntr=20/12"
  SecRule IP:EXPCNTR "@gt 100" "phase:2,id:210013,deny,status:429,setenv:RATELIMITED,skip:1,log"
  Header always set Retry-After "12" env=RATELIMITED
</LocationMatch>

<LocationMatch "^/cgi/exportview/">
  SecAction id:210021,initcol:ip=0.0.0.0,pass,nolog
  SecRule REQUEST_URI "^/cgi/" "id:210022,phase:2,nolog,setvar:ip.expvcntr=+1,deprecatevar:ip.expvcntr=20/12"
  SecRule IP:EXPVCNTR "@gt 100" "phase:2,id:210023,deny,status:429,setenv:RATELIMITED,skip:1,log"
  Header always set Retry-After "12" env=RATELIMITED
</LocationMatch>

Technically the LocationMatch is not needed as the SecRule REQUEST_URI could do this job.  However, for readability and debugging using the LocationMatch keeps things tidier, if not necessarily normal practice.  I am not experienced with potential nuances in SecRule regexps, so I kept this simple with just "^/cgi/".   As you need to make this a SecRule and have something to match against otherwise mod_security throws the following warning message:

ModSecurity: Warning. Unconditional match in SecAction.

As you can see there are separate rule chains for search, export and exportview.  I have contemplated adding view as well but in theory that should be cached.  

The first line inside every location match is a bit of a hack.  You would normally do something like "initcol:ip=%{REMOTE_ADDR}" but as we want to match all IPs, due to the DDoS nature of the attacks, then I have gone for a placeholder of '0.0.0.0'.

The second line is just a perfunctory match but the remainder of the line is the interesting bit. setvar does the incrementing and deprecatevar does the depreciation of the counter over time, in this case 20 every 12 seconds.

The third line enforces the 429 when the number of request is greater than 100.  I have have also added the following line to my code and designed an EPrints' lang/en/static/rate_limited.xpage to display a message saying this service is "getting too many requests right now".

ErrorDocument 429 /rate_limited.html

The rest of the third line says that you should deny the request and send a 429 (Too Many Requests) response and set the environment RATELIMITED, which is just a useful hint to tell the client it is worth trying again 12 seconds later after the latest depreciation.  If you have mod_security logging switched on and at a suitable log level, you should be able to get logging for which requests and from what IP are getting 429-ed.  You could also grep the logs for 429 responses in the access logs, (although that can require a clever regexp to not match responses of size 429 bytes).

I have been running this on several of repositories today and although I have been still seeing high numbers of Apache processes / MySQL connections on occasion, I have not seen elevated CPU load or poor responsiveness on a monitoring check that I have setup to regularly request the homepage of these repositories and report back how long it takes to get a response.

This is only early days, so I expect some tweaking of the parameters will be needed and inclusion of extra paths, (e.g. /cgi/stats/... and possibly /views/...).  I will keep people post on my results and update the mod_security configuration I need to make.

Regards

David Newman

On 21/07/2025 10:21, John Salter wrote:

CAUTION: This e-mail originated outside the University of Southampton.

CAUTION: This e-mail originated outside the University of Southampton.

Hi All,

An aspect of the way EPrints deals with search requests may be causing us some of these issues.


What is happening:

  1. Something searches your site
  2. It extracts all the links from the results page – including the paginated links
  3. These links are saved, and at some point (weeks/months in the future) will be requested from a network of devices
  4. Each link contains both the cacheid of the original search, and all the parameters needed to re-run the search

 

When the paginated links are farmed out to the network of devices (hence the spread of IP addresses), the original EPrints search cache has expired.

Each paginated link then triggers the same original search to be run – with each request making a new cache table.

 

If the original search _expression_ returned 1000 results, presented with 20 links on each page, the follow-up crawl of those paginated links will cause 50 new individual cache tables.

 

I’ve documented it here: https://github.com/eprints/eprints3.4/issues/479 .

 

A quick short-term fix would be to stop EPrints auto-re-running a search if the search contained an old cache id.

This changes the current user experience, but I think would be better than systems becoming unresponsive.

 

NB the above has been observed using the ‘internal’ search methods (rather than Xapian/ElasticSearch).

 

Cheers,

John

 

From: eprints-tech-request@ecs.soton.ac.uk <eprints-tech-request@ecs.soton.ac.uk> On Behalf Of Yuri Carrer
Sent: 18 July 2025 07:24
To: eprints-tech@ecs.soton.ac.uk
Subject: Re: [EP-tech] DDoS on simple and advanced search

 

CAUTION: External Message. Use caution opening links and attachments.


CAUTION: This e-mail originated outside the University of Southampton.

CAUTION: This e-mail originated outside the University of Southampton.

Botnets doesn't use the same IPs.

 

The solution in simpler: rename the search script and update internal links (or let Eprints use some config for it). You can do it weekly, nobody will notice but bots won't be able to keep up with it.

 

Il 17/07/25 19:32, Tomasz Neugebauer ha scritto:

Any comments on that solution?  It seems elegant, if it works?

 

Tomasz

 

__

 
-- 
Yuri Carrer
 
 CAB - Centro di Ateneo per le Biblioteche, Università di Padova
 Tel: 049/827 9712 - Via Beato Pellegrino, 28 - Padova



*** Options: https://wiki.eprints.org/w/Eprints-tech_Mailing_List
*** Archive: https://www.eprints.org/tech.php/
*** EPrints community wiki: https://wiki.eprints.org/
 

*** Options: https://wiki.eprints.org/w/Eprints-tech_Mailing_List
*** Archive: https://www.eprints.org/tech.php/
*** EPrints community wiki: https://wiki.eprints.org/



*** Options: https://wiki.eprints.org/w/Eprints-tech_Mailing_List
*** Archive: https://www.eprints.org/tech.php/
*** EPrints community wiki: https://wiki.eprints.org/



*** Options: https://wiki.eprints.org/w/Eprints-tech_Mailing_List
*** Archive: https://www.eprints.org/tech.php/
*** EPrints community wiki: https://wiki.eprints.org/



*** Options: https://wiki.eprints.org/w/Eprints-tech_Mailing_List
*** Archive: https://www.eprints.org/tech.php/
*** EPrints community wiki: https://wiki.eprints.org/