EPrints Technical Mailing List Archive

Message: #05838


< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Seeing unusually high downloads in IRStats


Forwarding from JISC-REPOSITORIES list – we’ve been seeing this in California too, and our IRStats2 counts are through the roof for the last couple of weeks.

 

Can anyone tell me how to filter out these robots in IRStats2?  And how to clean the access file so that our irstats2 reports are not distorted by this deluge?  I assume I’d want to delete all entries with a requester_id in the table below and rerun IRstats2 setup from scratch.

 

Thanks,

Betsy Coles

Caltech – Digital Library Development

bcoles@caltech.edu

 

From: Repositories discussion list [mailto:JISC-REPOSITORIES@JISCMAIL.AC.UK] On Behalf Of Hilary Jones
Sent: Friday, July 15, 2016 3:43 AM
To: JISC-REPOSITORIES@JISCMAIL.AC.UK
Subject: Seeing unusually high downloads in IRStats - IRUS-UK's explanation and why this isn't affecting IRUS-UK stats

 

Hi everyone,

 

There was a discussion, via UKCORR mailing list, on why there are exceptionally high downloads being seen this week in IRStats and what might be causing it.

 

After some investigation we have found that the unusually high downloads are down to four IP ranges:

 

IP range

Organisation

Location

No. IP addresses

103.25.156.*

Microsoft Bingbot

China

128

103.36.96.*

Microsoft Corporation

China

216

111.221.28.*

Microsoft Bingbot

China

256

202.89.235.*

Microsoft Bingbot

China

80

 

These IPs have been systematically trawling and downloading files from many UK repositories. Looking at their User Agent strings they do not declare themselves as bots but masquerade as normal users.

 

Happily, the IRUS-UK ingest has been filtering out these robotic downloads, so you won’t see a massive spike in your IRUS-UK stats.

 

We hope this is of help.

 

Best wishes

Hilary

 

Jisc

Hilary Jones
Services and Projects Support

0161 413 7541
Skype hilary.jones@jisc.ac.uk
Twitter @JonesHilaryJ
6th Floor Churchgate House, 56 Oxford Street, Manchester, M1  6EU

jisc.ac.uk

Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 882 5529 90. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. jisc.ac.uk