[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[EP-tech] IRstats2 "origin of downloads"



Hi Tomasz,

In fact, I think I found an issue even with that amended code in one 
scenario, so in a development version of IRStats2 are changed that to:

 ? ? foreach my $pkg ( 'Geo::IP', 'Geo::IP::PurePerl' )
 ?? ?{
 ?? ???? if( EPrints::Utils::require_if_exists( $pkg ) )
 ?? ???? {
 ?? ???? ??? if( $pkg !~ /PurePerl/ )
 ?? ???? ??? {
 ?? ???? ??? ??? $self->{geoip} = $pkg->new( $dat_file ) if $dat_file eq 
'1';
 ?? ???? ??? ??? $self->{geoip} = $pkg->open( $dat_file ) if $dat_file 
ne '1';
 ?? ???? ??? }
 ?? ???? ??? else
 ?? ???? ??? {
 ?? ???? ??? ??? $self->{geoip} = $pkg->new( $dat_file );
 ?? ???? ??? }
 ?? ???? ??? last if( defined $self->{geoip} );
 ?? ???? }
 ?? ?}

Regards

David Newman

On 20/04/2023 11:23 pm, David R Newman via Eprints-tech wrote:
> Hi Tomasz,
>
> Ah, now you say this that rings a bell.? We have been looking at new 
> release of IRstats2 and remember that there was a (pretty old) change 
> since the last release that that uses ->new for one of the GeoIP 
> libraries and ->open for the other:
>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feprints%2Firstats2%2Fcommit%2Fa84f22bf6d8a7faa9b6593afa97ef1a0cd360fcc&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C99f15a4056524e4d22b908db423a6393%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638176590786071565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=onw05culW80jEJ2pWwGrUEOlDJ9I6f9L2j%2F3Ejxddz0%3D&reserved=0
>
> Regards
>
> David Newman
>
>
> On 20/04/2023 11:03 pm, Tomasz Neugebauer wrote:
>> *CAUTION:* This e-mail originated outside the University of Southampton.
>>
>> Hm? I think that one thing that might have happened here is that when 
>> I upgraded to 3.4, I removed the GeoIP.dat file from the ?lib? 
>> folder, thinking that the library is supposed to use ?the global one? 
>> instead.? I?m talking about this part of the code:
>>
>> my $dat_file = $self->{session}->config( "lib_path").'/geoip/GeoIP.dat';
>>
>> # alternatively use the global one
>> $dat_file = 1 if( !-e $dat_file );
>>
>> However, now that I look at this code, I find myself really 
>> confused!? $dat_file is set to 1 if the path doesn?t exist?! ?Let?s 
>> just say that doesn?t look solid to me. What is setting that path to 
>> 1 supposed to accomplish? The loop later on, I needed to use the 
>> ?open? command if I specify a path, not the ?new? command here:
>>
>> $self->{geoip} = $pkg->open( $dat_file );
>>
>> However, if that path in lib doesn?t exist, I?m opening ?1?, whatever 
>> that means ? so that fails with an error; ?new? works on it, but not 
>> sure what it does, then it defaults to the global one? I think that?s 
>> where possibly things are just failing.? When the file is there in 
>> the lib folder, it needs to be ?open? command, not new, but when the 
>> file is not there, that?s when it?s ?new? with that magic number ?1??
>>
>> Tomasz
>>
>> *From:*David R Newman <drn at ecs.soton.ac.uk>
>> *Sent:* April 19, 2023 3:23 PM
>> *To:* Tomasz Neugebauer <Tomasz.Neugebauer at concordia.ca>; 
>> eprints-tech at ecs.soton.ac.uk
>> *Subject:* Re: [EP-tech] IRstats2 "origin of downloads"
>>
>> *Attention* This email originates from outside the concordia.ca 
>> domain. // Ce courriel provient de l'ext?rieur du domaine de concordia.ca
>>
>> Hi Tomasz,
>>
>> Another possibility is that the irstats2_countries table has crashed 
>> an is therefore no longer updated.? However, I think you would have 
>> likely noticed this.? As I suspect it would have prevented other 
>> parts of IRStats2 updating, as one day the irstats2_internal table 
>> will retain the flag saying stats are still being updated and then 
>> subsequent days the process_stats would not be able to run.? However, 
>> maybe a crash table does not cause process_stats to fail without 
>> clearing the update flag.
>>
>> Otherwise, there is various checks in the code that will stop country 
>> or origin data being generated.? Here is the whole block of code.? 
>> Hopefully, you can work out from this whether something on your 
>> repository server does not meet the required criteria:
>>
>> # if possible, use the GeoIP data file shipped with EPrints
>> ?? ?my $dat_file = $self->{session}->config( 
>> "lib_path").'/geoip/GeoIP.dat';
>>
>> ?? ?# alternatively use the global one
>> ?? ?$dat_file = 1 if( !-e $dat_file );
>>
>> ?? ?#Test Geo::IP first - it's faster!
>> ?? ?foreach my $pkg ( 'Geo::IP', 'Geo::IP::PurePerl' )
>> ?? ?{
>> ?? ???? if( EPrints::Utils::require_if_exists( $pkg ) )
>> ?? ???? {
>> ?? ???? ??? $self->{geoip} = $pkg->new( $dat_file );
>> ?? ???? ??? last if( defined $self->{geoip} );
>> ?? ???? }
>> ?? ?}
>>
>> ?? ?if( !defined $self->{geoip} )
>> ?? ?{
>> ?? ???? $self->{advertise} = 0;
>> ?? ???? $self->{disable} = 1;
>> ?? ???? $self->{error} = "Failed to load required module for 
>> Processor::Access::Country. Country information will not be available.";
>> ?? ???? return $self;
>> ?? ?}
>>
>> This code is in the file 
>> EPRINS_PATH/plugins/EPrints/Plugin/Stats/Processor/Access/Country.pm 
>> so if you cannot work out whether the criteria is meet, you could try 
>> adding some debug and then running process_stats (or wait for it to 
>> run overnight).? If this was failing I would hope that wherever you 
>> are logging the output (STDOUT and STDERR) fro process_stats would 
>> show the $self->{error} message above.
>>
>> Regards
>>
>> David Newman
>>
>> On 19/04/2023 5:33 pm, Tomasz Neugebauer wrote:
>>
>>     *CAUTION:*This e-mail originated outside the University of
>>     Southampton.
>>
>>     Hi David,
>>
>>     Thanks for that, but I do believe that we have those packages
>>     installed. We are running on Ubuntu 18.04 if that makes a difference.
>>
>>     Here is some output from apt:
>>
>>     ?apt list | grep geoip-database
>>
>>         geoip-database/bionic,now 20180315-1 all [installed]
>>
>>         geoip-database-contrib/bionic 1.19 all
>>
>>         geoip-database-extra/bionic,now 20180315-1 all
>>         [installed,automatic]
>>
>>     ?apt list | grep libgeo-ip-perl
>>
>>     ??????libgeo-ip-perl/bionic,now 1.51-1 amd64 [installed]
>>
>>     Tomasz
>>
>>     ------------------------------------------------------------------------
>>
>>     *From:*David R Newman <drn at ecs.soton.ac.uk>
>>     <mailto:drn at ecs.soton.ac.uk>
>>     *Sent:* Wednesday, April 19, 2023 5:42 AM
>>     *To:* eprints-tech at ecs.soton.ac.uk <eprints-tech at ecs.soton.ac.uk>
>>     <mailto:eprints-tech at ecs.soton.ac.uk>; Tomasz Neugebauer
>>     <Tomasz.Neugebauer at concordia.ca>
>>     <mailto:Tomasz.Neugebauer at concordia.ca>
>>     *Subject:* Re: [EP-tech] IRstats2 "origin of downloads"
>>
>>     *Attention*This email originates from outside the concordia.ca
>>     domain. // Ce courriel provient de l'ext?rieur du domaine de
>>     concordia.ca
>>
>>     Hi Tomasz,
>>
>>     You will need an operating system package and a Perl library
>>     (which can probably be installed as an OS package) for that to
>>     work.? On Ubuntu 20.04 that is:
>>
>>     apt install geoip-database libgeo-ip-perl
>>
>>     On Enterprise Linux (RHEL, CentOS, Fedora, Rocky Linux, Alma
>>     Linux, etc.) it is:
>>
>>     yum install GeoIP perl-Geo-IP
>>
>>     I do not think you will be able to get back missing country of
>>     origin data without regenerating your stats from scratch but this
>>     should start being added in future.
>>
>>     Regards
>>
>>     David Newman
>>
>>     On 18/04/2023 21:29, Tomasz Neugebauer via Eprints-tech wrote:
>>
>>         *CAUTION:*This e-mail originated outside the University of
>>         Southampton.
>>
>>         I?m trying to figure out why the ?origin of downloads?
>>         country data has stopped working for our repository.
>>
>>         I see much of the old (pre-2022) data is still there, but
>>         starting in 2022, ?origin of downloads? (the map
>>         visualization) contains no data.
>>
>>         Any idea as to what could be causing that or where to look
>>         for the issue?
>>
>>         It?s not just the visualization of the map, because that
>>         still works for ?all? and earlier date ranges.
>>
>>         It looks like the country data is not there starting in 2022?
>>
>>         Tomasz
>>
>>
>>
>>
>>         *** Options:
>>         http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech  <http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech>
>>
>>         *** Archive:https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C99f15a4056524e4d22b908db423a6393%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638176590786071565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vlH9TZTiO2h6E72Q3PQa6gRpawM9vTO4dOVMOvObZNg%3D&reserved=0  <https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C99f15a4056524e4d22b908db423a6393%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638176590786071565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vlH9TZTiO2h6E72Q3PQa6gRpawM9vTO4dOVMOvObZNg%3D&reserved=0>
>>
>>         *** EPrints community wiki:https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C99f15a4056524e4d22b908db423a6393%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638176590786071565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hIzYzxYsURmOfSKc3k4%2BGgZSq8t3AZaFfkm8vvgtM6s%3D&reserved=0  <https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C99f15a4056524e4d22b908db423a6393%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638176590786071565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hIzYzxYsURmOfSKc3k4%2BGgZSq8t3AZaFfkm8vvgtM6s%3D&reserved=0>
>>
>
>
> *** Options:http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive:https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C99f15a4056524e4d22b908db423a6393%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638176590786071565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vlH9TZTiO2h6E72Q3PQa6gRpawM9vTO4dOVMOvObZNg%3D&reserved=0
> *** EPrints community wiki:https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C99f15a4056524e4d22b908db423a6393%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638176590786071565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hIzYzxYsURmOfSKc3k4%2BGgZSq8t3AZaFfkm8vvgtM6s%3D&reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20230421/23b35c0c/attachment-0001.html