EPrints Technical Mailing List Archive

Message: #05620

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Antwort: Re: Problems with view generation: EPrints System Error

Hi Verena,

did you try out the --generate menus  option of generate_views? This reduces the number of files considerably.

Also, in our nightly cron job, we delete all files that are older than 24 hours.
 
Best regards,

Martin

--
Dr. Martin Brändle
Zentrale Informatik
Universität Zürich
Stampfenbachstr. 73
CH-8006 Zürich

mail: martin.braendle@id.uzh.ch
phone: +41 44 63 56705
fax: +41 44 63 54505
http://www.zi.uzh.ch

Inactive hide details for Verena Mattes ---27/04/2016 10:54:23---Hi Adam, For now, my colleagues in the IT service centre solveVerena Mattes ---27/04/2016 10:54:23---Hi Adam, For now, my colleagues in the IT service centre solved the problem by

Von: Verena Mattes <verena.mattes@ub.uni-bayreuth.de>
An: eprints-tech@ecs.soton.ac.uk
Datum: 27/04/2016 10:54
Betreff: Re: [EP-tech] Problems with view generation: EPrints System Error
Gesendet von: eprints-tech-bounces@ecs.soton.ac.uk




Hi Adam,

For now, my colleagues in the IT service centre solved the problem by
raising the number of files permitted in one directory on our Netapp,
but that is only a temporary solution. Almost 340.000 files in one
directory are just too many, so I'll have to find a way to split the
person view into groups somehow.

Your suggestion of checking the number of variations should be a first
step in reducing the number of files. Here's our current view
configuration, which I'm going to change by taking away the DEFAULT
variation:

>        {
>                 id => "person",
>                 hideempty => 0,
>                 allow_null => 0,
>                 menus => [
>                         {
>                                 fields => [ "person_view_name" ],
>                                 mode => "sections",
>                                 grouping_function => "EPrints::Update::Views::group_by_first_character",
>                                 group_range_function => "EPrints::Update::Views::cluster_ranges_40",
>                                 new_column_at => [40],
>                                 open_first_section => 1,
>                         },
>                         ],
>                 order => "-date;res=year/title/publication/book_title",
>                 hideup => 0,
>                 nocount => 0,
>                 notimestamp => 0,
>                 include => 1,
>                 variations => [
>                         "type",
>                         "date;truncate=4,reverse",
>                         "DEFAULT",
>                 ],
>                 citation => "view",            # Views mit Volltext-Hinweis!
>         },

Thanks for all your help!

Verena

>
> Message: 1
> Date: Tue, 26 Apr 2016 08:35:49 +0000
> From: Adam Field <Adam.Field@jisc.ac.uk>
> Subject: Re: [EP-tech] Problems with view generation: EPrints System
> Error
> To: "eprints-tech@ecs.soton.ac.uk" <eprints-tech@ecs.soton.ac.uk>
> Message-ID: <C2A1398C-702E-4C79-B4CD-2BAD5987789F@jisc.ac.uk>
> Content-Type: text/plain; charset="utf-8"
>
> My previous suggestion wasn't right and won't work exactly as expected on reflection.
>
> What happens when you run generate_views from the command line?
> Just how many files do you have in the directory on the hard disk?
>
> Can you paste in the browse view configuration?  It may be that turning of variations will solve this.
>
> [Jisc]<http://www.jisc.ac.uk/>
>
> Adam Field
> SHERPA services analyst developer
>
>
> From: <eprints-tech-bounces@ecs.soton.ac.uk<mailto:eprints-tech-bounces@ecs.soton.ac.uk>> on behalf of Verena Mattes <verena.mattes@ub.uni-bayreuth.de<mailto:verena.mattes@ub.uni-bayreuth.de>>
> Reply-To: "eprints-tech@ecs.soton.ac.uk<mailto:eprints-tech@ecs.soton.ac.uk>" <eprints-tech@ecs.soton.ac.uk<mailto:eprints-tech@ecs.soton.ac.uk>>
> Date: Tuesday, 26 April 2016 07:48
> To: "eprints-tech@ecs.soton.ac.uk<mailto:eprints-tech@ecs.soton.ac.uk>" <eprints-tech@ecs.soton.ac.uk<mailto:eprints-tech@ecs.soton.ac.uk>>
> Subject: Re: [EP-tech] Problems with view generation: EPrints System Error
>
> Hi Alan and Adam,
>
> Thanks for your suggestions. We are running EPrints 3.3.11 and the free
> space on our hard disk is not a problem. I checked with a colleague and
> he confirmed my guess that the problem is caused by the number of files
> in the directory, obviously the maximum number was reached last week and
> no further files can be created.
> Does anybody have a quick idea on how to divide files for one view, the
> person view, to different directories? Would it be possible to define
> different views for groups of letters, e.g. A-D or something like that?
>
> Thanks!
>
> Verena
>
>
>
> Date: Mon, 25 Apr 2016 10:58:47 +0000
> From: Adam Field <Adam.Field@jisc.ac.uk<mailto:Adam.Field@jisc.ac.uk>>
> Subject: Re: [EP-tech] Problems with view generation: EPrints System
> Error
> To: "eprints-tech@ecs.soton.ac.uk<mailto:eprints-tech@ecs.soton.ac.uk>" <eprints-tech@ecs.soton.ac.uk<mailto:eprints-tech@ecs.soton.ac.uk>>
> Message-ID: <3D933692-5E6D-4B33-9499-1DD23E2B4C62@jisc.ac.uk<3D933692-5E6D-4B33-9499-1DD23E2B4C62@jisc.ac.uk">mailto:3D933692-5E6D-4B33-9499-1DD23E2B4C62@jisc.ac.uk>>
> Content-Type: text/plain; charset="utf-8"
>
> What specific version of EPrints are you running?  What is line 1550 of /usr/share/eprints3/perl_lib/EPrints/Update/Views.pm
>
> Also, silly question, but how much free space do you have on your hard disk?
>
> [Jisc]<http://www.jisc.ac.uk/>
>
> Adam Field
> SHERPA services analyst developer
>
>
> From: <eprints-tech-bounces@ecs.soton.ac.uk<mailto:eprints-tech-bounces@ecs.soton.ac.uk><mailto:eprints-tech-bounces@ecs.soton.ac.uk>> on behalf of Verena Mattes <verena.mattes@ub.uni-bayreuth.de<mailto:verena.mattes@ub.uni-bayreuth.de><mailto:verena.mattes@ub.uni-bayreuth.de>>
> Reply-To: "eprints-tech@ecs.soton.ac.uk<mailto:eprints-tech@ecs.soton.ac.uk><mailto:eprints-tech@ecs.soton.ac.uk>" <eprints-tech@ecs.soton.ac.uk<mailto:eprints-tech@ecs.soton.ac.uk><mailto:eprints-tech@ecs.soton.ac.uk>>
> Date: Monday, 25 April 2016 09:51
> To: "eprints-tech@ecs.soton.ac.uk<mailto:eprints-tech@ecs.soton.ac.uk><mailto:eprints-tech@ecs.soton.ac.uk>" <eprints-tech@ecs.soton.ac.uk<mailto:eprints-tech@ecs.soton.ac.uk><mailto:eprints-tech@ecs.soton.ac.uk>>
> Subject: [EP-tech] Problems with view generation: EPrints System Error
>
> Hello,
>
> since last week, we've had problems with the view generation for our
> author/person view. These problems specifically concern the views for
> new author names, which are not generated, while the views for already
> existing author names are updated.
>
> For each author name concerned, there is an entry in the apache error log:
>
> Error writing to /usr/share/eprints3/archives/ubt_eref/html/de/view/person/Kaiser=3AMario=3A=3A.export: File too large
> ------------------------------------------------------------------
>      at /usr/share/eprints3/perl_lib/EPrints/Update/Views.pm line 1550
>             EPrints::Update::Views::output_files('EPrints::Repository=HASH(0x7fb0663688d0)', '/usr/share/eprints3/archives/ubt_eref/html/de/view/person/Kai...', 'XML::LibXML::DocumentFragment=SCALAR(0x7fb06a7e5938)', '/usr/share/eprints3/archives/ubt_eref/html/de/view/person/Kai...', 'XML::LibXML::Element=SCALAR(0x7fb06a45f160)', '/usr/share/eprints3/archives/ubt_eref/html/de/view/person/Kai...', 'XML::LibXML::DocumentFragment=SCALAR(0x7fb06a7e5938)', '/usr/share/eprints3/archives/ubt_eref/html/de/view/person/Kai...', 'XML::LibXML::DocumentFragment=SCALAR(0x7fb06a668aa8)', ...) called at /usr/share/eprints3/perl_lib/EPrints/Update/Views.pm line 935
>             EPrints::Update::Views::update_view_list('EPrints::Repository=HASH(0x7fb0663688d0)', '/usr/share/eprints3/archives/ubt_eref/html/de/view/person/Kai...', 'de', 'EPrints::Update::Views=HASH(0x7fb06a6863f0)', 'ARRAY(0x7fb064b78c70)') called at /usr/share/eprints3/perl_lib/EPrints/Update/Views.pm line 259
>             EPrints::Update::Views::update_view_file('EPrints::Repository=HASH(0x7fb0663688d0)', 'de', '/view/person/Kaiser=3AMario=3A=3A.html', '/view/person/Kaiser=3AMario=3A=3A.html') called at /usr/share/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 513
>             EPrints::Apache::Rewrite::handler('Apache2::RequestRec=SCALAR(0x7fb06a471710)') called at -e line 0
>             eval {...} called at -e line 0
>
> In this specific case, the author's name is linked to only 2 EPrints
> entries, so it is unlikely that this file would be particularly large.
>
> Does anybody have an idea concerning this? I would appreciate any help.
>
> Thanks!
>
> Verena
>
> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: http://www.eprints.org/tech.php/
> *** EPrints community wiki: http://wiki.eprints.org/
> *** EPrints developers Forum: http://forum.eprints.org/
>
> Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc?s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.
>
> Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20160426/87b9ab89/attachment.html
> -------------- next part --------------

*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/
*** EPrints developers Forum: http://forum.eprints.org/