EPrints Technical Mailing List Archive

Message: #07878

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] generate_views

On Wed, 5 Jun 2019 at 19:00, Matthew Kerwin <matthew@kerwin.net.au> wrote:
> We've put a lot of energy into this problem over the years. I've
> variously: overridden the default Search generation for some
> problematic views with custom SQL queries, and added a wrapper around
> the generate_views script that runs four instances in parallel (one
> for each core on our server).  It still takes a couple of hours every
> week, but it's no longer 12+ hours.
> I'm not at work right now to get my hands on the code.  I'll see if I
> can get to it tomorrow.
> Cheers

For those playing along at home, here's the script we use to run the
generation in parallel:


The parallelisation is a bit magical, and requires you to specify the
`MAX_PARALLEL_CHILDREN=x` environment variable.  I've tried to
sanitise it of all our particular quirks, but might have missed

  Matthew Kerwin