[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[EP-tech] IRStats2: process_stats job with InnoDB



Thanks John. You're a life saver!

Now, where's that kill job..... :)

George

From: eprints-tech-bounces at ecs.soton.ac.uk [mailto:eprints-tech-bounces at ecs.soton.ac.uk] On Behalf Of John Salter
Sent: 16 May 2017 13:14
To: eprints-tech at ecs.soton.ac.uk
Subject: Re: [EP-tech] IRStats2: process_stats job with InnoDB

I think so - at least it did for us.
Using the transactions halved the time it took our stats to run.

>From memory, I *think* that I could have been more brutal with the transactions. I think there's a loop outside the begin/commit additions I made.
If the begin/commit were added outside this loop instead, there may be even bigger gains - but I was happy with halving the runtime!

Cheers,
John

From: eprints-tech-bounces at ecs.soton.ac.uk<mailto:eprints-tech-bounces at ecs.soton.ac.uk> [mailto:eprints-tech-bounces at ecs.soton.ac.uk] On Behalf Of George Macgregor
Sent: 16 May 2017 12:55
To: eprints-tech at ecs.soton.ac.uk<mailto:eprints-tech at ecs.soton.ac.uk>
Subject: Re: [EP-tech] IRStats2: process_stats job with InnoDB

Thanks as always, John!

So, making the updates to Handler.pm and Sets.pm should oil the wheels? I'm desperate to kill this current job!

Cheers

George

From: eprints-tech-bounces at ecs.soton.ac.uk<mailto:eprints-tech-bounces at ecs.soton.ac.uk> [mailto:eprints-tech-bounces at ecs.soton.ac.uk] On Behalf Of John Salter
Sent: 16 May 2017 12:22
To: eprints-tech at ecs.soton.ac.uk<mailto:eprints-tech at ecs.soton.ac.uk>
Subject: Re: [EP-tech] IRStats2: process_stats job with InnoDB

Hi George,
See: https://github.com/eprints/irstats2/pull/84

NB There are two forks of IRStats2:

-          https://github.com/eprints/irstats2/

-          https://github.com/eprintsug/irstats2/
I'm not sure which is the most up-to-date. Can anyone comment on this?
Should we standardise on one of them?

Cheers,
John

From: eprints-tech-bounces at ecs.soton.ac.uk<mailto:eprints-tech-bounces at ecs.soton.ac.uk> [mailto:eprints-tech-bounces at ecs.soton.ac.uk] On Behalf Of George Macgregor
Sent: 16 May 2017 12:10
To: eprints-tech at ecs.soton.ac.uk<mailto:eprints-tech at ecs.soton.ac.uk>
Subject: [EP-tech] IRStats2: process_stats job with InnoDB

Hello colleagues!

I am looking for some IRStats2 insights. We have recently shifted our MySQL DB to InnoDB from MyISAM. I am in the process of running the process_stats job for IRStats2 in order to reflect local changes but the job is taking an aeon - and significantly longer than it ever did under MyISAM.

I was informed that it would run more slowing with InnoDB but I had no idea it would be quite so bad. So far it has been running for the best part of a week, which has been impacting on end users (e.g. sporadic unavailability of EPrints, slow responses, etc.). It is so bad that I am almost tempted to kill the job, even though it has been running for the best part of a week.

Has a similar issue affected others and, if so, do you have any tips for speeding up the processing? Or do I simply need to take the hit?!

Cheers

George
--
George Macgregor
Information Services Directorate | University of Strathclyde
Curran Building | 101 St James Road
Glasgow G4 ONS
Tel: 0141 548 3496
Email: george.macgregor at strath.ac.uk<mailto:george.macgregor at strath.ac.uk>
Web: https://purl.org/g3om4c
Twitter: @g3om4c<https://twitter.com/g3om4c>
[iD icon]orcid.org/0000-0002-8482-3973<http://orcid.org/0000-0002-8482-3973>
--
Ensure that your research outputs are eligible for submission in the next REF.  Authors' accepted manuscripts of journal articles and conference proceedings accepted after 01 April 2016 must be deposited in PURE as soon as possible after acceptance for publication - email openaccess at strath.ac.uk<mailto:openaccess at strath.ac.uk> for further information or visit http://www.strath.ac.uk/openaccess.

[twitter logo]<https://twitter.com/StrathclydeOA>
--
The University of Strathclyde is a charitable body, registered in Scotland, with registration number SC015263

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20170516/ad292054/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 1261 bytes
Desc: image001.png
Url : http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20170516/ad292054/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 2196 bytes
Desc: image002.png
Url : http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20170516/ad292054/attachment-0003.png