[index] [prev] [next] [options] [help]
See the Contact page for how to subscribe and unsubscribe.

eprints_tech messages

Please note: this page shows emails that have been sent to the eprints_tech mailing list. Some of these may be spam emails we have failed to filter.

Re: [EP-tech] Antwort: Re: Antwort: IRStats2: Innodb tables; forks

From: John Salter <J.Salter AT leeds.ac.uk>
Date: Wed, 21 Dec 2016 11:06:26 +0000




Hi Martin,
Just a note: I would not expect any change to the MyISAM timings - as MyISAM 
doesn't support transactions - so the inserts would just happen as they do now.

Cheers,
John


From: eprints-tech-bounces AT ecs.soton.ac.uk [mailto:eprints-tech-bounces AT 
ecs.soton.ac.uk] On Behalf Of martin.braendle AT id.uzh.ch
Sent: 21 December 2016 09:11
To: eprints-tech AT ecs.soton.ac.uk
Subject: [EP-tech] Antwort: Re: Antwort: IRStats2: Innodb tables; forks


Hi John,

just to report back: the IRStats2 update on your InnoDB test server went down 
from 17 hours to about 4 hours. Let's see what effect your changes will have on 
the MyISAM production system.

Thank you,

Martin

--
Dr. Martin Brändle
Zentrale Informatik
Universität Zürich
Stampfenbachstr. 73
CH-8006 Zürich


[Inactive hide details for John Salter ---16/12/2016 15:56:48---Hi Martin, The 
changes in https://github.com/eprints/irstats2/pu]John Salter ---16/12/2016 
15:56:48---Hi Martin, The changes in 
https://github.com/eprints/irstats2/pull/84 seem to help us quite a bit. O

Von: John Salter <J.Salter AT leeds.ac.uk<mailto:J.Salter AT 
leeds.ac.uk>>
An: "eprints-tech AT ecs.soton.ac.uk<mailto:eprints-tech AT 
ecs.soton.ac.uk>" <eprints-tech AT 
ecs.soton.ac.uk<mailto:eprints-tech AT ecs.soton.ac.uk>>
Datum: 16/12/2016 15:56
Betreff: Re: [EP-tech] Antwort:  IRStats2: Innodb tables; forks
Gesendet von: eprints-tech-bounces AT 
ecs.soton.ac.uk<mailto:eprints-tech-bounces AT ecs.soton.ac.uk>

________________________________



Hi Martin,
The changes in https://github.com/eprints/irstats2/pull/84 seem to help us 
quite a bit. Out daily stats start run at 06:30, and were still running at 
12:00.
Implementing this fix yesterday (and with two days of data to process), the 
stats completed processing just after 09:00.

Some of the tables (details in https://github.com/eprints/irstats2/issues/83) 
are re-created each time the stats are processed.
These tables may be using InnoDB as the engine isn't specified in the CREATE 
TABLE statements (if you're running MySQL >= v5.5.5 the default engine is 
InnoDB).

The structure of these tables means that the InnoDB engine has to do a lot of 
work (maintaining the index? moving data?) for each insert or transaction. If 
the inserts are in a few transactions, there's a lot less work to be done (so 
it's quicker!).

I'll continue to monitor how quickly our stats process - it looks like it helps 
us, but it might not help everyone.

Cheers,
John

From: eprints-tech-bounces AT ecs.soton.ac.uk<mailto:eprints-tech-bounces AT 
ecs.soton.ac.uk> [mailto:eprints-tech-bounces AT ecs.soton.ac.uk] On Behalf 
Of martin.braendle AT id.uzh.ch<mailto:martin.braendle AT id.uzh.ch>
Sent: 16 December 2016 13:49
To: eprints-tech AT ecs.soton.ac.uk<mailto:eprints-tech AT 
ecs.soton.ac.uk>
Subject: [EP-tech] Antwort: IRStats2: Innodb tables; forks


Hi John,

we migrated our test system to InnoDB about a year ago, did some tuning using 
the MySQL performance tuning primer script, and carried out some performance 
tests with the Apache Benchmark tool on both the test and production system.

Result: For sequential accesses, response times with InnoDB were 930-960 ms, 
with MyISAM they were 700ms. For concurrent accesses, response times dropped to 
250ms on average with InnoDB. But you need many concurrent accesses to achieve 
this number. Also, when Web browsing, the InnoDB test system felt sluggish 
compared to the MyISAM production system.
IRStats2 weekly update runs take about 10 hours on the MyISAM system, and 17 
hours on the InnoDB system, although there are more updates to process on the 
production system.  (I see that you have added transactions to IRStats2 on 
GitHub, we will check whether this helps).

To bring InnoDB to fly, you would need a huge innodb_buffer_pool_size, around 
the total size of your databases (which in our case is around 30-40 GB), 
requiring a hefty DB server.

Since we were not sure, in as much the current EPrints version supports InnoDB 
transactions, we decided to stay with MyISAM on the production system for the 
moment.

Cheers,

Martin


[Inactive hide details for John Salter ---13/12/2016 12:59:06---Hi, Our 
IRStats2 processing runs slowly, and I suspect there mig]John Salter 
---13/12/2016 12:59:06---Hi, Our IRStats2 processing runs slowly, and I suspect 
there might be some improvements to be made -

Von: John Salter <J.Salter AT leeds.ac.uk<mailto:J.Salter AT 
leeds.ac.uk>>
An: "'eprints-tech AT ecs.soton.ac.uk'" <eprints-tech AT 
ecs.soton.ac.uk<mailto:eprints-tech AT ecs.soton.ac.uk>>
Datum: 13/12/2016 12:59
Betreff: [EP-tech] IRStats2: Innodb tables; forks
Gesendet von: eprints-tech-bounces AT 
ecs.soton.ac.uk<mailto:eprints-tech-bounces AT ecs.soton.ac.uk>

________________________________




Hi,
Our IRStats2 processing runs slowly, and I suspect there might be some 
improvements to be made - especially as we're using InnoDB tables.
Before I start trying to work out how to solve this, has anyone done any work 
in this area?

Also, there are two forks of IRStats2 that might need to be merged:
https://github.com/eprints/irstats2
https://github.com/eprintsug/irstats2

(Most other forks come from the head of eprints/irstats2).

Comparing these two forks:
https://github.com/eprints/irstats2/compare/master...eprintsug:master
it looks like there has been some re-arrangement of files (these seem sensible 
to me), and a couple of fixes.

Does anyone have any thoughts on whether these should be merged, so we have one 
main master again? Will doing this cause problems for people who have installed 
the package using the previous file-layout?

Cheers,
John*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/
*** EPrints developers Forum: http://forum.eprints.org/
*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/
*** EPrints developers Forum: http://forum.eprints.org/


ATTACHMENT: message.html!

*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech *** Archive: http://www.eprints.org/tech.php/ *** EPrints community wiki: http://wiki.eprints.org/ *** EPrints developers Forum: http://forum.eprints.org/

[index] [prev] [next] [options] [help]