EPrints Technical Mailing List Archive

Message: #02865

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Re: Thousands of old eprints repropagated via OAI after epadmin redo_thumbnails &co.

Hola!

On 09/04/14 10:31, Florian Heß wrote:
Am 08.04.2014 11:57, schrieb Sebastien Francois:

- recommit: by definition, this action should touch lastmod

Hi Sebastien,

I am afraid, I disagree here partly. Recommits should touch lastmod only 
*if* there are dirty substantial = user-editable metadata columns. This 
admittedly is difficult to decide by epadmin recommit tool as the 
changes often have taken place directly in advance, bypassing the API, 
for this I assume is the main purpose of that tool. Hence, what about 
--non-volatile-change alias --no-touch-lastmod switches (and/or their 
respective positive counterparts) to epadmin recommit and alike?

Fair point!...

So when someone runs epadmin recommit, the script forces the recommit ($dataobj->commit( 1 )) which *always* writes data to the database, regardless of whether the data-obj was modified during the re-commit.

So if I understand correctly your case-studies, it seems like you need:

1- epadmin recommit
2- epadmin force-recommit

v1 could do $dataobj->commit() -> no change in the dataobj = no DB write, no update of lastmod, revision, history. Change in the dataobj => lastmod, revision, history updated

v2 could do $dataobj->commit( 1 ) -> same as the current behaviour

Look, there are so many actions an eprint commit trigger (e.g. 
/cfg.d/eprint_fields_automatic.pl) might include that you developers 
possibly cannot forsee, that you maybe would [not] consider an 
anti-conception feature misuse, and that might need a "recommit" 
sometimes e.g. when code just added requires all older eprints to be 
reprocessed. Touching lastmod no matter if a specific data object meets 
any seldomly occurring conditions for a given action, can result in 
problems. A guy from China had problems accessing our OAI server after - 
and maybe just because - we regenerated the thumbnails, thus potentially 
making they swallow half of HeiDOK.

Shouldn't touch lastmod if regenerating thumbnails, we definitely agree on this.

But for other cases, if the metadata is updated then lastmod must be updated. And the OAI protocol is consistent with this behaviour:

A repository must update the datestamp of a record if a change occurs, the result of which would be a change to the metadata part of the XML-encoding of the record. Such changes include, but are not limited to, changes to the metadata of the record, changes to the metadata format of the record, introduction of a new metadata format, termination of support for a metadata format, etc.

Volatile changes shouldn't cause an update of the lastmod datestamp (by definition... it's a volatile change, not a content/metadata change). This is how EPrints works (but I agree stuff must be fine-tuned... hence our discussion :-)).

So what you cannot have (if you want to respect OAI) is updating some item's metadata and not update the lastmod field (you can of course find ways to achieve this result but not OAI-friendly).


Some more info on my OAI-harvesting aggregator scenario so you 
understand my problem:
The aggregator database is kept small by dropping items that have not 
been modified for more than 100 days. Practically, epadmin recommit is 
therefore a superb tool to make our "new media" service advertise rather 
old if not obsolete stuff. According to OAI specification (as is how I 
remember once having read), OAI-compliant repositories should bear in 
mind harvesters not mirroring all of a data provider. This includes in 
my eyes that the data provider should repropagate records with some 
caution in order to not irritate "bleeding edge stuff" harvesters. Sure, 
one can still argue that those are better off considering dc:date more, 
but this is not always an appropriate filtering criterion.

So... given the conversations we've had so far - would the suggested changes (the "don't touch lastmod if no metadata change has occurred") be compatible with what you're trying to achieve?

Thanks for keeping the conversation up. Once we're both happy, let's propagate this to github.

Seb.