[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[EP-tech] Main/filename not set in staging area

Subject: [EP-tech] Main/filename not set in staging area
From: jkerwin2101 at gmail.com (James Kerwin)
Date: Wed, 14 Dec 2022 11:54:08 +0000
In-reply-to: <DU2PR03MB7895685F4897A66F1AAFE394C41D9@DU2PR03MB7895.eurprd03.prod.outlook.com>
References: <CAKkNZ9AoPrBy4N83ueALCvvS5-65J50FFKEW=o+rrM2o6udSdA@mail.gmail.com> <4cf14eec-4951-a3e9-cd49-9671ac6c3349@ecs.soton.ac.uk> <EMEW3|86f397faaeeae20bb11dd0d9cd146371yB7E5703drn|ecs.soton.ac.uk|4cf14eec-4951-a3e9-cd49-9671ac6c3349@ecs.soton.ac.uk> <DU2PR03MB7895685F4897A66F1AAFE394C41D9@DU2PR03MB7895.eurprd03.prod.outlook.com> <CAKkNZ9BfHPUd3G8ZKDCWDC51FqJthKn2ZpG8HKNf6ZkzN=kX5g@mail.gmail.com>

CAUTION: This e-mail originated outside the University of Southampton.
Hi John and David,

Thank you both for the advice and willingness to help.

It would appear the problem was "me". I fixed one problem where filenames weren't showing as we expected on one page, but this is not related to "main" not being set. This is expected behaviour and my colleagues do it via the EPrints front-end by clicking a button on the file upload workflow that allows them to "select the main file" and populates the missing fields.

Discovered our Staging.pm doesn't actually do anything. It is our "URL.pm" that has been heavily edited to provide a staging area. It kicks of adding the items by using the Document method "add_directory" which specifically states that it does not set main.

Though not a total waste since after having a good look at how it works, I'm thinking of adding some new methods that better reflect how my team use the staging area (e.g. if uploading several files at once and setting main/pos etc).

Thanks,
James


On Thu, Dec 8, 2022 at 2:28 PM John Salter <J.Salter at leeds.ac.uk<mailto:J.Salter at leeds.ac.uk>> wrote:
Hi James,
Sounds like a curious issue?

What does your 'staging' actually do? Does it e.g. pull the files from a mounted filesystem (staging area)?

If possible, sharing the 'Staging.pm' code might help us diagnose.
Just checking - if the filesize is less than 4GB, does it all work as expected, but when it's over 4GB it's broken?

Cheers,
John

From: eprints-tech-bounces at ecs.soton.ac.uk<mailto:eprints-tech-bounces at ecs.soton.ac.uk> [mailto:eprints-tech-bounces at ecs.soton.ac.uk<mailto:eprints-tech-bounces at ecs.soton.ac.uk>] On Behalf Of David R Newman via Eprints-tech
Sent: 08 December 2022 14:05
To: eprints-tech at ecs.soton.ac.uk<mailto:eprints-tech at ecs.soton.ac.uk>; James Kerwin <jkerwin2101 at gmail.com<mailto:jkerwin2101 at gmail.com>>
Subject: Re: [EP-tech] Main/filename not set in staging area


Hi James,

Having a look at File.pm and URL.pm in 3.4.4, the only change for File.pm since the initial 3.4 import that relates to how files upload is some code that strips out trailing and leading spaces before the file is saved to disk and its filename saved to the database.  There is no such change for URL.pm.

One thing that has been altered about file uploads to help deal with problematic filenames is in perl_lib/EPrints/Plugin/Storage/Local.pm.  This change allowed files to be renamed to <fileid>.bin on disk and EPrints would check this as a secondary filename if it could not find the file under its recorded filename in the database.  The mismatch may have been due to a special character being transposed differently in the database to the filesystem, the previous leading/trailing spaces issue or some other problem.  There is a setting ($c->{generic_filenames}) which is disabled by default but if enabled would save files to disk as <fileid>.bin in future rather than the original filename.  This was first implemented for EPrints 3.4.3.

None of what I describe above seems like it would lead to the specific issue you are reporting.  I am assuming that Staging.pm was based on File.pm at some point in the past.  If this diverged pre-3.4 then there may have been other changes between 3.3 and 3.4.

Regards

David Newman
On 08/12/2022 12:30, James Kerwin via Eprints-tech wrote:
CAUTION: This e-mail originated outside the University of Southampton.
Hi All,

Once again I've done something to our poor Data Repository. We're on EPrints 3.4.4.

If we upload files the "normal" way they go on with no problem.

If we do an upload of files over 4GB in our "Staging Area" then "Main" is not set in the Documents database table. I'm almost certain that this is something specific to my repository as we have some slightly different files in:

/usr/share/eprints/perl_lib/EPrints/Plugin/Screen/EPrint/UploadMethod

compared to the github for EPrints 3.4.4:

https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feprints%2Feprints3.4%2Ftree%2Fmaster%2Fperl_lib%2FEPrints%2FPlugin%2FScreen%2FEPrint%2FUploadMethod&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Ca271276025d94b5b35e608daddc9f016%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638066157079035530%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=aOmylkqoV9Ns%2F6dBHZmD87cqBmuYEKPL3zjfx7BuFxA%3D&reserved=0<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feprints%2Feprints3.4%2Ftree%2Fmaster%2Fperl_lib%2FEPrints%2FPlugin%2FScreen%2FEPrint%2FUploadMethod&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Ca271276025d94b5b35e608daddc9f016%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638066157079035530%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=aOmylkqoV9Ns%2F6dBHZmD87cqBmuYEKPL3zjfx7BuFxA%3D&reserved=0>

We have File.pm, URL.pm and Staging.pm. Some changes have been made to these files to enable the staging area. I've no idea how customised this is compared to other repositories. Other values appear to be getting set and the "filename" is being set in the Files table.

Before I go through a semi-destructive process of trying to set this value, is there something very obvious that I'm missing?

Thanks,
James



*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ecs.soton.ac.uk%2Fmailman%2Flistinfo%2Feprints-tech&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Ca271276025d94b5b35e608daddc9f016%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638066157079035530%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=jlIRR9RJbGnAfsSOY3Hv2XKjG2CxT%2FAjufXp3ttpwSQ%3D&reserved=0>

*** Archive: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Ca271276025d94b5b35e608daddc9f016%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638066157079035530%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=u7skNuoVIWwdhj7%2B8XzKYiDb7T8uc6kLAH1KiV9Oi3I%3D&reserved=0<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Ca271276025d94b5b35e608daddc9f016%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638066157079035530%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=u7skNuoVIWwdhj7%2B8XzKYiDb7T8uc6kLAH1KiV9Oi3I%3D&reserved=0>

*** EPrints community wiki: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Ca271276025d94b5b35e608daddc9f016%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638066157079035530%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=vji0Qwpx8Clu7%2FveYLP%2FBkpJaeJGnOQhHQo3PJiVIRc%3D&reserved=0<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Ca271276025d94b5b35e608daddc9f016%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638066157079035530%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=vji0Qwpx8Clu7%2FveYLP%2FBkpJaeJGnOQhHQo3PJiVIRc%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20221214/7a5b7224/attachment-0001.html

Follow-Ups:
- [EP-tech] Main/filename not set in staging area
  - From: jkerwin2101 at gmail.com (James Kerwin)

References:
- [EP-tech] Main/filename not set in staging area
  - From: jkerwin2101 at gmail.com (James Kerwin)
- [EP-tech] Main/filename not set in staging area
  - From: drn at ecs.soton.ac.uk (David R Newman)
- [EP-tech] Main/filename not set in staging area
  - From: J.Salter at leeds.ac.uk (John Salter)
- [EP-tech] Main/filename not set in staging area
  - From: jkerwin2101 at gmail.com (James Kerwin)

Prev by Date: [EP-tech] permission for "manage records"
Next by Date: [EP-tech] permission for "manage records"
Next by thread: [EP-tech] Re: Eprints divisions problem
Index(es):
- Date
- Thread