[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[EP-tech] Eprints-tech Digest, Vol 157, Issue 23



CAUTION: This e-mail originated outside the University of Southampton.

Dear Paul,


Many thanks for your prompt and very helpful response to my query.

I have tried in the past to upload data files the size of 10 GB or a bit larger. Indeed, it is a very slow and time-consuming process and in case of success in uploading, the same cumbersome process will be mirrored during the downloading when someone attempts to access these data files in a future time. Thus, it might be better to split the files for the sake of functionality and accessibility.

The detailed technical recommendations and instructions you provided in respect of a size extension through the Javascript code are really great and much appreciated. I am tempted to give it a try and I will also investigate the web browser capability, though, I do not expect the issue is rooted there, as I have uploaded large size data files at some point a few months ago.


Best wishes,

Angeliki






________________________________
From: eprints-tech-bounces at ecs.soton.ac.uk <eprints-tech-bounces at ecs.soton.ac.uk> on behalf of eprints-tech-request at ecs.soton.ac.uk <eprints-tech-request at ecs.soton.ac.uk>
Sent: 18 October 2021 11:36:41
To: eprints-tech at ecs.soton.ac.uk
Subject: Eprints-tech Digest, Vol 157, Issue 23

Send Eprints-tech mailing list submissions to
        eprints-tech at ecs.soton.ac.uk

To subscribe or unsubscribe via the World Wide Web, visit
        http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
or, via email, send a message with subject or body 'help' to
        eprints-tech-request at ecs.soton.ac.uk

You can reach the person managing the list at
        eprints-tech-owner at ecs.soton.ac.uk

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Eprints-tech digest..."


Today's Topics:

   1. Datafiles size limit of 10 GB (AndrikAZ at adf.bham.ac.uk)
   2. Re: Datafiles size limit of 10 GB (David R Newman)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Oct 2021 08:58:41 +0000
From: "AndrikAZ at adf.bham.ac.uk" <a.andrikopoulou at bham.ac.uk>
Subject: [EP-tech] Datafiles size limit of 10 GB
To: "eprints-tech at ecs.soton.ac.uk" <eprints-tech at ecs.soton.ac.uk>
Message-ID: <2f84fc74cda3409db1d6ff4ebc0b6da2 at adf.bham.ac.uk>
Content-Type: text/plain; charset="iso-8859-1"

CAUTION: This e-mail originated outside the University of Southampton.

Hello All,


In our Ubira eData<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fedata.bham.ac.uk%2F%3F_ga%3D2.216318934.1906795383.1634547270-1716269736.1580930717&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=vbBqm3ef62gIAjbd0KgFrr%2F1dMg7GvOd%2B%2F0RQPIF9Bk%3D&amp;reserved=0> repository, we have a limit on the uploaded datasets of 10 GB per record. There are cases though, with much larger datasets that have to be deposited in a single record. Would it be possible to exceed this 10 GB limit?


Any recommendation will be very appreciated.


Many thanks,

Angeliki


Angeliki Andrikopoulou
Research Data Librarian

Library Services
Skype: A.Andrikopoulou at bham.ac.uk<mailto:A.Andrikopoulou at bham.ac.uk>
e: A.Andrikopoulou at bham.ac.uk<mailto:A.Andrikopoulou at bham.ac.uk>
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fintranet.birmingham.ac.uk%2Flibrary&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=5RJVo1gTWWQBpfJ05UlltINshTpFoJ5RBk6UD2E23zA%3D&amp;reserved=0
Twitter<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fuoblibservices&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=Wb%2FBgQCn9farAnaf7B5ph5ZC3%2Bn9K%2FRHoiOZk6t9PEE%3D&amp;reserved=0> / Instagram<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fuoblibservices&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=MBrX5%2BQqbOp6mYdChYPoumW8MYvPOuWKt7IxeC9YDGE%3D&amp;reserved=0> / Facebook<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.facebook.com%2FUniBirminghamLibrary&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=HipmWxEEj5I4RsuJ%2BrIwgTrT2pCZZTbTQTrG7xOQQF0%3D&amp;reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20211018/3d492284/attachment-0001.html

------------------------------

Message: 2
Date: Mon, 18 Oct 2021 11:36:36 +0100
From: David R Newman <drn at ecs.soton.ac.uk>
Subject: Re: [EP-tech] Datafiles size limit of 10 GB
To: <eprints-tech at ecs.soton.ac.uk>, "AndrikAZ at adf.bham.ac.uk"
        <a.andrikopoulou at bham.ac.uk>
Message-ID:
        <EMEW3|e60a30ee0a9b05f7cdddf750e5cbf8e7x9HBad03drn|ecs.soton.ac.uk|2aedd881-12ee-bed1-c4e2-e7a54cf1e69d at ecs.soton.ac.uk>

Content-Type: text/plain; charset="windows-1252"

Hi Angeliki,


I would normally advise against using EPrints for files larger than
10GB, as the time to upload can be quite long and as they is no in-built
failure recovery, there is a higher chance that uploads may fail and it
will ultimately take longer to upload than the same amount of data in
smaller files.? However, if the data is being uploaded from fairly
locally over a high bandwidth connections you may be OK.


The Javascript that manages uploads can configured to limit the max
upload size but this is not the case by default.? However, if you check
88_uploadmethod_file.js and there is code like:


checkFilesize: function(file){
 ??????? if (file.size < max_size)
 ??????? {
 ??????????? return 1;
 ??????? }
 ??????? eprints.currentRepository().phrase(
{'Plugin/Screen/EPrint/UploadMethod/File:filesize_too_big':{}
},function(phrase){ alert (file.name+": \n"+
phrase["Plugin/Screen/EPrint/UploadMethod/File:filesize_too_big"]); } ) ;
 ??????? return 0;
 ??? },


and max_size is set somewhere else in the script.? Then you can modify
the value for max_size to allow for bigger files.? If there is no
max_size set then you should be OK to upload any file size. However, be
sure to edit the right version of 88_uploadmethod_file.js as this can
appear on lib/, site_lib/ and archive level subdirectories.? You want to
edit the one at archive level.? If it does not exist copy the one from
site_lib or if that does not exist lib and then edit it at the archive
level
(archives/ARCHIVE_NAME/cfg/static/javascript/auto/88_uploadmethod_file.js).


I don't know if web browsers have built in upload limits anymore but if
they do I would not be surprised if 10GB was a limit they may have in
place.? So if you still cannot upload large files it may be worth
investigating if it is due to your web browser. However, if the upload
limit is not restricted by the setting of max_size, I would not be
surprised if the upload did fail at some point because such a large file
has more opporunity due to network and other issues.? There is also a
small possibility that 10GB leads to some integer overflow somewhere.?
It may be in the Javascript in 88_uploadmethod_file.js but I am not
aware of this being an issue but I have not tried to upload files over
2-3GB.


Regards


David Newman


On 18/10/2021 09:58, AndrikAZ--- via Eprints-tech wrote:
> *CAUTION:* This e-mail originated outside the University of Southampton.
>
> Hello All,
>
>
> In our Ubira eData
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fedata.bham.ac.uk%2F%3F_ga%3D2.216318934.1906795383.1634547270-1716269736.1580930717&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=vbBqm3ef62gIAjbd0KgFrr%2F1dMg7GvOd%2B%2F0RQPIF9Bk%3D&amp;reserved=0>?repository,
> we have a limit on the uploaded datasets?of 10 GB per record. There
> are cases though, with much larger datasets that have to be deposited
> in a single record. Would it be possible to exceed this 10 GB?limit?
>
>
> Any recommendation will be very appreciated.
>
>
> Many thanks,
>
> Angeliki
>
>
>
> Angeliki Andrikopoulou
>
> Research Data Librarian
>
> Library Services
>
> Skype:A.Andrikopoulou at bham.ac.uk <mailto:A.Andrikopoulou at bham.ac.uk>
>
> e: A.Andrikopoulou at bham.ac.uk <mailto:A.Andrikopoulou at bham.ac.uk>
>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fintranet.birmingham.ac.uk%2Flibrary&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=5RJVo1gTWWQBpfJ05UlltINshTpFoJ5RBk6UD2E23zA%3D&amp;reserved=0
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fintranet.birmingham.ac.uk%2Flibrary&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=5RJVo1gTWWQBpfJ05UlltINshTpFoJ5RBk6UD2E23zA%3D&amp;reserved=0>
>
> Twitter
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fuoblibservices&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=Wb%2FBgQCn9farAnaf7B5ph5ZC3%2Bn9K%2FRHoiOZk6t9PEE%3D&amp;reserved=0>?/?Instagram
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fuoblibservices&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=MBrX5%2BQqbOp6mYdChYPoumW8MYvPOuWKt7IxeC9YDGE%3D&amp;reserved=0>?/
> Facebook
> <https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.facebook.com%2FUniBirminghamLibrary&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=HipmWxEEj5I4RsuJ%2BrIwgTrT2pCZZTbTQTrG7xOQQF0%3D&amp;reserved=0>
>
>
> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=pIxgzKb%2FBKrUR%2BKghD9uBMP2bEfFiVQ8DfE3hM0lBjU%3D&amp;reserved=0
> *** EPrints community wiki: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=BUgdK5DzuGllbP9e0dW2IZwOic8Oj3kv7KbI22c%2FMTw%3D&amp;reserved=0


--
This email has been checked for viruses by AVG.
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.avg.com%2F&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6d7e7044c8246ae974e08d992e8e8f3%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702353857869387%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=UMC1vI8IQLV81lQWwW7ZIrWRsr7QwhTr0StVOjNlpFc%3D&amp;reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20211018/82c9899a/attachment.html

------------------------------

_______________________________________________
Eprints-tech mailing list
Eprints-tech at ecs.soton.ac.uk
http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech


End of Eprints-tech Digest, Vol 157, Issue 23
*********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20211019/54dc54c0/attachment-0001.html