[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[EP-tech] Datafiles size limit of 10 GB



Hi Angeliki,


I would normally advise against using EPrints for files larger than 
10GB, as the time to upload can be quite long and as they is no in-built 
failure recovery, there is a higher chance that uploads may fail and it 
will ultimately take longer to upload than the same amount of data in 
smaller files.? However, if the data is being uploaded from fairly 
locally over a high bandwidth connections you may be OK.


The Javascript that manages uploads can configured to limit the max 
upload size but this is not the case by default.? However, if you check 
88_uploadmethod_file.js and there is code like:


checkFilesize: function(file){
 ??????? if (file.size < max_size)
 ??????? {
 ??????????? return 1;
 ??????? }
 ??????? eprints.currentRepository().phrase( 
{'Plugin/Screen/EPrint/UploadMethod/File:filesize_too_big':{} 
},function(phrase){ alert (file.name+": \n"+ 
phrase["Plugin/Screen/EPrint/UploadMethod/File:filesize_too_big"]); } ) ;
 ??????? return 0;
 ??? },


and max_size is set somewhere else in the script.? Then you can modify 
the value for max_size to allow for bigger files.? If there is no 
max_size set then you should be OK to upload any file size. However, be 
sure to edit the right version of 88_uploadmethod_file.js as this can 
appear on lib/, site_lib/ and archive level subdirectories.? You want to 
edit the one at archive level.? If it does not exist copy the one from 
site_lib or if that does not exist lib and then edit it at the archive 
level 
(archives/ARCHIVE_NAME/cfg/static/javascript/auto/88_uploadmethod_file.js).


I don't know if web browsers have built in upload limits anymore but if 
they do I would not be surprised if 10GB was a limit they may have in 
place.? So if you still cannot upload large files it may be worth 
investigating if it is due to your web browser. However, if the upload 
limit is not restricted by the setting of max_size, I would not be 
surprised if the upload did fail at some point because such a large file 
has more opporunity due to network and other issues.? There is also a 
small possibility that 10GB leads to some integer overflow somewhere.? 
It may be in the Javascript in 88_uploadmethod_file.js but I am not 
aware of this being an issue but I have not tried to upload files over 
2-3GB.


Regards


David Newman


On 18/10/2021 09:58, AndrikAZ--- via Eprints-tech wrote:
> *CAUTION:* This e-mail originated outside the University of Southampton.
>
> Hello All,
>
>
> In our Ubira eData 
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fedata.bham.ac.uk%2F%3F_ga%3D2.216318934.1906795383.1634547270-1716269736.1580930717&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cbeb57c41c1af4ef4e2d308d992232ab6%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637701501994636845%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=lki4X5GanG9mCAvixPvo4nL77xof0HEMtRyPOzeG%2Bc4%3D&amp;reserved=0>?repository, 
> we have a limit on the uploaded datasets?of 10 GB per record. There 
> are cases though, with much larger datasets that have to be deposited 
> in a single record. Would it be possible to exceed this 10 GB?limit?
>
>
> Any recommendation will be very appreciated.
>
>
> Many thanks,
>
> Angeliki
>
>
>
> Angeliki Andrikopoulou
>
> Research Data Librarian
>
> Library Services
>
> Skype:A.Andrikopoulou at bham.ac.uk <mailto:A.Andrikopoulou at bham.ac.uk>
>
> e: A.Andrikopoulou at bham.ac.uk <mailto:A.Andrikopoulou at bham.ac.uk>
>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fintranet.birmingham.ac.uk%2Flibrary&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cbeb57c41c1af4ef4e2d308d992232ab6%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637701501994636845%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=kaVMUBCyUEWJlty5nLhGCnbEtXw%2FGQv3Q5q4gwoV3%2Fw%3D&amp;reserved=0 
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fintranet.birmingham.ac.uk%2Flibrary&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cbeb57c41c1af4ef4e2d308d992232ab6%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637701501994636845%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=kaVMUBCyUEWJlty5nLhGCnbEtXw%2FGQv3Q5q4gwoV3%2Fw%3D&amp;reserved=0>
>
> Twitter 
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fuoblibservices&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cbeb57c41c1af4ef4e2d308d992232ab6%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637701501994646801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=1yF3UY5Dkf4vH9FKgLutgBndHZ2KLeYOr7ZBYm5DyUc%3D&amp;reserved=0>?/?Instagram 
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fuoblibservices&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cbeb57c41c1af4ef4e2d308d992232ab6%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637701501994646801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=yFPvC0M9zv5SxmDKCb3fSczwt97ZrJWxLSi%2BMtpToTg%3D&amp;reserved=0>?/ 
> Facebook 
> <https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.facebook.com%2FUniBirminghamLibrary&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cbeb57c41c1af4ef4e2d308d992232ab6%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637701501994646801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=pk5rtfT%2B55BB3affJuSW2UasbQXu2JxIOfTvp5OWVwE%3D&amp;reserved=0>
>
>
> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cbeb57c41c1af4ef4e2d308d992232ab6%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637701501994646801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=0CGL45e3l9RGhVUyxaNYW%2BMOVCVQU5lX8hND6v9kNZQ%3D&amp;reserved=0
> *** EPrints community wiki: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cbeb57c41c1af4ef4e2d308d992232ab6%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637701501994646801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=%2B6zd3YWIJ%2FTEDoD9DG4Shwo0mUPnzN8kcESSNOknAbo%3D&amp;reserved=0


-- 
This email has been checked for viruses by AVG.
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.avg.com%2F&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cbeb57c41c1af4ef4e2d308d992232ab6%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637701501994646801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=4pU6We2BEtr1vP98%2FqCtxvcRSJvI9SFqpqCTTE5cTJA%3D&amp;reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20211018/82c9899a/attachment-0001.html