Open File larger than the amount of ram you have

Just starting out? Need help? Post your questions and find answers here.
User avatar
ts-soft
Always Here
Always Here
Posts: 5756
Joined: Thu Jun 24, 2004 2:44 pm
Location: Berlin - Germany

Re: Open File larger than the amount of ram you have

Post by ts-soft »

Yes, Filemapping is the way to go, see here:
http://www.purebasic.fr/german/viewtopi ... 94#p152894
PureBasic 5.73 | SpiderBasic 2.30 | Windows 10 Pro (x64) | Linux Mint 20.1 (x64)
Old bugs good, new bugs bad! Updates are evil: might fix old bugs and introduce no new ones.
Image
User avatar
PureLeo
Enthusiast
Enthusiast
Posts: 221
Joined: Fri Jan 29, 2010 1:05 pm
Location: Brazil

Re: Open File larger than the amount of ram you have

Post by PureLeo »

Thanks very much, I didn't know about that :)
User avatar
netmaestro
PureBasic Bullfrog
PureBasic Bullfrog
Posts: 8451
Joined: Wed Jul 06, 2005 5:42 am
Location: Fort Nelson, BC, Canada

Re: Open File larger than the amount of ram you have

Post by netmaestro »

I ran the ts-soft test code (no debugger) on a 3.1g file, it couldn't open. PB opens it ok. I tried on a 183mb file and pb was around 10% slower. I was expecting a bigger difference than that. (PB 4.60b3)
BERESHEIT
User avatar
PureLeo
Enthusiast
Enthusiast
Posts: 221
Joined: Fri Jan 29, 2010 1:05 pm
Location: Brazil

Re: Open File larger than the amount of ram you have

Post by PureLeo »

Hm... I haven't tested it on very large files yet
User avatar
netmaestro
PureBasic Bullfrog
PureBasic Bullfrog
Posts: 8451
Joined: Wed Jul 06, 2005 5:42 am
Location: Fort Nelson, BC, Canada

Re: Open File larger than the amount of ram you have

Post by netmaestro »

I'm working on a SHA 224-256-384-512 FileFingerprint implementation and I discovered that my posted code worked fine for files with size <= #MAXLONG but failed for larger sizes. So I worked with the code until I had a version that seemed to work correctly. (unposted as yet) I dowloaded a free file hash generator, hashslash, and tested it against my code. No differences. So I was happy until I discovered that hashslash was processing large files in about 1/3 of the time my code took. Ouch. I know my code is efficient (it completes in around 60% of the time the posted version takes) so I looked to the filemapping code by ts-soft for a possible solution to the speed difference and so far I only see an improvement of around 10% for filesizes <= #MAXLONG and no success opening files larger than that.
BERESHEIT
El_Choni
TailBite Expert
TailBite Expert
Posts: 1007
Joined: Fri Apr 25, 2003 6:09 pm
Location: Spain

Re: Open File larger than the amount of ram you have

Post by El_Choni »

Maybe using ts-soft's ReadData_M instead of ReadByte_M would help? (sorry if you already tried this)

Ragards,
El_Choni
Post Reply