Page 1 of 1
Faster file handling
Posted: Sat Aug 05, 2006 1:31 am
by Tipperton
I re-wrote a program I had originally written in PowerBasic into PureBasic because I wanted to give it a GUI to make it easier to use. PowerBasic's DDT system is overly complex compared to PureBasic's Gadget system.
End result: The PowerBasic program finished the file in less time than it took the PureBasic program to do one percent of the file!
So I know file operations (specifically string reading and writing) can be a lot faster.
Posted: Sat Aug 05, 2006 2:33 am
by Flype
Did you try ' FileBuffersSize(#File, size) '
Posted: Sat Aug 05, 2006 2:37 am
by inc.
Also a piece of you code would be nice to see your approach.
And bigger files always should be handled blockwise - if not done already.
Posted: Sat Aug 05, 2006 3:05 am
by Tipperton
Code: Select all
ReadFile(0, "filein.txt")
CreateFile(1, "fileout.txt")
Repeat
sInputLine=ReadString(0)
<process record>
WriteStringN(1, sOutputLine)
Until Eof(0)
CloseFile(0)
CloseFile(1)
My point is based on other languages, PowerBasic, and Visual Basic, both of which can process the file in this fashion much faster than PureBasic does, that faster reading and writing of files is possible and shouldn't require extra libraries or special coding.
I had ran into this before (as did other PureBasic users) and switched to PowerBasic. The problem with PowerBasic is that it assumes you know and understand Windows API programming because that's all their DDT (Dynamic Dialog Tools) system is is just built in commands that mirror the API. When I discovered that PureBasic had released version 4 I decided to try it. It's nice but it appears the file reading and writting is still as slow as it was before.
So I decided to add improved file reading and writing to the wish list/feature requests.
Posted: Sat Aug 05, 2006 1:03 pm
by Flype
Is FileBuffersSize(#File, size) really hard to use ?
Posted: Sat Aug 05, 2006 1:19 pm
by Tipperton
Flype wrote:Is FileBuffersSize(#File, size) really hard to use ?
Tried it, increasing the buffer size from the default 4096 to 65536.
Considering the records average about 300 bytes each, it made little (if any) difference.
BTW: I saw references to Rings' FastFile library but it doesn't appear to be on any of the PureBasic support sites like PureProject or PureArea. Is it no longer available?
Posted: Sat Aug 05, 2006 1:24 pm
by Fred
It's definitely much faster than before, and to me it sounds quite hard to reduce a lot the string reading speed anymore. Can you put the PowerBasic and VB executable + code available so i could test ? I assume you disabled the debugger as well.
Posted: Sat Aug 05, 2006 1:31 pm
by Nik
My tip is an enabled debugger as well^^
PB Is deffinitely very fast when it comes to file access, don't think VB can beat that, especially their .Net versions
Posted: Sat Aug 05, 2006 1:32 pm
by PB
Maybe this is actually the cause of the delay, rather than the disk read/write?
Posted: Sat Aug 05, 2006 2:44 pm
by Tipperton
Yes, the debugger is disabled.
PB wrote:
Maybe this is actually the cause of the delay, rather than the disk read/write?
Possibly, I've seen comments that PureBasic's string handling isn't all that fast though I'm not really doing anything all that fancy.
The file is in comma separated fields and all I'm doing is combining a couple of the fields into one field, formatting a couple of others, and deleting a few others. Then writing the result to an output file.
Posted: Sat Aug 05, 2006 3:04 pm
by Tipperton
Continued in this thread:
http://www.purebasic.fr/english/viewtopic.php?p=155678
Because this is becomming more of a discussion rather than an enhancement request.
Posted: Sat Aug 05, 2006 3:51 pm
by Fred
Tipperton wrote:Possibly, I've seen comments that PureBasic's string handling isn't all that fast though I'm not really doing anything all that fancy.
Please stop doing such assumptions, as you seem to not be very experienced with PB (strings handling in PB is very fast). We can move a topic when it's needed, so no need to create a second one (for the next time).
Posted: Sat Aug 05, 2006 9:25 pm
by Tipperton
Fred wrote:Please stop doing such assumptions, as you seem to not be very experienced with PB (strings handling in PB is very fast).
What assumptions? I've seen a number of posts that say that even in v4 that string functions aren't as fast as they could be. I'm just quoting those posts. I figure they have more PureBasic experience than I do so they must know something that I don't.
Fred wrote:We can move a topic when it's needed, so no need to create a second one (for the next time).
Noted: If this forum software is anything like what I use for the forum I run, you should be able to move this topic and merge it with the other if you wish.
Posted: Sat Aug 05, 2006 9:56 pm
by blueznl
show code... please?
Posted: Sat Aug 05, 2006 10:14 pm
by Tipperton
blueznl wrote:show code... please?
The code is in this thread:
http://www.purebasic.fr/english/viewtopic.php?p=155678
The problem has been solved, it turned out to be how often I was calling a function that allowed Windows to process events in general.
Thanks anyway.