Faster file handling

Tipperton · Post by **Tipperton** » Sat Aug 05, 2006 1:31 am

I re-wrote a program I had originally written in PowerBasic into PureBasic because I wanted to give it a GUI to make it easier to use. PowerBasic's DDT system is overly complex compared to PureBasic's Gadget system.

End result: The PowerBasic program finished the file in less time than it took the PureBasic program to do one percent of the file!

So I know file operations (specifically string reading and writing) can be a lot faster.

Flype · Post by **Flype** » Sat Aug 05, 2006 2:33 am

Did you try ' FileBuffersSize(#File, size) '

inc. · Post by **inc.** » Sat Aug 05, 2006 2:37 am

Also a piece of you code would be nice to see your approach.

And bigger files always should be handled blockwise - if not done already.

Tipperton · Post by **Tipperton** » Sat Aug 05, 2006 3:05 am

Code: Select all

ReadFile(0, "filein.txt")
CreateFile(1, "fileout.txt")
Repeat
  sInputLine=ReadString(0)
  <process record>
  WriteStringN(1, sOutputLine)
Until Eof(0)
CloseFile(0)
CloseFile(1)

My point is based on other languages, PowerBasic, and Visual Basic, both of which can process the file in this fashion much faster than PureBasic does, that faster reading and writing of files is possible and shouldn't require extra libraries or special coding.

I had ran into this before (as did other PureBasic users) and switched to PowerBasic. The problem with PowerBasic is that it assumes you know and understand Windows API programming because that's all their DDT (Dynamic Dialog Tools) system is is just built in commands that mirror the API. When I discovered that PureBasic had released version 4 I decided to try it. It's nice but it appears the file reading and writting is still as slow as it was before.

So I decided to add improved file reading and writing to the wish list/feature requests.

Flype · Post by **Flype** » Sat Aug 05, 2006 1:03 pm

Is FileBuffersSize(#File, size) really hard to use ?

Tipperton · Post by **Tipperton** » Sat Aug 05, 2006 1:19 pm

Flype wrote:Is FileBuffersSize(#File, size) really hard to use ?

Tried it, increasing the buffer size from the default 4096 to 65536.

Considering the records average about 300 bytes each, it made little (if any) difference.

BTW: I saw references to Rings' FastFile library but it doesn't appear to be on any of the PureBasic support sites like PureProject or PureArea. Is it no longer available?

Post by **Fred** » Sat Aug 05, 2006 1:24 pm

It's definitely much faster than before, and to me it sounds quite hard to reduce a lot the string reading speed anymore. Can you put the PowerBasic and VB executable + code available so i could test ? I assume you disabled the debugger as well.

Nik · Post by **Nik** » Sat Aug 05, 2006 1:31 pm

My tip is an enabled debugger as well^^
PB Is deffinitely very fast when it comes to file access, don't think VB can beat that, especially their .Net versions

PB · Post by PB » Sat Aug 05, 2006 1:32 pm

Tipperton wrote:
Code: Select all
<process record>

Maybe this is actually the cause of the delay, rather than the disk read/write?

Tipperton · Post by **Tipperton** » Sat Aug 05, 2006 2:44 pm

Yes, the debugger is disabled.

PB wrote:
Tipperton wrote:
Code: Select all
<process record>
Maybe this is actually the cause of the delay, rather than the disk read/write?

Possibly, I've seen comments that PureBasic's string handling isn't all that fast though I'm not really doing anything all that fancy.

The file is in comma separated fields and all I'm doing is combining a couple of the fields into one field, formatting a couple of others, and deleting a few others. Then writing the result to an output file.

Tipperton · Post by **Tipperton** » Sat Aug 05, 2006 3:04 pm

Continued in this thread:

http://www.purebasic.fr/english/viewtopic.php?p=155678

Because this is becomming more of a discussion rather than an enhancement request.

Post by **Fred** » Sat Aug 05, 2006 3:51 pm

Tipperton wrote:Possibly, I've seen comments that PureBasic's string handling isn't all that fast though I'm not really doing anything all that fancy.

Please stop doing such assumptions, as you seem to not be very experienced with PB (strings handling in PB is very fast). We can move a topic when it's needed, so no need to create a second one (for the next time).

Tipperton · Post by **Tipperton** » Sat Aug 05, 2006 9:25 pm

Fred wrote:Please stop doing such assumptions, as you seem to not be very experienced with PB (strings handling in PB is very fast).

What assumptions? I've seen a number of posts that say that even in v4 that string functions aren't as fast as they could be. I'm just quoting those posts. I figure they have more PureBasic experience than I do so they must know something that I don't.

Fred wrote:We can move a topic when it's needed, so no need to create a second one (for the next time).

Noted: If this forum software is anything like what I use for the forum I run, you should be able to move this topic and merge it with the other if you wish.

blueznl · Post by **blueznl** » Sat Aug 05, 2006 9:56 pm

show code... please?

Tipperton · Post by **Tipperton** » Sat Aug 05, 2006 10:14 pm

blueznl wrote:show code... please?

The code is in this thread: http://www.purebasic.fr/english/viewtopic.php?p=155678

The problem has been solved, it turned out to be how often I was calling a function that allowed Windows to process events in general.

Thanks anyway.