ReadByte/WriteByte incredibly slow

mp303 · Post by **mp303** » Mon Jul 14, 2003 4:06 am

"This library use buffered functions to increase the writing/reading speed"

Maybe buffering doesn't work for ReadByte/WriteByte functions? ... I'm doing some file conversion, and relying on what the helpfile said, I figured why do my own buffering? buffering "on top" of routines that are already buffered would just be extra overhead - so I'm reading/writing one byte at a time, like so:

Code: Select all

      While ~Eof(#FileIn)
        UseFile(#FileIn)
        i = ReadByte()
        UseFile(#FileOut)
        WriteByte(i)
      Wend

for a 5 MB file, such a copy operation takes nearly 20 seconds on a 2.5 GHz Pentium 4, which is crazy - if these routines are buffered, surely they don't work as intended.

of course my routines do way more than that, but to see whether the 20 seconds of overhead were coming from the actual work or from reading/writing, I removed all the code ... the actual work doesn't make up for even 1% of the CPU usage! 8O ... in other words, if not for the incredibly slow read/write operations, my program would finish in about 0.2 seconds instead of 20!

so now I do have to do my own buffering.

which is incredibly tedious, because I was counting on the library to do what the helpfile promises, and I hate having to reinvent the wheel...

*sigh*

Paul · Post by **Paul** » Mon Jul 14, 2003 4:29 am

and I hate having to reinvent the wheel...

So don't reinvent the wheel, use the FastFile ASM library written by Rings

Gantry · Post by **Gantry** » Mon Jul 14, 2003 11:17 am

I have had the same experience. While it is a good thing not having to reinvent the wheel, I still wonder why we should have to use someone elses wheels. Any plans for rewriting the file routines included in PB?

Still, having completed my first major project in PB I can only say that I love it. My previous releases of the same program had been coded first in c++ builder and then delphi. I rewrote the program in PB. It took me less time to code and the executable was 21 kB instead of 250 kB and with added functionality too.

Regards,

Gantry

Post by **Fred** » Mon Jul 14, 2003 12:12 pm

Yes, I plan to redo these routines.

Post by **Rings** » Mon Jul 14, 2003 3:35 pm

The use of the Fastfile - Lib is free, for everyone using a legal copy of Purebasic .
Of Course the lib is Windows-based only, so a native implement in Purebasic is not so easy and i haven't enough expierence under Linux-apis .
I do not want publish the Source of the library to the public, but i'm open to discuss that later with Fred personally

matthew180 · Post by **matthew180** » Mon Jul 14, 2003 5:16 pm

Fred wrote:Yes, I plan to redo these routines.

When? Can anyone help?

mp303 wrote:so now I do have to do my own buffering.

I already had a run in with the file functions and wrote a buffered version of ReadString(). My code could be easily modified to work like ReadByte() if you don't want to spend time coding your own buffered read functions.

Matthew

mp303 · Post by **mp303** » Mon Jul 14, 2003 6:12 pm

thanks, but I need both read and wrtite functions... besides, this can't (shouldn't) be implemented as functions - the buffers should be integrated into the reading/writing code itself, otherwise I still have four function calls for ever processed byte, which is way too much...

matthew180 · Post by **matthew180** » Mon Jul 14, 2003 7:16 pm

Well, if you need a work around right now, I guess you don't have many choices, unless you think you can wait for Fred to rewrite the functions... But I would not hold my breath on that.

My function uses ReadData() and a 64K buffer, so the amount of system IO is minimal, and the data can be treated however you like. I don't have a write function, but it would be trivial compared to the read function. Actaully, now that I think about it, a byte read would be pretty eacy too, compared to reading one line at a time (since I had to look for end-of-line and all that crap.)

Matthew

mp303 · Post by **mp303** » Tue Jul 15, 2003 6:10 pm

Matt,

On second thought, I'd like to take a look at your code, is it online anywhere?

Rings,

I checked out your FastFile library ... buuuuut, uhm ... is it a beta/prototype, or did you just not think anyone would need to open more than one file at a time? 8O

matthew180 · Post by **matthew180** » Tue Jul 15, 2003 8:00 pm

Okay, let me detangle it from the project I wrote it for and I'll send it your way.

Matthew

matthew180 · Post by **matthew180** » Thu Jul 17, 2003 4:21 am

Okay, I've got some functions for you, but I have to sleep... I'll put them up tomorrow (I'm in GMT -5). I added a single byte read and write for you to use, as well as cleaned up the readstring function.

Matthew

Post by **Rings** » Thu Jul 17, 2003 7:18 am

mp303 wrote:Rings,
I checked out your FastFile library ... buuuuut, uhm ... is it a beta/prototype, or did you just not think anyone would need to open more than one file at a time? 8O

No, no beta or Prototype, just a first version that works.
You can use 2 Files if you tackle the second file with the 'B' Functions.
For example FastOpenFileB, FastReadByteB etc...
That is of course only a workaround, but i will not loos speed whicle switching between files.
For a next version there is be planned to use a Filepointer like
FastOpenFile(Filenumber,Filename) or FastReadByte(FileNumber,Offset)
and other goodies like FastCrypting .... .
Well, let me see what holidays offer free time for coding..

matthew180 · Post by **matthew180** » Thu Jul 17, 2003 4:58 pm

My file code is up:

http://www.purebasic.org

Hit the Article link to FishLib. Let me know if you have any questions.

Matthew