FastFile released

BackupUser · Post by **BackupUser** » Fri Apr 04, 2003 9:54 am

Restored from previous forum. Originally posted by Rings.

Okay folks, next Beta of FastFileText

is downloadable at:

http://www.terrorkommando.de/fastfiletext2.zip

with fixed CRLFCRL bug

Its a long way to the top if you wanna .....CodeGuru

Post by **Rings** » Wed Jun 11, 2003 12:10 pm

i fixed a small bug with internal buffer (Line 32669-32670 Bug , thx pcfreak ).
check update here:

FastFileText Version 3

the developing of the FastFileLibrary continues......

Post by **Rings** » Wed Jun 11, 2003 12:38 pm

I fixed the FastFileCRC32 to work with ReadOnly File (cd-related)
download this update here

FastFileCRC32

Searhin · Post by **Searhin** » Sun Jul 06, 2003 11:45 am

Rings, your FastFile lib is amazing - until now i thought i wouldn't need it (what an error...)

i modified your example file slightly

some test results are at the bottom. using small or very large ASCII files FastFile is about 50..150 times faster, but the maximum speed gain seems to be with mid-sized files (a few MBs).

Code: Select all

;FastFile with Textfiles
;
;PureBasicLibrary (C) 2003 by Siegfried Rings
;
;BETA
;
;Commandset:
;
; LinesCount.l = FastOpenTextFile( Filename )  ;Opens the File for FastTEXTReading
;                                              ;Result is the lines available
;
; FastCloseTextFile()                          ;Close the File
;
; LineContent.s = FastReadLine(LineNumber)     ;Get the Content(String) of a Line
;
; Tested with following Textfile: http://www.holybiblecentral.org/holybiblekjv.zip 
;
;(this help file modified by Searhin)


;filename.s = "C:\PB_data for TreeTest.txt"   ; 281 KB
;filename.s = "C:\holybiblekjv.txt"           ; 4.23 MB
filename.s = "C:\test.txt"                   ; 20.5 MB

size = FileSize(filename)

If size > 0  And  OpenConsole() <> 0  And  CreateFile(2,"C:\results.txt")
                                            : WriteStringN(";PERFORMANCE TEST OF RINGS' FAST_TEXT_LIB")
  PrintN("file = "+filename)                : WriteStringN(";file = "+filename)
  PrintN("file size = "+Str(size)+" bytes") : WriteStringN(";file size = "+Str(size)+" bytes")
  PrintN(""): PrintN("")                    : WriteStringN(";"): WriteStringN(";")
  PrintN("first with Rings' FastFile lib:") : WriteStringN(";first with Rings' FastFile lib:")
  
  t1 = GetTickCount_()
  lines = FastOpenTextFile(filename)
  t2 = GetTickCount_() 
  PrintN(Space(2)+StrU(lines,2)+" lines counted in "+StrF((t2-t1)/1000,3)+" secs")
  WriteStringN(";"+Space(2)+StrU(lines,2)+" lines counted in "+StrF((t2-t1)/1000,3)+" secs")
  
  string.s = ""
  i.l = 0
  t1 = GetTickCount_()
  While i <= lines
    INC i
    string = FastReadLine(i)
  Wend
  t2 = GetTickCount_()
  fast.f = (t2-t1)/1000
  PrintN(Space(2)+"all lines read in "+StrF(fast,3)+" seconds")
  WriteStringN(";"+Space(2)+"all lines read in "+StrF(fast,3)+" seconds")
  
  sel_line = Int(lines/2)
  
  PrintN(""): WriteStringN(";")
  Repeat
    Print(Space(2)+"which line should be read? ")
    WriteString(";"+Space(2)+"which line should be read? ")
    input.s = Input(): line = Val(input)
    PrintN("")
    If line > 0  And  line <= lines
      WriteStringN(Str(line))
      string = FastReadLine(line)
      PrintN(Space(4)+"content of line_"+StrU(line,2)+" = "+Chr(34)+string+Chr(34))
      WriteStringN(";"+Space(4)+"content of line_"+StrU(line,2)+" = "+Chr(34)+string+Chr(34))
    ElseIf line = 0
    Else
      PrintN(Space(4)+"(invalid line number)")
      WriteStringN(";"+Space(4)+"(invalid line number)")
    EndIf
  Until input = ""
  FastCloseTextFile()
  PrintN(""): PrintN(""): WriteStringN(";"): WriteStringN(";")
  PrintN("now with generic PB functions:")
  WriteStringN(";now with generic PB functions:")
  
  ; count lines with PureBasic commands
  ; (counting and reading simultaneously is not slower compared to executing both functions in separate loops)
  lines = 0
  If ReadFile(1,filename)
    t1 = GetTickCount_()
    While Eof(1) = 0
      string = ReadString()
      INC lines
    Wend
    t2 = GetTickCount_()
    slow.f = (t2-t1)/1000
    CloseFile(1)
    UseFile(2)
    PrintN(Space(2)+StrU(lines,2)+" lines counted")
    PrintN(Space(2)+"all lines read in "+StrF(slow,3)+" seconds")
    WriteStringN(";"+Space(2)+StrU(lines,2)+" lines counted")
    WriteStringN(";"+Space(2)+"all lines read in "+StrF(slow,3)+" seconds")
    PrintN(""): PrintN(""): WriteStringN(";"): WriteStringN(";")

    If fast = 0: fast = 0.001: EndIf
    PrintN("FastFile was "+Str(slow/fast)+" times faster than generic PB")
    PrintN("Cheers Rings!")
    WriteStringN(";FastFile was "+Str(slow/fast)+" times faster than generic PB")
    WriteStringN(";Cheers Rings!")
    CloseFile(2)
    Repeat: Until Inkey()
    
  EndIf 
EndIf

End

; R E S U L T S :
; ---------------

;PERFORMANCE TEST OF RINGS' FAST_TEXT_LIB
;file = C:\holybiblekjv.txt
;file size = 4436584 bytes
;
;
;first with Rings' FastFile lib:
;  31347 lines counted in 0.030 secs
;  all lines read in 0.030 seconds
;
;  which line should be read? 1212
;    content of line_1212 = "Ge41:9 Then spake the chief butler unto Pharaoh, saying, I do remember my faults this day:"
;  which line should be read? ;
;
;now with generic PB functions:
;  31347 lines counted
;  all lines read in 7.431 seconds
;
;
;FastFile was 248 times faster than generic PB
;Cheers Rings!

;PERFORMANCE TEST OF RINGS' FAST_TEXT_LIB
;file = C:\test.txt
;file size = 21554877 bytes
;
;
;first with Rings' FastFile lib:
;  439668 lines counted in 1.662 secs
;  all lines read in 0.341 seconds
;
;  which line should be read? 250000
;    content of line_250000 = "     "AF432785""
;  which line should be read? 100000
;    content of line_100000 = "/Reference:"
;  which line should be read? ;
;
;now with generic PB functions:
;  439668 lines counted
;  all lines read in 34.460 seconds
;
;
;FastFile was 101 times faster than generic PB
;Cheers Rings!

[/code]

Num3 · Post by **Num3** » Sun Jul 06, 2003 12:51 pm

Hi Rings,

Could you please post a all-in-one zip file with all the fastfile modules?

I'm a bit lost with all the updates

Post by **Rings** » Sun Jul 06, 2003 5:06 pm

Num3 wrote:Hi Rings,

Could you please post a all-in-one zip file with all the fastfile modules?

I'm a bit lost with all the updates

Yes you are right num (and not alone),
i post a update next week with all new fixes and updates included

@searhin: thx for the flowers. That is the prefix 'FastFile' in the lib...

Tipperton · Post by **Tipperton** » Sun Jul 06, 2003 5:28 pm

I'm writing a program that updates a database from a very large (>500MB) text file and need something a lot faster than ReadString().

I've seen the FileBuffer functions by Horst but they want to load the whole file into memory. I think this file is a bit big for that not to mention the amount of time it would take to read the whole thing to memory in the first place.

Question for Rings:

Does the FastFileText functions try to load the entire file into memory or does it use a buffer and just reads sections of the file into memory at a time?

Thanks.

Post by **Rings** » Sun Jul 06, 2003 7:42 pm

Tipperton wrote:I'm writing a program that updates a database from a very large (>500MB) text file and need something a lot faster than ReadString().

I've seen the FileBuffer functions by Horst but they want to load the whole file into memory. I think this file is a bit big for that not to mention the amount of time it would take to read the whole thing to memory in the first place.

Question for Rings:

Does the FastFileText functions try to load the entire file into memory or does it use a buffer and just reads sections of the file into memory at a time?

Thanks.

There are more functions in FastFile.
For normaly binary file-access you can use the 'normaly' FastFile to read/write Bytes,Longs and Data etc.
For Textfiles use FasFileText which scans whole file for CRLF and their corresponding lines.
For Filechecking you can Use FastFileCRC32 and FastFileMD5 .
And of course, if FastFileText is not fast enough for normaly Textfiles, buy a faster Workstation.....or feel free to create your own FastDataaccess

Tipperton · Post by **Tipperton** » Sun Jul 06, 2003 7:57 pm

Rings wrote:if FastFileText is not fast enough for normaly Textfiles, buy a faster Workstation

Well blow me down! I'm doing a major upgrade this week! (From a 450MHz AMD K6-III+ to an AMD Athlon XP 2000+ (1.6GHz I think))...

The problem is that the target system that this will run on (I've been contracted to write this program) will be a 500MHz system so every bit of speed I can get will be a plus...

Rings wrote:or feel free to create your own FastDataaccess

I may just have to do that, what would be ideal is something like Horst's routines but using a buffer that only loads a part of the file at a time.

Initially all I need is sequential access through the file so it shouldn't be too hard to do. Eventually I will want to give it some ability to move around in the file at will. That will be the challenge...

Justin · Post by **Justin** » Mon Jul 28, 2003 5:34 pm

How about a way to display the progress in Fastfilecrc() ?, maybe suppling the address of a PB callback function. Also a way to stop the process using that callback would be great.

thanks for the lib, really usefull.

Tipperton · Post by **Tipperton** » Mon Jul 28, 2003 5:43 pm

Justin wrote:maybe suppling the address of a PB callback function. Also a way to stop the process using that callback would be great.

Stopping it should be fairly easy, just pass NULL for that parameter, then have the library check the callback address, if it's NULL, don't call it.

Insomniac · Post by **Insomniac** » Tue Jul 29, 2003 10:03 am

Hi Rings,

Can you check FastFileMD5 works with read only files now also please.

The version I have returns the correct value but also "beeps" every time a read only file is encountered.

FastfileCRC32 had a similar problem until you fixed it (as you state in this thread).

There might be a later one around than the one I have so I'm looking forward to your consolidated update.

Keep up the good work.

Insomniac

Post by **Rings** » Fri Aug 01, 2003 11:02 am

Insomniac wrote:Hi Rings,

Can you check FastFileMD5 works with read only files now also please.

The version I have returns the correct value but also "beeps" every time a read only file is encountered.

FastfileCRC32 had a similar problem until you fixed it (as you state in this thread).

There might be a later one around than the one I have so I'm looking forward to your consolidated update.

Keep up the good work.

Insomniac

Readonly-(bug) FastFileMD5 fixed for the next release.

newbie · Post by **newbie** » Fri Aug 01, 2003 12:11 pm

Hi there,

i would just want to know if the use of FastRead would be faster than ReadFile to read remote files (on a LAN) ?
Of course it is better to test it myself but i can't find link to download FastFile library, if i understand right there are under developpement and next release will be available soon ?

Paul · Post by **Paul** » Fri Aug 01, 2003 3:01 pm

Rings posted a version of FastFile on the PB Resources Site a while back

PureBasic Forums - English

FastFile released

single update file

Re: single update file