PureBasic Forums - English

http://www.purebasic.com
https://www.purebasic.fr/english/

Read text file is very slow

https://www.purebasic.fr/english/viewtopic.php?t=48665

Page 1 of 1

Read text file is very slow

Posted: Fri Dec 23, 2011 8:44 pm
by moob
hi...

this code are slow reading 8940 lines of text to ListIcon.

made some mistake to be so slow???

here is text file: http://www.mediafire.com/?tdpzkb2bvhulwpr

anyone can text this.

thanks...

Code: Select all


Procedure OpenPlayerLista(*retinort)
  Pattern$ = "playerlist.txt (*.txt)|playerlist.txt"
  Pattern = 0
  FileR$ = OpenFileRequester("Please choose the playerlist...", "", Pattern$, Pattern)
  If OpenFile(6, FileR$)
    If ReadFile(6, FileR$)   ; if the file could be read, we continue...
      ClearGadgetItems(#ListIcon_41)
      StatusBarText(#StatusBar_wndMain, 1, "Loading Playerlist wait...", #PB_StatusBar_Center)
      DisableMenuItem(#Menu_wndMain, #Menu_wndMain_OpenPlayerlist, 1) 
      IDN = 0
      length = Lof(6)
      While Eof(6) = 0 
        IDN = IDN + 1
        Texto$ = Trim(ReadString(6)) 
        NomesID$ = Trim(StringField(Texto$, 1, ":"))
        NsID$ = Trim(StringField(Texto$, 2, ":"))
        StatusBarProgress(#StatusBar_wndMain, 2, Loc(6) , #PB_StatusBar_Raised, 0,length)
        AddGadgetItem(#ListIcon_41, -1, Str(IDN)+Chr(10)+NomesID$+Chr(10)+NsID$)
      Wend
      CloseFile(6)
      DisableMenuItem(#Menu_wndMain, #Menu_wndMain_OpenPlayerlist, 0) 
      StatusBarText(#StatusBar_wndMain, 2, "PlayerList Load Successfully", #PB_StatusBar_Center)
      StatusBarText(#StatusBar_wndMain, 1, "PlayerList Loaded", #PB_StatusBar_Center)
    EndIf
  EndIf
EndProcedure

Re: Read text file is very slow

Posted: Fri Dec 23, 2011 9:14 pm
by Kiffi
this is a little bit faster:

Code: Select all

    HideGadget(#ListIcon_41, #True) ; <--- !
    While Eof(6) = 0 
        [...]
    Wend
    HideGadget(#ListIcon_41, #False) ; <--- !
and: you don't need the OpenFile()-Line

Greetings ... Kiffi

Re: Read text file is very slow

Posted: Fri Dec 23, 2011 9:34 pm
by moob
Kiffi wrote:this is a little bit faster:

Code: Select all

    HideGadget(#ListIcon_41, #True) ; <--- !
    While Eof(6) = 0 
        [...]
    Wend
    HideGadget(#ListIcon_41, #False) ; <--- !
and: you don't need the OpenFile()-Line

Greetings ... Kiffi
yes is little bit, but continues slowly

if i disable is the same :(

thanks Kiffi

Re: Read text file is very slow

Posted: Fri Dec 23, 2011 9:52 pm
by Arctic Fox
Try replacing this line

Code: Select all

Texto$ = Trim(ReadString(6))
with

Code: Select all

Texto$ = ReadString(6)
The Trim() isn't necessary for Texto$ since you trim NomesID$ and NsID$ later.

Also consider only updating the status progress bar for example every tenth line.

Code: Select all

If IDN % 10 = 0 : StatusBarProgress(#StatusBar_wndMain, 2, Loc(6) , #PB_StatusBar_Raised, 0,length) : EndIf

Re: Read text file is very slow

Posted: Fri Dec 23, 2011 10:10 pm
by netmaestro
Try this, you'll have to point it to your textfile:

Code: Select all

OpenWindow(0,0,0,640,480,"",#PB_Window_ScreenCentered|#PB_Window_SystemMenu)
CreateStatusBar(0, WindowID(0))
AddStatusBarField(#PB_Ignore)

ListIconGadget(0,0,0,300,400,"Number",140,#PB_ListIcon_GridLines)
AddGadgetColumn(0, 1, "Name", 156)

ButtonGadget(1, 100,410,80,20,"Load")

Repeat
  EventID = WaitWindowEvent()  
  Select EventID
    Case #PB_Event_Gadget
      If EventGadget() = 1
        SendMessage_(GadgetID(0), #WM_SETREDRAW, 0,0)
        If ReadFile(0, "c:\playerlist.txt")
          length = Lof(0)
          While Not Eof(0)
            Texto$ = Trim(ReadString(0)) 
            NomesID$ = Trim(StringField(Texto$, 1, ":"))
            NsID$ = Trim(StringField(Texto$, 2, ":"))
            StatusBarProgress(0, 0, Loc(0) , #PB_StatusBar_Raised, 0, length)
            AddGadgetItem(0, -1, NomesID$+Chr(10)+NsID$)
          Wend
        EndIf
        SendMessage_(GadgetID(0), #WM_SETREDRAW, 1, 0)
      EndIf
  EndSelect
  
Until EventID = #PB_Event_CloseWindow

Re: Read text file is very slow

Posted: Fri Dec 23, 2011 10:51 pm
by ts-soft
A faster version of netmaestros code:

Code: Select all

EnableExplicit

OpenWindow(0,0,0,640,480,"",#PB_Window_ScreenCentered|#PB_Window_SystemMenu)
CreateStatusBar(0, WindowID(0))
AddStatusBarField(#PB_Ignore)

ListIconGadget(0,0,0,300,400,"Number",140,#PB_ListIcon_GridLines)
AddGadgetColumn(0, 1, "Name", 156)

ButtonGadget(1, 100,410,80,20,"Load")

Define EventID, length, *mem, Text.s, Count, i
Define tmp.s, NomesID$, NsID$

Repeat
  EventID = WaitWindowEvent() 
  Select EventID
    Case #PB_Event_Gadget
      If EventGadget() = 1
        SendMessage_(GadgetID(0), #WM_SETREDRAW, 0,0)
        If ReadFile(0, "c:\playerlist.txt")
          length = Lof(0)
          *mem = AllocateMemory(length)
          If *mem
            ReadData(0, *mem, length)
            CloseFile(0)
            Text = PeekS(*mem, length, #PB_UTF8)
            FreeMemory(*mem)
            RemoveString(Text, #CR$)
            Count = CountString(Text, #LF$)
            For i = 1 To count
              tmp = StringField(Text, i, #LF$)
              NomesID$ = Trim(StringField(tmp, 1, ":"))
              NsID$ = Trim(StringField(tmp, 2, ":"))
              StatusBarProgress(0, 0, i, #PB_StatusBar_Raised, 0, count)
              AddGadgetItem(0, -1, NomesID$ + Chr(10) + NsID$)
            Next
          EndIf

        EndIf
        SendMessage_(GadgetID(0), #WM_SETREDRAW, 1, 0)
      EndIf
  EndSelect
 
Until EventID = #PB_Event_CloseWindow
Reading the text at ones in memory is always faster than line by line.

Greetings - Thomas

Re: Read text file is very slow

Posted: Fri Dec 23, 2011 11:04 pm
by rsts
In my experience, building/displaying a 9000 item listicongadget is a little slow even if everything is in memory.

There is a way to do it almost instantly, but it's not a standard PB listicongadget.

cheers

Re: Read text file is very slow

Posted: Sat Dec 24, 2011 12:44 am
by netmaestro
Let's strap a nice ACME rocket to the back of this coyote:

Code: Select all

#LVSICF_NOINVALIDATEALL = 1 
#LVN_ODCACHEHINT = #LVN_FIRST - 13

Global Dim Items.s(9000,2) 

If ReadFile(0, "c:\playerlist.txt")
  length = Lof(0)
  IDN=0
  While Not Eof(0)
    Items(IDN, 0) = Str(IDN)
    Texto$ = Trim(ReadString(0)) 
    Items(IDN, 1) = Trim(StringField(Texto$, 1, ":"))
    Items(IDN, 2) = Trim(StringField(Texto$, 2, ":"))
    IDN + 1
  Wend
  CloseFile(0)
Else
  Debug "Could not open the file!"
EndIf

Procedure WinProc(hwnd, msg, wParam, lParam) 
  result = #PB_ProcessPureBasicEvents 
  Select msg 
    Case #WM_NOTIFY 
      *pnmh.NMHDR = lParam 
      Select *pnmh\code 
        Case #LVN_ODCACHEHINT 
          result = 0  
        Case #LVN_GETDISPINFO 
          *pnmlvdi.NMLVDISPINFO = lParam 
          If *pnmlvdi\item\mask & #LVIF_TEXT = #LVIF_TEXT
            *pnmlvdi\item\pszText = @Items(*pnmlvdi\item\iItem, *pnmlvdi\item\iSubItem) 
          EndIf 
      EndSelect 
  EndSelect 
  ProcedureReturn result 
EndProcedure 

OpenWindow(0, 0, 0, 640, 480, "", #PB_Window_SystemMenu | #PB_Window_ScreenCentered)
SetWindowCallback(@WinProc(),0) 
ListIconGadget(0,0,0,300,400,"Number",50,#PB_ListIcon_GridLines|#LVS_OWNERDATA) 

SendMessage_(GadgetID(0), #LVM_SETITEMCOUNT, IDN, #LVSICF_NOINVALIDATEALL) 
AddGadgetColumn(0, 1, "ID",   60) 
AddGadgetColumn(0, 2, "Name", 140) 

Repeat : Until WaitWindowEvent() = #PB_Event_CloseWindow
Thanks to Sparkie for guidance and hints from here: http://www.purebasic.fr/english/viewtop ... 38&start=4

Re: Read text file is very slow

Posted: Sat Dec 24, 2011 2:07 am
by moob
thanks guys...

the netmaestro is fast and ts-soft too.


thanks to all

Re: Read text file is very slow

Posted: Fri Jan 13, 2012 9:25 am
by oryaaaaa
Fast reading method
1. ReadData use All text data
2. List Buffer use separeting text data
3. Foreach next

Code: Select all

Procedure OpenPlayerLista(*retinort)
  ; You must use Protected, coz safety
  Protected Pattern.s, Pattern_no.l, FileR.s, IDN.l, length.l, Texto.s, NomesID.s, NsID.s
  Protected *TextBuffer, TextPos, File.l
  Protected NewList TextBuffer.s()
  ; 1st Read All data to Memory
  Pattern = "playerlist.txt (*.txt)|playerlist.txt"
  Pattern_no = 0
  FileR = OpenFileRequester("Please choose the playerlist...", "", Pattern, Pattern_no)
  ; FileSize_check
  If FileSize(FileR)>0
    ClearGadgetItems(#ListIcon_41)
    StatusBarText(#StatusBar_wndMain, 1, "Loading Playerlist wait...", #PB_StatusBar_Center)
    DisableMenuItem(#Menu_wndMain, #Menu_wndMain_OpenPlayerlist, 1) 
    ; My Tips, Very fast
    File = ReadFile(#PB_Any, FileR)
    length = Lof(File)
    *TextBuffer = AllocateMemory(length)
    ReadData(File, *TextBuffer, length)
    CloseFile(File) 
    ; If FASM use, more fast. -1 is abc chr13+10. 0 is abc chr13+10 def.
　　SetPriorityClass_(GetCurrentProcess_(),#REALTIME_PRIORITY_CLASS) 
    For TextPos = 0 To CountString(PeekS(*TextBuffer,length), Chr(13)+Chr(10))-1
      AddElement(TextBuffer())
      TextBuffer() = StringField( PeekS(*TextBuffer,length,#PB_UTF8), TextPos, Chr(13)+Chr(10) )
    Next
    FreeMemory(*TextBuffer)
    ; This is Safety.
    ForEach TextBuffer()
      TextBuffer() = Trim( TextBuffer() )
      NomesID = Trim( StringField( TextBuffer(), 1, ":") )
      NsID = Trim( StringField( TextBuffer(), 2, ":") )
      AddGadgetItem(#ListIcon_41, -1, Str( ListIndex( TextBuffer() ) )+Chr(10)+NomesID+Chr(10)+NsID)
    Next
　　SetPriorityClass_(GetCurrentProcess_(),NORMAL_PRIORITY_CLASS) 
    DisableMenuItem(#Menu_wndMain, #Menu_wndMain_OpenPlayerlist, 0) 
    StatusBarText(#StatusBar_wndMain, 2, "PlayerList Load Successfully", #PB_StatusBar_Center)
    StatusBarText(#StatusBar_wndMain, 1, "PlayerList Loaded", #PB_StatusBar_Center)
  EndIf
EndProcedure

Re: Read text file is very slow

Posted: Fri Jan 13, 2012 9:44 am
by Fred
I don't think then readstring() is the bottleneck here, since a while all file operation are heavily cached by PureBasic for maximum efficiency. You can still put a larger internal filebuffer if you think it could help: http://www.purebasic.com/documentation/ ... ssize.html

Re: Read text file is very slow

Posted: Fri Jan 13, 2012 10:02 am
by oryaaaaa
Danger Tips
coz
Return value : This function has no return value.

"Invalid memory Accesss error" never?

I used FileBuffersSize() before, but very slow.
Benchmark test result ReadData win.

Re: Read text file is very slow

Posted: Sat Jan 14, 2012 3:08 pm
by Vitor_Boss®
In my application the ReadData() is 3-5 times faster than ReadText().

For this kind of application I use 2 procedures, one to get the text file and another to split a delimiter with unlimited size.

Seems the author of the thread speak portuguese, I have some code here if you want.
All times are UTC+01:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited