PureBasic Forums - English

http://www.purebasic.com
https://www.purebasic.fr/english/

Going Loopy

https://www.purebasic.fr/english/viewtopic.php?t=74687

Page 2 of 2

Re: Going Loopy

Posted: Wed Feb 26, 2020 6:05 pm
by Evil1
Marc56us wrote:Hum, bug :oops:
Otherwise the program will always look for the same term (the one given at the opening).
Yes I noticed that :-)

Re: Going Loopy

Posted: Wed Feb 26, 2020 7:00 pm
by mk-soft
mk-soft wrote:Don't forget...
That not work with Linux or maOS
I don't have mac and rarely code PB on Linux. What part is not compatible ?
You can't change gadget properties from threads... This crashed
Show signature ThreadToGUI

Re: Going Loopy

Posted: Thu Feb 27, 2020 9:16 am
by Marc56us
You can't change gadget properties from threads... This crashed
Show signature ThreadToGUI
Hi,

Thank for this information, but is this still valid with recent versions of PB?

I have tested my program (and yet not with the latest version of PB)

- PB 5.61 x64 (demo)
- Linux Slackware 14
- XFCE 4.12

+ No crashes with this program never.
+ The status bar (progress + text) is updated throughout the program by the Thread. (Add Delay(10) to realy see)

So Question 2020:
1. Does it apply to all gadgets or only to some? (which ones?) StatusBar Work (is it a gadget ?)
2. Can anyone test with other (Linux) configurations?

PS. I don't have a Mac and I don't intend to buy one since Apple has announced that next will be running on ARM CPUs which will cause problems for PB. The question that arises is whether to continue developing the Mac version of PB (or put on hold)

When I have time, I'll test other gadgets to see if it's just the SB that has a particular mode of operation.

Re: Going Loopy

Posted: Thu Feb 27, 2020 10:05 am
by mk-soft
With the statement of Fred, you must not change gadgets from threads.

With Linux this will sporadically crash.
With macOS it usually crashes immediately.

Re: Going Loopy

Posted: Thu Feb 27, 2020 2:52 pm
by Evil1
Hi MarcUS56, I found some code that Wilbert wrote in ASM for counting lines in files and replaced your code block with his :-
While Not Eof(hFile)
ReadString(hFile)
Lines + 1
Wend
CloseFile(hFile)

Code: Select all

Procedure CountFileByte(filename.s, byte)
  Protected result, bytes_read
  Protected buffer_size = 65536
  Protected f = ReadFile(#PB_Any, filename)
  Protected *mem = AllocateMemory(buffer_size + 48)
  Protected *pos = (*mem + 15) & -16
  While Not Eof(f)
    bytes_read = ReadData(f, *pos, buffer_size)
    If bytes_read <> buffer_size
      FillMemory(*pos + bytes_read, 32, ~byte)
      bytes_read = (bytes_read + 31) & -32
    EndIf
    !pxor xmm5, xmm5
    !movzx edx, byte [p.v_byte]
    !imul edx, 0x01010101
    !movd xmm4, edx
    !pshufd xmm4, xmm4, 0
    !mov edx, 0x01010101
    !movd xmm3, edx
    !pshufd xmm3, xmm3, 0
    !movd xmm0, [p.v_result]
    CompilerIf #PB_Compiler_Processor = #PB_Processor_x86
      !mov edx, [p.p_pos]
      !mov ecx, [p.v_bytes_read]
      !cfb_loop32b:
      !movdqa xmm1, [edx]
      !movdqa xmm2, [edx + 16]
    CompilerElse
      !mov rdx, [p.p_pos]
      !mov rcx, [p.v_bytes_read]
      !cfb_loop32b:
      !movdqa xmm1, [rdx]
      !movdqa xmm2, [rdx + 16]
    CompilerEndIf
    !pcmpeqb xmm1, xmm4
    !pcmpeqb xmm2, xmm4
    !pand xmm1, xmm3
    !psubb xmm1, xmm2
    !psadbw xmm1, xmm5
    !paddq xmm0, xmm1
    CompilerIf #PB_Compiler_Processor = #PB_Processor_x86
      !add edx, 32
      !sub ecx, 32
    CompilerElse
      !add rdx, 32
      !sub rcx, 32
    CompilerEndIf
    !jnz cfb_loop32b
    !pshufd xmm1, xmm0, 00001110b
    !paddq xmm0, xmm1
    !movd [p.v_result], xmm0
  Wend
  CloseFile(f)
  FreeMemory(*mem)
  ProcedureReturn result
EndProcedure

Code: Select all

lines = CountFileByte(file$, 13)
I can now read the amount of lines in a 1 GB file in 5 seconds on my laptop compared to 52 seconds using the PB code!

Overall to load the 1 GB file, read the lines and then loop through the log looking for ERROR xx now takes a total of 1m47s with the ASM down from nearly 3mins without.

it's mainly due to the findstring and mid commands in my loop that is slowing the program down so much :-

Code: Select all

If FindString(text$,"ERROR") = 21 And FindString(text$,":\")
        st=FindString(text$,"(0x")
        err$=Mid(text$,27,st-1)
        AddGadgetItem(Editor,-1,text$):err+1
      Endif
The above is looking for the word ERROR on each line of the log file and if ERROR appears 21 characters in, it then checks the starting position of (0x) so it can determine where the ERROR No. stops so it can store it in string err$

2020/01/25 11:58:09 ERROR 0 (0x00000000)
2020/02/06 16:07:09 ERROR 31 (0x0000001F) Copying NTFS Security to Destination File
2020/01/23 20:06:46 ERROR 1920 (0x00000780) Copying File

Do you know of a quicker way to do this check? - obviously ASM but that is far too complicated for this. Just want to know if my logic is the quickest.

Re: Going Loopy

Posted: Thu Feb 27, 2020 3:46 pm
by Marc56us
Interesting :)
With PB I'll search (I don't know the assembler at all), but the ASM pros will probably find it first.

Difficult to compare since we don't have the same machine I think.
To get an idea, can you test with the Windows command line tools?
Here's what I get on my PC (i7) with wc.exe and findstr.exe

Count lines with wc.exe (word count) port for windows (present in GIT tools if installed, otherwise, just load it on the net (unix tools))
https://en.wikipedia.org/wiki/Wc_(Unix)

Code: Select all

C:\>"C:\Program Files\Git\usr\bin\wc.exe" -l Txt_1_GB.txt
6008007 Txt_1_GB.txt
2 sec

Search text with Windows tool findstr.exe
I have added your sample at end of my 1 Gb test file (6 008 007 lines)

Code: Select all

C:\>findstr "20:06:46" Txt_1_GB.txt
3 sec

My numbers are probably not very significant, because my file is on a SSD and I have a lot of RAM

For question 2: If the word ERROR is still at same position, it will be faster to compare string instead of use findstring

Code: Select all

If Mid(Txt$, 21, 5) = "ERROR"
    Debug "Error: " + Mid(Txt$, 32, 8)
EndIf
To extract error code, it's also possible to use RegEx "ERROR.+0x(.+?)\)"

:wink:

Edit: Excellent ASM (40x faster), I keep in templates :P

Code: Select all

Counting lines, Please Wait...
EOF
8.60 secs
ASM
0.22 secs
Log analysis operations are often very long, that's why most of the time we do it in batch at night.
(There was a time when the nights were never long enough to finish the jobs.)
Now one of the main concerns of system administrators and DBAs is to prevent some users from running statistics all the time and not knowing how to wait for the result. 8)

Re: Going Loopy

Posted: Fri Feb 28, 2020 11:19 pm
by Evil1
So the initial thread issue slowdown when compiled vs running in the IDE was down to my attempt at a progressbar :-

Code: Select all

If ElapsedMilliseconds()-starttime >=20:starttime.q=ElapsedMilliseconds():SetGadgetState(ProgressBar_0,pg):EndIf
          If ElapsedMilliseconds()-starttime >=20:starttime.q=ElapsedMilliseconds():SetGadgetText(tim,Str(ElapsedMilliseconds()/1000-ltim/1000)+" Second(s)"):EndIf
using the progressbar code from Marc56US's code did not have this issue, I thought I was being smart, I guess not!

I created a quick and dirty Robocopy log generator :-

Code: Select all


s$="-------------------------------------------------------------------------------"+#CRLF$+
    "ROBOCOPY     ::     Robust File Copy For Windows                              "+#CRLF$+
    "-------------------------------------------------------------------------------"+#CRLF$+#CRLF$+
    
    "  Started : 22 January 2020 08:00:07"+#CRLF$+
    "   Source : C:\test\test"+#CRLF$+
    "     Dest : \\testfiler\testfiler\test\"+#CRLF$+#CRLF$+
    
    "   Files : *.*"+#CRLF$+#CRLF$+
    	    
    "Exc Files : autorun.inf"+#CRLF$+
                              " "+#CRLF$+	    
                              "Options : *.* /V /TEE /S /E /COPYALL /PURGE /MIR /ZB /XO /MT:4 /R:1 /W:1 "+#CRLF$+
                              " "+#CRLF$+
                              "------------------------------------------------------------------------------"+#CRLF$
    

  e$="------------------------------------------------------------------------------"+#CRLF$+#CRLF$+

  "              Total     Copied    Skipped  Mismatch    FAILED    Extras"+#CRLF$+
  "    Dirs :    161280    161273    161272          0        21         0"+#CRLF$+
  "    Files :   1240450   1240290        17         0       143         0"+#CRLF$+
   "   Bytes :   4.414 t   4.399 t       850         0  15.803 g         0"+#CRLF$+
   "   Times : 377:26:21  60:49:51                       0:00:00  15:28:13"+#CRLF$+
   "   Ended : 25 January 2020 12:18:12"+#CRLF$+#CRLF$
  
err$+"2020/01/21 08:32:46 ERROR 31 (0x0000001F) Copying NTFS Security To Destination File c:\test\test\test.doc"+#CRLF$+
"2020/01/21 08:32:46 ERROR 1 (0x0000001F) Copying NTFS Security To Destination File c:\test\test\test.doc"+#CRLF$+
"2020/01/21 08:32:46 ERROR 1920 (0x0000001F) Copying NTFS Security To Destination File c:\test\test\test.doc"+#CRLF$

r$="random line"+#CRLF$

CreateFile(0,"C:\data\Robocopytest.log")
WriteString(0,s$)
While FileSize("C:\data\Robocopytest.log")<1073741821
  x=Random(500,1)
    For a = 1 To x
      WriteString(0,r$)
    Next a

    x=Random(1500,0)
   If x=20
    For a = 1 To x
      WriteString(0,err$)
    Next a
   EndIf
Wend

WriteString(0,e$)
CloseFile(0)
Using the above code I generated a 1GB (ish) dummy Robocopy log file :-

https://1drv.ms/u/s!AjXyvTjHDjqbgohEN2AgibbULcmjig

Because the above log file is full of repetitions, when compressed the 1GB log file is only 62KB

So with the help of Marcus56US's code and Wilberts ASM:-

Code: Select all

lines = CountFileByte(file$, 13) ;All Wilbert
     Protected Factor = Lines / 20
                
      If ReadFile(hfile,file$)
          s1=0:length2 = FileSize(File$)
                              
          While Not Eof(hfile)
             
          x+1 : x2+1
          If x / factor = x % Factor
          StatusBarProgress(0,0,x,#PB_StatusBar_Raised, 0, lines)
          EndIf
          Text$ = ReadString(hfile)
                                    
         ps=FindString(text$,"Started : "):If ps<>0:s1+1:started(s1)=Mid(text$,ps+10,Len(text$)):spos(s1)=(x):Debug "started : "+started(s1):ps=0:EndIf
         ps=FindString(text$,"Ended : ")  :If ps<>0:s2+1:ended(s2)=Mid(text$,ps+8,Len(text$)):epos(s2)=(x):Debug "ended : "+ended(s2):ps=0:EndIf
         ps=FindString(Text$,"Dest : ")   :If ps<>0:s3+1:Destination(s3)=Mid(text$,ps+7,Len(text$)):Debug "Dest : "+ Destination(s3):ps=0:EndIf
         ps=FindString(text$,"Options : "):If ps<>0:s4+1:Options(s4)=Mid(text$,ps+10,Len(text$)):Debug "Options : "+Options(s4):ps=0:EndIf
             
      If x>=spos(curon) And s1=curon
       If epos(curon)=0 Or x<=epos(curon)
                         
    If FindString(text$,"ERROR") = 21 And FindString(text$,":\")
      st=FindString(text$,"(0x")
      err$=Mid(text$,27,st-1)
      
        If run=0:errcount(Val(err$))+1
           AddGadgetItem(Editor,-1,text$):err+1
        EndIf
        
        If run=1
          If checked(Val(err$))=0
            AddGadgetItem(Editor,-1,text$):err+1
          EndIf
        EndIf
      EndIf
    EndIf
  EndIf
 Wend
I did a little experimentation with Left, Mid, Findstring and Regex and Findstring is definitely faster overall
Interested to see if anyone has a better/faster and smarter approach to searching a file with the search criteria I am using.
All times are UTC+01:00
Page 2 of 2

Powered by phpBB® Forum Software © phpBB Limited