StringField - Alterable functions - Feasibility demos

Share your advanced PureBasic knowledge/code with the community.
User avatar
mk-soft
Always Here
Always Here
Posts: 5313
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: StringField_BF - A alterable function - Feasibility demo

Post by mk-soft »

Maybe you can use this as a basis ...

Before that, clean up the beginning and the end with Trim(...).
Link: SplitString to list and array with option double-quotes
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: StringField_BF - A alterable function - Feasibility demo

Post by Saki »

My StringField tool is ready, these test codes here are no longer needed by me.
They just serve as code example.
地球上の平和
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: StringField - Alterable functions - Feasibility demos

Post by Saki »

Well, despite all the prophecies of doom,
in the first real-world application of creating data lines, I achieve an enormous increase in speed when using my tool StringFields_BF compared to the StringField of PB.
This is due to the intended different way of working of my tool.
Instead of creating about 800 data rows in one second, I can more as 100,000 data rows in one second with my tool.

This achieves exactly what should be achieved.

https://www.purebasic.fr/english/viewto ... 12&t=77248

https://www.purebasic.fr/english/viewto ... 12&t=77219
地球上の平和
User avatar
idle
Always Here
Always Here
Posts: 5018
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

Re: StringField - Alterable functions - Feasibility demos

Post by idle »

There seems to be something wrong with your code as I'm getting around 9000ms for stringField_BF vs 3ms for OK_StringField_BS. It does the same thing in a different way.

Code: Select all

DeclareModule StringField_BF
  EnableExplicit
  Declare.s StringField_BF(string$, index, separator$)
EndDeclareModule

Module StringField_BF
  Global NewList index() : AddElement(index())
  Global result$, length
  
   Procedure.s StringField_BF(string$, index, separator$)
    ; StringField - By Saki - Unicode - This code is free for using and enhancing
    Protected i, ii, pos_1, pos_2, length_result, comp, amount_indexes, count_index
    Protected len_separator=StringByteLength(separator$), *pointer.word, jump_in
    Protected *string=@string$, *separator=@separator$
    If Not PeekW(*string) : ProcedureReturn "" : EndIf
    If index<1 : ProcedureReturn "" : EndIf
    ClearList(index()) : AddElement(index())
    If comp=CompareMemory(*string, *separator, len_separator)
      jump_in=1 : index+1
    EndIf
    i=-2
    Repeat
      i+2
      comp=CompareMemory(*string+i, *separator, len_separator)
      If comp+jump_in
        count_index+1
        jump_in=0 : ii+1 : i+len_separator-2 : amount_indexes+1
        AddElement(index()) : index()=i+2
      EndIf 
      *pointer=*string+i+len_separator
    Until count_index>index Or Not *pointer\w
    AddElement(index()) 
    If index>count_index : index=count_index : EndIf
    amount_indexes=ii : i=0 
    If amount_indexes
      Repeat
        SelectElement(index(), i) : pos_1=index()
        SelectElement(index(), i+1) : pos_2=index()
        length_result=pos_2-pos_1-len_separator 
          If length_result>0 And pos_2-pos_1>0
            If length<length_result
              length=length_result
              result$=Space(length_result>>1)
            EndIf
            FillMemory(@result$, length, 0, #PB_Word)
            CopyMemory(*string+pos_1, @result$, length_result)
            If i=index : ProcedureReturn result$ : EndIf
          EndIf
        i+1
      Until i>index Or i>amount_indexes-1
    EndIf
  EndProcedure
EndModule
UseModule StringField_BF

Procedure OK_StringField_BS(*source,List StringFields.s(),separator=' ') 
  Protected *inp.Character,key.s    
  
  If *source 
    *inp = *source 
    While *inp\c <> 0
      While (*inp\c <> separator) 
        *inp+2 
      Wend 
      key = PeekS(*source,(*inp-*source)>>1)
      AddElement(StringFields()) 
      StringFields() = key 
      If *inp\c <> 0
        *inp+2
        *source = *inp 
      Else 
        Break 
      EndIf   
    Wend 
  EndIf 
   
  ProcedureReturn ListSize(StringFields()) 
  
EndProcedure 

Define separator$=" "
Define index, result$, string$

string$=" Hello I am a splitted string "

CompilerIf #PB_Compiler_Debugger 
  Define multiplier=2  
CompilerElse
  Define multiplier=9 
CompilerEndIf   

Define i
For i=1 To multiplier
  string$+string$
Next i

Debug Len(string$)
Debug "================"

Define len_string=Len(string$)

Define time=ElapsedMilliseconds()

For index=0 To len_string
  result$ = stringField_BF(string$, index, separator$)
  If result$<>"" : Debug result$ : EndIf
Next 

Define et = ElapsedMilliseconds() 

MessageRequester("", Str(et-time))

Debug "++++++OK_StringField_BS ++++++++"

Define time1=ElapsedMilliseconds()

Define NewList MySplitStringBS.s() 

OK_StringField_BS(@string$,MySplitStringBS()) 

ForEach MySplitStringBS() 
  result$ = MySplitStringBS()
  If result$ <> "" : Debug result$ : EndIf
Next 

Define et1 = ElapsedMilliseconds() 

MessageRequester("", Str(et1-time1))


User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: StringField - Alterable functions - Feasibility demos

Post by Saki »

Hi, I'm looking,
I don't have more than 5ms here on a first test with debugger and purifier.
Try converting a 3mb image into data lines with the Data Encoder.
For me it takes about half a second and create about 50,000 lines.
I am only tested on Windows 573

Should there be a strange problem.
Did you provide the solution right away, that's cool. 8)

But your variant is definitely faster as it seems, then it comes in anyway.

I have to take a closer look.
地球上の平和
User avatar
idle
Always Here
Always Here
Posts: 5018
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

Re: StringField - Alterable functions - Feasibility demos

Post by idle »

Saki wrote: Tue May 18, 2021 11:07 am Hi, I'm looking,
I don't have more than 3ms here.
Try converting a 3mb image into data lines with the Data Encoder.
For me it takes about half a second and create about 50,000 lines.
I'm talking about the 2nd code example in the 1st post of this topic that's supposed to be twice as fast, I copied it and compared in the previous post. What's the intent of your code? As it's not doing a good job of StringFields, change the multiplier=10 and it takes 50000ms vs 5ms for the OK_StringField_BS.
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: StringField - Alterable functions - Feasibility demos

Post by Saki »

Ah, I see.
It's been a while now, but the point was, as I recall, that constantly recreating a string consumed a lot of time.
A larger empty string was then created once before, zeroed and the lines copied into it.
Then the string was zeroed again to include the next line.
This method is dirty, but it has never caused any problems.
The code is purely experimental and has been quickly discarded.
It is only left at the top of the thread in fairness to show people how to do things and then let them decide if it is OK or wrong.
This code has also only been tested on Windows.
Usually I make everything at least runnable on Windows first, then the test on Linux and then on Mac.
Sometimes I think this has to run on all OS because it doesn't contain anything special and then one test doesn't work after all.
The code above has been discarded very early on, it hasn't seen Linux or Mac yet.
The codes above are purely experimental, just to see what works well and what doesn't.
In the final tool only the well working code parts were combined, which was then very quickly finished.

The problem with the StringField function in PB is that the entire string must be re-read for each new substring to be selected.
This problem can be very serious with large files.

The original idea was to use CopyMemory and some tricks to speed up this process without fundamentally changing the function.

However, it became clear very quickly that this does not work.
Besides, it is not easy to program such a function without errors if you also want to try code tricks.

the-weavster split the code into two parts, which I wanted to avoid.
The way he did it, I always did it with my CSV crypters, that's the usual way.

So I combined the old knowledge with the new code parts to create a universally applicable new tool.
In the new data lines tool I then replaced the there used
PB StringField function with my StringFields_BF function to see what it does.
This was extremely faster.
As a test, more as 700,000 lines created in about 1 second with the BF version, the PB Version about 800 lines in one second.
地球上の平和
User avatar
idle
Always Here
Always Here
Posts: 5018
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

Re: StringField - Alterable functions - Feasibility demos

Post by idle »

still think you should fix up the code in this thread as your example doesn't work as expected

I've posted examples of StringField methods here
https://www.purebasic.fr/english/viewto ... 43#p570243
List=1759ms Array=2094ms PB=7633ms Len = 928 reps 10000
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: StringField - Alterable functions - Feasibility demos

Post by Saki »

I removed it, not worth bothering with it further as it was purely experimental.
地球上の平和
Post Reply