[Solved] Fast split a string of text into lines

[BarryG]

Hi, I need to quickly split a very large string of Base64 text into separate lines of X characters each (so StringField() is no good here). The code below works great, but is extremely slow on large strings of 1 MB or more, and the "crop" variable can be anything the user specifies. Can someone please optimize it for me? I believe you may need to use "SizeOf(Character)" and While/Wend with memory blocks, from what I've seen before?

Code: Select all

For n=1 To 52
  s$+Chr(Random(90,65))
Next
o$=s$

crop=5
While Len(s$)>crop
  c$+Left(s$,crop)+#CRLF$
  s$=Mid(s$,crop+1)
Wend
If s$
  c$+s$
EndIf

MessageRequester("Result","Original:"+#LF$+o$+#LF$+#LF$+"Cropped:"+#LF$+c$)

[Edit] I worked it out! Here's my working code, that I based off other examples in this forum:

Code: Select all

#SOC=SizeOf(Character)

Procedure.s SplitTextIntoEvenLines(text$,length=100)
  *i.Character=@text$
  *o.Character=AllocateMemory(StringByteLength(text$)+Len(text$)*(2*SizeOf(Character)),#PB_Memory_NoClear)
  *o_orig=*o
  max=length+1
  While *i\c
    n+1
    If n=max
      PokeS(*o,#CRLF$)
      *o+2*#SOC
      n=0
    Else
      *o\c=*i\c
      *o+#SOC
      *i+#SOC
    EndIf
  Wend
  text$=PeekS(*o_orig)
  FreeMemory(*o_orig)
  ProcedureReturn text$
EndProcedure

For n=1 To 52
  s$+Chr(Random(90,65))
Next

MessageRequester("Result","Original:"+#LF$+s$+#LF$+#LF$+"Cropped:"+#LF$+SplitTextIntoEvenLines(s$,5))

[skywalk]

Search the forum with the text of your post topic. There are many flavors of split() and join() functions in the forum.

[infratec]

Code: Select all

EnableExplicit

Procedure.s SplitTextIntoEvenLines(text$, length.i=100)
  
  Protected *in, *out, InPtr.i, OutPtr.i, Result$, ByteLength.i, CRLFSize.i
  
  
  ByteLength = length * SizeOf(Character)
  CRLFSize = 2 * SizeOf(Character)
  *in = @text$
  *out = AllocateMemory(StringByteLength(text$) + Len(text$) * CRLFSize, #PB_Memory_NoClear)
  If *out
    While OutPtr + ByteLength < MemorySize(*out)
      CopyMemory(*in + InPtr, *out + OutPtr, ByteLength)
      InPtr + ByteLength
      OutPtr + ByteLength
      PokeS(*out + OutPtr, #CRLF$, -1, #PB_String_NoZero)
      OutPtr + CRLFSize
    Wend
    If OutPtr + ByteLength < MemorySize(*out)
      CopyMemory(*in + InPtr, *out + OutPtr, MemorySize(*out) - OutPtr)
    EndIf
    Result$ = PeekS(*Out, MemorySize(*out) / SizeOf(Character))
    FreeMemory(*out)
  EndIf
  
  ProcedureReturn Result$
  
EndProcedure

Define n.i, s$

For n = 1 To 52
  s$ + Chr(Random(90,65))
Next

Debug "Original: "
Debug s$
Debug ""
Debug "Cropped:"
Debug SplitTextIntoEvenLines(s$, 5)

[BarryG]

infratec, I edited my first post because I woke up today with a fresh mind and worked out what I was doing wrong. But I want to thank you for taking the time to help me anyway! I appreciate your willingness to assist, and it's always good to see alternate versions as well. Thumbs up!

[infratec]

Hi,

a bit speed optimized.
Please compare the speed of the different versions.
Debugger disabled! (Messagerequester for output)

Code: Select all

EnableExplicit

Procedure.s SplitTextIntoEvenLinesBKK(text$, length.i=100)
 
  Protected *in, *out, Result$, ByteLength.i, CRLFSize.i, *outHelp, *outEnd
 
 
  ByteLength = length * SizeOf(Character)
  CRLFSize = 2 * SizeOf(Character)
  *in = @text$
  *out = AllocateMemory(StringByteLength(text$) + (Len(text$) / length) * CRLFSize, #PB_Memory_NoClear)
  If *out
    *outHelp = *out
    *outEnd = *out + MemorySize(*out)
    While *outHelp + ByteLength < *outEnd
      CopyMemory(*in, *outHelp, ByteLength)
      *in + ByteLength
      *outHelp + ByteLength
      PokeS(*outHelp, #CRLF$, -1, #PB_String_NoZero)
      *outHelp + CRLFSize
    Wend
    If *outHelp < *outEnd
      CopyMemory(*in, *outHelp, *outEnd - *outHelp)
    EndIf
    Result$ = PeekS(*out, MemorySize(*out) / SizeOf(Character))
    FreeMemory(*out)
  EndIf
 
  ProcedureReturn Result$
 
EndProcedure

Define n.i, s$

For n = 1 To 52
  s$ + Chr(Random(90,65))
Next

Debug "Original: "
Debug s$
Debug ""
Debug "Cropped:"
Debug SplitTextIntoEvenLines(s$, 5)

[infratec]

I did a bugfix above at AllocateMemory()

If the length parameter is larger, my version is a bit faster.
For lower values of length, your version is faster.

[BarryG]

Thanks for the bug fix and speed tests. My length parameter will always be 100, so I'll use your code after all.