renamed: General Scintilla Questions

[Zach]

I have renamed the thread since its expanded outside the scope of my initial question

I've been trying to solve a curious anomaly I'm running into where the reported positions need to have offsets applied to them to correctly affect things in the document.
It seems to be related to how I'm sending new line commands. One thing to keep in mind here is I am working entirely through the SendMessage system, and I am NOT operating on the buffer at the byte level - this is beyond my ability at the moment.


That being said. During initial testing I have been using

Chr(10)

Where I wish to start new lines. When doing operations that rely on document positions (setting styling / find and replace, etc) I have to use an offset of +1
So then I thought I would try the CRLF$ constant. Well that went in the other direction, now I needed to offset operations by +2

I'm looking for a solution I can use that requires no offset, if possible. Something I could set as a constant that works the same across Windows/Mac/Linux if possible.
My main fear is that the behavior will be unpredictable on search matches, and in some cases it may actually report the starting position of the match, instead of the gap in front of it.

I've noticed even GoScintilla (what little I understand of it) appears to use position offsets, so maybe I am barking up a tree uselessly...
Is there an actual solution? Or is it going to come down to "find what works on each OS and build your offset into the top of your text handling process"

[spikey]

Is there a BOM/not a BOM at the top of the file? (Just wondering.)

[Zach]

No idea, to be honest. I'm not opening files, I'm just printing direct to the document (treating it like an EditorGadget) with append commands, whatever Scintilla's default may be.
I set initial text with SETTEXT, then use APPENDTEXT after that (seems to require a SET to work, but I have an idea on how to get around that, I will try out later)

If I completely remove any kind of newline inserts from the code it seems to function properly, with the exception of search/replace, which still requires a +1 offset, I suspect in that case because its locating the gap in front of the match instead (docs say it will match either the word or the gap in front of it, something to do with multi-byte characters idk... )

edit:

gotta get some sleep before work, but here is a scratchpad of test code I've slowly built up. There are a lot of potential red herrings as I have unfinished ideas in the code, empty procedures, commented out stuff, etc. Apologies, haven't coded in ~5 years or so. But this should compile and highlight the problem at the very least.

Code: Select all

;// Enumerate some Constants for object handles
Enumeration Window
  #MainWindow
EndEnumeration

;// Scintilla specific
Enumeration
  #SCINT_VISIBLE
  #SCINT_SCRATCHPAD
EndEnumeration




;// Structures
Structure ColorTable
  name.s
  style.i
EndStructure

;// Make a list to hold our map for translating text styling markup
;// to style indexes

NewList Color_Table.ColorTable()

Procedure BuildColorTable(name.s, style.i)
  Shared Color_Table()                ;// give access to the translation table
  
  ForEach Color_Table()               ;// iterate through the list
    If Color_Table()\name = name.s
      ProcedureReturn #False          ;// If the entry already exists, break out of the loop and end procedure
    EndIf
  Next
  
  AddElement(Color_Table())           ;// Add a new entry to the color table
  Color_Table()\name = name.s         ;// assign color name  (red, blue, whatever)
  Color_Table()\style = style.i       ;// assign the style index number used for Scintilla's styles
  ProcedureReturn #True               ;// after iterating entire list without error, end procedure.
EndProcedure



;// Define some useful shit like a procedure
;// to automatically feed a string to a Scintilla
;// gadget without having to type all this crap


Procedure SciPrint(Text.s)    ;// Prints text to a new line when added
  
  ;// originally this was simply   Text.s = Chr(10) + Text.s
  CRLF.s = Chr(10)
  *NewLine = UTF8(CRLF.s)
  ScintillaSendMessage(0, #SCI_APPENDTEXT, Len(CRLF.s), *NewLine)
  Text.s =  Text.s
  *Text=UTF8(Text.s)
  ScintillaSendMessage(0, #SCI_APPENDTEXT, Len(Text.s), *Text)
  FreeMemory(*Text)
  ScintillaSendMessage(0, #SCI_DOCUMENTEND)
EndProcedure



Procedure SciPrintStyled(Text.s)
  
EndProcedure






If OpenWindow(0, 0, 0, 640, 480, "ScintillaGadget", #PB_Window_SystemMenu | #PB_Window_ScreenCentered)
   
    If InitScintilla()
      ContainerGadget(10,10,10,310, 70,#PB_Container_Flat)
        ScintillaGadget(#SCINT_VISIBLE, -2, -2, 310, 70, 0)
        
        ;// Make a second Scintilla Gadget
        ScintillaGadget(#SCINT_SCRATCHPAD, 0, 0, 100, 50, 0)
        ;// Hide the scratchpad
        HideGadget(#SCINT_SCRATCHPAD, #True)
        
      CloseGadgetList()
      
      
      
      ;//==========================================
      ;// Set all Styles to a default state
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_STYLESETBACK, 32, RGB(0,0,0))
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_STYLECLEARALL)
      
      ;// Output set to red color
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_STYLESETFORE, 0, RGB(255, 0, 0))
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_STYLESETFORE, 1, RGB(255, 0, 255))
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_STYLESETFORE, 2, RGB(255, 255, 255))
      
      ;// Build Color Table
      
      BuildColorTable("red", 0)
      BuildColorTable("hotpink", 1)
      BuildColorTable("white", 2)
      
      ;//==========================================
      
      ;// Set some styles up for the scratchpad
      ScintillaSendMessage(#SCINT_SCRATCHPAD, #SCI_STYLESETFORE, 0, RGB(137, 137, 200))
      
      
      ;// Turn on Word Wrap.
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETWRAPMODE, #SC_WRAP_WORD)
      
      ;// Set Margin to 0px
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETMARGINWIDTHN, 1, 0)
      
      ;// Set Caret to Invisible
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETCARETWIDTH, 1)
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETCARETFORE, RGB(0, 255, 0))
      
      ; Set the initial text to the ScintillaGadget
      *Text=UTF8("This is a simple ScintillaGadget with text...")
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETTEXT, 0, *Text)
      FreeMemory(*Text) ; The buffer made by UTF8() has to be freed, to avoid memory leak
     
      ; Adding a second line of text with linebreak before
      Text$ = Chr(10) + Chr(10) + "Second line"
      *Text=UTF8(Text$)
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_APPENDTEXT, Len(Text$), *Text)
      FreeMemory(*Text)
      
      ;//Test Scrolling
      
      SciPrint("Hey This is a test")
      
      SciPrint("And some more stuff added to the screen")
      SciPrint("")
      SciPrint("The line before this was a blank line. I inserted a character return")
      SciPrint("But this line should print directly below the line above it with no gap at all")
      SciPos = ScintillaSendMessage(#SCINT_VISIBLE, #SCI_GETCURRENTPOS, 0, 0)
      SciPrint(Str(SciPos))
      SciPos2 = ScintillaSendMessage(#SCINT_VISIBLE, #SCI_GETCURRENTPOS, 0, 0) ;// + 1
      SciPrint(Str(SciPos2))
      SciPos3 = ScintillaSendMessage(#SCINT_VISIBLE, #SCI_GETCURRENTPOS, 0, 0) ;// + 1
      SciPrint(Str(SciPos3))
      
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_STARTSTYLING, SciPos2, 0)
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETSTYLING, Len(Str(SciPos2)), 1)
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_STARTSTYLING, SciPos3, 0)
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETSTYLING, Len(Str(SciPos3)), 2)
      
      DbgOut$ = "*SciPos2 is " + Len(Str(SciPos2)) + " characters long."
      Debug DbgOut$
      
      
      ;// Here we are going to test sending text to the scratchpad, then reading
      ;// it back as styled, using the default style settings.  Then we will try
      ;// and copy and paste it over to the visible control
      *Text = UTF8("This line should appear styled when it is copied over to the visible Scintilla control.")
      ScintillaSendMessage(#SCINT_SCRATCHPAD, #SCI_APPENDTEXT, Len(Str(*Text)), *Text)
      FreeMemory(*Text)
      
      LineCount.i = ScintillaSendMessage(#SCINT_VISIBLE, #SCI_GETLINECOUNT)
      Debug "Raw Line Count: " + Str(LineCount)
      WrapCount.i = ScintillaSendMessage(#SCINT_VISIBLE, #SCI_WRAPCOUNT, 7)
      Debug "Wrapped Line Count: " + Str(WrapCount)
      
      
      ;ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETCURRENTPOS, 0, 0)
      ;ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETANCHOR, 0, 0)
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_DOCUMENTEND)
      SciPrint("<color=white>Adding a new line")
      
      ;// Set search start to entire document
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETTARGETSTART, 0, 0)
      ;// Set the search end to the end of the document
      ;ScintillaSendMessage(#SCINT_VISIBLE, #SCI_DOCUMENTEND)
      SelEndPos = ScintillaSendMessage(#SCINT_VISIBLE, #SCI_GETCURRENTPOS)
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETTARGETEND, SelEndPos)
      Debug "Document Start Position: " + Str( ScintillaSendMessage(#SCINT_VISIBLE, #SCI_GETTARGETSTART) )
      Debug "Document End Position: " + Str( ScintillaSendMessage(#SCINT_VISIBLE, #SCI_GETTARGETEND) )
      
      ;// It took me forever to figure this out, some of this API is retarded
      ;// the following code will establish a search term, and then search
      ;// inside the Scintilla control for instances of that term.
      ;// When the instance is found, the position within the document is
      ;// returned.  Length must be set by TARGET search term.
      
      TextSearch$ = "<color=" + Color_Table()\name
      
      *txtSearch = UTF8(TextSearch$)  ;// Test String to search for
      MatchLocation = ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SEARCHINTARGET, Len(TextSearch$), *txtSearch)
      MatchEnd = MatchLocation + Len(TextSearch$)
      Debug ~"Search Term \" " + TextSearch$ + ~" \"" + ", located at: " + MatchLocation
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETTARGETSTART, MatchLocation)
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SETTARGETEND, MatchEnd)
      EmptyStr$ = ""
      *EmptyStr = UTF8(EmptyStr$)
      
      ;// Unless you add a +1 to SCI_TARGETSTART or SCI_TARGETEND
      ;// a >  symbol is still left from the color tag.
      ScintillaSendMessage(#SCINT_VISIBLE, #SCI_REPLACETARGET, -1, *EmptyStr)
      Debug "Start Location: " + MatchLocation
      Debug "End Location: " + MatchEnd
      
      FreeMemory(*txtSearch)          ;// Free the memory allocated to the pointer
      FreeMemory(*EmptyStr) 
      
      
    EndIf
   
    Repeat : Until WaitWindowEvent() = #PB_Event_CloseWindow
  EndIf

[spikey]

Ok - now I'm up to speed!

When the caret is at a line end via a cursor movement, #SCI_DOCUMENTEND in this case, it will be positioned before the line end mark not after it (doing otherwise doesn't make sense - it's the end of the line!) When you use an LF this is one character long, when you use CRLF it's two characters long.

What you are doing is stying the line end mark - you just can't see this because it's invisible:-

Code: Select all

...above it with no 
266{start styling pink here}{LF}
27{stop styling here}0{start styling white here}{LF}
24{stop styling here}0{LF}...

You need to adjust the styling start point to be after the line end mark, or before the next line of text if you prefer - this is why you have to add 1 for an LF and two for a CRLF. That's the start position of the next piece of text. As long as your line ends are consistent the offsets will also be consistent.

There are a couple of ways to do it - just know that you need to adjust at the end of a line; or set the caret position to the start of the following line (once it exists - which would be a downside for this method).

But this line should print directly below the line above it with no gap at all

I can't replicate this problem, so I don't know what you're seeing there or what's causing it.

a > symbol is still left from the color tag.

This is because you haven't included the closing bracket in the search term - you get exactly the positions that you search for.
Either search for the full term - or adjust the terminating position if you don't want to.

[Zach]

Thanks, always good to have a second set of eyes.

I kind of figured the linefeed problem was what you mentioned, so I'll just plan it out so I'm account for the adjustment at a single point in the entire process.

Thanks for spotting the error with the closing bracket. That was pretty retarded on my part


edit: There is no "gap problem" with the above line you quoted. I think you misunderstood. That line was for the benefit of visual testing. It is rendered properly if there is no gap between it and the line above it. you have to consider the entire context

[Zach]

Ok now I've got another weird problem.

I booted into my VM, running PureBasic 5.72 on linux mint. Copy and pasted the code, compiles and runs fine.

But the scintilla control will not scroll... at all. I've tried sending it every command I could think of from the official docs, the caret is updating its position, etc. I just can't get the control to actually scroll and keep up with the new inputs/line feeds.


Can anyone replicate this using the code provided earlier?
I haven't set anything special with regards to rendering subsystems so I think its using GTK, if that matters.. (would have to double check what my desktop is).

[spikey]

There is no "gap problem" with the above line you quoted.

That would explain why I can't replicate it then!

I've tried sending it every command I could think of...

When you say this - what exactly have you tried? SCI_SCROLLCARET or SCI_SCROLLRANGE should do what you want.

[Zach]

I just did all my tests again. For simplicity I just tacked them on to the end of my code example, as that was an ideal place to programmatically force a scroll.
I have just tried everything listed, to the best of my ability. Something is either seriously wrong on my end, or maybe this is a weird bug

Code: Select all

#SCINT_VISIBLE = 0   ;// constanr for the gadget number
ScintillaSendMessage(#SCINT_VISIBLE, #SCI_LINESCROLL, 0, 10)    ;//  attempt to Scroll by number of columns, lines.
ScintillaSendMessage(#SCINT_VISIBLE, #SCI_LINESCROLLDOWN)   ;//  I stacked this one 10 times to simulate 10 down arrow key pushes
ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SCROLLTOEND)       ;//  Scroll to document end
ScintillaSendMessage(#SCINT_VISIBLE, #SCI_SCROLLCARET)      ;//  also fails, even when setting different positions

I did not try SCROLLRANGE because I'm having a problem mentally grasping how its intended for me to write that out in code... but I suspect it would not be very useful to me, as it would be extra steps and at this point nothing else is working so I think something is fundamentally wrong with Purebaisc... whether its a bug, or something I am not setting up correctly (I have never used PB in linux before), or something I need to be doing differently related to scintilla that is linux specific, I don't know.

But all of the above quoted methods work fine on Windows.
Of particular note is this screenshot. Look at the Vertical scrollbar for Scintilla, and then look at the Vertical scrollbar for the PB Editor; for some reason the scintilla gadget is not generating any whitespace around the content size indicator, and the indicator itself takes up the entire vertical height of the scrollbar. If I click on the scrollbar the Scintilla document will immediately jump down to the bottom/end of the document. I can scroll up/down with the mousewheel and up/down arrow keys only.

[Zach]

Ok, so a couple new developments.

The problem in the post directly above this one, relating to scrolling not working on properly Linux has been halfway resolved.
I installed the requirements for the QT subsystem and compiled the program using QT, Scintilla has proper scrolling and pretty closely matches the windows version of the control. I'm not sure why the default PB compile options would break it so bad... maybe my system is missing something required but not badly enough to halt compilation?

I have also been experimenting with cleaning up and simplifying some things, related to New Line inserts, etc. I took out everything and then started issuing ScintillaSendMessage(0, #SCI_NEWLINE) commands. This -appears- to insert the same as a #CRLF$ constant would if I inserted it into a string.
I've also seen some puzzling behavior changes in Scintilla itself, depending on where you insert them.

If you insert them at the end of the procedure like so

Code: Select all

Procedure SciPrint(Text.s)    ;// Prints text to a new line when added
 
  *Text=UTF8(Text.s)
  ScintillaSendMessage(0, #SCI_ADDTEXT, Len(Text.s), *Text)
  FreeMemory(*Text)
  ScintillaSendMessage(0, #SCI_DOCUMENTEND)
  ScintillaSendMessage(0, #SCI_NEWLINE)
EndProcedure

There will be no position offset issues with regards to setting targets for styling text, etc. The only caveat is that the final line printed always has a character return inserted, creating a blank space below it. My OCD dislikes this very much.
Similarly, if I move the NEWLINE command to the top of the procedure, it will break the position values and require an offset - and also the very first line will have a space in front of it. Again driving my OCD bonkers.
Furthermore, on Windows the required offset is +2 while on Linux the required offset is only +1
My OCD has rapidly become annoyed with this problem.

I tried getting clever and writing a test in, if it was position 0 (first line) then don't put the NEWLINE before the inserted text (or insert one at all). Otherwise insert the NEWLINE as normal, at the end of the procedure.
That idea quickly caused my OCD to declare nuclear war because for reasons unknown to my genius brain, doing this would COMPLETELY break the Scintilla widget. Document end position would throw itself into -1 other garbage may show up, styling is way off, etc.


So all in all, a rather unproductive day for me.

[JHPJHP]

Hi Zach,

See the following demo: Google Translate Service
- adding, inserting, replacing, and deleting lines of text on the fly

One option to avoid the LF, CRLF, etc. discrepancy between OS is to force End Of Line mode:

Code: Select all

ScintillaSendMessage(#HelpScintillaGadget, #SCI_SETEOLMODE, #SC_EOL_LF)
ScintillaSendMessage(#HelpScintillaGadget, #SCI_CONVERTEOLS, #SC_EOL_LF)

[Zach]

Thanks, I appreciate this snippet. I'll see if I can't make use of anything from it, after I've finished testing some other ideas

[Zach]

This is what I've ultimately come up with. I've spent too long trying to trackle the problem and want to move on, so I went with something I came up with that I could understand if I was looking at the code 5 years from now.

Code: Select all

Procedure SciPrint(Text.s)    ;// Prints text to a new line when added
  
  CaretPos.i = ScintillaSendMessage(#SCINT_VISIBLE, #SCI_GETCURRENTPOS)
  If CaretPos = 0
    *Text=UTF8(Text.s)
    ScintillaSendMessage(0, #SCI_ADDTEXT, Len(Text.s), *Text)
    FreeMemory(*Text)
    ScintillaSendMessage(0, #SCI_DOCUMENTEND)
    
  ElseIf CaretPos > 0
    ScintillaSendMessage(0, #SCI_DOCUMENTEND)
    ScintillaSendMessage(0, #SCI_NEWLINE)
    *Text=UTF8(Text.s)
    ScintillaSendMessage(0, #SCI_ADDTEXT, Len(Text.s), *Text)
    FreeMemory(*Text)
    ScintillaSendMessage(0, #SCI_DOCUMENTEND)
   
  EndIf
  
EndProcedure

I did try several experiments I thought had some merit but for unknown reasons did not work out.
A slightly different version of the above code involves keeping a NewLine at the top of the procedure each time text is added, and to prevent the gap with line 0 I'd issue a delete command to remove that line. IT still had issues with requiring different offsets on Linux vs Windows though. Not sure why.
Another version of that idea was me trying to keep the new line at the end of the procedure and then issuing the line delete there, resulting in no bottom gap - but that flat out wouldn't work. Not sure why.

For whatever reason, if you add a newline to Scintilla it completely breaks the position value and requires an offset. This can be mitigated by inserting your text FIRST, however even in an IF/ELSE chain where the NewLine is issued only in the second case (so its writing -something- to the control first) it will STILL break the position offsets. I'm not sure why it does that.

I settled on the above method, since it was going to break the offset anyway, and I'll just account for it in a spot in the code that handles the custom styling. The DocumentEnd may be redundant so I'll probably strip it out on the last line of the Else clause

[Jac de Lad]

Code: Select all

Procedure SciPrint(Text.s) ;// Prints text to a new line when added

If ScintillaSendMessage(#SCINT_VISIBLE, #SCI_GETCURRENTPOS) > 0
ScintillaSendMessage(0, #SCI_DOCUMENTEND)
ScintillaSendMessage(0, #SCI_NEWLINE)
EndIf

*Text=UTF8(Text.s)
ScintillaSendMessage(0, #SCI_ADDTEXT, Len(Text.s), *Text)
FreeMemory(*Text)
ScintillaSendMessage(0, #SCI_DOCUMENTEND)

EndProcedure

? A not shorter, but does the same.

[Jac de Lad]

*bit