It is currently Mon Aug 21, 2017 8:29 pm

All times are UTC + 1 hour




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: [5.44,5.60] Broken unicode support in ReadProgramString()
PostPosted: Thu Apr 13, 2017 9:56 am 
Offline
Enthusiast
Enthusiast
User avatar

Joined: Wed Jan 12, 2011 3:48 pm
Posts: 182
Location: Serbia
Try this code:
https://pastebin.com/dTWtVxwP

Results:
Image

It's the same for Linux.
If someone can test on Windows and other PB versions that will be great.

P.S. I tried to paste the code here, but it looks like phpBB also doesn't support emojis :)

_________________
PB examples collection
pb-osx-globalhotkeys
pb-osx-notifications


Last edited by deseven on Wed Apr 26, 2017 8:54 pm, edited 2 times in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: [5.44] Broken unicode support in ReadProgramString()
PostPosted: Wed Apr 19, 2017 11:24 am 
Offline
Enthusiast
Enthusiast
User avatar

Joined: Wed Jan 12, 2011 3:48 pm
Posts: 182
Location: Serbia
So, no one can confirm? Or no one is actually bothered?
Come on guys, it looks important enough.

Looks like PB community is slowly dying. Or is it just me?

_________________
PB examples collection
pb-osx-globalhotkeys
pb-osx-notifications


Top
 Profile  
Reply with quote  
 Post subject: Re: [5.44] Broken unicode support in ReadProgramString()
PostPosted: Wed Apr 19, 2017 11:47 am 
Offline
PureBasic Expert
PureBasic Expert

Joined: Sun Aug 08, 2004 5:21 am
Posts: 2960
Location: Netherlands
Yes, I can confirm the problem.

_________________
MacOS 10.12, PB 5.51 x64


Top
 Profile  
Reply with quote  
 Post subject: Re: [5.44] Broken unicode support in ReadProgramString()
PostPosted: Wed Apr 19, 2017 12:25 pm 
Offline
Addict
Addict
User avatar

Joined: Tue Dec 23, 2003 3:54 am
Posts: 1444
I'm not on Mac/Linux to test right now,
but if you want to post the example here,
try this to avoid embedding Unicode characters (or just use a DataSection + PeekS() of UTF-8 or UTF-16 encoded characters).

Works on Windows, I assume on Mac too:
Code:
Procedure.s ChrU(Codepoint.i)
  If (Codepoint > $FFFF)
    Result.s = "  "
    Codepoint - $10000
    PokeU(@Result, $D800 + ((Codepoint >> 10) & $3FF))
    PokeU(@Result + 2, $DC00 + (Codepoint & $3FF))
    ProcedureReturn Result
  ElseIf (Codepoint >= $0000)
    ProcedureReturn Chr(Codepoint)
  Else
    ProcedureReturn ""
  EndIf
EndProcedure

string.s = "["
string.s + ChrU($26C4)  ; snowman
string.s + ChrU($2708)  ; plane
string.s + ChrU($1F600) ; face
string.s + ChrU($1F6B6) ; walker
string.s + ChrU($2733)  ; burst
string.s + ChrU($203C)  ; exclamation
string.s + ChrU($1F310) ; globe
string.s + ChrU($1F3C2) ; snowboard
string.s + ChrU($1F464) ; person
string.s + ChrU($1F50D) ; magnifier
string.s + "]"

Debug "Raw string:"
Debug string


Top
 Profile  
Reply with quote  
 Post subject: Re: [5.44] Broken unicode support in ReadProgramString()
PostPosted: Wed Apr 19, 2017 9:17 pm 
Offline
Enthusiast
Enthusiast
User avatar

Joined: Wed Jan 12, 2011 3:48 pm
Posts: 182
Location: Serbia
Thanks kenmo, that's a clever approach.

Meanwhile, i adapted my example to Windows (which was a pain in the ass actually, since there is no easy way to output a unicode line) and it seems that PB for Windows doesn't have this issue.

Code:
Procedure.s runToolReadProgramString(path.s,args.s = "",workdir.s = "")
  Protected tool.i,out.s,error.s
  Protected bytes,oldBytes
  Protected *buf
  tool = RunProgram(path,args,workdir,#PB_Program_Open|#PB_Program_Read|#PB_Program_Error|#PB_Program_Hide)
  If tool
    While ProgramRunning(tool)
      If AvailableProgramOutput(tool)
        out + ReadProgramString(tool) + #LF$
      EndIf
      error + ReadProgramError(tool)
      Delay(10)
    Wend
    error + ReadProgramError(tool)
    CloseProgram(tool)
  Else
    Debug "failed to start"
  EndIf
  If Len(error)
    Debug "error: " + error
  EndIf
  out = RTrim(RTrim(out,#LF$),#CR$)
  ProcedureReturn out
EndProcedure
 
Procedure.s runToolReadProgramData(path.s,args.s = "",workdir.s = "")
  Protected tool.i,out.s,error.s
  Protected bytes,oldBytes
  Protected *buf
  tool = RunProgram(path,args,workdir,#PB_Program_Open|#PB_Program_Read|#PB_Program_Error|#PB_Program_Hide)
  If tool
    While ProgramRunning(tool)
      bytes = AvailableProgramOutput(tool)
      If bytes
        If Not *buf
          *buf = AllocateMemory(bytes)
        Else
          *buf = ReAllocateMemory(*buf,oldBytes+bytes)
        EndIf
        ReadProgramData(tool,*buf+oldBytes,bytes)
        oldBytes = MemorySize(*buf)
      EndIf
      error + ReadProgramError(tool)
      Delay(10)
    Wend
    error + ReadProgramError(tool)
    If *buf
      out = PeekS(*buf,MemorySize(*buf),#PB_UTF8|#PB_ByteLength)
      FreeMemory(*buf)
    EndIf
    CloseProgram(tool)
  Else
    Debug "failed to start"
  EndIf
  If Len(error)
    Debug "error: " + error
  EndIf
  out = RTrim(RTrim(out,#LF$),#CR$)
  ProcedureReturn out
EndProcedure

Procedure.s ChrU(Codepoint.i)
  If (Codepoint > $FFFF)
    Result.s = "  "
    Codepoint - $10000
    PokeU(@Result, $D800 + ((Codepoint >> 10) & $3FF))
    PokeU(@Result + 2, $DC00 + (Codepoint & $3FF))
    ProcedureReturn Result
  ElseIf (Codepoint >= $0000)
    ProcedureReturn Chr(Codepoint)
  Else
    ProcedureReturn ""
  EndIf
EndProcedure

string.s = "["
string.s + ChrU($26C4)  ; snowman
string.s + ChrU($2708)  ; plane
string.s + ChrU($1F600) ; face
string.s + ChrU($1F6B6) ; walker
string.s + ChrU($2733)  ; burst
string.s + ChrU($203C)  ; exclamation
string.s + ChrU($1F310) ; globe
string.s + ChrU($1F3C2) ; snowboard
string.s + ChrU($1F464) ; person
string.s + ChrU($1F50D) ; magnifier
string.s + "]"

CompilerIf #PB_Compiler_OS <> #PB_OS_Windows
  ReadProgramDataResult.s = runToolReadProgramData("echo",string)
  ReadProgramStringResult.s = runToolReadProgramString("echo",string)
CompilerElse
  ReadProgramDataResult.s = runToolReadProgramData("powershell",~"-Command \"[Console]::OutputEncoding = [System.Text.Encoding]::UTF8; Write-Host " + string + ~"\"")
  ReadProgramStringResult.s = runToolReadProgramString("powershell",~"-Command \"[Console]::OutputEncoding = [System.Text.Encoding]::UTF8; Write-Host " + string + ~"\"")
CompilerEndIf

Debug "Raw string:"
Debug string
Debug "ReadProgramData:"
If ReadProgramDataResult = string
  Debug ReadProgramDataResult + " - ok"
Else
  Debug ReadProgramDataResult + " - failed"
EndIf
Debug "ReadProgramString:"
If ReadProgramStringResult = string
  Debug ReadProgramStringResult + " - ok"
Else
  Debug ReadProgramStringResult + " - failed"
EndIf

_________________
PB examples collection
pb-osx-globalhotkeys
pb-osx-notifications


Top
 Profile  
Reply with quote  
 Post subject: Re: [5.44] Broken unicode support in ReadProgramString()
PostPosted: Thu Apr 20, 2017 2:26 pm 
Offline
Addict
Addict
User avatar

Joined: Tue Dec 23, 2003 3:54 am
Posts: 1444
One more comment, still on Windows (sorry)

In your updated code, on Windows 7, I get "failed" and "failed" because both strings are returned prefixed with a $FEFF BOM character.

If I add this check, I get "ok" and "ok".

Code:
If (Left(ReadProgramDataResult, 1) = Chr($FEFF))
  ReadProgramDataResult = Mid(ReadProgramDataResult, 2)
EndIf
If (Left(ReadProgramStringResult, 1) = Chr($FEFF))
  ReadProgramStringResult = Mid(ReadProgramStringResult, 2)
EndIf


Top
 Profile  
Reply with quote  
 Post subject: Re: [5.44] Broken unicode support in ReadProgramString()
PostPosted: Thu Apr 20, 2017 10:19 pm 
Offline
Enthusiast
Enthusiast
User avatar

Joined: Wed Jan 12, 2011 3:48 pm
Posts: 182
Location: Serbia
That's probably because of the older powershell version in windows 7.
Thanks!

_________________
PB examples collection
pb-osx-globalhotkeys
pb-osx-notifications


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

 


Powered by phpBB © 2008 phpBB Group
subSilver+ theme by Canver Software, sponsor Sanal Modifiye