Am I having a UNICODE issue?

[Trond]

You're right, it should say Bytes*SizeOf(Character).

[netmaestro]

I don't believe SendNetworkString is sending a unicode string even when compiled in unicode mode. Using Trond's example code with both programs compiled as unicode, the server is receiving 31 bytes of data with the test string, whereas if it were actually coming over as unicode it should be 62, right? 2 bytes per letter. Or am I missing something?

If I'm right, this should be classed as a bug imho.

[srod]

I agree. In Unicode mode SendNetworkString() still seems to be sending the string in Ascii as I just had a look at the bytes received!!!

Either this is a bug or SendNetworkString() is not designed for unicode use!

[netmaestro]

Well I can't be right else I would've noticed a bug that Trond missed and really - what are the odds of that???

[Trond]

Now it should work:

Code: Select all

; Client

Macro SendNetworkStringCorrectly(Connection, String)
  SendNetworkData(Connection, @String, Len(String)*SizeOf(Character))
EndMacro

InitNetwork() 
Connection = OpenNetworkConnection("127.0.0.1", 999) 

Debug SendNetworkStringCorrectly(Connection, "fiver") 

; if I leave this out the string isn't sent correctly for some reason. 
Repeat 
  NetworkClientEvent(Connection) 
  
ForEver

Code: Select all

; Server

Procedure.s ReceiveNetworkString()
  #BufferSize = 1024
  Protected Buffer = AllocateMemory(#BufferSize)
  Protected Result.s
  Repeat
    Bytes = ReceiveNetworkData(EventClient(), Buffer, #BufferSize)
    If Bytes > 0
      Result + PeekS(Buffer, Bytes)
    EndIf
  Until Bytes <> #BufferSize
  FreeMemory(Buffer)
  ProcedureReturn Result
EndProcedure

InitNetwork()

OpenWindow(0, 100, 100, 100, 100, "Server", #PB_Window_SystemMenu)
CreateGadgetList(WindowID(0))

Connection = CreateNetworkServer(0, 999)

Repeat
 
  Event  = WaitWindowEvent(12)
  Gadget = EventGadget()
  Type   = EventType()
  Window = EventWindow()
 
  N = NetworkServerEvent()
  Select N
    Case #PB_NetworkEvent_Connect
      Debug "SERVER: client connected"
    Case #PB_NetworkEvent_Data
      Debug "SERVER: receive: " + ReceiveNetworkString()
  EndSelect
Until Event = #PB_Event_CloseWindow

[Rescator]

I hope I didn't make any mistakes, kind of a rush coding, haven't tested it yet. So please do and let me know if it works ok.

SendNetworkStringUTF8() and ReceiveNetworkStringUTF8() provides proper UTF8 based string send and receive.

This means that a server can be unicode and a client can be ascii, the text will automatically be converted properly. (This is the true brilliance of UTF8 )

The send procedure is little more than a wrapper for SendNetworkData(), the receive procedure however is a lot more clever. It will fetch new data in 1KB blocks, reallocate memory as more and more data is available, up to 32KB (the default, you can adjust this easily if you expect longer strings) Once no more data is available the received data will be converted to unicode or ascii depending on the compile option of the program.

The reason why the string is converted after it has been collected is due to UTF8 using 2 or more bytes for beyond standard ascii characters, and if the string was converted as it is received you may end up with half a character, and thus a broken string. This way however only the completed string is converted.

I'm sure these two procedures can be improved further, so go for it if anyone have any ideas.

Fred, any chance you can change the current SendNetworkString() to behave similar to my SendNetworkStringUTF8() and add a ReceiveNetworkString() similar my ReceiveNetworkStringUTF8() ?

The current one does not handle unicode, only ascii right? And considering that ascii is able to carry UTF8 a new SendNetworkString() will still remain backwards compatible with old/current ascii programs. As for old unicode programs...those are broken anyway with the old SendNetworkString()

Considering how robust this would be, might I also suggest a #PB_NetworkEvent_String=5

Code: Select all

EnableExplicit

Procedure.l SendNetworkStringUTF8(ClientID.l,String$)
 Protected result.l=#False,*mem,length.l
 length=StringByteLength(String$,#PB_UTF8)
 If length
  *mem=AllocateMemory(length)
  If *mem
   PokeS(*mem,String$,#PB_UTF8)
   result=SendNetworkData(ClientID,*mem,length)
   FreeMemory(*mem)
  EndIf
 EndIf
 ProcedureReturn result
EndProcedure

Procedure.s ReceiveNetworkStringUTF8(ClientID.l,MaxBytes.l=32768)
 Protected string$,*mem,length.l,*pos,bytes.l,total.l
 length=1024
 *mem=AllocateMemory(length)
 If *mem
  *pos=*mem
  Repeat
   bytes=ReceiveNetworkData(ClientID,*pos,length)
   If bytes>0
    total+bytes
    If total<=MaxBytes
     If total>MemorySize(*mem)
      *pos=ReAllocateMemory(*mem,MemorySize(*mem)+length)
      If *pos
       *mem=*pos
       *pos=*pos+total
      Else
       Break
      EndIf
     EndIf
    Else
     Break
    EndIf
   EndIf
  Until bytes=0
  If total
   string$=PeekS(*mem,total,#PB_UTF8)
  EndIf
  FreeMemory(*mem)
 EndIf
 ProcedureReturn string$
EndProcedure

[Rook Zimbabwe]

Thank you all! I will play with this new code, it might be the right thing. I had no idea there was a unicode limitation in SendNetworkString()

Not until I had to use it.

[srod]

I had no idea there was a unicode limitation in SendNetworkString()

Yes, I am still wondering if that is a bug; although the fact that SendNetworkString() seems to convert to ascii first suggest that it is by design!

[Rook Zimbabwe]

Converts to AScii... that would explain the weird characters that arrive when compiled BOTH in unicode. The server gets what looks like BOXES and Chinese... though both should be sending and receiving the same thing!

I don't think this is a feature... ergo a bug! I found one!