Networking Send and Receive

[Foz]

Well, I've been putting this off for as long as possible because it is still far from perfect. However if you find some better ways of doing things than I have, then please share them. This is my first major project after all, and this code has been rewritten about 4 times now. Trust me, it was a LOT worse in the beginning

Note that this is cross platform, and I would like to keep it this way.

*** edit 19 Feb 2014
- Updated for PB 5.21 LTS
- Rearranged the data structure, so it's now [Length][CRC32][Data]
- Now fully unicode compatible : Added SendNetworkStringEx for sending strings - this sends the string in UTF8, with the null character at the end. The reason for this is so the string data is unified, and adds a minimal amount of data for reading strings.
- Changed the names of the received structure so it makes more sense
- the data pointer of the received structure is now at the ready to parse spot
- added a data size, which is the total data size - 8 so you can use that if you need it.
- added memory leak fix caught by ruslanx - thanks!

Networking.pbi

Code: Select all

EnableExplicit

;{- Enums, Structures and Globals
; Possible return codes
Enumeration
  #NetworkError_Client_NoError
  #NetworkError_Client_CouldNotConnect
  #NetworkError_Client_NoDataToSend
  #NetworkError_Client_Timeout
EndEnumeration

Enumeration
  #NetworkError_Server_NoError
  #NetworkError_Server_Corrupted
  #NetworkError_Server_TotalDataCRC
EndEnumeration

Enumeration 
  #PacketMaxSize = 1500
EndEnumeration

Structure structReceivedData
  ClientID.i
  BytesReceived.l
  TotalSize.l
  DataSize.l
  
  *ptr_total
  *data
EndStructure

Global NewList ReceivedData.structReceivedData()

;}

Procedure.l SendNetworkDataEx(ConnectionID.i, *MemoryBuffer, Length.l)
  If ConnectionID = 0
    ProcedureReturn #NetworkError_Client_CouldNotConnect
  EndIf
  If Length = 0
    ProcedureReturn #NetworkError_Client_NoDataToSend
  EndIf

  ; create the footer
  Protected DataCRC.l = CRC32Fingerprint(*MemoryBuffer, Length)

  ; create the buffer to send
  Protected *Buffer = AllocateMemory(Length + 8)
  ; a general pointer to move around in the buffer
  Protected *ptrBuffer = *Buffer

  ; first, in goes the header: data length + 8 (for header)
  PokeL(*ptrBuffer, Length + 8)
  *ptrBuffer + 4
  ; then the CRC check
  PokeL(*ptrBuffer, DataCRC)
  *ptrBuffer + 4
  ; then the data
  CopyMemory(*MemoryBuffer, *ptrBuffer, Length)
  
  ; transmit the buffer
  Protected TimeStarted.l = ElapsedMilliseconds()
  Protected DataToSend.l = Length + 8
  *ptrBuffer = *Buffer
  Repeat
    Protected DataSent.l = SendNetworkData(ConnectionID, *ptrBuffer, DataToSend)
    If DataSent > 0
      *ptrBuffer + DataSent
      DataToSend - DataSent
    EndIf
  Until DataToSend = 0 Or (ElapsedMilliseconds() > TimeStarted + 30000)

  ; release the allocated memory
  FreeMemory(*Buffer)
  
  If DataToSend > 0
    ProcedureReturn #NetworkError_Client_Timeout
  EndIf

  ProcedureReturn #NetworkError_Client_NoError
EndProcedure

Procedure.l SendNetworkStringEx(ConnectionID.i, Value.s)
  Protected StringLength.i = StringByteLength(Value, #PB_UTF8) + 1 ; plus a null character
  Protected *StringMemory = AllocateMemory(StringLength)
  PokeS(*StringMemory, Value, -1, #PB_UTF8)
  Protected rv.l = SendNetworkDataEx(ConnectionID, *StringMemory, StringLength)
  FreeMemory(*StringMemory)
  ProcedureReturn rv
EndProcedure

Procedure.l ReceiveNetworkDataEx(ClientID.i)
  Static NewList ReceivedPartialData.structReceivedData()

  If ClientID = 0
    ProcedureReturn 0
  EndIf

  Protected i.l
  Protected found.b = #False
  For i = 0 To ListSize(ReceivedPartialData()) - 1
    SelectElement(ReceivedPartialData(), i)
    If ReceivedPartialData()\ClientID = ClientID
      found = #True
      Break
    EndIf
  Next

  If Not found
    AddElement(ReceivedPartialData())
    ReceivedPartialData()\ClientID = ClientID
    ReceivedPartialData()\BytesReceived = 0

    ReceivedPartialData()\data = AllocateMemory(4)
    ReceivedPartialData()\ptr_total = ReceivedPartialData()\data
  EndIf
  
  Protected actualReceivedData.l
  Protected *localBuffer = AllocateMemory(#PacketMaxSize)
  actualReceivedData = ReceiveNetworkData(ClientID, *localBuffer, #PacketMaxSize)
  
  If ReceivedPartialData()\BytesReceived < 4 And ReceivedPartialData()\BytesReceived + actualReceivedData > 4
    ; reallocate the memory to be the data size needed by checking against the first long received
    ReceivedPartialData()\TotalSize = PeekL(*localBuffer)
    If ReceivedPartialData()\TotalSize > 0 And ReceivedPartialData()\TotalSize < 10000000
      ReceivedPartialData()\data = ReAllocateMemory(ReceivedPartialData()\data, ReceivedPartialData()\TotalSize)
      ReceivedPartialData()\ptr_total = ReceivedPartialData()\data

      If ReceivedPartialData()\data = 0
        ; The program cannot allocate any more memory!
        ; If this ever occurs on modern systems, especially with a 10mb limit to send,
        ; we are royally foo barred.
        PrintN("Out Of Memory!")
        PrintN("Reallocation of memory failed at NetworkDataReceived.")
        End 1
      EndIf

    Else
      ; if the TotalSize is less than 0 or more than 10mb, it's corrupted
      ; (for me anyway, if I go over 1mb I'm in trouble)
      ; need to return a corruption message
      FreeMemory(ReceivedPartialData()\data)
      DeleteElement(ReceivedPartialData())

      SendNetworkStringEx(ClientID, "Corrupted")
      FreeMemory(*localBuffer)
      ProcedureReturn 0
    EndIf
  EndIf

  ReceivedPartialData()\BytesReceived + actualReceivedData ; increase the amount of bytes received

  CopyMemory(*localBuffer, ReceivedPartialData()\ptr_total, actualReceivedData)
  ReceivedPartialData()\ptr_total + actualReceivedData ; move the pointer on

  FreeMemory(*localBuffer)
  
  If ReceivedPartialData()\BytesReceived = ReceivedPartialData()\TotalSize
    Protected CRC = PeekL(ReceivedPartialData()\data + 4)
    If CRC = CRC32Fingerprint(ReceivedPartialData()\data + 8, ReceivedPartialData()\TotalSize - 8)
      AddElement(ReceivedData())
      ReceivedData()\ClientID = ReceivedPartialData()\ClientID
      ReceivedData()\TotalSize = ReceivedPartialData()\TotalSize
      ReceivedData()\data = AllocateMemory(ReceivedPartialData()\TotalSize)
      CopyMemory(ReceivedPartialData()\data, ReceivedData()\data, ReceivedPartialData()\TotalSize)
      
      FreeMemory(ReceivedPartialData()\data)
      DeleteElement(ReceivedPartialData())

      ;SendNetworkString(ClientID, "Received")
    Else
      FreeMemory(ReceivedPartialData()\data)
      DeleteElement(ReceivedPartialData())

      ;SendNetworkString(ClientID, "Corrupted")
      ProcedureReturn 0
    EndIf
  EndIf
  
  ProcedureReturn actualReceivedData
EndProcedure

Procedure.l NetworkDataEvent()
  If ListSize(ReceivedData()) = 0
    ProcedureReturn 0
  EndIf
  
  FirstElement(ReceivedData())
  
  ProcedureReturn ReceivedData()\ClientID
EndProcedure

Procedure NetworkDataReceived(ClientID.l, *NetworkedData.structReceivedData)
  If ListSize(ReceivedData()) = 0
    ProcedureReturn
  EndIf

  ForEach ReceivedData()
    If ReceivedData()\ClientID = ClientID
      Break
    EndIf 
  Next

  *NetworkedData\ClientID = ReceivedData()\ClientID
  *NetworkedData\TotalSize = ReceivedData()\TotalSize
  
  *NetworkedData\data = AllocateMemory(ReceivedData()\TotalSize)
  CopyMemory(ReceivedData()\data, *NetworkedData\data, ReceivedData()\TotalSize)
  *NetworkedData\ptr_total = *NetworkedData\data
  *NetworkedData\data + 8 ; move the data point to start at the data
  *NetworkedData\DataSize = *NetworkedData\TotalSize - 8
  
  FreeMemory(ReceivedData()\data)
  DeleteElement(ReceivedData(), 1)
EndProcedure

Server.pb

Code: Select all

EnableExplicit

; main server
XIncludeFile "Networking.pbi"

OpenConsole()

If InitNetwork() = 0
  PrintN("Error - Can't initialize the network !")
  Input()
  End
EndIf

#Port = 6300

Procedure Rec(l.l)
    Protected  ClientID.l = NetworkDataEvent()
    If ClientID
      Protected rd.structReceivedData
      NetworkDataReceived(ClientID, @rd)
      PrintN("  Full data has been received: " + Str(rd\TotalSize))
      
      Protected testdata.s = PeekS(rd\data, -1, #PB_UTF8)
      
      PrintN(testdata)

      SendNetworkStringEx(ClientID, "Received")

      FreeMemory(rd\ptr_total)
    EndIf
EndProcedure

PrintN("Opening port "+Str(#Port)+").")

If CreateNetworkServer(0, #Port)
  PrintN("Server created (Port "+Str(#Port)+").")
 
  Repeat
    Define SEvent.l = NetworkServerEvent()
 
    If SEvent
      Define ClientID.l = EventClient()
 
      Select SEvent
        Case 1
          ; A new client has connected
          PrintN("Connected")
 
        Case 2
          ; Packet received
          PrintN("Packet received")
          ;CreateThread(@ReceiveNetworkDataEx(), ClientID)
          ReceiveNetworkDataEx(ClientID)
 
        Case 4
          ; Client has closed the connection
          PrintN("Closed")
      EndSelect
    EndIf
    
    ;CreateThread(@Rec(),0)
    Rec(0)
    Delay(50)
  ForEver

  CloseNetworkServer(0)
Else
  PrintN("Error: Can't create the server (port in use ?).")
EndIf

PrintN("Press enter to quit the server.")
Input()
End

Client.pb

Code: Select all

XIncludeFile "Networking.pbi"

If InitNetwork() = 0
  MessageRequester("ERROR", "Error: Can't initialize the network !")
  End
EndIf

Global Window_0, Editor_0, Button_0

Procedure Button_0_LeftClick()
  Define ConnectionID.l = OpenNetworkConnection("127.0.0.1", 6300)
  If ConnectionID
    Define str.s = GetGadgetText(Editor_0)
    SendNetworkStringEx(ConnectionID, str)
  Else
    MessageRequester("ERROR", "Can't find the server (Is it launched ?).")
    ProcedureReturn
  EndIf

  Repeat
    Define ev.l = NetworkClientEvent(ConnectionID)
    If ev = #PB_NetworkEvent_Data
      ReceiveNetworkDataEx(ConnectionID)
    EndIf
    
    ev = NetworkDataEvent()
    If ev
      Protected rd.structReceivedData
      NetworkDataReceived(ConnectionID, @rd)

      MessageRequester("Message from server:", PeekS(rd\data, -1, #PB_UTF8))

      FreeMemory(rd\ptr_total)
    EndIf
  Until ev > 0

  CloseNetworkConnection(ConnectionID)
EndProcedure

Procedure Open_Window_0()
  Window_0 = OpenWindow(#PB_Any, 206, 24, 621, 471, "Window 0",  #PB_Window_SystemMenu | #PB_Window_SizeGadget | #PB_Window_TitleBar )
  If Window_0
    Editor_0 = EditorGadget(#PB_Any, 10, 60, 600, 400)
    Button_0 = ButtonGadget(#PB_Any, 170, 10, 250, 40, "Send Everything!")
    BindGadgetEvent(Button_0, @Button_0_LeftClick(), #PB_EventType_LeftClick)
  EndIf
EndProcedure

Open_Window_0()

Repeat
Until WaitWindowEvent() = #PB_Event_CloseWindow

[Dare]

Thanks! Will study and learn.

[electrochrisso]

Nice one Foz! I'm learning too.

[Tranquil]

I have coded something similar some years ago and tried to explain exactly what happens on networking:

http://www.purebasic.fr/english/viewtop ... networking

[Rook Zimbabwe]

Foz that is some good code there!

I am considering parsing the String$ I send to the tiny server telling FindString() to look for the ";" and cutting that out of the buffer as another method since I am basically just sending SQL queries to the server. However your idea has serious merit.

[Foz]

I'm glad it might help some people. It was a bugger to write well.
The initial version had dynamic arrays of dynamic arrays of pointers to hold everything that had been transmitted and then once everything was transmitted, it was then going to reassemble them all. It was NASTY!

The purpose behind this code is that I want to be able to send *anything* to the server which can then receive it with minimum fuss. As it is the building block of my RPC tests, I want to be able to either send XML-RPC across for simple functions, or, in the case of a lot of data (such as arrays), just transmit a mass block of memory for the server to deal with.

The theory is sound. Once everything has been received, throw a new event to say who it is from and the complete data. However this is not bug free, as it hasn't been tested in The Real World(tm), i.e. bad networks, multiple clients, fast clients, etc.

For instance, if you make the client a console app that waits for Input(), and then paste a web page in, the server crashes with memory errors. This is because (I am fairly certain) the server is not expecting a massive churning of different data transmissions from the one client at the same time, so the networking queue gets confused. I'm still working on this one, it's a bugger to solve Then again, there might not be an actual problem and I'm just finding a problem that would never occur in The Real World(tm). If someone can solve this one please let me know

[ruslanx]

Code: Select all

Procedure NetworkDataReceived(ClientID.l, *NetworkedData.structReceivedData)
  If ListSize(ReceivedData()) = 0
    ProcedureReturn
  EndIf

  ForEach ReceivedData()
    If ReceivedData()\ClientID = ClientID
      Break
    EndIf 
  Next

  *NetworkedData\ClientID = ReceivedData()\ClientID
  *NetworkedData\DataSize = ReceivedData()\DataSize
  
  *NetworkedData\data = AllocateMemory(ReceivedData()\DataSize)
  CopyMemory(ReceivedData()\data, *NetworkedData\data, ReceivedData()\DataSize)
  *NetworkedData\ptr_data = *NetworkedData\data
  
  FreeMemory(ReceivedData()\data)   ; <------
  DeleteElement(ReceivedData(), 1)
EndProcedure

FreeMemory(ReceivedData()\data) ; You forgot to add Free Memory, to exclude Memory Leak

[Foz]

I've updated this tidbit - it should work a bit better now

[coco2]

Thanks very much. Useful for noobs like me. New to PB and also new to coding networking.

[a_carignan]

I would like to know how I read a pourais BinaryReader.ReadInt64 and a BinaryReader.ReadInt32 c #. during a transfer of network file. the client software is a phone. The information is the size of a file and the size of the file name. I tried some of peek, but it gives results much too high. solutions slopes.

[a_carignan]

I found if étaint of integer. it was difficult to find since the content of the data also contained junk and had to go through it.
Thank you anyway.