Network data server/client SendNetworkString/ReceiveNetworkData

Just starting out? Need help? Post your questions and find answers here.
Oso
Enthusiast
Enthusiast
Posts: 595
Joined: Wed Jul 20, 2022 10:09 am

Network data server/client SendNetworkString/ReceiveNetworkData

Post by Oso »

Hi all, I have been using SendNetworkString/ReceiveNetworkData with good results, but I notice that it can suffer from data loss, especially when I run my client across the network and also if the server sends a high volume of data back to the client. I had hoped that PB handled the buffering of data automatically (handshaking used to be the old term for this), but I suspect I have to do this myself. Perhaps others can confirm.

To @infratec — I remember your comment some weeks ago when you helped me to implement UDP communication between two processes, where you mentioned, "Your sended messages should not be longer than 1500 bytes. Else it will be more complicated". I guess this is relevant. viewtopic.php?p=590452#p590452

To demonstrate, I have used the example in the PB documentation, adding code to transmit a directory listing (because it's a useful test). It is equivalent to the loss of data that I see in my main application, but simplified so others can see. If running the below, add your IP address in client.pb, run the server first, then start the client. The client will send a one-line message to the server and the server will respond, sending back a directory listing which happens to be of C:\Windows\System32.

I have found other posts on this subject, one of which includes a comprehensive module for dealing with what it refers to as data over 64kb, but I'm keen to keep this to a set of changes to the below sample from the docs if I can, because then it becomes possible for me to understand a manageable amount of detail. Thanks in advance.

Here is a link to an example of the data loss... https://ibb.co/C2ttVHw

server.pb

Code: Select all

; ------------------------------------------------------------
;
;   PureBasic - Network (Server) example file
;
;    (c) Fantaisie Software
;
; ------------------------------------------------------------
;
; Note: run the Client.pb file to send some data to this server
;
OpenConsole()
EnableExplicit

Define *Buffer = AllocateMemory(1000)
Define Quit
Define ClientID,  ServerEvent

Declare Send_A_Lot(ClientID)                                            ; Procedure sends a lot of 'dir' data to client as a test

If CreateNetworkServer(0, 6832, #PB_Network_IPv4 | #PB_Network_TCP)
  
  PrintN("PureBasic - Server - Server created")
  Repeat
    ServerEvent = NetworkServerEvent()                                  ; Server event loop
    If ServerEvent                                                      ; Server event triggered
      ClientID = EventClient()                                          ; Which client ID ?
      Select ServerEvent
          
        Case #PB_NetworkEvent_Connect                                   ; Client connects
          PrintN("PureBasic - Server - A new client has connected !")
          
        Case #PB_NetworkEvent_Data                                      ; Received data from client
          PrintN("PureBasic - Server - Client " + ClientID + " has send a packet !")
          ReceiveNetworkData(ClientID, *Buffer, 1000)
          PrintN("Recevied data from client - string : "+PeekS(*Buffer, -1, #PB_Ascii))
          
          Send_A_Lot(ClientID)                                          ; We've received data from client, so send 'a lot' back
          Quit = 1                                                      ; Quit the server process now we've done
          
        Case #PB_NetworkEvent_Disconnect                                ; Client has disconnected
          PrintN("PureBasic - Server - Client " + ClientID + " has closed the connection...")
          Quit = 1                                                      ; Quit the server process
          
      EndSelect
    EndIf
  Until Quit = 1
  CloseNetworkServer(0)                                                 ; Close this server's connection
Else
  PrintN("Error - Can't create the server (perhaps port in use ?)")
EndIf
PrintN("PureBasic - Server - Press Enter to quit the server.")
Input()

Procedure Send_A_Lot(ClientID)
  
  Define dirpath.s = "C:\Windows\System32"                              ; 'Dir' provides a good source of test data
  Define directory.l                                                    ; Directory id.
  Define type.s                                                         ; Type 'blank' for files or <DIR>
  
  PrintN("PureBasic - Server - Sending a long directory listing to the client...")
  
  SendNetworkString(ClientID, "Directory for : " + dirpath.s + #CRLF$, #PB_Ascii)
  
  directory.l = ExamineDirectory(#PB_Any, dirpath.s, "*.*")
  If directory.l
    While NextDirectoryEntry(directory.l)
      type.s = "     "
      If DirectoryEntryType(directory.l) = #PB_DirectoryEntry_Directory
        type.s = "<DIR>"
      EndIf
      SendNetworkString(ClientID, LSet(DirectoryEntryName(directory.l),40) + " " + type.s + "  " + FormatDate("%dd/%mm/%yyyy", DirectoryEntryDate(directory.l, #PB_Date_Modified)) + "  " + FormatDate("%hh:%ii:%ss", DirectoryEntryDate(directory.l, #PB_Date_Modified)) + "  " + FormatNumber(DirectoryEntrySize(directory.l),0) + " bytes" + #CRLF$, #PB_Ascii)
 
    Wend
    FinishDirectory(directory.l)
  Else
    PrintN("Unable to open directory " + dirpath.s)
  EndIf
  
EndProcedure
client.pb

Code: Select all

; ------------------------------------------------------------
;
;   PureBasic - Network (Client) example file
;
;    (c) Fantaisie Software
;
; ------------------------------------------------------------
; Note: run the Server.pb file first To launch the server
OpenConsole()
EnableExplicit

Define *Buffer = AllocateMemory(1000)
Define IP.s = "192.168.0.181"                                           ; Change this to your server's hostname or IP address
Define Quit
Define ConnectionID,  ClientEvent

ConnectionID = OpenNetworkConnection(IP.s, 6832)                        ; Establish a connection to the server
If ConnectionID
  PrintN("PureBasic - Client - Client has connected to the server...")
  
  SendNetworkString(ConnectionID, "Hello from the client", #PB_Ascii)   ; Send a 'hello' from the client to the server
  PrintN("PureBasic - Client - A string has been sent to the server, please check it...")

  Repeat
    ClientEvent = NetworkClientEvent(ConnectionID)                      ; Client event loop
    If clientEvent
      Select ClientEvent
        Case #PB_NetworkEvent_Data                                      ; Client has received data
          
          ReceiveNetworkData(ConnectionID, *Buffer, 1000)               ; Receive the data buffer
          Print(PeekS(*Buffer, -1, #PB_Ascii))                          ; Print contents of buffer without CRLF
          
        Case #PB_NetworkEvent_Disconnect
          PrintN("")
          PrintN("Server has closed the connection...")
          Quit = 1                                                      ; Quit the client process
          
      EndSelect
    EndIf
  Until Quit = 1
  CloseNetworkConnection(ConnectionID)                                  ; Close the client connection
Else
  PrintN("PureBasic - Client - Can't find the server (Is it launched ?).")
EndIf

PrintN("PureBasic - Client - Press Enter to quit the client.")
Input()
User avatar
STARGÅTE
Addict
Addict
Posts: 2067
Joined: Thu Jan 10, 2008 1:30 pm
Location: Germany, Glienicke
Contact:

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by STARGÅTE »

Dear Oso,
There is no data loss in PureBasic, but you must respect the following rules:
  • If you sent data with one SendNetworkString or SendNetworkData command, there is no guarantee that you will receive all this data with just one ReceiveNetworkData call. You have to check the returned value of ReceiveNetworkData how many bytes are received and if you need more calls to complete the package.
  • If you sent data with multiple calls of SendNetworkString or SendNetworkData, there is no guarantee that the data is received in multiple calls of ReceiveNetworkData too. It is also possible that two or more sent data are received within one
    ReceiveNetworkData call concatenated.
What does this mean for you: You have to manage your sent data packages yourself with things like a header with length information.
During a #PB_NetworkEvent_Data event, you have to check yourself, if the data is complete or if multiple packages are within one buffer and split them.

There are several codes with includes or modules here in the forum that will help you with such things.
Just using SendNetworkString and ReceiveNetworkData one-by-one is very negligent.
PB 6.01 ― Win 10, 21H2 ― Ryzen 9 3900X, 32 GB ― NVIDIA GeForce RTX 3080 ― Vivaldi 6.0 ― www.unionbytes.de
Lizard - Script language for symbolic calculations and moreTypeface - Sprite-based font include/module
User avatar
mk-soft
Always Here
Always Here
Posts: 5335
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by mk-soft »

This is the same in all programming languages and is a feature of network communication.
PureBasic uses the standard socket interface in all three OS.

SendNetworkData/String passes the data to the send buffer. The return value indicates how many bytes were passed. Not that the data has been sent.

After the event #PB_NetworkEvent_Data, ReceiveNetworkData fetches the data accumulated until then from the receive buffer. Each connection has its own receive buffer.

For UDP the packet size is set to 2048 bytes. With TCP, the data is divided into individual packets to 1500 (somewhat smaller over the Internet). The sum of the packets is limited to 64KB and it is guaranteed for the 64KB that they arrive in the correct order but not for completeness.
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
User avatar
the.weavster
Addict
Addict
Posts: 1537
Joined: Thu Jul 03, 2003 6:53 pm
Location: England

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by the.weavster »

As I understand it the 64k limit is to do with IP not TCP.
And as far as an application is concerned a TCP socket just exposes a byte stream, the fact large data may have been split into 64kb chunks en route isn't a limit on the size of data you can send and receive, it just means you shouldn't create a buffer larger than 65535 or your code may not work as expected ...

So Oso's client Case #PB_NetworkEvent_Data code would be something like this (untested):

Code: Select all

; BufferSize should not be > 65535
ChunkSize = ReceiveNetworkData(ConnectionID, *Buffer, BufferSize)
ResponseText.s + PeekS(*Buffer, ChunkSize, #PB_Ascii)
If ChunkSize < BufferSize ; we've reached the end of the byte stream
  Print(ResponseText)
EndIf
Have I got that wrong? It's certainly always worked for me :?
User avatar
mk-soft
Always Here
Always Here
Posts: 5335
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by mk-soft »

If you send more than 64KB via TCP/IP, it can happen that the packets are not in the correct order in the receive queue.
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
Oso
Enthusiast
Enthusiast
Posts: 595
Joined: Wed Jul 20, 2022 10:09 am

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by Oso »

Thanks very much STARGÅTE mk-soft and the.weavster — this is indeed now functioning as intended, simply with your replacement code, @the.weavster, for the client Case #PB_NetworkEvent_Data

The reason for the previous erroneous result, appears to have been due to one small thing — simply the absence of checking the number of characters.

Code: Select all

        Case #PB_NetworkEvent_Data                                      ; Client has received data
          ReceiveNetworkData(ConnectionID, *Buffer, 1000)               ; Receive the data buffer
          Print(PeekS(*Buffer, -1, #PB_Ascii))                          ; Print contents of buffer without CRLF
Looking at this now, with the benefit of having read what you've explained, I note that the number of incoming bytes was not stored. Instead, it used PeekS() with a -1 length, to read everything. This was not my code incidentally, but the server-side example in the documentation.

Am I missing something? It seems straightforward and the requirement is only to print the data received. It appears to be working as intended.
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by NicTheQuick »

mk-soft wrote: Sat Jan 07, 2023 5:26 pm If you send more than 64KB via TCP/IP, it can happen that the packets are not in the correct order in the receive queue.
What? That's not true. Packets are always in the right order using TCP. The protocol is exactly there for this reason.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
User avatar
mk-soft
Always Here
Always Here
Posts: 5335
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by mk-soft »

NicTheQuick wrote: Sat Jan 07, 2023 5:38 pm
mk-soft wrote: Sat Jan 07, 2023 5:26 pm If you send more than 64KB via TCP/IP, it can happen that the packets are not in the correct order in the receive queue.
What? That's not true. Packets are always in the right order using TCP. The protocol is exactly there for this reason.
That is difficult to describe.

Sending up to 64KB is the correct order. No matter which routing path the data takes. They are entered in the correct order in the receive buffer. Even if a single packet (MTU sizes) arrives later because of a different routing path.
Last edited by mk-soft on Sat Jan 07, 2023 6:01 pm, edited 2 times in total.
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
Oso
Enthusiast
Enthusiast
Posts: 595
Joined: Wed Jul 20, 2022 10:09 am

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by Oso »

Alas, it seems I spoke too soon when I said it appeared to be fine. Running the client and server on the same machine results in a varying number of bytes received. Moving the client onto another Windows desktop on the LAN, results in a much larger deficit :(

Code: Select all

Local :
zipfldr.dll                                     16/07/2016  11:42:23  388,608 bytes
ztrace_maps.dll                                 16/07/2016  11:42:09  30,720 bytes
Server has closed the connection...
Total bytes received = 1,051,460

Code: Select all

Local (second time):
zipfldr.dll                                     16/07/2016  11:42:23  388,608 bytes
ztrace_maps.dll                                 16/07/2016  11:42:09  30,720 bytes
Server has closed the connection...
Total bytes received = 1,053,746

Code: Select all

LAN :
zipfldr.dll                                     16/07/2016  11:42:23  388,608 bytes
ztrace_maps.dll                                 16/07/2016  11:42:09  30,720 bytes
Server has closed the connection...
Total bytes received = 449,660
More work to do. Can I ask STARGÅTE, when you say "received in multiple calls of ReceiveNetworkData too", does this mean that it is necessary to repeatedly call ReceiveNetworkData() within a single #PB_NetworkEvent_Data event? If so, I think this is where my understanding was incorrect. I had been under the impression that if there is more data to be received, then there will be another #PB_NetworkEvent_Data event triggered anyway. Perhaps you could confirm on that.
Last edited by Oso on Sat Jan 07, 2023 6:02 pm, edited 1 time in total.
User avatar
mk-soft
Always Here
Always Here
Posts: 5335
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by mk-soft »

Example of an error:
If you send 64KB twice and an error or delay occurs on the way, it can happen that the second 64KB send arrives before the first 64KB send.

Therefore, a separate handshake is important.

P.S.
Link: Module NetworkTCP - Send and Receive Data over 64KB
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
User avatar
the.weavster
Addict
Addict
Posts: 1537
Joined: Thu Jul 03, 2003 6:53 pm
Location: England

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by the.weavster »

My understanding is the same as NicTheQuick, the disassembly and reassembly of the data in the right order is precisely what TCP does.

TCP sits between the network layer and the application layer and your application interacts with what are effectively byte streams.

When you read data using ReceiveNetworkData() that will trigger TCP to feed more data into your buffer, on and on until all data is consumed at which point the size of the data retrieved will be less than the size of the buffer.

My guess for why your buffer should be < 64kb is because that's the largest chunk TCP will write into the byte stream in one hit so if your buffer were larger you wouldn't be able to accurately determine if you'd reached the end of the data.
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by NicTheQuick »

The receiving end of an TCP connection always reassembles the data in the correct order. Each TCP packet consists of a sequence number. If packets arrive in the wrong order the receiver automatically buffers these packets until their sequence number has come. If a packet is missing after a certain timeout it tells the sender to resend the packet with the missing sequence number. Because of this packets never arrive in the wrong order. Your application does not need to reinvent the protocol by itself. Also every packet has a error correction code inside which can help to fix a few bit flips and in doubt can request the packet again from the sender.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
User avatar
the.weavster
Addict
Addict
Posts: 1537
Joined: Thu Jul 03, 2003 6:53 pm
Location: England

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by the.weavster »

Oso wrote: Sat Jan 07, 2023 5:28 pm The reason for the previous erroneous result, appears to have been due to one small thing — simply the absence of checking the number of characters.
Looking at this now, with the benefit of having read what you've explained, I note that the number of incoming bytes was not stored. Instead, it used PeekS() with a -1 length, to read everything. This was not my code incidentally, but the server-side example in the documentation.

Am I missing something?
Yes, there's something else... If you look at my code again you'll see each new chunk of data that arrives gets appended to ResponseText, it's only when ChunkSize < BufferSize (i.e. we've reached the end of the byte stream) that Print(ResponseText) gets called.
Oso
Enthusiast
Enthusiast
Posts: 595
Joined: Wed Jul 20, 2022 10:09 am

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by Oso »

the.weavster wrote: Sat Jan 07, 2023 7:18 pm Yes, there's something else... If you look at my code again you'll see each new chunk of data that arrives gets appended to ResponseText, it's only when ChunkSize < BufferSize (i.e. we've reached the end of the byte stream) that Print(ResponseText) gets called.
I've just got it working, the.weavster :D The reason I found varying results earlier, is because the string ResponseText isn't being cleared after printing, so it gets longer with each received data and that in turn is also dependent on a certain amount of variation in the number of bytes in the chunk. Below is the amended section...

Code: Select all

         ChunkSize = ReceiveNetworkData(ConnectionID, *Buffer, BufferSize)
         ResponseText.s + PeekS(*Buffer, ChunkSize, #PB_Ascii)
         If ChunkSize < BufferSize ; we've reached the end of the byte stream
           Print(ResponseText)
           totalbytes + Len(ResponseText)
           ResponseText = ""
         EndIf
I added a totalbytes counter at the server side and the client side and they match perfectly — 332,033 every time. I get this result when running the client and server on separate machines, as well as on the same machine.

Incidentally, I also proved that it was not necesary to keep calling ReceiveNetworkData() repeatedly inside the single #PB_NetworkEvent_Data event so therefore this answers my earlier question about whether a further event is trigged if the transmitted data is not read in one buffer. In other words, the answer is yes — if we don't consume all the bytes received, another event is triggered by PB.

I apologise for this in my first post, by the way... :cry:
ReceiveNetworkData with good results, but I notice that it can suffer from data loss,
On re-reading my text, I see that it implies that I thought PureBasic was responsible for data loss. This wasn't the case — I was well aware that my own code was responsible — and my question was really to ask, how can I improve my code to deal with that. Hope that helps to clarify anyway.

Thanks to all for the help with this, and especially for your code, as it's that which has enabled me to resolve this.
User avatar
the.weavster
Addict
Addict
Posts: 1537
Joined: Thu Jul 03, 2003 6:53 pm
Location: England

Re: Network data server/client SendNetworkString/ReceiveNetworkData

Post by the.weavster »

STARGÅTE wrote: Sat Jan 07, 2023 9:50 am
  • If you sent data with multiple calls of SendNetworkString or SendNetworkData, there is no guarantee that the data is received in multiple calls of ReceiveNetworkData too. It is also possible that two or more sent data are received within one
    ReceiveNetworkData call concatenated.
@NickTheQuick
Do you know what happens with TCP in this scenario described by STARGÅTE where instead of a message being sent with one call to SendNetworkData() it is sent in multiple snippets by repeated calls to that command?
Post Reply