Reliably send large amounts of data via network

merendo · **Posted:** Thu Apr 05, 2012 10:24 am

Hello folks!

My application will need to communicate with other instances of itself running on remote machines (sometimes even connected through a slow VPN tunnel). To that end, I need to write a reliable and robust system to send large amounts of data via the network (SendNetworkData, that is). We're talking magnitude of 25 MBytes or more.

I have written some basic code, both client and server. The client produces some random data in the memory, then transmits that to the server. Interestingly, the supposed limit of 65535 bytes documented in the manual don't seem to produce a problem. Perhaps you could make some comments as to where this script might fail or what external events could produce trouble.

Any thoughts appreciated!
merendo

Server:

Code:

If OpenConsole()

   If InitNetwork() : PrintN("TCP/IP-Stack initialised.") : Else : End : EndIf
   
   server = CreateNetworkServer(0, 8506)
   large_reclen.l = 0
   large_recbuffer$ = "" ; Large reception buffer.
   small_reclen.w = 0
   *small_recbuffer = AllocateMemory(1024) ; Received networkdata goes in here first, then into the large reception buffer.
   
   If server
      PrintN("Server running on port 8506")
      
      
      Repeat
         Delay(5)
         
         ServerEvent = NetworkServerEvent()
         If ServerEvent
            EventClient = EventClient()
         
            Select ServerEvent
               Case #PB_NetworkEvent_Connect
                  PrintN("Client connected.")

               Case #PB_NetworkEvent_Data
                  Repeat ; Keep looping until all data has been received
                     small_reclen = ReceiveNetworkData(EventClient, *small_recbuffer, 1024)
                     If small_reclen
                        large_recbuffer$ = large_recbuffer$ + PeekS(*small_recbuffer, small_reclen)
                        large_reclen + small_reclen
                     EndIf
                  Until Not small_reclen = 1024
                  
                  PrintN("Received some data. Size: "+Str(large_reclen)+" characters. MD5: "+MD5Fingerprint(@large_recbuffer$, large_reclen))
                  large_reclen.l = 0
                  large_recbuffer$ = ""
                  small_reclen.w = 0
               
               Case #PB_NetworkEvent_Disconnect
                  PrintN("Client disconnected.")
                  Input()
                  End
               
            EndSelect
         EndIf
         
      ForEver
      
   EndIf

EndIf

Client:

Code:

If OpenConsole()

   If InitNetwork() : PrintN("TCP/IP-Stack initialised.") : Else : End : EndIf
   
   Print("Input requested data length: ")
   datenlaenge = Val(Input())
   If datenlaenge < 1 : End : EndIf
   
   *baseAddress = AllocateMemory(datenlaenge)
   *currentAddress = *baseAddress
   
   Print("Connect to: ")
   target$ = Input()
   If target$ = "" : End : EndIf
   
   networkConn = OpenNetworkConnection(target$, 8506)
   
   If networkConn
      For x = 1 To datenlaenge
         PokeS(*currentAddress, Chr(Random(223)+32), 1)
         *currentAddress + 1
      Next
   
      PrintN("Connection established, random data is ready to transmit. Waiting 2.5 seconds before transmitting data...")
      Delay(2500)
      SendNetworkData(networkConn, *baseAddress, datenlaenge)
      PrintN("Data sent. MD5: "+MD5Fingerprint(*baseAddress, datenlaenge))
      Input()
   EndIf

EndIf

merendo · **Posted:** Thu Apr 05, 2012 10:40 am

Okay, I have found an odd bug already. When I send some data with a length of 1048576 bytes (1024*1024 Bytes = 1 MByte), the transmission is incorrect (one character is missing and the md5 checksums don't match as a result). When I send some data with 1200000, which is larger than 1 MByte, it works just fine. Any ideas why that might be?

merendo · **Posted:** Wed Apr 11, 2012 8:11 am

Anyone, any thoughts?

Foz · **Posted:** Wed Apr 11, 2012 11:44 am

My take on it would be a chunking system.

First of all, take a checksum (MD5 or something) of the entire file, and then decide how you want to split it (8, 64, 128kb blocks - your choice of size), and then you send to the receiver the filename, file size, the number of chunks and the checksum.

The receiver can then create the empty file, and it knows that it will be waiting for the chunks to fill in

Once you get a confirmation from the receiver, send the file numbers with a chunk number for a header.

The receiver can then write the chunks to the correct location in the waiting file as the chunks will be of a fix size (except for the last one, but that doesn't matter). Once done, check the received file against the checksum and you have a complete file.

You can take this further by having checksums per chunk and if the chunk checksums fail you can add in requests to the sender for specific chunks to be resent. If you do threading, then you could also do multiple sending of chunks at once - like the download managers.

merendo · **Posted:** Wed Apr 11, 2012 12:16 pm

Okay, that would make sense. I'll try to write such a chunking system, shouldn't be too much effort I guess.

Do you think a md5-checksum system would make sense for every chunk, though? After all, TCP/IP performs such checks already, so my guess would be I can be sure that the data sent is not corrupted.

Foz · **Posted:** Wed Apr 11, 2012 12:37 pm

Well, that is taking it to the extreme - but on a poor network with dropped packets... would you want to take the chance of having to retransmit the entire file all over again when you only need to resend that one corrupt chunk?

Shield · **Joined:** Fri Jan 21, 2011 8:25 am **Posts:** 549

TCP packages will always be verified before you'll be able to read the data on the application level.
The only way something could go wrong is if they are forged (i.e. the content has been changed on purpose),
but that's a different story.

merendo · **Posted:** Wed Apr 11, 2012 12:57 pm

Well... The network my application will be running in is a standard LAN, either 100MBit or GBit, so transmission quality should be fine, and I also don't expect someone tinkering with the contents of the packets.

Anyway, for my question why sending chunks of data larger than 65535 bytes (in a single transaction) doesn't always produce consistent results: Can it be said with reasonable certainty that sending such large packets of data is simply unsafe and should be avoided?

PureBasic Forum

Reliably send large amounts of data via network

Who is online