Dynamic Resizing Of Buffer With ReceiveNetworkData

[the.weavster]

As you probably don't know how much data there is before you've received it it would be good if the buffer would just grow to the appropriate size.

[AND51]

Yes, this would be a really nice feature!

[Kaeru Gaman]

and it is simply impossible...

[the.weavster]

and it is simply impossible...

Well both REALbasic and Hotbasic can do the impossible.

*edit*
I think I may have been wrong about Hotbasic.

[gnozal]

and it is simply impossible...

Well both REALbasic and Hotbasic can do the impossible.

Well you can do it (yourself) with CopyMemory() and ReAllocateMemory() for example.

[the.weavster]

I tried doing it with ReallocateMemory() but it kept causing a crash.
Maybe I was doing it wrong, could you give me a code snippet?

Thanks
Weave

[gnozal]

I tried doing it with ReallocateMemory() but it kept causing a crash.
Maybe I was doing it wrong, could you give me a code snippet?

Thanks
Weave

Something like this ?

Code: Select all

  Repeat
    If NetworkClientEvent(ConnectionID) = #PB_NetworkEvent_Data
      BytesRead = ReceiveNetworkData(ConnectionID, *NetworkDataBuffer, 5000)
      If BytesRead > 0
        *NewBuffer = ReAllocateMemory(*BigBuffer, TotalBytesRead + BytesRead)
        If *NewBuffer
          *BigBuffer = *NewBuffer
          CopyMemory(*NetworkDataBuffer, *BigBuffer + TotalBytesRead, BytesRead)
          TotalBytesRead + BytesRead
          ; /////////////////////////////////////////
          ; Test if end of data (depends on protocol)
          If EndOfTransmission = #True
            Debug "* BREAK (END)"
            Break
          EndIf
          ; /////////////////////////////////////////
        Else
          Debug "* BREAK (ReAllocate failed)"
          Break
        EndIf
      EndIf
    EndIf
  ForEver
  ; All the data is in *BigBuffer (length = TotalBytesRead)

[Kaeru Gaman]

and it is simply impossible...

Well both REALbasic and Hotbasic can do the impossible.

Well you can do it (yourself) with CopyMemory() and ReAllocateMemory() for example.

yes of course, but this will end up
- either
setting a huge buffer first and shrinkening it after recieve
- or
starting with a small buffer and making it bigger while recieving,
wich will mean recieving really small packages.

both don't sound really advantageous to me...

or did I overlook some serious fact?
afaik network-data does not have a header describing the length of incoming data, have they?

[the.weavster]

@gnozal
Thank you very much.

yes of course, but this will end up
- either
setting a huge buffer first

Which is what you have to do with ReceiveNetworkData() because

afaik network-data does not have a header describing the length of incoming data

- or
starting with a small buffer and making it bigger while recieving,
wich will mean recieving really small packages.

Which is more like I was hoping to achieve.

both don't sound really advantageous to me...

Adventageous to what? What is your third alternative?

[AND51]

Well both REALbasic and Hotbasic can do the impossible.

Yes, why should PureBasic renounce this, while other languages have this feature?

[netmaestro]

Agreed it can be done, but I must say I don't think your logic will work. This seems more logical to me:

Code: Select all

Repeat 
  If NetworkClientEvent(ConnectionID) = #PB_NetworkEvent_Data 
    BytesRead = ReceiveNetworkData(ConnectionID, *NetworkDataBuffer, MaxBytes) 
    overruns = 0
    While BytesRead = MaxBytes ; This means more data to be read
      overruns + 1 
      *result = ReAllocateMemory(*NetworkDataBuffer, MaxBytes + MaxBytes * overruns)
      If *result = *NetworkDataBuffer
        ; successfully enlarged the buffer
        ; reset the insertion point of the new larger buffer to past the already-read data
        BytesRead = ReceiveNetworkData(ConnectionID, *NetworkDataBuffer + (MaxBytes + MaxBytes * (overruns-1)), MaxBytes)   
      Else
        MessageRequester("OOPS!", "Insufficient memory to complete communication")
        Break
      EndIf
    Wend
  EndIf 
ForEver 

[Kaeru Gaman]

to explain myself:

when I said "impossible" I meant, that a real dynamic buffer would need previous information about the amound of data to receive.

anything else is "workaround" wich can be done, but is questionable of reason.

[Joakim Christiansen]

Code: Select all

BytesRead = ReceiveNetworkData(ConnectionID, *NetworkDataBuffer, MaxBytes)
While BytesRead = MaxBytes ; This means more data to be read
   ...
Wend

Sometimes I've noticed that BytesRead can actually be smaller than MaxBytes but still there is more data to be received! It's kinda weird/insane that the help file doesn't warn anyone about this. I found out that I have to include the packet size in a header and never stop reading until I got it all.

Edit:
Probably ReceiveNetworkData is a wrapper for recv()?

Code: Select all

    // Receive until the peer closes the connection
    do {

        iResult = recv(ConnectSocket, recvbuf, recvbuflen, 0);
        if ( iResult > 0 )
            printf("Bytes received: %d\n", iResult);
        else if ( iResult == 0 )
            printf("Connection closed\n");
        else
            printf("recv failed: %d\n", WSAGetLastError());

    } while( iResult > 0 );

Edit: Actually this example sucks..

[Joakim Christiansen]

lol
I send 98688 bytes of data and use this function to receive it just to test:

Code: Select all

Repeat
  DataLength = ReceiveNetworkData(ServerID,*Buffer,BufferLength)
  TotalSize + DataLength
  Debug "recieving..."+Str(TotalSize)+" "+Str(DataLength)
Until DataLength = 0
Debug "done"

And it outputs this:

recieving...7504 7504
recieving...18760 11256
recieving...22512 3752
recieving...37520 15008
recieving...45024 7504
recieving...48776 3752
recieving...52528 3752
recieving...56280 3752
recieving...60032 3752
recieving...63784 3752
recieving...67536 3752
recieving...71288 3752
recieving...75040 3752
recieving...78792 3752
recieving...82544 3752
recieving...86296 3752
recieving...90048 3752
recieving...93800 3752
recieving...98688 4888

As you can see it never debugged "done", which means there is no way for me too see when it has received all the data. This code just hangs after it received the data.

But sometimes it actually manage to receive it all in one buffer, the buffer is at 100000 bytes.

Next test:
Changing code to this (as manual suggests):

Code: Select all

Until DataLength <> BufferLength

And setting the buffer to 1000 bytes seems to help maybe, but still it will often end in unfinished transfer:

...
recieving...27000 1000
recieving...28000 1000
recieving...29000 1000
recieving...30000 1000
recieving...30660 660
done

(should be 98688)

Conclusion:
What I need to do is to rewrite the ReceiveNetworkData function...
And in these tests the SendNetworkData ALWAYS returned 98688 bytes sent, so it was NOT the problem.

Sometimes the buffer is not filled to the top: (now 2000 bytes)

recieving...94000 2000
recieving...96000 2000
recieving...96360 360
recieving...98360 2000
recieving...98688 328

What I start to understand is that this is how networks work (it can be unstable) and the help file should let us know.

[Rescator]

I forget how much exactly, but internet packets are around 1K. (!)
And obviously even if there IS more data to be received, it can't fill up the buffer fully unless it has received that much data. If a long delay or even temporary timeout or packet loss occur, the socket or api may simply give you what it's got so far.

So you have to keep fetching data until it returns 0.
And if the buffer is 100% full there may still be more data waiting, so again, keep reading until you get 0.
Try to imagine it all as a stream of data not unlike streaming audio or video files.

I don't think the PureBasic team should invest any time in this, it would lead to very bad programming practices, memory bloat and fragmentation.
Putting it into a temporary file, or into a fixed size final destination buffer (if you expect the file to be small and then just abort with an error if you receive way to much for some odd reason) would probably be better for performance.