PureBasic Forums - English

Posted: **Wed Feb 11, 2009 9:16 am**

I'm programming a client-server application using the networking commands included with PB.

Afer doing some test in the local machine, with both , client and server, I have found ReceiveNetworkData is a very slow command in my application, checked with EllapsedTime before and after the funciton. It does unusable the the application so I need to improve this.

Is there any trick to speed up the function? I suppose the function waits a lot of time for timeout or something like that but I have not found any parameter to configure the timeout, bytes to be read... in order to speed up the receiving data. Any API function usueful for this?

I don't put code in the post because I have not cutted yet to show easily the problem but it is like the samples included in the help of PB.

Any idea?

Thanks!

Posted: **Wed Feb 11, 2009 10:00 am**

Jordi wrote:... I have found ReceiveNetworkData is a very slow command ...

... but I have not found any parameter to configure the timeout, bytes to be read...

Hi! Since you mentioned timeout etc. I just wondered if you've been using the NetworkServerEvent() and/or NetworkClientEvent() functions?

The conseptual flow of a PB network program is (pseudo-code):

Code: Select all

Connect()

repeat
  if Network[Client or Server]Event() = #PB_NetworkEvent_Data
    while ReceiveNetworkData(*buffer, BytesToRead) = BytesToRead
      [Add data to mainbuffer.]
    wend
  endif

forever

Disconnect()

So basically you don't need any timeouts or checking any buffers because the NetworkServerEvent() and/or NetworkClientEvent() functions tell you when new data is available, and then you call ReceiveNetworkData() until it indicates that there is no more data by returning less bytes read than the BytesToRead param of the call.

Posted: **Wed Feb 11, 2009 2:52 pm**

Hi! Thks though we have not success.

At the beginning, reading your post, I thought oops I didn't understand the way to use the command, I thought that DataBufferLength is the size of the buffer but looking your pseudocode I thought it is the length of the frame so I implemented, but not success.

Used time to do a simple transfer for less than 50 bytes is around 15ms with client and server in the same machine, I need to decrease this.

Any other idea? THANKS.

Posted: **Wed Feb 11, 2009 3:22 pm**

checked with EllapsedTime before and after the funciton

The timing commands aren't that precise, so you should do the transfer a few thousand times in a loop and get the average time to get a proper result.

And additionally everything works much slower with the debugger enabled, so turn off that for benchmarking.

Posted: **Wed Feb 11, 2009 3:34 pm**

Used time to do a simple transfer for less than 50 bytes is around 15ms with client and server in the same machine, I need to decrease this.

1.) (as Trond mentioned): When you've "warmed up" your application + the TCP/IP stack, is the latency affected somehow? (i.e. if you send lots of messages does latency improve?)

2.) 15 ms/50 bytes may sound as much, but how fast does 1024 bytes travel? 10kb? Just to get some more measurements.

3.) What is the purpose of such low latency, over a network? If you don't want to "lock up" program waiting for network messages you might write the networking in a separate thread, to keep the main thread free to do it's work in full speed. Or are you trying to do very fast IPC?

Posted: **Wed Feb 11, 2009 5:38 pm**

I have done more tests and the global time is too high. Client show a "gui form" and it needs around 1s to fill all fields... so it doesn't good performance. I suppose I will improve the comms if in "reports" I send together some rows but this is not possible in a form gui.

Perhaps a way to improve transmissions is doing a third layer program, packaging all data before transmission, decreasing the queries then streams will be bigger and I suppose performance higher.

Debugger is disabled, from IDE.

Anybody knows the ReceiveNetworkData internals or the API related to it?

Thanks for your answers.

Posted: **Wed Feb 11, 2009 6:03 pm**

You should definetely pack it all in one big chunk before sending. Are you sure the part slowing it down isn't the repainting of the GUI?

Posted: **Wed Feb 11, 2009 11:39 pm**

You should definetely pack it all in one big chunk before sending.

I agree 100% probably you will get 15-20ms for the whole package on a local machine, so shorter than you can blink your eye.

Another very big advantage is that once you have a package (serializing and deserializing of your data) it's often very trivial to add compression as another layer on top, which will help even more on remote network connections.

One way to do serializing is to make a mirror Structure that has fields for all fields of your form, and then write two procedures:
1) Form2Struct() - moves all field-values from the form to a struct
2) Struct2Form() - (vice versa.)

This way you can threat your complete struct as a memory area and send through a network, or save to a file or anything you desire. (i.e. these mechanisms can often be reused to other good stuff once they are in place)

Posted: **Thu Feb 12, 2009 8:38 am**

Thnks Mahan and Trond.

I am starting with a "packing" or "serializing" function after that I will add a function to crypt the data and after a compression routine, or perhaps mixing both.

Anyone knows which algorithm is better for a short data length?

Thanks

Posted: **Thu Feb 12, 2009 11:07 am**

in TCP your data will be split up in MTU Sized Fragments. So packing it up to that size gives maximum speed.

15 ms is normal for a windows system ( provided you use a delay(1) somewhere - can be also a waitfor event - the delay / sleep function of XP eg has a precision of 10ms ).

so, just use a loop/delay when no time critical things going on - eg noone is on - or no events fired after a while.
PBs Networking commands arent much slower than the native API networking cmds of windows - just very much easer to use and crossplatform

for short datapackets group what correlates together and serialize it. -> send in some chunk - and for alot of small data ( eg events ) - serialize em and if possible chunk em up in bigger packets. Alot small packets = alot of router load and more latency because per packet 48 bytes are added from TCP. etc etc.

What do you want to achieve mainly ?

Cheers,
Thalius

Posted: **Thu Feb 12, 2009 12:21 pm**

Client show a "gui form" and it needs around 1s to fill all fields... so it doesn't good performance.

If you have 1000000000000000000000 fields, then filling them in 1 second is really good performance. (In other words: your post told us nothing since you didn't tell us the number of fields.)

Posted: **Thu Feb 12, 2009 12:45 pm**

I can send and receive 100 000 50-byte messages in 400ms here. If I add the messages to a listview as they are received I can receive 10 000 50-byte strings in 266ms.

Oups, that was with the debugger enabled. Without it I can send and receive one million 50-byte strings in 3.4 seconds, which equals almost 300 messages per millisecond.

Are you sure it's ReceiveNetworkData() which has a problem and not your code? I think it's your code.

Posted: **Thu Feb 12, 2009 1:51 pm**

I will double check again the code but I have not found any error til today.

Fields are around 20, 1 million or more is technically impossible on a nowdays monitor Trond

I will create a test program, with server and client simplifying the code to get the speed of the system because Trond performance is perfect. Anyone has done it before?

Anyway when client and server are in different machines I will take care of the length of the frame, to have the MTU length maximum.

The program is a database based one, is a gateway between an ERP and our "plugins".

How stop the debugger? Only by unchecking it in Compiler Options/Compiler-Debbuger?

Thanks to all!

Posted: **Thu Feb 12, 2009 3:51 pm**

Here is a test code (hardcoded to 50-byte strings):

Code: Select all


; Server
InitNetwork()


OpenWindow(0, 0, 0, 900, 384, "", #PB_Window_ScreenCentered | #PB_Window_SystemMenu)
ListViewGadget(0, 10, 10, 900-20, 384-20)

Port = 6832
*Buffer = AllocateMemory(1000)

If CreateNetworkServer(0, Port)
  Repeat
    SEvent = NetworkServerEvent()
    If SEvent
      ClientID = EventClient()
      Select SEvent
        Case #PB_NetworkEvent_Connect
          AddGadgetItem(0, -1, "Server: A new client (" + Str(ClientID) + ") has connected")
          T = ElapsedMilliseconds()
        Case #PB_NetworkEvent_Data
          ReceiveMore:
          ReceiveLen = ReceiveNetworkData(ClientID, *Buffer, 50)
            C + 1
            ; Comment these to see the speed without adding to the listview all the time
            AddGadgetItem(0, -1, "Got " + PeekS(*Buffer))
            AddGadgetItem(0, -1, "Total time used: " + Str(ElapsedMilliseconds()-T))
            AddGadgetItem(0, -1, "Total count: " + Str(C))
          If ReceiveLen = 50
            Goto ReceiveMore
          EndIf
          ; Uncomment this to see the time if you uncomment the above lines
          ; AddGadgetItem(0, -1, "Total time used: " + Str(ElapsedMilliseconds()-T))
        Case 4
          AddGadgetItem(0, -1, "Server: Client "+Str(ClientID)+" has closed the connection.")
          AddGadgetItem(0, -1, "Total time used: " + Str(ElapsedMilliseconds()-T))
      EndSelect
    Else
      Event = WindowEvent()
      Select Event
        Case #PB_Event_CloseWindow
          Break
        Default
          Delay(1)
      EndSelect
    EndIf
  ForEver
  CloseNetworkServer(0)
Else
  MessageRequester("Error", "Can't create the server (port in use ?)", 0)
EndIf

Code: Select all

; Client
InitNetwork()
Port = 6832
ConnectionID = OpenNetworkConnection("127.0.0.1", Port)
If ConnectionID
  For I = 0 To 1000
    S.s = "1234567890abcdeghijklmnopqrstuvABCDEFGHIJKLMNOPQR"
    L = Len(S) + 1
    While SendNetworkData(ConnectionID, @S, L) <> L
      Delay(1)
    Wend
  Next
  ;Delay(1000)
  CloseNetworkConnection(ConnectionID)
Else
  MessageRequester("Client", "Can't find the server (Is it launched?).", 0)
EndIf

To disable the debugger, uncheck "Use debugger" in the debugger menu, but I find that the speed is more than adequate even with the debugger enabled.

Fields are around 20, 1 million or more is technically impossible on a nowdays monitor Trond

You could place them in a scrollable list, I assumed there was a lot of them since it took so long to fill them. Filling 20 fields should easily be done in under 50 milliseconds.

Posted: **Thu Feb 12, 2009 5:36 pm**

Many many thanks Trond for the code, I seem very lazy...

I have done some tests, these are the results:

Showing all the message, in every step: 110ms - 150 ms total time
Showing only few steps with modulo % 10 : 32ms total time
Showing only final step: 15ms total time

So it seems is very fast, I am wrong... the debugger on... I don't need more speed so thinking in this I think the problem is in the server routine, perhaps doesn't showing data by console I can speed up the system or doing it in another thread is better. Though the server is a service and I had problems doing it and I don't want change a lot this also I have to recode with some new code from this forum for NTServices.

Any other idea for the delays different to show data by console or gui?

PureBasic Forums - English

ReceiveNetworkData issue

ReceiveNetworkData issue

Re: ReceiveNetworkData issue