Module NetworkData - Send strings and data over 64kb

[mk-soft]

Update v1.15
- Added: SetAESData (...). Send data encrypted.
- Added: SetUserData (...) and GetUserData (...) for each Connection.

Note UserData!
Access to the UserData is only possible within the NewDataCallback since each server or client has its own environment (threaded).
If a *pointer is used on your own data, you may have to release it again at event #PB_NetworkEvent_Disconnect.

[mk-soft]

Update v1.18
- Added some Mutex
- Optimize code
- Update examples server and client

[mk-soft]

Update v1.21
- Fixed SendList: Now ClearList on exists DataSet
- Optimize: Protocol header and data block are now also encrypted

Update examples too

[mk-soft]

Update v1.23
- Fixed MacOS - App Nap: Process go not longer to sleep. Thank to Danilo and Wilbert
- DataSet now intern a pointer to structure. We can free dataset later with new function FreeDataSet(...)
- Added: FreeDataSet(...)
- Remove: CopyDataSet(...)
- Added new constants for result of Callback
-- #NetResult ; Hold DataID and DataSet
-- #NetResultFreeData ; Free DataID and DataSet
-- #NetResultFreeDataWithoutDataSet ; Free DataID without DataSet


Thanks to Danilo and Wilbert for deactivate AppNap (MacOS)

[novablue]

Sorry for digging up an old thread but i find this module really useful so thanks for sharing this with us!

The only issue i am having with it is that sending data or files is very slow and nowhere near exceeding my upload bandwidth, it is sending at a speed of roughly 50kb/s out of the 1mb/s upload bandwidth that i have. Does anyone have an idea why this could be and if it can be improved? Or could this be an limitation with the pb network lib itself?

[novablue]

I have been using this for a long time now and i am still getting occasional crashes on SendNetworkData (particulary in the SendString() procedure). Any idea why this happens or how this could be fixed? From what i can tell SendNetworkData is being locked with a mutex so it should not happen but yet it does when run in a thread.

[Bitblazer]

@novablue

Maybe it is time to test it with the new PB 6 C backend. Bug fixing seems to be focussed there right now.

Make a minimal example and do a bug report. If it happens in windows - use WinAPIOverride to trace where exactly the crash happens. If you can trigger it in linux, use dmesg or something similar, the name can differ depending on the distribution.

Does it happen if you disable virtual memory? Did you do a memory / hardware / CPU test? I have had problems with VMM paging of large memory areas in the past, but that had to do with hardware or VMM driver bugs. Did you try software like CPU-Z, MemTest86 or AIDA64 to thoroughly test your hardware?

Does it depend on the involved devices? Does it happen with other software too?

[novablue]

It is hard to make an example. i am using this include in a large project and then just sometimes the program will IMA randomly on SendNetworkData. I am pretty sure it is because of threads being used and not necessarily because this include has a bug. From my experience using threads in PureBasic no matter how well written it will crash sooner than later. It cant be my system either I am running my program using this include on many different computers and on all it will just crash occasionally. I will try the new c back end soon if that makes any difference.

[mk-soft]

@novablue

I assume that you have set the compiler option 'ThreadSafe' in the main file. Only with this option string are thread safe.

[novablue]

@novablue

I assume that you have set the compiler option 'ThreadSafe' in the main file. Only with this option string are thread safe.

Yes i have ThreadSafe mode on, i will investigate stability further with the new c back end.

I have some side questions that don't really have anything to do with this include but they might be fitting here:

On windows is there any way to close a stuck network connection on the server side? For example: a client connects to the server and the clients internet connection gets cut off. On the server the #PB_NetworkEvent_Disconnect event will not happen sometimes and the client will be stuck forever it seems. I can use a ping / pong method to test if the client still responds and then ignore lost connections but Is there any way to actually close the client connection manually server sided?

My second question: Is it possible to verify if a connection handle is still valid? Something like isConnection() would be really useful... "ConnectionID()" will return an unique system identifier maybe there is a way with Winapi to check if that handle is still valid?

[mk-soft]

You can also close a connection on the server side. The PB function CloseNetwerkConnection also works on the server side. Even the event PB_Event_Disconnet arrives at the client most of the time, even if it is not described in the PB help.

However, it is not yet implemented in the module that the server disconnects a single connection.
A timeout monitoring must still be programmed in the server thread for this. If no data comes from the client after time x, the connection could be disconnected from the server. However, the client must at least send data after time x or a Lifetelegram .

[novablue]

You can also close a connection on the server side. The PB function CloseNetwerkConnection also works on the server side. Even the event PB_Event_Disconnet arrives at the client most of the time, even if it is not described in the PB help.

However, it is not yet implemented in the module that the server disconnects a single connection.
A timeout monitoring must still be programmed in the server thread for this. If no data comes from the client after time x, the connection could be disconnected from the server. However, the client must at least send data after time x or a Lifetelegram .

OK thanks i did not know CloseNetwerkConnection works server sided. I tried to implement it myself and it seems to be working. For now the pinging/timeout is handled by my program itself, maybe adding it internally to the module would be better in the future. If you have a better way of doing this please let me know.

Code: Select all

Structure udtServerList
    ServerID.i
    ThreadID.i
    ExitServer.i
    CloseClientConnection.i ; New Added Changes
    NewDataCB.ProtoNewDataCB
EndStructure

Procedure ServerCloseClientConnnection(ServerID, ConnectionID) ; New Added Changes
    Protected keyServerID.s = Hex(ServerID)
    
    If (FindMapElement(ServerList(), keyServerID)) 
        While (ServerList(keyServerID)\CloseClientConnection) : Delay(10) : Wend
        ServerList(keyServerID)\CloseClientConnection = ConnectionID
    EndIf
EndProcedure

Procedure ThreadServer(*this.udtServerList) 
    Protected Event, ConnectionID, keyConnectionID.s, count
    
    With *this
        Repeat 
            LockMutex(LockServer)
            Event = NetworkServerEvent(\ServerID)
            If (Event) 
                ConnectionID = EventClient()
            EndIf
            
            ; New Added Changes
            If (\CloseClientConnection) 
                keyConnectionID = Hex(\CloseClientConnection)
                
                If (FindMapElement(DataConnection(), keyConnectionID)) 
                    CloseNetworkConnection(DataConnection()\ConnectionID)
                    If (\NewDataCB) 
                        \NewDataCB(#PB_NetworkEvent_Disconnect, DataConnection()\ConnectionID, 0)
                    EndIf
                    ForEach DataConnection()\DataPacket() 
                        FreeDataPacket(DataConnection()\DataPacket())
                    Next
                    DeleteMapElement(DataConnection(), keyConnectionID)
                EndIf
                
                \CloseClientConnection = #False
            EndIf
            ; New Added Changes ^^
            
            UnlockMutex(LockServer)
            Select (Event) 
                Case #PB_NetworkEvent_Connect 
                    ; Create DataConnection
                    keyConnectionID = Hex(ConnectionID)
                    If (FindMapElement(DataConnection(), keyConnectionID)) 
                        ForEach DataConnection()\DataPacket() 
                            FreeDataPacket(DataConnection()\DataPacket())
                        Next
                        DeleteMapElement(DataConnection(), keyConnectionID)
                    
                    Else 
                        AddMapElement(DataConnection(), keyConnectionID)
                        DataConnection()\ConnectionID = ConnectionID
                        ; 							Logging("Network: Client connected: ID " + IPString(GetClientIP(ConnectionID)))
                    EndIf
                    If (\NewDataCB) 
                        \NewDataCB(#PB_NetworkEvent_Connect, ConnectionID, 0)
                    EndIf
                
                Case #PB_NetworkEvent_Data 
                    ReceiveData(ConnectionID,\NewDataCB)
                
                Case #PB_NetworkEvent_Disconnect 
                    ; Destroy DataConnection
                    keyConnectionID = Hex(ConnectionID)
                    ; 						Logging("Network: Client disconnected: ID " + keyConnectionID)
                    If (\NewDataCB) 
                        \NewDataCB(#PB_NetworkEvent_Disconnect, ConnectionID, 0)
                    EndIf
                    If (FindMapElement(DataConnection(), keyConnectionID)) 
                        ForEach DataConnection()\DataPacket() 
                            FreeDataPacket(DataConnection()\DataPacket())
                        Next
                        DeleteMapElement(DataConnection(), keyConnectionID)
                    EndIf
                
                Default 
                    Delay(20)
                
            EndSelect
        Until (\ExitServer)
        
        ; Clear all DataConnection. We can delete all the data, because each server have their own DataConnection. DataConnection is threaded
        ForEach DataConnection() 
            If (\NewDataCB) 
                \NewDataCB(#PB_NetworkEvent_Disconnect, DataConnection()\ConnectionID, 0)
            EndIf
            ForEach DataConnection()\DataPacket() 
                FreeDataPacket(DataConnection()\DataPacket())
            Next
            ClearMap(DataConnection()\DataPacket())
        Next
        ClearMap(DataConnection())
        ; Exit Thread
        \ExitServer = 0
    EndWith
EndProcedure