Buffered file transfer with unlimited filesize

Konne · Post by **Konne** » Sat Aug 12, 2006 1:35 pm

Inf0Byt3 wrote:It's a bit harder than it seems... Now I handle the network events in a thread. The problem is that it's using too much CPU (98-99%) and if I put a delay() in it it does not receive the whole file . Any advices?

[Edit]

I got it... Delay when the events are 0 . It's working ok now (0-1%).

Or just use Api.

Inf0Byt3 · Post by **Inf0Byt3** » Sat Aug 12, 2006 1:44 pm

Well yeah... But for now i just gotta get used to networking. It's the first time I use the networking functions ^^.

Bonne_den_kule · Post by **Bonne_den_kule** » Sat Aug 12, 2006 4:21 pm

Link to the source of my program:
http://www.photo-host.org/img/618544fil ... rogram.gif
Save it as .zip.

It contains for program: A file list server (make and sends file lists), a file server (sends the file), and a client.

File list server sources:
*ListServerMain.pb <---compile this
*ServerInclude.pb
*GlobalInclude.pb

File server sources:
*FileServerMain.pb <---compile this
*ServerInclude.pb
*GlobalInclude.pb

Client sources:
*ClientMain.pb <---compile this
*ClientInclude.pb
*ClientEvents.pb
*GlobalInclude.pb
*Logo.jpg

Don't compile as threadsafe and turn off the debugger to get rid of some invalid memory access errors. Hard to read, and it contains a lot of nested macros and threads. You will find the send/recieve stuff in the ServerInclude and ClientIncludeFile. The -main files contains the network loop/events handling and UI (in the -include file for the client). Works OK. The file list server is 99% stable, but the other is still a bit unstable. Tested on both local network and over internet. The file sending procedure is a bit slow on high speed transfer because of the data send/recieve system I have made. My first time using network functions.

EDIT: Just delete the .ini file or set host2=127.0.0.1

Inf0Byt3 · Post by **Inf0Byt3** » Sat Aug 12, 2006 4:30 pm

Same here ^^. Thanks very much

. My program is a bit slow too but now it seems to work perfectly. I am implementing now the sanity check for the packets.

Bonne_den_kule · Post by **Bonne_den_kule** » Sat Aug 12, 2006 4:32 pm

Inf0Byt3 wrote:Same here ^^. Thanks very much . My program is a bit slow too but now it seems to work perfectly. I am implementing now the sanity check for the packets.

I will try to make some code too, and at the end we can mix it together.

Inf0Byt3 · Post by **Inf0Byt3** » Sat Aug 12, 2006 4:33 pm

Perfect!

Bonne_den_kule · Post by **Bonne_den_kule** » Sun Aug 13, 2006 4:22 pm

My code at this moment.
The code is buffered and with functions to ensure that all data is sent/received. The speed on my local computer was 3200kB/s average on a ca 300 MB file.

Server:

Code: Select all

EnableExplicit
IncludeFile "Include.pb"
EnableGraphicalConsole(1)
OpenConsole()
ConsoleTitle("Server")

Define Quit.b, DataReceived.l, Buffer.l=AllocateMemory(#BufferSize), FileSize.q, FileOffset.q, FileName.s, Reserved.l
Define Path.s, TotaltDataReceived.q, DataInBuffer.l, ClientID.l, StartTime.l, Timer.l, TempDataReceived.l

If InitNetwork()
  PrintN("Init network done")
  If CreateNetworkServer(#Server, #Server_Port)
    PrintN("Server created")
    
    Repeat
    
      Select NetworkServerEvent()
        
        Case #PB_NetworkEvent_Connect
          PrintN("Client connected")
        
        Case #PB_NetworkEvent_Disconnect
          PrintN("Client disconnected")
          Quit=1
        
        Case #PB_NetworkEvent_Data
          ClientID=EventClient()
          DataReceived=ReceiveNetworkDataEx(ClientID, Buffer)
          If DataReceived
            PrintN("Valid data received")
            Select PeekB(Buffer+4) ;Reading network message
              
              Case #SendFile ;A message sent from the client
                FileSize=PeekQ(Buffer+5) ;Length of file or the length of the part of the file to be received
                FileOffset=PeekQ(Buffer+13) ;File offset (the offset in the file to write the received datas)
                Reserved=PeekL(Buffer+21) ;Reserved space in the packet for other info (to be used in the future maybe)
                FileName=PeekS(Buffer+25, DataReceived-25) ;File name
                
                If CheckFilename(FileName) And FileSize<>0
                  PrintN("File transfer request accepted")
                  PrintN("Name: "+FileName+", size of data/file: "+Str(Int(FileSize/1024))+"kB")
                  Path.s=PathRequester("Where to save "+FileName, "")
                  If Path
                    If OpenFile(#File, Path+FileName) 
                      FileBuffersSize(#File, #BufferSize*10)
                      ;Making a packet to send to the client to say that the request is accepted
                      PokeL(Buffer, 5) ;The size of the packet
                      PokeB(Buffer+4, #RequestAccepted)
                      If SendNetworkDataEx(ClientID, Buffer)
                        Repeat
                          Delay(1)
                        Until NetworkServerEvent()=#PB_NetworkEvent_Data
                        
                        PrintN("Receiving the file")
                        FileSeek(#File, FileOffset)
                        StartTime=ElapsedMilliseconds()
                        Timer.l=ElapsedMilliseconds()
                        
                        ;The file receive loop
                        Repeat
                        
                          Repeat
                            Delay(1)
                          Until NetworkServerEvent()=#PB_NetworkEvent_Data
                          
                          DataReceived=ReceiveNetworkData(ClientID, Buffer+DataInBuffer, Min(#BufferSize-DataInBuffer, FileSize-TotaltDataReceived))
                          If DataReceived<=-1
                            PrintN("Connection error")
                            WriteData(#File, Buffer, DataInBuffer)
                            DataInBuffer=0
                            Break
                          EndIf
                          TempDataReceived+DataReceived
                          TotaltDataReceived+DataReceived
                          DataInBuffer+DataReceived
                          
                          If DataInBuffer>Int(#BufferSize*0.9) Or TotaltDataReceived=FileSize
                            WriteData(#File, Buffer, DataInBuffer)
                            DataInBuffer=0
                          EndIf
                          If ElapsedMilliseconds()-Timer>1000
                            PrintN("Progress: "+StrD((TotaltDataReceived/FileSize)*100, 1)+"%, speed: "+StrD((TempDataReceived/1024)/((ElapsedMilliseconds()-Timer)/1000), 2)+"kB/s")
                            Timer=ElapsedMilliseconds()
                            TempDataReceived=0
                          EndIf
                                                
                        Until TotaltDataReceived=FileSize 
                         
                        If DataReceived>-1
                          PrintN("File sent")
                          PrintN("Average speed: "+StrD((FileSize/1024)/((ElapsedMilliseconds()-StartTime)/1000), 2)+"kB/s, time used: "+FormatTime(Int(ElapsedMilliseconds()-StartTime)/1000))
                        EndIf
                        CloseFile(#File)
                         
                      EndIf
                        
                    Else
                      PrintN("Cant create file")
                      CloseNetworkConnection(ClientID)
                    EndIf
                  Else
                    PrintN("Wrong path")
                    CloseNetworkConnection(ClientID)
                  EndIf
                Else
                  PrintN("Wrong file name or size")
                  CloseNetworkConnection(ClientID)    
                EndIf
                
            EndSelect
          Else
            PrintN("Garbarge received from client")
            CloseNetworkConnection(ClientID)          
          EndIf
          
      EndSelect    
      Delay(1)
    Until Quit=1
    
  Else
    PrintN("Error creating server")
  EndIf
Else
  PrintN("Error initing network")
EndIf

CloseNetworkServer(#Server)
FreeMemory(-1)
PrintN("Server closed")
Input()

Client:

Code: Select all

EnableExplicit
IncludeFile "Include.pb"
EnableGraphicalConsole(1)
OpenConsole()
ConsoleTitle("Client")

Define ConnectionID, Buffer.l=AllocateMemory(#BufferSize), File.s, DataRead.l, DataSent.l, TotaltDataSent.l, TotalBufferSent.q
Define StartTime.l, Timer.l, TempDataSent.l

If InitNetwork()
  PrintN("Init network done")
  ConnectionID=OpenNetworkConnection(#HostName, #Server_Port)
  
  If ConnectionID
    PrintN("Connected to server")
    File=OpenFileRequester("File to send", "", "", 0)
    
    If FileSize(File)>0 And ReadFile(#File, File)
      FileBuffersSize(#File, #BufferSize*10)
      ;A packet to send to server:
      PokeL(Buffer, Len(GetFilePart(File))+25)
      PokeB(Buffer+4, #SendFile)
      PokeQ(Buffer+5, FileSize(File)) ;Length of file to send
      PokeQ(Buffer+13, 0) ;File offset, zero in this example
      PokeL(Buffer+21, 0) ;Reserved for use in the future
      PokeS(Buffer+25, GetFilePart(File))
      SendNetworkDataEx(ConnectionID, Buffer)
      
      Repeat
        Delay(1)
      Until NetworkClientEvent(ConnectionID)=#PB_NetworkEvent_Data
      
      If ReceiveNetworkDataEx(ConnectionID, Buffer)=5 And PeekB(Buffer+4)=#RequestAccepted
        PrintN("Server accepted the request")
        ;The file send loop
        PrintN("Starting sending the file, name: "+GetFilePart(File)+", size: "+Str(Int(FileSize(File)/1024))+"kB")
        StartTime=ElapsedMilliseconds()
        Timer=ElapsedMilliseconds()
        
        Repeat
          DataRead=ReadData(#File, Buffer, Min(#BufferSize,Lof(#File)-TotaltDataSent))
          TotalBufferSent=0
          Repeat
          
            DataSent=SendNetworkData(ConnectionID, Buffer, DataRead-TotalBufferSent)
            If DataSent<=-1
              PrintN("Connection error")
              Break 2
            EndIf
            TotalBufferSent+DataSent
            TempDataSent+DataSent
            
            If ElapsedMilliseconds()-Timer>1000
              PrintN("Progress: "+StrD((TotaltDataSent/FileSize(File))*100, 1)+"%, speed: "+StrD((TempDataSent/1024)/((ElapsedMilliseconds()-Timer)/1000), 2)+"kB/s")
              Timer=ElapsedMilliseconds()
              TempDataSent=0
            EndIf
            
          Until DataRead=TotalBufferSent
          TotaltDataSent+TotalBufferSent
          
        Until TotaltDataSent=Lof(#File)
        
        If DataSent>-1
          PrintN("File sent to server")
          PrintN("Average speed: "+StrD((FileSize(file)/1024)/((ElapsedMilliseconds()-StartTime)/1000), 2)+"kB/s, time used: "+FormatTime(Int(ElapsedMilliseconds()-StartTime)/1000))
        EndIf
          
      Else
        PrintN("Server could not download file")
      EndIf
              
    Else
      PrintN("Wrong file, empty file or no file specified")
    EndIf 
       
  Else
    PrintN("Error connecting to server")
  EndIf
  
Else
  PrintN("Error initing network")
EndIf

If ConnectionID
  CloseNetworkConnection(ConnectionID)
EndIf
If IsFile(#File)
  CloseFile(#File)
EndIf

FreeMemory(-1)
PrintN("Client closed")
Input()

Include file for both server and client:

Code: Select all

#Server_Port=2585
#HostName="127.0.0.1" ;localhost
#BufferSize=1024*1024
#TimeOut=5000

Enumeration
  #Server
  #File
  #SendFile
  #RequestAccepted
EndEnumeration


Procedure ReceiveNetworkDataEx(ConnectionID.l, Buffer.l) ;Function to prevent some kinds of buffer overflow's and ensure that all data is received
  Define DataRead.l, TotaltDataRead.l, Length.l=4, StartTime.l=ElapsedMilliseconds()
  Repeat
  
    DataRead=ReceiveNetworkData(ConnectionID, Buffer+TotaltDataRead, Length-TotaltDataRead)
    TotaltDataRead+DataRead
    
    If TotaltDataRead>=4
      Length=PeekL(Buffer)
    ElseIf DataRead<=-1 Or Length>#BufferSize Or (ElapsedMilliseconds()-StartTime>#TimeOut And TotaltDataRead<=4)
      ProcedureReturn 0
    EndIf
    
    Delay(1)
  Until TotaltDataRead=Length And Length<>0
  ProcedureReturn TotaltDataRead
EndProcedure
Procedure SendNetworkDataEx(ConnectionID.l, Buffer.l) ;Function to ensure that all data is sent
  Define DataSent.l, TotaltDataSent.l, StartTime.l=ElapsedMilliseconds(), Length=PeekL(Buffer)
  Repeat
  
    DataSent.l=SendNetworkData(ConnectionID, Buffer+TotaltDataSent, Length-TotaltDataSent)
    TotaltDataSent+DataSent
    
    If DataSent<=-1  Or (ElapsedMilliseconds()-StartTime>#TimeOut And TotaltDataSent=0) Or Length=0
      ProcedureReturn 0
    EndIf
    
    Delay(1)      
  Until TotaltDataSent=Length And TotaltDataSent>0
  ProcedureReturn 1
EndProcedure

Procedure.q Min(val1.q, val2.q) ;Returns the lowest value
  If val1<val2
    ProcedureReturn val1
  Else
    ProcedureReturn val2
  EndIf
EndProcedure
Procedure.s FormatTime(seconds.l)
  Define Date.s=FormatDate("%hhh %iim %sss", AddDate(0, #PB_Date_Second, seconds))
  Date=RemoveString(Date, "00h")
  Date=RemoveString(Date, "00m")
  Date=RemoveString(Date, "00s")
  Date=Trim(Date)
  If Left(Date, 1)="0"
    Date=Right(Date, Len(Date)-1)
  EndIf
  ProcedureReturn Date
EndProcedure

Inf0Byt3 · Post by **Inf0Byt3** » Sun Aug 13, 2006 6:26 pm

Wow this is awesome

. I will now study your code to see how it exactly works. I personally had some problems implementing the 'resend if wrong' stuff. What i've done was working only on my computer, but if i wanted to send a file to somebody over LAN it failed to deliver all the packets. Thanks for the code, this is much appreciated

Bonne_den_kule · Post by **Bonne_den_kule** » Mon Aug 14, 2006 8:39 am

Have you tested my code over LAN?
I haven't done it yet.

EDIT: Tested my code on LAN (wireless, 54Mb/s), got ca 1400 kB/s in average.

Inf0Byt3 · Post by **Inf0Byt3** » Mon Aug 14, 2006 12:24 pm

I haven't tested it over LAN yet, because my neighbour is not home for a few days

. Locally it works fantastic

Konne · Post by **Konne** » Mon Aug 14, 2006 12:40 pm

If want to test it over lan I can help u just send me a PM. (And if u're not German the time difference)

Bonne_den_kule · Post by **Bonne_den_kule** » Mon Aug 14, 2006 1:11 pm

I will make my code threaded to increase the speed, and post the result when I am finished.

Bonne_den_kule · Post by **Bonne_den_kule** » Mon Aug 14, 2006 11:17 pm

Well, the threads did only slow down the process a bit, no speed increasement. Strange, this means that the buffered file functions are damn fast and maybe thread too. (Thanks Fred).

Inf0Byt3 · Post by **Inf0Byt3** » Tue Aug 15, 2006 12:06 am

This is strange

. I've tested your old code over the net and it works brilliantly.

Bonne_den_kule · Post by **Bonne_den_kule** » Tue Aug 15, 2006 7:33 pm

I finished with my code and will post it as a new thread at the forum.
I got the threads to work fast now, and you will see a major speed increasement when using bigger buffers while sending large files.