Page 1 of 3

Buffered file transfer with unlimited filesize

Posted: Thu Aug 10, 2006 11:22 pm
by Inf0Byt3
Code updated For 5.20+


Hi, i just managed to make a buffered file transfer test and it seems to work great. The file was sent successfully here. There is not too much error-checking inside but this can be optimized and improved by anyone. Don't forget to enter a valid file path in the client's code and after executing the server and the client to press a key in the client screen to begin the transfer. If you manage to make it better please share it because this is a pretty good improvement to the network functions. We can add a callback too :D.

The server.pb file

Code: Select all

  ;by Inf0Byt3
  OpenConsole()
  
  #Command = "!ASLKLK#@*)SAUJ!"
  #DataBuf = "!SI#)()U!J:LSAJ!"
  
  Global FileHandle

  *Buffer = AllocateMemory(1024)

  If InitNetwork()
   CreateNetworkServer(0, 6443, #PB_Network_TCP)
   PrintN("Server created...")
   Repeat
    SEvent = NetworkServerEvent()
     If SEvent
      ClientID = EventClient()
      Select SEvent
       
       Case 1

       Case 2
        ReceiveNetworkData(ClientID, *Buffer, 1024)
        Command.s = PeekS(*Buffer,128)
        
       ;00000000000000000000000000000
        Select Left(Command,16)
         
         Case #Command
          DataRecv.s = Mid(Command,17,Len(Command)-16)
          Filesize.l = Val(StringField(DataRecv,1,"|"))
          Filename.s = StringField(DataRecv,2,"|")
          FileHandle = CreateFile(#PB_Any,"Received.txt")
         
         Case #DataBuf
          LengthRec = Val(RemoveString(Mid(Command,17,8),"X"))
          WriteData(FileHandle,*Buffer+24,LengthRec)
        
        EndSelect
        FreeMemory(*Buffer)
       ;00000000000000000000000000000

       Case 4
        Quit = 1
      
      EndSelect
    EndIf
   Until Quit = 1 
   CloseNetworkServer(0)
  Else
   MessageRequester("Error", "Can't create the server (port in use ?).", 0)
  EndIf
The client.pb file (execute after the server and press a key in the console)

Code: Select all


OpenConsole()

#Command = "!ASLKLK#@*)SAUJ!"
#DataBuf = "!SI#)()U!J:LSAJ!"

If InitNetwork() = 0
  PrintN("Can't initialize the network !")
  Input()
  End
EndIf

;!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
File.s = "yourfilepathhere"
;!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Global Sent,Length

ConnectionID = OpenNetworkConnection("127.0.0.1", 6443, #PB_Network_TCP)

If ConnectionID

  ;Repeat
   
   SendNetworkString(ConnectionID, #Command + Str(FileSize(File))+"|"+GetFilePart(File))
   Input()
   If ReadFile(0,File)
    Length = Lof(0)
    
    While Sent < Length
     *DataSend = AllocateMemory(1024)
     *Temp = AllocateMemory(1000)
     DataR = ReadData(0,*Temp,1000)
     BufferLength.s = LSet(Str(DataR),8,"X")
     Signature.s = #DataBuf+BufferLength+"!"
     PokeS(*DataSend,Signature)
     CopyMemory(*Temp,*DataSend+24,DataR)
     SendNetworkData(ConnectionID,*DataSend,DataR+24)
     FreeMemory(*Temp)
     FreeMemory(*DataSend)
     Sent + DataR
     PrintN(Str(Sent)+" -> "+Str(Length)+" -> "+Str(x))
     x + 1
    Wend
    
    CloseFile(0)
    CloseNetworkConnection(ConnectionID)
   EndIf
   Input()

Else
  MessageRequester("Client", "Can't find the server!!! Is it launched ?", 0)
EndIf


Re: Buffered file transfer with unlimited filesize

Posted: Fri Aug 11, 2006 3:18 am
by NoahPhense
nice code..

- np

Posted: Fri Aug 11, 2006 8:59 am
by Anden
Yipieeh. At last someone (thanks Inf0Byt3) started the revolution :-)

Things to do:

Supply "filepart number" and check it (must be +1 for each part. That ensures that the file is always put together in the right sequence). Could also be used as a progress indicator!

Unique #Command and #DataBuf values (never underestimate the odds)

Optimize :-) E.g. Move Allocate and FreeMem in client out of the While loop

Posted: Fri Aug 11, 2006 9:22 am
by Inf0Byt3
Thanks! I'm on it... For now i'm implementing a hash check on the parts and the filepart number.

Posted: Fri Aug 11, 2006 12:18 pm
by Bonne_den_kule
I made a similar program with compression, start-stop function and progress bar.

Posted: Fri Aug 11, 2006 12:21 pm
by Inf0Byt3
Wow that's very nice... Any chance to share it (or the source) ?. I am doing something similar but not with compression for now. Although it can be easily implemented.

Posted: Fri Aug 11, 2006 1:13 pm
by Konne
Ok I think there's a problem with your Receive Part.
The PB-Help tells u that the Result of ReceiveNetworkData() is te Size of Data realy read. So if the program didn't receive 1024 Byte yet u have wrong values in the file. I can tell u It's much easyer to use the Api (And it's Linux etc compatible too!).

Posted: Fri Aug 11, 2006 1:20 pm
by Inf0Byt3
Oh, I know that... This is just a skeleton... It can be improved, but that parts must be coded inside your program to make them work as you wish. It would be nice to make it send a confirmation after the bytes were received and if it is wrong (e.g. not all bytes received or a hash fail) to resend that piece.

Posted: Fri Aug 11, 2006 10:28 pm
by Bonne_den_kule
Inf0Byt3 wrote:Wow that's very nice... Any chance to share it (or the source) ?. I am doing something similar but not with compression for now. Although it can be easily implemented.
I can send you the source tomorrow.
But is still a bit unstable.
PM me your email or Skype user if any.

Posted: Fri Aug 11, 2006 10:30 pm
by Bonne_den_kule
Inf0Byt3 wrote:Oh, I know that... This is just a skeleton... It can be improved, but that parts must be coded inside your program to make them work as you wish. It would be nice to make it send a confirmation after the bytes were received and if it is wrong (e.g. not all bytes received or a hash fail) to resend that piece.
I think that the TCP/IP protocol ensures that the data is not corrupted.

Posted: Fri Aug 11, 2006 10:54 pm
by Shannara
Cool, would be sweet to have updated and/or bug fixed code :)

Posted: Fri Aug 11, 2006 10:56 pm
by Inf0Byt3
I'm on it :).

Posted: Fri Aug 11, 2006 11:04 pm
by Tranquil
Bonne_den_kule wrote:
I think that the TCP/IP protocol ensures that the data is not corrupted.
TCP ensures this, you are correct. But you have also to check for TCP errors (eg buffer overflows). If not, you will loose datas. Probably not on localhost/localhost connections but for sure on internet ones.

Posted: Sat Aug 12, 2006 12:13 am
by Inf0Byt3
It's a bit harder than it seems... Now I handle the network events in a thread. The problem is that it's using too much CPU (98-99%) and if I put a delay() in it it does not receive the whole file :(. Any advices?

[Edit]

I got it... Delay when the events are 0 :D. It's working ok now (0-1%).

Posted: Sat Aug 12, 2006 12:23 pm
by Pantcho!!
Very Nice!
Looking forward to see it done with hash error checking :)