Page 1 of 1

speed up procedure

Posted: Wed Jan 12, 2011 3:10 pm
by Primus1
Hello,

I want to check a list of several urls, I'm using this method:

<test snippet from the full program, works directly in PB>

Code: Select all

InitNetwork()

Procedure.s ReceiveHTTPString(URL$, TimeOut=2000)
   Protected Event, Time, Size, String$, Inhalt
   Protected BufferSize = $1000, *Buffer = AllocateMemory(BufferSize)
   Protected ServerName$ = GetURLPart(URL$, #PB_URL_Site)
   Protected ConnectionID = OpenNetworkConnection(ServerName$, 80)
   If ConnectionID
      SendNetworkString(ConnectionID, "GET "+URL$+" HTTP/1.0"+#LFCR$+#LFCR$)
      Time = ElapsedMilliseconds()
      Repeat
         Delay(10)
         Event = NetworkClientEvent(ConnectionID)
         If Event = #PB_NetworkEvent_Data
            Repeat
               Size = ReceiveNetworkData(ConnectionID, *Buffer, BufferSize)
               String$ + PeekS(*Buffer, Size, #PB_Ascii)
            Until Not Size
            Inhalt = FindString(String$, #LFCR$, 1)
            If Inhalt
               String$ = Mid(String$, Inhalt+3)
            EndIf
         EndIf   
      Until ElapsedMilliseconds()-Time > TimeOut Or String$
      CloseNetworkConnection(ConnectionID)
   EndIf
   FreeMemory(*Buffer)
   ProcedureReturn String$
EndProcedure

content.s=""

  StartTime.l = ElapsedMilliseconds()

For i=1 To 20
  content=ReceiveHTTPString("http://www.example.com")
Next i  

  Debug "Duration: "+Str(ElapsedMilliseconds() - StartTime)+" milliseconds."
  
Debug(content)
In the for next loop different urls will be placed in the content variable.


Is there anything that can speed this up? currently the program takes around 6 seconds to complete, to be precise: 6422 miliseconds. In the end it has to check url lists of 2000-10000 url's.

Re: speed up procedure

Posted: Wed Jan 12, 2011 4:59 pm
by Trond
The most obvious way to speed it up would be to do several requests at the same time. This could be done by changing ReceiveHTTPString(), or by running it in parallell with multiple threads.

Re: speed up procedure

Posted: Wed Jan 12, 2011 5:11 pm
by Primus1
Thanks, I tried the multiple threads but did not succeed, would speed things up if some or all code in the procedure was changed into asm?

Re: speed up procedure

Posted: Wed Jan 12, 2011 8:04 pm
by Trond
would speed things up if some or all code in the procedure was changed into asm?
Absolutely not, the bottleneck is the connection to the server. The only way to speed it up is to connect to multiple servers at once.

Re: speed up procedure

Posted: Thu Jan 13, 2011 5:13 pm
by Primus1
Thanks, I thought so. I've finally succeeded in making the program multithreaded and got a 29% speed increase! Working with threads the first time is pretty complex. :?

Re: speed up procedure

Posted: Mon Jan 24, 2011 1:39 am
by Joakim Christiansen
Primus1 wrote:Thanks, I thought so. I've finally succeeded in making the program multithreaded and got a 29% speed increase! Working with threads the first time is pretty complex. :?
Yeah, threaded programming takes some time to learn.
Make sure you enable "threadsafe" in the compiler options, this will help you a little.

For HTTP requests like these I don't recommend using more than 10 threads, but feel free to experiment.

Re: speed up procedure

Posted: Mon Jan 24, 2011 8:27 am
by DarkDragon
You should also check whether the connection is still alive (So it won't wait for timeout anymore after the file was sent):
http://www.purebasic.fr/english/viewtop ... disconnect