speed up procedure

[Primus1]

Hello,

I want to check a list of several urls, I'm using this method:

<test snippet from the full program, works directly in PB>

Code: Select all

InitNetwork()

Procedure.s ReceiveHTTPString(URL$, TimeOut=2000)
   Protected Event, Time, Size, String$, Inhalt
   Protected BufferSize = $1000, *Buffer = AllocateMemory(BufferSize)
   Protected ServerName$ = GetURLPart(URL$, #PB_URL_Site)
   Protected ConnectionID = OpenNetworkConnection(ServerName$, 80)
   If ConnectionID
      SendNetworkString(ConnectionID, "GET "+URL$+" HTTP/1.0"+#LFCR$+#LFCR$)
      Time = ElapsedMilliseconds()
      Repeat
         Delay(10)
         Event = NetworkClientEvent(ConnectionID)
         If Event = #PB_NetworkEvent_Data
            Repeat
               Size = ReceiveNetworkData(ConnectionID, *Buffer, BufferSize)
               String$ + PeekS(*Buffer, Size, #PB_Ascii)
            Until Not Size
            Inhalt = FindString(String$, #LFCR$, 1)
            If Inhalt
               String$ = Mid(String$, Inhalt+3)
            EndIf
         EndIf   
      Until ElapsedMilliseconds()-Time > TimeOut Or String$
      CloseNetworkConnection(ConnectionID)
   EndIf
   FreeMemory(*Buffer)
   ProcedureReturn String$
EndProcedure

content.s=""

  StartTime.l = ElapsedMilliseconds()

For i=1 To 20
  content=ReceiveHTTPString("http://www.example.com")
Next i  

  Debug "Duration: "+Str(ElapsedMilliseconds() - StartTime)+" milliseconds."
  
Debug(content)

In the for next loop different urls will be placed in the content variable.


Is there anything that can speed this up? currently the program takes around 6 seconds to complete, to be precise: 6422 miliseconds. In the end it has to check url lists of 2000-10000 url's.

[Trond]

The most obvious way to speed it up would be to do several requests at the same time. This could be done by changing ReceiveHTTPString(), or by running it in parallell with multiple threads.

[Primus1]

Thanks, I tried the multiple threads but did not succeed, would speed things up if some or all code in the procedure was changed into asm?

[Trond]

would speed things up if some or all code in the procedure was changed into asm?

Absolutely not, the bottleneck is the connection to the server. The only way to speed it up is to connect to multiple servers at once.

[Primus1]

Thanks, I thought so. I've finally succeeded in making the program multithreaded and got a 29% speed increase! Working with threads the first time is pretty complex.

[Joakim Christiansen]

Thanks, I thought so. I've finally succeeded in making the program multithreaded and got a 29% speed increase! Working with threads the first time is pretty complex.

Yeah, threaded programming takes some time to learn.
Make sure you enable "threadsafe" in the compiler options, this will help you a little.

For HTTP requests like these I don't recommend using more than 10 threads, but feel free to experiment.

[DarkDragon]

You should also check whether the connection is still alive (So it won't wait for timeout anymore after the file was sent):
http://www.purebasic.fr/english/viewtop ... disconnect