RunProgram(): Weird ReadProgramString() behaviour?

[ergrull0]

Hello everybody!
I'm experimenting a bit with the RunProgram() procedure and I noticed a strange behaviour with the "curl" executable. This is the code:

Code: Select all

Process = RunProgram("curl", "http://purebasic.com/documentation/PureBasicSmall.pdf --stderr - -o PureBasicSmall.pdf", 
                     GetTemporaryDirectory(), 
                     #PB_Program_Open | #PB_Program_Read | #PB_Program_Error)
If Process
  While ProgramRunning(Process)
    If AvailableProgramOutput(Process)
      Debug ReadProgramString(Process)
    EndIf 
  Wend
  CloseProgram(Process)
EndIf

This is the only debug output I get while the program is running:

Code: Select all

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

But once the program finished it outputs all together the output of the download progress of curl instead of outputting it during the download i.e.

Code: Select all

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  2 3276k    2 90471    0     0   179k      0  0:00:18 --:--:--  0:00:18  232k
 25 3276k   25  837k    0     0   561k      0  0:00:05  0:00:01  0:00:04  607k
 48 3276k   48 1577k    0     0   631k      0  0:00:05  0:00:02  0:00:03  661k
 69 3276k   69 2261k    0     0   647k      0  0:00:05  0:00:03  0:00:02  668k
 89 3276k   89 2945k    0     0   655k      0  0:00:04  0:00:04 --:--:--  672k
100 3276k  100 3276k    0     0   662k      0  0:00:04  0:00:04 --:--:--  715k

Any idea about how to prevent this to happen and get the output in "real time"?

Thanks!!

[IdeasVacuum]

Try putting a short delay in the While loop: Delay(10)

See also: http://www.purebasic.fr/english/viewtop ... =3&t=56382

[ergrull0]

Try putting a short delay in the While loop: Delay(10)

See also: http://www.purebasic.fr/english/viewtop ... =3&t=56382

Nope. It didn't work. But I found what the problem was: while downloading the curl displays a string that ends with a CR only instead of a LF. I guess that ReadProgramString() expects a LF as End Of Line character.
So I came up with this workaround to make it work:

Code: Select all

*Buffer = AllocateMemory(SizeOf(Double))

Dim Result$(0)

Process = RunProgram("curl", "http://speedtest.webair.com/5mb.bin --stderr - -o 5mb.bin", 
                     GetTemporaryDirectory(), 
                     #PB_Program_Open | #PB_Program_Read | #PB_Program_Error)
If Process
  While ProgramRunning(Process)
    If AvailableProgramOutput(Process)
       NumOfBytes = ReadProgramData(Process, *Buffer, SizeOf(*Buffer)) ;                    ; Saves the program output to a buffer
        OutputString$ + PeekS(*Buffer, NumOfBytes, #PB_UTF8)                                ; concatenates the buffered data into a string
        If FindString(OutputString$, #CR$) Or FindString(OutputString$, #LF$)               ; until it finds a CR or LF character in it
          Debug StringField(OutputString$, 1, #CR$)                                         ; splits the string before the CR and displays the first segment
          OutputString$ = StringField(OutputString$, 2, #CR$)                               ; the remaining characters will be used to start the new string
        EndIf
    EndIf 
  Wend
  CloseProgram(Process)
EndIf

[Shardik]

I want to present another solution which uses libcurl.dylib and imports the required functions. The libcurl.dylib doesn't need to be installed because it is contained in /usr/lib.

Code: Select all

EnableExplicit

#ServerFile = "http://purebasic.com/documentation/PureBasicSmall.pdf"

#CURL_ERROR_SIZE          =   256
#CURLOPT_ERRORBUFFER      = 10010
#CURLOPT_NOPROGRESS       =    43
#CURLOPT_PROGRESSFUNCTION = 20056
#CURLOPT_URL              = 10002
#CURLOPT_WRITEFUNCTION    = 20011

ImportC "/usr/lib/libcurl.dylib"
  curl_easy_cleanup(*CurlSession)
  curl_easy_init()
  curl_easy_setopt(*CurlSession, Option.I, Parameter.I)
  curl_easy_strerror(ErrorNumber.I)
  curl_easy_perform(*CurlSession) 
EndImport

Define ClientFile.S = GetTemporaryDirectory() + "PureBasicSmall.pdf"
Define i.I
Define ReceivedData.S
Define ServerFile.S

ProcedureC DataReceivedCallback(*Data, Size.I, NMemb.I, *UserData)
  WriteData(0, *Data, Size * NMemb)

  ProcedureReturn Size * NMemb
EndProcedure

ProcedureC ProgressCallback(*ProgressData, DownloadTotal.D, DownloadNow.D,
  UploadTotal.D, UploadNow.D)
  Debug StrD(DownloadNow) + " Bytes from " + StrD(DownloadTotal) +
    " Bytes have been downloaded"
EndProcedure

Procedure SetCurlOption(CurlSession.I, Option.I, Parameter.I)
  CompilerSelect #PB_Compiler_Processor
    CompilerCase #PB_Processor_x86
      curl_easy_setopt(CurlSession, Option, Parameter)
    CompilerCase #PB_Processor_x64
      !EXTERN _curl_easy_setopt
      !MOV    RDI,QWORD [p.v_CurlSession]
      !MOV    RSI,QWORD [p.v_Option]
      !MOV    RDX,QWORD [p.v_Parameter]
      !CALL   _curl_easy_setopt
  CompilerEndSelect
EndProcedure

Procedure DownloadFile(URL.S)
  Shared ReceivedData.S
  
  Protected CurlSession.I
  Protected ErrorCode.I
  Protected ErrorMsg.S = Space(#CURL_ERROR_SIZE)
  Protected *ErrorMsgBuffer = AllocateMemory(#CURL_ERROR_SIZE)
  Protected Result.I = #False
  
  CurlSession = curl_easy_init()
  
  If CurlSession
    SetCurlOption(CurlSession, #CURLOPT_URL, @URL)
    SetCurlOption(CurlSession, #CURLOPT_WRITEFUNCTION, @DataReceivedCallback())
    SetCurlOption(CurlSession, #CURLOPT_ERRORBUFFER, *ErrorMsgBuffer)
    SetCurlOption(CurlSession, #CURLOPT_PROGRESSFUNCTION, @ProgressCallback())
    SetCurlOption(CurlSession, #CURLOPT_NOPROGRESS, #False)
    
    ErrorCode = curl_easy_perform(CurlSession)
    
    If ErrorCode = 0
      Result = #True
    Else
      CompilerIf #PB_Compiler_Unicode
        ErrorMsg = Space(StringByteLength(PeekS(*ErrorMsgBuffer, -1,
          #PB_Ascii), #PB_Unicode))
        PokeS(@ErrorMsg, PeekS(*ErrorMsgBuffer, -1, #PB_Ascii), -1, #PB_Unicode)
      CompilerElse
        ErrorMsg = PeekS(*ErrorMsgBuffer)
      CompilerEndIf

      If Trim(ErrorMsg) = ""
        CompilerIf #PB_Compiler_Unicode
          ErrorMsg = Space(StringByteLength(PeekS(curl_easy_strerror(ErrorCode),
            -1, #PB_Ascii), #PB_Unicode))
          PokeS(@ErrorMsg, PeekS(curl_easy_strerror(ErrorCode), -1, #PB_Ascii), -1,
            #PB_Unicode)
        CompilerElse
          ErrorMsg = PeekS(curl_easy_strerror(ErrorCode))
        CompilerEndIf
      EndIf

      MessageRequester("Error", ErrorMsg)
    EndIf
    
    FreeMemory(*ErrorMsgBuffer)
    curl_easy_cleanup(CurlSession)
  EndIf

  ProcedureReturn Result
EndProcedure

If OpenFile(0, ClientFile)
  If #PB_Compiler_Unicode
    ServerFile = Space(StringByteLength(#ServerFile))
    PokeS(@ServerFile, #ServerFile, -1, #PB_Ascii)
  Else
    ServerFile = #ServerFile
  EndIf

  If DownloadFile(ServerFile)
    MessageRequester("HTTP-Download", "The file" + #CR$ + #CR$ + #ServerFile + #CR$ + #CR$ + "was downloaded successfully and stored as" + #CR$ + #CR$ + ClientFile)
  EndIf

  CloseFile(0)
EndIf

Update 1: I have modified the code to work in both ASCII and Unicode mode.
Update 2: I have added the new procedure SetCurlOption() which uses x64 assembly instructions (when compiled on a x64 processor) to correctly move the arguments into the correct registers and call curl_easy_setopt(). I have verified this code to successfully run on MacOS X Snow Leopard and Mavericks in x86 and x64 mode and in both ASCII and Unicode mode.

[ergrull0]

I want to present another solution which uses libcurl.dylib and imports the required functions. Currently it only works with the PB x86 compiler and with Unicode mode disabled (the program recognizes these conditions and displays an info message). I have tested the program with OS X 10.6.8 (Snow Leopard). The libcurl.dylib doesn't need to be installed because it is contained in /usr/lib.

Code: Select all

...

Great piece of code Shardik! Unfortunately I can't test it right now because I have just the x64 version of PureBasic installed. Can you explain why your code works only on x86 platform and without unicode? Is it a limitation of the libcurl library itself?

Thanks!

[Shardik]

Can you explain why your code works only on x86 platform and without unicode? Is it a limitation of the libcurl library itself?

To implement Unicode I was just too lazy...
I have therefore modified the above code to work in both ASCII and Unicode mode.

The culprit for only working when compiled to x86 code is the PB compiler. PB doesn't support variable argument functions which libcurl requires. For a solution you have to program a customized C wrapper. You may take a look into this thread:

You can't call such variable argument functions in PB directly, you will need a wrapper.

PB doesn't support this notation, so args are arbitrary pushed on the stacks on x86 and using regs on x64, but the interpretation depends of the type. Better write a small wrapper in C to handle that.

Fred's last statement would mean that even the x86 version is working only by luck, but I never experienced any problems with my x86 programs using libcurl on Windows XP SP3, Windows Server 2000, Windows 7 SP1, different Linux distributions and MacOS X Snow Leopard...

It's not a problem of the libcurl version contained in MacOS X which is compiled as a universal binary for x64, x86 and PPC. You may proof this with the following terminal command:

Code: Select all

file /usr/lib/libcurl.dylib

[ergrull0]

[...]The culprit for only working when compiled to x86 code is the PB compiler. PB doesn't support variable argument functions which libcurl requires[...]

So, just to clarify my mind, the problem is that the curl_easy_setopt() function requires different types of parameters (a function pointer, a long etc...) based on the CURLOPT_[...] option, and this kind of "variability" is not supported by PB. Am I right?

[Shardik]

So, just to clarify my mind, the problem is that the curl_easy_setopt() function requires different types of parameters (a function pointer, a long etc...) based on the CURLOPT_[...] option, and this kind of "variability" is not supported by PB. Am I right?

Yes, you are right!

[Shardik]

I have modified my example code from above to also work with PB's x64 compiler. I have added the new procedure SetCurlOption() which uses x64 assembly instructions (when compiled on a x64 processor) to correctly move the arguments into the correct registers and call curl_easy_setopt(). I have verified this code to successfully run on MacOS X Snow Leopard and Mavericks in x86 and x64 mode and in both ASCII and Unicode mode.