Uncompress a zipped blob from a website returns -1

[RichAlgeni]

I have a zipped file as part on an installer. I've sent the zipped file to a hidden url on the internet. I do a GET on the file, and have it in memory. When I do an UncompressMemory() against it, with an output 100 times the size of the zip file, I get a -1 returned. I have used the Fingerprint() function against the blob, and is EXACTLY equal to the original file. Do I need to prepare the blob in some way for it to be uncompressed?

Here is the relevant program portion:

Code: Select all

internetHandle = InternetOpen_(#Null, #INTERNET_OPEN_TYPE_DIRECT, "", "", 0)
If internetHandle
    dispString = "InternetOpen_ returned: " + Str(internetHandle)
    PrintN(dispString)

    dispString = "Attempting to open webServer with path = /" + httpPath
    PrintN(dispString)

    sessionHandle = InternetConnect_(internetHandle, webServer, #INTERNET_DEFAULT_HTTPS_PORT, "", "", #INTERNET_SERVICE_HTTP, 0, 0)
    If sessionHandle
        dispString = "InternetConnect_ returned: " + Str(sessionHandle)
        PrintN(dispString)

; then open a handle to the get request

        requestHandle = HttpOpenRequest_(sessionHandle, "GET", httpPath, "", "", 0, #INTERNET_FLAG_SECURE, 0)
        If requestHandle
            dispString = "HttpOpenRequest_ returned: " + Str(requestHandle)
            PrintN(dispString)

; now send the get request

            requestResponse = HttpSendRequest_(requestHandle, "", 0, "", 0)
            If requestResponse
                dispString = "HttpSendRequest_ returned: " + Str(requestResponse)
                PrintN(dispString)

; now get the page itself

                totalRead = 0
                Repeat
                    returnValue  = InternetReadFile_(requestHandle, *receiveBuffer, readLength, @amountRead)
                    If amountRead > 0
                        CopyMemory(*receiveBuffer, @responseText + totalRead, amountRead)
                        totalRead + amountRead
                    Else
                        Break
                    EndIf
                Until (returnValue = 1 And amountRead = 0) Or returnValue = 0

                dispString = "Total bytes for blob read from connection: " + Str(totalRead)
                PrintN(dispString)

                sha256Hash = Fingerprint(@responseText, totalRead, #PB_Cipher_SHA2, 256)

                dispString = "sha256Hash of blob using 'Fingerprint' = " + sha256Hash
                PrintN(dispString)

                fileNumber = CreateFile(#PB_Any, fileName)
                If fileNumber > 0
                    result = WriteData(fileNumber, @responseText, totalRead)
                    CloseFile(fileNumber)

                    dispString = "Wrote " + Str(totalRead) + " to " + fileName
                    PrintN(dispString)

                    sha256Hash = FileFingerprint(fileName, #PB_Cipher_SHA2, 256)
                    dispString = "sha256Hash of file using 'FileFingerprint' = " + sha256Hash
                    PrintN(dispString)
                Else
                    dispString = "Unable to open: " + fileName
                    PrintN(dispString)
                EndIf

                result = UncompressMemory(@responseText, totalRead, *outputMemory, outputSize, #PB_PackerPlugin_Zip)

                dispString = "Result of memory uncompressed: " + Str(result) + ", size uncompressed: " + Str(outputSize)
                PrintN(dispString)
            Else
                dispString = "HttpSendRequest_() failed for '" + webServer + "/" + httpPath + "'"
                PrintN(dispString)
            EndIf
        Else
            dispString = "HttpOpenRequest_() failed for '" + webServer + "/" + httpPath + "'"
            PrintN(dispString)
        EndIf
    Else
        dispString = "InternetConnect_() failed for '" + webServer + "/" + httpPath + "'"
        PrintN(dispString)
    EndIf
Else
    dispString = "InternetOpen_() failed for '" + webServer + "/" + httpPath + "'"
    PrintN(dispString)
EndIf

Here is the output:

Code: Select all

InternetOpen_ returned: 13369348
Attempting to open webServer with path = /NortheastMountain/nemtn_sites.zip
InternetConnect_ returned: 13369352
HttpOpenRequest_ returned: 13369356
HttpSendRequest_ returned: 1
Total bytes for blob read from connection: 5982
sha256Hash of blob using 'Fingerprint' = 4bca11f068a957c5e02fbc9fba0167bb8cce87c4f2a93bba2b1ac067e24d8b31
Wrote 5982 to C:\Users\Rich Algeni\Downloads\nemtn_sites.zip
sha256Hash of file using 'FileFingerprint' = 4bca11f068a957c5e02fbc9fba0167bb8cce87c4f2a93bba2b1ac067e24d8b31
Result of memory uncompressed: -1, size uncompressed: 524287
Done

[boddhi]

I don't see *outputMemory allocation and outputSize assignation in your code...

[idle]

Also have you tried using HttpRequest?

Code: Select all

HttpRequest = HTTPRequest(#PB_HTTP_Get, "https://www.purebasic.com/download/PureBasic_Demo.zip", "", #PB_HTTP_Asynchronous)
  If HttpRequest
    Debug "StatusCode: " + HTTPInfo(HTTPRequest, #PB_HTTP_StatusCode)
    
    Repeat
      Progress = HTTPProgress(HttpRequest)
      Select Progress
        Case #PB_HTTP_Success
          Debug "Download finished"
          *Buffer = HTTPMemory(HTTPRequest)
          If *Buffer
            Debug "Memory buffer size: "+MemorySize(*buffer) ; Here the buffer can be written to disk or used directory
            FreeMemory(*Buffer)
          EndIf
          FinishHTTP(HTTPRequest) ; Always call FinishHTTP() when request is finished
          Break
          
        Case #PB_HTTP_Failed
          Debug "Download failed"
          FinishHTTP(HTTPRequest) ; Always call FinishHTTP() when request failed
          Break
          
        Case #PB_HTTP_Aborted
          Debug "Download aborted"
          FinishHTTP(HTTPRequest) ; Always call FinishHTTP() when request is aborted
          
        Default
          Debug "Current download: " + Progress ; The current download progress, in bytes
          Delay(100)
      EndSelect
    ForEver
  Else
    Debug "Request creation failed"
  EndIf

[RichAlgeni]

Here are the variable assignments. I always use EnableExplicit:

Code: Select all

Define sha256Hash.s
Define readLength.i = $FFFF
Define outputSize.i = $7FFFF
Define *receiveBuffer = AllocateMemory(readLength + 1)
Define responseText.s = Space(readLength + 1)
Define *outputMemory  = AllocateMemory(outputSize)

I didn't show them because it would have displayed sensitive information.

I will try that Idle.

[boddhi]

Another way if you download only ZIP files:

Code: Select all

EnableExplicit

Global.s ZipFileName

Procedure   Pc_DownloadFile(ArgURL.s)
  Protected *DownloadedFile
  Protected FileSize
  
  Print("File downloading: ")
  *DownloadedFile=ReceiveHTTPMemory(ArgURL)
  If *DownloadedFile
    PrintN("OK")
    ;
    Print("File format verification: ")
    If PeekL(*DownloadedFile)=$04034B50 ; Check with ZIP header to make sure it's a ZIP file. If not, it's an error 4xx, 5xx, etc
                                        ; For header, see https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.9.TXT
      PrintN("Good format")
      ;
      Print("Creation of file backup: ")
      FileSize=MemorySize(*DownloadedFile)
      ZipFileName=GetTemporaryDirectory()+"Temp-"+Str(Date())+"-PureBasic_Demo.zip"
      If CreateFile(0,ZipFileName)
        PrintN("OK")
        Print("Data writing: ")
        If WriteData(0,*DownloadedFile,FileSize)=FileSize
          PrintN("OK")
        Else
          PrintN("Error !!!!")
        EndIf
        CloseFile(0)
      Else
        PrintN("Error !!!!")
        ZipFileName="" ; This for delete or no the file after example code execution 
      EndIf
      ;
      ;
      ; Here, If needed, you can use UncompressedMemory() to process your data.
      ;
      ;
    Else
      PrintN("Bad format (not ZIP) !!!!")
    EndIf
    FreeMemory(*DownloadedFile)
  Else
    PrintN("Error !!!!")
  EndIf
EndProcedure

OpenConsole("Download ZIP")
;
PrintN("A file that that doesn't exist")
Pc_DownloadFile("HTTPS://www.purebasic.com/download/X_PureBasic_Demo.zip")
Print("Press any key")
Input()
PrintN("---------------------------------")
PrintN("And now a file that exists")
Pc_DownloadFile("HTTPS://www.purebasic.com/download/PureBasic_Demo.zip")
;
If ZipFileName
  PrintN("Start ZIP default tool...")
  RunProgram(ZipFileName)
EndIf
Print("Press any key to delete file and exit: ")
Input()
If ZipFileName
  DeleteFile(ZipFileName,#PB_FileSystem_Force)
EndIf
CloseConsole()

[RichAlgeni]

Ok, so I took an old text file, and zipped it up using 7z and online winzip. The files both sowed they were valid zip files, but both could not be uncompressed. Is there consensus that this is a bug? Or am I doing something wrong?

Code: Select all

EnableExplicit

UseZipPacker()
UseSHA2Fingerprint()

#ProgramName = "test_uncompress"

Define result.i
Define totalRead.i
Define dispString.s
Define fileNumber.i
Define sha256Hash.s
Define fileName.s      = "C:\Users\Rich Algeni\Downloads\test_json_online.zip"
Define readSize.i      = 100000
Define responseText.s  = Space(readSize)
Define outputSize.i    = readSize * 100
Define *outputMemory   = AllocateMemory(outputSize)
Define zipHeader.l     = $04034B50

sha256Hash = FileFingerprint(fileName, #PB_Cipher_SHA2, 256)
dispString = "sha256Hash of '" + fileName + "' using 'FileFingerprint' = " + sha256Hash
dispString + #LF$ + "filezize of '" + fileName + "' = " + Str(FileSize(fileName))

fileNumber = ReadFile(#PB_Any, fileName)
If fileNumber > 0
    totalRead = ReadData(fileNumber, @responseText, readSize)
    CloseFile(fileNumber)

    dispString + #LF$ + "Read " + Str(totalRead) + " bytes from " + fileName
; should be $04034B50 per boddhi 2024/07/14
    If PeekL(@responseText) = zipHeader
        dispString + #LF$ + "PeekL(@responseText) = $04034B50, file is a valid zip file"
    Else
        dispString + #LF$ + "PeekL(@responseText) <> $04034B50, file is NOT a valid zip file"
        MessageRequester(#ProgramName, dispString)
        End
    EndIf

    sha256Hash = Fingerprint(@responseText, totalRead, #PB_Cipher_SHA2, 256)
    dispString + #LF$ + "sha256Hash of memory area using 'Fingerprint' = " + sha256Hash

    result = UncompressMemory(@responseText, totalRead, *outputMemory, outputSize, #PB_PackerPlugin_Zip)

    dispString + #LF$ + "Result of memory uncompressed: " + Str(result)
    If result > 0
        dispString + #LF$ + "Size uncompressed: " + Str(outputSize)
    EndIf
Else
    dispString = "Unable to open: " + fileName
EndIf

MessageRequester(#ProgramName, dispString)

End

[RichAlgeni]

I believe that I am trying to use a function, UncompressMemory(), in a way which it was not designed: to unpack a packed file with directories and files. I'd rather have not had to write the packed file to disk, then read it again as a packed file, but I believe that is what is required. Maybe in the future, there could be a RecognizePackedMemory() function, that precludes us from having to write to disk, and reread as a packed file. It would return a Pack Number that could be used with the other Pack functions.

[RichAlgeni]

Ok, so I'm an idiot! Fred, Freak and the team have already given us CatchPack()!

In the immortal words of Roseanne Roseannadanna..., nevermind!!!

[RichAlgeni]

CatchPack() works perfectly!!! Forgive me! My mind has been mush since I lost my wife.