Page 1 of 1
Packer - error with files >4GB size
Posted: Sun Aug 11, 2024 7:08 pm
by nalor
Hi!
(UPDATE: changed the title as it better shows the findings)
I just want to clarify if I'm doing something wrong of if really all plugins are limited to 2GB source file size.
I tried it with:
And none of them is able to compress files >2GB.
The ZIP plugin mentions a max archive size of 2GB, but I think this is not related here.
To me it looks as if the source-file-size must be smaller than 2GB.
Is this correct? Is there an option to compress larger files?
Re: Packer - 2GB size limit for all plugins
Posted: Sun Aug 11, 2024 8:08 pm
by User_Russian
I do not confirm.
This is the test code (PB 6.11 x64 windows)
Code: Select all
UseZipPacker()
; Create the packed file
If CreatePack(0, "mycompressedfiles.zip")
; add your files
Debug AddPackFile(0, "File.iso", "File.iso")
ClosePack(0)
EndIf
successfully compressed file size 3.5 GB.
I checked the created archive using "7-Zip File Manager" and it was correct (no errors).
Did you use the x64 version of PB?
Does your computer have enough free RAM (2 times more than the size of the compressed file)?
Re: Packer - 2GB size limit for all plugins
Posted: Sat Aug 17, 2024 10:35 pm
by nalor
Here's a small example which shows my problem:
- Creates a 15GB file
- creates a new pack
- adds the 15gb file
- closes the pack and reopens it (else I get an error during execution)
- examines the pack and the added file is only 3gb in size?
Please adjust the filenames before executing.
The result with a 3GB file inside the zip file is reproducible in both PB 6.11 x86 and x64.
When I reduce the source file size to e.g. 13GB, the packed file is only 1GB.
So I agree it's not a "2GB" limit - but at least I have no explanation.
Code: Select all
EnableExplicit
UseZipPacker()
CompilerIf Defined(FormatMessage, #PB_Procedure) = #False
Procedure.s FormatMessage(ErrorNumber.l)
Protected *Buffer, len, result.s
len = FormatMessage_(#FORMAT_MESSAGE_ALLOCATE_BUFFER|#FORMAT_MESSAGE_FROM_SYSTEM,0,ErrorNumber,0,@*Buffer,0,0)
If len
result = PeekS(*Buffer, len - 2)
LocalFree_(*Buffer)
ProcedureReturn result
Else
ProcedureReturn "Errorcode: " + Hex(ErrorNumber)
EndIf
EndProcedure
CompilerEndIf
Procedure CreateFileSize(sFilename.s, qFilesize.q)
; Purpose: instantly create a file with a given size (file has random content)
; 20161009 .. nalor .. created
Protected iFileHdl.i=0
Protected lLoSize.l
Protected lHiSize.l
Protected iReturn.i
Protected iLastError.i
Protected iTmp.i
Repeat ; single repeat
iFileHdl = CreateFile_(sFilename, #GENERIC_WRITE, 0, 0, #CREATE_ALWAYS, 0, 0)
iLastError=GetLastError_()
If iFileHdl=#INVALID_HANDLE_VALUE
Debug "ERROR! CreateFile_ failed ("+FormatMessage(iLastError)+")"
iReturn=-1
Break
EndIf
lLoSize=qFilesize & $FFFFFFFF
lHiSize=(qFilesize>>32) & $FFFFFFFF
iTmp=SetFilePointer_(iFileHdl, lLoSize, @lHiSize, #FILE_BEGIN)
iLastError=GetLastError_()
If iTmp=#INVALID_SET_FILE_POINTER And iLastError<>#NO_ERROR
Debug "ERROR! SetFilePointer_ failed ("+FormatMessage(iLastError)+")"
iReturn=-1
Break
EndIf
iTmp=SetEndOfFile_(iFileHdl)
iLastError=GetLastError_()
If iTmp=0
Debug "ERROR! SetEndOfFile_ failed ("+FormatMessage(iLastError)+")"
iReturn=-2
Break
EndIf
iReturn=#True
Until #True
If iFileHdl<>0
iTmp=CloseHandle_(iFileHdl)
iLastError=GetLastError_()
If iTmp=0
Debug "ERROR! CloseHandle_ failed ("+FormatMessage(iLastError)+")"
iReturn=-3
EndIf
EndIf
If iReturn<0
DeleteFile(sFilename)
EndIf
ProcedureReturn iReturn
EndProcedure
Procedure.s CMSTR_FormatFileSizeAuto(qSizeByte.q)
; 20160807 ... nalor ... created
Protected fValue.f=qSizeByte
Protected sUnit.s=" B"
If qSizeByte>=1024
fValue=qSizeByte/1024
sUnit="KB"
Else
ProcedureReturn Str(fValue)+sUnit
EndIf
If fValue>=1024
fValue=fValue/1024
sUnit="MB"
EndIf
If fValue>=1024
fValue=fValue/1024
sUnit="GB"
EndIf
If fValue>=1024
fValue=fValue/1024
sUnit="TB"
EndIf
ProcedureReturn StrF(fValue, 2)+sUnit
EndProcedure
Define iFileHdl.i
Define sBigfile.s="c:\tmp\bigfile.test"
Define qBigFileSize.q=1024 * 1024 * 1024 * 15 ; 15 GB
Define iPkgHdl.i
Define sDstFile.s="c:\tmp\bigfile.zip"
; create huge test-file
Debug "#### Create big file with size >"+CMSTR_FormatFileSizeAuto(qBigFileSize)+"<"
If Not CreateFileSize(sBigfile, qBigFileSize)
Debug "File NOT Created"
End
EndIf
Debug "---- file created"
Debug "#### Create pack"
iPkgHdl=CreatePack(#PB_Any, sDstFile, #PB_PackerPlugin_Zip)
If Not iPkgHdl
Debug "error create pack"
End
EndIf
Debug "---- pack created"
Debug "#### add big file to pack"
If Not AddPackFile(iPkgHdl, sBigfile, GetFilePart(sBigfile))
Debug "error add file"
End
EndIf
Debug "---- file added"
Debug "#### close pack"
ClosePack(iPkgHdl)
Debug "---- pack closed"
Debug "#### (re)open pack"
iPkgHdl=OpenPack(#PB_Any, sDstFile, #PB_PackerPlugin_Zip)
If Not iPkgHdl
Debug "error open pack"
End
EndIf
Debug "---- pack opened"
Debug "#### list all the entries"
If ExaminePack(iPkgHdl)
While NextPackEntry(iPkgHdl)
Debug "Name: " + PackEntryName(iPkgHdl) + ", Size >" + CMSTR_FormatFileSizeAuto(PackEntrySize(iPkgHdl))+"<"
Wend
EndIf
Debug "---- all entries listed"
Debug "#### close pack"
ClosePack(iPkgHdl)
Debug "---- pack closed"
Re: Packer - 2GB size limit for all plugins
Posted: Sat Aug 17, 2024 10:43 pm
by nalor
Maybe it's a 4GB problem
- 1GB file -> 1GB packed file
- 2GB file -> 2GB packed file
- 3GB file -> 3GB packed file
- 4GB file -> 0 byte packed file ??
- 5GB file -> 1GB packed file
- 6GB file -> 2GB packed file
- 7GB file -> 3GB packed file
- 8GB file -> 0 byte packed file ????
And when I create a 4GB-1Byte file and add it to the pack, the filesize is still correct ... so beginning with "4GB" size the problems start.
Re: Packer - 2GB size limit for all plugins
Posted: Mon Aug 19, 2024 2:29 pm
by miso
Just a stab in the dark (can't test the code right now), but maybe the disk format system causes the problem. Isn't it FAT32, maybe?
Edit:(No, I tested it [6.11 LTS beta 2 win7 64bit], same behavior, you described. )
Re: Packer - error with files >4GB size
Posted: Sun Aug 25, 2024 10:41 pm
by nalor