Packer - error with files >4GB size

[nalor]

Hi!
(UPDATE: changed the title as it better shows the findings)

I just want to clarify if I'm doing something wrong of if really all plugins are limited to 2GB source file size.

I tried it with:

• BriefLZ
• TAR
• LZMA
• ZIP

And none of them is able to compress files >2GB.

The ZIP plugin mentions a max archive size of 2GB, but I think this is not related here.

To me it looks as if the source-file-size must be smaller than 2GB.

Is this correct? Is there an option to compress larger files?

[User_Russian]

I do not confirm.
This is the test code (PB 6.11 x64 windows)

Code: Select all

UseZipPacker()

; Create the packed file
If CreatePack(0, "mycompressedfiles.zip") 
  ; add your files 
  Debug AddPackFile(0, "File.iso", "File.iso")
  ClosePack(0) 
EndIf

successfully compressed file size 3.5 GB.
I checked the created archive using "7-Zip File Manager" and it was correct (no errors).

Did you use the x64 version of PB?
Does your computer have enough free RAM (2 times more than the size of the compressed file)?

[nalor]

Here's a small example which shows my problem:

• Creates a 15GB file
• creates a new pack
• adds the 15gb file
• closes the pack and reopens it (else I get an error during execution)
• examines the pack and the added file is only 3gb in size?

Please adjust the filenames before executing.

The result with a 3GB file inside the zip file is reproducible in both PB 6.11 x86 and x64.
When I reduce the source file size to e.g. 13GB, the packed file is only 1GB.

So I agree it's not a "2GB" limit - but at least I have no explanation.

Code: Select all

EnableExplicit

UseZipPacker()

CompilerIf Defined(FormatMessage, #PB_Procedure) = #False

  Procedure.s FormatMessage(ErrorNumber.l)

    Protected *Buffer, len, result.s
   
    len = FormatMessage_(#FORMAT_MESSAGE_ALLOCATE_BUFFER|#FORMAT_MESSAGE_FROM_SYSTEM,0,ErrorNumber,0,@*Buffer,0,0)
    If len
      result = PeekS(*Buffer, len - 2)
      LocalFree_(*Buffer)
      ProcedureReturn result
    Else
      ProcedureReturn "Errorcode: " + Hex(ErrorNumber)
    EndIf
   
  EndProcedure

CompilerEndIf

Procedure CreateFileSize(sFilename.s, qFilesize.q)
	; Purpose: instantly create a file with a given size (file has random content)
	
	; 20161009 .. nalor .. created
	
	Protected iFileHdl.i=0
	Protected lLoSize.l
	Protected lHiSize.l
	Protected iReturn.i
	Protected iLastError.i
	Protected iTmp.i
	
	Repeat ; single repeat
		iFileHdl = CreateFile_(sFilename, #GENERIC_WRITE, 0, 0, #CREATE_ALWAYS, 0, 0)
		iLastError=GetLastError_()
		If iFileHdl=#INVALID_HANDLE_VALUE
			Debug "ERROR! CreateFile_ failed ("+FormatMessage(iLastError)+")"
			iReturn=-1
			Break
		EndIf
		
		lLoSize=qFilesize & $FFFFFFFF
		lHiSize=(qFilesize>>32) & $FFFFFFFF
		
		iTmp=SetFilePointer_(iFileHdl, lLoSize, @lHiSize, #FILE_BEGIN)
		iLastError=GetLastError_()
		If iTmp=#INVALID_SET_FILE_POINTER And iLastError<>#NO_ERROR
			Debug "ERROR! SetFilePointer_ failed ("+FormatMessage(iLastError)+")"
			iReturn=-1
			Break
		EndIf
		
		iTmp=SetEndOfFile_(iFileHdl)
		iLastError=GetLastError_()
		If iTmp=0
			Debug "ERROR! SetEndOfFile_ failed ("+FormatMessage(iLastError)+")"
			iReturn=-2
			Break
		EndIf
		
		iReturn=#True
		
	Until #True
	
	If iFileHdl<>0
		iTmp=CloseHandle_(iFileHdl)
		iLastError=GetLastError_()
		If iTmp=0
			Debug "ERROR! CloseHandle_ failed ("+FormatMessage(iLastError)+")"
			iReturn=-3
		EndIf
	EndIf
	
	If iReturn<0
		DeleteFile(sFilename)
	EndIf
	
	ProcedureReturn iReturn
	
EndProcedure

Procedure.s CMSTR_FormatFileSizeAuto(qSizeByte.q)
	
	; 20160807 ... nalor ... created
	
	Protected fValue.f=qSizeByte
	Protected sUnit.s=" B"
	
	If qSizeByte>=1024
		fValue=qSizeByte/1024
		sUnit="KB"
	Else
		ProcedureReturn Str(fValue)+sUnit
	EndIf
	
	If fValue>=1024
		fValue=fValue/1024
		sUnit="MB"
	EndIf
	
	If fValue>=1024
		fValue=fValue/1024
		sUnit="GB"
	EndIf
	
	If fValue>=1024
		fValue=fValue/1024
		sUnit="TB"
	EndIf
	
	ProcedureReturn StrF(fValue, 2)+sUnit	
	
EndProcedure

Define iFileHdl.i
Define sBigfile.s="c:\tmp\bigfile.test"
Define qBigFileSize.q=1024 * 1024 * 1024 * 15 ; 15 GB

Define iPkgHdl.i
Define sDstFile.s="c:\tmp\bigfile.zip"


; create huge test-file


Debug "#### Create big file with size >"+CMSTR_FormatFileSizeAuto(qBigFileSize)+"<"

If Not CreateFileSize(sBigfile, qBigFileSize)
	Debug "File NOT Created"
	End
EndIf

Debug "---- file created"

Debug "#### Create pack"

iPkgHdl=CreatePack(#PB_Any, sDstFile, #PB_PackerPlugin_Zip)
If Not iPkgHdl
	Debug "error create pack"
	End
EndIf

Debug "---- pack created"

Debug "#### add big file to pack"

If Not AddPackFile(iPkgHdl, sBigfile, GetFilePart(sBigfile))
	Debug "error add file"
	End
	
EndIf

Debug "---- file added"

Debug "#### close pack"

ClosePack(iPkgHdl)

Debug "---- pack closed"


Debug "#### (re)open pack"

iPkgHdl=OpenPack(#PB_Any, sDstFile, #PB_PackerPlugin_Zip)
If Not iPkgHdl
	Debug "error open pack"
	End
EndIf

Debug "---- pack opened"


Debug "#### list all the entries"
If ExaminePack(iPkgHdl)
	While NextPackEntry(iPkgHdl)
		Debug "Name: " + PackEntryName(iPkgHdl) + ", Size >" + CMSTR_FormatFileSizeAuto(PackEntrySize(iPkgHdl))+"<"
	Wend
EndIf


Debug "---- all entries listed"

Debug "#### close pack"

ClosePack(iPkgHdl)

Debug "---- pack closed"

[nalor]

Maybe it's a 4GB problem

• 1GB file -> 1GB packed file
• 2GB file -> 2GB packed file
• 3GB file -> 3GB packed file
• 4GB file -> 0 byte packed file ??
• 5GB file -> 1GB packed file
• 6GB file -> 2GB packed file
• 7GB file -> 3GB packed file
• 8GB file -> 0 byte packed file ????

And when I create a 4GB-1Byte file and add it to the pack, the filesize is still correct ... so beginning with "4GB" size the problems start.

[miso]

Just a stab in the dark (can't test the code right now), but maybe the disk format system causes the problem. Isn't it FAT32, maybe?

Edit:(No, I tested it [6.11 LTS beta 2 win7 64bit], same behavior, you described. )

[nalor]

Created a bug report for the problem:

https://www.purebasic.fr/english/viewtopic.php?t=85194