Write block of data to file - fastest way?

Just starting out? Need help? Post your questions and find answers here.
oO0XX0Oo
User
User
Posts: 78
Joined: Thu Aug 10, 2017 7:35 am

Write block of data to file - fastest way?

Post by oO0XX0Oo »

Hi,

I need to create a lot of files with individual file sizes. I'm running over
a list with an underlying structure that contains the size of the original file.

Code: Select all

          itemToCreate = dstFolder + ReplaceString(ScanDirInfos()\Item, srcFolder, "", #PB_String_NoCase)
          If CreateFile(0, itemToCreate, #PB_File_SharedRead|#PB_File_SharedWrite|#PB_File_NoBuffering|#PB_Ascii)
            For c = 1 To ScanDirInfos()\Size
              WriteString(0, Chr(32), #PB_Ascii)
            Next
            CloseFile(0)
          EndIf
This part works by filling a file with the necessary amount of spaces. But ofc this is a slow process...

It doesn't really matter what is written into the file (it doesn't need to be spaces).

Is there any efficient way (without allocating GBs of memory if a file has a size in that range) to do it quicker?
User avatar
Sicro
Enthusiast
Enthusiast
Posts: 538
Joined: Wed Jun 25, 2014 5:25 pm
Location: Germany
Contact:

Re: Write block of data to file - fastest way?

Post by Sicro »

Code: Select all

#Kilobytes = 1024
#Megabytes = #Kilobytes * 1024
#Gigabytes = #Megabytes * 1024

#MaxUseOfRAM = #Megabytes ; Set here the maximum size limit of RAM consumption

itemToCreate = dstFolder + ReplaceString(ScanDirInfos()\Item, srcFolder, "", #PB_String_NoCase)
If CreateFile(0, itemToCreate, #PB_File_SharedWrite|#PB_File_NoBuffering)
  
  Define RemainingLengthOfData = ScanDirInfos()\Size
  Define *Memory
  
  If RemainingLengthOfData > #MaxUseOfRAM
    
    *Memory = AllocateMemory(#MaxUseOfRAM, #PB_Memory_NoClear)
    If *Memory
      While RemainingLengthOfData >= #MaxUseOfRAM
        WriteData(0, *Memory, MemorySize(*Memory))
        RemainingLengthOfData - MemorySize(*Memory)
      Wend
      FreeMemory(*Memory)
    EndIf
    
  EndIf
  
  If RemainingLengthOfData > 0
    
    *Memory = AllocateMemory(RemainingLengthOfData, #PB_Memory_NoClear)
    If *Memory
      WriteData(0, *Memory, MemorySize(*Memory))
      FreeMemory(*Memory)
    EndIf
    
  EndIf
  
  CloseFile(0)
  
EndIf
Image
Why OpenSource should have a license :: PB-CodeArchiv-Rebirth :: Pleasant-Dark (syntax color scheme) :: RegEx-Engine (compiles RegExes to NFA/DFA)
Manjaro Xfce x64 (Main system) :: Windows 10 Home (VirtualBox) :: Newest PureBasic version
oO0XX0Oo
User
User
Posts: 78
Joined: Thu Aug 10, 2017 7:35 am

Re: Write block of data to file - fastest way?

Post by oO0XX0Oo »

Thanks a lot Sicro!

On a limited set of files where creating is done in RAM disk I have an acceleration
factor of about 30x with your code... :mrgreen:
User avatar
Bisonte
Addict
Addict
Posts: 1226
Joined: Tue Oct 09, 2007 2:15 am

Re: Write block of data to file - fastest way?

Post by Bisonte »

But you have to be careful! If there are strings in the structure, only the addresses of the strings are stored, not the strings themselves. To do this, you should use fixed strings in the structure.

Code: Select all

Structure mystruct
  Size.i
  String.s
  FixString.s{20}
EndStructure
PureBasic 6.04 LTS (Windows x86/x64) | Windows10 Pro x64 | Asus TUF X570 Gaming Plus | R9 5900X | 64GB RAM | GeForce RTX 3080 TI iChill X4 | HAF XF Evo | build by vannicom​​
English is not my native language... (I often use DeepL to translate my texts.)
oO0XX0Oo
User
User
Posts: 78
Joined: Thu Aug 10, 2017 7:35 am

Re: Write block of data to file - fastest way?

Post by oO0XX0Oo »

This is the structure that I'm using:

Code: Select all

Structure _ScanDirInfo
  Item.s                        ; Full file / folder name
  Type.b                        ; 1 = file, 2 = folder
  Size.q                        ; Size of file (not used for folders)
  DateC.i                       ; Creation date
  DateM.i                       ; Last modified date
  DateA.i                       ; Last accessed date
  Attributes.i                  ; Attributes
EndStructure
Doesn't look like that there is something I should be aware of, does it?
User avatar
Sicro
Enthusiast
Enthusiast
Posts: 538
Joined: Wed Jun 25, 2014 5:25 pm
Location: Germany
Contact:

Re: Write block of data to file - fastest way?

Post by Sicro »

@Bisonte:
As I understand him, it isn't a data structure, but a directory structure with files.
He wants to copy this structure without the real file contents (only the file size should be retained).
Image
Why OpenSource should have a license :: PB-CodeArchiv-Rebirth :: Pleasant-Dark (syntax color scheme) :: RegEx-Engine (compiles RegExes to NFA/DFA)
Manjaro Xfce x64 (Main system) :: Windows 10 Home (VirtualBox) :: Newest PureBasic version
User avatar
Bisonte
Addict
Addict
Posts: 1226
Joined: Tue Oct 09, 2007 2:15 am

Re: Write block of data to file - fastest way?

Post by Bisonte »

oO0XX0Oo wrote:This is the structure that I'm using:

Code: Select all

Structure _ScanDirInfo
  Item.s                        ; Full file / folder name
  Type.b                        ; 1 = file, 2 = folder
  Size.q                        ; Size of file (not used for folders)
  DateC.i                       ; Creation date
  DateM.i                       ; Last modified date
  DateA.i                       ; Last accessed date
  Attributes.i                  ; Attributes
EndStructure
Doesn't look like that there is something I should be aware of, does it?
If you want to save the "Item.s", you have to use fixedStrings like "Item.s{#MAX_PATH}" (on Windows). Otherwise
you save only a integer value (the adress of the string). You can make a little test.

Code: Select all

Debug SizeOf(_ScanDirInfo)
These are the bytes, that can be stored in this structure.... and this is also the size you save on disk...
PureBasic 6.04 LTS (Windows x86/x64) | Windows10 Pro x64 | Asus TUF X570 Gaming Plus | R9 5900X | 64GB RAM | GeForce RTX 3080 TI iChill X4 | HAF XF Evo | build by vannicom​​
English is not my native language... (I often use DeepL to translate my texts.)
oO0XX0Oo
User
User
Posts: 78
Joined: Thu Aug 10, 2017 7:35 am

Re: Write block of data to file - fastest way?

Post by oO0XX0Oo »

No Bisonte, I'm not writing the structure to disk. It just contains
the information that the recursive scan dir procedure uses for each
found file / directory while putting them into a list
wilbert
PureBasic Expert
PureBasic Expert
Posts: 3870
Joined: Sun Aug 08, 2004 5:21 am
Location: Netherlands

Re: Write block of data to file - fastest way?

Post by wilbert »

oO0XX0Oo wrote:Is there any efficient way (without allocating GBs of memory if a file has a size in that range) to do it quicker?
I found this approach online.
Not sure if it works on all Windows versions. :?

Code: Select all

CompilerIf #PB_Compiler_OS = #PB_OS_Windows
  
  ImportC ""
    fopen(filename.p-utf8, mode.p-ascii)
    ftruncate(fd, size) As "_chsize"
    fileno(fp) As "_fileno"
    fclose(fp)  
  EndImport
  
CompilerElse
  
  ImportC ""
    fopen(filename.p-utf8, mode.p-ascii)
    ftruncate(fd, size)
    fileno(fp)
    fclose(fp)  
  EndImport
  
CompilerEndIf



; Create 64 MB file named myfile.dat

fp=fopen("myfile.dat", "w");
ftruncate(fileno(fp), 64*1024*1024)
fclose(fp)
Windows (x64)
Raspberry Pi OS (Arm64)
oO0XX0Oo
User
User
Posts: 78
Joined: Thu Aug 10, 2017 7:35 am

Re: Write block of data to file - fastest way?

Post by oO0XX0Oo »

@wilbert

Trying to compile it leads to:

Code: Select all

PureBasic - Linker error
---------------------------
POLINK: error: Unresolved external symbol '_fileno'.
POLINK: error: Unresolved external symbol '_chsize'.
POLINK: fatal error: 2 unresolved external(s).
Windows Server 2012 R2 U3 x64
wilbert
PureBasic Expert
PureBasic Expert
Posts: 3870
Joined: Sun Aug 08, 2004 5:21 am
Location: Netherlands

Re: Write block of data to file - fastest way?

Post by wilbert »

When I try the 32 bit version of PureBasic, I get the same unresolved symbol errors.
When I try the 64 bit version of PureBasic it works fine on my computer (Windows 10).
I don't know how to make it work with the 32 bit version.
Windows (x64)
Raspberry Pi OS (Arm64)
oO0XX0Oo
User
User
Posts: 78
Joined: Thu Aug 10, 2017 7:35 am

Re: Write block of data to file - fastest way?

Post by oO0XX0Oo »

Ok, thanks wilbert
Simo_na
Enthusiast
Enthusiast
Posts: 177
Joined: Sun Mar 03, 2013 9:01 am

Re: Write block of data to file - fastest way?

Post by Simo_na »

wilbert wrote: Sun Feb 18, 2018 7:02 pm I found this approach online.
Not sure if it works on all Windows versions. :?
the same code but with fopen64 ?

it seems that the 64-bit Purebasic compiler (on various occasions) doesn't like it very much the 64 bit.... :shock:
ZX80
Enthusiast
Enthusiast
Posts: 330
Joined: Mon Dec 12, 2016 1:37 pm

Re: Write block of data to file - fastest way?

Post by ZX80 »

Hi, all!

I found an old code by User_Russian on my hdd (on your/the topic). See if this helps you:

Code: Select all

Procedure CreateSpaceFile(FileName.s, Size.q)
  Protected Result, h.l, l.l, FileID
  
  Result=#False
  h = (Size>>32)&$FFFFFFFF
  l = Size&$FFFFFFFF
  
  FileID = CreateFile(#PB_Any, FileName)
  If FileID
    hMapping=CreateFileMapping_(FileID(FileID), 0, #PAGE_READWRITE, h, l, 0)
    If hMapping
      Result=#True
      CloseHandle_(hMapping)
    EndIf
    CloseFile(FileID)
  EndIf
  
  ProcedureReturn Result
EndProcedure

; Quickly create an 8 GB file. Blank.
Debug CreateSpaceFile("C:\Test.bin", 8*1024*1024*1024)
All credit rightfully has to go to User_Russian.
I'm out of business :D
Simo_na
Enthusiast
Enthusiast
Posts: 177
Joined: Sun Mar 03, 2013 9:01 am

Re: Write block of data to file - fastest way?

Post by Simo_na »

ZX80 wrote: Tue Mar 08, 2022 12:17 pm Hi, all!
I found an old code by User_Russian on my hdd (on your/the topic). See if this helps you:
G R E A T !

Thank you
Post Reply