TreeSize: Get sizes of a directory and its sub-directories

Share your advanced PureBasic knowledge/code with the community.
Little John
Addict
Addict
Posts: 4777
Joined: Thu Jun 07, 2007 3:25 pm
Location: Berlin, Germany

TreeSize: Get sizes of a directory and its sub-directories

Post by Little John »

The decisive point is here, to do a so-called post-order traversal of the directory tree. That is, first visit all subtrees of the given directory, then visit its root.

//edit 2022-02-21: added error handling

Code: Select all

; -- Get sizes of a directory and its sub-directories
; tested with PB 5.73 LTS (32 bit and 64 bit)
; <https://www.purebasic.fr/english/viewtopic.php?t=78722>

EnableExplicit

Structure Node
   name$
   level.i
   bytes.q
EndStructure


Procedure.q _TreeSize (path$, List dir.Node(), level.i=0)
   ; -- recursive post-order traversal of a directory tree
   ; in : path$: path *without* trailing separator
   ;      level: recursion level
   ; out: dir()       : list containing 'path$' and its subdirectories
   ;      return value: total size of all files in 'path$'
   ;                    (including all sub-directories);
   ;                    -1 on error reading 'path$' itself
   ;                    -2 on error reading sub-tree of 'path$'
   Protected id.i, subSize.q, localSize.q=0, ret.q=0
   
   id = ExamineDirectory(#PB_Any, path$, "")
   If id
      While NextDirectoryEntry(id)
         If DirectoryEntryType(id) & #PB_DirectoryEntry_File
            localSize + DirectoryEntrySize(id)
         ElseIf (DirectoryEntryType(id) & #PB_DirectoryEntry_Directory) And
                (FindString("..", DirectoryEntryName(id)) = 0)
            subSize = _TreeSize(path$ + #PS$ + DirectoryEntryName(id), dir(), level+1)
            If ret >= 0 And subSize >= 0
               ret + subSize
            Else
               ret = -2
            EndIf
         EndIf
      Wend
      FinishDirectory(id)
      If ret >= 0
         ret + localSize
      EndIf
   Else
      ret = - 1
   EndIf
   
   AddElement(dir())
   dir()\name$ = path$
   dir()\level = level   
   dir()\bytes = ret
   
   ProcedureReturn ret
EndProcedure


CompilerIf #PB_Compiler_OS = #PB_OS_Windows
   #PSChars$ = "\/"    ; characters used as path separators
CompilerElse
   #PSChars$ = "/"
CompilerEndIf

Macro RTrimPS (_path_)
   ; _path_ must not be an expression, but a variable.
   If FindString(#PSChars$, Right(_path_,1)) <> 0
      _path_ = Left(_path_, Len(_path_)-1)
   EndIf
EndMacro


Procedure TreeSize (path$, List dir.Node(), sortBySize.i=#False)
   ; The paths can only be sorted meaningfully *without* a trailing separator.
   RTrimPS(path$)
   
   _TreeSize(path$, dir())
   
   If ListSize(dir()) > 1
      SortStructuredList(dir(), #PB_Sort_Ascending|#PB_Sort_NoCase, OffsetOf(Node\name$), TypeOf(Node\name$))
      If sortBySize
         SortStructuredList(dir(), #PB_Sort_Descending, OffsetOf(Node\bytes), TypeOf(Node\bytes))
      EndIf
   EndIf
EndProcedure


;-- Demo
Procedure.s StrR (n.q, width.i)
   ; -- convert a quad number into a right-aligned string
   Protected s$ = Str(n)
   
   If Len(s$) < width
      ProcedureReturn RSet(s$, width)
   Else
      ProcedureReturn s$
   EndIf
EndProcedure

Define path$
NewList dir.Node()

; --------------------------------------------------
path$ = ...   ; put your path here
; --------------------------------------------------

If FileSize(path$) <> -2
   Debug "Directory '" + path$ + "' not found."
   End   
EndIf

TreeSize(path$, dir(), #True)

Debug "Size of '" + path$ + "' (including sub-directories):"
Debug ""
Debug "Level          Bytes    Name"
Debug "-----          -----    ----"
ForEach dir()
   Debug StrR(dir()\level, 4) + Space(5) + StrR(dir()\bytes, 11) + Space(4) + dir()\name$ + #PS$
Next
Last edited by Little John on Mon Feb 21, 2022 10:05 am, edited 2 times in total.
davido
Addict
Addict
Posts: 1890
Joined: Fri Nov 09, 2012 11:04 pm
Location: Uttoxeter, UK

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by davido »

@Little John,

I'll find this very useful. Thank you for sharing. :D
Tested on a MacBook m1, PB5.73LTS, X64
DE AA EB
BarryG
Addict
Addict
Posts: 4128
Joined: Thu Apr 18, 2019 8:17 am

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by BarryG »

Slight bug: the debug output shows a negative values for some subdirectories.

I'm using 32-bit PureBasic and tested with "C:\Windows\System32\". Works fine with 64-bit PureBasic.

Code: Select all

Level          Bytes    Name
-----          -----    ----
   0      -273025660    C:\Windows\System32\
   1     -1631535995    C:\Windows\System32\DriverStore\
   2     -1633497011    C:\Windows\System32\DriverStore\FileRepository\
Little John
Addict
Addict
Posts: 4777
Joined: Thu Jun 07, 2007 3:25 pm
Location: Berlin, Germany

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by Little John »

@davido:
You are welcome! I'm glad that the code is useful for you. And thank you for testing it on a MacBook!

@BarryG:
Thank you for the bug report! I only tested with PB 64 bit. :oops:
In the demo part of the code, please replace

Code: Select all

Procedure.s StrR (n.i, width.i)
with

Code: Select all

Procedure.s StrR (n.q, width.i)
It's fixed now in the first post here.
BarryG
Addict
Addict
Posts: 4128
Joined: Thu Apr 18, 2019 8:17 am

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by BarryG »

Hi Little John! So I checked it again, but some calculations are still wrong on 32-bit (sorry!).

I tried to bug-fix it but can't work out where it's going wrong. So it's back over to you, lol.

Image

Image
Little John
Addict
Addict
Posts: 4777
Joined: Thu Jun 07, 2007 3:25 pm
Location: Berlin, Germany

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by Little John »

Hi BarryG, thanks again!
The program doesn't have reading permission for all sub-directories of "C:\Windows\System32\".
I added error handling now.
BarryG
Addict
Addict
Posts: 4128
Joined: Thu Apr 18, 2019 8:17 am

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by BarryG »

We're getting there... but not all folders are returning a size.

One example is "C:\Windows\System32\Tasks\". This is the debug output when run as Admin:

Code: Select all

Size of 'C:\Windows\System32\Tasks\' (including sub-directories):

Level          Bytes    Name
-----          -----    ----
   0               0    C:\Windows\System32\Tasks\
   1               0    C:\Windows\System32\Tasks\Microsoft\
   2               0    C:\Windows\System32\Tasks\Microsoft\Windows\
   3               0    C:\Windows\System32\Tasks\Microsoft\Windows\PLA\
   4               0    C:\Windows\System32\Tasks\Microsoft\Windows\PLA\System\
   3               0    C:\Windows\System32\Tasks\Microsoft\Windows\RemoteApp and Desktop Connections Update\
   3               0    C:\Windows\System32\Tasks\Microsoft\Windows\SyncCenter\
   3               0    C:\Windows\System32\Tasks\Microsoft\Windows\TaskScheduler\
   3               0    C:\Windows\System32\Tasks\Microsoft\Windows\WCM\
But WizTree can read that folder just fine with Admin rights:

Image

If WizTree can read that folder with Admin rights, then there must be a way for PureBasic to read it as Admin as well?

Sorry for being a thorn in your side!
Little John
Addict
Addict
Posts: 4777
Joined: Thu Jun 07, 2007 3:25 pm
Location: Berlin, Germany

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by Little John »

Hi again!
I can confirm your results (on Windows 11) when running my code with PB 5.73 LTS 32 bit.
However, when running the code with PB 5.73 LTS 64 bit, I'm getting different results.
Sorry, I don't know the reason why.
User avatar
ChrisR
Addict
Addict
Posts: 1466
Joined: Sun Jan 08, 2017 10:27 pm
Location: France

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by ChrisR »

Hi,
Isn't it linked to the file system redirection enabled by default.
Disables file system redirection by using Wow64DisableWow64FsRedirection

Code: Select all

CompilerIf #PB_Compiler_Processor = #PB_Processor_x86
  Procedure DisableWow64FsRedirection()
    Protected IsWow64ProcessFlag, Flag
    OpenLibrary(0, "Kernel32.dll")
    If IsLibrary(0)  
      GetFunction(0, "IsWow64Process")
      CallFunction(0, "IsWow64Process", GetCurrentProcess_(), @IsWow64ProcessFlag)
      If IsWow64ProcessFlag <> 0 And SizeOf(Integer) = 4
        GetFunction(0, "Wow64DisableWow64FsRedirection") 
        CallFunction(0, "Wow64DisableWow64FsRedirection", Flag)
      EndIf
      CloseLibrary(0)
    EndIf
  EndProcedure
  
  Procedure RevertWow64FsRedirection()
    Protected IsWow64ProcessFlag, Flag
    OpenLibrary(0, "Kernel32.dll")
    If IsLibrary(0)  
      GetFunction(0, "IsWow64Process")
      CallFunction(0, "IsWow64Process", GetCurrentProcess_(), @IsWow64ProcessFlag)
      If IsWow64ProcessFlag <> 0 And SizeOf(Integer) = 4
        GetFunction(0, "Wow64RevertWow64FsRedirection") 
        CallFunction(0, "Wow64RevertWow64FsRedirection", Flag)
      EndIf
      CloseLibrary(0)
    EndIf
  EndProcedure
  ;#KEY_WOW64_64KEY	0x0100	Access a 64-bit key from either a 32-bit Or 64-bit application.
  ;#KEY_WOW64_32KEY	0x0200	Access a 32-bit key from either a 32-bit Or 64-bit application.
CompilerEndIf

CompilerIf #PB_Compiler_Processor = #PB_Processor_x86  
  DisableWow64FsRedirection()
CompilerEndIf
AZJIO
Addict
Addict
Posts: 2143
Joined: Sun May 14, 2017 1:48 am

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by AZJIO »

Code: Select all

Import "Kernel32.lib"
	Wow64DisableWow64FsRedirection(*OldValue) ; значение *OldValue является параметром вывода, и его не следует изменять
	Wow64EnableWow64FsRedirection(bValue)	  ; bValue - 0 или 1 (0 - следует отключить виртуализацию)
EndImport
Wow64EnableWow64FsRedirection(0)
; your code
Wow64EnableWow64FsRedirection(1)

ChrisR
#PB_Processor_x86
Does the processor determine which system is installed on it?

Code: Select all

If FileSize(GetEnvironmentVariable("WinDir")+"\SysWow64") + 1
	x64 = 1
Else
	x64 = 0
EndIf

; Or
x64 = FileSize(GetEnvironmentVariable("WinDir")+"\SysWow64") + 1
User avatar
ChrisR
Addict
Addict
Posts: 1466
Joined: Sun Jan 08, 2017 10:27 pm
Location: France

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by ChrisR »

Hi AZJIO,
No, #PB_Processor_x86 is the architecture for which the program is created, not the architecture of the installed OS.
To check the OS Architecture under which the program is running is a 64 bit OS or not, I use this procedure:

Code: Select all

Procedure.i Is64BitOS()
  Protected HDLL, IsWow64Process_
  
  If SizeOf(Integer) = 8
    Is64BitOS = 1   ; this is a 64 bit exe
  Else
    HDll = OpenLibrary(#PB_Any, "kernel32.dll")
    If HDll
      IsWow64Process_ = GetFunction(HDll, "IsWow64Process")
      If IsWow64Process_
        CallFunctionFast(IsWow64Process_, GetCurrentProcess_(), @Is64BitOS)
      EndIf
      CloseLibrary(HDll)
    EndIf
  EndIf
  
  ProcedureReturn Is64BitOS
EndProcedure
.
It should work also indeed, by testing the SysWow64 folder, except in the case of a pure x64 Windows OS (without the x86 sub-system and without SysWow64 folder) as for some winPEs.

Code: Select all

x64 = FileSize(GetEnvironmentVariable("WinDir")+"\SysWow64") + 1
It Returns 0 or -1, which works if you test with "If x64 : code : Endif"
It's probably a little better with 0 or 1 as return using

Code: Select all

x64 = Bool(FileSize(GetEnvironmentVariable("WinDir")+"\SysWow64") = -2)
AZJIO
Addict
Addict
Posts: 2143
Joined: Sun May 14, 2017 1:48 am

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by AZJIO »

ChrisR wrote: Mon Feb 21, 2022 5:21 pm Returns 0 or -1
Bool / Abs
BarryG
Addict
Addict
Posts: 4128
Joined: Thu Apr 18, 2019 8:17 am

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by BarryG »

So using ChrisR's DisableWow64FsRedirection() procedure helps because Little John's code can now access and read the "C:\Windows\System32\Tasks\" folder (when run as Admin), but the returned folder size is still not correct:

Image
Little John
Addict
Addict
Posts: 4777
Joined: Thu Jun 07, 2007 3:25 pm
Location: Berlin, Germany

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by Little John »

When dealing with Windows system stuff like this on a 64 bit system, it's generally better to use a 64 bit program.
Running my above code with PB 5.73 64 bit, it returns exactly the same size for "C:\Windows\System32\Tasks\" as Windows itself.
BarryG
Addict
Addict
Posts: 4128
Joined: Thu Apr 18, 2019 8:17 am

Re: TreeSize: Get sizes of a directory and its sub-directories

Post by BarryG »

None of those sizes require 64-bit calculations and have anything to do with a 64-bit system, though. They could be a group of files anywhere. I'll keep trying to find another way. Thanks anyway!
Post Reply