[Solved] Get all files on disk without recursion?

Just starting out? Need help? Post your questions and find answers here.
BarryG
Addict
Addict
Posts: 3293
Joined: Thu Apr 18, 2019 8:17 am

[Solved] Get all files on disk without recursion?

Post by BarryG »

Hi all, I remember seeing a procedure for getting a list of files on a disk without recursion, but can't find it now. You could call the procedure and it never called itself, but was able to get all subfolder files too. I didn't save it for some reason, but now I need it. Anyone remember how it was done? Thanks.

[Edit] I need to get each filename on-the-fly to do something with them; not just get a big list at the end.

[Edit 2] Found the post I was looking for - and it was by Freak! -> viewtopic.php?p=297865#p297865
Last edited by BarryG on Thu Feb 25, 2021 5:11 pm, edited 3 times in total.
RASHAD
PureBasic Expert
PureBasic Expert
Posts: 4636
Joined: Sun Apr 12, 2009 6:27 am

Re: Get all files on disk without recursion?

Post by RASHAD »

Hi BarryG

Code: Select all

dir c: /w /s >> c:\users\barry??\list.txt 
Egypt my love
Marc56us
Addict
Addict
Posts: 1477
Joined: Sat Feb 08, 2014 3:26 pm

Re: Get all files on disk without recursion?

Post by Marc56us »

Yes, and to have full filename

Code: Select all

dir /s /b
BarryG
Addict
Addict
Posts: 3293
Joined: Thu Apr 18, 2019 8:17 am

Re: Get all files on disk without recursion?

Post by BarryG »

Sorry, see my first post edit.
User avatar
kpeters58
Enthusiast
Enthusiast
Posts: 341
Joined: Tue Nov 22, 2011 5:11 pm
Location: Kelowna, BC, Canada

Re: Get all files on disk without recursion?

Post by kpeters58 »

BarryG wrote:Sorry, see my first post edit.
I don't understand...

So take Rashad's list and iterate over it, doing to each/some file(s) whatever you need to do.
PB 5.73 on Windows 10 & OS X High Sierra
BarryG
Addict
Addict
Posts: 3293
Joined: Thu Apr 18, 2019 8:17 am

Re: Get all files on disk without recursion?

Post by BarryG »

Waiting until the list populates (from "dir") takes too long (I have almost half a million files on a 2 TB slow USB drive), and results in a 15 MB text file to parse. I need to start displaying the files immediately, in real-time. I can't make the user wait 20 seconds for their file list.

Plus, the user needs to be able to cancel the list build at any time, which "dir" won't let me do while that long 20-second wait is happening.

Plus, I need to filter by size and/or date, so it means touching all the files anyway. Better just to do one big non-recursive loop and filter them on-the-fly.
Last edited by BarryG on Thu Feb 25, 2021 4:49 pm, edited 2 times in total.
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Get all files on disk without recursion?

Post by NicTheQuick »

In principle you can just use a recursing version and then handling the stack by yourself. Or you also can define a callback which is called for every file. I will try something. Gimme a few minutes.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
User avatar
jacdelad
Addict
Addict
Posts: 1436
Joined: Wed Feb 03, 2021 12:46 pm
Location: Planet Riesa
Contact:

Re: Get all files on disk without recursion?

Post by jacdelad »

Use two lists. Put the base path in the first list. Start a loop that processes all elements of the list, not a for-loop (use a while-loop, since you don't know how many entries it will finally contain). Whole processing said elements write the files into the second list, subdirs into the first. When done, process the next subdir in the first list, if available.

Code: Select all

EnableExplicit
Define counter,examine,NewList List1.s(),NewList List2.s(),dirtemp.s
AddElement(List1())
List1()="D:\PureBasic"
While ListSize(List1())>0
  SelectElement(List1(),0)
  dirtemp=List1()
  examine=ExamineDirectory(#PB_Any,dirtemp,"*.*")
  If examine
    While NextDirectoryEntry(examine)
      Select DirectoryEntryType(examine)
        Case #PB_DirectoryEntry_Directory
          If Len(ReplaceString(DirectoryEntryName(examine),".",""))>0
            AddElement(List1())
            List1()=dirtemp+"\"+DirectoryEntryName(examine)
          EndIf
        Case #PB_DirectoryEntry_File
          AddElement(List2())
          List2()=dirtemp+"\"+DirectoryEntryName(examine)
          Debug List2()
      EndSelect
    Wend
    FinishDirectory(examine)
  EndIf
  SelectElement(List1(),0)
  DeleteElement(List1())
Wend
Last edited by jacdelad on Thu Feb 25, 2021 5:13 pm, edited 1 time in total.
PureBasic 6.04/XProfan X4a/Embarcadero RAD Studio 11/Perl 5.2/Python 3.10
Windows 11/Ryzen 5800X/32GB RAM/Radeon 7770 OC/3TB SSD/11TB HDD
Synology DS1821+/36GB RAM/130TB
Synology DS920+/20GB RAM/54TB
Synology DS916+ii/8GB RAM/12TB
Axolotl
Enthusiast
Enthusiast
Posts: 435
Joined: Wed Dec 31, 2008 3:36 pm

Re: Get all files on disk without recursion?

Post by Axolotl »

Hi,

if I understand your question right, than this could be of help:

Code: Select all

Procedure MakeDirectoryListwithoutRecursion(Directory$)  ;' returns number of found items 
  Protected NewList folders$()  ;' temp directories 
  Protected dir$, name$, count, QuitLoop, DirID = 0  

  If Directory$ = ""  ;' missing start directory 
    Directory$ = PathRequester("Please select the drive and directory to catalogue", "")  
    If Directory$ = ""  
      ProcedureReturn 0  ;' nothing to examine 
    EndIf  
  EndIf  

; ClearGadgetItemList(#GADGET_Items)  
Debug "Start with " + Directory$ 

  ClearList(folders$())  
  AddElement(folders$())   ;' init first element 
  folders$() = RTrim(Directory$, "\")  ;' remove trailing backslash    
  count = 0

  Repeat  
    If ListSize(folders$()) = 0 : Break : EndIf 

    FirstElement(folders$()) 
    dir$ = folders$()  
    DeleteElement(folders$(), 1)  ;' current is the next element 

    If ExamineDirectory(DirID, dir$, "*.*")  
      While NextDirectoryEntry(DirID) 
        If DirectoryEntryType(DirID) = #PB_DirectoryEntry_File 
          name$ = DirectoryEntryName(DirID)  
          Debug "List[" + RSet(Str(count), 4) + "] = '" + dir$ + "\" + name$ + "'" 
;         AddGadgetItem(#GADGET_Items, - 1, dir$ + name$)  
;         FlushEvents()  
;         SetStatus(1, "File(s) " + Str(count)) 
;         SendMessage_(GadgetID(#GADGET_Items), #LVM_ENSUREVISIBLE, count, 0)  
          count + 1  

        Else ; = #PB_DirectoryEntry_Directory 
          name$ = DirectoryEntryName(DirID)  
          If name$ <> ".." And name$ <> "."  ;' i dont want the special windows directories 
            AddElement(folders$())           ;' keep directory for further processing 
            folders$() = dir$ + "\" + name$             ;:Debug "add to folders() = '" + folders$() + "'  " + ListSize(folders$()) 
          EndIf  
        EndIf
      Wend 
    EndIf  
  Until ListSize(folders$()) = 0 

  ProcedureReturn count 
EndProcedure ;() 

;' call some sample directories for test -- I made only some short ones :-)  
;Debug MakeDirectoryListwithoutRecursion("C:\temp\") 
;Debug MakeDirectoryListwithoutRecursion(<StartDirectory>) 

Mostly running PureBasic <latest stable version and current alpha/beta> (x64) on Windows 11 Home
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Get all files on disk without recursion?

Post by NicTheQuick »

This should do the trick:

Code: Select all

EnableExplicit

#PATH_SEP = "/"

Structure iterStack
	hDir.i
	path.s
EndStructure
Procedure iterateFiles(path.s)
	Protected NewList stack.iterStack()
	Protected fullpath.s, name.s, nextLevel.i = #False
	
	AddElement(stack())
	stack()\path = path
	stack()\hDir = 0
	
	While FirstElement(stack())
		With stack()
			If \hDir = 0
				\hDir = ExamineDirectory(#PB_Any, \path, "*")
			EndIf
			If \hDir
				nextLevel = #False
				While NextDirectoryEntry(\hDir)
					name = DirectoryEntryName(\hDir)
					If name = ".." Or name = "."
						Continue
					EndIf
					fullpath = \path + #PATH_SEP + name
					
					Debug fullpath  ; Here you will get you file!
					
					If DirectoryEntryType(\hDir) = #PB_DirectoryEntry_Directory
						AddElement(stack())
						stack()\path = fullpath
						stack()\hDir = 0
						nextLevel = #True
						Break
					EndIf
				Wend
				If nextLevel
					Continue
				EndIf
				FinishDirectory(\hDir)
			EndIf
			DeleteElement(stack())
		EndWith
	Wend
EndProcedure

iterateFiles("/home/nicolas/tmp")
I do not remember what the correct constant is for the operating system dependent path separator, so I named it #PATH_SEP.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
BarryG
Addict
Addict
Posts: 3293
Joined: Thu Apr 18, 2019 8:17 am

Re: Get all files on disk without recursion?

Post by BarryG »

Thanks everyone - your codes do the job indeed, and I've edited my first post with the code I was looking for (I found it!).
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: [Solved] Get all files on disk without recursion?

Post by NicTheQuick »

I've implemented a Depth-first search if this is what you are looking for. That's the reason why I also need to push the handle of ExamineDirectory to the stack. This way it is comparable to a naive recursion search.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
BarryG
Addict
Addict
Posts: 3293
Joined: Thu Apr 18, 2019 8:17 am

Re: [Solved] Get all files on disk without recursion?

Post by BarryG »

Thanks, NicTheQuick. Also, the constant for the path separator is #PS$.
AZJIO
Addict
Addict
Posts: 1319
Joined: Sun May 14, 2017 1:48 am

Re: [Solved] Get all files on disk without recursion?

Post by AZJIO »

NicTheQuick
A week ago I posted the same code, but I have no stack deletion, but an array is used. You can embed the nesting depth, for example, request a depth of 2 or 3 levels. In the archive, I made the formation of a list including, for example, size and date. If you enable sorting, it works slowly, while I think how to optimize.

Your function is called iterateFiles, but it contains files and folders. I used mine and your function, I ended up with different lists. I sorted and compared in Meld, the difference is in adding folders.

I made a change to my code and began to receive only files in the specified folder. Now you can use the number as the search depth.

Code: Select all

; AZJIO
; https://www.purebasic.fr/english/viewtopic.php?p=566355#p566355
EnableExplicit

Procedure FileSearch(*Result.string, sPath.s, Mask$ = "*", depth=130)
	Protected Len, *Point
	Protected NewList Files.s()
	
	Protected sName.s, c = 0
	Protected Dim aExaDir(depth)
	Protected Dim aSePath.s(depth)
	
	If  Right(sPath, 1) <> #PS$
		sPath + #PS$
	EndIf
	
	aSePath(c) = sPath
	aExaDir(c) = ExamineDirectory(#PB_Any, sPath, Mask$)
	If Not aExaDir(c)
		ProcedureReturn
	EndIf
	
	Repeat
		While NextDirectoryEntry(aExaDir(c))
			sName=DirectoryEntryName(aExaDir(c))
			If sName = "." Or sName = ".."
				Continue
			EndIf
			If DirectoryEntryType(aExaDir(c)) = #PB_DirectoryEntry_Directory
				If c >= depth
					Continue
				EndIf
				sPath = aSePath(c)
				c + 1
				aSePath(c) = sPath + sName + #PS$
				aExaDir(c) = ExamineDirectory(#PB_Any, aSePath(c), Mask$)
				If Not aExaDir(c)
					c - 1
				EndIf
			Else
				If AddElement(Files())
					Files() = aSePath(c) + sName
				EndIf
			EndIf
		Wend
		FinishDirectory(aExaDir(c))
		c - 1
	Until c < 0
	
	Debug "Depth = " + Str(depth)
	Debug "Count = " + Str(ListSize(Files()))
	Len=0
	ForEach Files()
		Len + Len(Files())+2
	Next
	
	*Result\s = Space(Len)
	*Point = @*Result\s
	ForEach Files()
		CopyMemoryString(Files()+#CRLF$, @*Point)
	Next
	
	ClearList(Files())
EndProcedure

Define StartTime=ElapsedMilliseconds()
Define Path$ = "/home/user"
Define Result.string
FileSearch(@Result, Path$, "*", 0) ; Wildcard, *.exe
Define Time.s = "Time = " + Str(ElapsedMilliseconds()-StartTime) + " ms"

Debug Time
Debug "_______________"
Debug Result\s
Post Reply