Bug in long path support?

Got an idea for enhancing PureBasic? New command(s) you'd like to see?
mikejs
Enthusiast
Enthusiast
Posts: 160
Joined: Thu Oct 21, 2010 9:46 pm

Bug in long path support?

Post by mikejs »

Suppose LongPath$ is the full path to a folder, using the form "\\?\C:\Folder\....\Folder", where Len(Fullpath$) is somewhere over 260 characters. The "\\?\" prefix is the syntax for long paths as per MS docs here: https://msdn.microsoft.com/en-us/librar ... s.85).aspx

Also, suppose that all parts of that path except the last one exist already, and the part that already exists is less than 255 characters.

Then, CreateDirectory(LongPath$) returns success, but actually creates a folder with the name truncated to a total path length around 257 characters.

Using the native Win32 API call directly creates the folder correctly without truncation, i.e. CreateDirectory_(LongPath$, #NUL) works fine.

It's as if PB's CreateDirectory() function is doing its own truncation of the supplied path, on the expectation that paths that long aren't valid.

Other parts of the FileSystem library do the same. e.g. FileSize(LongPath$) returns -2 to indicate a directory when the truncated version exists, but the full version does not - it's similarly truncating the path.

But, things like ExamineDirectory() work fine, so not everything is doing the truncation.

(PB 5.60)
Fred
Administrator
Administrator
Posts: 16619
Joined: Fri May 17, 2002 4:39 pm
Location: France
Contact:

Re: Bug in long path support?

Post by Fred »

PB doesn't supports such syntax for path for now
mikejs
Enthusiast
Enthusiast
Posts: 160
Joined: Thu Oct 21, 2010 9:46 pm

Re: Bug in long path support?

Post by mikejs »

Fred wrote:PB doesn't supports such syntax for path for now
In the meantime, I've been able to work around it with a small library of functions that use the Win32 API directly, but it would be useful to have native support for it at some point.
Little John
Addict
Addict
Posts: 4519
Joined: Thu Jun 07, 2007 3:25 pm
Location: Berlin, Germany

Re: Bug in long path support?

Post by Little John »

mikejs wrote:[...] but it would be useful to have native support for it at some point.
Absolutely!
+1
GenRabbit
Enthusiast
Enthusiast
Posts: 118
Joined: Wed Dec 31, 2014 5:41 pm

Re: Bug in long path support?

Post by GenRabbit »

There is alot of programs out there not supporting Long filenames. I think this is much to Microsoft's own fault.
if you pass a Path+Filename with a length over 259 chars, only 256 of them will be parsed over as microsoft CMD doesn't support it.
In windows10, it has support for it, but you have to set alot a few settings around for it to work.(Never tested it myself)
oO0XX0Oo
User
User
Posts: 78
Joined: Thu Aug 10, 2017 7:35 am

Re: Bug in long path support?

Post by oO0XX0Oo »

@mikejs
I've been able to work around it with a small library of functions that use the Win32 API directly
Would you be so kind to make this public?
GenRabbit
Enthusiast
Enthusiast
Posts: 118
Joined: Wed Dec 31, 2014 5:41 pm

Re: Bug in long path support?

Post by GenRabbit »

I read somewhere that the 256 byte lengt api is ANSI, while if you wish to use the NTFS (32768 byte long) api, you have to use unicode
mikejs
Enthusiast
Enthusiast
Posts: 160
Joined: Thu Oct 21, 2010 9:46 pm

Re: Bug in long path support?

Post by mikejs »

oO0XX0Oo wrote:@mikejs
I've been able to work around it with a small library of functions that use the Win32 API directly
Would you be so kind to make this public?
This was put together for a particular application, and so only does the bits needed by that application. I've not done a thorough examination of which bits of the PB filesystem library are long path safe. Based on my own testing, the following are OK:
  • SetFileAttributes() and GetFileAttributes(). Which also work for directory attributes.
  • ExamineDirectory(), plus IsDirectory(), NextDirectoryEntry(), DirectoryEntry####(), FinishDirectory()
I've done long path safe versions of:
  • FileSize()
  • CopyFile()
  • DeleteFile()
  • DeleteDirectory()
  • CreateDirectory()
All the long path aware versions have an LP suffix. So CopyFileLP() is the long path safe version of CopyFile(). I also needed long path aware versions of a couple of my own library functions - DirExists() and FileExists(), so I also have DirExistsLP() which is a wrapper around FileSizeLP(), etc.

One complication is that the application that needed these also needed proper error detection and reporting of what went wrong, which means calling GetLastError_() as close as possible to where an error occurred. This means GetLastError_() has to be called within the lib function if an error happens. For my stuff I used a global variable to pass the error number back to the caller in that case. In the code below th_errcode is what is used for that. I've left that in, but removed some bits related to my own logging of whats going on.

Sometimes you can get away with a simple wrapper around the corresponding API call - e.g. CopyFileLP() has nothing else going on. Some of the others are more complicated - e.g. the API call RemoveDirectory_() does not do the recursion for you so we have to handle that ourselves, and dealing with errors gets more complicated (see comments).

Code: Select all

Procedure.q FileSizeLP(file$)
  Protected.s parent$, name$
  Protected.i dhnd
  Protected.q out

  ; Follow the PB approach of -1 for not found, -2 for a directory.
  ; Start with it at -1 and update if we find anything.
  out=-1
  
  parent$=GetPathPart(file$)
  name$=GetFilePart(file$)
  
  dhnd=ExamineDirectory(#PB_Any, parent$, name$)
  If IsDirectory(dhnd)
    While NextDirectoryEntry(dhnd)
      If DirectoryEntryType(dhnd)=#PB_DirectoryEntry_File
        If LCase(Trim(name$))=LCase(Trim(DirectoryEntryName(dhnd)))
          out=DirectoryEntrySize(dhnd)
        EndIf
      EndIf
      If DirectoryEntryType(dhnd)=#PB_DirectoryEntry_Directory
        If LCase(Trim(name$))=LCase(Trim(DirectoryEntryName(dhnd)))
          ; -2 for a directory.
          out=-2
        EndIf
      EndIf
    Wend
    FinishDirectory(dhnd)
  EndIf
  
  ProcedureReturn out
EndProcedure


Procedure.l DirExistsLP(dir$)
  Protected.l out
  out=#False
  
  ; Using long path aware version of filesize, which follows the PB approach of -1 for not found
  ; and -2 for a directory.
  If FileSizeLP(dir$)=-2
    out=#True
  EndIf
  
  ProcedureReturn out
EndProcedure


Procedure.l FileExistsLP(file$)
  Protected.l out
  out=#False
  Protected.q tmp
  
  ; Using long path aware version of filesize, which follows the PB approach of -1 for not found
  ; and -2 for a directory.
  tmp=FileSizeLP(file$)
  If tmp=-2
    ; dir
  Else
    If tmp=-1
      ; not found
    Else
      ; found, not a dir.
      out=#True
    EndIf
  EndIf
  
  ProcedureReturn out
EndProcedure


Procedure.l CreateDirectoryLP(dir$)
  Protected.l out
  
  ; Just call CreateDirectory_() instead. #NUL gives us default security, which is what the PB function would have done.
  out=CreateDirectory_(dir$, #NUL)
  ; update th_errcode
  ; 0 indicates error
  If out=0
    th_errcode=GetLastError_()
  EndIf
  
  ProcedureReturn out
EndProcedure


Procedure.l DeleteFileLP(file$, mode.l=0, recursion.b=#False)
  Protected.l out
  
  If (mode & #PB_FileSystem_Force)
    ; explicitly remove attributes first.
    SetFileAttributes(file$, #PB_FileSystem_Normal)
  EndIf
  
  ; Can now call DeleteFile_()
  out=DeleteFile_(file$)
  ; update th_errcode
  ; 0 indicates error
  If out=0
    th_errcode=GetLastError_()
  EndIf
  
  ProcedureReturn out
EndProcedure


Procedure.l DeleteDirectoryLP(dir$, pattern$, mode.l=0, recursion.b=#False)
  Protected.l out
  Protected.l result
  Protected.i dhnd
  Protected.s name$
  Protected.l errcode
  
  ; Handling the errcode value here is tricky.
  ; If we have errors during the recursion, we'll leave something behind in the folder,
  ; and the final RemoveDirectory_() call will always return a "directory not empty" error,
  ; which is technically correct but hides the underlying problem.
  
  ; The errorcode for this is 145.
  
  ; So, there are two things we need to do to provide a more meaningful error.
  ; 1. On initial entry to the function (i.e. not on recursive calls), set th_errcode to 0.
  ; 2. On getting a 0 returned from RemoveDirectory_(), we only override th_errcode if it's still 0
  ;    or the errcode is something other than 145.
  
  ; Firstly, if we're not calling ourselves recursively, set th_errcode to 0.
  If recursion=#False
    th_errcode=0
  EndIf
  
  ; also to match PB, we have a default pattern$, but the parameter is not optional.
  If pattern$="" : pattern$="*.*" : EndIf
  
  ; Need to do our own recursion, before calling RemoveDirectory_()
  dhnd=ExamineDirectory(#PB_Any, dir$, pattern$)
  If IsDirectory(dhnd)
    While NextDirectoryEntry(dhnd)
      name$=DirectoryEntryName(dhnd)
      If name$="." Or name$=".."
        ; skip it
      Else
        ; If it's a directory, recurse into it, if the mode says to.
        If DirectoryEntryType(dhnd)=#PB_DirectoryEntry_Directory
          If (mode & #PB_FileSystem_Recursive)
            DeleteDirectoryLP(dir$+"\"+name$, pattern$, mode, #True)
          EndIf
        EndIf
        ; If it's a file, delete it with DeleteFileLP()
        If DirectoryEntryType(dhnd)=#PB_DirectoryEntry_File
          DeleteFileLP(dir$+"\"+name$, mode)
        EndIf
      EndIf
    Wend
    FinishDirectory(dhnd)
  EndIf
  
  ; Finally, call RemoveDirectory_()
  out=RemoveDirectory_(dir$)
  ; 0 indicates error
  If out=0
    errcode=GetLastError_()
    ; Use this to overwrite th_errcode if it' still 0.
    If th_errcode=0
      th_errcode=errcode
    EndIf
    ; or if errcode is something other than 145
    If errcode<>145
      th_errcode=errcode
    EndIf
  EndIf
  
  ProcedureReturn out
EndProcedure


Procedure.l CopyFileLP(src$, dest$)
  Protected.l out
  
  ; Just call CopyFile_ instead. #False means we don't fail if the dest exists.
  out=CopyFile_(src$, dest$, #False)
  ; update th_errcode
  ; 0 indicates error
  If out=0
    th_errcode=GetLastError_()
  EndIf
  
  ProcedureReturn out
EndProcedure
mikejs
Enthusiast
Enthusiast
Posts: 160
Joined: Thu Oct 21, 2010 9:46 pm

Re: Bug in long path support?

Post by mikejs »

GenRabbit wrote:There is alot of programs out there not supporting Long filenames. I think this is much to Microsoft's own fault.
if you pass a Path+Filename with a length over 259 chars, only 256 of them will be parsed over as microsoft CMD doesn't support it.
In windows10, it has support for it, but you have to set alot a few settings around for it to work.(Never tested it myself)
Windows has had long path support for a long time - you just had to specify the path in such a way that you were telling windows that you were fine with long paths and weren't limited by by the 256-ish character limit. Long paths using the prefix ("\\?\") work fine in CMD, as do utilities that prepend that prefix as needed (RoboCopy; the thing I'm writing here), and they work fine on Win7 and probably earlier.

What's changed in Windows 10 is that MS is starting to have long path support on by default without the application needing to say that it's OK with paths over 256 characters. This means that more things "just work" with long paths without having to do anything special, with the risk that a long path might be passed to an application that genuinely cannot handle it.
oO0XX0Oo
User
User
Posts: 78
Joined: Thu Aug 10, 2017 7:35 am

Re: Bug in long path support?

Post by oO0XX0Oo »

Thanks a lot for providing these functions (and explanations), mikejs!
mikejs
Enthusiast
Enthusiast
Posts: 160
Joined: Thu Oct 21, 2010 9:46 pm

Re: Bug in long path support?

Post by mikejs »

Forgot to mention - when using the LP aware functions, the path you pass has to have already been prepended by the appropriate prefix to indicate to Windows that you're OK with long paths (unless you're on a new enough Win10 build for that to be implicit).

I have a function for ensuring this prefix is present and in the right format:

Code: Select all

Procedure.s EnsureLongPath(path$)
  Protected out$
  
  ; Take the supplied path and ensure that it has the necessary prefix to work as a long path
  
  ; See this doc: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx
  
  ; Essentially, drive letter paths need to end up as \\?\C:\whatever
  ; i.e. a simple prefix of \\?\
  ; Meanwhile, unc paths need to convert:
  ; \\server\share
  ; to
  ; \\?\UNC\server\share
  ; This is a bit trickier as it isn't a simple prefix - we need to lop off one of the \ from the original
  ; before prefixing.
  
  ; Either way, if it already starts with \\?\, we have no work to do.  
  If Left(path$,4)="\\?\"
    ; path already has long path support prefix
    out$=path$
  Else
    ; Is it a UNC?
    If Left(path$,2)="\\"
      out$="\\?\UNC"+Mid(path$,2)
    Else
      ; Is it a drive letter?
      If Mid(path$,2,1)=":"
        ; Add long path support to the path.
        out$="\\?\"+path$
      Else
        ; Dunno what kind of path this is. Leave it alone.
        out$=path$
      EndIf
    EndIf
  EndIf
  
  ProcedureReturn out$
EndProcedure
GenRabbit
Enthusiast
Enthusiast
Posts: 118
Joined: Wed Dec 31, 2014 5:41 pm

Re: Bug in long path support?

Post by GenRabbit »

mikejs wrote:
GenRabbit wrote:There is alot of programs out there not supporting Long filenames. I think this is much to Microsoft's own fault.
if you pass a Path+Filename with a length over 259 chars, only 256 of them will be parsed over as microsoft CMD doesn't support it.
In windows10, it has support for it, but you have to set alot a few settings around for it to work.(Never tested it myself)
Windows has had long path support for a long time - you just had to specify the path in such a way that you were telling windows that you were fine with long paths and weren't limited by by the 256-ish character limit. Long paths using the prefix ("\\?\") work fine in CMD, as do utilities that prepend that prefix as needed (RoboCopy; the thing I'm writing here), and they work fine on Win7 and probably earlier.

What's changed in Windows 10 is that MS is starting to have long path support on by default without the application needing to say that it's OK with paths over 256 characters. This means that more things "just work" with long paths without having to do anything special, with the risk that a long path might be passed to an application that genuinely cannot handle it.
Ok, I always thought longpathandfilename where not supported in CMD.
I knew NTFS support it.

Form Win10, version 1607, you no longer need the prefix ("\\?\") as far as I can tell from MSDN

"Tip Starting with Windows 10, version 1607, you can opt-in to remove the MAX_PATH limitation without prepending "\\?\". See the "Maximum Path Length Limitation" section of Naming Files, Paths, and Namespaces for details."
Post Reply