It is currently Fri Apr 20, 2018 8:54 pm

All times are UTC + 1 hour




Post new topic Reply to topic  [ 12 posts ] 
Author Message
 Post subject: Bug in long path support?
PostPosted: Fri Oct 20, 2017 11:51 am 
Offline
Enthusiast
Enthusiast

Joined: Thu Oct 21, 2010 9:46 pm
Posts: 118
Suppose LongPath$ is the full path to a folder, using the form "\\?\C:\Folder\....\Folder", where Len(Fullpath$) is somewhere over 260 characters. The "\\?\" prefix is the syntax for long paths as per MS docs here: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

Also, suppose that all parts of that path except the last one exist already, and the part that already exists is less than 255 characters.

Then, CreateDirectory(LongPath$) returns success, but actually creates a folder with the name truncated to a total path length around 257 characters.

Using the native Win32 API call directly creates the folder correctly without truncation, i.e. CreateDirectory_(LongPath$, #NUL) works fine.

It's as if PB's CreateDirectory() function is doing its own truncation of the supplied path, on the expectation that paths that long aren't valid.

Other parts of the FileSystem library do the same. e.g. FileSize(LongPath$) returns -2 to indicate a directory when the truncated version exists, but the full version does not - it's similarly truncating the path.

But, things like ExamineDirectory() work fine, so not everything is doing the truncation.

(PB 5.60)


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Thu Nov 02, 2017 10:14 am 
Offline
Administrator
Administrator

Joined: Fri May 17, 2002 4:39 pm
Posts: 13252
Location: France
PB doesn't supports such syntax for path for now


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Thu Nov 02, 2017 2:46 pm 
Offline
Enthusiast
Enthusiast

Joined: Thu Oct 21, 2010 9:46 pm
Posts: 118
Fred wrote:
PB doesn't supports such syntax for path for now


In the meantime, I've been able to work around it with a small library of functions that use the Win32 API directly, but it would be useful to have native support for it at some point.


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Thu Nov 02, 2017 5:34 pm 
Offline
Addict
Addict
User avatar

Joined: Thu Jun 07, 2007 3:25 pm
Posts: 3145
Location: Berlin, Germany
mikejs wrote:
[...] but it would be useful to have native support for it at some point.

Absolutely!
+1

_________________
Please excuse my flawed English. My native language is PureBasic.
Search
RSBasic's backups


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Wed Jan 10, 2018 8:42 pm 
Offline
User
User

Joined: Wed Dec 31, 2014 5:41 pm
Posts: 84
There is alot of programs out there not supporting Long filenames. I think this is much to Microsoft's own fault.
if you pass a Path+Filename with a length over 259 chars, only 256 of them will be parsed over as microsoft CMD doesn't support it.
In windows10, it has support for it, but you have to set alot a few settings around for it to work.(Never tested it myself)


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Wed Jan 10, 2018 10:02 pm 
Offline
User
User

Joined: Thu Aug 10, 2017 7:35 am
Posts: 75
@mikejs

Quote:
I've been able to work around it with a small library of functions that use the Win32 API directly


Would you be so kind to make this public?


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Thu Jan 11, 2018 4:14 am 
Offline
User
User

Joined: Wed Dec 31, 2014 5:41 pm
Posts: 84
I read somewhere that the 256 byte lengt api is ANSI, while if you wish to use the NTFS (32768 byte long) api, you have to use unicode


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Thu Jan 11, 2018 12:08 pm 
Offline
Enthusiast
Enthusiast

Joined: Thu Oct 21, 2010 9:46 pm
Posts: 118
oO0XX0Oo wrote:
@mikejs
Quote:
I've been able to work around it with a small library of functions that use the Win32 API directly

Would you be so kind to make this public?


This was put together for a particular application, and so only does the bits needed by that application. I've not done a thorough examination of which bits of the PB filesystem library are long path safe. Based on my own testing, the following are OK:

  • SetFileAttributes() and GetFileAttributes(). Which also work for directory attributes.
  • ExamineDirectory(), plus IsDirectory(), NextDirectoryEntry(), DirectoryEntry####(), FinishDirectory()

I've done long path safe versions of:

  • FileSize()
  • CopyFile()
  • DeleteFile()
  • DeleteDirectory()
  • CreateDirectory()

All the long path aware versions have an LP suffix. So CopyFileLP() is the long path safe version of CopyFile(). I also needed long path aware versions of a couple of my own library functions - DirExists() and FileExists(), so I also have DirExistsLP() which is a wrapper around FileSizeLP(), etc.

One complication is that the application that needed these also needed proper error detection and reporting of what went wrong, which means calling GetLastError_() as close as possible to where an error occurred. This means GetLastError_() has to be called within the lib function if an error happens. For my stuff I used a global variable to pass the error number back to the caller in that case. In the code below th_errcode is what is used for that. I've left that in, but removed some bits related to my own logging of whats going on.

Sometimes you can get away with a simple wrapper around the corresponding API call - e.g. CopyFileLP() has nothing else going on. Some of the others are more complicated - e.g. the API call RemoveDirectory_() does not do the recursion for you so we have to handle that ourselves, and dealing with errors gets more complicated (see comments).

Code:
Procedure.q FileSizeLP(file$)
  Protected.s parent$, name$
  Protected.i dhnd
  Protected.q out

  ; Follow the PB approach of -1 for not found, -2 for a directory.
  ; Start with it at -1 and update if we find anything.
  out=-1
 
  parent$=GetPathPart(file$)
  name$=GetFilePart(file$)
 
  dhnd=ExamineDirectory(#PB_Any, parent$, name$)
  If IsDirectory(dhnd)
    While NextDirectoryEntry(dhnd)
      If DirectoryEntryType(dhnd)=#PB_DirectoryEntry_File
        If LCase(Trim(name$))=LCase(Trim(DirectoryEntryName(dhnd)))
          out=DirectoryEntrySize(dhnd)
        EndIf
      EndIf
      If DirectoryEntryType(dhnd)=#PB_DirectoryEntry_Directory
        If LCase(Trim(name$))=LCase(Trim(DirectoryEntryName(dhnd)))
          ; -2 for a directory.
          out=-2
        EndIf
      EndIf
    Wend
    FinishDirectory(dhnd)
  EndIf
 
  ProcedureReturn out
EndProcedure


Procedure.l DirExistsLP(dir$)
  Protected.l out
  out=#False
 
  ; Using long path aware version of filesize, which follows the PB approach of -1 for not found
  ; and -2 for a directory.
  If FileSizeLP(dir$)=-2
    out=#True
  EndIf
 
  ProcedureReturn out
EndProcedure


Procedure.l FileExistsLP(file$)
  Protected.l out
  out=#False
  Protected.q tmp
 
  ; Using long path aware version of filesize, which follows the PB approach of -1 for not found
  ; and -2 for a directory.
  tmp=FileSizeLP(file$)
  If tmp=-2
    ; dir
  Else
    If tmp=-1
      ; not found
    Else
      ; found, not a dir.
      out=#True
    EndIf
  EndIf
 
  ProcedureReturn out
EndProcedure


Procedure.l CreateDirectoryLP(dir$)
  Protected.l out
 
  ; Just call CreateDirectory_() instead. #NUL gives us default security, which is what the PB function would have done.
  out=CreateDirectory_(dir$, #NUL)
  ; update th_errcode
  ; 0 indicates error
  If out=0
    th_errcode=GetLastError_()
  EndIf
 
  ProcedureReturn out
EndProcedure


Procedure.l DeleteFileLP(file$, mode.l=0, recursion.b=#False)
  Protected.l out
 
  If (mode & #PB_FileSystem_Force)
    ; explicitly remove attributes first.
    SetFileAttributes(file$, #PB_FileSystem_Normal)
  EndIf
 
  ; Can now call DeleteFile_()
  out=DeleteFile_(file$)
  ; update th_errcode
  ; 0 indicates error
  If out=0
    th_errcode=GetLastError_()
  EndIf
 
  ProcedureReturn out
EndProcedure


Procedure.l DeleteDirectoryLP(dir$, pattern$, mode.l=0, recursion.b=#False)
  Protected.l out
  Protected.l result
  Protected.i dhnd
  Protected.s name$
  Protected.l errcode
 
  ; Handling the errcode value here is tricky.
  ; If we have errors during the recursion, we'll leave something behind in the folder,
  ; and the final RemoveDirectory_() call will always return a "directory not empty" error,
  ; which is technically correct but hides the underlying problem.
 
  ; The errorcode for this is 145.
 
  ; So, there are two things we need to do to provide a more meaningful error.
  ; 1. On initial entry to the function (i.e. not on recursive calls), set th_errcode to 0.
  ; 2. On getting a 0 returned from RemoveDirectory_(), we only override th_errcode if it's still 0
  ;    or the errcode is something other than 145.
 
  ; Firstly, if we're not calling ourselves recursively, set th_errcode to 0.
  If recursion=#False
    th_errcode=0
  EndIf
 
  ; also to match PB, we have a default pattern$, but the parameter is not optional.
  If pattern$="" : pattern$="*.*" : EndIf
 
  ; Need to do our own recursion, before calling RemoveDirectory_()
  dhnd=ExamineDirectory(#PB_Any, dir$, pattern$)
  If IsDirectory(dhnd)
    While NextDirectoryEntry(dhnd)
      name$=DirectoryEntryName(dhnd)
      If name$="." Or name$=".."
        ; skip it
      Else
        ; If it's a directory, recurse into it, if the mode says to.
        If DirectoryEntryType(dhnd)=#PB_DirectoryEntry_Directory
          If (mode & #PB_FileSystem_Recursive)
            DeleteDirectoryLP(dir$+"\"+name$, pattern$, mode, #True)
          EndIf
        EndIf
        ; If it's a file, delete it with DeleteFileLP()
        If DirectoryEntryType(dhnd)=#PB_DirectoryEntry_File
          DeleteFileLP(dir$+"\"+name$, mode)
        EndIf
      EndIf
    Wend
    FinishDirectory(dhnd)
  EndIf
 
  ; Finally, call RemoveDirectory_()
  out=RemoveDirectory_(dir$)
  ; 0 indicates error
  If out=0
    errcode=GetLastError_()
    ; Use this to overwrite th_errcode if it' still 0.
    If th_errcode=0
      th_errcode=errcode
    EndIf
    ; or if errcode is something other than 145
    If errcode<>145
      th_errcode=errcode
    EndIf
  EndIf
 
  ProcedureReturn out
EndProcedure


Procedure.l CopyFileLP(src$, dest$)
  Protected.l out
 
  ; Just call CopyFile_ instead. #False means we don't fail if the dest exists.
  out=CopyFile_(src$, dest$, #False)
  ; update th_errcode
  ; 0 indicates error
  If out=0
    th_errcode=GetLastError_()
  EndIf
 
  ProcedureReturn out
EndProcedure


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Thu Jan 11, 2018 12:16 pm 
Offline
Enthusiast
Enthusiast

Joined: Thu Oct 21, 2010 9:46 pm
Posts: 118
GenRabbit wrote:
There is alot of programs out there not supporting Long filenames. I think this is much to Microsoft's own fault.
if you pass a Path+Filename with a length over 259 chars, only 256 of them will be parsed over as microsoft CMD doesn't support it.
In windows10, it has support for it, but you have to set alot a few settings around for it to work.(Never tested it myself)


Windows has had long path support for a long time - you just had to specify the path in such a way that you were telling windows that you were fine with long paths and weren't limited by by the 256-ish character limit. Long paths using the prefix ("\\?\") work fine in CMD, as do utilities that prepend that prefix as needed (RoboCopy; the thing I'm writing here), and they work fine on Win7 and probably earlier.

What's changed in Windows 10 is that MS is starting to have long path support on by default without the application needing to say that it's OK with paths over 256 characters. This means that more things "just work" with long paths without having to do anything special, with the risk that a long path might be passed to an application that genuinely cannot handle it.


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Thu Jan 11, 2018 1:17 pm 
Offline
User
User

Joined: Thu Aug 10, 2017 7:35 am
Posts: 75
Thanks a lot for providing these functions (and explanations), mikejs!


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Thu Jan 11, 2018 1:30 pm 
Offline
Enthusiast
Enthusiast

Joined: Thu Oct 21, 2010 9:46 pm
Posts: 118
Forgot to mention - when using the LP aware functions, the path you pass has to have already been prepended by the appropriate prefix to indicate to Windows that you're OK with long paths (unless you're on a new enough Win10 build for that to be implicit).

I have a function for ensuring this prefix is present and in the right format:

Code:
Procedure.s EnsureLongPath(path$)
  Protected out$
 
  ; Take the supplied path and ensure that it has the necessary prefix to work as a long path
 
  ; See this doc: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx
 
  ; Essentially, drive letter paths need to end up as \\?\C:\whatever
  ; i.e. a simple prefix of \\?\
  ; Meanwhile, unc paths need to convert:
  ; \\server\share
  ; to
  ; \\?\UNC\server\share
  ; This is a bit trickier as it isn't a simple prefix - we need to lop off one of the \ from the original
  ; before prefixing.
 
  ; Either way, if it already starts with \\?\, we have no work to do. 
  If Left(path$,4)="\\?\"
    ; path already has long path support prefix
    out$=path$
  Else
    ; Is it a UNC?
    If Left(path$,2)="\\"
      out$="\\?\UNC"+Mid(path$,2)
    Else
      ; Is it a drive letter?
      If Mid(path$,2,1)=":"
        ; Add long path support to the path.
        out$="\\?\"+path$
      Else
        ; Dunno what kind of path this is. Leave it alone.
        out$=path$
      EndIf
    EndIf
  EndIf
 
  ProcedureReturn out$
EndProcedure


Top
 Profile  
Reply with quote  
 Post subject: Re: Bug in long path support?
PostPosted: Sun Jan 14, 2018 5:19 am 
Offline
User
User

Joined: Wed Dec 31, 2014 5:41 pm
Posts: 84
mikejs wrote:
GenRabbit wrote:
There is alot of programs out there not supporting Long filenames. I think this is much to Microsoft's own fault.
if you pass a Path+Filename with a length over 259 chars, only 256 of them will be parsed over as microsoft CMD doesn't support it.
In windows10, it has support for it, but you have to set alot a few settings around for it to work.(Never tested it myself)


Windows has had long path support for a long time - you just had to specify the path in such a way that you were telling windows that you were fine with long paths and weren't limited by by the 256-ish character limit. Long paths using the prefix ("\\?\") work fine in CMD, as do utilities that prepend that prefix as needed (RoboCopy; the thing I'm writing here), and they work fine on Win7 and probably earlier.

What's changed in Windows 10 is that MS is starting to have long path support on by default without the application needing to say that it's OK with paths over 256 characters. This means that more things "just work" with long paths without having to do anything special, with the risk that a long path might be passed to an application that genuinely cannot handle it.


Ok, I always thought longpathandfilename where not supported in CMD.
I knew NTFS support it.

Form Win10, version 1607, you no longer need the prefix ("\\?\") as far as I can tell from MSDN

"Tip Starting with Windows 10, version 1607, you can opt-in to remove the MAX_PATH limitation without prepending "\\?\". See the "Maximum Path Length Limitation" section of Naming Files, Paths, and Namespaces for details."


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

 


Powered by phpBB © 2008 phpBB Group
subSilver+ theme by Canver Software, sponsor Sanal Modifiye