[Done] GetFileDate() and UTC

[mikejs]

Hi,

I've been trying to move all my stuff over to 6.10, and part of that means doing away with separate libraries for things that can now be done natively. Previously I had a dedicated function for getting the UTC timestamp of a file using windows API calls (based on the date64 library). In 610, we now have UTC dates, and although there is no GetFileDateUTC(), I assumed I could use plain GetFileDate() to get the timestamp in my current local time, and then call ConvertDate() to convert that to UTC.

I'm in the UK, and daylight savings is in effect, so for practical purposes my local timezone is UTC+1, and converting local to UTC means subtracting an hour.

But, this does not seem to work as I would expect. For the file I was looking at, these both return the same value

Code: Select all

Debug GetFileDate(file$, #PB_Date_Modified)
Debug ConvertDate(GetFileDate(file$, #PB_Date_Modified), #PB_Date_UTC)

After a bit of poking around with different files, this seems to be because the timestamp of the file is from a time when daylight savings was not in effect. i.e. the above code returns the same value, or different values, depending on the timestamp of the file you choose.

This is a) not how I would have expected this to work, and b) not how the old date64 library did things, which I think means it differs from Windows own idea of what the UTC timestamp of the file is.

If my current timezone is UTC+1, then I would expect ConvertDate() to always result in a 1hr adjustment, as so long as daylight savings is in effect there is always a 1 hour difference between my local time and UTC, and if I want the UTC value of something that is being presented in local time, that correction needs to be made.

Am I doing something wrong here, or misunderstanding how this is meant to work?

Either way, is there a native 610 way to get the UTC timestamp of a file that agrees with the value you get if you go direct to the win32 API? (I.e. I found this issue because the new code was giving different values to the old code, and I need them to agree...)

(This is all on Windows, PB 610 x64, same in ASM and C backends)

[RASHAD]

Hi
Maybe GetFileTime_() will do the job
https://www.purebasic.fr/english/viewto ... etfiletime

[mikejs]

Hi
Maybe GetFileTime_() will do the job
https://www.purebasic.fr/english/viewto ... etfiletime

Yes, that's the Win32 API way of doing it, and essentially what I was doing earlier (and provides a way to get the timestamp before it gets converted into the local timezone). I was hoping to be able to do the same in native PB, as it looks possible now.

My post above isn't very clear though, as I wrote it while I was still trying to work out what was going on. The issue isn't so much GetFileDate(), as the fact that ConvertDate() doesn't do what I expected it to do.

I was expecting ConvertDate() to apply the currently applicable offset from UTC. What it seems to do instead is apply the offset from UTC that was applicable at the time of the timstamp you give it.

I have no idea if this is correct behaviour or not, but on a practical level it results in discrepancies between my old code and my new code that's doing things the PB 610 way.

Is that how ConvertDate() is meant to work?

[mikejs]

OK, here's some actual code demonstrating the differences I see (there are more than I thought). For this to work, you need two files, one with a date inside the daylight savings period, one outside. I have these, as seen from a command prompt in windows:

Code: Select all

 Directory of C:\temp

16/04/2024  10:37                 2 BSTFile.txt
08/11/2023  13:43                48 GMTFile.txt

With these files in place, I have this code:

Code: Select all

EnableExplicit

#df$="%yyyy.%mm.%dd %hh:%ii"

Procedure.q FileTimeToDate64(*FT.FILETIME, utc.l=#False)
  Protected stUTC.SYSTEMTIME, st.SYSTEMTIME 
  Protected out.q
  
  FileTimeToSystemTime_(*FT, stUTC) 
  
  If utc
    ; Return the UTC time with no TZ conversion
    out=Date(stUTC\wYear, stUTC\wMonth, stUTC\wDay, stUTC\wHour, stUTC\wMinute, stUTC\wSecond)  
  Else
    ; Convert to local
    SystemTimeToTzSpecificLocalTime_(#Null, stUTC, st)
    out=Date(st\wYear, st\wMonth, st\wDay, st\wHour, st\wMinute, st\wSecond)  
  EndIf
  
  ProcedureReturn out
EndProcedure

Procedure.q GetFileDate64(file$, type.l, utc.l=#False)
  Protected.q out
  Protected.i hnd, rc
  Protected ft.FILETIME
  
  hnd = CreateFile_(@File$, #GENERIC_READ, #FILE_SHARE_READ|#FILE_SHARE_WRITE, 0, #OPEN_EXISTING, #FILE_ATTRIBUTE_NORMAL|#FILE_FLAG_BACKUP_SEMANTICS, 0) 
  If hnd <> #INVALID_HANDLE_VALUE 
    Select type 
      Case #PB_Date_Created  : rc = GetFileTime_(hnd, ft, 0, 0) 
      Case #PB_Date_Accessed : rc = GetFileTime_(hnd, 0, ft, 0) 
      Case #PB_Date_Modified : rc = GetFileTime_(hnd, 0, 0, ft) 
    EndSelect 
    If rc = 0                                                                  
      ; error
    Else 
      out = FileTimeToDate64(ft, utc) 
    EndIf 
    CloseHandle_(hnd)
  EndIf 
  
  ProcedureReturn out
EndProcedure

Define.s gmtfile$="C:\Temp\GMTFile.txt"
Define.s bstfile$="C:\Temp\BSTFile.txt"

Debug "File with a timestamp in GMT:"
Debug "PB date local:   "+FormatDate(#df$, GetFileDate(gmtfile$, #PB_Date_Modified))
Debug "PB date UTC:     "+FormatDate(#df$, ConvertDate(GetFileDate(gmtfile$, #PB_Date_Modified), #PB_Date_UTC))
Debug "API date local:  "+FormatDate(#df$, GetFileDate64(gmtfile$, #PB_Date_Modified, #False))
Debug "API date UTC:    "+FormatDate(#df$, GetFileDate64(gmtfile$, #PB_Date_Modified, #True))
Debug ""
Debug "File with a timestamp in BST:"
Debug "PB date local:   "+FormatDate(#df$, GetFileDate(bstfile$, #PB_Date_Modified))
Debug "PB date UTC:     "+FormatDate(#df$, ConvertDate(GetFileDate(bstfile$, #PB_Date_Modified), #PB_Date_UTC))
Debug "API date local:  "+FormatDate(#df$, GetFileDate64(bstfile$, #PB_Date_Modified, #False))
Debug "API date UTC:    "+FormatDate(#df$, GetFileDate64(bstfile$, #PB_Date_Modified, #True))

This produces this output:

Code: Select all

File with a timestamp in GMT:
PB date local:   2023.11.08 13:43
PB date UTC:     2023.11.08 13:43
API date local:  2023.11.08 12:43
API date UTC:    2023.11.08 12:43

File with a timestamp in BST:
PB date local:   2024.04.16 10:37
PB date UTC:     2024.04.16 09:37
API date local:  2024.04.16 10:37
API date UTC:    2024.04.16 09:37

For the file with a BST timestamp, both methods give the same answers and I think they are right. Whether they would still agree and still be right after daylight savings ends is harder to test.

For the file with the timestamp in GMT, the PB and API methods differ, but there are oddities in both directions. PB gives the local time for both local and UTC times. But the API approach gives the UTC time for both. (Outside of this test program, I only cared about the UTC time, and so had not noticed this discrepancy in the other direction.)

I find it interesting that both the API and PB approaches agree that converting to UTC has no effect on this timestamp. i.e. they agree that, given that the timestamp and system timezone refer to a time when the offset from UTC was 0, there is no adjustment to make. This implies that PB's ConvertDate() is doing the right thing (and maybe using the same API calls). But, they are still 1 hour offset from one another, and this is the cause of the discrepancy I was seeing originally that lead to me poking around at this.

The question then is, who's right here - the PB output or the API? As I understand the API, the UTC timestamp is what is actually stored on disk, and by reading that as-is and not converting it, we should have the UTC timestamp as directly as possible. But, this does not agree with the PB output.

Can anyone shed any light on this?

[mk-soft]

Unfortunately there is a bug in PB.
GetFileDate returns the local datetime of the file. However, the timestamp is stored in the file system as UTC.
Unfortunately PB calculates the returned datetime with the actual datetime difference, instead of calculating with the datetime difference to the timestamp of the file. Thus the datetime is not correct because of the summer/winter time changeover

[mikejs]

Thanks for this - it's what I was suspecting...

So for now, if you want the UTC date, the API is the only reliable way to get it, I think.

In practice I was going to put this code back to using the API because I have existing code rolled out to quite a few PCs and existing timestamps in indexes and elsewhere that need to come out the same. So even if PB was right here, I need to match existing behaviour.

Having done a bit more testing, getting the date in PB via ExamineDirectory() has the same issue.

[highend]

Wouldn't it be better to report this as a bug?

With the new date library in place this should be handled correctly (and I hope it can do this for ExamineDirectory() as well)...

[hoerbie]

Are you sure, that this is a bug in PB?
I know wrong calculated times from winter/summer time also from the Windows Explorer...

[mk-soft]

Windows 10 Explorer calculates the date correctly.
I have compared

[Axolotl]

Even if the OP is solved I would like to share some related information about filetimes.
Because the file time is (unfortunately) dependent on the file system used. FAT == local time and NTFS == universal time.
But see youself on MSDN pages:
1. File-Times
2. GetFileTime

[Fred]

Fixed. DirectoryEntryDate() and ExplorerListGadget() have been also been patched.