What should ReadStringFormat(#File) return?

[simkot]

The help says that ReadStringFormat(#File) should return strings. For example, #PB_UTF16BE. But when I check a file in PB_UTF16BE encoding, it returns 4. For a file in UTF8, BOM returns 2. For ANSI - 24. How is that possible?

[STARGÅTE]

The help says that ReadStringFormat(#File) should return strings.

ReadStringFormat()

In the documentation it is written:

Result = ReadStringFormat(#File)

So, the return value is a number.
#PB_UTF16BE is not a string, it is a constant, representing 4.

[simkot]

How to find out the values ​​of other constants?
This function is kind of strange. It probably defines the encoding, but not the whole one. It doesn't define the most common UTF8 without BOM. So what's the point of it?? In other languages, such obvious and everyday functions are presented much more widely and work more clearly.

[STARGÅTE]

The values ​​don't matter (and they could change from one PureBasic version to the next).
You always compare the returned value with the constants specified in the documentation.

Code: Select all

If ReadStringFormat(#File) = #PB_UTF8
	Debug "File is UTF8"
EndIf

[simkot]

Thank you! Now it's clear. But how do you determine UTF8 without BOM? Can PureBasic do this?

[STARGÅTE]

But how do you determine UTF8 without BOM? Can PureBasic do this?

Without BOM, ReadStringFormat() returns #PB_Ascii.
There is no safe method to distinguish between ASCII and UTF8 without reading the whole file and check if all characters are valid UTF8 encoded. Even then, it could be still normal ASCII.

Please note, PureBasic always tries to read the file as UTF8, if you omit the additional flag in ReadFile(#File, Filename$ [, Flags]) which specifies the behaviour of ReadString() etc., even if the file is not valid UTF8.

[AZJIO]

But how do you determine UTF8 without BOM?

https://www.purebasic.fr/english/viewto ... 06#p636306

It doesn't define the most common UTF8 without BOM

How can he identify it if the BOM is missing? The whole essence of the definition is stored in the BOM.

[Piero]

How to find out the values ​​of other constants?

This is a job for the  RegExx0r!!! 

[HeX0R]

How to find out the values ​​of other constants?
This function is kind of strange. It probably defines the encoding, but not the whole one. It doesn't define the most common UTF8 without BOM. So what's the point of it?? In other languages, such obvious and everyday functions are presented much more widely and work more clearly.

Interesting, which other language can detect a UTF8 text file without BOM by a quick look at it?
I guess you didn't understand that function.
If it had been named "ReadFileBOM()" (which is, what it does) it might have been clearer.

This is a job for the  RegExx0r!!! 

Dunno why, but somehow I'm feeling triggered

[Piero]

This is a job for the  RegExx0r!!! 

Dunno why, but somehow I'm feeling triggered

Please don't: it's just for fun!

PS: I just "updated" it for n3wbi3z

[Demivec]

Thank you! Now it's clear. But how do you determine UTF8 without BOM? Can PureBasic do this?

https://www.purebasic.fr/english/viewtopic.php?t=64385

[simkot]

There is no safe method to distinguish between ASCII and UTF8

But it is necessary to define.

Interesting, which other language can detect a UTF8 text file without BOM by a quick look at it?

For example, in Autotit it is solved by one line

Code: Select all

FileGetEncoding ( "filehandle/filename" [, mode = 1] )

In this case, ANSI is also defined inUTF 8 without BOM.

PureBasic always tries to read the file as UTF8

PureBasic is acting somehow illogically. It encodes its .pb files in UTF8 with BOM, but reads files without BOM. It is necessary to act somehow the same way.

[Piero]

Well… don't you have Well-Done Text Editors on windows?
TextMate (Mac) in case of doubts proposes you to select the encoding, with a preview of the resulting text, optionally "saving your choice" to "better guess next time"…
I mean, so you can popup a red msgbox saying: "wtf is this? please resave it with a BOM using [freeware link]!"

PS: Dear Wise Admins, I know; I'm getting repetitive about TextMate

[simkot]

Writing a program to convert text that offers to convert it in another program... I don't think this is serious.

[Piero]

Writing a program to convert text that offers to convert it in another program... I don't think this is serious.

I understand you, but I was talking in general (it's obvious it would be shameful if your progam is a text converter)
Anyway the best tradition always was to do better "shamelessly" using others' great (possibly opensource) stuff