An easy to use solution for multilanguage programs

Share your advanced PureBasic knowledge/code with the community.
freak
PureBasic Team
PureBasic Team
Posts: 5929
Joined: Fri Apr 25, 2003 5:21 pm
Location: Germany

An easy to use solution for multilanguage programs

Post by freak »

I promised Fangles to post this one a long time ago. So before he continues throwing
smelly fish and whatnot at me, i thought i'd better post this... ;)

This is the way the PureBasic IDE manages its languages. The IDE uses a bit more code
to also store information about the language (translators etc) in the files,
but the basic code is the same.

The older solution we used was a simple array and strings were refered to by their index.
This is of course a good solution for a small programs, but it became a problem
as the project grew. The array indexes are not descriptive of what is inside,
so it is hard to tell what a particular string in the code actually contains, and it
also becomes a nightmare to add new strings, especially with the updating of
the external translated files.
The code you see here (which is in use since 3.94) has proven to make
managing the languages very simple and solves the above mentioned problems.

Here it is:

Code: Select all

; -----------------------------------------------------------------
;   Example of a simple language management for programs
; -----------------------------------------------------------------
;
; Benefits of this solution:
;  - Strings are identified by a Group and Name string, which allows to
;    organize them with descriptive names and makes coding easier.
;
;  - Strings are sorted and indexed when they are loaded, which gives
;    a fast access even though they are accessed by name.
;
;  - A default language is defined in the code (DataSection), so even if
;    external language files are missing or outdated, there is always a 
;    fallback string present.
;
;  - The list of strings is easy to extend. Simply add a new entry in the
;    DataSection and the language files and use the new Group and Name.
;
;  - Additional language files are in the PB Preference format which makes
;    them easy to maintain.
;
; Usage:
;  - define the default language in the DataSection like shown below
;  - Use LoadLanguage() at least once to load the default language or external file
;  - Use Language(Group$, Name$) to access the language strings
;
; -----------------------------------------------------------------

; Some misc stuff...
;
Global NbLanguageGroups, NbLanguageStrings

Structure LanguageGroup
  Name$
  GroupStart.l
  GroupEnd.l
  IndexTable.l[256]
EndStructure

; This procedure loads the language from a file, or the default language.
; It must be called at least once before using any language strings.
;
; It does this:
;  - load & sort the included default language
;  - load any additional language from file
;
; This way you always get a language string, even if the file is not found
; or a string entry is missing in the file. You will still get the default
; language when using tha Language() command.
;
; This function can be called multiple times to change the used language
; during runtime.
;
Procedure LoadLanguage(FileName$ = "")

  ; do a quick count in the datasection first:
  ;
  NbLanguageGroups = 0
  NbLanguageStrings = 0

  Restore Language
  Repeat

    Read.s Name$
    Read.s String$

    Name$ = UCase(Name$)

    If Name$ = "_GROUP_"
      NbLanguageGroups + 1
    ElseIf Name$ = "_END_"
      Break
    Else
      NbLanguageStrings + 1
    EndIf
   
  ForEver

  Global Dim LanguageGroups.LanguageGroup(NbLanguageGroups)  ; all one based here
  Global Dim LanguageStrings.s(NbLanguageStrings)
  Global Dim LanguageNames.s(NbLanguageStrings)

  ; Now load the standard language:
  ;  
  Group = 0
  StringIndex = 0  

  Restore Language
  Repeat

    Read.s Name$
    Read.s String$

    Name$ = UCase(Name$)

    If Name$ = "_GROUP_"
      LanguageGroups(Group)\GroupEnd   = StringIndex
      Group + 1

      LanguageGroups(Group)\Name$      = UCase(String$)
      LanguageGroups(Group)\GroupStart = StringIndex + 1
      For i = 0 To 255
        LanguageGroups(Group)\IndexTable[i] = 0
      Next i
      
    ElseIf Name$ = "_END_"
      Break

    Else
      StringIndex + 1
      LanguageNames(StringIndex)   = Name$ + Chr(1) + String$  ; keep name and string together for easier sorting

    EndIf
   
  ForEver

  LanguageGroups(Group)\GroupEnd   = StringIndex ; set end for the last group!
  
  ; Now do the sorting and the indexing for each group
  ;
  For Group = 1 To NbLanguageGroups
    If LanguageGroups(Group)\GroupStart <= LanguageGroups(Group)\GroupEnd  ; sanity check.. check for empty groups
      
      SortArray(LanguageNames(), 0, LanguageGroups(Group)\GroupStart, LanguageGroups(Group)\GroupEnd)
 
      char = 0
      For StringIndex = LanguageGroups(Group)\GroupStart To LanguageGroups(Group)\GroupEnd
        LanguageStrings(StringIndex) = StringField(LanguageNames(StringIndex), 2, Chr(1)) ; splitt the value from the name
        LanguageNames(StringIndex)   = StringField(LanguageNames(StringIndex), 1, Chr(1))

        If Asc(Left(LanguageNames(StringIndex), 1)) <> char
          char = Asc(Left(LanguageNames(StringIndex), 1))
          LanguageGroups(Group)\IndexTable[char] = StringIndex
        EndIf
      Next StringIndex
      
    EndIf
  Next Group

  ; Now try to load an external language file
  ;       
  If FileName$ <> ""
      
    If OpenPreferences(FileName$)
      For Group = 1 To NbLanguageGroups
        If LanguageGroups(Group)\GroupStart <= LanguageGroups(Group)\GroupEnd  ; sanity check.. check for empty groups
          PreferenceGroup(LanguageGroups(Group)\Name$)
         
          For StringIndex = LanguageGroups(Group)\GroupStart To LanguageGroups(Group)\GroupEnd
            LanguageStrings(StringIndex) = ReadPreferenceString(LanguageNames(StringIndex), LanguageStrings(StringIndex))
          Next StringIndex
        EndIf
      Next Group
      ClosePreferences()   
      
      ProcedureReturn #True
    EndIf    

  EndIf
  
  ProcedureReturn #True
EndProcedure


; This function returns a string in the current language
; Each string is identified by a Group and a Name (both case insensitive)
;
; If the string is not found (not even in the included default language), the
; return is "##### String not found! #####" which helps to spot errors in the
; language code easily.
;
Procedure.s Language(Group$, Name$)
  Static Group.l  ; for quicker access when using the same group more than once
  Protected String$, StringIndex, Result

  Group$  = UCase(Group$)
  Name$   = UCase(Name$)
  String$ = "##### String not found! #####"  ; to help find bugs

  If LanguageGroups(Group)\Name$ <> Group$  ; check if it is the same group as last time   
    For Group = 1 To NbLanguageGroups
      If Group$ = LanguageGroups(Group)\Name$
        Break
      EndIf
    Next Group

    If Group > NbLanguageGroups  ; check if group was found
      Group = 0
    EndIf
  EndIf
  
  If Group <> 0
    StringIndex = LanguageGroups(Group)\IndexTable[ Asc(Left(Name$, 1)) ]
    If StringIndex <> 0

      Repeat
        Result = CompareMemoryString(@Name$, @LanguageNames(StringIndex))

        If Result = 0
          String$ = LanguageStrings(StringIndex)
          Break

        ElseIf Result = -1 ; string not found!
          Break

        EndIf

        StringIndex + 1
      Until StringIndex > LanguageGroups(Group)\GroupEnd

    EndIf

  EndIf

  ProcedureReturn String$
EndProcedure


DataSection

  ; Here the default language is specified. It is a list of Group, Name pairs,
  ; with some special keywords for the Group:
  ;
  ; "_GROUP_" will indicate a new group in the datasection, the second value is the group name
  ; "_END_" will indicate the end of the language list (as there is no fixed number)
  ;
  ; Note: The identifier strings are case insensitive to make live easier :)
  
  Language:

    ; ===================================================
    Data$ "_GROUP_",            "MenuTitle"
    ; ===================================================

      Data$ "File",             "File"
      Data$ "Edit",             "Edit"
                       
    ; ===================================================
    Data$ "_GROUP_",            "MenuItem"
    ; ===================================================

      Data$ "New",              "New"
      Data$ "Open",             "Open..."
      Data$ "Save",             "Save"
         
    ; ===================================================
    Data$ "_END_",              ""
    ; ===================================================

EndDataSection

; -----------------------------------------------------------------
; Example:
; -----------------------------------------------------------------

LoadLanguage()                ; load default language
;LoadLanguage("german.prefs") ; uncomment this to load the german file

; get some language strings
;
Debug Language("MenuTitle", "Edit")
Debug Language("MenuItem", "Save")

; -----------------------------------------------------------------
The german.prefs file would look like this:

Code: Select all

[MenuTitle]
File = Datei
Edit = Bearbeiten

[MenuItem]
New  = Neu
Open = Öffnen
Save = Speichern  
btw, for a unicode program, the language files should be in UTF8 with the UTF8 byte order mark (see WriteStringFormat()) at
the beginning, so the Preference lib loads unicode strings correctly.
quidquid Latine dictum sit altum videtur
Thalius
Enthusiast
Enthusiast
Posts: 711
Joined: Thu Jul 17, 2003 4:15 pm
Contact:

Post by Thalius »

So before he continues throwing
smelly fish and whatnot at me
Oh don't bet on peace -- you missed him by 10 minutes with this post *g*. Besides theres a fresh supply of undead Hampster's ( somehow coming from Dracflamocs direction ... ) :lol:

Thalius
"In 3D there is never enough Time to do Things right,
but there's always enough Time to make them *look* right."
"psssst! i steal signatures... don't tell anyone! ;)"
User avatar
Joakim Christiansen
Addict
Addict
Posts: 2452
Joined: Wed Dec 22, 2004 4:12 pm
Location: Norway
Contact:

Post by Joakim Christiansen »

I was thinking about using an array like this that would return the correct string for the language used:

Code: Select all

L(#L_Press_any_button_to_continue)
I like logic, hence I dislike humans but love computers.
User avatar
Fangbeast
PureBasic Protozoa
PureBasic Protozoa
Posts: 4749
Joined: Fri Apr 25, 2003 3:08 pm
Location: Not Sydney!!! (Bad water, no goats)

But, but, but!!!

Post by Fangbeast »

But Freak, I love throwing dead fish at you!!! And I wasted an entire supply of decayed hamsters thrown all over Thalius just because you weren't there!!!! Oh, the inhumanity.

I'll throw some of the dead fish at Thalius but I was saving the good quality decayed ones for you (evil grin).

Seriously, this is great. Some of us wannabe coders like myself need better solutions than Lang(1285) and this is it and my brain couldn't work it out.

Shuddup Thalius, I did say "Brain", lemme alone damnit!!

Thank so much for this!!
Amateur Radio, D-STAR/VK3HAF
Thalius
Enthusiast
Enthusiast
Posts: 711
Joined: Thu Jul 17, 2003 4:15 pm
Contact:

Post by Thalius »

I did say "Brain", lemme alone damnit!!
mmmmmhmmmm ... Brrrraainnssssss !!!! :lol:
"In 3D there is never enough Time to do Things right,
but there's always enough Time to make them *look* right."
"psssst! i steal signatures... don't tell anyone! ;)"
User avatar
Fangbeast
PureBasic Protozoa
PureBasic Protozoa
Posts: 4749
Joined: Fri Apr 25, 2003 3:08 pm
Location: Not Sydney!!! (Bad water, no goats)

Post by Fangbeast »

Just spent 15 minutes adding freak's code and some strings, tested it and it went great.

I got all cocky and spent another 8 hours adding all the string conversions int he right places and fired up. Now I get this error and no idea why..

"Invalid memory Access"

at this command.

If LanguageGroups(Group)\Name$ <> Group$ ; Check if it is the same group as last time

arrrrghghghg.

I need sleep, not more debugging!!
Amateur Radio, D-STAR/VK3HAF
User avatar
Fangbeast
PureBasic Protozoa
PureBasic Protozoa
Posts: 4749
Joined: Fri Apr 25, 2003 3:08 pm
Location: Not Sydney!!! (Bad water, no goats)

I have no idea.

Post by Fangbeast »

I ripped out my strings and out them just in their own datasection and no crash.

Checked every string I had, still "Invalid memory access" and I am stumped.


**EDIT** It dies after this line:

Hits this okay >>> Global Dim NameOfDay.s(7)
Hits this okay >>> NameOfDay(0) = Language("DaysMonths","Sunday")

Goes to language manager to pull the string out and crashes at

" If LanguageGroups(Group)\Name$ <> Group$"

with Invalid memory access error
Amateur Radio, D-STAR/VK3HAF
freak
PureBasic Team
PureBasic Team
Posts: 5929
Joined: Fri Apr 25, 2003 5:21 pm
Location: Germany

Post by freak »

You did call LoadLanguage() before, right ?

You can comment out that whole if block and see if it works then
(it will be a bit slower but should work)
quidquid Latine dictum sit altum videtur
User avatar
Fangbeast
PureBasic Protozoa
PureBasic Protozoa
Posts: 4749
Joined: Fri Apr 25, 2003 3:08 pm
Location: Not Sydney!!! (Bad water, no goats)

Soitanlysoir wizard (Said in a 3 stooges voice)

Post by Fangbeast »

I made sure I did that. All of the multilanguage features that you designed are loaded at the very front of my code.

I made the compiler stop and stepped through the code and it always dies at that section. I am wondering if the problem is that it is being interfered with by the array functions there?

If I comment out that block of code, the entire multilanguage thing (all 450 strings) works perfectly! What could be the problem. A timing issue with arrays when calling other procedures.

Maybe there is a better way to avoid the below code but I am using it to create a date using English words instead of numbered dates in the program title field.

Code: Select all

;============================================================================================================================
; Setup day and month literal names
;============================================================================================================================

Global Dim NameOfDay.s(7)                                                     ; Fill an array with the names of the days (Terry Hough I think)

   NameOfDay(0)      = Language("DaysMonths","Sunday")
   NameOfDay(1)      = Language("DaysMonths","Monday")
   NameOfDay(2)      = Language("DaysMonths","Tuesday")
   NameOfDay(3)      = Language("DaysMonths","Wednesday")
   NameOfDay(4)      = Language("DaysMonths","Thursday")
   NameOfDay(5)      = Language("DaysMonths","Friday")
   NameOfDay(6)      = Language("DaysMonths","Saturday")

Global Dim DaysPerMonth(12)                                                   ; Fill an array on how many days per month there are

  For X = 0 To 11     
    DaysPerMonth(X) = 31 
  Next

  DaysPerMonth(1)   = 28
  DaysPerMonth(3)   = 30
  DaysPerMonth(5)   = 30
  DaysPerMonth(8)   = 30

  DaysPerMonth(10)  = 30

Global Dim NameOfMonth.s(12)                                                  ; Fill an array with the names of the months

   NameOfMonth(0)    = Language("DaysMonths","January")
   NameOfMonth(1)    = Language("DaysMonths","February")
   NameOfMonth(2)    = Language("DaysMonths","March")
   NameOfMonth(3)    = Language("DaysMonths","April")
   NameOfMonth(4)    = Language("DaysMonths","May")
   NameOfMonth(5)    = Language("DaysMonths","June")
   NameOfMonth(6)    = Language("DaysMonths","July")
   NameOfMonth(7)    = Language("DaysMonths","August")
   NameOfMonth(8)    = Language("DaysMonths","September")
   NameOfMonth(9)    = Language("DaysMonths","October")
   NameOfMonth(10)   = Language("DaysMonths",November")
   NameOfMonth(11)   = Language("DaysMonths",December")

Global Dim Years.s(7)                                                         ; Fill an array with the years

  Years(0)          = "2002"
  Years(1)          = "2003"
  Years(2)          = "2004"
  Years(3)          = "2005"
  Years(4)          = "2006" 
  Years(5)          = "2007"
  Years(6)          = "2008"

The datasection for that block looks like this. The only one giving such trouble.

Code: Select all

  Data$ "_GROUP_"       ,  "DaysMonths"
  Data$ "DMonday"       ,  "Monday"
  Data$ "DTuesday"      ,  "Tuesday"
  Data$ "DWednesday"    ,  "Wednesday"
  Data$ "DThursday"     ,  "Thursday"
  Data$ "DFriday"       ,  "Friday"
  Data$ "DSaturday"     ,  "Saturday"
  Data$ "DSunday"       ,  "Sunday"
  Data$ "Mjanuary"      ,  "January"
  Data$ "Mfebruary"     ,  "February"
  Data$ "Mmarch"        ,  "March"
  Data$ "Mapril"        ,  "April"
  Data$ "Mmay"          ,  "May"
  Data$ "Mjune"         ,  "June"
  Data$ "Mjuly"         ,  "July"
  Data$ "Maugust"       ,  "August"
  Data$ "Mseptember"    ,  "September"
  Data$ "Moctober"      ,  "October"
  Data$ "Mnovember"     ,  "November"
  Data$ "Mdecember"     ,  "December"

**EDIT** I just tried DEBUG Language("DaysMonths","Dmonday") and that also crashes it. So the array had bugger all to do with it, sorry about that.
Amateur Radio, D-STAR/VK3HAF
User avatar
Fangbeast
PureBasic Protozoa
PureBasic Protozoa
Posts: 4749
Joined: Fri Apr 25, 2003 3:08 pm
Location: Not Sydney!!! (Bad water, no goats)

**SOLVED**

Post by Fangbeast »

I'll give up for a while as I am not very well lately. When I get my strength back, will beat this up again.

**EDIT** After going through 15,00 lines of code, one line at a time (I didn't feel as sick today thank God), I found one call asking for a language string before the language was loaded.

ARRRGHGHG.

Timo, thanks so much for this, you have no idea how much help this is. It's worth me even fainting last night to get all this finished (and I am not kidding)
Amateur Radio, D-STAR/VK3HAF
User avatar
thyphoon
Enthusiast
Enthusiast
Posts: 327
Joined: Sat Dec 25, 2004 2:37 pm

Post by thyphoon »

Thanks Freak for this code. I use it and it work fine.
I add a little procedure to save current language in a file. it's more easy to translate
best regards (and excuse me for my bad english)

Code: Select all

Procedure SaveLanguage(FileName.s)
  Protected z.l, StringIndex.l
  If CreatePreferences(FileName.s)
    For z = 1 To NbLanguageGroups
      PreferenceGroup(LanguageGroups(z)\Name$)
      StringIndex = 0;
      For StringIndex = LanguageGroups(z)\GroupStart To LanguageGroups(z)\GroupEnd
        WritePreferenceString(LanguageNames(StringIndex), LanguageStrings(StringIndex))
      Next
    Next
    ClosePreferences()
  EndIf
EndProcedure
User avatar
Rook Zimbabwe
Addict
Addict
Posts: 4326
Joined: Tue Jan 02, 2007 8:16 pm
Location: Cypress TX
Contact:

Post by Rook Zimbabwe »

Timmo,

This code has me astounded... I can conceptualize it now but I could not see it before I started reading the code!

Your way was beter than mine! :D
Binarily speaking... it takes 10 to Tango!!!

Image
http://www.bluemesapc.com/
IdeasVacuum
Always Here
Always Here
Posts: 6425
Joined: Fri Oct 23, 2009 2:33 am
Location: Wales, UK
Contact:

Re: An easy to use solution for multilanguage programs

Post by IdeasVacuum »

This method has been a joy to use compared to what I have had to do when using C/C++ and others :D

Rather than have all the language options available from my program's GUI, I offer them at install time. The installer deposits the relevant prefs file and the program loads the file on start up. The PB support for Unicode is excellent for obvious reasons, but perhaps a less obvious one is that you can quickly check to see that the text fits the GUI well in every language.

I have hit an issue though, a tricky one. I have been unable to get the Hindi and Urdu languages to work. I initially use the Google/Yahoo translators to translate from English to other languages (fairly accurate but of course some not so good sentences are churned out too :| ). Works fine for everything, including multi-byte languages such as Chinese, Japanese and Korean, but Hindi and Urdu do not even display as characters in the Google translation, never mind in my prefs files (UTF-8 text files).

Has anyone hit this problem before and found a solution?
IdeasVacuum
If it sounds simple, you have not grasped the complexity.
c4s
Addict
Addict
Posts: 1981
Joined: Thu Nov 01, 2007 5:37 pm
Location: Germany

Re: An easy to use solution for multilanguage programs

Post by c4s »

Better create a new topic for your question instead of asking in the "Tips & Tricks" section - but hey everyone is learning :wink:

So to your question:
It's possible that you have to install the correct character set first. Well, wikipedia has something to say about this too: here.
Maybe it helps.
If any of you native English speakers have any suggestions for the above text, please let me know (via PM). Thanks!
IdeasVacuum
Always Here
Always Here
Posts: 6425
Joined: Fri Oct 23, 2009 2:33 am
Location: Wales, UK
Contact:

Re: An easy to use solution for multilanguage programs

Post by IdeasVacuum »

...ah yes, this is in the wrong section, sorry everyone. Brain is getting smaller by the minute! Dare I start again as a new topic? As a newbie I don't know the etiquette of the PB forum - on others, you'd be instantly flamed for multi-posting.

I have studied the Wiki info and it seems to suggest that Hindi at least should be covered by UTF-8, same as any other language, a step away from specific character sets/code pages.
IdeasVacuum
If it sounds simple, you have not grasped the complexity.
Post Reply