Category Archives: Debugger

Debugger related articles

A little bit of history about the debugger

In the last entry i mentioned that the debugger is compatible between 32bit and 64bit (you can debug a 64bit program with the 32bit debugger and vice versa). Why is that and why do we have 3 different debuggers anyway?

Its interesting actually: a lot of these things were not planned to be like that. In fact the whole debugger was not planned at all. It was one of these things that start as a simple idea and just grow into something huge. To understand more about that we have to go all the way back to when the current debugger was written. It was the year 2004, the current PB version was 3.91. The Windows debugger consisted of a small window which only showed the global variables and allowed stepping through the program. PureBasic for Linux was much worse. There was no IDE at all and the debugger did notning more than print the output of Debug statements to the command line.

At that time i decided to write the new IDE. Thats another project that vastly outgrew the original plans even on its initial release. The plan was to write a simple crossplatform IDE with the same set of features as the IDE for Windows at the time. Most of the IDE was working and we planned to include it in the 3.93 release. All it did for debugging on Linux was to make sure the program runs in a terminal window so you can see the debug output. To improve that, Fred gave me the source to the Linux debugger lib (just a collection of printf statements) so i could improve it.

The Windows debugger was just a library that got compiled into the executable which launched a new window in a thread. The idea was to do the same for Linux and just duplicate the feature set of the Windows debugger. Man am i glad that this didn’t work out! If that worked, we would probably be stuck with two very platform specific debuggers with probably a similary limited feature set as we had back then, and maybe no OSX debugger at all. Well, it turns out that Gtk doesn’t play so well with threads, especially when the main program doesn’t even know about the debugger thread.

New plan then: the debugger had to be an external program. Usually debuggers get access to their target programs by reading their memory directly, even patching the target program’s code as needed. Since this approach is quite platform specific we decided against that. After all, if we write a new debugger it should be available for Windows as well. So we still have the design of a special debugger library which gets compiled into the executable even today. This library allows the debugger to connect via pipes to exchange the needed data. This design requires a minimum of platform specific code and it works quite well.

Now we come to the reason for the three debuggers. The console debugger exists because I started this work on Linux with the given minimalistic library that just printed the output. This just grew to be the console debugger. To ease the testing i first implemented a new feature such as array display in console mode to make sure i got all the interaction with the compiler generated tables and stuff like that right and only as a last step i also added a GUI version of that. I still do it the same way today. So the main purpose of the console debugger is the development of the debugger itself. I also use it a lot to debug the IDE because I do not want to load that beast into the gui debugger every time.

I almost didn’t write the standalone gui debugger. After all, why add an extra window when you can access all the features from the IDE. The only reason i did it was to not kill the jaPBe project. Without any way to debug a program other than the IDE, people would have abandoned it fairly quickly. I was tempted though. After all, it would have been an easy way to get rid of the “competition”. 🙂 Looking back, the standalone debugger has proven useful in many more situations than that. For example this way programs can be debugged with elevated privileges on Vista or 7 without the need for the whole IDE to run at that level. Also it has again proven helpful in the development of the IDE and debugger itself. The codebase of the gui debugger is much smaller than that of the IDE so it is much easier to find a bug in there. Since most of the code is shared with the IDE we get that fixed as well.

The three debuggers also help in porting to a new platform like with the x64 port. As soon as the compiler works at least at a minimum, the first order of businesses is to get the console debugger up and running. This is then used to further debug the compiler output. When that works reliably enough the next step is to get the gui debugger and IDE running. Getting the IDE to run is quite the milestone for every port, as it is quite huge and makes use of a lot of different libraries.

Back to the communication between executable and debugger. As i said, it works through pipes. Its a simple package based binary protocol that sends a header structure and then data with a variable length. It hasn’t changed much since the first version although the code for handling the pipes has changed quite a lot. With the 64bit port we faced the question what to do with this communication protocol. Changing all values to 64bit would have been rather simple and the obvious choice. The reason why i choose not to do it was again for development purposes. The 64bit version was still quite unstable and having a stable 32bit debugger there to analyse it was a big help. At one point i was running the 32bit gui debugger to debug the 64bit gui debugger while debugging a 32bit executable. 🙂

So this is why the debuggers are compatible. The communication protocol keeps its 32bit header structure and the debuggers can adapt to the executable’s bit size when working with the contained data. Now with the 4.50 this design pays off once again because you can now actually load the 64bit compiler into the 32bit IDE for compiling and debugging. The final step was to make the PowerPC and x86/x64 communication compatible. This is something i just did recently because its a bit more of a change because the different byte-order has to be taken into account and so far there was just no need for this. Anyway, i did not do all this work just so you don’t have to use Rosetta on the IDE on OSX. Now that the debugger communication is compatible between all platforms, the way is finally open for one of my longer term goals: network debugging across all platforms.

Now with the next version we will have that as well: Network debugging across all platforms and processor types. I recently ran the debugger in a 32bit Linux VM, debugging a PPC executable on the Mac, while itself being debugged by another debugger instance on 64bit Vista. That just rocks! 🙂

To all those not really interested in this kind of stuff: you benefit from this as well. All these steps to new platforms and crossplatform compatibility have made the debugger more stable as a whole too, because many bugs become more apparent when you switch platforms frequently. So everybody wins.

Merry Christmas

I want to wish everybody in the PureBasic community a Merry Christmas and a happy New Year! Thank you for taking part in the making of PureBasic by giving feedback and helping find bugs. It was a fun year. Although there was only one major release this year its still quite the accomplishment in my opinion as it is one of the largest updates we did so far.

It has become somewhat of a custom for us to do a release around Christmas or New Year as a present. There won’t be anything like that this year.  The 4.40 isn’t that far behind, but calling it a Christmas-release is kind of wrong because it has been out there (in beta) for quite a while now. Well, i guess you can’t get it right every time. We’ll try to hit the spot again next time 🙂

Our plan is to go for a shorter release cycle with smaller releases in the future to reduce the waiting time for new features and bug fixes and also to keep the whole think more manageable. The next thing in line is a 4.41 bug fix release somewhen in January and then there is the 4.50 coming next. We are working on that already. I want to talk a bit about what i am currently working on so you can see what is ahead:

Data Breakpoints

Data breakpoints are like normal breakpoints, only they are not defined by a line number but by a condition. As soon as this condition becomes true the program will halt. This is a great feature for debugging a problem like a variable having an unexpected value but you have no idea where the change is made. So in this case you would just define ChristmasText$ <> “Merry Christmas” as the condition and the program will halt as soon as the value changes.

The conditions are evaluated by the debuggers expression parser which provides access to variables, arrays, lists etc but also some functions from the Memory library so you can even check for things inside memory buffers. Everything you can put behind an “If” should be possible as a data breakpoint.

Of course this feature comes at a price and that is speed. The conditions have to be re-evaluated for every executed line to notice all changes correctly. This is quite the slowdown. However, if you develop on a fairly recent machine it should not be a big deal at all. Also keep in mind that you won’t run your programs with data breakpoints all the time. They are only for tracking down specific problems and there speed is not really critical.

Multiple compiler support

Ever since we introduced the 64bit version in 4.30 people have requested that we merge the 32bit and 64bit version somehow so they can easily compile projects for both platforms. While a merge of the versions won’t happen any time soon (the versions are just very different internally and it would be a bit of a mess), there will now be a feature to provide a similar functionality. Basically there will be a list of “additional compilers” in the preferences where you can add the paths to other compilers installed on your system. These compilers are then available for selection in the compiler options. This will work with compilers all the way back to version 4.10.

So by adding the 64bit compiler to this list, you can easily build both 32bit and 64bit targets from the same IDE and from a single project. Even better, the debuggers of the x86 and x64 versions are compatible so as long as the compiler’s version number matches you can even debug poth from within the same IDE (if the version does not match, all debugging will be in the standalone debugger). The same is true now for the PowerPC and x86 versions on OSX. You can load the PowerPC compiler into the x86 IDE and compile/debug from there. This way you won’t have to run the entire IDE in Rosetta which is quite the slowdown.

Well, this is it for this little peak at the current development. Have a Merry Christmas every one!

New debugger features for userlibraries

A number of new functions were added to the DebuggerModule.h file in the SDK with 4.30 and since there is no documentation for them except the header file itself, i am going to explain them here briefly. They are mostly about localizing the debugger messages for the libraries but there are also some useful functions to validate the function parameters. I am using C to explain things even though most userlibraries are probably created using Tailbite, but C is what this stuff was written in/for and not everything can be directly translated to PB code (like the variable arguments functions). If you use TailBite you’ll have to use some tricks to make it all work. Note that all Message and Key parameters take ascii strings as input, even in unicode mode.

Error reporting:

In addition to the PB_DEBUGGER_SendError() which was there before, there is now also PB_DEBUGGER_SendWarning(). It works just like the SendError one, but issues a warning and not an error. Warnings are handled depending on the level that the user chooses. They can be ignored, just added to the log or handled like an error (program stop etc). The intention is to report anything that is probably wrong, but could also be valid input as a warning so the user is not constantly annoyed by errors that are none. The PB libraries currently report the following things as warnings:

  • Missing files for LoadImage(), LoadSprite(), etc. This often indicates that the user just mistyped the filename, but it is also perfectly valid to use these functions to try to load a nonexisting file and handle the failure properly.
  • Passing something that isn’t a PB procedure to a function that expects a callback. This helps catch cases like CreateThread(@Function, 0) where the ‘()’ is simply forgotten which is a very common mistake. It is however perfectly legal to pass a pointer to an external (maybe dll) function or label here, this is why this is not an error. A 0-pointer or a procedure pointer with the wrong prototype are however handled as errors, because those clearly are not valid input.
  • If OnError functions are used with enabled debugger, or with disabled OnError lines support.
  • On Linux: Gtk warnings/errors. Those mostly do not cause a program to fail, that is why they are just warnings. The debugger catches these since 4.30 because this way you can get a PB linenumber for the warnings which was not possible before.

There is the SendDebuggerWarning macro to keep the same style as the SendDebuggerError() before.

Parameter validation:

PB_DEBUGGER_CheckLabel() takes a pointer and returns true if the given pointer represents a label in the code. This does not detect labels in direct Asm, so there should be only a warning if this check fails.

PB_DEBUGGER_CheckProcedure() can check if a pointer is a procedure in the code and if so, wether the procedure matches a given prototype. This is a vararg function and takes the following arguments:

  • The pointer to check
  • The expected return type. The type is specified using the values of the PB type constants (#PB_Long, #PB_String etc). They are defined for C in the PureLibrary.h header.
  • The expected number of arguments to the procedure.
  • The type of each argument, the same way as the return type.

The possible returnvalues are listed in the DebuggerModule.h and indicated wether the function was found and if the prototype matches. Note that the automatic promoting of parameters into account. For example a word is always expanded to a long on the stack on x86, so passing PB_Word as expected parameter will also match a procedure that has a long at this parameter position (as the stack layout is the same). As mentioned above, a mismatching prototype or 0-pointer should be seen as an error, but a nonzero pointer that does not point to a PB procedure may still be valid, so better just issue a warning there.

PB_DEBUGGER_FileExists() was there before, and simply checks if the given file exists. The input to this has to be a unicode string in unicode mode unlike the other functions.

To make the list complete, there are also the PB_DEBUGGER_FullScreen, PB_DEBUGGER_Unicode and PB_DEBUGGER_Thread variables to indicate the settings/state of the program. There is also the threadlocal structure PB_DebuggerGlobals which currently only tells you if the program is in a StartDrawing() block. To get this info, use PB_Object_GetThreadMemory() from the Object.h with the PB_DEBUGGER_Globals variable to get the actual pointer to the threadlocal structure. All this stuff was there also before 4.30

Localisation of debugger messages: Using the common error messages

There is a list of very common error message for checking input parameters. These can be used within your library even without providing your own translation files. You can see the possible messages in the Debugger.Catalog file in the PureBasic folder in the [Common] section. The functions to access these are:

char *PB_DEBUGGER_GetCommonLanguage(char *Key, ...);
void  PB_DEBUGGER_SendCommonError(char *Key, ...);
void  PB_DEBUGGER_SendCommonWarning(char *Key, ...);

As you can see, they are vararg functions. They work in fact just like printf(). Most common messages have placeholders for strings/numbers and you have to pass arguments to fill them. The GetCommonLanguage() just returns the translated language string, the other two directly send a warning/error. There are again the SendCommonError, SendCommonWarning and GetCommonLanguage macros for better readability.

Localisation of debugger messages: Using custom messages

You can also provide your own messages and your own Catalog files. This involves the following steps:

First you have to include the information about your language data and the default strings with your debugger functions in the form of a PB_Language structure. Here is an example:

static PB_Language LanguageTable =
{
  "Libraries.catalog",
  "PB_Libraries",
  "OnError",
  0,
  {
    "DebuggerPresent", "The OnError library may not catch all errors with enabled debugger.",
    "LinesDisabled",   "The OnError lines support is not enabled.",
    "NoHandler",       "This function can only be called inside an eror handler.",
    "", "",
  }
};

These are the fields:

  • Default Filename for your catalog file (without path). This filename is checked first, but if it cannot be loaded, all Catalog files are searched for the correct data. So this is more of a hint.
  • The value of the “Application” key in the [LanguageInfo] group inside the Catalog file. This is what truely identifies your Catalog, so it has to be unique.
  • The name of the group inside the Catalog file in which the following Keys are located. Note that the group names of all libraries share the same namespace, so group names have to be unique too. Either use your library name here (as the PB libs do), or prefix your names with something unique. Multiple Libraries can share the same Catalog file (as the PB libs do), but one PB_Language structure has to correspond to one group. If two libraries refere to the same group, their values will overwrite each other.
  • The ‘Initialized’ field must be set to 0 initially. The debugger uses this to track wether it already loaded this language data or not.
  • Following is the list of Key-Value pairs with the default language. The list is terminated by two empty strings.

Now you can use this defined language (even without any actual Catalog files). Like with the common language, there are 3 functions to get the translations:

M_PBFUNCTION(char *) PB_DEBUGGER_GetLanguage(PB_Language *data, char *Key);
M_PBFUNCTION(void)   PB_DEBUGGER_SendTranslatedError(PB_Language *data, char *Key);
M_PBFUNCTION(void)   PB_DEBUGGER_SendTranslatedWarning(PB_Language *data, char *Key);

These are not varargs, and don’t support the printf-like syntax. The reason is that we access them mostly through macros, and support for varargs macros is not very good among C compilers. The first argument is a pointer to the PB_Language structure and the second one is the language key to look for. If you use C and name your PB_Language structure ‘LanguageTable’ as in the example above, then you can use the GetLanguage, SendTranslatedError and SendTranslatedWarning macros to leave out the pointer argument for simplicity.

If all this works, you can start creating translations in Catalog files. The format is the simple preference format. Just look at the Libraries.Catalog for an example. The [LanguageInfo] group and its “Application” and “Language” keys are mandatory, the other fields in [LanguageInfo] are for information only and not used by the debugger. Place your catalogs in the PB folder where the other ones are.

The actual language that will be used is determined by the compiler or ide, not the debugger. You can set it for the commandline compiler with the /LANGUAGE switch. The ide sets the language to the one it uses itself.