Author Archives: freak

PureBasic 4.31 beta1 released

A new PureBasic release has entered the beta stage. This is a bugfix-only release, there are no new features. The purpose of this release is to give those that only switch to newer releases once the beta-period is over a stable base to work with while the v4.40 release is going through the alpha and beta stages. We tried to focus on the major bugs in the 4.30 release and skip minor issues. Note that some even bigger bugs could not be resolved without major changes to the code and so we postponed these to the next release as well.

You can download it as usual on your user account.

Note: Since there are no new features and we want to get our focus back on the v4.40 release soon, this beta-period will be very short (maybe only one beta release). So if you reported bugs that are now marked as fixed, please test soon to verify that things work as they should now.

The v4.40 release is still a bit away, but we are working very hard on it. The list of new things is quite long again and we still need to work a few things out. Stay tuned.

Alpha-channel improvements

Alpha-channel improvements on Windows

Alpha-channel improvements on Windows

This one is for all those who are quick to complain when a new feature is added that they themselves don’t need. As it turns out, sometimes changes made in areas you may not care about at all can trigger changes in places that you may find useful after all. One of these areas is the IDE.

It all started very small and insignificant: Inspired by the recently reanimated thread here, i decided to add the ability to easily switch out the IDE provided icon set for a custom icon theme. How hard could this be? It was a rather small thing really, and quite quickly implemented in the IDE code. Here is where you would say: “Why do i need themes? Better focus on something else!”, well better read on…

Of course i had to test this, and for that purpose i created a theme with the Silk icons also mentioned in that thread (the result is quite nice btw, and will be included in the PB package). They use a lot of semi-transparency and converting them to icons (non 32bit, for older Windows versions) just didn’t look good. So i tried loading and using the png files directly. Well ups, the PB ImageMenu doesn’t support the alpha-channel. In fact, it doesn’t even support non-icon images at all. So i’ll just change the drawing code to support alpha-channel images, how hard could this be?

Those who know me a little closer know that i tend to think way too far ahead with these kinds of things. What good is alpha-channel support in the menu when not even the ImageGadget can properly display alpha-channel images? So lets just add it there as well, and then there is the ButtonImageGadget too. Seriously, how hard could this be? 🙂 (turns out this one was actually a tough one. The nice solution i had worked up for Vista/XP just didn’t want to work on the older Windows versions)

Alpha-channel improvements on Linux

Alpha-channel improvements on Linux

The line doesn’t end there though. There is also the Linux version. The Image lib on Linux didn’t support the full alpha-channel so far, it only supported full transparent pixels through a 1-bit mask. And there i am, right back at the beginning with a set of ugly looking icons in the menu. So, lets just add the full alpha-channel to the Linux Image lib. How hard… well, this really is a bigger deal. It means a rewrite of the entire lib, plus large parts of the 2DDrawing lib and every other command that deals with images. We had this one on our list for quite a while already though because the lib still uses a GdkPixmap to represent an image and not a GdkPixbuf which is the much better choice. So i went ahead and rewrote these commands as well, and it was worth it. The lib can now use the full alpha-channel and since the Gtk widgets all bring support for GdkPixbu’s, every Gadget that deals with images can use it too. (for example ListIcon, Tree etc). Oh and while i was at it, i dived a bit into SDL and implemented DrawAlphaImage() for Linux too. 🙂

This brings a new “problem”. With Linux supporting the alpha-channel in every GUI element, the Windows version is lacking behind again because it still requires Icon files for some GUI things. That cannot stand! XP and Vista actually solve this quite well with their 32bit icon support so this wasn’t so hard to do. Older versions however don’t support that, so to get the best possible result an alpha-channel image is converted to an icon with a mask to get at least some transparency. You can now pass non-icon images to all functions that take an Image on Windows too and it will be converted to an icon as needed.

For once, the Mac version was the easy target in all this because it already came with full alpha-channel support from the start. It only lacked the ImageMenu, but i added that as well.

So, with v4.40 you will be able to use alpha-channel images for all GUI elements and it will look good on all OS. No more converting to ico to get transparency on Windows! If you plan to target Windows versions older than XP, you might want to avoid semi-transparent pixels though for things like toolbars so it will look good also without the 32bit icon support. If you plan to target Windows 95 or NT4, you are out of luck as they don’t have alpha-channel support at all. 😛

Not so useless after all, eh?

Using PureBasic (32bit) on 64bit Linux

This has been asked a couple of times. Until now i wasn’t much help there because i never tried this myself. So now i tried it and had some success. I was able to compile the PureBasic IDE, which means that pretty much all non-multimedia libs must be working. If you don’t want to wait for PureBasic 64bit for Linux (planned for 4.40) then here is a step by step guide.

There is a guide for Ubuntu 8.10 and OpenSUSE 11.1 64bit (always starting from a clean installation). If you have a slightly different version of one of these distributions then it may work for you as well with some luck. If you have a different distribution, then read this anyway. There are some tips on the bottom on how you could make this work for your distribution as well.

PureBasic on Ubuntu 8.10 64bit

First download and unpack the PureBasic package. Then open a console and set up the environment variables for the compiler and try running it “pbcompiler -h”. If you started from a clean install, then not even this will work (you will get a weird error like “No such file or directory”). So we first need the runtime environment for 32bit programs:

sudo apt-get install ia32-libs

Note that this installs a lot of stuff (120Mb). After this is completed, the compiler should at least be able to run. At this point you should also be able to run the IDE. But even trying to compile an empty program will cause linker errors. So what we need next is the basic gcc and libc6 support for compiling 32bit programs:

sudo apt-get install gcc-multilib libc6-i386 libc6-dev-i386

After that, you should be able to compile your first (console only) programs, and also use the PureBasic debugger.

The next big thing are Gtk and SDL. Unfortunately there are no special 32bit development versions of these packages available (something like “libgtk2.0-dev-i386”), so we have to use some tricks. To compile with PB, we actually only need two things: The library files and the properly working pkg-config sdl-config tools to actually find them (and tell the linker which libraries to link).

Lets start with the configuration tools: here we can actually use the ones that come with the regular 64bit packages, so just install those:

sudo apt-get install libgtk2.0-dev libsdl1.2-dev

If you try to compile a program with a MessageRequester() before and after this step, you will notice that the linker errors are reduced from a huge pile of unresolved symbols to just not finding a compatible library. So all we need now is some compatible library files. It turns out that all the needed libraries are already installed with the “ia32-libs” package, because you can use a dynamic library (.so) for linking as well (the linker will automatically generate a static lib from that). The only thing that is missing really are some symbolic links to help the linker find those dynamic ones, thats all.

I found a nice script here that creates these links and extended it by a few more links needed for PureBasic. Put the following into a textfile and execute it as root (“sudo sh myfile.sh”):

#!/bin/sh
cd /usr/lib32
for lib in gio-2.0 gdk-x11-2.0 atk-1.0 gdk_pixbuf-2.0 \
           pangocairo-1.0 pango-1.0 pangoft2-1.0 \
           gobject-2.0 gmodule-2.0 glib-2.0 gtk-x11-2.0; do
  ln -s -f lib$lib.so.0 lib$lib.so
done
ln -s -f libcairo.so.2 libcairo.so
ln -s -f libfreetype.so.6 libfreetype.so
ln -s -f libz.so.1 libz.so
ln -s -f libfontconfig.so.1 libfontconfig.so
ln -s -f /usr/lib32/libX11.so.6 /usr/lib32/libX11.so
ln -s -f /usr/lib32/libXrender.so.1 /usr/lib32/libXrender.so
ln -s -f /usr/lib32/libXext.so.6 /usr/lib32/libXext.so
ln -s -f libgthread-2.0.so.0 libgthread-2.0.so
ln -s -f libSDL-1.2.so.0 libSDL.so
ln -s -f libstdc++.so.5 libstdc++.so

Thats it! Now most libraries should work fine. The IDE compiled fine with this setup. As far as i understand the page above, this step will actually not be needed for the next stable Ubuntu release.

Some PureBasic libs that have special dependencies will still not work however. The commands from the following libraries will compile/link fine, but not work with this setup:

  • Sound, SoundPlugin, Module – I am not sure why these do not work. Could be because i am running VMWare and have no tools installed yet. Somebody else will have to try this.
  • All 3D Engine related libraries – I did not look deeper into why the engine does not load. Maybe another time. It actually starts loading, but you never see a screen.
  • the WebGadget() – This does not even work out of the box on some 32bit systems. I did not bother looking deeper into that.

The following Libraries will not even link. The reason is simply that the “ia32-libs” package does not include 32bit versions of them. To get them to work you would have to compile them manually in 32bit mode. Maybe i will try that another time.

  • PrinterOutput() – needs libgnomeprint2.2
  • Movie lib – needs libxine
  • Database lib with ODBC – needs libiodbc2 (Note: SQLite works fine as it is included in PB)
PureBasic on OpenSUSE 11.1 64bit

I started from a clean installation with Gnome desktop and no other packages added in the setup.

Open the “Software Management” in Yast and install the following packages:

  • gcc-32bit
  • gtk2-devel
  • sdl-devel-32bit
  • libgnomeprint-devel
  • libgnomeprint-32bit
  • libgnomeprintui-devel
  • libgnomeprintui-32bit
  • libxine-devel
  • libxine1-32bit

Then copy the code below to a text file and run it in a shell as root (“sudo sh myfile.sh”):

#!/bin/bash
cd /usr/lib
for lib in gio-2.0 gdk-x11-2.0 atk-1.0 gdk_pixbuf-2.0 \
           pangocairo-1.0 pango-1.0 pangoft2-1.0 \
           gobject-2.0 gmodule-2.0 glib-2.0 gtk-x11-2.0; do
  ln -s -f lib$lib.so.0 lib$lib.so
done
ln -s -f libcairo.so.2 libcairo.so
ln -s -f libfreetype.so.6 libfreetype.so
ln -s -f libfontconfig.so.1 libfontconfig.so
ln -s -f libstdc++.so.6 libstdc++.so
ln -s -f /lib/libz.so.1 /lib/libz.so
ln -s -f /lib/libgcc_s.so.1 /lib/libgcc_s.so
ln -s -f libgthread-2.0.so.0 libgthread-2.0.so
ln -s -f libgnomeprint-2-2.so.0 libgnomeprint-2-2.so
ln -s -f libart_lgpl_2.so.2 libart_lgpl_2.so
ln -s -f libxml2.so.2 libxml2.so
ln -s -f libgnomeprintui-2-2.so.0 libgnomeprintui-2-2.so
ln -s -f libgnomecanvas-2.so.0 libgnomecanvas-2.so
ln -s -f libxine.so.1 libxine.so
ln -s -f libpng12.so.0 libpng.so

This should do the trick to allow PB to compile. Note that unlike on Ubuntu, the Movie lib and PrinterOutput() will work just fine here. The other mentioned library problems are the same as on Ubuntu though.

PureBasic on other 64bit Linux distributions

Ok, i cannot do this for all distributions, but the steps on your Linux of choice should be fairly similar, just with slightly different package/filenames:

  1. Install the 32bit runtime environment (sometimes this is preinstalled). You know you succeeded when the IDE and compiler can be started.
  2. Install what is needed to build basic 32bit programs. This is usually a special gcc package and a special libc6-dev package. You know you succeeded if you can compile an empty sourcecode without errors.
  3. Install the required libraries. If they exist as a 32bit-devel package then that is perfect. If they only exist as a 32bit runtime version (maybe included in a larger 32bit package) then install that plus the 64bit devel package. Then add any missing symbolic links in your 32bit lib directory. Usually the linker looks for a file like “libSDL.so” and the folder will contain a “libSDL1.2.so.0” or similar. (usually just with the extra .0) Just add a link as the scripts above do. If there is not even a 32bit runtime package for a library then you can only try compiling them manually. Use a small testcode like a MessageRequester() and try the above until the linker no longer complains about any libraries.

The required packages (all as devel version, Gtk and SDL are the most important):

  • libgtk2.0
  • libsdl1.2
  • libstdc++ (for ScintillaGadget)
  • libgnomeprint2.2 and libgnomeprintui (for PrinterOutput())
  • libxine (for Movie lib)
  • libiodbc (for ODBC database)

Thats pretty much it. I hope you can get this working on your system. If there are more problems just ask. I would not call myself a Linux expert but i will try to help if possible. If you get PureBasic working on another distribution, please publish your exact steps somewhere so others can benefit too.

New debugger features for userlibraries

A number of new functions were added to the DebuggerModule.h file in the SDK with 4.30 and since there is no documentation for them except the header file itself, i am going to explain them here briefly. They are mostly about localizing the debugger messages for the libraries but there are also some useful functions to validate the function parameters. I am using C to explain things even though most userlibraries are probably created using Tailbite, but C is what this stuff was written in/for and not everything can be directly translated to PB code (like the variable arguments functions). If you use TailBite you’ll have to use some tricks to make it all work. Note that all Message and Key parameters take ascii strings as input, even in unicode mode.

Error reporting:

In addition to the PB_DEBUGGER_SendError() which was there before, there is now also PB_DEBUGGER_SendWarning(). It works just like the SendError one, but issues a warning and not an error. Warnings are handled depending on the level that the user chooses. They can be ignored, just added to the log or handled like an error (program stop etc). The intention is to report anything that is probably wrong, but could also be valid input as a warning so the user is not constantly annoyed by errors that are none. The PB libraries currently report the following things as warnings:

  • Missing files for LoadImage(), LoadSprite(), etc. This often indicates that the user just mistyped the filename, but it is also perfectly valid to use these functions to try to load a nonexisting file and handle the failure properly.
  • Passing something that isn’t a PB procedure to a function that expects a callback. This helps catch cases like CreateThread(@Function, 0) where the ‘()’ is simply forgotten which is a very common mistake. It is however perfectly legal to pass a pointer to an external (maybe dll) function or label here, this is why this is not an error. A 0-pointer or a procedure pointer with the wrong prototype are however handled as errors, because those clearly are not valid input.
  • If OnError functions are used with enabled debugger, or with disabled OnError lines support.
  • On Linux: Gtk warnings/errors. Those mostly do not cause a program to fail, that is why they are just warnings. The debugger catches these since 4.30 because this way you can get a PB linenumber for the warnings which was not possible before.

There is the SendDebuggerWarning macro to keep the same style as the SendDebuggerError() before.

Parameter validation:

PB_DEBUGGER_CheckLabel() takes a pointer and returns true if the given pointer represents a label in the code. This does not detect labels in direct Asm, so there should be only a warning if this check fails.

PB_DEBUGGER_CheckProcedure() can check if a pointer is a procedure in the code and if so, wether the procedure matches a given prototype. This is a vararg function and takes the following arguments:

  • The pointer to check
  • The expected return type. The type is specified using the values of the PB type constants (#PB_Long, #PB_String etc). They are defined for C in the PureLibrary.h header.
  • The expected number of arguments to the procedure.
  • The type of each argument, the same way as the return type.

The possible returnvalues are listed in the DebuggerModule.h and indicated wether the function was found and if the prototype matches. Note that the automatic promoting of parameters into account. For example a word is always expanded to a long on the stack on x86, so passing PB_Word as expected parameter will also match a procedure that has a long at this parameter position (as the stack layout is the same). As mentioned above, a mismatching prototype or 0-pointer should be seen as an error, but a nonzero pointer that does not point to a PB procedure may still be valid, so better just issue a warning there.

PB_DEBUGGER_FileExists() was there before, and simply checks if the given file exists. The input to this has to be a unicode string in unicode mode unlike the other functions.

To make the list complete, there are also the PB_DEBUGGER_FullScreen, PB_DEBUGGER_Unicode and PB_DEBUGGER_Thread variables to indicate the settings/state of the program. There is also the threadlocal structure PB_DebuggerGlobals which currently only tells you if the program is in a StartDrawing() block. To get this info, use PB_Object_GetThreadMemory() from the Object.h with the PB_DEBUGGER_Globals variable to get the actual pointer to the threadlocal structure. All this stuff was there also before 4.30

Localisation of debugger messages: Using the common error messages

There is a list of very common error message for checking input parameters. These can be used within your library even without providing your own translation files. You can see the possible messages in the Debugger.Catalog file in the PureBasic folder in the [Common] section. The functions to access these are:

char *PB_DEBUGGER_GetCommonLanguage(char *Key, ...);
void  PB_DEBUGGER_SendCommonError(char *Key, ...);
void  PB_DEBUGGER_SendCommonWarning(char *Key, ...);

As you can see, they are vararg functions. They work in fact just like printf(). Most common messages have placeholders for strings/numbers and you have to pass arguments to fill them. The GetCommonLanguage() just returns the translated language string, the other two directly send a warning/error. There are again the SendCommonError, SendCommonWarning and GetCommonLanguage macros for better readability.

Localisation of debugger messages: Using custom messages

You can also provide your own messages and your own Catalog files. This involves the following steps:

First you have to include the information about your language data and the default strings with your debugger functions in the form of a PB_Language structure. Here is an example:

static PB_Language LanguageTable =
{
  "Libraries.catalog",
  "PB_Libraries",
  "OnError",
  0,
  {
    "DebuggerPresent", "The OnError library may not catch all errors with enabled debugger.",
    "LinesDisabled",   "The OnError lines support is not enabled.",
    "NoHandler",       "This function can only be called inside an eror handler.",
    "", "",
  }
};

These are the fields:

  • Default Filename for your catalog file (without path). This filename is checked first, but if it cannot be loaded, all Catalog files are searched for the correct data. So this is more of a hint.
  • The value of the “Application” key in the [LanguageInfo] group inside the Catalog file. This is what truely identifies your Catalog, so it has to be unique.
  • The name of the group inside the Catalog file in which the following Keys are located. Note that the group names of all libraries share the same namespace, so group names have to be unique too. Either use your library name here (as the PB libs do), or prefix your names with something unique. Multiple Libraries can share the same Catalog file (as the PB libs do), but one PB_Language structure has to correspond to one group. If two libraries refere to the same group, their values will overwrite each other.
  • The ‘Initialized’ field must be set to 0 initially. The debugger uses this to track wether it already loaded this language data or not.
  • Following is the list of Key-Value pairs with the default language. The list is terminated by two empty strings.

Now you can use this defined language (even without any actual Catalog files). Like with the common language, there are 3 functions to get the translations:

M_PBFUNCTION(char *) PB_DEBUGGER_GetLanguage(PB_Language *data, char *Key);
M_PBFUNCTION(void)   PB_DEBUGGER_SendTranslatedError(PB_Language *data, char *Key);
M_PBFUNCTION(void)   PB_DEBUGGER_SendTranslatedWarning(PB_Language *data, char *Key);

These are not varargs, and don’t support the printf-like syntax. The reason is that we access them mostly through macros, and support for varargs macros is not very good among C compilers. The first argument is a pointer to the PB_Language structure and the second one is the language key to look for. If you use C and name your PB_Language structure ‘LanguageTable’ as in the example above, then you can use the GetLanguage, SendTranslatedError and SendTranslatedWarning macros to leave out the pointer argument for simplicity.

If all this works, you can start creating translations in Catalog files. The format is the simple preference format. Just look at the Libraries.Catalog for an example. The [LanguageInfo] group and its “Application” and “Language” keys are mandatory, the other fields in [LanguageInfo] are for information only and not used by the debugger. Place your catalogs in the PB folder where the other ones are.

The actual language that will be used is determined by the compiler or ide, not the debugger. You can set it for the commandline compiler with the /LANGUAGE switch. The ide sets the language to the one it uses itself.

Parallel frenzy

Ever since i got a QuadCore, i have been looking for ways to make it simple for people to make use of this extra power in today’s computers. Only a handful of programs available today can really use this power. Thread programming is not easy. It is hard to do it without introducing hidden bugs, and as i found out myself it is even harder to do it in a way that fully utilizes the available resources and does not spend all its time waiting on synchronisation points. Writing a piece of code that can max out 4 cores and still perform something useful is quite a challange 🙂 And if trends continue, even 8 core machines are not unimaginable.

So how do we make this simpler ? Even if writing a fully threaded program remains a challange, maybe we can introduce commands that make it simpler to parallelize smaller portions of the program. The most obvious place to start are the sort commands, as these contain CPU intensive tasks which can be parallelized well. So i wrote some parallel sort routines. The implementation can scale up to as many cores as are available. So in 4.40, at least the sorting commands can utilize the full power of the CPU. I wrote a small framework to make parallelizing PB commands easier, so other commands may benefit as well. (although most commands are just not CPU intensive enough to make this useful).

SortArray() vs. parallel SortArray()

Here you see some test results for SortArray() on my QuadCore (showing the sort times for an array of longs). I tested the original version, as well as a 2 and a 4 thread run of the command to see how it would perform on a DualCore or a QuadCore. The
improvement of the parallel variants is quite constant. The 2 thread test took on average 52% of the execution of the serial version while the 4 thread version takes on average 31% of the time. The improvement actually increases a little bit the larger the array gets.

Even while this is a good improvement, why not more ? After all, 31% is only about a factor of 3 while we would expect a factor of 4 in the 4 thread variant. Some of this is due to the management overhead, but there is also a part which simply cannot be parallelized. Not all threads can get to work immediately. The first partition step has to be done in one thread, before 2 threads can get to work on the second step of the sorting (on each half of the data). Only in the third level (and all subsequent ones) can 4 threads even find work. This serial part is always there, and it limits the improvement that can be made by parallelizing the other part, no matter how many cores are present (this is called Amdahl’s law).

SortList() vs. parallel SortList()

The results for SortList() are not as good. Here the results vary depending on the list size, with a best of 55% but average of 71% for the 2 thread test and a best of 47% and average of 67% for the 4 thread version. Parallelizing a mergesort takes some more overhead than quicksort, as there is some waiting time involved until two smaller sort steps can be merged into a larger one. Since the larger sort steps come last and depend on the smaller ones (unlike with quicksort) this cannot be avoided. Also since a list is spread more through memory than an array, cache effects will play a role too. Still, both the results for SortList() and SortArray() are a good improvement in my opinion.

How hard it is to write code that fully utilizes the CPU power can be seen on the parallel SortList(). I went through a number of different implementations for this command. Trying to come up with more compact versions (which use less code or less memory reads/writes) only to find out that they performed worse than the original serial version due to cache effects. For example i found a mergesort code on the net witch uses only a handful of variables and a minimum amount of read operations. I implemented it in asm, using exclusively registers to store the variables. The instincts tell us that this is an optimal solution. But because of the way the code reads the list makes it read and overwrite values in the cache on every run. This makes it run slow as hell for large lists, even if executed on 4 cores. The key word here is “locality“. Memory locations which are close together must be processed together so that the values are still in the cache (or at least in many memory for large lists). The current implementation performs quite well in this respect.

By the way, some of this locality optimisation is already present in the 4.30 release. One of the reason why i wrote the block allocator about which i talked in my first blog post was to make sure that list elements are closer together. This way both the sort, but also a normal walk through the list can be executed in a more cache-friendly way. The block allocation was working in time for 4.30, the parallel stuff did not make it to that release.

We have some more new things planned related threads and parallel programming which should make it simpler to write threaded programs in PB, but the rest is a surprise… 😉

PureBasic 4.30 final is out

It is now official. PureBasic 4.30 has reach the final status. It is available for download for all registered users on their personal download account on http://www.purebasic.com/ .

The list of new features is quite impressive:

16th December 2008 : Version 4.30

- Added: Full 64 bits version of PureBasic (Windows)
- Added: MacOS X x86 support
- Added: 'JPEG 2000' library
- Added: 'Node' library
- Added: 'Gadget3D' library
- Added: 'Window3D' library
- Added: 'Sound3D' library
- Added: Full unicode support to the engine3d
- Added: MacOS X support for engine3d
- Added: Absolute/relative rotation for all engine3d rotate commands
- Added: Absolute move for engine3d physic body
- Added: EntityID(), EntityLookAt()
- Added: CameraProjectionX(), CameraProjectionY(), WorldDebug(), Sun()
- Added: MousePick(), PointPick(), PickX/Y/Z(), CreateWater(), AntialiasingMode()
- Added: CatchModule(), ArraySize(), WindowBounds(), DesktopX(), DesktopY()
- Added: FinishDatabaseQuery(), DatabaseID()
- Added: CreateSemaphore(), FreeSemaphore(), SignalSemaphore(), WaitSemaphore(), TrySemaphore()
- Added: '.i' native type for 32/64 bits integer size
- Added: Read/WriteInteger(), Read/WritePreferenceInteger(), PeekI(), PokeI()
- Added: Language support for compiler and debugger
- Added: Style parameter to FontRequester()
- Added: Support for 'And/Or' in CompilerIf statements
- Added: #PB_Event_MinimizeWindow, #PB_Event_MaximizeWindow, #PB_Event_RestoreWindow
- Added: #PB_Ignore support to AddStatusBarField()
- Added: #PB_Explorer_DisplayMode to ExplorerListGadget()
- Added: Multiple joystick support
- Added: #PB_Compiler_FilePath compiler constant 

- Enhanced: Major rework of OnError library (crossplatform)
- Enhanced: InitScintilla(), parameter is now optional
- Enhanced: Complete debugger rewrite for support of the new platforms 

- Optimized: XML library to handle very big files quickly
- Optimized: LinkedList library to use memory efficiently 

- Changed: 'Read' keyword now requiers a type (Read.l, Read.q etc.) to avoid 64 bits migration problems
- Changed: CountList() to ListSize()
- Changed: ClearGadgetItemList() to ClearGadgetItems()
- Changed: CameraProjection() to CameraProjectionMode()
- Changed: AddElement() and such now return a pointer to the start of data
- Changed: ComboBoxGadget() height is now the real gadget height
- Changed: #Long, #Byte, #Word etc. to #PB_Long, #PB_Byte, #PB_Word etc. for consistency
- Changed: RotateEntity(), RotateCamera() and RotateBillboard() x,y rotation axis
- Changed: CameraFOV() angle from radian to degree
- Changed: Array and list parameter now requiers the 'List' or 'Array' keyword 

- Removed: StrQ(), HexQ(), BinQ(), ValQ()
- Removed: ChangeListIconDisplay(), replaced by a gadget attribute
- Removed: ButtonImageGadget() backward compatibility
- Removed: CreateGadgetList(), now automatically done in OpenWindow()
- Removed: CopyTexture(), TextureOutput()
- Removed: Removed CPU monitor from the IDE, as all OS provide the same functionality
- Fixed: DX9 subsystem on Windows
- Fixed: Many compiler and libraries related bugs

Have a lot of fun with this new version!

Debugging tips – Part 2: Stack problems

Another common problem that the debugger does not really identify are stack problems. Fortunately, other than the heap problems discussed before, these are not hard to narrow down.

Symptoms:
  • A piece of code works on the main level, but fails when called inside a procedure or inside a Select block
  • The crash is usually when leaving the procedure (ProcedureReturn or EndProcedure)
  • Usually a call to a Dll or imported function is involved
Reason:

The x86 processor has several different so called calling conventions which define how a function call is made (were the arguments go and who is responsible for cleaning up after the call). In fact everybody can invent their own convention for this, and quite a number of compilers do this for performance reasons. (PureBasic itself does it for Library functions written in ASM. They use the stdcall convention, but receive their first argument in the EAX register instead of the stack). When calling an external function, the calling convention must match or there will be problems like the one we have here.

The most popular conventions on x86 are cdecl and stdcall. cdecl is the standard used by the C language, stdcall is most used by Windows dlls. PureBasic uses stdcall by default in most places. They both provide the function arguments on the stack in reverse order. The only main difference is in the fact that cdecl requires the caller to remove the arguments from the stack and stdcall requires the function to do this before returning (there is also a difference in symbol naming, but thats not relevant here). So if the caller assumes the wrong calling convention then either both try to remove the parameters from the stack or nobody does it. In both cases the stack is not the same after the function call than before, and this is a big problem.

Here we get to the reason why it works outside of a procedure and not inside. A call to a procedure stores the address where execution continues after the procedure on the stack as well. If a function call inside the procedure leaves the stack in a mess, then the ProcedureReturn cannot find the address to continue and uses a wrong value instead which leads to a crash. Outside of a procedure, nobody will notice the stack mess, as the stack is only used for function calls.

Solution:
  • PB provides support for both stdcall (the default) and cdecl calling conventions (the functions/keywords with a C in the name)
  • If you used CallFunction() or CallFunctionFast() try CallCFunction() or CallCFunctionFast() instead. (these are the cdecl versions of the same function)
  • If you used Prototype, try PrototypeC instead.
  • If the crash happens in a callback procedure which you passed to an external function, try defining the callback with ProcedureC instead.
  • Also check if the number of arguments and the argument types match, as this can lead to the same problem. (remember that CallFunction() and CallFunctionFast() default to the long type, so for quads and doubles you have to use prototypes for a correct call.)

Some rules of thumb:

  • On Windows all Dlls that come with Windows use the stdcall convention. Most other Dlls follow this model, so there are only a few that use cdecl. With static libraries (used with Import/ImportC), the division is not so clear. Just try it.
  • On Linux / Mac OSX, the dominating calling convention is cdecl. So pretty much every library you use will require the C type of function.

Stack problems can also arise when InlineASM code is used and the stack is changed. Those that use this should know what they are doing though. The x64 and PowerPC processors do not have the calling convention problems by the way, as the force one common calling convention on everybody. So a parameter mismatch is the most probable explanation here.

A sidenote: Window callbacks

On Windows, another condition which can cause a crash on the ProcedureReturn keyword is the window callback. This is not a stack issue then though. What it means is that the crash happened somewhere in the PureBasic event processing that happens after your window callback returns. The debugger just shows the ProcedureReturn line because this was the last correctly executed line in non-library code.

What this probably means is that you did something with Windows API calls that conflicts with the way PureBasic handles its events. This may be in the window callback directly, or by modifying some Gadget/Window related data that PureBasic is expecting to remain unchanged. If you have no clue what causes this, just post it on the forum. There are a lot of capable people around that can help with these things.

Debugging tips – Part 1: Heap corruption

If you are wondering why this blog is silent for so long, its because we are in the beta phase of 4.30, and writing about bugfixing and documentation work is just boring.

I want to talk a bit about the most common problems i have seen on the forums where the PureBasic debugger is of not much help and which therefore leave most users quite confused. The usual reaction is to post it as a bugreport. So in the future, if somebody posts a bugreport about AllocateMemory(), you can point him to this article and it should explain everything 🙂

Symptoms:
  • A crash at AllocateMemory() or FreeMemory() even though the given input is valid.
  • A crash on a simple string assignment
  • Modifying the code in seemingly unrelated places makes the problem go away.
Reason:

First of all: AllocateMemory() is never the problem. Its a direct wrapper to the HeapAlloc() API function and also a heavily used function. If it had a bug we would know by now. What you have gotten yourself into here is what is called a heap corruption. You destroyed part of the data Windows uses to manage allocated memory.

When Windows allocates memory, it keeps a data structure to manage the allocated memory (usually 12bytes on 32bit Windows). This data structure is normal, writable memory which means that you will not get any access error when accidentally writing over it. It is nowhere specified where this data is kept, but it is a fact that it sometimes ends up right after your allocated memory buffer. Now, if you happen to write over the end of this buffer by just a few bytes, you destroy the heap structure without getting any error. The crash only happens later when another attempt to allocate or free memory causes Windows to examine this piece of heap data and crash due to an invalid pointer. This fact, that the cause of the problem and the actual crash are in different places makes this kind of bug so hard to debug.

Why does this not happen always when you overwrite a buffer ?

Getting the heap data right at the end of the allocated buffer is a very rare condition. Windows often rounds the allocated buffer size up to page boundaries or for alignment purposes, so you often get more memory than you asked for, which makes a small overwrite have no effect. Another scenario is that the memory after your allocated buffer is simply marked as invalid, in which case you get an error when trying to write to it. Which of these scenarios actually happens depends on the sequence of memory allocations done by your program, which means that if you comment another totally unrelated program part, you may change the allocation sequence and the problem seemingly disappears. (Note that in this case only the symptom disappears, not the problem of writing over the end of a buffer.)

Solution:

As we saw above, the problem is not where the crash is. Fortunately, Windows provides a function to check if the heap structures. Below is a piece of code that can be used to check the most used memory heaps in PureBasic: (this works in PureBasic Windows 32bit and 64bit)

Procedure _TestHeaps(File$, Line)
  Protected StringHeap, MemoryBase, MemoryHeap

  CompilerIf #PB_Compiler_Processor = #PB_Processor_x86
    !extrn _PB_StringHeap
    !extrn _PB_Memory_Heap         

    !mov eax, dword [_PB_StringHeap]
    !mov [p.v_StringHeap], eax
    !mov eax, dword [_PB_MemoryBase]
    !mov [p.v_MemoryBase], eax
    !mov eax, dword [_PB_Memory_Heap]
    !mov [p.v_MemoryHeap], eax
  CompilerElse
    !extrn PB_StringHeap
    !extrn PB_Memory_Heap

    !mov rax, qword [PB_StringHeap]
    !mov [p.v_StringHeap], rax
    !mov rax, qword [_PB_MemoryBase]
    !mov [p.v_MemoryBase], rax
    !mov rax, qword [PB_Memory_Heap]
    !mov [p.v_MemoryHeap], rax
  CompilerEndIf

  If HeapValidate_(StringHeap, 0, 0) = 0
    MessageRequester("StringHeap corrupted !", File$+" : "+Str(Line))
  EndIf

  If HeapValidate_(MemoryBase, 0, 0) = 0
    MessageRequester("MemoryBase heap corrupted !", File$+" : "+Str(Line))
  EndIf

  If HeapValidate_(MemoryHeap, 0, 0) = 0
    MessageRequester("AllocateMemory heap corrupted !", File$+" : "+Str(Line))
  EndIf
EndProcedure

Macro TestHeaps
  _TestHeaps(#PB_Compiler_File, #PB_Compiler_Line)
EndMacro 

Steps to find the bug:

  • Place a TestHeaps call right before the line that crashes. If you get one of the message requesters, you have a heap corruption. If not, then the problem is something else and the above code will not help.
  • Start placing TestHeaps calls in places that are executed before the crashing line. Start with only a call every bunch of lines and narrow it down later.
  • You need to find the line of code, where TestHeaps reports nothing before, and reports a heap corruption after. This is the line that causes all the problems.
  • Make sure you remove all this test code after fixing the bug, as it can have a big performance hit on the program (see below).

So why doesn’t the PureBasic debugger make this check automatically ?

The reason is that HeapValidate() has an effect on the future run of the program. The documentation sais that it degrades performance and that this effect can last until the end of the program. My guess is that the check for a valid heap somehow reorganizes it into a less efficient state which means that future allocations will be slower. This is why this check is not done by the debugger. Maybe there will be an option for this somewhen in the future. who knows ?

Porting your programs to PureBasic 64bit

The first public beta of PureBasic 64bit is out today, so the quick ones of you will soon start porting their projects to it. Here are a few tips from our own experience of porting the PureBasic tools to x64:

1) Port your project first to 4.30 x86 and fully test it there before moving on

The 4.30 update introduces a number of incompatible changes (many because of the x64 introduction) which are not all visible as compiler errors when used the old way. So instead of jumping directly to the x64 version and thinking about wrong variable sizes etc, better first make sure the crashes you get are not caused by one of these changes and are totally unrelated to the fact that your program is a 64bit one now. If you plan to release both a 32bit and 64bit version of your program, this is a logical first step anyway.

The most important change in this respect is probably the Read command. To avoid problems with different variable sizes on x86 and x64 with Read (which could lead to hard to find bugs), we decided to change the command to not get its type from the variable but rather like Procedure and others from a type postfix. This way you can match the Read type with the type in the DataSection, independent from the type of the variable you read the data into (you just cannot read a float into an integer type directly, but reading a byte into a quad works for example). Unfortunately, this change means that a 4.20 program which does “Read a.b” will now read an integer (4/8 bytes) as that is the default type if no postfix is specified. So our decision was to have this problem now once, or for all eternity when developing a program in both 32bit and 64bit. We decided to go with the former. The best and most foolproof way to deal with this is just to do a “Find in Files” on all your code for the Read command and adding a postfix for everyone of them.

Another source for a possibly hard to locate error is the change in LinkedList commands. They now return the element pointer, and no longer the list header pointer. So if you used the returnvalue of AddElement() and the like, you should check them all and make the needed changes.

Less problematic (because easily visible) changes are the ComboBoxGadget() height parameter, and the removal of the ButtonImageGadget() backward compatibility (you now need SetGadgetAttribute() to change the image). All other incompatible changes should raise compiler errors or warnings, so these are easy to spot.

Once all this is dealt with and your program runs fine with 4.30 x86, its time to dive into x64 programming…

2) .l is evil… very, very, very evil!

Many people have the habit of appending .l to every variable, even though this has been the default type all the time. Well, now is the time to stop that practice. Not only will the speed be worse using a too small type on x64, you also need to constantly worry what you store in the variable, as many things need 8 bytes for storage on x64 (pointers, handles, #PB_Any values, etc). Do not try to get guided by the urge to “save memory” here. 64bit systems have a ton of memory, and for normal variables this is really no argument.

From now on, you should treat the long type as you treated words and bytes so far. Use them only when really needed inside structures, or for Arrays/Lists where memory concerns really become an issue. For every normal variable just leave out the type (which will default to .i), or if you really cannot shake the habit, use .i instead. Just do this consistently, even for counter variables and the like. This will save you a ton of trouble, trust me. Truncated pointers are the worst kind of bug to try to track down.

Another good habit is to simply use a real *Pointer instead of just an integer variable whenever a pointer is stored. This is no requirement, as the .i type will work just as well. But in the long term this increases readability of the code in my opinion.

3) Working directly with memory – keep the pointer size in mind

When working with raw memory, we all probably used “*Pointer + 4” at one point to skip a pointer or long value. You now need to keep in mind to us 8 here for 64bit mode. Using any fixed numbers is discouraged here. Either use the SizeOf() compiler function, or define yourself a nice #PointerSize constant to make this very transparent.

4) API stuff

The API is quite transparent between the 32bit and 64bit versions. We updated all API structure definitions to have the correct types, so most API code should work on x64 out of the box.

Some things need to be considered though:

  • A number of API types also switch sizes, such as HANDLE, LONG_PTR etc. However, a number of types do not change size, such as DWORD, LONG, etc. For “in”-Parameters that are passed byvalue to a function, this is not a real problem, but if you need to pass a pointer to the variable to receive some data you need to check which type is needed.
  • Do NOT use Get/SetWindowLong_() for subclassing anymore. These remain 32bit even on x64. Use Get/SetWindowLongPtr_() instead, which works correctly everywhere. There is really no need to use SetWindowLong_() over SetWindowLongPtr_() actually, so best do a search and replace and replace all of them right away.
  • If you happen to use some API structure that is not predefined by PB, be careful about the padding. There is a lot more padding involved in the x64 structures, and the padding behavior is a bit complex to be explained quickly. If possible, just check the PB structure size against the output of a C compiler like VC8 (the 64bit compiler is provided with the Windows SDK) or ask on the Forum if you are unsure about the makeup of the structure.
5) Other

There is not much more too it. Calling the PB functions is straight forward on x64, as long as you do not use a too small type to store the return values (#PB_Any etc).

If you use ASM code, the x64 fasm commands can be used on the x64 version without trouble. The #PB_Compiler_Processor constant is a good help here to have separate CompilerIf sections of code for this.

All in all, the x64 port of the PB Programs like the IDE was much easier than we expected. If you get the variable types right, its pretty much done. Now good luck with your x64 ports.