Category: Coding

API Programming: What’s behind the PB GUI objects

Posted by freak – September 7, 2010

For Windows, this question is quickly answered: HWND. Pretty much everything is of this basic “window”-type and most API commands that deal with windows can be used on most GUI objects. You can also find many code examples on the forums that deal with a lot of common API tasks. The situation is different for Linux and OSX. Here, different Gadgets may have different underlying objects and so it is not entirely clear what functions can be used. There are also a lot less examples around for these systems. Of course when developing programs that run on multiple OS, you will need solutions for all of them.

To help this situation a bit, i have compiled a list of what are the underlying objects behind the PB gadgets and other GUI objects. You can get access to these using functions such as GadgetID(), WindowID(), etc.

Disclaimer:

This information is internal stuff. While many of these things haven’t changed in many years, it is still possible that we decide to change something in the future. Also some Gadgets are heavily modified for PB with custom subclasses or callbacks which also may change in the future and break your code. So as usual: Feel free to use this information, but if it breaks, its not our problem.

So here it goes:
Windows Linux OSX
WindowID() HWND GtkWindow WindowRef
MenuID() HMENU GtkMenuBar / GtkMenu MenuRef
StatusBarID() HWND GtkHBox ControlRef (UserPane)
ToolBarID() HWND GtkToolbar HIToolbarRef
FontID() HFONT PangoFontDescription PB internal pointer
ImageID() HBITMAP GdkPixbuf CGImageRef
GadgetID() HWND GtkWidget ControlRef
ButtonGadget “Button” GtkButton / GtkToggleButton PushButton / BevelButton
ButtonImageGadget “Button” GtkButton / GtkToggleButton PushButton / BevekButton
CalendarGadget “SysMonthCal32″ GtkCalendar HIView
CheckBoxGadget “Button” GtkCheckButton CheckBox
ComboBoxGadget “ComboBox” / WC_COMBOBOXEX GtkComboBox / GtkComboBoxEntry PopupButton / HIComboBox
ContainerGadget custom class GtkFixed UserPane
DateGadget DATETIMEPICK_CLASS GtkEntry UserPane
EditorGadget RICHEDIT_CLASS GtkTextView UserPane (rendering a TNXObject)
ExplorerComboGadget WC_COMBOBOXEX GekComboBoxEntry HIComboBox
ExplorerListGadget WC_LISTVIEW GtkTreeView DataBrowser
ExplorerTreeGadget WC_TREEVIEW GtkTreeView DataBrowser
Frame3DGadget “Button” / “Static” GtkFrame GroupBox / UserPane / Separator
HyperLinkGadget “Static” GtkButton UserPane
IPAddressGadget WC_IPADDRESS GtkEntry EditUnicodeText
ImageGadget “Static” GtkImage UserPane
ListIconGadget WC_LISTVIEW GtkTreeView DataBrowser
ListIconGadget “ListBox” GtkTreeView DataBrowser
MDIGadget “MDICLIENT” - -
OptionGadget “Button” GtkRadioButton RadioButton
PanelGadget “SysTabControl32″ GtkNotebook Tabs
ProgressBarGadget PROGRESS_CLASS GtkProgressBar ProgressBar
ScrollAreaGadget custom class GtkScrolledWindow UserPane
ScrollBarGadget “SCROLLBAR” GtkVScrollBar / GtkHScrollBar ScrollBar
ShortcutGadget HOTKEY_CLASS GtkEntry EditUnicodeText
SpinGadget UPDOWN_CLASS + “Edit” GtkHBox containing others UserPane containing others
SplitterGadget custom class GtkVPaned / GtkHPaned UserPane
StringGadget “Edit” GtkEntry EditUnicodeText
TextGadget “Static” GtkLabel StaticText
TrackBarGadget TRACKBAR_CLASS GtkVScale / GtkHScale Slider
TreeGadget WC_TREEVIEW GtkTreeView DataBrowser
WebGadget custom class + ActiveX control GtkMozEmbed HIWebView
Notes:
  • Gadgets all have the same basic type noted in the GadgetID() row. However, Gadgets are devided into “window classes” on Windows, or subclasses of the general GtkWidget on Linux.
  • For Gadgets on Windows: If the entry is noted in “” then this is the class name, if it is not then this is the symbolic constant for the class name. You’ll have to look up the text value in the appropriate header files.
  • On OSX, there are no classes for the controls, its all a ControlRef. The names i noted here are derived from the control creation functions (ie CreateTabsControl()). If you are looking for information, you should start there.
  • Gadgets may have multiple classes depending on the flags on creation.
  • On Linux, many gadgets are placed inside their own container to add a frame or catch events. (GtkFrame, GtkEventBox). The GadgetID() command returns the real gadget in this case, not the container.
  • On OSX, the UserPane controls are drawn by PB itself, so you don’t have much ways to modify them other than the PB commands.

Thats it. I hope this information is useful to some people, and hopefully we will see some more cross-platform API examples on the forum in the future :)

The cost of empty lists

Posted by freak – April 24, 2010

So now we have the ability to embed lists, maps and arrays inside structures. Its a very useful feature and we can already see people building crazy data structures with all sorts of nesting in them, even tree like structures. I want to talk a little bit about the costs associated with these objects. The fact is, even empty lists or maps take up memory and if you create a lot of them, you should be aware of that. The sizes i give below are all for the x86 versions (so 4 bytes per pointer). For x64, you can usually double that. Not all fields are actually pointer sized, so it is less than that in reality, but it is a good approximation.  My base scenario is an array of a structure type with an embedded list. However, this applies to all other combinations as well.

Structure MyStruct
  Something.i
  List Elements.s()
EndStructure

Dim Table.MyStruct(99)
Automatic initialization on creation

First of all, its important to realize that structure content is automatically initialized on creation. This means that if you create a structured variable with an embedded list, you don’t have to call NewList in order to use the embedded list. However, this also means that simply by the process of creating the structured variable, you also created a (still empty) list which takes up some memory. The same thing applies to my example array: By creating the array with 100 elements (0-99), i also created 100 empty lists.

You could say this is a waste of memory, and you would not be entirely wrong. After all, if not all elements of this array are actually used to store data, that is a lot of unused memory. We discussed this at length when implementing the feature. An alternative approach would have been to leave all such dynamic elements uninitialized on creation and require the user to explicitly call NewList before using them in each element. This would conserve memory when not all array elements are intended to have items in their lists, but it puts the burden to ensure that all dynamic elements are initialized before they are accessed on the user and introduces a whole new source for bugs. Its also extra work for the user considering that if people define such a structure they intend to put things into it, and forcing them to do an extra initialization step each time when PB can easily do it for them is just wasting the programmer’s energy.

So in the interest of having a really easy to use feature we decided to go with automatic initialization. After all, if you really need the full control and do not shy away from some extra work then building your own linked data structures in memory is not that hard. If you want something simple that just works, then this feature is for you. Of course its never wrong to know about the costs so you know what happens when your data set gets large.

Memory usage

How much memory does an empty list take? Aside from the actual space it takes up inside the structure which is 2 pointers for a list (1 pointer for array and map), there is a header structure which is allocated as well. It stores information about the list such as its type (for sorting and cleanup), the pointers to the first and last element in the list as well as cached index information for a fast SelectElement() if possible. The list header is currently 48 bytes on x86. As described in another entry, we also have custom memory allocation for list elements for faster allocation/freeing of memory. These allocation routines also take up 24 bytes of header data per list.

So in my list in array example above i have created 100*12 bytes of actual element data in the array plus another 100*72 bytes for empty lists. While this extra data is still not very much, it is considerably more than the array element data itself, so it will become an issue if you make your array very large. If you knew for example that you will never have more than 18 elements in the embedded list, then you could replace the thing with a static array inside the structure with 18 elements and still have the same memory usage.

Empty maps have a much larger impact. They also come with a header and an allocator header, but an empty map also reserves space for its hash table even if there are no elements in it. The default table size is 512 entries, so that is another 2048 bytes for an empty map on x86.

Arrays are a little different as you can declare inside the structure how many elements it should have on creation. Arrays also come with a header, but nothing else. Here its more important to keep in mind that if the elements of the embedded array in turn contain embedded lists, maps or arrays that you have to take their costs into account as well as they will be initialized too when the embedded array is created.

One final thing to keep in mind is that lists and maps actually allocate the memory for their elements in blocks of 16 elements minimum (for speed and to reduce fragmentation), if you create many lists with just a few elements in them (say 2 per list for a binary tree structure), you end up with used memory of 16 elements per list instead. Note though that this is just allocation, not initialization. The used memory will be only for the 16 elements themselves, not for any contained dynamic list, map or array they may contain.

Bottom line

This post should be seen in the right context. The cost of the embedded object may seem large compared to the size of the structure they are put in, but that doesn’t change the fact that they are still very cheap objects. Lists, maps and arrays are all memory-only objects. This means that all they need is memory, no handles, no system objects, nothing else. Memory is not much of a concern these days, so you can create many of them and even create/destroy them frequently. This really isn’t a problem. Take my above array example: Even if we Dim the array to 10000 elements, we still only used up 820Kb of memory which is not much by today’s standards.

The only place where you really need to concern yourself with this is if your data sets get really large and memory does become an issue. In this case the advice is simple: Avoid “sparse” data structures where a large part of the dynamic elements are allocated but not used. You can also call FreeList(), FreeMap(), FreeArray() or ClearStructure() on any element you do not need. You then have to re-initialize it if you want to use it later with NewList, NewMap, Dim or InitializeStructure() respectively.

Always keep in mind the quote from Donald Knuth on optimization:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.

In this context, this means not to worry about this issue when designing your data structures. Do not get tempted to optimize away a few bytes of extra memory with funny tricks. A clear and simple layout of the data in your program is worth much more than that (and will save you many headaches from chasing bugs). Only if you see that your program really has a memory problem is it time to start thinking of ways to cut down on overhead like this.

In this spirit, have fun designing crazy nested data structures with the new dynamic elements! :)

2DDrawing in v4.40 under the hood

Posted by freak – August 26, 2009

A lot changed in the 2DDrawing library with the v4.40 version. In fact, it basically underwent a complete rewrite. Only very few code parts remain from the previous version. Since it has come up in various questions and bug reports on the forum i am going to explain the new design a little bit and what consequences it has on stuff like mixing API and PB commands for drawing.

Lets start off with what the 4.31 library looked like:

The Windows version of the library was based entirely on GDI (the Windows drawing API). GDI allows output on a window, image, printer, sprites and the screen with the same commands so for the most part only the code that set up the drawing needed to be specific and the rest was shared (hence the separation of the output functions like ImageOutput() from StartDrawing()). One detail that a lot of users relied on was the fact that the GDI handle to the drawing context (HDC) was returned for StartDrawing() which allowed to easily mix API and PB commands in the drawing operation even without much knowledge of GDI. The internal data structure of the library was published a long time ago in the PB Library SDK and since the library didn’t change much over time there is still some code that relies on them.

Linux was a different story. Here every output requires its own API (Gtk for window an image, libgnomeprint for printer and some pixel based routines for SDL output). So the Linux library had a plugin like system just like some other PB libraries have to select the right command depending on the output. The Mac OSX version basically had only one output for images. The printer library created a temporary image and sent it to the printer when drawing was done. There was no drawing for windows, sprites or the screen on OSX. The drawing functions shared the pixel based drawing code we used for SDL on Linux.

Why the library was rewritten:

The reason was the alphachannel. As explained in a previous entry, i worked on improving the capabilities of PB when it comes to displaying alphachannel images. After that was finished i felt that at least some support for 2DDrawing was needed in addition to the DrawAlphaImage() command as well. The problem here is that GDI does not support the alphachannel even though it can handle 32bit images. Since Windows 98 you can load images with alphachannel and display them with the AlphaBlend() API but that is pretty much it. Any manipulation of the image with GDI will cause the alpha information to be lost. The same problem exists with Gdk on Linux. The ony drawing functions that exist work on GdkDrawable objects and these do not support the alphachannel. GDI+ is the Windows replacement for GDI which can deal with the alphachannel, but we needed a crossplatform solution to this problem. So we made the decision to create our own drawing functions similar to the pixel based ones for SDL and OSX which could handle the alphachannel in all drawing commands. As you can see from the result i went a step further and added the gradient drawing support (which also would not be possible with the API functions).

The new design:

Since our pixel based routines can only draw on images we now need separate “subsystems” for drawing even on Windows. So now we have this plugin-architecture on all OS. The real drawing functions are called through pointers which are set when the drawing is started so the speed impact is minimal. For those concerned about executable sizes: A drawing subsystem is only linked when the corresponding output function is used in the code. So if you do not use ImageOutput() none of the image drawing code will be linked.

So there is now a GDI subsystem on Windows and the new “PixelBuffer” one. The PixelBuffer subsystem does all its drawing directly on the memory of the output image with no API involvement. The only exception is the drawing of text. We did not want to add the overhead and license troubles by including an external font rendering engine like freetype so DrawText() uses GDI to render the text to a special device context from which the text is then drawn to to the real output with the alphachannel and gradients taken into account. It works in a similar way on the other OS where the native API is used to render the text and then it is copied to the real output. There is of course a speed penalty from this, but it cannot be avoided and it is questionable how much faster a separate rendering engine would be with this task.

Things to watch out for for on Windows:

I tried my best to keep the library backward compatible. If you use PB drawing commands only then there should be no difference at all. If you mix them with API then there are some things to watch out for:

  • As the “HDC result from StartDrawing()” is very commonly used i kept that behavior the same. So even though the PB commands do not use it for the drawing, there is still a device context created and returned from StartDrawing() which you can use to draw on the image with API.
  • You can still mix API drawing functions with PB functions on the same output without problems. The only thing that changed here is that GDI functions that change the global state of the device context no longer affect the PB drawing functions like they used to because the PB functions do not actually used that device context. So a function like SetWorldTransform() will not cause DrawImage() to draw a rotated image anymore. However it should still work if you use the BitBlt() API.
  • You have to be aware that GDI functions will erase the alphachannel. So if you draw on a 32bit PB image and use a GDI function, the modified area will look transparent after that. The best way to avoid this problem is to use 24bit images as output when API commands are involved. (We changed the default image depth to 24bit in 4.40 beta2 to make the transition easier)
  • PB Images are now always DIBs (device independent bitmap) to allow for the pixel based drawing. The #PB_Image_DisplayFormat flag used to create a DDB (device dependent bitmap), but this is no longer supported. #PB_Image_DisplayFormat now has the value 32 to create a 32bit image instead. Some GDI functions (like GetDIBits()) expect a DDB, so you may get trouble there.
  • DrawImage() can still draw API created images (including DDBs and icons). Again you have to be aware that a 32bit API created bitmap will probably have all alpha values as 0 which PB will interpret as fully transparent. Using 24bit is the solution here as well. Also DrawImage() expects DIBs to be bottom-up (the default on Windows). If you use top-down DIBs then they will be drawn upside down. There is no way to avoid that as Windows does not provide a way to find out the pixel orientation of a DIB.
  • If you relied on the library’s internal data structures then you are out of luck. Nothing is as it was before on the library’s internals.

To sum it up: Stick to 24bit images and in most cases mixing PB drawing with API drawing should work just as it did before.

Things to watch out for on Linux:

I am not aware of code that mixed 2DDrawing with API on Linux. StartDrawing() used to return the GdkDrawable handle for Image+WindowOutput() before. It now returns that only for the WindowOutput() as there is no more drawable when drawing on an image. Backward compatibility wasn’t possible here.

One thing to note in general is that PB images are now GdkPixbuf objects and no longer GdkPixmap. This is a big improvement as many Gtk functions expect GdkPixbuf nowadays, so it is actually easier to use PB images with Gtk functions.

Things to watch out for on OSX:

Nothing to say here. Since the 2DDrawing support was so poor before, it only got better with this release. There is now a separate QuickDraw based subsystem for WindowOutput(). Yes i know that QuickDraw is deprecated, but it was much easier to implement this way than to go for a full Quartz based one. I have plans to do a Quartz subsystem one day to have better PrinterOutput() (for the moment, it still works with images as an intermediate).

One thing to note is the new output for sprites and the screen with OpenGL (this applies to all OS): There is no way to modify the data in video memory, so if you call StartDrawing() the whole sprite data or the entire screen content is transferred to main memory for the drawing. This is very slow. So doing 2DDrawing everytime you refresh the screen is too costly. The better way is to draw on sprites just once and then display them over and over. Still, this is better than having no 2DDrawing for sprite/screen at all.

Future plans:

There is still room for optimisation in the new 2DDrawing code. For now the focus was on getting a stable implementation. Also the plan is to eventually have the alpha drawing routines also for SpriteOutput() and ScreenOutput() but this will need some more work to support all possible pixel formats for the screen. Stay tuned…

Is your 64bit program really solid ?

Posted by freak – August 16, 2009

If you ask Windows for memory it will generally start with low addresses first. There is nothing wrong with that. For a 64bit program however, this means that under normal circumstances unless you actually use up 4Gb of memory, your pointers will still fit into a 32bit value. So if you store a pointer into a too small variable (say a long) you will not notice, because the upper half of the 64bit pointer value isn’t even used. So your program may look quite solid and usable right now but if you feed it something like a 4Gb file to load it will start crashing. This may sound like an unlikely case but it will become much more common even for programs that are not meant to deal with large amounts of data, simply because as disk space, memory and computer speeds grow, so do the amounts of data that users deal with.

Anyway, I recently found this link: http://msdn.microsoft.com/en-us/library/bb613473.aspx

Notice at the very bottom the mention of the AllocationPreference registry setting. If you develop 64bit applications (especially if you are porting code from 32bit to 64bit) then this key will help you a great deal. What it does is it instructs Windows to give you high addresses first. The result are pointers that don’t fit into 32bit right from the start. So these errors that usually go unnoticed will blow up right in your face and you can fix them.

The link doesn’t explicitly state the OS this works on but i tested both on XP 64bit and Vista 64bit and it works fine on both. You have to restart the computer after setting the registry value for the change to take effect. You can then verify with a simple program if the high addresses are returned (make sure you compile it with the 64bit compiler ;) ):

For i = 1 To 10
  a$ + RSet(Hex(AllocateMemory(1000), #PB_Integer), 16, "0") + Chr(13)
Next i
MessageRequester("Pointer values", a$)

In this mode you can easily distinguish a valid pointer from a truncated one by looking at it as hex like above. Since the virtual address space is limited to 8TB on current x64 Windows versions, a pointer allocated in this mode will look something like 000007FFFXXXXXXX because the addresses cannot go any higher. If a pointer got cut somewhere it will either look like 00000000XXXXXXXX or FFFFFFFFXXXXXXXX depending on wether the cut 32bit value was positive or negative (the sign-extension of the negative numbers will create all the F’s).

The PureBasic package itself is a good example of how easily such bugs go unnoticed. We payed a lot of attention to this when porting the libraries and also the programs like IDE, debugger and compiler to 64bit with PB 4.30. There was a lot of testing by the users after that too and PB 4.31 x64 is quite stable. However, if you turn on this mode the 4.31 IDE and debugger will not even start. Even the compiler won’t compile anything. So if you want to try this with your own programs you obviously cannot use PB 4.31 for it. The new 4.40 version was tested with this mode and these bugs are now fixed. Since it is still in beta, you may run into the odd crash though. If this happens please let us know.

Keep in mind that this setting is systemwide. So other x64 programs could well crash on you too if they have such unnoticed bugs.

Have fun fixing a ton of bug you never knew existed :P

Using PureBasic (32bit) on 64bit Linux

Posted by freak – February 26, 2009

This has been asked a couple of times. Until now i wasn’t much help there because i never tried this myself. So now i tried it and had some success. I was able to compile the PureBasic IDE, which means that pretty much all non-multimedia libs must be working. If you don’t want to wait for PureBasic 64bit for Linux (planned for 4.40) then here is a step by step guide.

There is a guide for Ubuntu 8.10 and OpenSUSE 11.1 64bit (always starting from a clean installation). If you have a slightly different version of one of these distributions then it may work for you as well with some luck. If you have a different distribution, then read this anyway. There are some tips on the bottom on how you could make this work for your distribution as well.

PureBasic on Ubuntu 8.10 64bit

First download and unpack the PureBasic package. Then open a console and set up the environment variables for the compiler and try running it “pbcompiler -h”. If you started from a clean install, then not even this will work (you will get a weird error like “No such file or directory”). So we first need the runtime environment for 32bit programs:

sudo apt-get install ia32-libs

Note that this installs a lot of stuff (120Mb). After this is completed, the compiler should at least be able to run. At this point you should also be able to run the IDE. But even trying to compile an empty program will cause linker errors. So what we need next is the basic gcc and libc6 support for compiling 32bit programs:

sudo apt-get install gcc-multilib libc6-i386 libc6-dev-i386

After that, you should be able to compile your first (console only) programs, and also use the PureBasic debugger.

The next big thing are Gtk and SDL. Unfortunately there are no special 32bit development versions of these packages available (something like “libgtk2.0-dev-i386″), so we have to use some tricks. To compile with PB, we actually only need two things: The library files and the properly working pkg-config sdl-config tools to actually find them (and tell the linker which libraries to link).

Lets start with the configuration tools: here we can actually use the ones that come with the regular 64bit packages, so just install those:

sudo apt-get install libgtk2.0-dev libsdl1.2-dev

If you try to compile a program with a MessageRequester() before and after this step, you will notice that the linker errors are reduced from a huge pile of unresolved symbols to just not finding a compatible library. So all we need now is some compatible library files. It turns out that all the needed libraries are already installed with the “ia32-libs” package, because you can use a dynamic library (.so) for linking as well (the linker will automatically generate a static lib from that). The only thing that is missing really are some symbolic links to help the linker find those dynamic ones, thats all.

I found a nice script here that creates these links and extended it by a few more links needed for PureBasic. Put the following into a textfile and execute it as root (“sudo sh myfile.sh”):

#!/bin/sh
cd /usr/lib32
for lib in gio-2.0 gdk-x11-2.0 atk-1.0 gdk_pixbuf-2.0 \
           pangocairo-1.0 pango-1.0 pangoft2-1.0 \
           gobject-2.0 gmodule-2.0 glib-2.0 gtk-x11-2.0; do
  ln -s -f lib$lib.so.0 lib$lib.so
done
ln -s -f libcairo.so.2 libcairo.so
ln -s -f libfreetype.so.6 libfreetype.so
ln -s -f libz.so.1 libz.so
ln -s -f libfontconfig.so.1 libfontconfig.so
ln -s -f /usr/lib32/libX11.so.6 /usr/lib32/libX11.so
ln -s -f /usr/lib32/libXrender.so.1 /usr/lib32/libXrender.so
ln -s -f /usr/lib32/libXext.so.6 /usr/lib32/libXext.so
ln -s -f libgthread-2.0.so.0 libgthread-2.0.so
ln -s -f libSDL-1.2.so.0 libSDL.so
ln -s -f libstdc++.so.5 libstdc++.so

Thats it! Now most libraries should work fine. The IDE compiled fine with this setup. As far as i understand the page above, this step will actually not be needed for the next stable Ubuntu release.

Some PureBasic libs that have special dependencies will still not work however. The commands from the following libraries will compile/link fine, but not work with this setup:

  • Sound, SoundPlugin, Module – I am not sure why these do not work. Could be because i am running VMWare and have no tools installed yet. Somebody else will have to try this.
  • All 3D Engine related libraries – I did not look deeper into why the engine does not load. Maybe another time. It actually starts loading, but you never see a screen.
  • the WebGadget() – This does not even work out of the box on some 32bit systems. I did not bother looking deeper into that.

The following Libraries will not even link. The reason is simply that the “ia32-libs” package does not include 32bit versions of them. To get them to work you would have to compile them manually in 32bit mode. Maybe i will try that another time.

  • PrinterOutput() - needs libgnomeprint2.2
  • Movie lib – needs libxine
  • Database lib with ODBC – needs libiodbc2 (Note: SQLite works fine as it is included in PB)
PureBasic on OpenSUSE 11.1 64bit

I started from a clean installation with Gnome desktop and no other packages added in the setup.

Open the “Software Management” in Yast and install the following packages:

  • gcc-32bit
  • gtk2-devel
  • sdl-devel-32bit
  • libgnomeprint-devel
  • libgnomeprint-32bit
  • libgnomeprintui-devel
  • libgnomeprintui-32bit
  • libxine-devel
  • libxine1-32bit

Then copy the code below to a text file and run it in a shell as root (“sudo sh myfile.sh”):

#!/bin/bash
cd /usr/lib
for lib in gio-2.0 gdk-x11-2.0 atk-1.0 gdk_pixbuf-2.0 \
           pangocairo-1.0 pango-1.0 pangoft2-1.0 \
           gobject-2.0 gmodule-2.0 glib-2.0 gtk-x11-2.0; do
  ln -s -f lib$lib.so.0 lib$lib.so
done
ln -s -f libcairo.so.2 libcairo.so
ln -s -f libfreetype.so.6 libfreetype.so
ln -s -f libfontconfig.so.1 libfontconfig.so
ln -s -f libstdc++.so.6 libstdc++.so
ln -s -f /lib/libz.so.1 /lib/libz.so
ln -s -f /lib/libgcc_s.so.1 /lib/libgcc_s.so
ln -s -f libgthread-2.0.so.0 libgthread-2.0.so
ln -s -f libgnomeprint-2-2.so.0 libgnomeprint-2-2.so
ln -s -f libart_lgpl_2.so.2 libart_lgpl_2.so
ln -s -f libxml2.so.2 libxml2.so
ln -s -f libgnomeprintui-2-2.so.0 libgnomeprintui-2-2.so
ln -s -f libgnomecanvas-2.so.0 libgnomecanvas-2.so
ln -s -f libxine.so.1 libxine.so
ln -s -f libpng12.so.0 libpng.so

This should do the trick to allow PB to compile. Note that unlike on Ubuntu, the Movie lib and PrinterOutput() will work just fine here. The other mentioned library problems are the same as on Ubuntu though.

PureBasic on other 64bit Linux distributions

Ok, i cannot do this for all distributions, but the steps on your Linux of choice should be fairly similar, just with slightly different package/filenames:

  1. Install the 32bit runtime environment (sometimes this is preinstalled). You know you succeeded when the IDE and compiler can be started.
  2. Install what is needed to build basic 32bit programs. This is usually a special gcc package and a special libc6-dev package. You know you succeeded if you can compile an empty sourcecode without errors.
  3. Install the required libraries. If they exist as a 32bit-devel package then that is perfect. If they only exist as a 32bit runtime version (maybe included in a larger 32bit package) then install that plus the 64bit devel package. Then add any missing symbolic links in your 32bit lib directory. Usually the linker looks for a file like “libSDL.so” and the folder will contain a “libSDL1.2.so.0″ or similar. (usually just with the extra .0) Just add a link as the scripts above do. If there is not even a 32bit runtime package for a library then you can only try compiling them manually. Use a small testcode like a MessageRequester() and try the above until the linker no longer complains about any libraries.

The required packages (all as devel version, Gtk and SDL are the most important):

  • libgtk2.0
  • libsdl1.2
  • libstdc++ (for ScintillaGadget)
  • libgnomeprint2.2 and libgnomeprintui (for PrinterOutput())
  • libxine (for Movie lib)
  • libiodbc (for ODBC database)

Thats pretty much it. I hope you can get this working on your system. If there are more problems just ask. I would not call myself a Linux expert but i will try to help if possible. If you get PureBasic working on another distribution, please publish your exact steps somewhere so others can benefit too.

Debugging tips – Part 2: Stack problems

Posted by freak – November 4, 2008

Another common problem that the debugger does not really identify are stack problems. Fortunately, other than the heap problems discussed before, these are not hard to narrow down.

Symptoms:
  • A piece of code works on the main level, but fails when called inside a procedure or inside a Select block
  • The crash is usually when leaving the procedure (ProcedureReturn or EndProcedure)
  • Usually a call to a Dll or imported function is involved
Reason:

The x86 processor has several different so called calling conventions which define how a function call is made (were the arguments go and who is responsible for cleaning up after the call). In fact everybody can invent their own convention for this, and quite a number of compilers do this for performance reasons. (PureBasic itself does it for Library functions written in ASM. They use the stdcall convention, but receive their first argument in the EAX register instead of the stack). When calling an external function, the calling convention must match or there will be problems like the one we have here.

The most popular conventions on x86 are cdecl and stdcall. cdecl is the standard used by the C language, stdcall is most used by Windows dlls. PureBasic uses stdcall by default in most places. They both provide the function arguments on the stack in reverse order. The only main difference is in the fact that cdecl requires the caller to remove the arguments from the stack and stdcall requires the function to do this before returning (there is also a difference in symbol naming, but thats not relevant here). So if the caller assumes the wrong calling convention then either both try to remove the parameters from the stack or nobody does it. In both cases the stack is not the same after the function call than before, and this is a big problem.

Here we get to the reason why it works outside of a procedure and not inside. A call to a procedure stores the address where execution continues after the procedure on the stack as well. If a function call inside the procedure leaves the stack in a mess, then the ProcedureReturn cannot find the address to continue and uses a wrong value instead which leads to a crash. Outside of a procedure, nobody will notice the stack mess, as the stack is only used for function calls.

Solution:
  • PB provides support for both stdcall (the default) and cdecl calling conventions (the functions/keywords with a C in the name)
  • If you used CallFunction() or CallFunctionFast() try CallCFunction() or CallCFunctionFast() instead. (these are the cdecl versions of the same function)
  • If you used Prototype, try PrototypeC instead.
  • If the crash happens in a callback procedure which you passed to an external function, try defining the callback with ProcedureC instead.
  • Also check if the number of arguments and the argument types match, as this can lead to the same problem. (remember that CallFunction() and CallFunctionFast() default to the long type, so for quads and doubles you have to use prototypes for a correct call.)

Some rules of thumb:

  • On Windows all Dlls that come with Windows use the stdcall convention. Most other Dlls follow this model, so there are only a few that use cdecl. With static libraries (used with Import/ImportC), the division is not so clear. Just try it.
  • On Linux / Mac OSX, the dominating calling convention is cdecl. So pretty much every library you use will require the C type of function.

Stack problems can also arise when InlineASM code is used and the stack is changed. Those that use this should know what they are doing though. The x64 and PowerPC processors do not have the calling convention problems by the way, as the force one common calling convention on everybody. So a parameter mismatch is the most probable explanation here.

A sidenote: Window callbacks

On Windows, another condition which can cause a crash on the ProcedureReturn keyword is the window callback. This is not a stack issue then though. What it means is that the crash happened somewhere in the PureBasic event processing that happens after your window callback returns. The debugger just shows the ProcedureReturn line because this was the last correctly executed line in non-library code.

What this probably means is that you did something with Windows API calls that conflicts with the way PureBasic handles its events. This may be in the window callback directly, or by modifying some Gadget/Window related data that PureBasic is expecting to remain unchanged. If you have no clue what causes this, just post it on the forum. There are a lot of capable people around that can help with these things.

Debugging tips – Part 1: Heap corruption

Posted by freak – October 30, 2008

If you are wondering why this blog is silent for so long, its because we are in the beta phase of 4.30, and writing about bugfixing and documentation work is just boring.

I want to talk a bit about the most common problems i have seen on the forums where the PureBasic debugger is of not much help and which therefore leave most users quite confused. The usual reaction is to post it as a bugreport. So in the future, if somebody posts a bugreport about AllocateMemory(), you can point him to this article and it should explain everything :)

Symptoms:
  • A crash at AllocateMemory() or FreeMemory() even though the given input is valid.
  • A crash on a simple string assignment
  • Modifying the code in seemingly unrelated places makes the problem go away.
Reason:

First of all: AllocateMemory() is never the problem. Its a direct wrapper to the HeapAlloc() API function and also a heavily used function. If it had a bug we would know by now. What you have gotten yourself into here is what is called a heap corruption. You destroyed part of the data Windows uses to manage allocated memory.

When Windows allocates memory, it keeps a data structure to manage the allocated memory (usually 12bytes on 32bit Windows). This data structure is normal, writable memory which means that you will not get any access error when accidentally writing over it. It is nowhere specified where this data is kept, but it is a fact that it sometimes ends up right after your allocated memory buffer. Now, if you happen to write over the end of this buffer by just a few bytes, you destroy the heap structure without getting any error. The crash only happens later when another attempt to allocate or free memory causes Windows to examine this piece of heap data and crash due to an invalid pointer. This fact, that the cause of the problem and the actual crash are in different places makes this kind of bug so hard to debug.

Why does this not happen always when you overwrite a buffer ?

Getting the heap data right at the end of the allocated buffer is a very rare condition. Windows often rounds the allocated buffer size up to page boundaries or for alignment purposes, so you often get more memory than you asked for, which makes a small overwrite have no effect. Another scenario is that the memory after your allocated buffer is simply marked as invalid, in which case you get an error when trying to write to it. Which of these scenarios actually happens depends on the sequence of memory allocations done by your program, which means that if you comment another totally unrelated program part, you may change the allocation sequence and the problem seemingly disappears. (Note that in this case only the symptom disappears, not the problem of writing over the end of a buffer.)

Solution:

As we saw above, the problem is not where the crash is. Fortunately, Windows provides a function to check if the heap structures. Below is a piece of code that can be used to check the most used memory heaps in PureBasic: (this works in PureBasic Windows 32bit and 64bit)

Procedure _TestHeaps(File$, Line)
  Protected StringHeap, MemoryBase, MemoryHeap

  CompilerIf #PB_Compiler_Processor = #PB_Processor_x86
    !extrn _PB_StringHeap
    !extrn _PB_Memory_Heap         

    !mov eax, dword [_PB_StringHeap]
    !mov [p.v_StringHeap], eax
    !mov eax, dword [_PB_MemoryBase]
    !mov [p.v_MemoryBase], eax
    !mov eax, dword [_PB_Memory_Heap]
    !mov [p.v_MemoryHeap], eax
  CompilerElse
    !extrn PB_StringHeap
    !extrn PB_Memory_Heap

    !mov rax, qword [PB_StringHeap]
    !mov [p.v_StringHeap], rax
    !mov rax, qword [_PB_MemoryBase]
    !mov [p.v_MemoryBase], rax
    !mov rax, qword [PB_Memory_Heap]
    !mov [p.v_MemoryHeap], rax
  CompilerEndIf

  If HeapValidate_(StringHeap, 0, 0) = 0
    MessageRequester("StringHeap corrupted !", File$+" : "+Str(Line))
  EndIf

  If HeapValidate_(MemoryBase, 0, 0) = 0
    MessageRequester("MemoryBase heap corrupted !", File$+" : "+Str(Line))
  EndIf

  If HeapValidate_(MemoryHeap, 0, 0) = 0
    MessageRequester("AllocateMemory heap corrupted !", File$+" : "+Str(Line))
  EndIf
EndProcedure

Macro TestHeaps
  _TestHeaps(#PB_Compiler_File, #PB_Compiler_Line)
EndMacro

Steps to find the bug:

  • Place a TestHeaps call right before the line that crashes. If you get one of the message requesters, you have a heap corruption. If not, then the problem is something else and the above code will not help.
  • Start placing TestHeaps calls in places that are executed before the crashing line. Start with only a call every bunch of lines and narrow it down later.
  • You need to find the line of code, where TestHeaps reports nothing before, and reports a heap corruption after. This is the line that causes all the problems.
  • Make sure you remove all this test code after fixing the bug, as it can have a big performance hit on the program (see below).

So why doesn’t the PureBasic debugger make this check automatically ?

The reason is that HeapValidate() has an effect on the future run of the program. The documentation sais that it degrades performance and that this effect can last until the end of the program. My guess is that the check for a valid heap somehow reorganizes it into a less efficient state which means that future allocations will be slower. This is why this check is not done by the debugger. Maybe there will be an option for this somewhen in the future. who knows ?

Porting your programs to PureBasic 64bit

Posted by freak – September 16, 2008

The first public beta of PureBasic 64bit is out today, so the quick ones of you will soon start porting their projects to it. Here are a few tips from our own experience of porting the PureBasic tools to x64:

1) Port your project first to 4.30 x86 and fully test it there before moving on

The 4.30 update introduces a number of incompatible changes (many because of the x64 introduction) which are not all visible as compiler errors when used the old way. So instead of jumping directly to the x64 version and thinking about wrong variable sizes etc, better first make sure the crashes you get are not caused by one of these changes and are totally unrelated to the fact that your program is a 64bit one now. If you plan to release both a 32bit and 64bit version of your program, this is a logical first step anyway.

The most important change in this respect is probably the Read command. To avoid problems with different variable sizes on x86 and x64 with Read (which could lead to hard to find bugs), we decided to change the command to not get its type from the variable but rather like Procedure and others from a type postfix. This way you can match the Read type with the type in the DataSection, independent from the type of the variable you read the data into (you just cannot read a float into an integer type directly, but reading a byte into a quad works for example). Unfortunately, this change means that a 4.20 program which does “Read a.b” will now read an integer (4/8 bytes) as that is the default type if no postfix is specified. So our decision was to have this problem now once, or for all eternity when developing a program in both 32bit and 64bit. We decided to go with the former. The best and most foolproof way to deal with this is just to do a “Find in Files” on all your code for the Read command and adding a postfix for everyone of them.

Another source for a possibly hard to locate error is the change in LinkedList commands. They now return the element pointer, and no longer the list header pointer. So if you used the returnvalue of AddElement() and the like, you should check them all and make the needed changes.

Less problematic (because easily visible) changes are the ComboBoxGadget() height parameter, and the removal of the ButtonImageGadget() backward compatibility (you now need SetGadgetAttribute() to change the image). All other incompatible changes should raise compiler errors or warnings, so these are easy to spot.

Once all this is dealt with and your program runs fine with 4.30 x86, its time to dive into x64 programming…

2) .l is evil… very, very, very evil!

Many people have the habit of appending .l to every variable, even though this has been the default type all the time. Well, now is the time to stop that practice. Not only will the speed be worse using a too small type on x64, you also need to constantly worry what you store in the variable, as many things need 8 bytes for storage on x64 (pointers, handles, #PB_Any values, etc). Do not try to get guided by the urge to “save memory” here. 64bit systems have a ton of memory, and for normal variables this is really no argument.

From now on, you should treat the long type as you treated words and bytes so far. Use them only when really needed inside structures, or for Arrays/Lists where memory concerns really become an issue. For every normal variable just leave out the type (which will default to .i), or if you really cannot shake the habit, use .i instead. Just do this consistently, even for counter variables and the like. This will save you a ton of trouble, trust me. Truncated pointers are the worst kind of bug to try to track down.

Another good habit is to simply use a real *Pointer instead of just an integer variable whenever a pointer is stored. This is no requirement, as the .i type will work just as well. But in the long term this increases readability of the code in my opinion.

3) Working directly with memory – keep the pointer size in mind

When working with raw memory, we all probably used “*Pointer + 4″ at one point to skip a pointer or long value. You now need to keep in mind to us 8 here for 64bit mode. Using any fixed numbers is discouraged here. Either use the SizeOf() compiler function, or define yourself a nice #PointerSize constant to make this very transparent.

4) API stuff

The API is quite transparent between the 32bit and 64bit versions. We updated all API structure definitions to have the correct types, so most API code should work on x64 out of the box.

Some things need to be considered though:

  • A number of API types also switch sizes, such as HANDLE, LONG_PTR etc. However, a number of types do not change size, such as DWORD, LONG, etc. For “in”-Parameters that are passed byvalue to a function, this is not a real problem, but if you need to pass a pointer to the variable to receive some data you need to check which type is needed.
  • Do NOT use Get/SetWindowLong_() for subclassing anymore. These remain 32bit even on x64. Use Get/SetWindowLongPtr_() instead, which works correctly everywhere. There is really no need to use SetWindowLong_() over SetWindowLongPtr_() actually, so best do a search and replace and replace all of them right away.
  • If you happen to use some API structure that is not predefined by PB, be careful about the padding. There is a lot more padding involved in the x64 structures, and the padding behavior is a bit complex to be explained quickly. If possible, just check the PB structure size against the output of a C compiler like VC8 (the 64bit compiler is provided with the Windows SDK) or ask on the Forum if you are unsure about the makeup of the structure.
5) Other

There is not much more too it. Calling the PB functions is straight forward on x64, as long as you do not use a too small type to store the return values (#PB_Any etc).

If you use ASM code, the x64 fasm commands can be used on the x64 version without trouble. The #PB_Compiler_Processor constant is a good help here to have separate CompilerIf sections of code for this.

All in all, the x64 port of the PB Programs like the IDE was much easier than we expected. If you get the variable types right, its pretty much done. Now good luck with your x64 ports.