Category Archives: Coding

Coding in PureBasic itself

New PureArea.net Interview with Fred and freak released

Hi folks, Happy New Year 2016!

I’m proudly presenting a new interview I made with the main PureBasic coders Frederic ‘alphaSND’ Laboureur and Timo ‘freak’ Harter.

You can read it on my website www.PureArea.net, just go to the ‘News’ section of 4th January 2016.

I hope you enjoy the reading of this (like I find) interesting interview!

(Direct link to the interview)

Enter GTK3

Since PureBasic 5.40 LTS, GTK3 is the default GUI toolkit used by PureBasic on Linux. It seems like a no brainer update for our users, but the fact is than GTK3 doesn’t fit well the PureBasic GUI library and we have several bad hacks here and here to make it work at all. We decided from the start in PureBasic to handle pixel perfect sizing of the GUI elements to allow maximum flexibity. For example, we added a Dialog library on top of it, which adds automatic layout depending of the font size, and allow to design complex resizable windows. It’s nice, but not mandatory, and depending of your needs, you will or will not use it. Up to GTK2, it was the same: you could define pixel perfect size and the toolkit would honor it. Now, it’s a different story: the toolkit can decide if the size you gave programmatically is too small or not. It’s a big change, as you could end up with a deformed GUI and more important, the running program doesn’t reflect the code entered by the programmer. You may wonder why such change has been implemented ? My guess is to force use of layout for everything, so the toolkit can support themes more efficiently and without change from the programmer. It may be true, but it’s one of the first toolkit to do so and forcing the programmer hand isn’t always a great thing, especially when it removes flexibility: you can write a layout engine using pixel perfect sizing, but you can’t do pixel perfect sizing with a layout engine. And for some specific apps, running on proprietary OS, it can be an issue as theming is irrelevant.

Getting the best out of the IDE when using Modules

Since the last beta version (5.20 beta8) the IDE finally has support for Modules as well (AutoComplete, QuickHelp, ProcedureBrowser etc). Please test these features well, as it was quite a change in the IDE to implement this. Now, to get the best experience when using the IDE and Modules together, there are some simple rules that have to be followed. I will explain them below. But first, let me explain why these limitations exist.

As the AutoComplete feature became smarter and more helpful in past versions, it had to grow from a simple scanning of the source for strings into a full grown analysis of the source code. This is especially required for modules, because the IDE needs to know what procedures/constants/structures are exposed from each module in order to present the correct choices in AutoComplete. So the IDE in essence needs to understand your code. The best way to fully understand code is to compile it. The IDE does not actually compile the code for the following reasons:

  • Speed: The AutoComplete feature cannot introduce delays because that would make it unusable. So compiling the entire code from beginning to end is not an option.
  • The code is usually not compilable anyway: Remember, the purpose of the IDE is to edit code. While you edit a line, it will be in a state that cannot be compiled most of the time. In fact, if you look at all characters typed in a source code, the instances in which that typed character puts the source in a compilable state are rather rare compared to those that put it in a state with errors (from a compiler perspective).
  • There are IDEs that actually have a builtin compiler that can tolerate this and still compile the code. Eclipse is such an example. However, writing something like that is quite complex and is actually harder for PB because of features such as macros and compiler directives.

Instead, the IDE tries its best to (quickly) understand the code without a full compilation. For that, it parses the code and extracts all information that can be understood without context (from a single line of code) into an indexed form for fast search. This organization has the benefit that if a line of code is changed, only this line needs to be parsed again and not the rest of the code which is quite fast. Then when the type of a variable is needed or something like that, the IDE searches through this indexed data to put it in the proper context to answer the question. So it is not a full compilation, but rather a “fuzzy” analysis of the code. In essence, the IDE collects the information it can understand, and stuff that only a compiler would know (like what an expanded macro would look like) is ignored.

So how does this affect Modules? One thing that the IDE does not do is resolve which source file is included where in the final program. It only looks at each source file individually. This was no big limitation before modules existed, but with modules it means you have to follow some rules so the IDE understands where a module begins and ends and what it contains.

These are the rules:

  1. The entire DeclareModule/EndDeclareModule block should be in the same source file.
  2. The entire Module/EndModule is best also written in a single file, but it does not have to be the same one as the one that contains the DeclareModule
  3. At least the file containing the DeclareModule block should be included in the Project and scanned for AutoComplete. You don’t have to include the actual module implementation since the stuff in there is not visible on the outside anyway.
  4. If you use UseModule, all modules that are open within a source file should be opened within that source file. The best was is simply to have the UseModule calls right at the top.
  5. Don’t hide the essential module commands behind macros. The IDE does not expand macros so it will not know that the keywords are in there.

The explanation is rather simple. Consider this code:

DeclareModule Foo
  XIncludeFile "FooDeclaration.pb"
EndDeclareModule

Since the IDE only looks at each source file individually, it does not know that the contents of FooDeclaration.pb are part of the module declaration. So the IDE sees an empty Foo module, plus a file FooDeclaration.pb with stuff that is outside of any module.

In the same way, if you have a UseModule call outside of the current source file, the IDE does not know that stuff from the opened module is available while you edit your file and therefore cannot help you with AutoComplete in this case.

Other languages enforce such rules on the compiler level (one class per file etc). We chose to not limit the compiler in this way so you can organize your code as you like. And you don’t have to follow these rules to use the IDE. It just cannot be as helpful as it could be if you don’t. Anyway, in my opinion these rules also make sense from a code organization standpoint so i don’t think it is that hard to follow them.

 

API Programming: What’s behind the PB GUI objects

For Windows, this question is quickly answered: HWND. Pretty much everything is of this basic “window”-type and most API commands that deal with windows can be used on most GUI objects. You can also find many code examples on the forums that deal with a lot of common API tasks. The situation is different for Linux and OSX. Here, different Gadgets may have different underlying objects and so it is not entirely clear what functions can be used. There are also a lot less examples around for these systems. Of course when developing programs that run on multiple OS, you will need solutions for all of them.

To help this situation a bit, i have compiled a list of what are the underlying objects behind the PB gadgets and other GUI objects. You can get access to these using functions such as GadgetID(), WindowID(), etc.

Disclaimer:

This information is internal stuff. While many of these things haven’t changed in many years, it is still possible that we decide to change something in the future. Also some Gadgets are heavily modified for PB with custom subclasses or callbacks which also may change in the future and break your code. So as usual: Feel free to use this information, but if it breaks, its not our problem.

So here it goes:
Windows Linux OSX
WindowID() HWND GtkWindow WindowRef
MenuID() HMENU GtkMenuBar / GtkMenu MenuRef
StatusBarID() HWND GtkHBox ControlRef (UserPane)
ToolBarID() HWND GtkToolbar HIToolbarRef
FontID() HFONT PangoFontDescription PB internal pointer
ImageID() HBITMAP GdkPixbuf CGImageRef
GadgetID() HWND GtkWidget ControlRef
ButtonGadget “Button” GtkButton / GtkToggleButton PushButton / BevelButton
ButtonImageGadget “Button” GtkButton / GtkToggleButton PushButton / BevekButton
CalendarGadget “SysMonthCal32” GtkCalendar HIView
CheckBoxGadget “Button” GtkCheckButton CheckBox
ComboBoxGadget “ComboBox” / WC_COMBOBOXEX GtkComboBox / GtkComboBoxEntry PopupButton / HIComboBox
ContainerGadget custom class GtkFixed UserPane
DateGadget DATETIMEPICK_CLASS GtkEntry UserPane
EditorGadget RICHEDIT_CLASS GtkTextView UserPane (rendering a TNXObject)
ExplorerComboGadget WC_COMBOBOXEX GekComboBoxEntry HIComboBox
ExplorerListGadget WC_LISTVIEW GtkTreeView DataBrowser
ExplorerTreeGadget WC_TREEVIEW GtkTreeView DataBrowser
Frame3DGadget “Button” / “Static” GtkFrame GroupBox / UserPane / Separator
HyperLinkGadget “Static” GtkButton UserPane
IPAddressGadget WC_IPADDRESS GtkEntry EditUnicodeText
ImageGadget “Static” GtkImage UserPane
ListIconGadget WC_LISTVIEW GtkTreeView DataBrowser
ListIconGadget “ListBox” GtkTreeView DataBrowser
MDIGadget “MDICLIENT”
OptionGadget “Button” GtkRadioButton RadioButton
PanelGadget “SysTabControl32” GtkNotebook Tabs
ProgressBarGadget PROGRESS_CLASS GtkProgressBar ProgressBar
ScrollAreaGadget custom class GtkScrolledWindow UserPane
ScrollBarGadget “SCROLLBAR” GtkVScrollBar / GtkHScrollBar ScrollBar
ShortcutGadget HOTKEY_CLASS GtkEntry EditUnicodeText
SpinGadget UPDOWN_CLASS + “Edit” GtkHBox containing others UserPane containing others
SplitterGadget custom class GtkVPaned / GtkHPaned UserPane
StringGadget “Edit” GtkEntry EditUnicodeText
TextGadget “Static” GtkLabel StaticText
TrackBarGadget TRACKBAR_CLASS GtkVScale / GtkHScale Slider
TreeGadget WC_TREEVIEW GtkTreeView DataBrowser
WebGadget custom class + ActiveX control GtkMozEmbed HIWebView
Notes:
  • Gadgets all have the same basic type noted in the GadgetID() row. However, Gadgets are devided into “window classes” on Windows, or subclasses of the general GtkWidget on Linux.
  • For Gadgets on Windows: If the entry is noted in “” then this is the class name, if it is not then this is the symbolic constant for the class name. You’ll have to look up the text value in the appropriate header files.
  • On OSX, there are no classes for the controls, its all a ControlRef. The names i noted here are derived from the control creation functions (ie CreateTabsControl()). If you are looking for information, you should start there.
  • Gadgets may have multiple classes depending on the flags on creation.
  • On Linux, many gadgets are placed inside their own container to add a frame or catch events. (GtkFrame, GtkEventBox). The GadgetID() command returns the real gadget in this case, not the container.
  • On OSX, the UserPane controls are drawn by PB itself, so you don’t have much ways to modify them other than the PB commands.

Thats it. I hope this information is useful to some people, and hopefully we will see some more cross-platform API examples on the forum in the future 🙂

The cost of empty lists

So now we have the ability to embed lists, maps and arrays inside structures. Its a very useful feature and we can already see people building crazy data structures with all sorts of nesting in them, even tree like structures. I want to talk a little bit about the costs associated with these objects. The fact is, even empty lists or maps take up memory and if you create a lot of them, you should be aware of that. The sizes i give below are all for the x86 versions (so 4 bytes per pointer). For x64, you can usually double that. Not all fields are actually pointer sized, so it is less than that in reality, but it is a good approximation.  My base scenario is an array of a structure type with an embedded list. However, this applies to all other combinations as well.

Structure MyStruct
  Something.i
  List Elements.s()
EndStructure

Dim Table.MyStruct(99)
Automatic initialization on creation

First of all, its important to realize that structure content is automatically initialized on creation. This means that if you create a structured variable with an embedded list, you don’t have to call NewList in order to use the embedded list. However, this also means that simply by the process of creating the structured variable, you also created a (still empty) list which takes up some memory. The same thing applies to my example array: By creating the array with 100 elements (0-99), i also created 100 empty lists.

You could say this is a waste of memory, and you would not be entirely wrong. After all, if not all elements of this array are actually used to store data, that is a lot of unused memory. We discussed this at length when implementing the feature. An alternative approach would have been to leave all such dynamic elements uninitialized on creation and require the user to explicitly call NewList before using them in each element. This would conserve memory when not all array elements are intended to have items in their lists, but it puts the burden to ensure that all dynamic elements are initialized before they are accessed on the user and introduces a whole new source for bugs. Its also extra work for the user considering that if people define such a structure they intend to put things into it, and forcing them to do an extra initialization step each time when PB can easily do it for them is just wasting the programmer’s energy.

So in the interest of having a really easy to use feature we decided to go with automatic initialization. After all, if you really need the full control and do not shy away from some extra work then building your own linked data structures in memory is not that hard. If you want something simple that just works, then this feature is for you. Of course its never wrong to know about the costs so you know what happens when your data set gets large.

Memory usage

How much memory does an empty list take? Aside from the actual space it takes up inside the structure which is 2 pointers for a list (1 pointer for array and map), there is a header structure which is allocated as well. It stores information about the list such as its type (for sorting and cleanup), the pointers to the first and last element in the list as well as cached index information for a fast SelectElement() if possible. The list header is currently 48 bytes on x86. As described in another entry, we also have custom memory allocation for list elements for faster allocation/freeing of memory. These allocation routines also take up 24 bytes of header data per list.

So in my list in array example above i have created 100*12 bytes of actual element data in the array plus another 100*72 bytes for empty lists. While this extra data is still not very much, it is considerably more than the array element data itself, so it will become an issue if you make your array very large. If you knew for example that you will never have more than 18 elements in the embedded list, then you could replace the thing with a static array inside the structure with 18 elements and still have the same memory usage.

Empty maps have a much larger impact. They also come with a header and an allocator header, but an empty map also reserves space for its hash table even if there are no elements in it. The default table size is 512 entries, so that is another 2048 bytes for an empty map on x86.

Arrays are a little different as you can declare inside the structure how many elements it should have on creation. Arrays also come with a header, but nothing else. Here its more important to keep in mind that if the elements of the embedded array in turn contain embedded lists, maps or arrays that you have to take their costs into account as well as they will be initialized too when the embedded array is created.

One final thing to keep in mind is that lists and maps actually allocate the memory for their elements in blocks of 16 elements minimum (for speed and to reduce fragmentation), if you create many lists with just a few elements in them (say 2 per list for a binary tree structure), you end up with used memory of 16 elements per list instead. Note though that this is just allocation, not initialization. The used memory will be only for the 16 elements themselves, not for any contained dynamic list, map or array they may contain.

Bottom line

This post should be seen in the right context. The cost of the embedded object may seem large compared to the size of the structure they are put in, but that doesn’t change the fact that they are still very cheap objects. Lists, maps and arrays are all memory-only objects. This means that all they need is memory, no handles, no system objects, nothing else. Memory is not much of a concern these days, so you can create many of them and even create/destroy them frequently. This really isn’t a problem. Take my above array example: Even if we Dim the array to 10000 elements, we still only used up 820Kb of memory which is not much by today’s standards.

The only place where you really need to concern yourself with this is if your data sets get really large and memory does become an issue. In this case the advice is simple: Avoid “sparse” data structures where a large part of the dynamic elements are allocated but not used. You can also call FreeList(), FreeMap(), FreeArray() or ClearStructure() on any element you do not need. You then have to re-initialize it if you want to use it later with NewList, NewMap, Dim or InitializeStructure() respectively.

Always keep in mind the quote from Donald Knuth on optimization:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.

In this context, this means not to worry about this issue when designing your data structures. Do not get tempted to optimize away a few bytes of extra memory with funny tricks. A clear and simple layout of the data in your program is worth much more than that (and will save you many headaches from chasing bugs). Only if you see that your program really has a memory problem is it time to start thinking of ways to cut down on overhead like this.

In this spirit, have fun designing crazy nested data structures with the new dynamic elements! 🙂

2DDrawing in v4.40 under the hood

A lot changed in the 2DDrawing library with the v4.40 version. In fact, it basically underwent a complete rewrite. Only very few code parts remain from the previous version. Since it has come up in various questions and bug reports on the forum i am going to explain the new design a little bit and what consequences it has on stuff like mixing API and PB commands for drawing.

Lets start off with what the 4.31 library looked like:

The Windows version of the library was based entirely on GDI (the Windows drawing API). GDI allows output on a window, image, printer, sprites and the screen with the same commands so for the most part only the code that set up the drawing needed to be specific and the rest was shared (hence the separation of the output functions like ImageOutput() from StartDrawing()). One detail that a lot of users relied on was the fact that the GDI handle to the drawing context (HDC) was returned for StartDrawing() which allowed to easily mix API and PB commands in the drawing operation even without much knowledge of GDI. The internal data structure of the library was published a long time ago in the PB Library SDK and since the library didn’t change much over time there is still some code that relies on them.

Linux was a different story. Here every output requires its own API (Gtk for window an image, libgnomeprint for printer and some pixel based routines for SDL output). So the Linux library had a plugin like system just like some other PB libraries have to select the right command depending on the output. The Mac OSX version basically had only one output for images. The printer library created a temporary image and sent it to the printer when drawing was done. There was no drawing for windows, sprites or the screen on OSX. The drawing functions shared the pixel based drawing code we used for SDL on Linux.

Why the library was rewritten:

The reason was the alphachannel. As explained in a previous entry, i worked on improving the capabilities of PB when it comes to displaying alphachannel images. After that was finished i felt that at least some support for 2DDrawing was needed in addition to the DrawAlphaImage() command as well. The problem here is that GDI does not support the alphachannel even though it can handle 32bit images. Since Windows 98 you can load images with alphachannel and display them with the AlphaBlend() API but that is pretty much it. Any manipulation of the image with GDI will cause the alpha information to be lost. The same problem exists with Gdk on Linux. The ony drawing functions that exist work on GdkDrawable objects and these do not support the alphachannel. GDI+ is the Windows replacement for GDI which can deal with the alphachannel, but we needed a crossplatform solution to this problem. So we made the decision to create our own drawing functions similar to the pixel based ones for SDL and OSX which could handle the alphachannel in all drawing commands. As you can see from the result i went a step further and added the gradient drawing support (which also would not be possible with the API functions).

The new design:

Since our pixel based routines can only draw on images we now need separate “subsystems” for drawing even on Windows. So now we have this plugin-architecture on all OS. The real drawing functions are called through pointers which are set when the drawing is started so the speed impact is minimal. For those concerned about executable sizes: A drawing subsystem is only linked when the corresponding output function is used in the code. So if you do not use ImageOutput() none of the image drawing code will be linked.

So there is now a GDI subsystem on Windows and the new “PixelBuffer” one. The PixelBuffer subsystem does all its drawing directly on the memory of the output image with no API involvement. The only exception is the drawing of text. We did not want to add the overhead and license troubles by including an external font rendering engine like freetype so DrawText() uses GDI to render the text to a special device context from which the text is then drawn to to the real output with the alphachannel and gradients taken into account. It works in a similar way on the other OS where the native API is used to render the text and then it is copied to the real output. There is of course a speed penalty from this, but it cannot be avoided and it is questionable how much faster a separate rendering engine would be with this task.

Things to watch out for for on Windows:

I tried my best to keep the library backward compatible. If you use PB drawing commands only then there should be no difference at all. If you mix them with API then there are some things to watch out for:

  • As the “HDC result from StartDrawing()” is very commonly used i kept that behavior the same. So even though the PB commands do not use it for the drawing, there is still a device context created and returned from StartDrawing() which you can use to draw on the image with API.
  • You can still mix API drawing functions with PB functions on the same output without problems. The only thing that changed here is that GDI functions that change the global state of the device context no longer affect the PB drawing functions like they used to because the PB functions do not actually used that device context. So a function like SetWorldTransform() will not cause DrawImage() to draw a rotated image anymore. However it should still work if you use the BitBlt() API.
  • You have to be aware that GDI functions will erase the alphachannel. So if you draw on a 32bit PB image and use a GDI function, the modified area will look transparent after that. The best way to avoid this problem is to use 24bit images as output when API commands are involved. (We changed the default image depth to 24bit in 4.40 beta2 to make the transition easier)
  • PB Images are now always DIBs (device independent bitmap) to allow for the pixel based drawing. The #PB_Image_DisplayFormat flag used to create a DDB (device dependent bitmap), but this is no longer supported. #PB_Image_DisplayFormat now has the value 32 to create a 32bit image instead. Some GDI functions (like GetDIBits()) expect a DDB, so you may get trouble there.
  • DrawImage() can still draw API created images (including DDBs and icons). Again you have to be aware that a 32bit API created bitmap will probably have all alpha values as 0 which PB will interpret as fully transparent. Using 24bit is the solution here as well. Also DrawImage() expects DIBs to be bottom-up (the default on Windows). If you use top-down DIBs then they will be drawn upside down. There is no way to avoid that as Windows does not provide a way to find out the pixel orientation of a DIB.
  • If you relied on the library’s internal data structures then you are out of luck. Nothing is as it was before on the library’s internals.

To sum it up: Stick to 24bit images and in most cases mixing PB drawing with API drawing should work just as it did before.

Things to watch out for on Linux:

I am not aware of code that mixed 2DDrawing with API on Linux. StartDrawing() used to return the GdkDrawable handle for Image+WindowOutput() before. It now returns that only for the WindowOutput() as there is no more drawable when drawing on an image. Backward compatibility wasn’t possible here.

One thing to note in general is that PB images are now GdkPixbuf objects and no longer GdkPixmap. This is a big improvement as many Gtk functions expect GdkPixbuf nowadays, so it is actually easier to use PB images with Gtk functions.

Things to watch out for on OSX:

Nothing to say here. Since the 2DDrawing support was so poor before, it only got better with this release. There is now a separate QuickDraw based subsystem for WindowOutput(). Yes i know that QuickDraw is deprecated, but it was much easier to implement this way than to go for a full Quartz based one. I have plans to do a Quartz subsystem one day to have better PrinterOutput() (for the moment, it still works with images as an intermediate).

One thing to note is the new output for sprites and the screen with OpenGL (this applies to all OS): There is no way to modify the data in video memory, so if you call StartDrawing() the whole sprite data or the entire screen content is transferred to main memory for the drawing. This is very slow. So doing 2DDrawing everytime you refresh the screen is too costly. The better way is to draw on sprites just once and then display them over and over. Still, this is better than having no 2DDrawing for sprite/screen at all.

Future plans:

There is still room for optimisation in the new 2DDrawing code. For now the focus was on getting a stable implementation. Also the plan is to eventually have the alpha drawing routines also for SpriteOutput() and ScreenOutput() but this will need some more work to support all possible pixel formats for the screen. Stay tuned…

Is your 64bit program really solid ?

If you ask Windows for memory it will generally start with low addresses first. There is nothing wrong with that. For a 64bit program however, this means that under normal circumstances unless you actually use up 4Gb of memory, your pointers will still fit into a 32bit value. So if you store a pointer into a too small variable (say a long) you will not notice, because the upper half of the 64bit pointer value isn’t even used. So your program may look quite solid and usable right now but if you feed it something like a 4Gb file to load it will start crashing. This may sound like an unlikely case but it will become much more common even for programs that are not meant to deal with large amounts of data, simply because as disk space, memory and computer speeds grow, so do the amounts of data that users deal with.

Anyway, I recently found this link: http://msdn.microsoft.com/en-us/library/bb613473.aspx

Notice at the very bottom the mention of the AllocationPreference registry setting. If you develop 64bit applications (especially if you are porting code from 32bit to 64bit) then this key will help you a great deal. What it does is it instructs Windows to give you high addresses first. The result are pointers that don’t fit into 32bit right from the start. So these errors that usually go unnoticed will blow up right in your face and you can fix them.

The link doesn’t explicitly state the OS this works on but i tested both on XP 64bit and Vista 64bit and it works fine on both. You have to restart the computer after setting the registry value for the change to take effect. You can then verify with a simple program if the high addresses are returned (make sure you compile it with the 64bit compiler 😉 ):

For i = 1 To 10
  a$ + RSet(Hex(AllocateMemory(1000), #PB_Integer), 16, "0") + Chr(13)
Next i
MessageRequester("Pointer values", a$)

In this mode you can easily distinguish a valid pointer from a truncated one by looking at it as hex like above. Since the virtual address space is limited to 8TB on current x64 Windows versions, a pointer allocated in this mode will look something like 000007FFFXXXXXXX because the addresses cannot go any higher. If a pointer got cut somewhere it will either look like 00000000XXXXXXXX or FFFFFFFFXXXXXXXX depending on wether the cut 32bit value was positive or negative (the sign-extension of the negative numbers will create all the F’s).

The PureBasic package itself is a good example of how easily such bugs go unnoticed. We payed a lot of attention to this when porting the libraries and also the programs like IDE, debugger and compiler to 64bit with PB 4.30. There was a lot of testing by the users after that too and PB 4.31 x64 is quite stable. However, if you turn on this mode the 4.31 IDE and debugger will not even start. Even the compiler won’t compile anything. So if you want to try this with your own programs you obviously cannot use PB 4.31 for it. The new 4.40 version was tested with this mode and these bugs are now fixed. Since it is still in beta, you may run into the odd crash though. If this happens please let us know.

Keep in mind that this setting is systemwide. So other x64 programs could well crash on you too if they have such unnoticed bugs.

Have fun fixing a ton of bug you never knew existed 😛

Using PureBasic (32bit) on 64bit Linux

This has been asked a couple of times. Until now i wasn’t much help there because i never tried this myself. So now i tried it and had some success. I was able to compile the PureBasic IDE, which means that pretty much all non-multimedia libs must be working. If you don’t want to wait for PureBasic 64bit for Linux (planned for 4.40) then here is a step by step guide.

There is a guide for Ubuntu 8.10 and OpenSUSE 11.1 64bit (always starting from a clean installation). If you have a slightly different version of one of these distributions then it may work for you as well with some luck. If you have a different distribution, then read this anyway. There are some tips on the bottom on how you could make this work for your distribution as well.

PureBasic on Ubuntu 8.10 64bit

First download and unpack the PureBasic package. Then open a console and set up the environment variables for the compiler and try running it “pbcompiler -h”. If you started from a clean install, then not even this will work (you will get a weird error like “No such file or directory”). So we first need the runtime environment for 32bit programs:

sudo apt-get install ia32-libs

Note that this installs a lot of stuff (120Mb). After this is completed, the compiler should at least be able to run. At this point you should also be able to run the IDE. But even trying to compile an empty program will cause linker errors. So what we need next is the basic gcc and libc6 support for compiling 32bit programs:

sudo apt-get install gcc-multilib libc6-i386 libc6-dev-i386

After that, you should be able to compile your first (console only) programs, and also use the PureBasic debugger.

The next big thing are Gtk and SDL. Unfortunately there are no special 32bit development versions of these packages available (something like “libgtk2.0-dev-i386”), so we have to use some tricks. To compile with PB, we actually only need two things: The library files and the properly working pkg-config sdl-config tools to actually find them (and tell the linker which libraries to link).

Lets start with the configuration tools: here we can actually use the ones that come with the regular 64bit packages, so just install those:

sudo apt-get install libgtk2.0-dev libsdl1.2-dev

If you try to compile a program with a MessageRequester() before and after this step, you will notice that the linker errors are reduced from a huge pile of unresolved symbols to just not finding a compatible library. So all we need now is some compatible library files. It turns out that all the needed libraries are already installed with the “ia32-libs” package, because you can use a dynamic library (.so) for linking as well (the linker will automatically generate a static lib from that). The only thing that is missing really are some symbolic links to help the linker find those dynamic ones, thats all.

I found a nice script here that creates these links and extended it by a few more links needed for PureBasic. Put the following into a textfile and execute it as root (“sudo sh myfile.sh”):

#!/bin/sh
cd /usr/lib32
for lib in gio-2.0 gdk-x11-2.0 atk-1.0 gdk_pixbuf-2.0 \
           pangocairo-1.0 pango-1.0 pangoft2-1.0 \
           gobject-2.0 gmodule-2.0 glib-2.0 gtk-x11-2.0; do
  ln -s -f lib$lib.so.0 lib$lib.so
done
ln -s -f libcairo.so.2 libcairo.so
ln -s -f libfreetype.so.6 libfreetype.so
ln -s -f libz.so.1 libz.so
ln -s -f libfontconfig.so.1 libfontconfig.so
ln -s -f /usr/lib32/libX11.so.6 /usr/lib32/libX11.so
ln -s -f /usr/lib32/libXrender.so.1 /usr/lib32/libXrender.so
ln -s -f /usr/lib32/libXext.so.6 /usr/lib32/libXext.so
ln -s -f libgthread-2.0.so.0 libgthread-2.0.so
ln -s -f libSDL-1.2.so.0 libSDL.so
ln -s -f libstdc++.so.5 libstdc++.so

Thats it! Now most libraries should work fine. The IDE compiled fine with this setup. As far as i understand the page above, this step will actually not be needed for the next stable Ubuntu release.

Some PureBasic libs that have special dependencies will still not work however. The commands from the following libraries will compile/link fine, but not work with this setup:

  • Sound, SoundPlugin, Module – I am not sure why these do not work. Could be because i am running VMWare and have no tools installed yet. Somebody else will have to try this.
  • All 3D Engine related libraries – I did not look deeper into why the engine does not load. Maybe another time. It actually starts loading, but you never see a screen.
  • the WebGadget() – This does not even work out of the box on some 32bit systems. I did not bother looking deeper into that.

The following Libraries will not even link. The reason is simply that the “ia32-libs” package does not include 32bit versions of them. To get them to work you would have to compile them manually in 32bit mode. Maybe i will try that another time.

  • PrinterOutput() – needs libgnomeprint2.2
  • Movie lib – needs libxine
  • Database lib with ODBC – needs libiodbc2 (Note: SQLite works fine as it is included in PB)
PureBasic on OpenSUSE 11.1 64bit

I started from a clean installation with Gnome desktop and no other packages added in the setup.

Open the “Software Management” in Yast and install the following packages:

  • gcc-32bit
  • gtk2-devel
  • sdl-devel-32bit
  • libgnomeprint-devel
  • libgnomeprint-32bit
  • libgnomeprintui-devel
  • libgnomeprintui-32bit
  • libxine-devel
  • libxine1-32bit

Then copy the code below to a text file and run it in a shell as root (“sudo sh myfile.sh”):

#!/bin/bash
cd /usr/lib
for lib in gio-2.0 gdk-x11-2.0 atk-1.0 gdk_pixbuf-2.0 \
           pangocairo-1.0 pango-1.0 pangoft2-1.0 \
           gobject-2.0 gmodule-2.0 glib-2.0 gtk-x11-2.0; do
  ln -s -f lib$lib.so.0 lib$lib.so
done
ln -s -f libcairo.so.2 libcairo.so
ln -s -f libfreetype.so.6 libfreetype.so
ln -s -f libfontconfig.so.1 libfontconfig.so
ln -s -f libstdc++.so.6 libstdc++.so
ln -s -f /lib/libz.so.1 /lib/libz.so
ln -s -f /lib/libgcc_s.so.1 /lib/libgcc_s.so
ln -s -f libgthread-2.0.so.0 libgthread-2.0.so
ln -s -f libgnomeprint-2-2.so.0 libgnomeprint-2-2.so
ln -s -f libart_lgpl_2.so.2 libart_lgpl_2.so
ln -s -f libxml2.so.2 libxml2.so
ln -s -f libgnomeprintui-2-2.so.0 libgnomeprintui-2-2.so
ln -s -f libgnomecanvas-2.so.0 libgnomecanvas-2.so
ln -s -f libxine.so.1 libxine.so
ln -s -f libpng12.so.0 libpng.so

This should do the trick to allow PB to compile. Note that unlike on Ubuntu, the Movie lib and PrinterOutput() will work just fine here. The other mentioned library problems are the same as on Ubuntu though.

PureBasic on other 64bit Linux distributions

Ok, i cannot do this for all distributions, but the steps on your Linux of choice should be fairly similar, just with slightly different package/filenames:

  1. Install the 32bit runtime environment (sometimes this is preinstalled). You know you succeeded when the IDE and compiler can be started.
  2. Install what is needed to build basic 32bit programs. This is usually a special gcc package and a special libc6-dev package. You know you succeeded if you can compile an empty sourcecode without errors.
  3. Install the required libraries. If they exist as a 32bit-devel package then that is perfect. If they only exist as a 32bit runtime version (maybe included in a larger 32bit package) then install that plus the 64bit devel package. Then add any missing symbolic links in your 32bit lib directory. Usually the linker looks for a file like “libSDL.so” and the folder will contain a “libSDL1.2.so.0” or similar. (usually just with the extra .0) Just add a link as the scripts above do. If there is not even a 32bit runtime package for a library then you can only try compiling them manually. Use a small testcode like a MessageRequester() and try the above until the linker no longer complains about any libraries.

The required packages (all as devel version, Gtk and SDL are the most important):

  • libgtk2.0
  • libsdl1.2
  • libstdc++ (for ScintillaGadget)
  • libgnomeprint2.2 and libgnomeprintui (for PrinterOutput())
  • libxine (for Movie lib)
  • libiodbc (for ODBC database)

Thats pretty much it. I hope you can get this working on your system. If there are more problems just ask. I would not call myself a Linux expert but i will try to help if possible. If you get PureBasic working on another distribution, please publish your exact steps somewhere so others can benefit too.

Debugging tips – Part 2: Stack problems

Another common problem that the debugger does not really identify are stack problems. Fortunately, other than the heap problems discussed before, these are not hard to narrow down.

Symptoms:
  • A piece of code works on the main level, but fails when called inside a procedure or inside a Select block
  • The crash is usually when leaving the procedure (ProcedureReturn or EndProcedure)
  • Usually a call to a Dll or imported function is involved
Reason:

The x86 processor has several different so called calling conventions which define how a function call is made (were the arguments go and who is responsible for cleaning up after the call). In fact everybody can invent their own convention for this, and quite a number of compilers do this for performance reasons. (PureBasic itself does it for Library functions written in ASM. They use the stdcall convention, but receive their first argument in the EAX register instead of the stack). When calling an external function, the calling convention must match or there will be problems like the one we have here.

The most popular conventions on x86 are cdecl and stdcall. cdecl is the standard used by the C language, stdcall is most used by Windows dlls. PureBasic uses stdcall by default in most places. They both provide the function arguments on the stack in reverse order. The only main difference is in the fact that cdecl requires the caller to remove the arguments from the stack and stdcall requires the function to do this before returning (there is also a difference in symbol naming, but thats not relevant here). So if the caller assumes the wrong calling convention then either both try to remove the parameters from the stack or nobody does it. In both cases the stack is not the same after the function call than before, and this is a big problem.

Here we get to the reason why it works outside of a procedure and not inside. A call to a procedure stores the address where execution continues after the procedure on the stack as well. If a function call inside the procedure leaves the stack in a mess, then the ProcedureReturn cannot find the address to continue and uses a wrong value instead which leads to a crash. Outside of a procedure, nobody will notice the stack mess, as the stack is only used for function calls.

Solution:
  • PB provides support for both stdcall (the default) and cdecl calling conventions (the functions/keywords with a C in the name)
  • If you used CallFunction() or CallFunctionFast() try CallCFunction() or CallCFunctionFast() instead. (these are the cdecl versions of the same function)
  • If you used Prototype, try PrototypeC instead.
  • If the crash happens in a callback procedure which you passed to an external function, try defining the callback with ProcedureC instead.
  • Also check if the number of arguments and the argument types match, as this can lead to the same problem. (remember that CallFunction() and CallFunctionFast() default to the long type, so for quads and doubles you have to use prototypes for a correct call.)

Some rules of thumb:

  • On Windows all Dlls that come with Windows use the stdcall convention. Most other Dlls follow this model, so there are only a few that use cdecl. With static libraries (used with Import/ImportC), the division is not so clear. Just try it.
  • On Linux / Mac OSX, the dominating calling convention is cdecl. So pretty much every library you use will require the C type of function.

Stack problems can also arise when InlineASM code is used and the stack is changed. Those that use this should know what they are doing though. The x64 and PowerPC processors do not have the calling convention problems by the way, as the force one common calling convention on everybody. So a parameter mismatch is the most probable explanation here.

A sidenote: Window callbacks

On Windows, another condition which can cause a crash on the ProcedureReturn keyword is the window callback. This is not a stack issue then though. What it means is that the crash happened somewhere in the PureBasic event processing that happens after your window callback returns. The debugger just shows the ProcedureReturn line because this was the last correctly executed line in non-library code.

What this probably means is that you did something with Windows API calls that conflicts with the way PureBasic handles its events. This may be in the window callback directly, or by modifying some Gadget/Window related data that PureBasic is expecting to remain unchanged. If you have no clue what causes this, just post it on the forum. There are a lot of capable people around that can help with these things.

Debugging tips – Part 1: Heap corruption

If you are wondering why this blog is silent for so long, its because we are in the beta phase of 4.30, and writing about bugfixing and documentation work is just boring.

I want to talk a bit about the most common problems i have seen on the forums where the PureBasic debugger is of not much help and which therefore leave most users quite confused. The usual reaction is to post it as a bugreport. So in the future, if somebody posts a bugreport about AllocateMemory(), you can point him to this article and it should explain everything 🙂

Symptoms:
  • A crash at AllocateMemory() or FreeMemory() even though the given input is valid.
  • A crash on a simple string assignment
  • Modifying the code in seemingly unrelated places makes the problem go away.
Reason:

First of all: AllocateMemory() is never the problem. Its a direct wrapper to the HeapAlloc() API function and also a heavily used function. If it had a bug we would know by now. What you have gotten yourself into here is what is called a heap corruption. You destroyed part of the data Windows uses to manage allocated memory.

When Windows allocates memory, it keeps a data structure to manage the allocated memory (usually 12bytes on 32bit Windows). This data structure is normal, writable memory which means that you will not get any access error when accidentally writing over it. It is nowhere specified where this data is kept, but it is a fact that it sometimes ends up right after your allocated memory buffer. Now, if you happen to write over the end of this buffer by just a few bytes, you destroy the heap structure without getting any error. The crash only happens later when another attempt to allocate or free memory causes Windows to examine this piece of heap data and crash due to an invalid pointer. This fact, that the cause of the problem and the actual crash are in different places makes this kind of bug so hard to debug.

Why does this not happen always when you overwrite a buffer ?

Getting the heap data right at the end of the allocated buffer is a very rare condition. Windows often rounds the allocated buffer size up to page boundaries or for alignment purposes, so you often get more memory than you asked for, which makes a small overwrite have no effect. Another scenario is that the memory after your allocated buffer is simply marked as invalid, in which case you get an error when trying to write to it. Which of these scenarios actually happens depends on the sequence of memory allocations done by your program, which means that if you comment another totally unrelated program part, you may change the allocation sequence and the problem seemingly disappears. (Note that in this case only the symptom disappears, not the problem of writing over the end of a buffer.)

Solution:

As we saw above, the problem is not where the crash is. Fortunately, Windows provides a function to check if the heap structures. Below is a piece of code that can be used to check the most used memory heaps in PureBasic: (this works in PureBasic Windows 32bit and 64bit)

Procedure _TestHeaps(File$, Line)
  Protected StringHeap, MemoryBase, MemoryHeap

  CompilerIf #PB_Compiler_Processor = #PB_Processor_x86
    !extrn _PB_StringHeap
    !extrn _PB_Memory_Heap         

    !mov eax, dword [_PB_StringHeap]
    !mov [p.v_StringHeap], eax
    !mov eax, dword [_PB_MemoryBase]
    !mov [p.v_MemoryBase], eax
    !mov eax, dword [_PB_Memory_Heap]
    !mov [p.v_MemoryHeap], eax
  CompilerElse
    !extrn PB_StringHeap
    !extrn PB_Memory_Heap

    !mov rax, qword [PB_StringHeap]
    !mov [p.v_StringHeap], rax
    !mov rax, qword [_PB_MemoryBase]
    !mov [p.v_MemoryBase], rax
    !mov rax, qword [PB_Memory_Heap]
    !mov [p.v_MemoryHeap], rax
  CompilerEndIf

  If HeapValidate_(StringHeap, 0, 0) = 0
    MessageRequester("StringHeap corrupted !", File$+" : "+Str(Line))
  EndIf

  If HeapValidate_(MemoryBase, 0, 0) = 0
    MessageRequester("MemoryBase heap corrupted !", File$+" : "+Str(Line))
  EndIf

  If HeapValidate_(MemoryHeap, 0, 0) = 0
    MessageRequester("AllocateMemory heap corrupted !", File$+" : "+Str(Line))
  EndIf
EndProcedure

Macro TestHeaps
  _TestHeaps(#PB_Compiler_File, #PB_Compiler_Line)
EndMacro 

Steps to find the bug:

  • Place a TestHeaps call right before the line that crashes. If you get one of the message requesters, you have a heap corruption. If not, then the problem is something else and the above code will not help.
  • Start placing TestHeaps calls in places that are executed before the crashing line. Start with only a call every bunch of lines and narrow it down later.
  • You need to find the line of code, where TestHeaps reports nothing before, and reports a heap corruption after. This is the line that causes all the problems.
  • Make sure you remove all this test code after fixing the bug, as it can have a big performance hit on the program (see below).

So why doesn’t the PureBasic debugger make this check automatically ?

The reason is that HeapValidate() has an effect on the future run of the program. The documentation sais that it degrades performance and that this effect can last until the end of the program. My guess is that the check for a valid heap somehow reorganizes it into a less efficient state which means that future allocations will be slower. This is why this check is not done by the debugger. Maybe there will be an option for this somewhen in the future. who knows ?