Author Archives: freak

The Future of the Desktop & PureBasic

Another year has gone by, an its another guest post by Syed Ibrahim (@TI-994A) that saves us from going post-less in 2018. Thanks 🙂

Another year has swooshed by, and 2019 will see PureBasic turning 21, and still going strong.

New Technologies & Platforms

As we all know, the current eco-system is choc-full of APIs, SDKs, new languages, and even more RAD tools and frameworks, all struggling to stay ahead of the curve, and remain relevant. An industry once dominated by the desktop has now splintered into an array of platforms, from the web, to the mobile phone, to smart devices and wearables, and soon on IOT everywhere.

The technologies vying to be the development tool of choice are also ever-growing, with Microsoft jumping into the Mac & iPhone arena with their acquisition of Xamarin, Apple and Google introducing BASIC-like languages with Swift & Kotlin to lure more developers onto their platforms, and Google introducing their very own cross-platform mobile SDK called Flutter, for reasons still quite unclear (for Fuchsia, perhaps?).

The Mighty Desktop

Despite all these shake-ups, the desktop platform will continue to be the indispensable backbone of the consumer, business, and educational sectors. They will also be the cornerstone of every new emerging platform, as integral hosts and back-end servers.

Sadly however, this foundational platform has been neglected, seeing almost little to no progress in development tools or technologies. While all the players have been busy in the cross-platform race, they offer only clumsy and bloated solutions for desktop development as part of so-called single code-base suites. Needless to say, the results have been dismal.

While there have been many champions in the desktop development platforms over the decades, many have fallen behind, and even more have simply thrown in the towel. Loyal users have been left frustrated and in the lurch, having to migrate their skills and code bases to other development platforms and languages.

The Stable & Solid Tool of Choice

The one (yes, ONE!) exception to this is PureBasic, which has strongly been forging ahead on all three of the major desktop platforms (Windows, MacOS, and Linux), growing and evolving year after year, staying relevant and on top of all the current technologies, always offering its loyal user-base the best development experience. PureBasic has continuously remained on the cutting-edge for the past twenty years, diligently incorporating new features and functions as they become available on the respective platforms, all the while keeping close to the core, to deliver the most optimized and native results. And all this under a single and fully portable code base!

Today, no other development tool offers the same native feature-set on as many platforms, with the speed, size, and performance that PureBasic does. Its binaries remain highly competitive even with the likes of platform-specific compilers, but with the ease and simplicity of the BASIC language. Even among the most formidable players, stability has been a major issue. Google has been developing and touting multiple languages, from Go, to Kotlin, and now Dart. And Microsoft axed the original Visual Basic DOS/Windows suite hardly a decade from its inception.

The Future of PureBasic

Our collective hats off to TEAM PureBasic as we look forward to the new and exciting magic that they hold for us all. At 21, PureBasic is the oldest actively-developed cross-platform tool on the market today. At this pace, we can be sure of many, many Merry Christmases to come!

Support for Ascii compilation ends after the next LTS cycle

As Fred has explained here, supporting both the creation of ascii and unicode executables in the compiler is becoming a burden and we would like to end support for ascii compilation in order to streamline the library code and make it easier to maintain the PureBasic package in the future. However, the above thread has shown that people wish for a longer transition period, and we would like to honor this wish.

So we decided to remove the ability to compile in ascii mode in the PB version that follows after the next LTS version (that is, the LTS version coming after the 5.2x LTS cycle ends).

There are no exact dates for the releases, but the timeline looks like this:

  • The current 5.2x LTS version will be supported until at least 09/17/2015
  • After that date, a new LTS version will be released with support for 2 years. This version will still have full ascii compilation support.
  • The first non-LTS version released after the next LTS cycle starts will have no ascii compilation support

This means that the first non-LTS version without ascii support will be released in about a year. By staying with LTS releases, you have the ability to use an ascii mode compiler with a fully supported PB version for at least another 3 years starting from today. This should give enough time for a smooth transition.

No changes will be made to the language or data types (The .c, .a and .u as well as the pseudo-types and library commands will remain as they are). The only difference is that the “compile in unicode mode” switch in the compiler will be permanently set to “on”.

Please understand that we do listen to the concerns voiced in the discussion thread and that we do not make this decision lightly. I think we have a quite good track record of supporting older technology as is evidenced by the very long support for Windows 9x which just recently ended and by the fact that we still support Windows XP even after MS dropped all support for it. However, in order for us to be able to introduce new technologies (like x64 compilation in the past, or now support for the web with Spiderbasic), we simply cannot support old stuff indefinitely. In order to move forward into the future, we have to leave some stuff behind from time to time. We hope you can understand that.

PureBasic 5.30 beta 1 released

The first public beta version of the upcoming PureBasic version is available for download on your download account.

Some notable new features include:

  •  A new JSON library including functions to serialize complex data structures directly from PB code to JSON and back. The same functionality has also been added to the XML library.
  • New functions for the 3D engine.
  • The IDE and debugger are now full unicode programs
  • Improvements for the IDE such as a “plain-text” editing mode, the highlighting of repeated occurrences of the current selection, a new ‘Issues’ tool to track TODOs in the source code and more.
  • A new /PREPROCESS compiler flag which will generate a PB code with all compiler directives and macros resolved.
  • … and much more!

The english documentation is already up to date for the new features, so you can find a description and examples there.

This version also includes some changes that may break compatibility with existing code:

  • The (X)IncludeFile and IncludeBinary have changed their behavior for relative paths.
  • The #PB_Event_SizeWindow and #PB_Event_MoveWindow events are no longer reported in realtime in the main event loop. This is to reduce problems with flickering on resize. To get realtime events you have to use the BindEvent() function and a callback in the future.
  • CreateXMLNode() has a new mandatory parameter.[/list]

This is regular release which means it brings new features and will have the regular support cycle. The 5.2x LTS release will continue to be maintained and a 5.23 LTS release is planned for shortly after this release becomes final.

Here is the full list of changes:

Added: New "Issue" tool for IDE to build todo/issue lists easily from comments.
Added: JSON library
Added: ParseXML(), ComposeXML(), InsertXMLArray/List/Map/Structure(), ExtractXMLArray/List/Map/Structure() to XML lib
Added: OpenGLGadget() with native opengl commands ande constants support for Windows, OSX and Linux
Added: ExamineRegularExpression() and related commands to process regex matches step by step (with group support)
Added: ClipOutput(), UnclipOutput(), SetOrigin(), GetOriginX(), GetOriginY()
Added: GetWindowData(), SetWindowData()
Added: AllocateStructure(), FreeStructure()
Added: #PB_Default support to WindowsBound() to reset min/max size
Added: 'Format' parameter to Read/WriteProgramString(), WriteProgramStringN() and ReadProgramError()
Added: /PREPROCESS compiler flag to create a big single source with all macros, compilerif and file include resolved. Can be combined with /COMMENTED to get the original source with comments as well.
Added: optional '#Server' parameter to NetworkServerEvent() to check events only on a specific server
Added: #PB_String_NoZero flag support to PokeS() to avoid writting the ending null character
Added: #PB_Enumeration support for Defined()
Added: Optional 'Type' parameter to CreateBillboardGroup()
Added: BillboardGroupCommonDirection(), BillboardGroupCommonUpVector()
Added: #PB_Entity_MinVelocity and #PB_Entity_ForceVelocity to SetEntityAttribute()
Added: SetMaterialAttribute() with #PB_Material_DepthCheck and #PB_Material_DepthWrite constants
Added: #PB_Material_DepthCheck support for GetMaterialAttribute().
Added: Engine3DStatus() with these constants: #PB_Engine3D_NbRenderedTriangles, #PB_Engine3D_NbRenderedBatches
       #PB_Engine3D_CurrentFPS, #PB_Engine3D_MaximumFPS, #PB_Engine3D_MinimumFPS, #PB_Engine3D_AverageFPS, #PB_Engine3D_ResetFPS
Added: #PB_Absolute / #PB_Relative support to CameraDirectionX/Y/Z(), CameraX/Y/Z(), EntityX/Y/Z(), LightX/Y/Z(), LightDirectionX/Y/Z()
       BillBoardGroupX/Y/Z(), NodeX/Y/Z(), ParticleEmitterX/Y/Z() and FetchOrientation()
Added: ParticleSpeedFactor(), DisableParticleEmitter()
Added: GetEntityCollisionMask(), GetEntityCollisionGroup(), SetEntityCollisionFilter()
Added: WaterHeight(), FreeWater()
Added: Fully unicode IDE
Added: Highlighting of repeated occurrences of the currently selected word in the IDE
Added: Plain-text editing mode to edit non-PB files in the IDE
Added: 'Issues' IDE tool to collect and display TODO/FIXME markers inside the code
Added: Ctrl+E and Ctrl+Shift+E shortcut to align/shift comments in a selected code block
Added: Ctrl+M and Ctrl+Shift+M shortcut to select the current code block (repeated presses select the next code block)
Added: PopupMenu to IDE error log for clear/copy operation
Added: %HOME and %PROJECT to IDE tool commandline options, added PB_TOOL_Project to available env vars
Added: Automatic code indentation can align comments at the end of code lines in the IDE
Added: AutoComplete remembers last selection for Structure/Module AutoComplete
Added: Context sensitivity for current module/procedure for variable display and expression eval in the debugger

Changed: FormatXML() with #PB_XML_ReFormat no longer adds newlines inside single-line elements for a more readable output
Changed: DeleteElement() now returns the data pointer to the new current element (if any)
Changed: SetXMLAttribute() to accept newline characters in attributes (will be encoded as character entities)
Changed: Added a mandatory "name" parameter to CreateXMLNode() as some parser needs it at node creation time
Changed: The way (X)IncludeFile and IncludeBinary works: it's now relative to the file which contains these statements (which is easier to handle)
Changed: #PB_Event_SizeWindow and #PB_Event_MoveWindow are no more realtime on Windows, use BindEvent() to get real time update. It should fixes ugly flickering when realtime resizing on Windows.
Changed: DataSection label within Procedure are now local labels.
Changed: ASM local label prefix has been changed from "l_" to "ll_" ("ll" for local label), to avoid possible clash with main labels.
Changed: #PB_LinkedList constant has been renamed to #PB_List for better consistancy
Changed: Removed the "Billboard" parameter from AddBillboard() as it was not used. Now returns the new billboard index.
Changed: Updated Scintilla to version 3.4.2

Removed: MaterialDepthWrite() (replaced by SetMaterialAttribute())
Removed: CountRenderedTriangles() and Engine3DFrameRate(), replaced by Engine3DStatus()

Getting the best out of the IDE when using Modules

Since the last beta version (5.20 beta8) the IDE finally has support for Modules as well (AutoComplete, QuickHelp, ProcedureBrowser etc). Please test these features well, as it was quite a change in the IDE to implement this. Now, to get the best experience when using the IDE and Modules together, there are some simple rules that have to be followed. I will explain them below. But first, let me explain why these limitations exist.

As the AutoComplete feature became smarter and more helpful in past versions, it had to grow from a simple scanning of the source for strings into a full grown analysis of the source code. This is especially required for modules, because the IDE needs to know what procedures/constants/structures are exposed from each module in order to present the correct choices in AutoComplete. So the IDE in essence needs to understand your code. The best way to fully understand code is to compile it. The IDE does not actually compile the code for the following reasons:

  • Speed: The AutoComplete feature cannot introduce delays because that would make it unusable. So compiling the entire code from beginning to end is not an option.
  • The code is usually not compilable anyway: Remember, the purpose of the IDE is to edit code. While you edit a line, it will be in a state that cannot be compiled most of the time. In fact, if you look at all characters typed in a source code, the instances in which that typed character puts the source in a compilable state are rather rare compared to those that put it in a state with errors (from a compiler perspective).
  • There are IDEs that actually have a builtin compiler that can tolerate this and still compile the code. Eclipse is such an example. However, writing something like that is quite complex and is actually harder for PB because of features such as macros and compiler directives.

Instead, the IDE tries its best to (quickly) understand the code without a full compilation. For that, it parses the code and extracts all information that can be understood without context (from a single line of code) into an indexed form for fast search. This organization has the benefit that if a line of code is changed, only this line needs to be parsed again and not the rest of the code which is quite fast. Then when the type of a variable is needed or something like that, the IDE searches through this indexed data to put it in the proper context to answer the question. So it is not a full compilation, but rather a “fuzzy” analysis of the code. In essence, the IDE collects the information it can understand, and stuff that only a compiler would know (like what an expanded macro would look like) is ignored.

So how does this affect Modules? One thing that the IDE does not do is resolve which source file is included where in the final program. It only looks at each source file individually. This was no big limitation before modules existed, but with modules it means you have to follow some rules so the IDE understands where a module begins and ends and what it contains.

These are the rules:

  1. The entire DeclareModule/EndDeclareModule block should be in the same source file.
  2. The entire Module/EndModule is best also written in a single file, but it does not have to be the same one as the one that contains the DeclareModule
  3. At least the file containing the DeclareModule block should be included in the Project and scanned for AutoComplete. You don’t have to include the actual module implementation since the stuff in there is not visible on the outside anyway.
  4. If you use UseModule, all modules that are open within a source file should be opened within that source file. The best was is simply to have the UseModule calls right at the top.
  5. Don’t hide the essential module commands behind macros. The IDE does not expand macros so it will not know that the keywords are in there.

The explanation is rather simple. Consider this code:

DeclareModule Foo
  XIncludeFile "FooDeclaration.pb"

Since the IDE only looks at each source file individually, it does not know that the contents of FooDeclaration.pb are part of the module declaration. So the IDE sees an empty Foo module, plus a file FooDeclaration.pb with stuff that is outside of any module.

In the same way, if you have a UseModule call outside of the current source file, the IDE does not know that stuff from the opened module is available while you edit your file and therefore cannot help you with AutoComplete in this case.

Other languages enforce such rules on the compiler level (one class per file etc). We chose to not limit the compiler in this way so you can organize your code as you like. And you don’t have to follow these rules to use the IDE. It just cannot be as helpful as it could be if you don’t. Anyway, in my opinion these rules also make sense from a code organization standpoint so i don’t think it is that hard to follow them.


PureBasic 4.60 Beta1 released!

Its been quiet around here for a while, but now there is big news: The first public beta of the upcoming 4.60 release is out!

This release is mostly about the 3D side of things, but there are new features for application programmers as well. The OGRE engine was updated to version 1.7.0 and we switched the physics engine from ODE to Bullet. Furthermore, there are a ton of new 3D libraries and commands. You can see the full list below. The 3D commands are not documented yet, but we will put up some examples to demonstrate them.

A special thanks to G-Rom and TMyke from the french forum for their help on the new 3D commands!

On the application side of things, the most notable addition is the CanvasGadget() command. It is a simple drawing surface that provides detailed mouse and keyboard events to easily implement custom data views or controls (all cross-platform of course). You can see this new gadget in action in the IDE already: The new file panel and the new color picker tool both make use of it. Other than that, there are some smaller new commands like additions to the LinkedList library for example. All new non-3D commands are already documented in the manual.

The IDE got some additions as well, like the ability to build projects directly from the command-line, or the option to be warned when a file is changed on disk while open in the IDE. Unfortunately, the automation framework that I talked about here on the forum did not make it into this release. It wasn’t finished in time, and its not a big enough priority to justify delaying this release any longer.

Finally, we are in the process of changing the PureBasic documentation to be more precise and readable when it comes to function parameters and return values. This is still a work in progress which should be finished for the final release. Comments on the new structure are welcome.

These are the changes:

– added CanvasGadget(), CanvasOutput()
– added SetGadgetItemData() for PanelGadget
– added MoveElement(), MergeLists(), SplitList() commands
– added RandomizeList(), RandomizeArray()
– added PushListPosition(), PopListPosition(), PushMapPosition(), PopMapPosition()
– added ImageID parameter to OpenSubMenu()
– added #PB_ListIcon_ThreeState and #PB_ListIcon_Inbetween
– added #PB_Tree_ThreeState and #PB_Tree_Inbetween
– added crossplatform ComboBox events
– added ‘Joint’ library
– added ‘SpecialEffect’ library
– added ‘StaticGeometry’ library
– added CameraPitch(), CameraRoll(), CameraYaw(), SwitchCamera()
– added ApplyEntityForce(), ApplyEntityImpulse(), EntityPitch(), EntityRoll(), EntityYaw(), GetEntityAttribute(),
GetEntityMaterial(), SetEntityAttribute(), SetEntityMaterial()
– added LightDiffuseColor(), SpotLightRange(), LightLookAt(), LightPower(), DisableLightShadows()
– added MaterialDepthWrite(), MaterialSelfIlluminationColor(), MaterialShininess(), GetScriptMaterial()
– added BuildMeshShadowVolume(), CreateLine3D(), CreateCube(), CreateSphere(), CreateCylinder(), CreatePlane(), AddSubMesh()
MeshVertexCount(), UpdateMeshBoundingBox(), MeshRadius(), AddMeshVertex(), MeshVertexNormal(), MeshVertexColor()
MeshVertexTextureCoordinate(), AddMeshFace(), FinishMesh(), NormalizeMesh(), SaveMesh(), SetMeshMaterial(), SubMeshCount()
– added NodePitch(), NodeRoll(), NodeYaw()

– changed FindString() ‘StartPosition’ parameter to be optional
– changed WebGadget to use WebKitGtk on Linux
– changed ContainerGadget with #PB_Container_Borderless to no longer add a 2px invisible border on Linux
– changed EntityPhysicBody(), CreateLight(), RenderWorld(), ShowGUI, WorldShadows()

– removed: Get/SetEntityMass(), Get/SetEntityFriction()
– removed: SetMeshData() (temporary, needs to be updated)

– added IDE options for commandline project building
– added monitoring of files for changes on disk while they are open in the IDE
– added FilePanel custom implementation with Drag & Drop, Dropdown menu, scrolling also on OSX
– added Diff tool for directories/files
– added Html help viewer for Linux/OSX
– added Help viewer in toolspanel
– new Color Picker tool
– enabled font selection in IDE for OSX
– ProjectPanel now remembers node expanded states

– added maximize button to all debugger windows
– added all IsXxx() and XxxID() functions to the expression parser (for data breakpoints)

– The manual is being updated to a new format which more clearly describes
individual function parameters and return values. This is not yet complete,
but will be finished for the final release.

Visit the announcement on the PureBasic forums here.

API Programming: What’s behind the PB GUI objects

For Windows, this question is quickly answered: HWND. Pretty much everything is of this basic “window”-type and most API commands that deal with windows can be used on most GUI objects. You can also find many code examples on the forums that deal with a lot of common API tasks. The situation is different for Linux and OSX. Here, different Gadgets may have different underlying objects and so it is not entirely clear what functions can be used. There are also a lot less examples around for these systems. Of course when developing programs that run on multiple OS, you will need solutions for all of them.

To help this situation a bit, i have compiled a list of what are the underlying objects behind the PB gadgets and other GUI objects. You can get access to these using functions such as GadgetID(), WindowID(), etc.


This information is internal stuff. While many of these things haven’t changed in many years, it is still possible that we decide to change something in the future. Also some Gadgets are heavily modified for PB with custom subclasses or callbacks which also may change in the future and break your code. So as usual: Feel free to use this information, but if it breaks, its not our problem.

So here it goes:
Windows Linux OSX
WindowID() HWND GtkWindow WindowRef
MenuID() HMENU GtkMenuBar / GtkMenu MenuRef
StatusBarID() HWND GtkHBox ControlRef (UserPane)
ToolBarID() HWND GtkToolbar HIToolbarRef
FontID() HFONT PangoFontDescription PB internal pointer
ImageID() HBITMAP GdkPixbuf CGImageRef
GadgetID() HWND GtkWidget ControlRef
ButtonGadget “Button” GtkButton / GtkToggleButton PushButton / BevelButton
ButtonImageGadget “Button” GtkButton / GtkToggleButton PushButton / BevekButton
CalendarGadget “SysMonthCal32” GtkCalendar HIView
CheckBoxGadget “Button” GtkCheckButton CheckBox
ComboBoxGadget “ComboBox” / WC_COMBOBOXEX GtkComboBox / GtkComboBoxEntry PopupButton / HIComboBox
ContainerGadget custom class GtkFixed UserPane
DateGadget DATETIMEPICK_CLASS GtkEntry UserPane
EditorGadget RICHEDIT_CLASS GtkTextView UserPane (rendering a TNXObject)
ExplorerComboGadget WC_COMBOBOXEX GekComboBoxEntry HIComboBox
ExplorerListGadget WC_LISTVIEW GtkTreeView DataBrowser
ExplorerTreeGadget WC_TREEVIEW GtkTreeView DataBrowser
Frame3DGadget “Button” / “Static” GtkFrame GroupBox / UserPane / Separator
HyperLinkGadget “Static” GtkButton UserPane
IPAddressGadget WC_IPADDRESS GtkEntry EditUnicodeText
ImageGadget “Static” GtkImage UserPane
ListIconGadget WC_LISTVIEW GtkTreeView DataBrowser
ListIconGadget “ListBox” GtkTreeView DataBrowser
OptionGadget “Button” GtkRadioButton RadioButton
PanelGadget “SysTabControl32” GtkNotebook Tabs
ProgressBarGadget PROGRESS_CLASS GtkProgressBar ProgressBar
ScrollAreaGadget custom class GtkScrolledWindow UserPane
ScrollBarGadget “SCROLLBAR” GtkVScrollBar / GtkHScrollBar ScrollBar
ShortcutGadget HOTKEY_CLASS GtkEntry EditUnicodeText
SpinGadget UPDOWN_CLASS + “Edit” GtkHBox containing others UserPane containing others
SplitterGadget custom class GtkVPaned / GtkHPaned UserPane
StringGadget “Edit” GtkEntry EditUnicodeText
TextGadget “Static” GtkLabel StaticText
TrackBarGadget TRACKBAR_CLASS GtkVScale / GtkHScale Slider
TreeGadget WC_TREEVIEW GtkTreeView DataBrowser
WebGadget custom class + ActiveX control GtkMozEmbed HIWebView
  • Gadgets all have the same basic type noted in the GadgetID() row. However, Gadgets are devided into “window classes” on Windows, or subclasses of the general GtkWidget on Linux.
  • For Gadgets on Windows: If the entry is noted in “” then this is the class name, if it is not then this is the symbolic constant for the class name. You’ll have to look up the text value in the appropriate header files.
  • On OSX, there are no classes for the controls, its all a ControlRef. The names i noted here are derived from the control creation functions (ie CreateTabsControl()). If you are looking for information, you should start there.
  • Gadgets may have multiple classes depending on the flags on creation.
  • On Linux, many gadgets are placed inside their own container to add a frame or catch events. (GtkFrame, GtkEventBox). The GadgetID() command returns the real gadget in this case, not the container.
  • On OSX, the UserPane controls are drawn by PB itself, so you don’t have much ways to modify them other than the PB commands.

Thats it. I hope this information is useful to some people, and hopefully we will see some more cross-platform API examples on the forum in the future 🙂

How we make decisions

Since we are on the issue of bug reports (see last post), here is how we made the decision to put “[Done]” in the title of fixed bugreports:

[12:32] <fr34k> this is the point were a bugtracker would be more helpful. there we could filter all solved bugs easily :)
[12:35] <AlphaSND> we already talked about that
[12:35] <AlphaSND> btw, we could change the title of the topic with [Fixed] in it, no ?
[12:35] <fr34k> true
[12:36] <AlphaSND> ok, let's do that starting from now
[12:36] <fr34k> thats a good idea actually, as often people keep posting after the "fixed", so it is no longer obvious from the last post
[12:37] <fr34k> what do we do with stuff that we determine not to fix? (not a bug, not solvable)
[12:37] <fr34k> should be marked as well
[12:37] <AlphaSND> yes
[12:37] <AlphaSND> we could put [Issue] -
[12:37] <fr34k> issue ?
[12:37] <AlphaSND> and move that in Coding Question
[12:38] <AlphaSND> Issue :p
[12:38] <AlphaSND> or [Feature] -
[12:38] <fr34k> lets just put [done] ... its shorter than [fixed] anyway :D
[12:38] <AlphaSND> [Done]
[12:38] <AlphaSND> ok
[12:39] <fr34k> [done] :p
[12:39] <AlphaSND> [Done] Damit !
[12:39] <AlphaSND> :p
[12:39] <fr34k> why ?
[12:40] <AlphaSND> because
[12:40] <fr34k> lets do it differently, so we see who fixed it :D
[12:40] <AlphaSND> man, let's be consistent :p
[12:40] <fr34k> berikco has to write [donE] :D
[12:40] <AlphaSND> :D
[12:40] <fr34k> i noticed, you also always write "Fixed" in the posts, i write "fixed."
[12:41] <fr34k> its not the beginning of a sentence, so there is no reason to write it capital. and its one more key to press damit :p
[12:42] <AlphaSND> It's a full sentence !
[12:42] <AlphaSND> Fixed.
[12:42] <AlphaSND> Uppercase, verb and dot
[12:42] <AlphaSND> :p
[12:43] <fr34k> lol, where have you been when grammar has been tought at school? :D
[12:43] <fr34k> coding probably
[12:43] <AlphaSND> :)
[12:44] <fr34k> now what? :)
[12:44] <AlphaSND> now what what ?
[12:44] <fr34k> i just fixed a bug, what do i write ?
[12:44] <fr34k> the rebell in me wants to write [done] :D
[12:45] <AlphaSND> [Done] -
[12:45] <AlphaSND> i'm the older here :p
[12:45] <fr34k> lol
[12:46] <fr34k> why the " - " btw ? 3 more keys!
[12:46] <AlphaSND> mannnnn !
[12:46] <AlphaSND> to differentiate of the original topic :)
[12:47] <fr34k> hey, i am just trying to optimze here
[12:47] <fr34k> "[Done] - 4.10 and 4.20 Beta 2 - Odd bug"
[12:47] <fr34k> "[done] 4.10 and 4.20 Beta 2 - Odd bug"
[12:47] <fr34k> so much better
[12:47] <AlphaSND> if you fix the bug faster than typing 2 two keys, ok
[12:47] <AlphaSND> the first one is indeed much more sexy ;)
[12:48] <fr34k> compromize: either the D or the "-" has to go ;)
[12:49] <AlphaSND> man, you're damn cra
[12:49] <AlphaSND> zy
[12:49] <fr34k> oh well, i have to go, food time. we'll continue this! :)
[12:49] <AlphaSND> let's remove the - then
[12:49] <fr34k> ha, success :D
[12:49] <AlphaSND> and eidt the 3 i did
[12:49] <fr34k> ok
[13:00] <AlphaSND>
[13:00] <AlphaSND> could you look at it, i can't find what's wrong with it
[13:21] <fr34k> re
[13:28] <fr34k> the mdi one is "[donE]" :D
[13:29] <AlphaSND> mouarf :)

In the end, i am glad we dropped the ” – “, because i am already having a hard enough time putting the “[Done]” in some of the bugreport titles. People use too descriptive titles in the bug forum it seems. Anyway, we are usually very much on the same page when it comes to the direction that PureBasic should take so big discussions are quite rare. But if we disagree, it gets tough! 😀

A Bug’s Life

Ok, its a cheesy title but it fits. I was asked to write about the process with which we handle bugs. The following is roughly how i handle this. I can’t speak for Fred but i don’t think the approaches are too different. This post is also intended as a guide on how to write effective bug reports about PureBasic, as the report can make a big difference here.

Each bug starts its life when it is posted in the bug reports section. If you want your bug report to be seen, write it in the (english) forums.  People sometimes try to tell me about bugs when they meet me on IRC, but this is a very bad idea. If i am unable to address the issue immediately the report will be buried in the log files and most likely forgotten. I am usually busy while i am on IRC so the chance to get something fixed immediately is very slim. Post it on the forum and it won’t be forgotten. Then we can discuss further details on IRC if you happen to catch me. The english forums is where i manage the bug reports. Anything reported somewhere else (IRC, email, german forums) starts at a huge disadvantage.

The first stage after a bug is posted could be called the “initial viability test”. I read every bug report that is posted the next time i visit the forums. I am not on the forum to specifically fix bugs at this point though so this is just a quick check. The relatively obvious non-bugs get filtered here. For example when the behavior is intentional or somebody confused a feature request with a bug. There are also a number of things that people often think are wrong where it is really just a lack of understanding (every few months somebody thinks GetCurrentDirectory() is wrong for example).

At this point i also run any provided example code if possible (if i have a copy of PureBasic in reach). This is why providing an example code to show the bug is very important even if it is just 5 lines of code and you think it should be plainly obvious. Its not that i am lazy. Remember that in this first stage i am not concentrating on fixing bugs. I am just browsing the forum in the lunch break for example. I am not going to write any test code at this stage, however if i just have to fire up the IDE and run it then i usually do that. When a bug can be reproduced at this stage, it starts its life with a big head start compared to the other reports because once i actually get around to fixing bugs i tend to start with these as i already have a place to start looking.

I usually do not comment the bug report at this point (unless it falls in one of the non-bug categories above) so if you post a report and do not hear anything for a while don’t be alarmed. I did read it and it will not be forgotten. Sometimes it takes very long for a bug to get actually fixed, sometimes it is done really fast. This varies a lot as do the reason for why some bugs take longer to fix. Sometimes we just don’t have a solution. Sometimes the solution would require a larger redesign of a library and is therefore postponed to when other planned changes to that library will be implemented. Sometimes the required work just outweighs the usefulness of the bugfix. In the later category i tend to give priority to bugs that cannot be easily worked around by the user. If a bug can be easily worked around with a few lines of code (but may be would be much more work to fix internally) then it may take longer than others. There is really no general rule here.

This brings me to the issue of “bumping” bug reports. Our usual response is “don’t bump” and sometimes it sounds a bit angry which some people might not understand so i will explain it a bit. To me bumping comes across as a bit rude and i am often very annoyed by it (hence the response). Its like saying “stop working on other stuff and fix my specific problem”. I can understand that a certain bugreport may be important to you and the lack of feedback from our side can be frustrating at times. But try to see the whole thing from my point of view: I am confronted with a large list of bugreports (from multiple OS) and i am trying to get them fixed as best as i can. My time is limited and i can only work on one thing at a time. It may be that i just spent a lot of time trying to fix one or more other bugs and then a comment like “still not fixed” or simply “bump” on a bugreport is just frustrating. I am well aware of the status of that bug because as i said above, i read everything that is posted in the bugs section. Bottom line: if a bug really is important to you and you think it should get more attention then ask nicely, don’t just bump. Explaining your reasons why it is important to you also helps, because if something is really a showstopper for you i may try to make an extra effort to get it fixed. In any case it gets you much further than just making me angry with a “bump” post, thats for sure.

Ok, back to the life of a PB bug. I do most of my bugfixing in larger batches at once (recently fixed 30 bugs for the OSX version in one weekend). You will notice this by a lot of [Done] appearing in the bugreport section. Sometimes Fred and I even have a kind of race about it to see who can dominate the bugs form (i own the OSX one right now obviously). For this i make a list of what i want to get fixed from the forum, sort it by priority and get to work on it. The method for fixing a bug is always the same, even for the very obvious ones: I first have to see the effects of the bug in action. Even if it is pretty clear from the bug description where the problem is in the actual code, i still have to make sure i can see whatever the problem is myself. This is the only way to really know that i fixed the bug afterwards.

This reproduction-phase is the most important (and often most difficult) one. Once again, if you provided a working code example that i could verify when i first read your report this problem is already solved, which is why i tend to fix such bugs first. If not then here is where the testing begins. You can make this alot easier for me by providing as much information as possible, such as the exact OS you are using. GUI related bugs are very often specific to some OS, or even to a specific set of settings. So include this information in your post.

My development PC runs Vista 64bit, but I also run VMWare and have a lot of different Windows versions set up for testing. The same thing goes for Linux: Linux distributions are so many and released so frequently that I usually cannot keep up with the latest release of the popular distributions. I am not somebody to install updates all the time. I do my development on OpenSUSE 11 right now. However, if you include your exact distribution and version, then i will download that and setup a VM for it in no time for testing (thanks to fast internet and VMWare this is really no big deal). With all this VM testing there is one common thing: If i have to test on a specific OS version, i most likely test in a clean install with just enough stuff installed to run PB. So if you have made any large changes (especially on Linux) from the base install which you might think could have an influence on the problem then please include that information.

The most important thing however in the whole reproduction-phase is to know what i am trying to reproduce. It may be plainly obvious to you what the bug you post about is, but it may not be to me, so always tell us what you think the problem is (don’t just post code). I have to know what i am looking for after all. It may also be that the bug does not appear with my testing configuration at all, so the code alone is no help here. Always include these two things in your report: What do you see when you run the code, and what do you think should happen instead. The clearer the report is the easier it is to work on it.

If i cannot reproduce it, i will post asking for more information or feedback from others. Here it helps if others respond with their results and also their system configuration. This usually helps a lot in figuring out bugs that are OS or configuration specific. I have seen posters who did not want any feedback from other users, probably because they feared that the existence of a workaround for their problem may diminish their chances of getting their problem fixed. It is the opposite actually: It would not be the first time that a posted workaround helped a lot in fixing a bug or implementing a new feature. Although it must be said that the existence of a workaround does not necessarily mean that it is a good idea to do it that way in the PB library. In any case, feedback on bugreports or feature requests is always welcome as long as it adds extra information. A simple “+1” is usually not helpful (except when the question was “can anybody else reproduce this”). A bug generally does not get fixed based on how much people provide feedback, but especially when we have trouble reproducing the problem then more information from others certainly helps.

If everything fails and i simply cannot get the bug to appear on any of my systems then things get tricky. I can do things like build special debug versions of libraries or the IDE so the user can help me track it down, but it involves a lot of back and forth. I try to avoid this if possible, but  from time to time there is just no other way.

When the bug is successfully reproduced, it is time to hunt down the cause and fix it. There is really not much to say about this part. It really depends on the kind of problem at hand. It is usually a back and forth between modifying the PB code that shows the problem and the code of the library in question until the problem is found.

When the problem is finally fixed, i do the same steps/run the same code i used to see the bug in the first place once again to make sure it is gone. Depending on the kind of bug, i may add a routine to our unit tests (which are run by Fred’s build script before every release) to make sure it does not come back. I don’t do that too often though, as most of the times it does not really make sense. For example, a bug in any of the many GUI libraries cannot be automatically tested because they need user interaction. Also, many bugs are quite unique and not very likely to appear again, so unittests are a bit of a waste of time here. Its different for the compiler. Here it is much easier to have work in a different area cause bugs that were once fixed to reappear, so Fred maintains a set of unittests for many compiler features. Anyway, this is the end of the PB bug. Now its the obligatory “fixed.” post (or “Fixed” as Fred would write :)) and that is it. One more bug in the “[Done]”-category.

The cost of empty lists

So now we have the ability to embed lists, maps and arrays inside structures. Its a very useful feature and we can already see people building crazy data structures with all sorts of nesting in them, even tree like structures. I want to talk a little bit about the costs associated with these objects. The fact is, even empty lists or maps take up memory and if you create a lot of them, you should be aware of that. The sizes i give below are all for the x86 versions (so 4 bytes per pointer). For x64, you can usually double that. Not all fields are actually pointer sized, so it is less than that in reality, but it is a good approximation.  My base scenario is an array of a structure type with an embedded list. However, this applies to all other combinations as well.

Structure MyStruct
  List Elements.s()

Dim Table.MyStruct(99)
Automatic initialization on creation

First of all, its important to realize that structure content is automatically initialized on creation. This means that if you create a structured variable with an embedded list, you don’t have to call NewList in order to use the embedded list. However, this also means that simply by the process of creating the structured variable, you also created a (still empty) list which takes up some memory. The same thing applies to my example array: By creating the array with 100 elements (0-99), i also created 100 empty lists.

You could say this is a waste of memory, and you would not be entirely wrong. After all, if not all elements of this array are actually used to store data, that is a lot of unused memory. We discussed this at length when implementing the feature. An alternative approach would have been to leave all such dynamic elements uninitialized on creation and require the user to explicitly call NewList before using them in each element. This would conserve memory when not all array elements are intended to have items in their lists, but it puts the burden to ensure that all dynamic elements are initialized before they are accessed on the user and introduces a whole new source for bugs. Its also extra work for the user considering that if people define such a structure they intend to put things into it, and forcing them to do an extra initialization step each time when PB can easily do it for them is just wasting the programmer’s energy.

So in the interest of having a really easy to use feature we decided to go with automatic initialization. After all, if you really need the full control and do not shy away from some extra work then building your own linked data structures in memory is not that hard. If you want something simple that just works, then this feature is for you. Of course its never wrong to know about the costs so you know what happens when your data set gets large.

Memory usage

How much memory does an empty list take? Aside from the actual space it takes up inside the structure which is 2 pointers for a list (1 pointer for array and map), there is a header structure which is allocated as well. It stores information about the list such as its type (for sorting and cleanup), the pointers to the first and last element in the list as well as cached index information for a fast SelectElement() if possible. The list header is currently 48 bytes on x86. As described in another entry, we also have custom memory allocation for list elements for faster allocation/freeing of memory. These allocation routines also take up 24 bytes of header data per list.

So in my list in array example above i have created 100*12 bytes of actual element data in the array plus another 100*72 bytes for empty lists. While this extra data is still not very much, it is considerably more than the array element data itself, so it will become an issue if you make your array very large. If you knew for example that you will never have more than 18 elements in the embedded list, then you could replace the thing with a static array inside the structure with 18 elements and still have the same memory usage.

Empty maps have a much larger impact. They also come with a header and an allocator header, but an empty map also reserves space for its hash table even if there are no elements in it. The default table size is 512 entries, so that is another 2048 bytes for an empty map on x86.

Arrays are a little different as you can declare inside the structure how many elements it should have on creation. Arrays also come with a header, but nothing else. Here its more important to keep in mind that if the elements of the embedded array in turn contain embedded lists, maps or arrays that you have to take their costs into account as well as they will be initialized too when the embedded array is created.

One final thing to keep in mind is that lists and maps actually allocate the memory for their elements in blocks of 16 elements minimum (for speed and to reduce fragmentation), if you create many lists with just a few elements in them (say 2 per list for a binary tree structure), you end up with used memory of 16 elements per list instead. Note though that this is just allocation, not initialization. The used memory will be only for the 16 elements themselves, not for any contained dynamic list, map or array they may contain.

Bottom line

This post should be seen in the right context. The cost of the embedded object may seem large compared to the size of the structure they are put in, but that doesn’t change the fact that they are still very cheap objects. Lists, maps and arrays are all memory-only objects. This means that all they need is memory, no handles, no system objects, nothing else. Memory is not much of a concern these days, so you can create many of them and even create/destroy them frequently. This really isn’t a problem. Take my above array example: Even if we Dim the array to 10000 elements, we still only used up 820Kb of memory which is not much by today’s standards.

The only place where you really need to concern yourself with this is if your data sets get really large and memory does become an issue. In this case the advice is simple: Avoid “sparse” data structures where a large part of the dynamic elements are allocated but not used. You can also call FreeList(), FreeMap(), FreeArray() or ClearStructure() on any element you do not need. You then have to re-initialize it if you want to use it later with NewList, NewMap, Dim or InitializeStructure() respectively.

Always keep in mind the quote from Donald Knuth on optimization:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.

In this context, this means not to worry about this issue when designing your data structures. Do not get tempted to optimize away a few bytes of extra memory with funny tricks. A clear and simple layout of the data in your program is worth much more than that (and will save you many headaches from chasing bugs). Only if you see that your program really has a memory problem is it time to start thinking of ways to cut down on overhead like this.

In this spirit, have fun designing crazy nested data structures with the new dynamic elements! 🙂