Author Archives: Fred

Turning ten !

If you take a quick look to the history page on www.purebasic.com, you will see this:

PureBasic 1.60 (AmigaOS) (09/09/2000)

9 september 2000. Seriously. So PureBasic is officially haunting the web since 10 years, which is kind of amazing, for different reasons. Let’s try to step back, catch a time machine, and see what and how it happened.

I got the chance to have a computer very young, at twelve. It was an amstrad CPC 6128, with a very low resolution but colored screen and strange disk format. My main attract was games, games and games. I spent countless hours playing many games (Barbarian anyone ?!). On a side note, in the book sold with the computer, there were some listings to type yourself a game using the build-in BASIC (an old school one, without procedure or even label, so you have to specify the line number to jump at when using GOTO or GOSUB). Out of curiousity, i tried to make it work. It was a ‘dart’ game, and it took forever to type back the printed listing to the computer. The result was rather disappointing, and the game was pretty bad.

A few years later, a friend got an Amiga 500 and when i looked at the games running on it, it was just unbelievable. The graphics and the sounds were way above anything else, and the games availables were just stunning. After a lot of negotiations, my brother and I get the precious, and a brand new Amiga 500+ hit our desk ! We used it a lot as an advanced gaming console, and had a lot of fun. Then the number of games released on the platform decreased to almost zero very quickly due to Commodore bankruptcy, and then my main point of interest changed.

I started programming somewhen in 1994. At first, it was some small tools programmed in, guess it, Basic. When the PureBasic project started, early in 1998, it was a quick attempt to support new PowerPC processors freshly available on the Amiga in a Basic language, with a syntax based on BlitzBasic which was discontinued. It was very small, with few features and very tied to the AmigaOS. It got about 200 build-in commands, an ‘IDE’ which was mostly only a text editing area and a ‘Compile/Run’ menu and a rudimentary runtime debugger. The whole commands were programmed in 680×0 asssembly code, which was the rules on the Amiga to have optimized code. And as the AmigaOS didn’t have memory protection, better check your code twice before running it, especially when dealing with pointers, or it will immediately reboot ! So far i got some positive comments from the internet community (yes, we had internet in 1998, even if it wasn’t fast) about PureBasic and the choice was made to do a Windows and Linux version as soon as possible. I was studing computer science and learned the C programming language, so it was a good time to recode everything in a ‘portable’ way. I didn’t knew so much about real world programming and i made a lot of mistake when porting it in C, and hopefully learned quite a bit from it.

Late in 2000, we publicly released the first Windows version and it was a real achievement, for me and for the future of PureBasic. The real deal started here, as the Windows audience was way larger than the AmigaOS one. We got a lot of feedback (compared to what i expected) and it was a great source of motivation. André jumped into the boat and since then handle the german part of PureBasic. The plan was to add as much as commands as possible to reach the critical mass which could attract users and let them do a complete software without hassle. The releases iteration was much faster than what it is now, because the shipped code was lower quality and we didn’t do synchronized releases between platforms. The development of the different versions on linux, windows and amigaos was very time consuming, as the build was not unified and nothing was virtualized. On Windows, all the libraries were coded in x86 assembly, which was obviously a wrong choice: much longer/harder to develop, not significant size/speed gain most of the time and more important not portable on x64. Timo joined the team and helped a lot to refactor the libs and the build structure.

Then we added quite a number of features, dealing with a lot of API and we reached in 2006 another important mark in PureBasic history: the 4.00 release.  I think than the project took a professional turn at this point. It took 6 years to reach it but then we had some interesting features:

– Flexible commandset, with more than 900 build in commands
– Support for the 3 major operating system (Windows, Linux, OS X), with an high compatibility level
– Buildin unicode & thread support
– Full featured, modern IDE
– Robust debugger, which comes in 3 flavours (console, standalone GUI and IDE integrated)

Four years later, we are at 4.51, which is a general refinement of 4.00. More commands, more stability, more fun. Internally, we have enhanced many things, making our development easier, faster and more robust. Time flies, but it’s great to see what have been achieved. We have very high expectations for the future, there is so much things left to do !

So far, thank you to all the PureBasic folks which are actually using it and make all this possible. See you in 10 years…

Purify it !

What a wierd title. Or may be not. Depending of your experience, you may or may not know these life-saver tools, when strange things happen to your program and you have absolutely no clue about what the hell is going on. No starting point to look at. It sometimes happens, sometimes not, as if the computer was an organic lifeform. Here enters purify like programs (‘valgrind’ on linux, ‘purify’ or ‘bound checker’ on Windows) , which adds realtime checks around almost every kind dynamically allocated memory block to detect the root of evils: buffer overflows.

You may think: “well that’s cool but i use PureBasic, and barely never dynamically allocated memory block, as all is managed by natives libraries”. You are right. But may be you use pointers, or even OS calls. And this is where things got a bit more complicated, as pointers is the ultimate power in programming. And as all power, it has to be used wisely. Are don’t get me wrong: you will use them wisely. Until something is changed in your code which impacts that and suddenly the pointer gets crazy, without any notification. On a big program, it could be very long to isolate the bug. I have some ‘souvenirs’ of very vicious cases which took me several days to fix. And as PureBasic executables uses a custom debugger format, it is not compatiable with any of the purifier tools. Too bad. Until now.

Before going further in the details, let’s try to really understand how a purifier works. Basically, it puts ‘cookies’ around every potientally dangerous code items: variables, arrays, strings, dynamically allocated memory block. What’s a cookie ? It’s just a predefined value (usually 32 bits), often something funny like $BADBEEF or $BADF00D. At every executed line, the purifier (which holds a list of all the installed cookies address) will check them, and if one is broken will stop. That means than something has erased this area which should never have been erased. Ever. That’s a bit like a code integrity check. As we could imagine, the bigger your program is, the slower the purifier will be. And it can be very very slow, that’s why it’s possible to tune it to check only some kind of data, and at which rate.

Consider this code:

Procedure a()

a.l = 10

*CrazyPointer.Long = @a
*CrazyPointer+1
*CrazyPointer\l = 152  ; stack corruption

EndProcedure
a()

This is an obvious error. But it does go unnoticed when running in standard release/debug mode. On some more retricrive OS, it could even crash. If you activate the purifier it catch it immediately and report a stack corruption. Let’s try another less obvious one:

*LocalDrive = AllocateMemory(#MAX_PATH)
GetModuleFileName_(0, *LocalDrive, #MAX_PATH)

This snippet uses Windows API, and get back the executable name. So far so good. Now what happen if you change the program mode from ASCII to Unicode ? You end up with a buffer which is twice smaller as requested, and chances are high than you will be hit with a nasty buffer overflow. Again the purifier will detect it as the ‘cookie’ put at the end of the allocated memory block will be overwritten.

And there is quite some cases where it can happen: a structure too small to recieve the expected data, wrong associated structure to a pointer, pointer manipulation error, wrong API call, etc. For most of them, the purifier will catch the error. There are some limitations tough: threaded program could report wrong line and  if a pointer is really filled with a random value, it won’t be noticed by it (but probably by the debugger as IMA – Invalid Memory Access).

The purifier is now available in PureBasic 4.50 !

PureBasic Museum

We just opened the PureBasic museum, a section in your download account to get access to previous version of PureBasic, up to PureBasic Windows 2.02. That’s a kind of time travel for new developper who missed the start of this big adventure !

Windows 95, the comeback !

I know, i know. But there is some stuffs i really hate, and one of them is when you upgrade your compiler version, and suddenly a software stop working on a specific plateform. That happened to the purebasic compiler executable, when we upgraded to VC8. Basically, VC8 adds a dependency to every executable created with it, which is not supported in Windows 95: the famous IsDebuggerPresent() API function (found in Kernel32.dll). So when you launch it on Win95, it just fails at loading time, telling there is a missing symbol. Well, who care about Win95 anymore, you think. I do. For fun. And it’s not a missing symbol (which is useless BTW, and added under the hood by linking libcmt.lib – which is requiered) which will stop PureBasic to work on lovely oldies plateforms.

So the fun began. I asked to my best virtual friend – google – to gently find me a workaround. And guess what ? There was one. But you had to link statically the c runtime and hack a lot with sources. I didn’t want to do that. So i tried to mess with the kernel32 import library (kernel32.lib) to remove the IsDebuggerPresent object. Total failure. Microsoft ‘lib’ tool just see tons on ‘Kernel32.dll’ object in the kernel32.lib when you launch:

prompt> lib /LIST kernel32.lib
KERNEL32.dll
KERNEL32.dll
KERNEL32.dll
KERNEL32.dll
KERNEL32.dll
KERNEL32.dll
....
prompt>

Ok. Not very useful to remove a specific object. Sounds like a bug btw…

So i tried with ‘polib’, which comes from PellesC package, without more success. It successfully list the objects name, but unfortunately can’t remove it. Well, i was out of luck. So i created a .def with kernel32 exports and tried to use ‘lib’ to create an import lib. But it doesn’t work as well, for some reason (seems like the lib created isn’t a real dll import lib). Finally i ended up to create a fake kernel32.dll, and reuse the import lib created by the linker. And voilà, it worked.. At least VC8 linked and complained about the missing IsDebuggerPresent import, so the hard part was done. Now we just need a small stub in our executable and it’s over. Here is the stub:

// The cool Win95 hack, so the compiler can launch on it, even compiled with VS2005
// Basically, this function is needed by the libcmt.lib/gs_report.obj
//
BOOL __stdcall _imp__IsDebuggerPresent()
{
return FALSE;
}

Timo kindly tested the new build and PureBasic launched and worked correctly on an old Win95. Victory !

Automation to the rescue !

PureBasic has grown step by step, like every other long time going project,  adding support for more platforms accross the time. With the forthcoming release (4.30), we have added 2 official distributions: Windows x64 and MacOS x86. This led us to a problem: 4 additional packages to manage (full and demo for each of the platforms). Until now, we handled the packaging phase on demand, with own created tools (on Windows) and scripts on Linux and OS X. This was very time consuming to do all the packaging stuff, as you have to boot the correct OS, update all the SVN repositories and pray to have everything still compiling/working. Then you have to check the resulting package, check if nothing is missing and finally upload it on the remote purebasic.com site. As you can see, there was room for improvement. And we did. Let’s see how we processed:

1) We killed the Windows ‘update’ package. It was a .zip package which contained the delta files against PureBasic version… 2.00. Yeah, it was  8 years ago. This package was useful at time, as it was way smaller, and the internet connections were much slower. Now, the .zip is actually bigger than the regular install shield package, as every single files have changed since the version 2.00, so there is no delta anymore. We modified the install shield package to support updating. Result: 1 package removed.

2) So starting from now, we will have 10 packages (Windows x86/x64, Linux x86, MacOS X PowerPC/x86 – all in both demo and full version). The obvious idea was to use virtualization to have a single OS running all the PureBasic supported platforms. We got an offer from a PureBasic user to lend us a specific computer to use it as a build server: that’s was a perfect timing.  We installed Windows Vista x64 on it, and setup the according virtual machines.

3) The next step was to script everything to do all the packaging phase automatically. Here are (roughly) the actions needed:

- Update all the SVN repositories (we have 5 differents)
- Compiles the compiler, libraries, IDE and all provided tools (DocumentationMaker, LibraryMaker, PureUnit etc.) from the sources
- Run the test units for the libraries and the compiler
- Create the documentation
- Create the full package
- Create the demo package (derived from the full package)
- Upload the packages on purebasic.com

And you can repeat it for every OS. We used ‘bash’ (from cygwin on Windows and the native one on linux/osx) to have a common base for scripting. Windows could natively compiles x64 and x86 version of PureBasic, so this case was sorted. Then we used ssh (plink.exe from the putty package) to connect to the various running virtual machines and launch remotely the appropriate packaging script. The automatic ssh connection is handled with ssh key authentification. After a week of work, we had our automated build server, and believe us, that’s really a time and headache saver ;). Ha, the whole build time for all versions when nothing is built is over 6h. On a Core2 duo – 3Gb. When all is built (ie: no change in the libraries and so), it’s just under one hour to build everything and put it on the web site – which is quite ok.

With this new tool, we will be able to publish synchronized alpha, beta and official releases without having to crawl all over the OS and we hope it will increase the overall PureBasic release quality. It should also help when new platforms will be available (linux/osx x64 anyone ?).

Have fun !