Monthly Archives: August 2009

2DDrawing in v4.40 under the hood

A lot changed in the 2DDrawing library with the v4.40 version. In fact, it basically underwent a complete rewrite. Only very few code parts remain from the previous version. Since it has come up in various questions and bug reports on the forum i am going to explain the new design a little bit and what consequences it has on stuff like mixing API and PB commands for drawing.

Lets start off with what the 4.31 library looked like:

The Windows version of the library was based entirely on GDI (the Windows drawing API). GDI allows output on a window, image, printer, sprites and the screen with the same commands so for the most part only the code that set up the drawing needed to be specific and the rest was shared (hence the separation of the output functions like ImageOutput() from StartDrawing()). One detail that a lot of users relied on was the fact that the GDI handle to the drawing context (HDC) was returned for StartDrawing() which allowed to easily mix API and PB commands in the drawing operation even without much knowledge of GDI. The internal data structure of the library was published a long time ago in the PB Library SDK and since the library didn’t change much over time there is still some code that relies on them.

Linux was a different story. Here every output requires its own API (Gtk for window an image, libgnomeprint for printer and some pixel based routines for SDL output). So the Linux library had a plugin like system just like some other PB libraries have to select the right command depending on the output. The Mac OSX version basically had only one output for images. The printer library created a temporary image and sent it to the printer when drawing was done. There was no drawing for windows, sprites or the screen on OSX. The drawing functions shared the pixel based drawing code we used for SDL on Linux.

Why the library was rewritten:

The reason was the alphachannel. As explained in a previous entry, i worked on improving the capabilities of PB when it comes to displaying alphachannel images. After that was finished i felt that at least some support for 2DDrawing was needed in addition to the DrawAlphaImage() command as well. The problem here is that GDI does not support the alphachannel even though it can handle 32bit images. Since Windows 98 you can load images with alphachannel and display them with the AlphaBlend() API but that is pretty much it. Any manipulation of the image with GDI will cause the alpha information to be lost. The same problem exists with Gdk on Linux. The ony drawing functions that exist work on GdkDrawable objects and these do not support the alphachannel. GDI+ is the Windows replacement for GDI which can deal with the alphachannel, but we needed a crossplatform solution to this problem. So we made the decision to create our own drawing functions similar to the pixel based ones for SDL and OSX which could handle the alphachannel in all drawing commands. As you can see from the result i went a step further and added the gradient drawing support (which also would not be possible with the API functions).

The new design:

Since our pixel based routines can only draw on images we now need separate “subsystems” for drawing even on Windows. So now we have this plugin-architecture on all OS. The real drawing functions are called through pointers which are set when the drawing is started so the speed impact is minimal. For those concerned about executable sizes: A drawing subsystem is only linked when the corresponding output function is used in the code. So if you do not use ImageOutput() none of the image drawing code will be linked.

So there is now a GDI subsystem on Windows and the new “PixelBuffer” one. The PixelBuffer subsystem does all its drawing directly on the memory of the output image with no API involvement. The only exception is the drawing of text. We did not want to add the overhead and license troubles by including an external font rendering engine like freetype so DrawText() uses GDI to render the text to a special device context from which the text is then drawn to to the real output with the alphachannel and gradients taken into account. It works in a similar way on the other OS where the native API is used to render the text and then it is copied to the real output. There is of course a speed penalty from this, but it cannot be avoided and it is questionable how much faster a separate rendering engine would be with this task.

Things to watch out for for on Windows:

I tried my best to keep the library backward compatible. If you use PB drawing commands only then there should be no difference at all. If you mix them with API then there are some things to watch out for:

  • As the “HDC result from StartDrawing()” is very commonly used i kept that behavior the same. So even though the PB commands do not use it for the drawing, there is still a device context created and returned from StartDrawing() which you can use to draw on the image with API.
  • You can still mix API drawing functions with PB functions on the same output without problems. The only thing that changed here is that GDI functions that change the global state of the device context no longer affect the PB drawing functions like they used to because the PB functions do not actually used that device context. So a function like SetWorldTransform() will not cause DrawImage() to draw a rotated image anymore. However it should still work if you use the BitBlt() API.
  • You have to be aware that GDI functions will erase the alphachannel. So if you draw on a 32bit PB image and use a GDI function, the modified area will look transparent after that. The best way to avoid this problem is to use 24bit images as output when API commands are involved. (We changed the default image depth to 24bit in 4.40 beta2 to make the transition easier)
  • PB Images are now always DIBs (device independent bitmap) to allow for the pixel based drawing. The #PB_Image_DisplayFormat flag used to create a DDB (device dependent bitmap), but this is no longer supported. #PB_Image_DisplayFormat now has the value 32 to create a 32bit image instead. Some GDI functions (like GetDIBits()) expect a DDB, so you may get trouble there.
  • DrawImage() can still draw API created images (including DDBs and icons). Again you have to be aware that a 32bit API created bitmap will probably have all alpha values as 0 which PB will interpret as fully transparent. Using 24bit is the solution here as well. Also DrawImage() expects DIBs to be bottom-up (the default on Windows). If you use top-down DIBs then they will be drawn upside down. There is no way to avoid that as Windows does not provide a way to find out the pixel orientation of a DIB.
  • If you relied on the library’s internal data structures then you are out of luck. Nothing is as it was before on the library’s internals.

To sum it up: Stick to 24bit images and in most cases mixing PB drawing with API drawing should work just as it did before.

Things to watch out for on Linux:

I am not aware of code that mixed 2DDrawing with API on Linux. StartDrawing() used to return the GdkDrawable handle for Image+WindowOutput() before. It now returns that only for the WindowOutput() as there is no more drawable when drawing on an image. Backward compatibility wasn’t possible here.

One thing to note in general is that PB images are now GdkPixbuf objects and no longer GdkPixmap. This is a big improvement as many Gtk functions expect GdkPixbuf nowadays, so it is actually easier to use PB images with Gtk functions.

Things to watch out for on OSX:

Nothing to say here. Since the 2DDrawing support was so poor before, it only got better with this release. There is now a separate QuickDraw based subsystem for WindowOutput(). Yes i know that QuickDraw is deprecated, but it was much easier to implement this way than to go for a full Quartz based one. I have plans to do a Quartz subsystem one day to have better PrinterOutput() (for the moment, it still works with images as an intermediate).

One thing to note is the new output for sprites and the screen with OpenGL (this applies to all OS): There is no way to modify the data in video memory, so if you call StartDrawing() the whole sprite data or the entire screen content is transferred to main memory for the drawing. This is very slow. So doing 2DDrawing everytime you refresh the screen is too costly. The better way is to draw on sprites just once and then display them over and over. Still, this is better than having no 2DDrawing for sprite/screen at all.

Future plans:

There is still room for optimisation in the new 2DDrawing code. For now the focus was on getting a stable implementation. Also the plan is to eventually have the alpha drawing routines also for SpriteOutput() and ScreenOutput() but this will need some more work to support all possible pixel formats for the screen. Stay tuned…

Is your 64bit program really solid ?

If you ask Windows for memory it will generally start with low addresses first. There is nothing wrong with that. For a 64bit program however, this means that under normal circumstances unless you actually use up 4Gb of memory, your pointers will still fit into a 32bit value. So if you store a pointer into a too small variable (say a long) you will not notice, because the upper half of the 64bit pointer value isn’t even used. So your program may look quite solid and usable right now but if you feed it something like a 4Gb file to load it will start crashing. This may sound like an unlikely case but it will become much more common even for programs that are not meant to deal with large amounts of data, simply because as disk space, memory and computer speeds grow, so do the amounts of data that users deal with.

Anyway, I recently found this link: http://msdn.microsoft.com/en-us/library/bb613473.aspx

Notice at the very bottom the mention of the AllocationPreference registry setting. If you develop 64bit applications (especially if you are porting code from 32bit to 64bit) then this key will help you a great deal. What it does is it instructs Windows to give you high addresses first. The result are pointers that don’t fit into 32bit right from the start. So these errors that usually go unnoticed will blow up right in your face and you can fix them.

The link doesn’t explicitly state the OS this works on but i tested both on XP 64bit and Vista 64bit and it works fine on both. You have to restart the computer after setting the registry value for the change to take effect. You can then verify with a simple program if the high addresses are returned (make sure you compile it with the 64bit compiler 😉 ):

For i = 1 To 10
  a$ + RSet(Hex(AllocateMemory(1000), #PB_Integer), 16, "0") + Chr(13)
Next i
MessageRequester("Pointer values", a$)

In this mode you can easily distinguish a valid pointer from a truncated one by looking at it as hex like above. Since the virtual address space is limited to 8TB on current x64 Windows versions, a pointer allocated in this mode will look something like 000007FFFXXXXXXX because the addresses cannot go any higher. If a pointer got cut somewhere it will either look like 00000000XXXXXXXX or FFFFFFFFXXXXXXXX depending on wether the cut 32bit value was positive or negative (the sign-extension of the negative numbers will create all the F’s).

The PureBasic package itself is a good example of how easily such bugs go unnoticed. We payed a lot of attention to this when porting the libraries and also the programs like IDE, debugger and compiler to 64bit with PB 4.30. There was a lot of testing by the users after that too and PB 4.31 x64 is quite stable. However, if you turn on this mode the 4.31 IDE and debugger will not even start. Even the compiler won’t compile anything. So if you want to try this with your own programs you obviously cannot use PB 4.31 for it. The new 4.40 version was tested with this mode and these bugs are now fixed. Since it is still in beta, you may run into the odd crash though. If this happens please let us know.

Keep in mind that this setting is systemwide. So other x64 programs could well crash on you too if they have such unnoticed bugs.

Have fun fixing a ton of bug you never knew existed 😛

PureBasic 4.40 Beta1 released!

The wait is finally over. The first beta of our brand new v4.40 release hits the public. It brings exciting new features in all areas:

  • On the compiler side there is the new ‘Threaded’ scope keyword to create thread-local variables and the new ‘.a’ and ‘.u’ native types for unsigned byte/word.
  • On the Library side there is the new ‘Map’ library to have easy hash-maps, and a completly rewritten 2DDrawing library with Sprite/ScreenOutput() even for OpenGl (slow though) and cool alpha-channel and gradient support for ImageOutput().
  • And finally, the IDE gets a much improved AutoComplete and Project management!
  • Not to forget, there is also a brand new x64 Version for Linux available!

You can find the new version on your download account on http://www.purebasic.com/

To better explain the new features, we put together some examples which can be found here: http://www.purebasic.com/beta/v440_examples.zip

Now it is up to you to help us make this new version as solid as possible. Please report any problems you have with this new version in our bug forums. Expecially the Linux x64 version needs much testing as it is brand new, and the IDE because of the massive changes that were needed for the Project management.

Here is the full changelog:

- Added: Linux x64
- Added: Map Library for hash tables
- Added: 'Threaded' Keyword for thread-local variables
- Added: Structure assignment copies the structure (a.point = b.point)
- Added: ClearStructure(*Pointer, Structure)
- Added: AES to cipher library: AESEncoder(), AESDecoder(), StartAESCipher(), AddCipherBuffer(), FinishCipher()
- Added: Dylib for OS X
- Added: Trim/LTrim/RTrim() got an optional character to trim
- Added: Added #PB_Function, #PB_OSFunction, #PB_Map to Defined()
- Added: ReverseString(String$), InsertString(String$, StringToInsert$, Position), RemoveString(String$, RemoveString$ [, Mode [, StartPosition [, NbOccurences]]])
- Added: Add blob support to databases
- Added: Added peephole optimizer to 64 bit versions to produce better code
- Added: '.a' (ascii) and '.u' (unicode) native type to provide native unsigned byte and word.
- Added: FileBuffersSize(#PB_Default, ...): #PB_Default support change the buffersize to the next opened files
- Added: #PB_URL_Protocol to Get/SetURLPart()
- Added: #PB_Shadow_TextureAdditive
- Added: FTPDirectoryEntryRaw(), FillMemory(Memory, Value, Size [, Type])
- Added: Global, Protected, Threaded, Shared and Static now accept a type to affect all default variable declaration.

- Added: AddWindowTimer(), RemoveWindowTimer(), #PB_Event_Timer, EventTimer()
- Added: #PB_Window_Tool - create tool windows
- Added: StatusBarProgress()
- Added: StatusBarImage() - now supports normal images (not just icons)
- Added: #PB_Checkbox_ThreeState flag to create a 3 state checkbox (state values: #PB_CheckBox_Checked, #PB_CheckBox_Unchecked, #PB_CheckBox_Inbetween)
- Added: ShortcutGadget()

- Added: CreateImageMenu(), CreatePopupImageMenu() for OSX
- Added: Full alphachannel support for all GUI elements that display images 
- Added: Full alphachannel support for Image lib 

- Added: New drawing modes for 2DDrawing in ImageOutput()
   - #PB_2DDrawing_AlphaBlend
   - #PB_2DDrawing_AlphaClip
   - #PB_2DDrawing_AlphaChannel
   - #PB_2DDrawing_Gradient

- Added: LinearGradient(), BoxedGradient(), CircularGradient(), EllipsicalGradient(), ConicalGradient(), 

CurtomGradient()
- Added: GradientColor(), ResetGradientColors()
- Added: DrawAlphaImage() for all outputs on all OS
- Added: DrawAlphaImage() has a new transparency parameter and can be used to draw non-alpha images part transparent

- Added: QuickDraw subsystem for WindowOutput() on OSX
- Added: Sprite/ScreenOutput for OpenGL (all OS)
- Added: DrawingFont() works for SDL output!
- Added: OutputWidth(), OutputHeight(), OutputDepth() for 2DDrawing
- Added: GrabDrawingImage(), DrawRotatedText()
- Added: RGBA(), Alpha()

- Added #PB_PixelFormat_ReversedY - for pixel buffers that are stored upside-down (Windows ImageOutput or OpenGl)
- Removed: #PB_Image_DisplayFormat is now deprecated (value set to 32 to have 32bit as default)

- Added: #PB_OS_Windows_7 for OSVersion()

- Updated: OGRE to 1.6.2, sqlite 3.6.14.2, PCRE to 7.9

- Changed: Call(C)Function(Fast) parameters have been changed from 'Any' to 'Integer'.
- Fixed: Image Decoders are now threadsafe

PureBasic IDE:
- Added: Highlight matching keyword for keyword under cursor (tied to BraceMatching color setting) 
- Added: Edit->Goto matching keyword
- Added: PB_TOOL_Preferences Environment variable for IDE tools
- Added: Theme management to customize menu icons
- Added: New 'Silk' theme based on the Iconsset by Mark James

- Added: Autocomplete scans implicit variable declaration and respects scope
- Added: Structure item autocomplete

- Added: Project Management:
      - Autocomplete for all files within a project (even if not opened)
      - Multiple compiler settings for different compile targets
      - Compile all compile targets at once
      - Per-Project list of last opened files
      - Project ToolsPanel tool for fast access to the project files

- Changed: moved encoding/newline setting from compiler options to file menu


Debugger:
- Improved: greatly improved VariableViewer update speeds
- Added: progressbar display if VariableViewer update takes long
- Added: column sort capability to VariableViewer (Windows Only)

If you are wondering where the parallel optimized sort functions are that i talked about in the blog, unfortunately these did not make it into this version. There were some problems and unfinished things in them and there just was not enough time to finish it for this version. Don’t worry though, it will be in the next one for sure 🙂

Have fun with this version!