When writing massive software like PureBasic, you need to have automated tests to validate it and ensures there is no regression when publishing a new version. We will try to explain how we achieve this for PureBasic and why it is essential for the product robustness.

First, let’s recap the main PureBasic components and their size:

• The compiler, which is about 250k lines of C, generating x86 asm, x64 asm and C
• The IDE, written in PureBasic which is about 130k lines of code
• The libraries, written for 3 platforms (Linux, Windows, OS X) in ASM, C, C++ or Objective C which are about 1.200k lines of code

Each time a compiler bug is squashed, we add a new test in the test suite to ensures it won’t happen again. For now, there is more than 2500 unit tests covering all public features of the PureBasic language, validating the syntax and ensuring the results are corrects. Theses tests are run in different context (Threaded mode ON/OFF, debugger ON/OFF, Optimizer ON/OFF) and if one of these tests fail, the build script is aborted. These tests are just written in standard PureBasic files, compiled into a single executable and run all at once. Each platform runs these tests. Here is a small example of how it looks from the AritmeticFloat.pb file:

; Float promotion
;
Check(Round(Val("2") * 1.2, #PB_Round_Nearest) = 2)

Integer.i = Round(Val("3") * 1.2, #PB_Round_Nearest)
Check(Integer = 4)

Integer = 1
Integer = Integer + Round(Val("2") * 1.5, #PB_Round_Nearest)
Check(Integer = 4)

Integer = 1
Integer = Integer - Round(Val("2") * 1.5, #PB_Round_Nearest)
Check(Integer = -2)

Integer = 2
Integer = Integer * Round(Val("2") * 1.5, #PB_Round_Nearest)
Check(Integer = 6)

Integer = 9
Integer = Integer / Round(Val("2") * 1.5, #PB_Round_Nearest)
Check(Integer = 3)

; Constant rounding
;
Integer.i = 0.6 * 2 ; rounded to 1
Check(Integer = 1)

Integer = 0.9 * 2 ; rounded to 2
Check(Integer = 2)

Integer = 0.6 ; rounded to 1
Check(Integer * 2 = 2) 

When we work on the libraries, it’s a bit different, as we use the PureUnit tool (which can be found freely in the SDK) which is similar to NUnit or JUnit if you know those tools. It is very good at writing single commands tests, and provide a solid way to run the tests randomly, in different modes (Threaded, with different Subsystems, etc.). When a bug is fixed in a library we add a test when it’s possible (testing UI, Sprites, 3D engine is complicated as you can’t really check the graphical output). The Syntax is very easy to use as you can see from this example taken from the String library unit test file:

ProcedureUnit InsertStringTest()
  Hello$ = "Hello"
  World$ = "World"
  AssertString(InsertString("Hello !", World$, 7), "Hello World!")
  AssertString(InsertString("Hello !", World$, 8), "Hello !World")
  AssertString(InsertString("Hello !", World$, 1), "WorldHello !")
  AssertString(InsertString("Hello !", World$, -1), "WorldHello !") ; Test with out of bounds value
  AssertString(InsertString(Null$, Null$, 1), "") ; Null values should be accepted as well
  AssertString(InsertString("", "Hello", 1), "Hello") ; Inserting empty string shouldn't touch the string
  AssertString(InsertString(Null$, "Hello", 1), "Hello") ; Inserting null string shouldn't touch the string
  
  AssertString(InsertString(Hello$+Hello$, World$+World$, 6), "HelloWorldWorldHello") ; Test with internal buffer use
EndProcedureUnit


ProcedureUnit LeftTest()
  Test$ = "Test"
  AssertString(Left(Test$, 1)      , "T")
  AssertString(Left(Test$, 0)     , "")
  AssertString(Left(Test$, -1)     , "")
  AssertString(Left(Test$, 10)     , "Test")
  AssertString(Left(Test$, 3)      , "Tes")
  AssertString(Left(Test$, 4)      , "Test")
  AssertString(Left(Null$, 0)      , "")
  Result$ = Left(Test$, 1)
EndProcedureUnit


ProcedureUnit LenTest()
  Test$ = "Test"
  Assert(Len(Test$) = 4)
  Assert(Len(Null$) = 0)
EndProcedureUnit

The test procedure are declared with ProcedureUnit and EndProcedureUnit, there is some Assert commands available to control the checks. You can declare some common setup or clear procedure to run before and after each test with ProcedureUnitBefore and ProcedureUnitAfter. There is much more, feel free to check the PureUnit documentation to get the most of it. I strongly recommand to give it a try if you need to write some unit tests for your own PureBasic application.

The next tests we have are ‘linker’ tests. We put all commands of a single library in a file, and compile it to ensures it links properly. Then we put all the commands of PureBasic in a single file and we link it as well.

Then we test if all the examples found the in PureBasic/Examples directory compiles properly with the new PureBasic version.

Once all these tests are run and successful, the final package can created and published.