PB and VB.NET speed comparison

[Olby]

I have been working on a massive prototype in PureBasic for more than 2 years now and due to release the final sometime between late 2012 early 2013. The application it self is a database driven, multi-user bespoke data processing solution. Obviously since it's a prototype we're planning to rewrite the v2.0 from scratch in more professional environment and as I used to work with VB6 back in the days I chose to redo it in VB.NET.

Am a big fan of PB and have been using it for a while and will keep using it when necessary, but there are certain limitations we've experienced while doing the prototype and VB.NET addresses them nicely. I'm pretty sure you all are aware of them, but here are some I find really annoying:

1. IDE (PB has a compact, swift and easy to use environment, but I find it rather basic for such a huge project we're doing)
2. Gadgets (Hands down VB.NET has a lot more to offer here and unless you want to rely on third-party developers PB fails to be as useful)
3. Inconsistencies (Here and there I find PB to bee a little rough around the edges for example no way to color a button unless you use PureColor which in turn falss under previous point)
4. User Base (PB has extremely helpful and knowledgeable community of developers, unfortunately we're nothing (quantitative) compared to .NET user base and some of the libraries we wanted to implement where not available directly for PB and time was spent wrapping them or fixing problems with existing wrappers)

People been complaining about how bloated Windows and .NET framework is and I totally agree. Luckily for all .NET developers Vista and 7 comes with pre-installed framework so I don't think thats an issue I should worry about unless I do games, but then I can use good old PB

Anyhow I'll stop my rant and return back to topic. I wanted to find out how VB.NET compares to PB in terms of pure runtime speed. As I was Googling this I stumbled upon an old thread that compared the speeds of VB6 to VB.NET using simple math calculations. Needless to say .NET was a lot faster.

So here it is ....

VB.NET source code:

Code: Select all

    Declare Function GetTickCount Lib "kernel32.dll" () As Long
    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        'Program to test relative speed of VB6 vs VB.NET
        'Tests mathematical operations
        'Coded 10.8.02

        Dim x As Long
        Dim z As Double
        Dim start As Long

        MessageBox.Show("Press OK to begin mathematical operations. Tests should take <1 minute.")

        start = GetTickCount

        'Basic operations, integer & floating point
        For x = 0 To 10000000
            z = 17 + 56 / (9 * 263) - 78 / (2.2 * 9)
        Next

        'Rnd
        For x = 0 To 1000000

            z = Rnd()

        Next

        'Trig functions
        For x = 0 To 10000000

            z = System.Math.Sin(23.5687)
            z = System.Math.Cos(23.5687)
            z = System.Math.Tan(23.5687)
            z = System.Math.Atan(23.5687)

        Next

        'Sqr
        For x = 0 To 10000000

            z = System.Math.Sqrt(2315.126)

        Next

        'Exp & log
        For x = 0 To 10000000

            z = System.Math.Exp(22.561)
            z = System.Math.Log(22.561)

        Next x

        MessageBox.Show((GetTickCount() - start) / 1000 & " seconds elapsed.")

    End Sub

PB Source code:

Code: Select all

Define x.l,z.d,start.l

MessageRequester("","Press OK To begin mathematical operations. Tests should take <1 minute.")

start = ElapsedMilliseconds()

;Basic operations, integer & floating point
For x = 0 To 10000000
    z = 17 + 56 / (9 * 263) - 78 / (2.2 * 9)
Next

;Rnd
For x = 0 To 1000000

    z = Random(1)

Next

;Trig functions
For x = 0 To 10000000

    z = Sin(23.5687)
    z = Cos(23.5687)
    z = Tan(23.5687)
    z = ATan(23.5687)

Next

;Sqr
For x = 0 To 10000000

    z = Sqr(2315.126)

Next

;Exp & log
For x = 0 To 10000000

    z = Exp(22.561)
    z = Log(22.561)

Next x

MessageRequester("",StrF((ElapsedMilliseconds() - start) / 1000,2) + " seconds elapsed.")

Building both projects resulted in following (on my system):

Visual Basic .NET Express 2008 - ~2.7 seconds
PureBasic v4.60 - ~4.77 seconds

I know such test are purely for "fanboys" and usually don't represent the real-life speed of the binary but I always thought that PB would beat VB.NET by a thick margin so this was quite a surprise for me.

[ts-soft]

Your test is only the runtime. The start of a .net app is slower than the start of a native exe.

[MachineCode]

A whole 2 seconds faster? Wow! Your test is not a real-world app anyway, so it's completely pointless.

Just so you know: you can use third-party IDEs with PureBasic source codes if you like, with full syntax coloring as well.

Oh, and buttons can be colored without PureCOLOR.

[Olby]

Your test is only the runtime. The start of a .net app is slower than the start of a native exe.

Yes, but only the first time you run it. Every other start-up is fast.

A whole 2 seconds faster? Wow!

Just so you know: you can use third-party IDEs with PureBasic source codes if you like, with full syntax coloring as well.

Oh, and buttons can be colored without PureCOLOR.

It adds up. If there is a 2 second difference now in such tiny application I can imagine the gain running something far more advanced in real-life scenario.

That's the thing, you just approved what I said earlier. PureBasic requires a lot of third-party stuff to be efficient. And I don't want to revert to WinAPI to color a button which is just appalling and breaks multi-platform compatibility.

The other day I found out that VB.NET IDE precompiles the source code as you write it. Basically it can detect mistakes as you make them without need to compile the whole solution (extremely useful for big projects). It can also, for example, point out if variable is defined but has not been used. Something I don't expect to see in PB or jaPBe.

[wilbert]

Since the outcome of expressions like

17 + 56 / (9 * 263) - 78 / (2.2 * 9)

or

System.Math.Sin(23.5687)

is always the same, it's very well possible a language that optimizes source code replaces such an expression with a constant.
That would make execution time much faster.

[moogle]

I think VB wins in this test because it does quite a lot more optimizations than PB does under the hood.

For example I remember that codes like this

Code: Select all

For x = 0 To 10000000

    z = Sin(23.5687)
    z = Cos(23.5687)
    z = Tan(23.5687)
    z = ATan(23.5687)

Next

VB detects that they are repeats so it will optimize this to run only one time as the values returned never change.

The first loop with Random will run like normal on VB as it's different every time.

C does this pretty quick too with it's optimizations.

PB doesn't have that (yet, I hope )

[Tenaja]

What kind of database are you creating that has such heavy use of floating point trig geometry? Are you creating airfoils or something? If not, why not use a benchmark that is applicable to your app? If the heavy lifting in your app is a database, then you should test that. Comparing trig libraries is rather pointless if you are not performing massive trig functions in your code. The DB library that is used that will matter more than the interface, and you can use the same one regardless of your language choice.

PB often has a speed advantage over the tokenizing languages, but not necessarily after startup on those that are compiled JIT. For a "true" compiler, PB output is not really optimized for speed...and likely never will, either. It has some built-in "features" that contradict speed optimizations.

I am guessing you have your debugger off since the cpu listed in your signature is similar speed to mine, and I am getting similar times (consistently 6.2 vs. 5.2 for VB). I am 99% certain the trig functions are the bottleneck, though.

Here is the asm code generated for the slowest section of your code, the Trig functions (comment it out, and my system cuts the execution time in half):

Code: Select all

; 
;Trig functions
; For x = 0 To 10000000
  MOV    dword [esp],0
_For5:
  MOV    eax,10000000
  CMP    eax,dword [esp]
  JL    _Next6
; 
; z = Sin(23.5687)
  FLD    qword [D2]
  FSIN
  FSTP   qword [esp+4]
; z = Cos(23.5687)
  FLD    qword [D2]
  FCOS
  FSTP   qword [esp+4]
; z = Tan(23.5687)
  FLD    qword [D2]
  FPTAN
  FSTP   dword [esp-4]
  FSTP   qword [esp+4]
; z = ATan(23.5687)
  FLD    qword [D2]
  SUB    esp,8
  FSTP   qword [esp]
  CALL  _PB_ATan_DOUBLE@8
  FSTP   qword [esp+4]
; 
; Next
_NextContinue6:
  INC    dword [esp]
  JNO   _For5
_Next6:
; 

In this section of code, there really is no room for any kind of optimization, except for maybe one instruction per loop if Fred were to reorganize the For loop so the comparison was at the top and also duplicated at the bottom. (That would eliminate one instruction per loop execution, at the cost of duplicating 3--but with only 20 instructions in the loop plus trig time, that is only a 5% improvement.) So, clearly, the calls to the trig functions take up almost all of the time. That puts us back where we started; test the speed of libraries you intend to use.

[Olby]

is always the same, it's very well possible a language that optimizes source code replaces such an expression with a constant.
That would make execution time much faster.

Must be it, I've heard it before.

Comparing trig libraries is rather pointless if you are not performing massive trig functions in your code.

Am also a hobbyist game designer and I use sin & cos quite often in my developments so it was kind of interesting to see where it compares.

test the speed of libraries you intend to use

I will, my intent is to test native ADO.NET with Adomate wrapper in PB written by srod, but I have the gut feeling that ADO.NET will outperform in this case as well. Anyway, I'll see if I can come up with some tests. Thanks.

[Tenaja]

It adds up. If there is a 2 second difference now in such tiny application I can imagine the gain running something far more advanced in real-life scenario.

In a real life scenario, nobody will ever loop 10M times through calculating the same values.

On the other hand, it is still the trig functions that are the bottleneck. Changing all the constants for an incremementing y variable, and the time goes to 8.0s PB, 6.0s VB. Clearly, VB is not optimizing the "constants". The only addition to the PB code is this:

Code: Select all

; y = y + 0.0001
  FLD    qword [v_y]
  FADD   qword [D2]
  FSTP   qword [v_y]

This is what I used:

Code: Select all

DisableDebugger
EnableExplicit

Global x.l,z.d, y.d, start.l

MessageRequester("","Press OK To begin mathematical operations. Tests should take <1 minute.")

start = ElapsedMilliseconds()

y= 0

;Basic operations, integer & floating point
For x = 0 To 10000000
	y = y + 0.0001
	z = y + 17 + 56 / (9 * 263) - 78 / (2.2 * 9)
Next

;Rnd
For x = 0 To 1000000
	
	z = Random(1)
	
Next

y = 0
;Trig functions
For x = 0 To 10000000
	y = y + 0.0001
	z = Sin(y)
	z = Cos(y)
	z = Tan(y)
	z = ATan(y)
	
Next

y = 0
;Sqr
For x = 0 To 10000000
	y = y + 0.0001
	z = Sqr(y)
	
Next

y = 0
;Exp & log
For x = 0 To 10000000
	y = y + 0.0001
	z = Exp(y)
	z = Log(y)
	
Next x

MessageRequester("",StrF((ElapsedMilliseconds() - start) / 1000,2) + " seconds elapsed.")

[moogle]

Am also a hobbyist game designer and I use sin & cos quite often in my developments so it was kind of interesting to see where it compares.

Using an array of precomputed values speeds things up pretty quickly I'd say.

[Olby]

In a real life scenario, nobody will ever loop 10M times through calculating the same values.

Maybe not ten mil but I do have to process data with an average of 2 to 4 thousand records and do some math and memory operations per record. Batch process is in a separate thread running in background while main thread is catering the user.

Using an array of precomputed values speeds things up pretty quickly I'd say.

Good idea, I'll try to see if I can do that for my game code.

[ts-soft]

I will, my intent is to test native ADO.NET with Adomate wrapper in PB written by srod, but I have the gut feeling that ADO.NET will outperform in this case as well. Anyway, I'll see if I can come up with some tests. Thanks.

Srod using late binding, compare this is useless. VB.NET will win
Is the same as you compare VB.NET with VB-Script.

[Tenaja]

Am also a hobbyist game designer and I use sin & cos quite often in my developments so it was kind of interesting to see where it compares.

Maybe not ten mil but I do have to process data with an average of 2 to 4 thousand records and do some math and memory operations per record. Batch process is in a separate thread running in background while main thread is catering the user.

Yes, but even in a game situation, sin & cos will be far from the majority of the code used. For a comparison that translates to the "real world," you should use code that resembles real world code. For instance, use the "math and memory operations" that you do on the record.


For this speed test, you could probably even find optimized sin & cos libraries online and use those. Either way, with code this simple, it is all about the library. Apart from the libarary, Fred can't squeeze much more out of this, unless he implements constant propagation and variable tracking. (He uses the stack for the trig calls, which requires the redundancy of pushing Y, then popping the result with every call. An optimized routine would let him put it in a register that is unmolested in the routine, then it only needs loading once per loop.)

[USCode]

This kind of real-world feedback is great and can only make PB better. Thanks!

Observations:
1. I never thought of PB as a multi-developer environment, I don't think that's their target market or focus?
2. There's certainly room here for improvement, and I think Fred's Gadget enhancement list is very large based on the feedback I've seen here on the forum. I'd like to be able to set the height of a drop-down's list, and set the width of the splitter separator bar, and detect a listicongadget column-width adjustment, and ... and ... However, I think many of the limitations come from the cross-platform nature of PB. You can do a lot of these things with API calls but its cross-platform nature is one of PB's core strengths IMHO.
3. Can you color buttons on OS X? Not sure if that's possible and might be the reason Fred decided not to support it on the other platforms. It's the lowest common denominator issue. Could you perhaps use the ButtonImageGadget here instead?
4. With the larger user base it also means there are more eyes to spot and report bugs, which results in a more stable product. PB is getting more stable lately with Fred working on PB full time again. He's really cranking out the bug fixes! That brings up another issue; Only 1 core developer works on PB. This can be good for the PB community as through the forum we have the direct attention of the primary developer but there's only so much 1 man can do and what happens if he needs to get a fulltime job again, or tires of PB, decides to have 6 kids, etc.?

Aren't there some issues with .NET? Start-up time? Even with MONO, is it truly cross-platform? Isn't it easy to decompile .NET applications to source? Others?

Once again, thanks for your feedback and please keep it coming.

[Olby]

1. I never thought of PB as a multi-developer environment, I don't think that's their target market or focus?

I think PB's Amiga heritage and overall focus on low level access is what makes it more like a sole programmer solution. Can't imagine someone doing ASM coding in multi-developer environment.

2. There's certainly room here for improvement, and I think Fred's Gadget enhancement list is very large based on the feedback I've seen here on the forum. I'd like to be able to set the height of a drop-down's list, and set the width of the splitter separator bar, and detect a listicongadget column-width adjustment, and ... and ... However, I think many of the limitations come from the cross-platform nature of PB. You can do a lot of these things with API calls but its cross-platform nature is one of PB's core strengths IMHO.

Agreed, but overwhelming use of native API calls for every platform makes it a real pain.

3. Can you color buttons on OS X? Not sure if that's possible and might be the reason Fred decided not to support it on the other platforms. It's the lowest common denominator issue. Could you perhaps use the ButtonImageGadget here instead?

Exactly I had to revert to image buttons. Fred states (in the help file) that "With activated Windows XP style the color settings will probably be ignored or overwritten by the style." which is not true, I can change button and other gadgets face colors in VB.NET easily.

4. With the larger user base it also means there are more eyes to spot and report bugs, which results in a more stable product. PB is getting more stable lately with Fred working on PB full time again. He's really cranking out the bug fixes! That brings up another issue; Only 1 core developer works on PB. This can be good for the PB community as through the forum we have the direct attention of the primary developer but there's only so much 1 man can do and what happens if he needs to get a fulltime job again, or tires of PB, decides to have 6 kids, etc.?

I totally appreciate Fred's and Freaks work so far, they're stars and done amazingly great job throughout these years. I wish only PB was more widely recognized and adopted.

Aren't there some issues with .NET? Start-up time? Even with MONO, is it truly cross-platform? Isn't it easy to decompile .NET applications to source? Others?

Of course there are, every language has it's pros and cons. Net framework based applications are slower to start for the first time because they pre-compile the code, luckily it happens only once. MS provides Dotfuscator to scramble the .NET code which makes it trickier to reverse-engineer, how good it is I don't know.