Implement #Embed to speed up C compilation when including binaries.

[pjay]

Hi,

I have a number of PB programs that use quite large amounts of embedded (IncludeBinary) files. The ASM backend compiles these very quickly, but the C backend labours, often taking over a minute on programs where the included binaries sum to over 30mb in size - They can also use over 6gb of RAM whilst doing so.

After a quick investigation it turns out that the C compiler simply takes a long time to work with the large unsigned char arrays that the IncludeBinary function generates.

Below is a snippet to highlight the longer compile times you get when including some large binaries (file chosen based on standard Windows PB install location, change as required):

Code: Select all

;/ ~35mb of included data... 50 seconds to compile on 13900h.
DataSection
  test1: ; ~8 seconds to compile, ~700mb ram usage during compile
  IncludeBinary "C:\Program Files\PureBasic\PureBasic.exe"
  test2: ; ~18 seconds to compile, ~1.4gb ram usage during compile
  IncludeBinary "C:\Program Files\PureBasic\PureBasic.exe"
  test3:
  IncludeBinary "C:\Program Files\PureBasic\PureBasic.exe"
  test4:
  IncludeBinary "C:\Program Files\PureBasic\PureBasic.exe"
  test5:
  IncludeBinary "C:\Program Files\PureBasic\PureBasic.exe"
EndDataSection

A possible solution for this has appeared in some relatively new C #Embed capability, as explained here: https://www.open-std.org/jtc1/sc22/wg14 ... /n2967.htm. Table 3.2.1 on the link provided shows example compile timings when using 40mb of embedded data going from 230s to just 1s, along with a large reduction in RAM usage whilst compiling.

Any chance of adopting this if it's available to you? Thanks.

[idle]

I don't see any issue on windows 11 with c backend, it's almost instantaneous, could it be that you have limited ram and it's thrashing the the page file?

[Caronte3D]

I have the same issue, with the C backend the compilation last many seconds.
Win 10 16GB Ram
With the asm backend it's instantaneous

[Paul]

I don't see any issue on windows 11 with c backend, it's almost instantaneous, could it be that you have limited ram and it's thrashing the the page file?

Doesn't matter what PC I use here or how powerful, the C backend compile time is slow.
The ASM backend is almost instant and always has been, even on the largest projects.

Are you sure you are actually using the C backend when compiling??

[idle]

I don't see any issue on windows 11 with c backend, it's almost instantaneous, could it be that you have limited ram and it's thrashing the the page file?

Doesn't matter what PC I use here or how powerful, the C backend compile time is slow.
The ASM backend is almost instant and always has been, even on the largest projects.

Are you sure you are actually using the C backend when compiling??

mybad, It takes a very long time!
I forgot to set my default compiler

[BarryG]

The ASM backend compiles these very quickly, but the C backend labours, often taking over a minute

Yep! The C backend takes 3 mins to compile my app, compared to 10 secs for ASM -> https://www.purebasic.fr/english/viewtopic.php?t=83731

[Fred]

That's could be great for GCC backend, but for clang (OSX, Win arm64) it's not supported.

[pjay]

Are you sure? I read it was added to clang 19 last year as part of the c23 implementation.

https://releases.llvm.org/19.1.0/tools/ ... Notes.html

Clang now supports N3017 #embed - a scannable, tooling-friendly binary resource inclusion mechanism.

[Fred]

Ha I missed that, so it could be a nice solution.

[blueb]

idle wrote...
mybad, It takes a very long time!
I forgot to set my default compiler
.
.
I had to double check myself.

I think it might be an idea for Fred to re-name the IDE's header to show the actual compiler in use.
Example:
https://prnt.sc/Q5HfrGfvXSLH

[pjay]

Thanks for considering it, appreciated.

[SMaag]

I think it might be an idea for Fred to re-name the IDE's header to show the actual compiler in use.

much better to integrate a compiler selection in a combobox in the Toolbar.

general notes on C compiling time:
C is a full optimizing multi pass compiler. That needs time!
Because of that issue Microsoft integrated in the Net Framework fast single pass compiler technolgy developed in the basiscs at ETH Zürich
(by Niklaus Wirth & Jürgen Gutknecht) -if you are interessted in background information use google to search for 'Compiler Construction
The Art of Niklaus Wirth'.
Compiling speed of fast single pass compiling compared with the full optimizing multi pass compilers is a big difference.
Purebasic ASM Backend is a kind of this fast singel pass compilers! The difference you can see in compiling times.

To speed up C-compiling time deactivate the code optimation in the compiler settings. Compile optimated only for the realease version!