Page 1 of 1
Implement #Embed to speed up C compilation when including binaries.
Posted: Thu Jan 30, 2025 11:44 am
by pjay
Hi,
I have a number of PB programs that use quite large amounts of embedded (IncludeBinary) files. The ASM backend compiles these very quickly, but the C backend labours, often taking over a minute on programs where the included binaries sum to over 30mb in size - They can also use over 6gb of RAM whilst doing so.
After a quick investigation it turns out that the C compiler simply takes a long time to work with the large unsigned char arrays that the IncludeBinary function generates.
Below is a snippet to highlight the longer compile times you get when including some large binaries (file chosen based on standard Windows PB install location, change as required):
Code: Select all
;/ ~35mb of included data... 50 seconds to compile on 13900h.
DataSection
test1: ; ~8 seconds to compile, ~700mb ram usage during compile
IncludeBinary "C:\Program Files\PureBasic\PureBasic.exe"
test2: ; ~18 seconds to compile, ~1.4gb ram usage during compile
IncludeBinary "C:\Program Files\PureBasic\PureBasic.exe"
test3:
IncludeBinary "C:\Program Files\PureBasic\PureBasic.exe"
test4:
IncludeBinary "C:\Program Files\PureBasic\PureBasic.exe"
test5:
IncludeBinary "C:\Program Files\PureBasic\PureBasic.exe"
EndDataSection
A possible solution for this has appeared in some relatively new C #Embed capability, as explained here:
https://www.open-std.org/jtc1/sc22/wg14 ... /n2967.htm. Table 3.2.1 on the link provided shows example compile timings when using 40mb of embedded data going from 230s to just 1s, along with a large reduction in RAM usage whilst compiling.
Any chance of adopting this if it's available to you? Thanks.
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Thu Jan 30, 2025 10:18 pm
by idle
I don't see any issue on windows 11 with c backend, it's almost instantaneous, could it be that you have limited ram and it's thrashing the the page file?
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Thu Jan 30, 2025 10:53 pm
by Caronte3D
I have the same issue, with the C backend the compilation last many seconds.
Win 10 16GB Ram
With the asm backend it's instantaneous
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Thu Jan 30, 2025 11:21 pm
by Paul
idle wrote: Thu Jan 30, 2025 10:18 pm
I don't see any issue on windows 11 with c backend, it's almost instantaneous, could it be that you have limited ram and it's thrashing the the page file?
Doesn't matter what PC I use here or how powerful, the C backend compile time is slow.
The ASM backend is almost instant and always has been, even on the largest projects.
Are you sure you are actually using the C backend when compiling??
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Fri Jan 31, 2025 12:52 am
by idle
Paul wrote: Thu Jan 30, 2025 11:21 pm
idle wrote: Thu Jan 30, 2025 10:18 pm
I don't see any issue on windows 11 with c backend, it's almost instantaneous, could it be that you have limited ram and it's thrashing the the page file?
Doesn't matter what PC I use here or how powerful, the C backend compile time is slow.
The ASM backend is almost instant and always has been, even on the largest projects.
Are you sure you are actually using the C backend when compiling??
mybad, It takes a very long time!
I forgot to set my default compiler
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Fri Jan 31, 2025 2:11 am
by BarryG
pjay wrote: Thu Jan 30, 2025 11:44 amThe ASM backend compiles these very quickly, but the C backend labours, often taking over a minute
Yep! The C backend takes 3 mins to compile my app, compared to 10 secs for ASM ->
https://www.purebasic.fr/english/viewtopic.php?t=83731
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Fri Jan 31, 2025 9:05 am
by Fred
That's could be great for GCC backend, but for clang (OSX, Win arm64) it's not supported.
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Fri Jan 31, 2025 10:09 am
by pjay
Are you sure? I read it was added to clang 19 last year as part of the c23 implementation.
https://releases.llvm.org/19.1.0/tools/ ... Notes.html
Clang now supports N3017 #embed - a scannable, tooling-friendly binary resource inclusion mechanism.
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Fri Jan 31, 2025 10:11 am
by Fred
Ha I missed that, so it could be a nice solution.
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Fri Jan 31, 2025 12:31 pm
by blueb
idle wrote...
mybad, It takes a very long time!
I forgot to set my default compiler
.
.
I had to double check myself.
I think it might be an idea for Fred to re-name the IDE's header to show the
actual compiler in use.
Example:
https://prnt.sc/Q5HfrGfvXSLH
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Sat Feb 01, 2025 10:50 am
by pjay
Thanks for considering it, appreciated.
Re: Implement #Embed to speed up C compilation when including binaries.
Posted: Mon Feb 03, 2025 10:19 am
by SMaag
I think it might be an idea for Fred to re-name the IDE's header to show the actual compiler in use.
much better to integrate a compiler selection in a combobox in the Toolbar.
general notes on C compiling time:
C is a full optimizing multi pass compiler. That needs time!
Because of that issue Microsoft integrated in the Net Framework fast single pass compiler technolgy developed in the basiscs at ETH Zürich
(by Niklaus Wirth & Jürgen Gutknecht) -if you are interessted in background information use google to search for 'Compiler Construction
The Art of Niklaus Wirth'.
Compiling speed of fast single pass compiling compared with the full optimizing multi pass compilers is a big difference.
Purebasic ASM Backend is a kind of this fast singel pass compilers! The difference you can see in compiling times.
To speed up C-compiling time deactivate the code optimation in the compiler settings. Compile optimated only for the realease version!