Code optimization

Got an idea for enhancing PureBasic? New command(s) you'd like to see?
User_Russian
Addict
Addict
Posts: 1441
Joined: Wed Nov 12, 2008 5:01 pm
Location: Russia

Code optimization

Post by User_Russian »

Why in executable file, add all of procedures of source, even those that have never called in the code (not used)?
Why the compiler can not find such procedures and to exclude them from the compilation?
User avatar
Shield
Addict
Addict
Posts: 1021
Joined: Fri Jan 21, 2011 8:25 am
Location: 'stralia!
Contact:

Re: Code optimization

Post by Shield »

Because PB doesn't do optimizations, apart form very simple ones such as constant folding
and replacing instructions by more efficient ones (e.g. replace ADD foo, 1 by INC foo).

Procedures that are not referenced won't be compiled, but this is of rather limited use.
Image
Blog: Why Does It Suck? (http://whydoesitsuck.com/)
"You can disagree with me as much as you want, but during this talk, by definition, anybody who disagrees is stupid and ugly."
- Linus Torvalds
User avatar
luis
Addict
Addict
Posts: 3876
Joined: Wed Aug 31, 2005 11:09 pm
Location: Italy

Re: Code optimization

Post by luis »

User_Russian wrote: Why the compiler can not find such procedures and to exclude them from the compilation?
Because for some reason it was not programmed to do so. Obviously it could.

The problem is twofold.

User procedures referenced in some way but not from your main code flow: -> http://www.purebasic.fr/english/viewtop ... 03#p300303

Library functions even if inside unreferenced user procedures : -> http://www.purebasic.fr/english/viewtop ... 01#p425401
http://www.purebasic.com/faq.php wrote: Is PureBasic really small and fast ?
Yes. All has been optimized to give maximum speed and compactness to the programs created with PureBasic.
Considering in PB you need to use includes to reuse code (the alternative is to manually cut/paste just what you need every time you start a new project), this is something that should be addressed IMO.
This, and variables defined and never used (zombies forgotten in your code).
"Have you tried turning it off and on again ?"
A little PureBasic review
c4s
Addict
Addict
Posts: 1981
Joined: Thu Nov 01, 2007 5:37 pm
Location: Germany

Re: Code optimization

Post by c4s »

+1 ...should be moved to the "feature request" (or "bug" ;)) forum though.
If any of you native English speakers have any suggestions for the above text, please let me know (via PM). Thanks!
User avatar
skywalk
Addict
Addict
Posts: 3960
Joined: Wed Dec 23, 2009 10:14 pm
Location: Boston, MA

Re: Code optimization

Post by skywalk »

It would be nice to summarize the most efficient tool(s) to spot unused variables and procedures? I remember luis and trond posted some logic.
It would help if PB created an expanded file from all macros & includes & modules.
The /commented asm achieves some of this, but parsing that output requires quite an investment.
The nice thing about standards is there are so many to choose from. ~ Andrew Tanenbaum
User avatar
blueznl
PureBasic Expert
PureBasic Expert
Posts: 6161
Joined: Sat May 17, 2003 11:31 am
Contact:

Re: Code optimization

Post by blueznl »

If you browse a bit there must be some tools that strip out unused procedures. CodeCaddy does but I haven't tested it on the latest versions of PureBasic...
( PB6.00 LTS Win11 x64 Asrock AB350 Pro4 Ryzen 5 3600 32GB GTX1060 6GB)
( The path to enlightenment and the PureBasic Survival Guide right here... )
User avatar
luis
Addict
Addict
Posts: 3876
Joined: Wed Aug 31, 2005 11:09 pm
Location: Italy

Re: Code optimization

Post by luis »

blueznl wrote:If you browse a bit there must be some tools that strip out unused procedures.
No one really works for every kind of code AFAIK, since it should be able to parse the source exactly as PB does (includes, constants substitution, macro expansions, conditional compilation, etc).

I'm waiting to write mine "stripper" when we will get this -> http://www.purebasic.fr/english/viewtop ... =3&t=48235, or for PB to do it from the start.
"Have you tried turning it off and on again ?"
A little PureBasic review
User avatar
skywalk
Addict
Addict
Posts: 3960
Joined: Wed Dec 23, 2009 10:14 pm
Location: Boston, MA

Re: Code optimization

Post by skywalk »

This is especially painful when I XIncludeFile for 1 or 2 member Procedures. My exe jumps by almost the full contents of the Xinclude and sure enough, the commented asm contains Procedures I have not referenced. :?:

I read in several places that a procedure that calls another procedure would be included even if the parent is never called. But, I fear that is not really what is happening in my case.

Does someone have a test case that shows the true exe inclusion process?
The nice thing about standards is there are so many to choose from. ~ Andrew Tanenbaum
User avatar
Danilo
Addict
Addict
Posts: 3037
Joined: Sat Apr 26, 2003 8:26 am
Location: Planet Earth

Re: Code optimization

Post by Danilo »

luis wrote:
blueznl wrote:If you browse a bit there must be some tools that strip out unused procedures.
No one really works for every kind of code AFAIK, since it should be able to parse the source exactly as PB does (includes, constants substitution, macro expansions, conditional compilation, etc).
If you don't want to write a full PB parser (incl. pre-processor and macro-processor), best thing for now could be
to process the resulting .asm output file. Scan the procedure macros, including calls to other procedures,
then filter which procedures are called in the main part (built a call graph). Exclude the PB procedure macros that are never called.
Processing this ASM is 'relatively' simple, as you just have to look for labels generally ("CALL label", "LEA ***, [label]", etc.)

PB procedure macros:

Code: Select all

macro MPnum{
_ProcedureNUM:   ; label inside procedure macro
   ; ...
   l_proc1_label1:  ; scan all labels inside procedures "[_A-Za-z0-9]:"
   ; ...
}
Example references:

Code: Select all

CALL _ProcedureNUM
LEA eax, [_ProcedureNUM]
*label_inside_procedure_macro* ; scan generally for any label names that were found inside procedure macros before [_A-Za-z0-9]
Procedure macros inserted at end (comment out unused macros):

Code: Select all

MP0
MP2
MP4
...
MP200
Of course the PB compiler already has all this infos and more (external calls to PB libraries from inside procedures, etc.)
Last edited by Danilo on Sat Dec 28, 2013 12:10 pm, edited 1 time in total.
User avatar
luis
Addict
Addict
Posts: 3876
Joined: Wed Aug 31, 2005 11:09 pm
Location: Italy

Re: Code optimization

Post by luis »

danilo wrote:If you don't want to write a full PB parser (incl. pre-processor and macro-processor
I don't, for the reasons explained here -> http://www.purebasic.fr/english/viewtop ... 61#p403061 :wink:
Danilo wrote:best thing for now could be to process the resulting .asm output file.
That was the first thing I tried, since it would have removed the need to parse the source expanding macro, evaluating conditional compilation etc.. It was working and was very easy to do but I wasn't happy with the result.
A lot of time is passed (a couple of years I think) so I don't remember it well, but I recall there were some external resources/dependencies imported by PB when it detects the usage of certain commands. When you just remove the calling macros you avoid the expansion of the body of your procedures, but the resources imported somewhere are staying there.
I would have to retry the process to be more specific, sorry, but I remember I thought "this is not working well, I would have to do this at the source level".
"Have you tried turning it off and on again ?"
A little PureBasic review
User avatar
skywalk
Addict
Addict
Posts: 3960
Joined: Wed Dec 23, 2009 10:14 pm
Location: Boston, MA

Re: Code optimization

Post by skywalk »

Sorry, but I am confused by the asm output with regards to unused and unreferenced procedures (called from unused procedures). I thought it was stated these would not be included in final exe?
I never checked the asm macro assignments before now, but I see many macros = completely unused procedures.

@Danilo, can you elaborate on your approach to parse the asm to build a table?
;-- FROM sample ASM --
; Procedure.i myProc(parameters...)
macro MP72{
_Procedure72:

;Pseudo code...
1. Build table of all 'CALL _ProcedureXXX' statements.
Don't I have the same problem PB has, since I am reporting CALL's from within unused procedures.
Or do I only search between:
'PureBasicStart:' and '_PB_EndFunctions:' ?
2. Build table of all '_ProcedureXXX' statements.
3. Build table of all 'macro MPxxx' statements.
What is different about the list of MPxxx's near the end of the ASM?
Many of those are also unused and unreferenced.
4. Build table of all original source code Procedure Names from preceding line 'macro MPxxx'.
; Procedure.x yyy(parameters...)
5. Search CALL table for all known '_ProcedureXXX' and dump a list containing only the used source code Procedures.

haha, which is faster to shrink my exe?
1. Manual cut and paste from many XIncludeFiles?
or
2. Write some code to parse the asm?
or
3. Write some code to parse the original code?
This seems the most difficult since Macros are not expanded.
The nice thing about standards is there are so many to choose from. ~ Andrew Tanenbaum
User avatar
Tenaja
Addict
Addict
Posts: 1948
Joined: Tue Nov 09, 2010 10:15 pm

Re: Code optimization

Post by Tenaja »

My guess would be that currently, PB has a boolean flag that just indicates if a Proc is called or not.

Instead of that boolean flag, procedure references (i.e. proc calls) need to be kept. One method would be in a map (the key being the names of procs) of lists (listing all procs called within the proc). Then, you iterate through, starting with the calls from outside of any proc, and only flag the procs that can be reached from the start.
User avatar
luis
Addict
Addict
Posts: 3876
Joined: Wed Aug 31, 2005 11:09 pm
Location: Italy

Re: Code optimization

Post by luis »

Tenaja wrote:My guess would be that currently, PB has a boolean flag that just indicates if a Proc is called or not.
It does not have to track anything.

As visible in the asm output here -> http://www.purebasic.fr/english/viewtop ... 03#p300303 PB generates the code for all the procedures but wraps them in asm macros. When a proc is referenced the macro is expanded and so its code is included in the final exe. If a proc is not referenced the macro is never called, the code is not expanded and the code is not included in the final exe.

The result is something good enough most of the times for small programs with zero efforts from the compiler, and at the same time the source of the problem discussed here.

Typical worst cases are included files with a lot of procedures referencing one another but with practically nothing called from the main program.
"Have you tried turning it off and on again ?"
A little PureBasic review
User avatar
skywalk
Addict
Addict
Posts: 3960
Joined: Wed Dec 23, 2009 10:14 pm
Location: Boston, MA

Re: Code optimization

Post by skywalk »

Exactly.
Struggling to scan only the "main" portion of the asm?
This is definitely a multi-pass problem. (all 5th element jokes go here :lol: )
If I exclude all macro MPxxx's{}, I am hoping to be left with just the main calls.
Then there is that curious list of MPxxx's which I don't understand why some are and some aren't included?
Either way, it would be better if this was solved by the IDE now that we have XIncludes and Modules.
Manually stripping include files/modules is not elegant.
The nice thing about standards is there are so many to choose from. ~ Andrew Tanenbaum
User avatar
Tenaja
Addict
Addict
Posts: 1948
Joined: Tue Nov 09, 2010 10:15 pm

Re: Code optimization

Post by Tenaja »

luis wrote:It does not have to track anything.

As visible in the asm output here -> http://www.purebasic.fr/english/viewtop ... 03#p300303 PB generates the code for all the procedures but wraps them in asm macros. When a proc is referenced the macro is expanded and so its code is included in the final exe. If a proc is not referenced the macro is never called, the code is not expanded and the code is not included in the final exe.
I believe I know how the macro wrapping works (and have, for several years)...and I do not believe your description above is entirely accurate, because your description omits a critical step, which is actually supported by your link to the 2009 thread...

The macro NOT expanded when (i.e. at the location) a proc is referenced. It is expanded at the very end of the asm file--this is supported by the fact that all of the macro "call" insertions--such as MP0, MP2, MP4, etc.--are listed on consecutive lines at the bottom of the asm code. Because of that, it follows that PB is somehow flagging (tracking) which procs require that macro expansion call to be inserted, and it does not insert the call (the MP0, MP2, etc) if the flag for that proc is not set.

The "root of the problem" (I believe) is that PB flags a proc to be expanded immediately once the proc is referenced (even, for instance, within an unused proc). This technique results in some unused procs being flagged for insertion. (I wouldn't be surprised if a map or list is used to store the flags, since the macro calls are not in ascending order, nor are they in the order called.)

The solution is to use a different technique--do not flag them for insertion during parsing, but rather merely track their use such as with a call tree (or dependency graph) during parsing. Then insert required expanding macro calls as needed. One possible method is to use a map [an element for each proc] of lists [listing all procs called within that proc]) and at the end, run through the lists and only insert the procs that can possibly be executed. The "inclusion loop" would start with the non-proc code, and flagging all procs called and recursively handle each called proc for other required procs.
Locked