Does subsystem flag matter for DLLs?

Tipperton · Post by **Tipperton** » Sat Aug 19, 2006 12:43 pm

A couple of months ago I wrote a DLL for a client using PowerBASIC. It worked perfectly except for one stupid mistake I made which was not closing a file it had openned....

But they didn't tell me there was a problem until just now (several months later).

In that time I had migrated from PowerBASIC to PureBasic because I found writing GUI's in PureBasic FAR EASIER in PureBasic than it was in PowerBASIC.

So I decided to re-write the DLL in PureBasic. Other than source code syntax differences and the code generated by the compilers, the DLL functions exactly the same (does the exact same steps in the exact same order) in the two versions. The client now tells me that the new DLL (which they call from their installer) kills the installer with no error messages.

I decided to use PE Explorer to see if there was anything different about the two DLLs that might help troubleshoot the problem. One thing I noticed is that the DLL compiled by PowerBASIC has subsystem flags set for WindowsGUI while the DLL compiled by PureBasic has them set for Console App.

I don't much (if anything) about what affect the subsystem flags might have or if they even apply to DLLs, so... I'll ask...

Could the subsystem flags cause this problem and would it be enough to use a hex editor to change the Console App flag to a WindowGUI flag, or is there more to it than that, or does it even apply to DLLs and is just their for header format consistancy?

Thanks!

Tipperton · Post by **Tipperton** » Sun Aug 20, 2006 1:46 pm

As a test, I wrote a small program to check the subsystem flag on all DLLs in the directory that the program was run from and to output the results to a text file.

It appears that normally DLLs should have the subsystem flag set to WindowsGUI rather than Consol App, so I'm considering this a minor bug in PureBasic. (for now).

So the next question is: Is the a way to force the compiler to set this flag properly when compiling DLLs?

Thanks!

remi_meier · Post by **remi_meier** » Sun Aug 20, 2006 2:01 pm

Didn't try, but you could

Create a plain text file with an @ as first character in name, like "@commands.txt".
Content:

Code: Select all

/SUBSYSTEM:WINDOWS

then add to the command line of the compiler (IDE options for the project)
this flag:

Code: Select all

/LINKER "C:/@commands.txt"

and compile a DLL with it.

Perhaps it works, perhaps it doesn't...

Tipperton · Post by **Tipperton** » Sun Aug 20, 2006 2:27 pm

Nice idea but it may have to be setup as some sort of "tool" because the only command line option I can see (unless I'm missing something) in the compiler options is for the executable itself.

PS: I'm using PureBasic v4.00 with the standard IDE that comes with it.

remi_meier · Post by **remi_meier** » Sun Aug 20, 2006 2:43 pm

Ok then, in jaPBe you could choose "compile manually" where you could
add the flags...
So you have to compile through command line!

pbcompiler "sourcefile.pb" /DLL "dllname.dll" /Linker "@commands.txt"

or something like that, again, not tested.

Tipperton · Post by **Tipperton** » Sun Aug 20, 2006 3:11 pm

remi_meier wrote:again, not tested.

No problem! I'm willing to "tinker"!

remi_meier · Post by **remi_meier** » Sun Aug 20, 2006 3:21 pm

Ok, I tested it, but not if the subsystem is now correct. Use this command
line:

pbcompiler "filename.pb" /DLL /EXE "dllname.dll" /LINKER "@commands.txt"

No error now.

Tipperton · Post by **Tipperton** » Sun Aug 20, 2006 3:50 pm

remi_meier wrote:Ok, I tested it, but not if the subsystem is now correct. Use this command line:
bcompiler "filename.pb" /DLL /EXE "dllname.dll" /LINKER "@commands.txt"
No error now.

Thanks, I'll set that up as a tool and give it a shot...

Tipperton · Post by **Tipperton** » Thu Aug 24, 2006 2:46 am

Well, well, well...

As it turns out, the installer (InstallShield) assumes that DLLs accept and return 32 bit integers... I was accepting a 16 bit integer and returning a byte since that was all that was needed. Changing it to accept and return 32 bit integers (longs) solved the problems they were having with it.

I also did a bunch of playing around with DLLs and have come to the conclusion that the subsystem flag in the PE headers is ignored for DLLs and is just there for header format consistency.

Makes sense in a way, DLLs are just a bunch of functions and procedures that can be called by other programs (both Console and WindowsGUI apps) so for them, what subsystem they are for doesn't really matter.

Learn something new every day....