I have first used a seperate thread with an application specific file in the global temp directory. The seperate thread writes a timestamp every 5 seconds, so if the file exists, you can read the timestamp and check if it is "outdated" (older then 11 seconds in my case). If it is outdated, you can just silently erase the tempfile and create your own. Otherwise a "the application seems in use" requester is opened with directions how to recover if this would be an error. We used to record the User + Machine into the file additionally, so we could show those info's in these rare cases of crash recovery.
The second method i used, was a base network port number and a seperate thread which communicates with other instances of the application. If the base port (for example 50000) is used, the new instance connects to the port and does an application specific handshake. For example saying "hi i am Ramihyns app V 0.25, who are you?" and my own app would reply "hey cool, i am Ramihyns app V 0.5". If the service doesnt correctly reply to the communication, there is a predetermined offset we use, for example 1000. So we do the same on port 51000 and we have a limit of 10 attempts. So the app tries 50000, 51000, 52000 till 60000. That method works well in the field, can recover from crashes (remember that ports can be in use for up to some minutes after your application crashed) and the protocol has grown a lot to include license managing, data exchange and other usefull stuff.
A major downside of the network based method, is that some virus scanners can act "funny" if you do it. Some interfere and ask the user who then will wonder why your software is trying to connect to the "internet" or even worse "starts a network service". That can result in support requests from paranoid users. Thats why we use global mutexes for our windows versions, because it regularly gave us trouble in windows. I guess another downside might be that you unintentionally crash some other network service if it cant deal with your "hello" message. It never happened to me, but you could just change it so the client doesnt say anything and waits 3 seconds for your app to send its "Hi i am ..." string and if nothing happens, just assumes that it is some other app and closes the port again.
In practise dont use something like 50000 with steps of 1000, but better like 48627 and steps of 692. Numbers that another developer is unlikely to choose "randomly"
