[Implemented] Remotely compile/debug on Server

Got an idea for enhancing PureBasic? New command(s) you'd like to see?
mdp
Enthusiast
Enthusiast
Posts: 115
Joined: Mon Apr 18, 2005 8:28 pm

[Implemented] Remotely compile/debug on Server

Post by mdp »

In case you want to write commands for a Linux server - which normally is a machine you cannot have the PureBasic IDE on - it is often not practical to build on some Desktop machine, transfer the file, test, repeat...
This way the procedure is pretty slow and you will miss the debugger.

It would be instead very useful to have a feature for compiling and debugging something on a remote machine via a SSH tunnel (or even some "pbcompilerd" server daemon on your favourite port, for that matter)

@PB Team: would that be feasible?
@All: Are there present alternatives I missed to my "LinuxDesktop + gFTP + run-command-on-server-and-check-output" environment?
Last edited by mdp on Sun Feb 07, 2010 11:28 am, edited 1 time in total.
freak
PureBasic Team
PureBasic Team
Posts: 5929
Joined: Fri Apr 25, 2003 5:21 pm
Location: Germany

Post by freak »

Remote debugging is planned for one of the next versions.

Dunno if remote compiling is worth the effort.
If you can login via ssh, compile and run the program, then connect the debugger via the network, it should be enough i think.
quidquid Latine dictum sit altum videtur
mdp
Enthusiast
Enthusiast
Posts: 115
Joined: Mon Apr 18, 2005 8:28 pm

Post by mdp »

Yes, having the remote CPU do the compiling job is not necessary. I should have written "compiling (and debugging) something to a remote machine".

The present situation: you can create a ssh connection, then go to "CompilerOptions" > "Compile/Run" and set "CurrentDirectory" to a remote folder (such as "/home/myuser/.gvfs/sftp on 11.22.33.44").
This way, supposing you have written eg. a Bash( command.s ) function, the following will work "as wished"

Code: Select all

Bash("ls -al home")   ; remote
The following will not, as "/" points to the local machine

Code: Select all

Bash("ls -al /home")  ; local
So, I guess the wish might be to have in the mentioned configuration page a "Create temporary executable in [browse for local/remote] directory" and keeping the debugging ability
mdp
Enthusiast
Enthusiast
Posts: 115
Joined: Mon Apr 18, 2005 8:28 pm

Post by mdp »

Sorry, I forgot:
Creating a sftp volume (the "Connect to Network" in at least Ubuntu Desktop) and doing "CreateExecutable" selecting a remote folder in the mounted volume as a path returns:
PureBasic - Linker error
---------------------------
/usr/bin/ld: final link failed: Operation not supported
Collect 2: ld returned 1 exit status
(Odd behaviour, my mistake or bug? I report that here consistent to the first half of the "wish", to directly put the executable in its remote place instead of creating it locally and then moving it)
freak
PureBasic Team
PureBasic Team
Posts: 5929
Joined: Fri Apr 25, 2003 5:21 pm
Location: Germany

Post by freak »

If you just set the current directory to a remote one, you still execute the actual program locally.

Routing the debugger data through a network connection is the easy part. I did that for Windows a while ago:
http://www.purebasic.fr/english/viewtopic.php?t=16415

I plan to add this functionality directly to the debugger for all OS. So the most basic solution would be this:
- compile with -d on the local machine
- move the exe to the remote machine manually
- start exe with a spechial parameter for network debugging
- connect from the local machine with the debugger

Of course thats not very convenient. Automating this would be better.

Automating the upload to the remote machine is not that big a deal.
I could add various ways to transfer the file (ftp, scp and such) to the IDE with some work or the use of libraries/external commands.

The question i am still debating in my head is how to best execute the program on the remote machine. I could put together a small sever program for this, but this does not sound like a good solution because having a program sit on your server that waits for outside commands
to execute local programs is quite the security risk.

Is there some standard way to do something like this in the Linux/Unix world ? (automated execution of a remote program, without manually logging in)

The perfect solution in my opinion would be to do the upload and execution though standard (and secure) means, so only the network protocol for the debugger would be something i implement myself.

What are your thoughts on this ?


About your bug: Looks like the linker cannot write to the folder, so its probably not a PB bug.
You could try compiling something to that folder with gcc, does that work ?
quidquid Latine dictum sit altum videtur
mdp
Enthusiast
Enthusiast
Posts: 115
Joined: Mon Apr 18, 2005 8:28 pm

Post by mdp »

One quick note, while I more properly reason:
Something like:

Code: Select all

pbcompilerd --listen=192.168.1.123 -p
might not be a major security breach:
- you add some more "firewalling" by telling the daemon to only accept connections from an address matching your developement machine;
- you ask for a password that should match one you specified in the configuration panel in the local IDE
And of course, as you finish your session, of course you will:

Code: Select all

pbcompilerd stop
mdp
Enthusiast
Enthusiast
Posts: 115
Joined: Mon Apr 18, 2005 8:28 pm

Post by mdp »

Transfering/executing a command in a remote entity with SSH services on (I guess that should be the minimum ground) is still ssh, implemented through 'ssh' in Linux and 'plink.exe' in Windows... The pass should be cached somewhere at your first login during the session
Edit: oops in Linux scp for transfer, ssh for executing, see below
Last edited by mdp on Sun Aug 03, 2008 6:08 pm, edited 2 times in total.
freak
PureBasic Team
PureBasic Team
Posts: 5929
Joined: Fri Apr 25, 2003 5:21 pm
Location: Germany

Post by freak »

The thing is: If the program could be executed on the remote machine without the need for a PB specific server program,
then you could remotely debug a program on a server that doesn't even have PB installed.
Installing PB just for the small server program to execute the debugged exe seems overkill to me.
quidquid Latine dictum sit altum videtur
mdp
Enthusiast
Enthusiast
Posts: 115
Joined: Mon Apr 18, 2005 8:28 pm

Post by mdp »

Timo, are you looking for a polished version of the following (an External Tool, with parameter %EXECUTABLE)?

Code: Select all

Procedure.s ShComplex( cmd.s )
   p=RunProgram("sh", "-c "+#DQUOTE$+cmd+#DQUOTE$, ".", #PB_Program_Read|#PB_Program_Open)
   While ProgramRunning(p)
      s.s = s+ReadProgramString(p) 
   Wend 
   ProcedureReturn s
EndProcedure

param.s=ProgramParameter(0) : If Not param : MessageRequester("Xfer2Server","No executable! Exiting...") : End : EndIf

ShComplex( "scp "+param+" root@11.22.33.44:/root/" ) ; transfer
ShComplex( "ssh root@11.22.33.44 '/root/"+GetFilePart(param)+"'" ) ; execute 

In my opinion, your way is:

Code: Select all

scp /path/to/my/exe user@address:remote/path
ssh user@address '/remote/path/to/exe --network-debugging'
Please not that the above works, but it is for some reason - maybe something related to the fact that the parameters are normally used within interactive console, not desktop manager - slow (it takes a while before asking you the passwords).
freak
PureBasic Team
PureBasic Team
Posts: 5929
Joined: Fri Apr 25, 2003 5:21 pm
Location: Germany

Post by freak »

Yes, something like this.

I'll experiment a bit with this and we'll see what comes from it.
I can't tell you when you are going to see anything solid though, as we have not finalized plans beyond PB 4.30 yet.

For the moment, the best advise is probably to compile from the commandline with the -d switch, copy to the remote directory and run the exe through ssh.
This way you atleast have the console debugger available on the server which is better than no debugging.
quidquid Latine dictum sit altum videtur
mdp
Enthusiast
Enthusiast
Posts: 115
Joined: Mon Apr 18, 2005 8:28 pm

Post by mdp »

Thank you Freak!

Well for those who may be interested, that's the environment I am now using. Steps:
* Mounting the / of the remote environment, linked from [...]/Desktop/pbtransfer (I personally used the "ConnectToServer" Gnome menu entry then made a symbolic link)
* Creating a script pbremote in the 'compilers' directory, containing

Code: Select all

#! /bin/bash
date 1>> /tmp/tmpdebug.log
/home/myuser/purebasic/compilers/pbcompiler $1 -d -e /home/myuser/Programming/pb.tmp  	2>> /tmp/tmpdebug.log
cp /home/myuser/Programming/pb.tmp /home/myuser/Desktop/pbtransfer/root/pb.tmp 		2>> /tmp/tmpdebug.log
chmod 777 /home/myuser/Desktop/pbtransfer/root/pb.tmp  					2>> /tmp/tmpdebug.log
* Setting an external tool with commandline pbremote and arguments %FILE, bound to [CTRL]+[F5]
* Keeping a ssh console ready (e.g. putty) to run the program there

Which is a proper environment for what I needed, until we will be able to directly use the IDE debugger :wink:
User avatar
mback2k
Enthusiast
Enthusiast
Posts: 257
Joined: Sun Dec 02, 2007 12:11 pm
Location: Germany

Post by mback2k »

freak, can you provide us with news on this topic, please?

I would really like to remote compile and debug on my brothers 64bit computer without disturbing him by taking over his screen.
freak
PureBasic Team
PureBasic Team
Posts: 5929
Joined: Fri Apr 25, 2003 5:21 pm
Location: Germany

Post by freak »

This is still on my list for things i like to do "soon" (note the "'s ;)). The list of things to do for v4.40 is pretty much complete now though, so i am hoping to do it for v4.50. This does not have the hightest priority for me as it is a bit specific and probably won't be used by most PB users.

btw, if you plan to debug on your brothers machine, you still have to connect to his screen (via VNC for example) unless your program has no gui at all to interact with.
quidquid Latine dictum sit altum videtur
mdp
Enthusiast
Enthusiast
Posts: 115
Joined: Mon Apr 18, 2005 8:28 pm

Re: Remotely complie/debug on Server

Post by mdp »

A little update on how I am (temporarily :wink: ) remote debugging

I create an sshfs mount to the remote test machine - that's inexpensive, you do it once per session - then use the following purebasic IDE tool, bound as:
Remote Debugger - Configure
Remote Debugger ____ Ctrl+F5

The latter will take %FILE as parameter, the former will provide an interface to record the needed data (connection locators, paths etc.)

the basic trick, after copying the temporary executable in the remote machine, is:

Code: Select all

RunProgram( "gnome-terminal" , "-x sshpass -f sshpass_passstorage_file ssh -t myuser@11.22.33.44 /path/to/exe -d ;read" , "" )
Sorry for the not so pretty code

Code: Select all

Enumeration
   #mwin : #g_dst : #l_dst : #g_pth : #l_pth : #g_tpth : #l_tpth : #g_pw : #l_pw : #b_sav
EndEnumeration

Global rd_dst.s      ; myuser@11.22.33.44
Global rd_pw.s       ; mYy53r
Global rd_pth.s      ; /remotehome/coder/tempexe/
Global rd_tpth.s     ; /local/sshfs_mountpoint/remotehome/coder/tempexe/


Procedure.s GetShellOutput( cmd.s )
   prg = RunProgram("sh","-c "+#DQUOTE$+cmd+#DQUOTE$,"",#PB_Program_Open|#PB_Program_Read)
   If prg
      While ProgramRunning(prg) 
         out.s=out+ReadProgramString(prg)+#LF$
      Wend
   EndIf
   ProcedureReturn out
EndProcedure

Procedure SaveCnf( user_at_ip.s , pw.s  , remote_path.s , tunnel_path.s , cfgpth.s )
   If OpenFile(0,cfgpth+"remotedbg.conf")
      WriteStringN(0,pw)
      WriteStringN(0,user_at_ip)
      WriteStringN(0,remote_path)
      WriteStringN(0,tunnel_path)
      CloseFile(0)
   EndIf
EndProcedure

Procedure LoadCnf( cfgpth.s )
   Debug cfgpth+"remotedbg.conf"
   If ReadFile(0,cfgpth+"remotedbg.conf")
      rd_pw =ReadString(0) 
      rd_dst=ReadString(0)
      rd_pth=ReadString(0)  : If Right(rd_pth,1)<>"/"  : rd_pth =rd_pth +"/" : EndIf  
      rd_tpth=ReadString(0) : If Right(rd_tpth,1)<>"/" : rd_tpth=rd_tpth+"/" : EndIf  
   EndIf
EndProcedure

home.s = StringField(ReplaceString(GetShellOutput("env | grep -e ^HOME="),#LF$,""),2,"=") : If Right(home,1)<>"/" : home=home+"/" : EndIf : Debug home
LoadCnf(home+".purebasic/")

; -

prm.s = ProgramParameter()       ;    "--config"   OR   "%FILE" from PB IDE tools

If prm="--config"

      If OpenWindow(#mwin,0,0,408,124,"Remote Debugging Configuration",#PB_Window_ScreenCentered)
         TextGadget(#l_dst,  4,  8, 80, 20,"user@ip_addr"): StringGadget(#g_dst,  84,  8,320, 20,"")
         TextGadget(#l_pw ,  4, 28, 80, 20,"pass")        : StringGadget(#g_pw ,  84, 28,320, 20,"",#PB_String_Password)
         TextGadget(#l_pth,  4, 48, 80, 20,"remote path") : StringGadget(#g_pth,  84, 48,320, 20,"")
         TextGadget(#l_tpth, 4, 68, 80, 20,"sshfs path")  : StringGadget(#g_tpth, 84, 68,320, 20,"")
         ButtonGadget(#b_sav,4, 92,400, 24,"SAVE")
      Else : MessageRequester("Error!","Cannot open window!") : End : EndIf
      
      SetGadgetText(#g_pw ,rd_pw) : SetGadgetText(#g_dst,rd_dst) : SetGadgetText(#g_pth,rd_pth) : SetGadgetText(#g_tpth,rd_tpth)
      
      Repeat
         EvID=WaitWindowEvent()
         Select EvID
            Case #PB_Event_Gadget
               EvG=EventGadget()
               Select EvG
                  Case #b_sav
                     SaveCnf( Trim(GetGadgetText(#g_dst)) , Trim(GetGadgetText(#g_pw)) , Trim(GetGadgetText(#g_pth)) , Trim(GetGadgetText(#g_tpth)) , home+".purebasic/" )
               EndSelect
         EndSelect
      Until EvID=#PB_Event_CloseWindow

Else
      
      If Not FindString(prm,".pb",1) : MessageRequester("Error!","Source %FILE is: "+prm) : EndIf
      src_pth.s=GetPathPart(prm) : cmplr.s = #PB_Compiler_Home : cmplr=cmplr+"compilers/pbcompiler"
      ok=RunProgram(cmplr,prm+" -d -e rdbgelf",src_pth,#PB_Program_Wait)
      ok=CopyFile(src_pth+"rdbgelf",rd_tpth+"rdbgelf") 
      If ok
         GetShellOutput("chmod 700 "+rd_tpth+"rdbgelf")
         RunProgram("gnome-terminal","-x sshpass -f "+home+".purebasic/remotedbg.conf ssh -t "+rd_dst+" "+rd_pth+"rdbgelf -d ;read","")
      Else
         MessageRequester("Copy failed from/to",src_pth+"rdbgelf"+#LF$+rd_tpth+"rdbgelf")
      EndIf

EndIf
This until tunnelling of debugging info allows for the use of a GUI debugger.
freak
PureBasic Team
PureBasic Team
Posts: 5929
Joined: Fri Apr 25, 2003 5:21 pm
Location: Germany

Re: Remotely complie/debug on Server

Post by freak »

> This until tunnelling of debugging info allows for the use of a GUI debugger.

You'll be glad to hear that I have implemented network debugging for the upcoming v4.50 release.

Features:
- both debugger and exe can act as either client or server (good if one is behind a router)
- works across platforms (you can connect the Windows x64 debugger to an OSX PPC program if you want)
- password protection/encryption built in (AES)

You still have to do the remote compiling or transfer of the file yourself for the moment. I want to add remote compiling as well at some point but its not a priority. This new network feature should help a good deal in debugging server programs already. You can just use the tool you posted with an extra line to launch the local gui debugger and connect it to the program.
quidquid Latine dictum sit altum videtur
Post Reply