RunProgram("sh",...) fails after 1000+ calls

[mdp]

I really need a hand on this.

I use a GetShellOutput( commandstring.s ) procedure to interface to the Linux shell commands (ls, du, df, env etc.).

If you iterate a RunProgram more than around 1000 times, RunProgram will return 0 (fail). On a production server it lasts 1016, on my local machine 1006 or 1003.

Code: Select all

Procedure.s GetShellOutput( cmd.s )
   Static tmpcount
   prg = RunProgram(cmd,"","",#PB_Program_Open|#PB_Program_Read) : tmpcount+1 : Debug tmpcount
   If prg
      While ProgramRunning(prg) 
         out.s=out+ReadProgramString(prg)+#LF$
      Wend
   Else
      Debug "!!! Cannot RunProgram "+#DQUOTE$+cmd+#DQUOTE$
   EndIf
   ProcedureReturn out
EndProcedure

For a=0 To 1100
   Debug GetShellOutput("date")
Next

I really do not understand why this happens. It looks like it encounters some OS limit, but the called (sub)program has to be over as it exits the ReadProgramString() loop.
Any help is very appreciated.

[mdp]

Through strace, it appears that a pipe() systemcall starts returning 1 EMFILE (Too many open files) as the issue happens

Code: Select all

access("/bin/date", X_OK)               = 0
pipe([1020, 1021])                      = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f9341c097c0) = 25273
[...]
access("/bin/date", X_OK)               = 0
pipe([1021, 1022])                      = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f9341c097c0) = 25275
[...]
access("/bin/date", X_OK)               = 0
pipe([1022, 1023])                      = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f9341c097c0) = 25277
[...]
access("/bin/date", X_OK)               = 0
pipe(0x7fff57569740)                    = -1 EMFILE (Too many open files)
write(1, "1021\n", 51021
)                   = 5
write(1, "!!! Cannot RunProgram \"date\"\n", 29!!! Cannot RunProgram "date"
) = 29

[UPDATE] - The pipe() fails at the 1024th supposed call

[Fred]

You need to call CloseProgram() if you use #PB_Program_Open: http://www.purebasic.com/documentation/ ... ogram.html

[ts-soft]

You need to call CloseProgram() if you use #PB_Program_Open: http://www.purebasic.com/documentation/ ... ogram.html

You should add this to the example in the doc
The example is bad

Greetings
Thomas

[mdp]

Now I-feel-so-dumb. (Actually like "That command was not there before..." ... and of course it was)
Sorry!

[Fred]

You need to call CloseProgram() if you use #PB_Program_Open: http://www.purebasic.com/documentation/ ... ogram.html

You should add this to the example in the doc
The example is bad

Greetings
Thomas

Good point, it's modified.

[MikeM]

Hi.
I've got the same result through valgrind under openSuse11.1:
#valgrind /tmp/purebasic_compilation0.out
// it's the executable.

Code: Select all

==15826== Warning: invalid file descriptor 1029 in syscall pipe()
[Debugger]  1021
[Debugger]  !!! Cannot RunProgram "date"
[Debugger]

To my mind, it's the limitation of OS LINUX( the size of pipe buffer). To check it out, you may try
the next code:
buffsize.c:

Code: Select all

#include <stdio.h>
#include <unistd.h>
int main() {
  int pipe_file_d[2];
  int value;
  pipe(pipe_file_d);
  // without handling error
 value = fpathconf(pipe_file_d[0], _PC_PIPE_BUF);
 printf("PIPE_BUF = %d\n", value);
 printf("SIZE_OF_INT = %d\n", sizeof(pipe_file_d[0]));
 return 0;
}

gcc buffsize.c; ./a.out
you will get:
PIPE_BUF = 4096
SIZE_OF_INT = 4
That's why you cannot start more then 1024 processes in one shell, IMHO.
Best regards, Mike.

[mdp]

Thanks Mike,
but the inability to start and keep more than 1024 processes should not be a problem - the need should not occur.
You can start and end processes as long as you need.
If you do a RunProgram with the #PB_Program_Open flag, a pipe is also created - and this one should also be ended as soon as it is not needed (my mistake was that I thought it would be killed together with the process end).

Thanks for the interesting input, I will try valgrind (best presentation pic ever).

[MikeM]

Hi,mdp.
Thanks for the clarification.Naturally, i've have been stupid last post.
I mean another thing: the number of simultaneously open-file descriptors
for one process in Linux(default is 1024).
To check it you may:
> ulimit -aS
To increase your limits, you may do the following steps:
1) At the end of /etc/security/limits.conf add two lines(as root).
* soft nofile 65535
* hard nofile 65535
2) #echo 65535 > /proc/sys/fs/file-max
3) #reboot
Then your example works fine, without any correct.

Best regards, Mike.