How to get stdout from Linux terminal?

Just starting out? Need help? Post your questions and find answers here.
AZJIO
Addict
Addict
Posts: 1312
Joined: Sun May 14, 2017 1:48 am

How to get stdout from Linux terminal?

Post by AZJIO »

How to get stdout from Linux terminal? I already broke my brain.

Code: Select all

Procedure.s ExeComStr(Execut$, ComStr$)
	Protected tmp, res$
	tmp = RunProgram(Execut$, ComStr$, "", #PB_Program_Open | #PB_Program_Read)
; 	Debug Execut$
; 	Debug ComStr$
	res$ = ""
	If tmp
		While ProgramRunning(tmp)
			If AvailableProgramOutput(tmp)
				res$ + ReadProgramString(tmp) + Chr(10)
			EndIf
		Wend
		CloseProgram(tmp)
	EndIf

	ProcedureReturn res$
EndProcedure

; find '/home/user/' -regex '.+/[A-Za-z]+\.\w\w' -type f
tmp$ = "'/home/user/' -regex '.+/[A-Za-z]+\.\w\w' -type f"

tmp$ = ExeComStr("gnome-terminal", "-- bash -c find " + tmp$ + " 2>&1")
; tmp$ = ExeComStr("gnome-terminal", "-- bash -c " + Chr(34) + "find " + tmp$ + " 2>&1" + Chr(34))
Debug tmp$
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: How to get stdout from Linux terminal?

Post by NicTheQuick »

Just execute "find" directly without any "gnome-terminal" or "bash". But you need the full path then. I try to make an example for you. Please wait.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: How to get stdout from Linux terminal?

Post by NicTheQuick »

Instead of ReadProgramData you also could use ReadProgramString I guess. But I had more fun with ReadProgramData. 8)

Code: Select all

EnableExplicit

#BUFFER_SIZE = 1024

Procedure.s getOutput(command.s, args.s)
	Protected hProg.i
	Protected stdout.s, stderr.s
	Protected avail.i, *buffer = AllocateMemory(#BUFFER_SIZE)
	
	hProg = RunProgram(command, args, "", #PB_Program_Open | #PB_Program_Read | #PB_Program_Error | #PB_Program_UTF8)
	If hProg
		While ProgramRunning(hProg)
			avail = AvailableProgramOutput(hProg)
			If avail
				If avail > #BUFFER_SIZE
					avail = #BUFFER_SIZE
				EndIf
				avail = ReadProgramData(hProg, *buffer, avail)
				stdout + PeekS(*buffer, avail, #PB_ByteLength | #PB_UTF8)
			EndIf
			stderr + ReadProgramError(hProg)
		Wend
		CloseProgram(hProg)
	EndIf

	FreeMemory(*buffer)
	
	If stderr
		Debug "======ERROR====="
		Debug stderr
		Debug "================"
	EndIf

	ProcedureReturn stdout
EndProcedure

Debug getOutput("/usr/bin/find", "/home/nicolas/tmp -type d")

Debug getOutput("/usr/bin/find", "/home/nicolas/tmp -type r")
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
AZJIO
Addict
Addict
Posts: 1312
Joined: Sun May 14, 2017 1:48 am

Re: How to get stdout from Linux terminal?

Post by AZJIO »

if i use "find" i can't add "|sort"

download: yandex, upload.ee

Image
Last edited by AZJIO on Fri Mar 05, 2021 4:35 pm, edited 3 times in total.
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: How to get stdout from Linux terminal?

Post by NicTheQuick »

Ah, you want to concatenate commands like bash can do it?
Then just call it like so:

Code: Select all

Debug getOutput("/bin/bash", ~"-c \"find /home/nicolas/tmp -type d | sort\"")
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
AZJIO
Addict
Addict
Posts: 1312
Joined: Sun May 14, 2017 1:48 am

Re: How to get stdout from Linux terminal?

Post by AZJIO »

I've done a call count (b). Initially it was b=1500, but when I increased #BUFFER_SIZE 10 times, then the number of calls became 150. Increasing the size even more did not reduce the number of calls.

Code: Select all

EnableExplicit

; #BUFFER_SIZE = 1024
#BUFFER_SIZE = 10240

Procedure getOutput(*Result.string)
	Protected NewList Files.s(), *Point, Len = 0
	Protected hProg.i;, c=0, b=0
	Protected stderr.s, tmp$
	Protected avail.i, *buffer = AllocateMemory(#BUFFER_SIZE)
	Debug *Result\s
	
	hProg = RunProgram("bash", *Result\s, "", #PB_Program_Open | #PB_Program_Read | #PB_Program_Error | #PB_Program_UTF8)
	If hProg
		While ProgramRunning(hProg)
; 			c+1
			avail = AvailableProgramOutput(hProg)
			If avail
; 				b+1
				If avail > #BUFFER_SIZE
					avail = #BUFFER_SIZE
				EndIf
				avail = ReadProgramData(hProg, *buffer, avail)
	            If AddElement(Files())
	            	Files() = PeekS(*buffer, avail, #PB_ByteLength | #PB_UTF8)
	            	Len+avail
	            EndIf
				tmp$ = ReadProgramError(hProg)
				If tmp$
					stderr + tmp$ + Chr(10)
				EndIf
			EndIf
		Wend
		CloseProgram(hProg)
	EndIf
	
	FreeMemory(*buffer)
	
	Debug Len
	*Result\s = Space(Len)
	*Point = @*Result\s
	ForEach Files()
		CopyMemoryString(Files(), @*Point)
	Next
	ClearList(Files())
	
; 	Debug c
; 	Debug b
	If stderr
		Debug "======ERROR====="
		Debug stderr
		Debug "================"
	EndIf

EndProcedure

; If you add "|sort -r" then "access denied" is printed to the error stream
Define Result.string
Result\s = "-c " + Chr(34) + "find '/home/user/' -regex '^.*?\.\(png\|jpg\|gif\)$' -type f|sort -r 2>&1" + Chr(34)
getOutput(@Result)
Debug Result\s
I read binary and line by line, then compare the results. Differ considerably. 52653 vs 54748 files. Compare in Meld. In the binary version, it is as if part of the data is cut off at each iteration step.
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: How to get stdout from Linux terminal?

Post by NicTheQuick »

I've done a call count (b). Initially it was b=1500, but when I increased #BUFFER_SIZE 10 times, then the number of calls became 150. Increasing the size even more did not reduce the number of calls.
I guess the standard buffer size is 8192 bytes (8kB) and because of that you'll see no difference in the ReadProgramData() calls if you the define the #BUFFER_SIZE bigger than that.
I read binary and line by line, then compare the results. Differ considerably. 52653 vs 54748 files. Compare in Meld. In the binary version, it is as if part of the data is cut off at each iteration step.
Well, that's bad. I just tried it out with a bigger directory, consisting of 406390 files and directories but Purebasic is so fucking slow at string concatenation that I have to wait minutes before I see any result. I expect the string to be round about 49 MB in size:

Code: Select all

nicolas@tp-w530:~$ find /home/nicolas/Install | wc -c
49013651 [bytes]
nicolas@tp-w530:~$ find /home/nicolas/Install | wc -l
406390 [lines]
I modified my version a bit and now it takes only 1.3 seconds to get the files (with Debugger on!) and the result is correct.

Code: Select all

EnableExplicit

#BUFFER_SIZE = 8192

Procedure.s getOutput(command.s, args.s)
	Protected hProg.i
	Protected stdout.s, stderr.s
	Protected avail.i, bytesRead.i, bufferSize.i
	Protected bufferCapacity.i = #BUFFER_SIZE, *buffer = AllocateMemory(bufferCapacity, #PB_Memory_NoClear), *tBuffer
	
	hProg = RunProgram(command, args, "", #PB_Program_Open | #PB_Program_Read | #PB_Program_Error | #PB_Program_UTF8)
	If hProg
		While ProgramRunning(hProg)
			avail = AvailableProgramOutput(hProg)
			If avail
				If avail > bufferCapacity - bufferSize
					While avail > bufferCapacity - bufferSize
						; Double up the buffer size each time it is too small for new data
						bufferCapacity * 2
					Wend
					*tBuffer = ReAllocateMemory(*buffer, bufferCapacity, #PB_Memory_NoClear)
					If Not *tBuffer
						DebuggerError("Out of memory. Can not allocate " + bufferCapacity + " bytes.")
						Break
					EndIf
					*buffer = *tBuffer
				EndIf
				Repeat
					bytesRead = ReadProgramData(hProg, *buffer + bufferSize, avail)
					avail - bytesRead
					bufferSize + bytesRead
				Until avail = 0
				
			EndIf
			stderr + ReadProgramError(hProg)
		Wend
		CloseProgram(hProg)
	EndIf
	
	stdout = PeekS(*buffer, bufferSize, #PB_ByteLength | #PB_UTF8)

	FreeMemory(*buffer)
	
	If stderr
		Debug "======ERROR====="
		Debug stderr
		Debug "================"
	EndIf

	ProcedureReturn stdout
EndProcedure

Define time.i = ElapsedMilliseconds()

Define files.s
files = getOutput("/bin/bash", ~"-c \"find /home/nicolas/Install\"")

Debug "Time: " + Str(ElapsedMilliseconds() - time) + " ms"

Debug "Count: " + Str(CountString(files, #LF$))

Debug "Bytes: " + StringByteLength(files, #PB_UTF8)
And the result is:
Debugger wrote:Time: 1371 ms
Count: 406390
Bytes: 49013649
The bytes length only differs in 2 bytes. I guess that is something `StringByteLength()` does wrong after the string was converted to Unicode in between. If you debug the variable `bufferSize` you will get the exact same amount of data as `wc -c` shows you.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
AZJIO
Addict
Addict
Posts: 1312
Joined: Sun May 14, 2017 1:48 am

Re: How to get stdout from Linux terminal?

Post by AZJIO »

I guess the standard buffer size is 8192 bytes (8kB)
I substituted 9024, it was not enough

at the moment I am unable to get the list of files using your code
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: How to get stdout from Linux terminal?

Post by NicTheQuick »

AZJIO wrote:
I guess the standard buffer size is 8192 bytes (8kB)
I substituted 9024, it was not enough

at the moment I am unable to get the list of files using your code
That's weird. What are you getting instead?
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
Post Reply