Slow down compression in purebasic?

[oakvalley]

I really like the compression ratio of purebasic's internal Packer (JCalG1), but, problem is that every compression scheme I search for here in forums and in google dont have any option for not using high amount of cpu.

All compressor schemes seems to be based on memory usage, compression speed and decompression.

In my project, i have every 10 second a need to compress a .bmp file without gearing the cpu up to 10%. If it can be slowed down, we might end up around 0.5% over let's say 8 seconds. Is that possible?

Basically, my project consists of grabbing the desktop, saving the bmp and send it over network (with no page faults, almost no cpu-time used every 10 second).

I have tried various 24bit image to 8 and 4 bit conversion, but they all either end up with page faults or using too much cpu. Even Purebasic's saveimage with compression and 4 as bit-flag bogs my loop down and simply is not suitable. Saving the bmp straight out, with no compression or bit-flag set, uses most likeley no cpu-time, but I end up around 80-147kb pr image. Sending that over network is not so cool. I have gotten the same file and suitable image data down to about 2-3kb, but that with the expense of page faults and too much cpu-time used every 10 second.

Right now, I have no page faults in my code and cpu-time even on slow machines is only about 1-2% every 10 second. But, again, the filesize is really to high.

Any ideas about slow compression schemes?

[c4s]

Just guessing: Maybe you could put your compression code into a thread and give it the lowest priority level?

[IdeasVacuum]

Have you tried PB's SaveImage() to output a jpg?

[ts-soft]

JCalG1 is the slowest in packing, use another one.

[buddymatkona]

c4s has the right idea. I think the easiest way to limit such things is to launch them with Task Scheduler to run in idle time.

[Rook Zimbabwe]

a 256 color PNG is pretty small...

sounds weird to keep grabbing screenshots and sending over the web... Is this some new method of video chat?

[IdeasVacuum]

I think the point is in danger of being missed, because slowing down the compression is not actually the goal if you read carefully.

The goal is to send a screenshot of the desktop over the network every 10 seconds without using a high amount of cpu. So, if the process of compressing the image could be avoided all together, that would be the Utopian condition since currently it is compressing the image that hikes-up the cpu usage.

One possibility, depending on what the data is to be used for, would be to send a complete screenshot as the first image, then send only the differences in subsequent files. Or perhaps a JPG or PNG could be sent without compression. Does the entire desktop have to be captured or could the capture region be reduced to something more specific? If the entire desktop image has to be sent, could the compression be applied more efficiently at the receiving end? (server?). For example, the uncompressed images could be stored until a latent period of time on the server could be used to batch-process them.

[oakvalley]

Hi all,

Some more background why I'm seeking to slow down compression:

The software I want to create is kind of similar to VNC, TeamViewer but without any attempts to control that machine remotely, just view. The reason is that sometimes I have several computers running, with various tasks such as rendering, Amiga/C64 recording, game machines, servers, robot-home pc's or whatever cool stuff etc., and they are mostly without any monitor and located different places in the house..uhum.

So, my first idea was. Do there exists a small hardware dongle-like connector that takes vga/dvi-i out signal of the computer port and send the image via lan?
So I can keep track of monitor-less machines. And fiddeling with VNC, teamviewer will eventually be too much hassle and they also are loaded with other features I do not need at all, not to mention the eat a heck alot more of cpu-time than my software do today naturally

Since I'm not sure if there existed any dongle-like (not much bigger than a DVI connector for instance) that sends whatever output coming out (be that blue screen, bios, booting, dos, windows or whatever signal the off-site computer with no monitor is showing) I decided to think software (of course, only windows desktop gets captured or full screen stuff, no dos, bios etc..naturally).

So, I thought why not grab the desktop in a 4x scaled down image, save that as BMP 4bit, compressed with RLE (like purebasic do?) and just save the file to a common network path that a receiver program would pick up and show tiled and kinda a like a "billboard", which is the name of the software too "Billboard". One server that captures the desktop every 10sec or whatever delay one want and save that file to a network path.

Today, it works great, without any page faults and the file is about 80kb-140kb BMP uncompressed as 24bit image. Also, saving it with purebasic 4bit, compressed it gets down to just a few kb, and thats not much data to save over the network every 10 sec for a machine, but eats cpu-time.

Things I discovered was:
- Compressing of any kind will eat cpu time for about 1sec, jumping the CPU up to 1%-6% depending on the hardware. It works on Win95 and up to Win7 even!
- Saving a 80kb-140kb over the network also eats cpu time I have noticed. 2 few kb don't show any sign of that.
- During a idle capture session for 2 days, I had only used 40sec cpu-time totall for 48hours saving just locally. With network saving, it gets over 1 minute. That was uncompressed BMP, then.

Since I cannot control the BMP compression loop I cannot control the about of CPU spike being used.

And that was my reason for slowing down compression loop so it don't eat spikes of cpu time that gets logged. I want my software to be as idle and non-eating resource-monster at all!

I also tried thread and priority level, but since there aren't many full-source-viewable around, I have still no control of for instance a external packer program or even purebasic's internal packer and bmp compression since that is just a line of code, not 2 pages of it that I could slow down in the main loop of it

Even if I set my own software to "low" priority,it will still eat cpu-time when doing compression atleast with packer, purebasic saveimage for instance. Or Trond's huffman code on the forums here too. Putting a delay(0) or delay(1) in crucial places will also not 100% reduce the cpu-spike.

Saving to JPG eats cpu-time and lowering the quality is also not desired.
Saving to PNG eats cpu-time with compression.
Jcalq may be slow, but speed is not the issue here. Compression size and amount of cpu-time spikes are.

Sure, one could do comparsion and differences sending over the bytes, but my belief is that too much of that code will even eat cpu-time over time.

My goal is to capture whatever resolution, not regions. Since my image is scaled down 6x (for instance), a 1024x768 is unreadable for the eye, but that's the idea of Billoard, I just want to know in a thumbnail-kind-of-way what the machine is doing, or if it finished off a process/rendering. Just try to grab your desktop scale it down to 6x as raw-pixel and tell me if you can tell the difference between a clean desktop or a yellow ballon popping up telling you about Microsoft updates, or even if the strange Flash Installer wants to grab your attention. Sure, its easy to see what's going on in Windows by several meters or just by looking at a very small thumbnail was my logic

In the end, my second goal is to put the receiver software on a panel-pc touchscreen located in my living room, with a billboard of what the other computers are doing, maybe they crashed, still working, finished off whatever I told them to do etc.

[oakvalley]

Just one more thing...

Is "idle" different from "low" priority? How do I set that?
This is what I use today to get "low":

handler=GetCurrentProcess_()
SetPriorityClass_(handler,64)

[graph100]

from what I've understand your post, I think the best is to re-code the function for compression, and slow it down by placing delay(0) function at strategically chosen point.
Here a little exemple, it palettize the bitmap to the number of color you want and write it's own image format.
the resulting file must be read with a special function that you have to write for the server side.

I've written this quickly, so there are many way to improve it. such as controlling the bits, the zero values, remove the use of the point() function, etc...
It reduce the size of a bitmap by nearly 3 times, and the image stay pretty readable (as you point earlier, the snapshot is reduce by 6)

Code: Select all

#dist_max = 442

Global snap.l, nb_color = 25


Procedure DesktopSnapShot(coef.d)
	Ecran_Largeur = GetSystemMetrics_(#SM_CXSCREEN)
	Ecran_Hauteur = GetSystemMetrics_(#SM_CYSCREEN)
	
	DC = GetDC_(0)
	
	img = CreateImage(#PB_Any, Ecran_Largeur, Ecran_Hauteur)
	
	Dessin = StartDrawing(ImageOutput(img))
	
	If Dessin
		BitBlt_(Dessin, 0, 0, Ecran_Largeur, Ecran_Hauteur, DC, 0, 0, #SRCPAINT | $40000000)
		
		StopDrawing()
	EndIf
	
	ReleaseDC_(0, DC)
	
	w.l = Ecran_Largeur / coef
	h.l = Ecran_Hauteur / coef
	
	CreateImage(snap, w, h)
	
	If StartDrawing(ImageOutput(snap))
		DrawImage(ImageID(img), 0, 0, w, h)
		
		StopDrawing()
	EndIf
	
	FreeImage(img)
	
	
EndProcedure


ProcedureDLL.l GetImageBits2(ImageID, HList) ; Transfert d'une image vers un tableau
	Protected bmi.BITMAPINFO, hdc.l, Resultat, Mem, n, nn, bm.BITMAP
	
	Resultat = 0
	
	GetObject_(ImageID, SizeOf(BITMAP), @bm.BITMAP)
	
	bmi\bmiHeader\biSize = SizeOf(BITMAPINFOHEADER)
	bmi\bmiHeader\biWidth = bm\bmWidth
	bmi\bmiHeader\biHeight = bm\bmHeight
	bmi\bmiHeader\biPlanes = 1
	bmi\bmiHeader\biBitCount = 32
	bmi\bmiHeader\biCompression = #BI_RGB
	
	Mem = AllocateMemory (bm\bmWidth * bm\bmHeight * 4)
	If Mem
		
		hdc = CreateCompatibleDC_(GetDC_(ImageID))
		If hdc
			GetDIBits_(hdc, ImageID, 0, bm\bmHeight, Mem, @bmi, #DIB_RGB_COLORS ) ; on envoie la liste dans l'image
			ReleaseDC_(0, hdc)
			Resultat = ImageID
		EndIf
		
		; On convertit la liste dans le bon format
		For n = 0 To bm\bmHeight - 1
			For nn = 0 To bm\bmWidth - 1
				CopyMemory (Mem + nn * 4 + (bm\bmHeight - 1 - n) * bm\bmWidth * 4, HList + n * 4 + nn * bm\bmHeight * 4, 4)
			Next
		Next
		
		FreeMemory (Mem)
	EndIf
	
	ProcedureReturn Resultat
EndProcedure

Structure colorarray
	r.a
	g.a
	b.a
	
	nb_elem.l
EndStructure


Macro Distance(r1, g1, b1, r2, g2, b2)
	Sqr((r1 - r2) * (r1 - r2) + (g1 - g2) * (g1 - g2) + (b1 - b2) * (b1 - b2))
EndMacro


Procedure SaveCompression(fichier$)
	
	If nb_color > 255
		nb_color = 255
	EndIf
	
	w.l = ImageWidth(snap) - 1
	h.l = ImageHeight(snap) - 1
	
; 	Dim img.l(w, h)
; 	GetImageBits2(ImageID(snap), @img())
	
	NewList color.colorarray()
	
	seuil = 5
	
	If StartDrawing(ImageOutput(snap))
		For x = 0 To w
			For y = 0 To h
				
				color = Point(x, y)
				
; 				r.a = Blue(img(x, y))
; 				g.a = Green(img(x, y))
; 				b.a = Red(img(x, y))
				r.a = Red(color)
				g.a = Green(color)
				b.a = Blue(color)
				
				ForEach color()
					d = Distance(r, g, b, color()\r, color()\g, color()\b)
					
					If d <= seuil
						color()\nb_elem + 1
						
						Break
					EndIf
				Next
				
				If d > seuil Or ListSize(color()) = 0
					AddElement(color())
					
					color()\r = r
					color()\g = g
					color()\b = b
					
					color()\nb_elem = 1
				EndIf
				
			Next
		Next
		StopDrawing()
	EndIf
	
	Debug "#### LISTE SIZE = " + Str(ListSize(color()))
	
	If ListSize(color()) > nb_color
		
		SortStructuredList(color(), #PB_Sort_Descending, OffsetOf(colorarray\nb_elem), #PB_Sort_Integer)
		
		SelectElement(color(), nb_color - 1)
		
		; 		Debug nb_element_seul
		
		
		
		While NextElement(color())
			; 			Debug "r=" + StrU(color()\r, #PB_Ascii) + "  g=" + StrU(color()\g, #PB_Ascii) + "  b=" + StrU(color()\b, #PB_Ascii) + "  nb=" + Str(color()\nb_elem)
			
			*pointer.colorarray = color()
			dist.l = #dist_max
			
			
			ForEach color()
				
				d = Distance(color()\r, color()\g, color()\b, *pointer\r, *pointer\g, *pointer\b)
				
				If d =< dist
					dist = d
					
					color()\nb_elem + *pointer\nb_elem
				EndIf
				
				If ListIndex(color()) - 1 >= nb_color : Break : EndIf
			Next
			
			ChangeCurrentElement(color(), *pointer)
			DeleteElement(color())
		Wend
		
	EndIf
	
	
	Debug "##############################"
	Debug "#### LISTE SIZE = " + Str(ListSize(color()))
	
	; 	ForEach color()
	; 		Debug "r=" + StrU(color()\r, #PB_Ascii) + "  g=" + StrU(color()\g, #PB_Ascii) + "  b=" + StrU(color()\b, #PB_Ascii) + "  nb=" + Str(color()\nb_elem)
	; 	Next
	
	
	; writting of the file
	
	If CreateFile(0, fichier$)
		
		
		; header : mind the format for decoding later (on the server)
		
		WriteAsciiCharacter(0, nb_color) ; number of color
		
		ForEach color()
			WriteLong(0, RGB(color()\r, color()\g, color()\b))
		Next
		
		If StartDrawing(ImageOutput(snap))
			For x = 0 To w
				For y = 0 To h
					
					dist.l = #dist_max
					*pointer = 0
					index = 0
					color = Point(x, y)
					
					
					ForEach color()
						d = Distance(color()\r, color()\g, color()\b, Red(color), Green(color), Blue(color))
						
						If d =< dist
							dist = d
							index = ListIndex(color())
							*pointer = color()
						EndIf
					Next
					
					Plot(x, y, RGB(*pointer\r, *pointer\g, *pointer\b))
					
					WriteAsciiCharacter(0, index)
				Next
			Next
			
		
		StopDrawing()
	EndIf
		
		CloseFile(0)
	EndIf
	
EndProcedure


If OpenWindow(0, 0, 0, 500, 500, "", #PB_Window_ScreenCentered | #PB_Window_SystemMenu)
	ImageGadget(0, 0, 0, WindowWidth(0), WindowHeight(0), 0)
	
EndIf


DesktopSnapShot(6)

SaveCompression("test2.image")
SaveImage(snap, "test.bmp")

SetGadgetState(0, ImageID(snap))

Repeat
	event = WaitWindowEvent()
	
	
	
	
Until event = #PB_Event_CloseWindow

End

I made a graphical output juste for the purpose of showing the result easily.

[Fig]

You can also use quadtrees to compress your image.Then you can sort them by color and inside each section of color sort them by size.

#color1#size1#quad1(X,Y)#quad2(X,Y)#size2#quad3(X,Y)...etc..#color2#size1#...

[graph100]

I don't quite understand your method. what represent the quad ? And isn't writing the location of each pixel too big ?
I have an idea of a compression, I will try to see if it's interessing.

[graph100]

Well, well,
http://www.purebasic.fr/english/viewtop ... 12&t=38606

In this topic you can find a compression algoryhme, so maybe, you can use it, and slow this as you want (I think this is the simpliest way)

[Thorium]

I think you should not be so concerned about little spikes in the CPU usage.
Setting the priority to low should be the best you can do. And better use zlib for compression, if you want to compress.

I guess you want that your program does not slow down other programs by grabbing to much CPU time. That will not happen with low priority. There will be spikes, but there is nothing wrong with spikes. Windows priority systems works very well. Having a spike does not mean that your program takes CPU time that other programs need. If the priotity is low it means that no other program needs the CPU time, so windows gives it to your program.

Idle priority means that your program gets only CPU time if no other program needs some at all. That means if a program takes 100% CPU your program will never do a screenshot, so stick with low priority.

Well, well,
http://www.purebasic.fr/english/viewtop ... 12&t=38606

Thats just for decompression, not for compression.

[oakvalley]

I downloaded the zlib and found a example code in C called "minigzip.c". I got my C compatible friend to add some buffer size and slowdown parameters for me:

I'm not sure exactly what he did, but here is some of it:

---------------------------------------------------------
for (;;) {
usleep(globalslowdown);
len = (int)fread(buf, 1, sizeof(buf), in);
if (ferror(in)) {
perror("fread");
exit(1);
}
if (len == 0) break;

if (gzwrite(out, buf, (unsigned)len) != len) error(gzerror(out, &err));
}
fclose(in);
if (gzclose(out) != Z_OK) error("failed gzclose");
-------------------------------------------

And it does work by compressing xx number of bytes with a small delay, making no cpu-spikes it seems. A standard compiling of the minigzip.c resulted in a 2% at least cpu usage during compression. The modified version worked longer naturally and showing 0% in cpu-usage.

For now, it was just compiled as an .exe which I use runprogram with the following modified command line
"minigzip.exe -b5000 -s50 -9 test.bmp", where -b and -s are new parameters.

Where it compress in each delay 5000 bytes with a 50 nanosecond delay. This is just an example number naturally, if I lower the -b to -b50, it takes longer to compress, several seconds actually which is what I'm after. I haven't found the golden values to use yet, but at least its going to a point where I want.

The next challenge would to be to trim out or cut out everything not needed in the minigzip and make a user library out of it, but that's way too complicated for me. Other solution is to create a static library somehow, but again, I'm totally blank of how to pull that off, or even how to get my friends slowdown code into that static lib, as it naturally would not rely on the original "minigzip.c" compiled to lib or exe...or....really, it's over my head.

Executing an external program is a bit messy and it causes a page fault counter to increase by 2 or so every time.

Graph: Your code worked great, however, putting delay(0) around the point and plot commands, still gave out 25% in cpu-usage during the entire process. Putting a delay(1) in there, and it took some minutes to complete. Is there a Delay(0.2) possibility...heh..or even a nanosecond delay available in purebasic?

The other issue is that DrawImage(ImageID(img), 0, 0, w, h) scales it with smooth pixels, and if I'm not mistaken this takes cpu-time along with the need for point command which in the loop is hard to slow down at least with standard purebasic commands and in which end even using other smart ways of doing the point command in API way still would to be run in a loop where I cannot slow down the loop beyond the minimum of 1 ms during each run. Is there time and space for requesting a nanosecond delay in the next version of Purebasic?