TSI include, new image file format

Applications, Games, Tools, User libs and useful stuff coded in PureBasic
Thorium
Addict
Addict
Posts: 1271
Joined: Sat Aug 15, 2009 6:59 pm

TSI include, new image file format

Post by Thorium »

I have created a new image file format optimized for decoding speed.
It is intended to be used on 2D games. So it have the following characteristics:
  • fast decoding
  • slow encoding
  • lossless
  • alpha channel support
  • small size, comparable to PNG, sometimes bigger, sometimes smaler, depands on the image
It is a bit comparable to PNG, it uses ZLib for compression and it uses 3 of the 4 filters that PNG uses to filter the image lossless for bedder compression.
I skiped the Paeth filter because it's to slow for a high performance format. I had it in and tested it and it was 10 times slower than the other filters.

I plan to implement my own compression that will be much slower on compression but better on compression ratio. And the include is mostly unoptimized. The filters for the decoder are a little bit optimzed but the remain is unoptimized so with the next versions the encoder and decoder will be much faster. The decoder is allready faster than the PB internal PNG decoder. Decodes up to 2 times faster, depanding on the plattform, subsystem and image.

However it does support 2 lossless transformations of the image data for better compression. Very simple transformations, just sorting the channels. So on some images it beats PNG in size on some it's bigger than PNG because it only can store 32bit pixel. Currently there is no support for indexed colors.

It should work on every plattform supported by PB and on every subsystem supporting alpha channel. But it is not tested very much for now and it is still in beta phase.

Here is the include: http://www.GameTreasure.de/downloads/TsiInclude.zip needs PB 4.40
Comes with a XnView plugin for viewing and converting of TSI files.
With a stand alone converter (only Windows). And with a stand alone viewer (only Windows).

Just let me explain the most important procedures:

Tsi_LoadTsi2Image(FileName.s)
Similar to PB's LoadImage. Loads a TSI from a file and decodes it into a PB image.

Tsi_CatchTsi2Image(*Tsi)
Similar to PB's CatchImage. Decodes a TSI in memory to a PB image.

Tsi_LoadTsi2Sprite(FileName.s)
Similar to PB's LoadSprite. Loads a TSI from a file and decodes it into a PB sprite.

Tsi_CatchTsi2Sprite(*Tsi)
Similar to PB's CatchSprite. Decodes a TSI in memory to a PB sprite.

Tsi_BestSaveFromImage(Image.i, FileName.s)
Similar to PB's SaveImage. Encodes a PB image to a TSI and saves it to the hard disk.

Tsi_BestSaveFromSprite(Sprite.i, FileName.s)
Similar to PB's SaveSprite. Encodes a PB sprite to a TSI and saves it to the hard disk.

Tsi_BestSave2MemoryFromImage(Image.i, *TsiSize.Integer)
Similar to PB's SaveImage, but saves the TSI to memory and not to the hard disk.

Tsi_BestSave2MemoryFromSprite(Sprite.i, *TsiSize.Integer)
Similar to PB's SaveSprite, but saves the TSI to memory and not to the hard disk.

So it's easy to use. There are more procedures, that offer you more controle but you can explore the include by your self. I have commented all procedures, what they do and what they return.

Hope someone can need it.
After i have optimized the include and added my own compression i will make a animation format (TSA).

File size and decoding speed comparison TSI against PNG:

Code: Select all

File Size Comparison

3,145,728b = 3,072kb 100.00% size : raw image data
2,200,459b = 2,149kb  69.95% size : PNG saved with PureBasic
1,470,636b = 1,436kb  46.75% size : PNG saved with Gimp
1,313,207b = 1,282kb  41.73% size : PNG optimized with PNG Optimizer
1.266.965b = 1,237kb  40.27% size : TSI saved with TSI include

Decoding Speed Comparison

image on x86 Windows
68ms 100.00% time needed : PNG saved with Gimp
56ms  82.35% time needed : PNG optimized with PNG Optimizer
39ms  57.35% time needed : PNG saved with PureBasic
29ms  42.65% time needed : TSI saved with TSI include

image on x64 Windows
62ms 100.00% time needed : PNG saved with Gimp
52ms  83.87% time needed : PNG optimized with PNG Optimizer
38ms  61.29% time needed : PNG saved with PureBasic
28ms  45.16% time needed : TSI saved with TSI include

sprite with DirectX7 on x86 Windows
79ms 100.00% time needed : PNG saved with Gimp
66ms  83.54% time needed : PNG optimized with PNG Optimizer
48ms  60.76% time needed : PNG saved with PureBasic
38ms  48.10% time needed : TSI saved with TSI include

sprite with DirectX7 on x64 Windows
71ms 100.00% time needed : PNG saved with Gimp
62ms  87.32% time needed : PNG optimized with PNG Optimizer
47ms  66.20% time needed : PNG saved with PureBasic
37ms  52.11% time needed : TSI saved with TSI include

sprite with DirectX9 on x86 Windows
74ms 100.00% time needed : PNG saved with Gimp
64ms  86.49% time needed : PNG optimized with PNG Optimizer
45ms  60.81% time needed : PNG saved with PureBasic
34ms  45.95% time needed : TSI saved with TSI include

sprite with DirectX9 on x64 Windows
66ms 100.00% time needed : PNG saved with Gimp
58ms  87.88% time needed : PNG optimized with PNG Optimizer
42ms  63.64% time needed : PNG saved with PureBasic
33ms  50.00% time needed : TSI saved with TSI include
Last edited by Thorium on Sat Oct 24, 2009 9:01 pm, edited 2 times in total.
Mistrel
Addict
Addict
Posts: 3415
Joined: Sat Jun 30, 2007 8:04 pm

Re: TSI include, new image file format

Post by Mistrel »

Do you have a performance comparison of decompression time compared to PNG?
Thorium
Addict
Addict
Posts: 1271
Joined: Sat Aug 15, 2009 6:59 pm

Re: TSI include, new image file format

Post by Thorium »

Mistrel wrote:Do you have a performance comparison of decompression time compared to PNG?
I have done a quick benchmark. Only with x86 Windows and PB images. I chosed a photo from Google Earth as test image and added a alpha channel to it.

You can download all the different versions of the image i have tested in this package: http://www.GameTreasure.de/downloads/TestImages.zip
So you can see for your self that this is no joke. ;)

Here are the results:
The smaller the numbers, the better.

Code: Select all

File Size Comparison

3,145,728b = 3,072kb 100.00% size : raw image data
2,200,459b = 2,149kb  69.95% size : PNG saved with PureBasic
1,470,636b = 1,436kb  46.75% size : PNG saved with Gimp
1,313,207b = 1,282kb  41.73% size : PNG optimized with PNG Optimizer
1.266.965b = 1,237kb  40.27% size : TSI saved with TSI include

Decoding Speed Comparison

image on x86 Windows

68ms 100.00% time needed : PNG saved with Gimp
56ms  82.35% time needed : PNG optimized with PNG Optimizer
39ms  57.35% time needed : PNG saved with PureBasic
33ms  48.53% time needed : TSI saved with TSI include
I have measured the time needed for decoding by using the QueryPerformanceCounter and decoded the files a 100 times in a loop and divided the result by 100. Of course decoding was been done with a PNG and TSI in memory. So the complete loading time is much bedder than any of the tested PNG's. The smaller the image the faster it can be loaded from the hard disk plus the fast decoding makes it realy realy fast to load.

As you can see TSI is the smallest and the fastest. It beats PNG Optimizer in size and the PB saved PNG in speed. Even with the fact that the PB saved PNG don't used filters and don't need to waste time on decoding a filter.

I am a little bit proud. ^^
But this is a "best case" test image. Images with less than 257 colors are very much smaller if saved with PNG Optimizer. But keep in mind that PNG Optimizer will remove the alpha channel and replace it with a transparent color. So if you load the image with PureBasic it will have no alpha channel!

But this is just the start. There is much more optimizing potential.
Last edited by Thorium on Sat Oct 24, 2009 9:04 pm, edited 1 time in total.
Thorium
Addict
Addict
Posts: 1271
Joined: Sat Aug 15, 2009 6:59 pm

Re: TSI include, new image file format

Post by Thorium »

I forgott to write that the include is currently (PB 4.40B5) not working with the OpenGL subsystem and sprites. It's because of this PB bug: http://www.purebasic.fr/english/viewtop ... =4&t=38626

Loading or saving a TSI with the OpenGL subsystem will fail and set Tsi_LastError to #Tsi_Error_PbSpriteWrongPixelFormat, which is the correct behavior. It's not a bug of the TSI include.
Thorium
Addict
Addict
Posts: 1271
Joined: Sat Aug 15, 2009 6:59 pm

Re: TSI include, new image file format

Post by Thorium »

Just uploaded TSI Include v0.02: http://www.gametreasure.de/downloads/TsiInclude.zip
Transformationdecoding for x86 and x64 are rewritten in assembler. Resulting in a 12% speed up on TSI decoding.

New comparison:

Code: Select all

File Size Comparison

3,145,728b = 3,072kb 100.00% size : raw image data
2,200,459b = 2,149kb  69.95% size : PNG saved with PureBasic
1,470,636b = 1,436kb  46.75% size : PNG saved with Gimp
1,313,207b = 1,282kb  41.73% size : PNG optimized with PNG Optimizer
1.266.965b = 1,237kb  40.27% size : TSI saved with TSI include

Decoding Speed Comparison

image on x86 Windows
68ms 100.00% time needed : PNG saved with Gimp
56ms  82.35% time needed : PNG optimized with PNG Optimizer
39ms  57.35% time needed : PNG saved with PureBasic
29ms  42.65% time needed : TSI saved with TSI include

image on x64 Windows
62ms 100.00% time needed : PNG saved with Gimp
52ms  83.87% time needed : PNG optimized with PNG Optimizer
38ms  61.29% time needed : PNG saved with PureBasic
28ms  45.16% time needed : TSI saved with TSI include

sprite with DirectX7 on x86 Windows
79ms 100.00% time needed : PNG saved with Gimp
66ms  83.54% time needed : PNG optimized with PNG Optimizer
48ms  60.76% time needed : PNG saved with PureBasic
38ms  48.10% time needed : TSI saved with TSI include

sprite with DirectX7 on x64 Windows
71ms 100.00% time needed : PNG saved with Gimp
62ms  87.32% time needed : PNG optimized with PNG Optimizer
47ms  66.20% time needed : PNG saved with PureBasic
37ms  52.11% time needed : TSI saved with TSI include

sprite with DirectX9 on x86 Windows
74ms 100.00% time needed : PNG saved with Gimp
64ms  86.49% time needed : PNG optimized with PNG Optimizer
45ms  60.81% time needed : PNG saved with PureBasic
34ms  45.95% time needed : TSI saved with TSI include

sprite with DirectX9 on x64 Windows
66ms 100.00% time needed : PNG saved with Gimp
58ms  87.88% time needed : PNG optimized with PNG Optimizer
42ms  63.64% time needed : PNG saved with PureBasic
33ms  50.00% time needed : TSI saved with TSI include
Mistrel
Addict
Addict
Posts: 3415
Joined: Sat Jun 30, 2007 8:04 pm

Re: TSI include, new image file format

Post by Mistrel »

Thanks for the benchmarks. This is all great information. :)
X
Enthusiast
Enthusiast
Posts: 311
Joined: Tue Apr 04, 2006 6:27 am

Re: TSI include, new image file format

Post by X »

This is VERY awesome! Im going to have to use this on my games, replacing PNG, and provide credit, of course :)
User avatar
idle
Always Here
Always Here
Posts: 5042
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

Re: TSI include, new image file format

Post by idle »

That's looking really good! :mrgreen:
Fred
Administrator
Administrator
Posts: 16619
Joined: Fri May 17, 2002 4:39 pm
Location: France
Contact:

Re: TSI include, new image file format

Post by Fred »

It's a very interesting initiative :wink:
User avatar
DoubleDutch
Addict
Addict
Posts: 3219
Joined: Thu Aug 07, 2003 7:01 pm
Location: United Kingdom
Contact:

Re: TSI include, new image file format

Post by DoubleDutch »

Will be amazing for games. Would be great if it (when finished) become one of the official native formats available in PB. :)
https://deluxepixel.com <- My Business website
https://reportcomplete.com <- School end of term reports system
User avatar
thyphoon
Enthusiast
Enthusiast
Posts: 327
Joined: Sat Dec 25, 2004 2:37 pm

Re: TSI include, new image file format

Post by thyphoon »

DoubleDutch wrote:Will be amazing for games. Would be great if it (when finished) become one of the official native formats available in PB. :)
+1

Thanks Thorium to share it ! It's very great !
JCV
Enthusiast
Enthusiast
Posts: 579
Joined: Fri Jun 30, 2006 4:30 pm
Location: Middle East

Re: TSI include, new image file format

Post by JCV »

nice! Thanks for sharing!

[Registered PB User since 2006]
[PureBasic 5.7][SpiderBasic 2.2] [Win 10 64bit]
[Intel i7 990x 4.20 Ghz] [18GB DDR3]
User avatar
luis
Addict
Addict
Posts: 3876
Joined: Wed Aug 31, 2005 11:09 pm
Location: Italy

Re: TSI include, new image file format

Post by luis »

Nice project, thank you for sharing it !
"Have you tried turning it off and on again ?"
A little PureBasic review
User avatar
djes
Addict
Addict
Posts: 1806
Joined: Sat Feb 19, 2005 2:46 pm
Location: Pas-de-Calais, France

Re: TSI include, new image file format

Post by djes »

Nice, thank you! :D
Thorium
Addict
Addict
Posts: 1271
Joined: Sat Aug 15, 2009 6:59 pm

Re: TSI include, new image file format

Post by Thorium »

Just a small update to let you people know that i am still working on this include. ^^

Currently i am learning SIMD programming: MMX, SSE and so on.
I tried to reprogram the filter decoding with MMX and SSE2. Well it's a big success. I reprogramed first the up filter decoding, because it's the easiest of the 3 to parallelize. And yes it works like a charm. The SSE2 implementation is 9 times faster than my original assembler implementation. Thats a big performance boost. But unfortunatly the decoding of the filters was allready very fast with my original assembler implementation. So the filter decoding is speeded up by 934% but the overall decoding of a TSI is only speeded up by 6.9%. I was hoping for more. But anyway i am very pleased with SIMD. ^^

Now i have to reprogram the other 2 filters with MMX and SSE2. That will be much more complicated. Especially with the Average filter. I even think it's not possible to use SIMD with the average filter but i will double check that.

Here is a little performance comparision with different implementations of the Up filter:
measured in MB/s

Code: Select all

PureBasic   197 MB/s  100%
Assembler  1178 MB/s  598%
MMX        7133 MB/s 3621%
SSE2      10997 MB/s 5582%
Post Reply