DATA Tool - 50% smaller output - First sample code

Developed or developing a new product in PureBasic? Tell the world about it.
User avatar
pf shadoko
Enthusiast
Enthusiast
Posts: 285
Joined: Thu Jul 09, 2015 9:07 am

Re: DATA Tool - 50% smaller output - First sample code

Post by pf shadoko »

I just wanted to point out that we could get 15 bits per character instead of 8
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: DATA Tool - 50% smaller output - First sample code

Post by Saki »

Ah, I think I understand what you mean now.
See, it goes like this:
I take the high byte to move around in the UCS-2 character list to a range where characters are declared, but they always have the same symbol assigned to them.
There are several such ranges, but they are usually not large.
The low byte then takes the desired byte value.
So the low byte fluctuates constantly around 256 bytes, the high byte stays.
If I take more bits, I move extremely far up and down the character list and quickly leave the desired range.
What I think is hardly mentioned is that the characters are also encrypted.
Defacto I must never leave a defined area, otherwise I quickly end up in Asian character areas, which can be very wide. It looks then also fast terrible.
I hope I haven't misunderstood you, but I don't see a feasible way at the moment.
The thing was also not easy to code, if the slightest thing is wrong, nothing works.
If you see a way, try to implement it, anything that improves the code is welcome.
地球上の平和
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: DATA Tool - 50% smaller output - First sample code

Post by Saki »

Annoncement - Compression Feature for the Data Codes added for testing and using.

I just added compression support for the data codes.
The codes are quite complex, so this is now the second attempt.
But it looks very good.
Please test it and let me know if anything stands out.
There are also now many different codes in the threads with different features.
Always in ascending order.
Here is quickly swapped what, you have to be very careful.

I think this is the easiest solution to create encrypted and compressed text in seconds and include it in an executable.
This example shows how to include a text as data directly into the code, without the need of a data section.


These few data lines contain this test text.
Created with Data_Tool_InCode_BF
viewtopic.php?f=12&t=77295
Image

Needed code for this output

Code: Select all

  
  Define Str_01$
Str_01$+""
Str_01$+""
Str_01$+""
Str_01$+""
Str_01$+""
Str_01$+""
Str_01$+""
  Debug PeekS(Data_InCode_Decoder_BF(Str_01$), -1, #PB_UTF8)
  
地球上の平和
deathmx
User
User
Posts: 27
Joined: Mon Feb 26, 2018 3:14 am

Re: DATA Tool - 50% smaller output - First sample code

Post by deathmx »

I also tried this a while ago, was quite funny... I used pretty much all of the unicode characters and while it certainly saved a bit space it really lagged the scrolling of the ide and other programs supporting unicode. :D
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: DATA Tool - 50% smaller output - First sample code

Post by Saki »

Hi, yes, many thanks, those lines are a lot of text.
Since there are no size limits, this can already slow down an editor when there are tens of thousands of lines.
But this is a normal behavior with very large texts.
Editing a text with hundreds of thousands of lines is usually no fun anymore.
It also depends on the editor, how fast it is and of course on the computer.
On Mac and Linux I have tested only rudimentary, my machines for this are quite slow, mostly I just check if everything runs, then I go back to Win10.
On my machine with fast processor, but 4K monitor I have no problems with large codes.
But as I said, huge texts always have their needs.

Normally you store these parts as Xinclude, then you can use them without them being annoying.
Static texts and images are no longer processed,
you don't have to have them in front of your nose all the time.

The important thing is whether it runs stable and has no more bugs.

However, suggestions for improvement are always welcome.
Yes, the free availability of almost arbitrary Unicode characters for encoding can look quite funny.
I have left this adjustable for the user, so that everyone can use what he wants.
Compatibility is always maintained.
This can also drive some people crazy trying to manipulate a text or image.
The methods used here are so new and unusual that hardly anyone can make sense of what is happening here. :x

Hint : The BaseU_BF Codes have now also LZMA compression, ready for testing and using.
viewtopic.php?f=12&t=77374
地球上の平和
deathmx
User
User
Posts: 27
Joined: Mon Feb 26, 2018 3:14 am

Re: DATA Tool - 50% smaller output - First sample code

Post by deathmx »

When I created mine it looked mostly chinese and worked great except for the extreme high amount of lag from the vast variety of characters... seems there are a lot a chinese characters in the unicode list. I was just running tests so that I would be able to post code with larger images without having to use as much data. When I realized that I might be lagging a forum for everyone posting such a thing I thought it might be considered kind of a trolling mechanism that might get me banned from forums lol, so I just left it alone. I am glad to see someone put some effort and created something reasonable that could prevent this problem.
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: DATA Tool - 50% smaller output - First sample code

Post by Saki »

Hi, thank you very much.
This morning I added a function to all codes that support compression, which returns the compression rate in percent.
This works with the encoders and decoders.

Interesting is that many GIF can still be compressed quite significantly.

Best Regards Saki
地球上の平和
Post Reply