Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Share your advanced PureBasic knowledge/code with the community.
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Post by Saki »

Yep, many thanks for the hint @Keya.

Here in this example I have to go back from UTF8 to UTF16.
地球上の平和
User avatar
Keya
Addict
Addict
Posts: 1864
Joined: Thu Jun 04, 2015 7:10 am

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Post by Keya »

yes but you're only using PeekS at the very end after all decoding/decrypting/etc is already complete (ie. the ready-to-use string buffer). It is slower, inefficient and wasteful on memory resources to create a second copy of the buffer (when you only need one) when you can simply use a string pointer to refer to the original buffer.

You should only create a copy of a buffer when you need to modify it while preserving the original buffer. Otherwise, just work with the original buffer.
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Post by Saki »

Hmm, the buffer contains only one-digit characters, so here correspondingly Ascii.
PB sees in it as string only "Chinese" since high and low byte would have to share two characters.

I create a demo text here with a simple text editor.
Under Windows, for example, this text is saved in Ansi format, so each letter is encoded with 8 bits.
I then encode this with the Data Encoder.
The encoder does not see any text but only binary data.

The embedding and encoding of these data by the encoder into Unicode characters makes them temporarily Unicode characters,
but these are only storage cells for the desired binary data.
The decoder makes pure binary data out of it again

I can read this binary data with Peeks as Ascii or UTF8.
So it becomes text again, but only if Peeks can create an interpretable text from it.

This copy effect also always results when strings are passed into a procedure.
But it is practically only noticeable when many thousands of calls of very large strings are made in a loop, because it is really extremely fast.
地球上の平和
User avatar
Keya
Addict
Addict
Posts: 1864
Joined: Thu Jun 04, 2015 7:10 am

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Post by Keya »

Saki wrote: Tue Jun 15, 2021 6:12 pm because it is really extremely fast.
Creating a second (and completely unnecessary) copy of the buffer will never be as fast as a simple string pointer to the original buffer.
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Post by Saki »

That is of course correct #Keya,
but practically it always depends on the situation.
The StringFields_BF can therefore optionally work with a pointer to a string.
But it is really only relevant if it is about thousands of very big strings which should be parsed in a loop.
One could also say that it makes no difference whether one drives with the car with 100kmh or 200kmh head-on against a wall.
Nevertheless, you are of course right.
If you can optimize the codes that would be nice, because I myself do not see everything optimally and quickly make mistakes.

I often don't have the nerve to deal with these things over and over again.
It was also a mistake to try to develop several codes practically in parallel, this quickly gives confusions.

In principle, the data codes have a high potential,
but the way of working seems to be very opaque, because you can't use already existing data to think into it.

It's just the karma of the new.
The wheel has already been invented and establishing something completely new is not easy.

The tool for the data section I only made because I wanted something absolutely simple that can still encrypt
and I don't have to study codes first to apply it.
地球上の平和
User avatar
Keya
Addict
Addict
Posts: 1864
Joined: Thu Jun 04, 2015 7:10 am

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Post by Keya »

Please don't be discouraged - we all push each other here to help all of us become better programmers. As a result, when we keep an open mind, we'll never stop learning.
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Post by Saki »

Hi, yes thank you very much #Keya.

Best Regards Saki
地球上の平和
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Post by Saki »

All Data Codes and BaseU codes with compression updated and enhanced

Fixes and enhancements.
The coders now also show the compression rate in percent via a function call.
地球上の平和
User avatar
Lunasole
Addict
Addict
Posts: 984
Joined: Mon Oct 26, 2015 2:55 am
Location: UA
Contact:

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Post by Lunasole »

Well it makes enough difference. I've used 64-kb jpeg file to test.
So my tool (it just packs data as it without any compression/encryption, using quads) produced 200kb file, and your produced 144kb.
But anyway for most cases I like more the first variant as it produces shorter datasection (24 lines vs 660 lines of your tool), and requires no extra code.
Maybe your tool will be nice anyway if lot of files need to be packed this way.
"W̷i̷s̷h̷i̷n̷g o̷n a s̷t̷a̷r"
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Post by Saki »

Hi Lunasole, thank you very much.
Feel free to introduce your tool here.
I don't know it, but that doesn't matter.
With the 24 lines, that was surely a typo.
With my tool you get uncompressed with the default settings 100 bytes in one line.
With the Quad variant comparably about 40 characters.
Compressing is always such a thing.
With Jpeg there is often not much more to get out.

Yes, without decoder nothing works with my tool.
But the complete module is only about 60 lines long, the bare needed decoder about 30 lines, which is very little.
But it does the decompression and the decryption of the data automatically.

Tip.
The easiest way to check the images is to use the BaseU codes,
because the encoder and decoder are automatically called one after the other.
The process is almost completely identical.

Best Regards Saki
地球上の平和
User avatar
Lunasole
Addict
Addict
Posts: 984
Joined: Mon Oct 26, 2015 2:55 am
Location: UA
Contact:

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Post by Lunasole »

Saki wrote: Mon Jun 21, 2021 8:44 am Hi Lunasole, thank you very much.
Feel free to introduce your tool here.
I don't know it, but that doesn't matter.
With the 24 lines, that was surely a typo.
Not a typo :) Just It produces a very long lines of quads, ~8-9kb per line. That may even slow down IDE sometimes, but I've made it that way to use less lines anyway.
There is nothing much special in my tool (I just made it quickly when it was needed) unlike yours, but if you want here is a link: https://lunasole.github.io/data/_4pb/datapacker/l
"W̷i̷s̷h̷i̷n̷g o̷n a s̷t̷a̷r"
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Post by Saki »

Hi, yes, I hadn't thought of that now.

you can also set with my codes, without limit.
But I think more than 150 characters wide finds not acceptance.

Best Regards Saki.

Hint.
All data codes have been updated.
The codes with AES still had a key bug.

The speed is now even higher.

The data codes are compatible with the BaseU_BF function.
This opens up completely new possibilities.
地球上の平和
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Post by Saki »

The codes have been revised again.
Artifacts and errors removed.
The speed could be increased by this once again.
地球上の平和
Post Reply