Page 2 of 2

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Posted: Tue Jun 15, 2021 2:11 pm
by Saki
Yep, many thanks for the hint @Keya.

Here in this example I have to go back from UTF8 to UTF16.

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Posted: Tue Jun 15, 2021 4:47 pm
by Keya
yes but you're only using PeekS at the very end after all decoding/decrypting/etc is already complete (ie. the ready-to-use string buffer). It is slower, inefficient and wasteful on memory resources to create a second copy of the buffer (when you only need one) when you can simply use a string pointer to refer to the original buffer.

You should only create a copy of a buffer when you need to modify it while preserving the original buffer. Otherwise, just work with the original buffer.

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Posted: Tue Jun 15, 2021 6:12 pm
by Saki
Hmm, the buffer contains only one-digit characters, so here correspondingly Ascii.
PB sees in it as string only "Chinese" since high and low byte would have to share two characters.

I create a demo text here with a simple text editor.
Under Windows, for example, this text is saved in Ansi format, so each letter is encoded with 8 bits.
I then encode this with the Data Encoder.
The encoder does not see any text but only binary data.

The embedding and encoding of these data by the encoder into Unicode characters makes them temporarily Unicode characters,
but these are only storage cells for the desired binary data.
The decoder makes pure binary data out of it again

I can read this binary data with Peeks as Ascii or UTF8.
So it becomes text again, but only if Peeks can create an interpretable text from it.

This copy effect also always results when strings are passed into a procedure.
But it is practically only noticeable when many thousands of calls of very large strings are made in a loop, because it is really extremely fast.

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Posted: Tue Jun 15, 2021 7:58 pm
by Keya
Saki wrote: Tue Jun 15, 2021 6:12 pm because it is really extremely fast.
Creating a second (and completely unnecessary) copy of the buffer will never be as fast as a simple string pointer to the original buffer.

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Posted: Tue Jun 15, 2021 8:21 pm
by Saki
That is of course correct #Keya,
but practically it always depends on the situation.
The StringFields_BF can therefore optionally work with a pointer to a string.
But it is really only relevant if it is about thousands of very big strings which should be parsed in a loop.
One could also say that it makes no difference whether one drives with the car with 100kmh or 200kmh head-on against a wall.
Nevertheless, you are of course right.
If you can optimize the codes that would be nice, because I myself do not see everything optimally and quickly make mistakes.

I often don't have the nerve to deal with these things over and over again.
It was also a mistake to try to develop several codes practically in parallel, this quickly gives confusions.

In principle, the data codes have a high potential,
but the way of working seems to be very opaque, because you can't use already existing data to think into it.

It's just the karma of the new.
The wheel has already been invented and establishing something completely new is not easy.

The tool for the data section I only made because I wanted something absolutely simple that can still encrypt
and I don't have to study codes first to apply it.

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Posted: Tue Jun 15, 2021 8:34 pm
by Keya
Please don't be discouraged - we all push each other here to help all of us become better programmers. As a result, when we keep an open mind, we'll never stop learning.

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Posted: Tue Jun 15, 2021 11:37 pm
by Saki
Hi, yes thank you very much #Keya.

Best Regards Saki

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Posted: Sun Jun 20, 2021 10:50 pm
by Saki
All Data Codes and BaseU codes with compression updated and enhanced

Fixes and enhancements.
The coders now also show the compression rate in percent via a function call.

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Posted: Mon Jun 21, 2021 7:26 am
by Lunasole
Well it makes enough difference. I've used 64-kb jpeg file to test.
So my tool (it just packs data as it without any compression/encryption, using quads) produced 200kb file, and your produced 144kb.
But anyway for most cases I like more the first variant as it produces shorter datasection (24 lines vs 660 lines of your tool), and requires no extra code.
Maybe your tool will be nice anyway if lot of files need to be packed this way.

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Posted: Mon Jun 21, 2021 8:44 am
by Saki
Hi Lunasole, thank you very much.
Feel free to introduce your tool here.
I don't know it, but that doesn't matter.
With the 24 lines, that was surely a typo.
With my tool you get uncompressed with the default settings 100 bytes in one line.
With the Quad variant comparably about 40 characters.
Compressing is always such a thing.
With Jpeg there is often not much more to get out.

Yes, without decoder nothing works with my tool.
But the complete module is only about 60 lines long, the bare needed decoder about 30 lines, which is very little.
But it does the decompression and the decryption of the data automatically.

Tip.
The easiest way to check the images is to use the BaseU codes,
because the encoder and decoder are automatically called one after the other.
The process is almost completely identical.

Best Regards Saki

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Posted: Wed Jun 23, 2021 4:47 pm
by Lunasole
Saki wrote: Mon Jun 21, 2021 8:44 am Hi Lunasole, thank you very much.
Feel free to introduce your tool here.
I don't know it, but that doesn't matter.
With the 24 lines, that was surely a typo.
Not a typo :) Just It produces a very long lines of quads, ~8-9kb per line. That may even slow down IDE sometimes, but I've made it that way to use less lines anyway.
There is nothing much special in my tool (I just made it quickly when it was needed) unlike yours, but if you want here is a link: https://lunasole.github.io/data/_4pb/datapacker/l

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Posted: Wed Jun 23, 2021 10:40 pm
by Saki
Hi, yes, I hadn't thought of that now.

you can also set with my codes, without limit.
But I think more than 150 characters wide finds not acceptance.

Best Regards Saki.

Hint.
All data codes have been updated.
The codes with AES still had a key bug.

The speed is now even higher.

The data codes are compatible with the BaseU_BF function.
This opens up completely new possibilities.

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Posted: Fri Jun 25, 2021 8:18 pm
by Saki
The codes have been revised again.
Artifacts and errors removed.
The speed could be increased by this once again.