PureBasic Forums - English

http://www.purebasic.com
https://www.purebasic.fr/english/

DATA Tool - 50% smaller output - First sample code

https://www.purebasic.fr/english/viewtopic.php?t=77246

Page 1 of 2

DATA Tool - 50% smaller output - First sample code

Posted: Sat May 15, 2021 3:36 pm
by Saki
DATA Tool - 50% smaller output - Coming soon

I have a new tool ready which can output data lines for images and icons reduced by 50%, for example.
in size.

It is still being tested a bit, it was quite fiddly to get it running, but it works fine already.
I think there are no more problems with the code.
If everything is OK I can share it.

You guys can say what you think about this.

It is Unicode based and can store the required data in one character instead of two.

I plan to make the encoder part so that it doesn't save the generated data lines,
but puts it directly into the clipboard.
That's how it's done now.
You can then simply copy and paste the data into the codes.

I plan to bundle both in a small module.

Image

Re: DATA Tool - 50% smaller output - Coming soon

Posted: Sat May 15, 2021 3:54 pm
by Tenaja
This will be interesting to see!

Re: DATA Tool - 50% smaller output - First sample code

Posted: Sat May 15, 2021 7:31 pm
by Saki
Hi Tenaja
Here is a first decoder for testing.
I don't know yet if all problems are solved.
Most of the time it's the details that throw a spanner in the works.
Or a thinking error, which happens quickly.

Code: Select all

EnableExplicit
; Sakis Unicode Data Generator - Decoder

UsePNGImageDecoder()
UseJPEGImageDecoder()
UseGIFImageDecoder()

Define i, ii, breaks, *buffer, len_buffer, len_line, data_size, string$

len_line=100 ; Length data lines

data_size=?image_1_b-?image_1_a
len_buffer=data_size/2
breaks=data_size/2/len_line
string$=Space(len_buffer+breaks)
*buffer=AllocateMemory((len_buffer-breaks))

ii=0
For i=0 To len_buffer-1-breaks
  If Not i%len_line And i : ii+2 : EndIf
  PokeA(*buffer+i, PeekA(@string$+ii)) : ii+2
Next

*buffer=AllocateMemory(data_size)
CopyMemory(?image_1_a, *buffer, data_size)
i=0 : breaks=0

Repeat
  If Not PeekW(*buffer+i) : PokeW(*buffer+i, 2560) : breaks+1 : EndIf : i+2
Until i>data_size

Define *catch_buffer_1=AllocateMemory(data_size/2-breaks)

ii=0
For i=0 To len_buffer-breaks-1
  If Not i%len_line And i : ii+2 : EndIf
  PokeA(*catch_buffer_1+i, PeekA(*buffer+ii))
  ii+2
Next

FreeMemory(*buffer)

Define image_ID=CatchImage(#PB_Any, *catch_buffer_1)

Define window_ID=OpenWindow(#PB_Any, 0, 0, 600, 400, "", #PB_Window_SystemMenu|#PB_Window_ScreenCentered)
Define image_gadget_ID=ImageGadget(#PB_Any,
                                   WindowWidth(window_ID)/2-ImageWidth(image_ID)/2,
                                   WindowHeight(window_ID)/2-ImageHeight(image_ID)/2,
                                   0,
                                   0,
                                   ImageID(image_ID))
Define win_event
Repeat
  win_event=WaitWindowEvent()
  If win_event=#PB_Event_Gadget
    Select EventGadget()
    EndSelect
  EndIf 
  If win_event=#PB_Event_CloseWindow
    End
  EndIf
ForEver 

DataSection
  image_1_a:
  
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s ""
  Data.s "#"
  
  image_1_b:
EndDataSection

Re: DATA Tool - 50% smaller output - First sample code

Posted: Sun May 16, 2021 4:06 am
by JHPJHP
Unless I'm missing something, your DataSection is over four times that of the standard method?

If you wanted a String DataSection, why not use the universally accepted Base64 which is still smaller?

DataSection created using Bin2Data by ts-soft

Code: Select all

UsePNGImageDecoder()

nSize = ?PB_End - ?PB_Start
*Memory = AllocateMemory(nSize)
CopyMemory(?PB_Start, *Memory, nSize)
PB_Image = CatchImage(#PB_Any, *Memory, nSize)
FreeMemory(*Memory)

Debug ?PB_End - ?PB_Start

If IsImage(PB_Image)
  If OpenWindow(0, 0, 0, 381, 68, "ImageGadget", #PB_Window_SystemMenu | #PB_Window_ScreenCentered)
    ImageGadget(0, 0, 0, 381, 68, ImageID(PB_Image))
    Repeat : Until WaitWindowEvent() = #PB_Event_CloseWindow
  EndIf
  FreeImage(PB_Image)
EndIf

DataSection
  PB_Start:
    Data.q $0A1A0A0D474E5089,$524448490D000000,$440000007D010000,$15A55A0000000308,$544C5000030000B2
    Data.q $1400001400001445,$0000140000140000,$0B1D08071A020116,$22100F210F0D1E0C,$1B16241512221211
    Data.q $1C27241C27211A26,$27241C27241C2724,$24212F241E2B241C,$29362B2834292532,$24362B24352B282D
    Data.q $362B25362B24362B,$2C2B342C2F372B25,$2D3A2F29302D3A35,$42373340352F3C31,$393A463938453935
    Data.q $3A47393A46393A46,$493C3C453A374739,$4351464049463D43,$5343435343435344,$4544514443534343
    Data.q $49435346444B454A,$46604C465F4C465A,$614D47614D46614D,$52505E504F5F4E49,$5165565160555460
    Data.q $715A506D584E6857,$6053785F53745C51,$557C62557A615579,$7E64567D63567D62,$656E67656D7F6557
    Data.q $7180665865656F66,$6E686A6A676B6666,$6873836859826859,$5A84695A84695A68,$84695A84695A8469
    Data.q $695A83695A84695A,$5A84695A84695A83,$84695A84695A8469,$695A84695A84695A,$6984695A84695A84
    Data.q $846A5A846A5A726A,$6B5C856A5B846A5B,$69786C696B6B7685,$896E60876D5F756C,$716B8A70637B6F6A
    Data.q $6471717D8B71637F,$8C73668C72658C72,$748085756D8C7467,$8074748074748074,$7474807474807474
    Data.q $767F8D7669747580,$7E89776D8C776B75,$7A7B828C786D7677,$867B84868A808386,$7D9B867A9A867A99
    Data.q $88898D9288859887,$8A7E8F89888B8A8B,$86A49085A38F839E,$A59186A59186A591,$948BA39288A49186
    Data.q $92A69890A6968DA5,$9D99999F9895A398,$9B9C9E9A9B9B999C,$9DA89F9CA39D9B9F,$B2A7A1B1A59FADA2
    Data.q $ADAEAFABAAB3AAA6,$B9B3B4B6B0B0B1AF,$BABABEB8B8BCB6B6,$BABFBABABFBABABF,$C1BBBCBFBBBBBFBA
    Data.q $C5C9C9C1C5C5BEC0,$D0D1CACDCFC8CBCC,$D9C7D6D8CAD3D4CA,$C6DCDEC6DBDDC8D8,$DFE1CDDEE0CBDEE0
    Data.q $E5D4E1E3D6E0E2D4,$CCE5E8CEE4E6CFE3,$EAEDCEE9EBCDE7EA,$EFD2EDF0D1ECEED0,$EEEEEFEDEEEEEBEE
    Data.q $F1F2F0EFF0D5EFF2,$F4D8F2F5D7F2F4E5,$DAF4F6F2F2F3DFF2,$FCFCFAFAFBF8F8F9,$FFFEFEFEFEFEFEFC
    Data.q $E3FDFFE3FDFEE5FE,$FFFFFFFFFFFFFFFF,$FFFFFFFFFFFFFFFF,$FFFFFFFFFFFFFFFF,$FFFFFFFFFFFFFFFF
    Data.q $FFFFFFFFFFFFFFFF,$F8FFFFFBFFFFFDFF,$FFFFF3FFFFF6FFFF,$FFEEFFFFF0FFFFF2,$E8FFFFEAFFFFECFF
    Data.q $FFFFFFFFFFFFFFFF,$0B0000106B7734FF,$EDDA78544144493D,$47C71A68D36FCF9C,$A936960E458D21EF
    Data.q $396569534E577072,$21564A7A87252C9C,$DEAF7054ABA844CB,$F9385CB61555522B,$9008D9A2AB320732
    Data.q $931F974D14582122,$E9DDA8D2D10522E9,$666043BBB02CD1AA,$AA88B80FF0A20287,$BF5FB6BEFBEC3D15
    Data.q $FAD21924CED21C4E,$7E3E8E3F6C4E5411,$13E4DF79E7DFBCFC,$43E08FB3DEED4078,$875D7557D930727C
    Data.q $37DDFA3C444B9730,$1E100B28B67FD09D,$377BFD3EEFD1E63F,$F51E447E883B8DF4,$9489F3B13E9D77E8
    Data.q $F879B3DDFA3DC420,$FA3CA5B0145FD298,$9FAF8512FA9BB2DD,$0EB3D77E8F65F426,$1CB55DFB1782C7B2
    Data.q $7EF2F41FC0957F4B,$D47E98436DBEA537,$FA3CB7ABE6372FBC,$14299BDE5C26D73F,$D8C175B44DEBFC20
    Data.q $5F243082CA4FD272,$E58F04548D177B46,$3D4E26E0EB61E4A6,$0805DCF0CBB4BBFA,$94219D6C9FF44A11
    Data.q $417497AFF5B0199F,$BAF24BC2C02EC616,$C783908F4FE0C78A,$331F445FFD1E69C9,$B1A5E430FF1BE834
    Data.q $F1FA615FF818BFC1,$08203AB2F182854F,$399C8E83337FDFD3,$865D9BEC7B6FD1E6,$19103C82FD12D901
    Data.q $BABE247D3CBFE8B2,$4F1C83FC16E97CF9,$49EA27F690F7F47B,$C176E15CFF791A4D,$500B8FC80CE7FBC8
    Data.q $131D343FA2DAFA18,$8BABB78D8BC6AB84,$4213D74BDB5B9B9B,$7E8F19DA7475C8E4,$5A4E74AC55FED59B
    Data.q $C5F3450A2E2FE3F4,$6D6D6E6DDF6FB7DB,$3DA7241CA38B9FBF,$09FE9893FA7ED3CD,$97F7F67F55647813
    Data.q $F3D677B7F01BE856,$7E8F71B1F898E883,$7F70CDF4C59FE9BB,$BFA23E14BD8C9A7A,$51690167677EFDBD
    Data.q $FD376FD1E6312061,$B3AA2FFE113E981B,$F0C2DFC26FF153DF,$8F51A01D21117F5F,$6C9C4EF6E3E9BB7E
    Data.q $4E22F5FC2B9CC1C1,$699FFD8BD864423C,$16C6B996CAA97157,$BABFE84990527D06,$FFE03BD935BFCC26
    Data.q $EB1B3F822E00C4F3,$2A184FE97FBBC168,$8FF59116C9F521DF,$5A5EC42E3EF2199F,$C29C9FF427C26F81
    Data.q $DFD1E69E663A2769,$DB15F78299F431BB,$67C0CD170822B9E7,$472FB4A3CF59BFE1,$C5D7E81B7E8F0552
    Data.q $247C08DA61116DBE,$5F5FC16EB0FF9F35,$D38C66A42C981540,$B70BC9B47BFA3D26,$5F80E8B5B1AE659D
    Data.q $31B377FB063B8595,$49FAFF84DD617A61,$FED38DC723FDE71C,$3FA0EC58D36FD1E4,$C9E73A48DCF8BF3B
    Data.q $F5FD66FF861CCB3C,$E5D9D22C8F315ADA,$17FDE6916670070E,$36EFF711ACD2C6FA,$E4489FA9C6398DE7
    Data.q $FDFF27C7EFFDADAF,$6FD1E6409E2143CF,$C6ECA342DC6FF063,$D8339B1E7D585546,$0605F5E9CD256E6F
    Data.q $A15E81CDE407A3FE,$9B4CB1490B59E29F,$67D98F009D7B4A1D,$55788E1CC12EA75B,$6CE94477F984E788
    Data.q $819D7851F404C96D,$2C557D0AF48FD26F,$DBA7081CB2D256DB,$0393DD5F811350E9,$819E275A2D9D6B29
    Data.q $D85D911C787D30E7,$0A33C48DB7B7EC24,$B0F47EFFF82DAF73,$2F4E7A9C555F62C7,$202AAC3AB6A97D06
    Data.q $C9533ACD710717B6,$A39BF88E36F7AAD1,$3F7FBFD8345B1082,$316784DD8FE8A324,$1F0B0115FC2627FC
    Data.q $135EBE8F7B93ACD9,$EE5EC167A875ED03,$B7FE56CA7F4D88E7,$47D8E28C2ADFCCEE,$1910B89D9DFB11E8
    Data.q $B07FC0C4BC028412,$7A15ACD06102878C,$D43AF6A9015EBE90,$B0BD556FA12FD0C5,$84FC3F73E226AE30
    Data.q $CB859CC09CECEFD9,$11F39213E3F6B4B0,$77A148FD09F707FC,$6535C18BB4A2EBE9,$8721FD2D8A8CA2DD
    Data.q $D377775C1C719025,$3F6337DE76230F0E,$AF0955BF7853AAA7,$7F62473EBEBF8923,$09772BF8183FECE6
    Data.q $6D85AEBEC70F68FD,$F9223118B7B511D9,$8A701FB0B8080880,$D43A9AC5080104A2,$58391F61AA7E3239
    Data.q $021623F759202004,$FD238789E0786820,$22E7FC45FA6FFCFD,$BE857990FC79F842,$BFD88ABBFD4653C3
    Data.q $FA7CB9050A76B379,$57F0243FE08F621E,$F72759849F6AC8E6,$CBC0E66B240D7D10,$F3CADCDE7C03F0D3
    Data.q $7E98E9AB400118B6,$1652A762E68B0414,$9E821CED0E755204,$150AF34528084CA6,$96C800A9D411F38D
    Data.q $9A9C8F2259CA83CC,$495A8571BCC10247,$45B6A84A40793C01,$F8BFC420B4E1F332,$39FFCFE70BF94ACE
    Data.q $5F473B6C6BC3AA7F,$03629E65775399F2,$F8BBF64BC362D97C,$2178ECF20C3F4440,$09BA55043AC4B562
    Data.q $AC59ED8EC4AE0133,$294794E2B13E3284,$663415D151465F68,$48F6625F63D0D97F,$CD11F7773FBA3CE9
    Data.q $489C3C3A3E02793A,$FA3BB6B0FD22F70C,$D4EB165C5001B215,$F4DB2FE870E32F92,$C383A1094EA55C43
    Data.q $47E83BF1C86EAD5C,$461CC4D8F480AE1B,$31D64875A9BE344F,$5F4D3A1935CEDF6A,$D5A7DF419AFD2DDA
    Data.q $FA6DE438724E558B,$2FAC57D839725069,$865B3C77D2CA800E,$15BD9B3599C18B1E,$6FA7BED481FD16F8
    Data.q $619A15B8158152EE,$7D3A0B414BE8CDFC,$E957287A8650B404,$62BBD1C93FD0C108,$5BD0897E0EFECFED
    Data.q $57CAEDE347C0CDEC,$33F726A7639CC3A4,$B4ADD22B01A1497D,$BD2D1AA3F4DBEF23,$1D00E8E55E25FC2C
    Data.q $BAD648FE8814F86C,$B1383EA5514AC065,$6F95FC31ECB3D18C,$67A76399E05E32F3,$61ADF46F687E897A
    Data.q $655066F581FB2F42,$872B7820AEC0F21C,$24558F4F1A7A0128,$2BE7DA1E9FBF46E0,$CE3D04BA8D01F75B
    Data.q $4F97976F95FB3639,$353318D9F9BD200C,$25DE6A56C3F443D9,$DA5A5B017748BE7D,$1FA8A4173D1ACB0E
    Data.q $83A9F4968329BA85,$4E3CC4E1503E7A7E,$85E5FB2607FB16F7,$7BC6A50C01541153,$A3BB941D8A8BF631
    Data.q $56D52E49AFA46E5F,$AFA6DD7D0BA1E399,$BD220AF48BE95CF9,$AFF77DAEB92A0A11,$5E72FE18F659F72B
    Data.q $353A51C8E592181E,$4AFD35BE6EB59BD9,$5A7FD00F8176635E,$F433F7D0C33E020C,$E67D2C658124AA45
    Data.q $0B32BE6CB8F9F6E2,$FD98BBE5FB1637E1,$1F1F89E891751E32,$0B68F128C3CD45EF,$B61A2B6A205608FA
    Data.q $33082CD5FB12E149,$9AA290204DFA39F4,$426A5DD9AFC54747,$2B917AF1FD36CF3F,$C798353C5BBFBE46
    Data.q $0457F67729C50DF0,$B3F416F1091F4F3A,$80D4947D2DDCE4A4,$057D7669CDA19BCA,$9CD63F80D35FBD58
    Data.q $BF810AEBC261B47E,$9ABFF7730B3B8D3C,$A71EF1E9F8638B1A,$AD4C122FA2B5F59F,$4BE30956E2ED57E8
    Data.q $E1802DB5B22C40F1,$3B914EE08CF1CD60,$496C18C97E4FC6FD,$D1E3C38F9F8B27C8,$01597E9C77E1A748
    Data.q $F45680098DF456B1,$03C3300317AF116D,$56B13ED30004EEA5,$64FBC9FD2DDA29DF,$54F531470C1A35AC
    Data.q $43597E9277EF1787,$D0FA4B61578070A6,$5327156351AABF41,$2CDC35CFC640556D,$453AEA560BE526E4
    Data.q $FED7965D03CD5749,$1485C58FD08E766D,$069D73DBDEDC6F79,$5777A91C66A6425E,$6B98E566897ECBF4
    Data.q $F296D83A310741A7,$1E01233000749BAC,$11B67D1A7EF5DC73,$676BBA9DF1CC7FF2,$00BF5A384C72C3D3
    Data.q $3D82B2F63CFAF7B1,$FD09A959FF5BB7D2,$246E7572A834D5B0,$BE9DDE34DAF4C71F,$5FA85AA0AE9F341A
    Data.q $3BDE1B46C70B0B68,$A7D2CFBCB735F448,$303E687DE8C6B185,$4FA1343B5418A680,$977388685FDFE2CB
    Data.q $5AB77D077D1EB7C5,$602893DBC44B0217,$EBAD4C294FEF35C4,$7ADF1661FF6B685C,$A79F76FA4B40A5F4
    Data.q $7D8EC79AAEA75B4A,$B52FD111F68AFE15,$74ABE92DC99B4FA2,$3B9F48D8BF41CC1A,$0EEC87E9CD03F448
    Data.q $FB6D2714390C6D33,$BE95DF635D47E83B,$1CC82DCB5106C3DD,$52FA46DF0D1CFA1C,$F7F6CF6150A9CE92
    Data.q $F2FACFD36AB75A99,$1955AACE3DA7DE6C,$6759CE8DFCE9F430,$837AB3F4F7DC0055,$C5E253A09FA46F38
    Data.q $F41E7AEBB4D18D3C,$4F34635669689474,$98EC556894F26AD1,$1B2B73799773A5D7,$06C00BA263F4F7D8
    Data.q $DDA99475598699FA,$177C6B487C695357,$2ACCC327070DC3C0,$C164F759C6BEA9DA,$4BBB70AD9DA54BED
    Data.q $A535AA83DFEC053F,$AA00189CD551990F,$A9F437DAC15C7D2B,$090150E95FAD46ED,$2B75AAE5BFAA7146
    Data.q $54174C7EFC350E4A,$C7F6927F43E9777E,$9BBD9C0610BA8F9A,$265F45ACB7BA883C,$B1D11B8CB2D73945
    Data.q $AD15760170AF4AC4,$7C53556D58E4399A,$DB8542EA4EDF86B9,$E5BA99AD9009FA65,$E69996483C33F443
    Data.q $34972B5A69B1B468,$554E7ECC84074F0E,$FCA1404297371854,$4DCFF41C64081AAB,$D8315AFF0ACD3805
    Data.q $4E7B03EC57F616A7,$969A92E3E63B798F,$50FB25E6EE9E020D,$3373485380F136FB,$7C7CC4874EA39B88
    Data.q $E04997112B3158C8,$3D37ED54B2E1F7B7,$DC61088FE8704524,$2001239380553FDE,$E3B4E1DF4015C926
    Data.q $BF463B70AB5DA4AB,$FD6B19E3B8126599,$BBD353112A8CA22D,$1D64D121EA689973,$76FAFE015E612A5A
    Data.q $BE8F3123FB057F40,$CB7401FFDAA0FF47,$00002FA76A0C96CB,$42AE444E45490000
    Data.b $60,$82
  PB_End:
EndDataSection

Code: Select all

Debug ?PB_End - ?PB_Start

DataSection
  PB_Start:
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s "#"
  PB_End:
EndDataSection

Re: DATA Tool - 50% smaller output - First sample code

Posted: Sun May 16, 2021 7:19 am
by Keya
Saki wrote: Sat May 15, 2021 3:36 pmIt is Unicode based and can store the required data in one character instead of two.
So it converts from unicode to ascii (in a lossy way) to achieve the 50% space reduction?
You could just use DATA.a instead of DATA.s for this ... ?

Re: DATA Tool - 50% smaller output - First sample code

Posted: Sun May 16, 2021 9:34 am
by Saki
Hi,
thanks for the interest and feedback.

@Keya
The basis of this code is the idea that you cannot store a complete byte in an ascii character, but in a 16bit unicode character.
Zero, for example, terminates the string and there are also many control characters that can completely destroy a text, depending on how and in which order they occur.

PB now uses only UCS-2 Unicode with over 60 000 defined characters.

So the idea is to use a single 16 bit Unicode character to store a byte in it.

In my tool here you can see characters that exist but are undefined.
Depending on the character, they are displayed as dots or small boxes.

However, it is also possible to select a different Unicode range for the output in which defined characters are located.
You set this with an offset.
Depending on how you choose the offset, the output then falls into the Ansi range or, for example, Javanese or Chinese.
This is only a simple parameter.
However, everything can still be read out in the same way.
You don't have to pay attention to anything else.
Start and end address of the data lines will suffice in the final code.
It doesn't matter which characters are mapped there.
Whether a procedure is established or not is irrelevant.
It is a tiny module that can create these lines and read them again.

@JHPJHP
Base 64 , I don't like the buffer handling and it's so incredibly ugly.
And it is also always about 35% bigger ! :o
You convert the BMP original PB template to PNG first,
then of course it becomes much smaller, but, that's not a fair comparison. :!:
I did the math, your PNG image is 3714 bytes, my BMP image is 7812 bytes.
That is 47.5% in contrast to my image.
Seven and three is eleven, I'm missing a chicken. :lol:

And it looks just chic. :wink:
The tool provides also a high level of protection against the manipulation or alteration of images embedded in the codes. :!:
The pictures cannot be changed or localised by strangers if one does not
know exactly what is there or what exactly is happening.

I still have to check if everything works without errors.

Re: DATA Tool - 50% smaller output - First sample code

Posted: Sun May 16, 2021 12:50 pm
by Saki
Now, the first complete module is ready and posted on T&T

https://www.purebasic.fr/english/viewto ... 97#p570097

Re: DATA Tool - 50% smaller output - First sample code

Posted: Sun May 16, 2021 4:30 pm
by JHPJHP
walbus/Saki wrote:I have a new tool ready which can output data lines for images and icons reduced by 50%
Not sure if you're intentionally being vague or misleading, but your results seem to be based on the BMP image format only? Comparing ICO, PNG, etc. file formats using your tool with Bin2Data by ts-soft, and the results continuously show your DataSection is over twice in size.

Code: Select all

Debug ?Icon_End - ?Icon_Start

DataSection
  Icon_Start:
    Data.q $2023000100010000,$12A8002000010000,$0028000000160000,$0040000000230000,$0000002000010000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$FF00000000000000
    Data.q $FE00FF00FF000300,$FF00FF00FE00FF00,$FF00FF00FF00FF00,$FE00FF00FF00FF00,$000000000000FF00
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$00FFFF0000FE0000
    Data.q $00FFFF0000FFFF00,$00FFFF0000FFFF00,$00FFFF0000FEFF00,$0000000000FEFF00,$0000000000000000
    Data.q $0000000000000000,$FF00000000000000,$FF00FF00FF00FF00,$FF00FF00FF00FF00,$FF00FF00FF00FF00
    Data.q $FF00FF00FF00FF00,$FF00FF00FF00FF00,$00000000FF00FF00,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$00FFFF0000FF0000,$00FFFF0000FFFF00,$00FFFF0000FFFF00,$00FFFF0000FFFF00
    Data.q $00FFFF0000FFFF00,$0000130000FFFF00,$0000000000000000,$FF00000000000000,$FF00FF00FE00FF00
    Data.q $FE00FF00FF00FF00,$FF00FF00FE00FF00,$FE00FF00FE00FF00,$FE00FF00FF00FF00,$FE00FF00FF00FF00
    Data.q $000000000000FF00,$0000000000000000,$0000000000000000,$00FFFF0000FF0000,$00FFFF0000FFFF00
    Data.q $00FEFF0000FFFF00,$00FFFF0000FFFF00,$00FFFF0000FEFF00,$00FEFF0000FFFF00,$000000000000FF00
    Data.q $FF00000000000000,$FF00FF00FF000000,$FF00FF00FF00FF00,$FE00FF00FE00FF00,$FF00FF00FF00FF00
    Data.q $FF00FF00FF00FF00,$FF00FF00FF00FF00,$FF00FF00FF00FF00,$000000000000FF00,$0000000000000000
    Data.q $00FFFF0000FE0000,$00FFFF0000FFFF00,$00FFFF0000FFFF00,$00FFFF0000FFFF00,$00FEFF0000FFFF00
    Data.q $00FFFF0000FFFF00,$00FFFF0000FEFF00,$000000000000FF00,$FF00FF00FE000000,$FF00FF00FE00FF00
    Data.q $FE00FF00FF00FF00,$0000FF00FE00FF00,$FE00000000000000,$FE00FF00FF00FF00,$FE00FF00FF00FF00
    Data.q $FF00FF00FF00FF00,$0000000000000100,$00FE000000000000,$00FFFF0000FFFF00,$00FFFF0000FFFF00
    Data.q $00FFFF0000FFFF00,$00FF000000002E00,$00FFFF0000FEFE00,$00FEFF0000FFFF00,$00FEFF0000FFFF00
    Data.q $FE0000000000FF00,$FF00FF00FF00FF00,$FF00FF00FF00FF00,$00000000FF00FF00,$0000000000000000
    Data.q $FF00000000000000,$FF00FF00FF00FF00,$FF00FF00FF00FF00,$0000FE00FE00FF00,$00FF000000000000
    Data.q $00FFFF0000FEFF00,$00FFFF0000FFFF00,$0000FD0000FFFF00,$0000000000000000,$0000000000000000
    Data.q $00FFFF0000FF0000,$00FFFF0000FEFF00,$0000FF0000FFFF00,$FF00FF00FE000000,$FF00FF00FE00FF00
    Data.q $0000FF00FF00FF00,$0000000000000000,$0000000000000000,$FE00000000000000,$FF00FF00FF00FF00
    Data.q $FF00FF00FF00FF00,$000000000000FF00,$00FEFF0000FE0000,$00FFFF0000FFFF00,$0000FF0000FFFF00
    Data.q $0000000000000000,$0000000000000000,$00FF000000000000,$00FEFF0000FFFF00,$00FFFF0000FFFF00
    Data.q $FE0000000000FF00,$FF00FF00FF00FF00,$FE00FF00FF00FF00,$000000000000D500,$0000000000000000
    Data.q $0000000000000000,$FF00FF00FE000000,$FE00FF00FE00FF00,$0000FF00FE00FF00,$00FE000000000000
    Data.q $00FFFF0000FEFF00,$00FFFF0000FFFF00,$000000000000FF00,$0000000000000000,$0000000000000000
    Data.q $00FFFA0000FF0000,$00FFFF0000FEFF00,$0000FF0000FFFF00,$FF00FF00FE000000,$FF00FF00FF00FF00
    Data.q $0000FC00FF00FF00,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$00FEFF0000FE0000,$00FFFF0000FFFF00,$0000FF0000FFFF00
    Data.q $0000000000000000,$0000000000000000,$00FF000000000000,$00FEFF0000FFFF00,$00FFFF0000FFFF00
    Data.q $FE0000000000FF00,$FE00FF00FF00FF00,$FF00FF00FF00FF00,$00000000FE00FF00,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$00FE000000000000
    Data.q $00FFFF0000FEFF00,$00FFFF0000FFFF00,$0000D60000FEFF00,$0000000000000000,$00FE000000000000
    Data.q $00FFFF0000FF0000,$00FFFF0000FEFF00,$0000FF0000FFFF00,$FF00FF00FE000000,$FE00FF00FE00FF00
    Data.q $FE00FF00FE00FF00,$FF00BB00FF00FF00,$FF000200FF000000,$0000000000002700,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$00FE060000FF0000,$00FEFF0000FEFF00,$00FFFF0000FFFF00
    Data.q $000000000000FF00,$0000000000000000,$00FFFF0000FE0000,$00FEFF0000FFFF00,$00FEFF0000FFFF00
    Data.q $000000000000FF00,$FF00FF00FF000000,$FF00FF00FF00FF00,$FF00FF00FE00FF00,$FF00FF00FF00FF00
    Data.q $0000FF00FF00FF00,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $00FFFF0000FF0000,$00FFFF0000FFFF00,$0000FF0000FFFF00,$0000000000000000,$00FF000000000000
    Data.q $00FFFF0000FFFC00,$00FFFF0000FFFF00,$000000000000FF00,$FF00000000000000,$FF00FF00FF00F900
    Data.q $FE00FF00FF00FF00,$FF00FF00FF00FF00,$FF00FF00FE00FF00,$0000FF00FE00FF00,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$00FE000000000000,$00FFFF0000FF0000,$0000FF0000FFFF00
    Data.q $0000000000000000,$0000000000000000,$00FF000000000000,$00FFFF0000FFFF00,$0000FF0000FFFF00
    Data.q $0000000000000000,$FF00000000000000,$FF00FF00FE00FD00,$FE00FF00FE00FF00,$FE00FF00FF00FF00
    Data.q $FF00FF00FE00FF00,$00000000FF00FF00,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$00FFFF0000FE0000,$000000000000FB00,$0000000000000000,$0000000000000000
    Data.q $00FF7C0000FF0000,$0000FF0000FEFF00,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $FE00FF00FF000000,$FF00FF00FE00FF00,$FE00FF00FE00FF00,$FF00FF00FF00FF00,$000000000000FB00
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$FE000000FF000000,$FE00FF00FF005A00
    Data.q $000000000000F400,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$000002FF00000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$00000000000000FF
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000FFFE00000000,$00003FFE0000FFFF,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000FFFF0000FFFF,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000FBFF00000000,$0000FFFF0000FFFE
    Data.q $000000000000FFFF,$0000000000000000,$0000000000000000,$0000FFFF0000FFFE,$0000FFFF0000FFFF
    Data.q $00000000000002FE,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000FFFE00000000,$0000FFFE0000FFFF,$0000FFFF0000FFFF,$00000000000015FF,$0000000000000000
    Data.q $0000FFFF00000000,$0000FFFF0000FFFF,$0000FFFF0000FFFF,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000FFFF0000FEFF,$0000FFFF0000FFFF
    Data.q $000017FF0000FFFF,$0000000000000000,$0000000000000000,$0000FFFF0000FFFE,$0000FFFF0000FFFF
    Data.q $000000000000FFFF,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000FFFF0000FFFE,$0000FFFF0000FFFF,$0000FFFF0000FFFF,$00000000000001FF,$0000000000000000
    Data.q $0000FFFF00000000,$0000FFFF0000FFFF,$0000FFFF0000FFFF,$000000000000FFFF,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000FFFE00000000,$0000FFFF0000FFFF,$0000FFFF0000FFFF
    Data.q $00000000000019FF,$0000000000000000,$0000000000000000,$0000FFFF00000000,$0000FFFF0000FFFF
    Data.q $0000FFFF0000FFFF,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000FFFE0000FFFE,$0000FFFE0000FFFF,$000001FF0000FFFF,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000FFFF0000FFFF,$0000FFFF0000FFFF,$000000000000FFFF,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000FFFE00000000,$0000FFFF0000FFFE,$0000FFFE0000FFFE
    Data.q $0000000000000BFF,$0000000000000000,$0000000000000000,$0000FFFE00000000,$0000FFFF0000FFFE
    Data.q $0000FFFF0000FFFF,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000FFFE0000FFFE,$0000FFFE0000FFFE,$0000FFFE0000FFFF,$0000000000000000,$0000000000000000
    Data.q $0000FFFF00000000,$0000FFFE0000FFFE,$0000FFFF0000FFFF,$000000000000FFFE,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000FFFE0000FDFF,$0000FFFF0000FFFE
    Data.q $0000FFFE0000FFFF,$00000000000000FE,$0000FFFE00000000,$0000FFFE0000FFFE,$0000FFFF0000FFFE
    Data.q $000000000000FFFF,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000FFFE00000000,$0000FFFE0000FFFE,$0000FFFF0000FFFF,$0000FFFE0000FFFE,$0000FFFF0000FFFF
    Data.q $0000FFFF0000FFFF,$0000FFFE0000FFFE,$0000FFFF0000FFFF,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000FFFE00000000,$0000FFFF0000FFFF
    Data.q $0000FFFF0000FFFF,$0000FFFF0000FFFF,$0000FFFF0000FFFF,$0000FFFF0000FFFF,$0000FFFF0000FFFF
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000FFFE00000000,$0000FFFF0000FFFF,$0000FFFF0000FFFF,$0000FFFF0000FFFF
    Data.q $0000FFFE0000FFFE,$0000FFFF0000FFFF,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000FFFE0000FFFF,$0000FFFF0000FFFE,$0000FFFE0000FFFF,$00005EFE0000FFFE,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$00006AFF0000FFFF
    Data.q $0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000,$0000000000000000
    Data.q $0000000000000000,$0000000000000000,$07F8000000000000,$03E0000000E001FE,$01C0000000E000FC
    Data.q $00C00000006000F8,$C0800000002000F0,$E0830000000030F0,$F083000000007C60,$F08300000000FC60
    Data.q $FF8300000000FC60,$FF8300000000FCE0,$FF80000000007CE0,$1FC00000000078F0,$0FC00000002078F0
    Data.q $0FE000000020FCF8,$07F800000060FEFC,$3FFF000000E0FFFF,$FFFF000000E0FFFF,$F3FF000000E0FFFF
    Data.q $E1FF000000E0FFF9,$C1FF000000E0FFF0,$C1FF000000E07FF0,$81FF000000E07FF0,$83FF000000E03FF0
    Data.q $83FF000000E03FF8,$83FF000000E03FF8,$81FF000000E03FF8,$C0FF000000E03FF0,$C0FF000000E07FE0
    Data.q $E0FF000000E07F00,$F0FF000000E0FF00,$FCFF000000E0FF01,$FFFF000000E0FF07
    Data.b $BF,$FF,$E0,$00,$00,$00
  Icon_End:
EndDataSection

Code: Select all

Debug ?Icon_End - ?Icon_Start

DataSection
  Icon_Start:
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s "."
  Icon_End:
EndDataSection

Re: DATA Tool - 50% smaller output - First sample code

Posted: Sun May 16, 2021 4:42 pm
by Demivec
JHPJHP wrote: Sun May 16, 2021 4:30 pm
I have a new tool ready which can output data lines for images and icons reduced by 50%
Not sure if you're intentionally being vague or misleading, but your results seem to be based on the BMP image format only? Comparing ICO, PNG, etc. file formats using your tool with Bin2Data by ts-soft, and the results continuously show your DataSections are over twice in size.
@Saki: Following up on JHPJHP post, you convert each byte of the input file to a 2 byte Unicode character. This does indeed generate output that is 200% + the size of the input. Is the description '50% smaller output' not to be compared with the input but instead to the output of some other method of conversion?

Re: DATA Tool - 50% smaller output - First sample code

Posted: Sun May 16, 2021 4:49 pm
by Saki
Hello, yes.
I'm only concerned about the space required in the source code.
You don't need two characters to represent one byte, but only one.
This means that the codes are not so visually bloated.
In an executable file, it has the advantage that the images almost certainly cannot be manipulated or replaced,
because nobody knows what they're looking at.
Since these applications are mostly small images or icons, it makes no difference whether 20kb or 40kb are needed.

That's why I write that the output is 50% smaller.

But I think JHPJHP has already recognised this very well, otherwise he would not have
have compressed the image of the example template by more than 50% without pointing it out.
I noticed this because his demo outputs were supposed to be much larger than they were.

But you also have to keep in mind that Base64 in Data lines is also stored in Unicode in an executable, so on average then you get a size of about 270% with Base64.

Binary in hexa results in no increase, but the data part in the source code becomes more than twice as big.

Yes, it can be difficult for me to get it right in English too.

Re: DATA Tool - 50% smaller output - First sample code

Posted: Sun May 16, 2021 6:46 pm
by netmaestro
Data.q is the most efficient way to store binary data.

Re: DATA Tool - 50% smaller output - First sample code

Posted: Sun May 16, 2021 8:25 pm
by Saki
That's right, it's about the source code.
Look a little further up.
In the sample code with the tool from ts-soft, there are 5 quads in a line, that is 40 bytes.
In my solution, there are 100 bytes in one line in the above examples, each character represents one byte.
That is 60% more !
So the resulting size of my code here is only 40%, which is quite a lot.

The problem is probably primarily that not everyone recognises how it works
and thus does not see what is actually happening here.
Then misunderstandings also occur automatically.

Compare :
Above standard 40 Bytes in one row
Below mine, 100 Bytes in one row.
Image

Re: DATA Tool - 50% smaller output - First sample code

Posted: Tue May 18, 2021 10:29 pm
by Saki
The final as modules , you found here :

https://www.purebasic.fr/english/viewto ... 97#p570097

Re: DATA Tool - 50% smaller output - First sample code

Posted: Wed May 19, 2021 8:57 am
by pf shadoko
clever
I don't understand why the reviews are so critical
but why don't you store 15 bits/character?
it's not complicated
it will be 70% smaller than quad

Re: DATA Tool - 50% smaller output - First sample code

Posted: Wed May 19, 2021 3:45 pm
by Saki
Hi,
these are 16 bit characters,
Look, they are all small brackets, each bracket is a 16 bit character.
All brackets look the same, but each bracket has a different content.
The brackets are like memory cells, each bracket can store one byte.
All times are UTC+01:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited