PureBasic Forums - English

http://www.purebasic.com
https://www.purebasic.fr/english/

Binary Serializer and Reflection system

https://www.purebasic.fr/english/viewtopic.php?t=71777

Page 1 of 1

Binary Serializer and Reflection system

Posted: Thu Nov 22, 2018 8:52 pm
by idle
I'm creating a Binary Serializer for an in memory DB project
the db engine is lmdb from the openldap project which is also used in sqlite3 for it's in memory function.
The end result will be something like this

lmdb_Set(*data,structure,key.s)
lmdb_Get(*data,structure,key.s)

It utilizes the structure layout tables which are used internally for copystructure.
This provides enough information to get at the non serial objects within a structure
and the rest of the data can be inferred from the size, offsets and current position.

Adding runtime reflection would make it a lot easier but I don't see how to do that
without writing a compiler tool. Any thoughts on how to do it without a compiler tool
would be really welcome.

It's going to take a while to do and I'm still working through enumerating
and resolving lists and maps.

In the mean time here's a routine to examine the structure layout table.
not much use of itself and I'm still undecided if there's any point of putting the info in a list

Wilbert if you read this, is there a fix for the GetStructAdr(structAdr,struct) macro for osx ?

Code: Select all

Structure PB_ListHeader 
  *next.PB_ListHeader
  *Previous.PB_ListHeader
EndStructure   
Structure PB_ListPosition
  *next.PB_ListPosition;
  *Element.PB_ListHeader;
EndStructure 

Structure PB_List
  *First.PB_ListHeader;
  *Last.PB_ListHeader;
  *Current.PB_ListHeader; // Don't move it, as it is used internally by the compiler
  *CurrentVariable.PB_ListHeader; **
  NbElements.i;
  Index.i;
  *StructureMap;
  *Allocator;
  *PositionStack.PB_ListPosition;
  *Object.PB_ListObject;
  ElementSize.i;      // moved here for better alignment on X64
  ElementType.l;
  IsIndexInvalid.b;
  IsDynamic.b;
  IsDynamicObject.b;
EndStructure
Structure PB_ListObject
  *list.PB_List;
  *CurrentElement.PB_ListElement;
EndStructure 
Structure PB_ListElement
  Header.PB_ListHeader;
	StructureUnion
	 	Long.i;
		*String;
	EndStructureUnion;
EndStructure 
Import ""
  PB_FirstElement(*List.PB_List)
  PB_ListSize(*list.PB_List);
  PB_NextElement(*List.PB_List) 
EndImport   


;
#ST_End   =      -1
#ST_StaticArray= -2
#ST_Structure =  -3
#ST_Array =      -4
#ST_List =       -5
#ST_Map =        -6
#ST_MultiArray = -7
#ST_String = 0 


Structure StructInfo 
  TypeStr.s   ;Array List Map String 
  Type.l      ;Array list map 
  TypeSize.l  ;Size of type  
  Offset.l    ;offset in structure 
  SLAdr.i     ;address of structure layout 
  NumElements.l ;Number of elements 
  NumDimentions.l ;number of dimentions
  SpanDimentions.l ; Array(2,3) 2*3 
  StuctType.l    ;type of contained structure 
EndStructure     

Structure structItr  
  index.i[0]
EndStructure 

Global NewList StInfo.StructInfo() 

CompilerIf #PB_Compiler_Processor = #PB_Processor_x64 
Macro eax : rax : EndMacro 
CompilerEndIf 

Macro GetStructAdr(structAdr,struct) 
  
  !if defined  s_#struct  
  !lea eax,[s_#struct]
  !else 
  !xor eax,eax 
  !end if 
  
  EnableASM  
  mov structAdr,eax 
  DisableASM
  
EndMacro 

#PB_FixedString = 10

Procedure.s GetTypeName(type) 
  Select type 
    Case #PB_Byte 
      ProcedureReturn "Byte" 
    Case #PB_Word
      ProcedureReturn "Word" 
    Case #PB_Long 
      ProcedureReturn "Long" 
    Case #PB_Structure 
      ProcedureReturn "Structure" 
    Case #PB_String
      ProcedureReturn "String" 
    Case #PB_Float
      ProcedureReturn "Float" 
    Case #PB_FixedString
      ProcedureReturn "FixedString" 
    Case #PB_Character
      ProcedureReturn "Character" 
    Case #PB_Double
      ProcedureReturn "Double" 
    Case #PB_Quad
      ProcedureReturn "Quad" 
    Case #PB_Integer
      ProcedureReturn "Integer" 
    Case #PB_Ascii
      ProcedureReturn "Ascii" 
    Case #PB_Unicode
      ProcedureReturn "Unicode" 
  EndSelect 
EndProcedure 

Procedure StructureInfo(structadr)
 Protected a,*itr.structItr
 
 *itr = structadr 
 
 If *itr 
   Repeat 
     x = *itr\index[a]
     Select x 
       Case #ST_StaticArray 
         AddElement(StInfo())
         stinfo()\TypeStr = "StaticArray" 
         stinfo()\Type = #ST_StaticArray 
         stinfo()\NumElements = *itr\index[a+1]
         stinfo()\Offset = *itr\index[a+2] 
         stinfo()\StuctType =  *itr\index[a+3] 
         stinfo()\SLAdr = *itr\index[a+4]
         Debug "Static Array: Offset =" + stinfo()\offset + " StructType=" + GetTypeName(stinfo()\StuctType) + " ArraySize =" + stinfo()\NumElements  + " Stucture Adr=" + stinfo()\SLAdr 
         a+4
         If stinfo()\SLAdr <> 0 
            StructureInfo(stinfo()\SLAdr)  
         EndIf  
        
       Case #ST_Structure   
         AddElement(StInfo())
         stinfo()\TypeStr = "Structure" 
         stinfo()\Type = #ST_Structure 
         stinfo()\Offset = *itr\index[a+1]
         stinfo()\TypeSize = *itr\index[a+2] 
         stinfo()\SLAdr = *itr\index[a+3]  
         Debug "Structure: Offset =" + StInfo()\Offset + " Typesize =" + StInfo()\TypeSize + " stucture adr=" + stinfo()\SLAdr 
         a+3
         If stinfo()\SLAdr <> 0 
            StructureInfo(stinfo()\SLAdr)  
         EndIf   
         
       Case #ST_Array  
         AddElement(StInfo()) 
         StInfo()\TypeStr = "Array"
         stinfo()\Type = #ST_Array 
         stInfo()\Offset = *itr\index[a+1] 
         StInfo()\TypeSize = *itr\index[a+2] 
         stInfo()\NumElements = *itr\index[a+3]
         StInfo()\StuctType = *itr\index[a+4] 
         StInfo()\SLAdr = *itr\index[a+5]
         Debug "Array: offset =" + stinfo()\offset + " typesize =" + stinfo()\TypeSize + " ArraySize =" + stinfo()\NumElements + " struct type=" + GetTypeName(StInfo()\StuctType) 
         a+5 
          If stinfo()\SLAdr <> 0 
           StructureInfo(stinfo()\SLAdr) 
         EndIf 
       Case #ST_List  
         AddElement(stInfo())
         StInfo()\TypeStr = "List"
         stinfo()\Type = #ST_List 
         stinfo()\Offset = *itr\index[a+1] 
         stinfo()\TypeSize = *itr\index[a+2] 
         StInfo()\SLAdr = *itr\index[a+3]
         stInfo()\StuctType = *itr\index[a+4]
         Debug "List: offset =" + stinfo()\offset + " typesize =" + stinfo()\TypeSize + " structure adr =" + stinfo()\SLAdr + " type =" + GetTypeName(stinfo()\StuctType)
         a+4
         If stinfo()\SLAdr <> 0 
           StructureInfo(stinfo()\SLAdr) 
         EndIf 
         
       Case #ST_Map 
         AddElement(stinfo())
         StInfo()\TypeStr = "Map" 
         StInfo()\Type = #ST_Map 
         StInfo()\offset = *itr\index[a+1] 
         stinfo()\TypeSize = *itr\index[a+2] 
         stinfo()\StuctType = *itr\index[a+3]
         StInfo()\SLAdr = *itr\index[a+4]
         stinfo()\NumElements = *itr\index[a+5]
         Debug "Map: offset =" + stinfo()\Offset + " typesize =" + stinfo()\TypeSize + " type =" + GetTypeName(stinfo()\StuctType) + " structure adr =" + stinfo()\SLAdr + " mapsize =" + stinfo()\NumElements   
         a+5
         If stinfo()\SLAdr <> 0 
            StructureInfo(stinfo()\SLAdr)
         EndIf
        
       Case #ST_MultiArray  
         AddElement(StInfo())
         StInfo()\TypeStr = "Multi Array"
         stinfo()\Type = #ST_MultiArray 
         stinfo()\offset = *itr\index[a+1] 
         stinfo()\NumDimentions = *itr\index[a+2]
         Stinfo()\TypeSize = *itr\index[a+3] 
         stinfo()\StuctType = *itr\index[a+4] 
         stinfo()\SLAdr = *itr\index[a+5] 
         Debug "Multi Array: offset =" + stinfo()\offset + " dimentions = " + stinfo()\NumDimentions + " typesize =" + stinfo()\TypeSize + " Type =" + GetTypeName(stinfo()\StuctType)  
         a+5
         stinfo()\SpanDimentions = 1
         For b= 1 To StInfo()\NumDimentions 
           stinfo()\SpanDimentions * *itr\index[a+b]   
           dims$ + Str(*itr\index[a+b]) + ","  
         Next
         Debug "dimentions(" + dims$  +")" 
         a+stinfo()\NumDimentions
         If stinfo()\SLAdr <> 0 
            StructureInfo(stinfo()\SLAdr)
         EndIf
         
       Default 
         If x >= 0 
           AddElement(StInfo()) 
           StInfo()\TypeStr = "String" 
           stinfo()\Type = #ST_String 
           stinfo()\Offset = x 
           Debug "string: offset =" + x     
         EndIf 
     EndSelect 
     a+1 
   Until x = #ST_End
   
   ProcedureReturn 1
 EndIf 
EndProcedure  

Structure noObject 
  a.i 
  b.i 
EndStructure   

Structure RecordStructure
  ID.l
  Name.s
  Age.l
  List country.s()
  Email.s
  no.noObject 
EndStructure

Structure foo 
  a.b
  w.w
  l.s[11]
  i.i
  s.s
  Map mp.l(4096)
  List ls.RecordStructure()
  Array ar.f(10)
  q.q
  Array ard.q(12,10,5) 
EndStructure   

Structure serial 
  a.i 
  foo.foo 
  b.s
EndStructure   


Define serial.serial 
Define noobject.noobject 

Global a.i 

GetStructAdr(a,serial)

Debug StructureInfo(a)

Debug "+++++++++++++++++++" 
ForEach StInfo() 
  Debug stInfo()\TypeStr 
  Debug stInfo()\Offset 
  Debug stinfo()\TypeSize
  
Next 
Debug "+++++++++++++++++++"  
ClearList(StInfo()) 

GetStructAdr(a,noObject) 
Debug StructureInfo(a)   
Debug "+++++++++++++++++++"

Re: Binary Serializer and Reflection system

Posted: Fri Nov 23, 2018 12:56 pm
by Kwai chang caine
Utility a little bit complicated for me :oops:
But apparently works on W10 X64 / v5.62 x86 :wink:
Thanks for sharing 8)
Structure: Offset =4 Typesize =95 stucture adr=4406692
Static Array: Offset =3 StructType= ArraySize =11 Stucture Adr=4406652
string: offset =0
string: offset =51
Map: offset =55 typesize =4 type =Long structure adr =0 mapsize =4096
List: offset =59 typesize =32 structure adr =4406660 type =Structure
string: offset =4
List: offset =12 typesize =4 structure adr =4406652 type =String
string: offset =0
string: offset =20
Array: offset =67 typesize =4 ArraySize =11 struct type=Float
Multi Array: offset =79 dimentions = 3 typesize =8 Type =Quad
dimentions(13,11,6,)
string: offset =99
1
+++++++++++++++++++
Structure
4
95
StaticArray
3
0
String
0
0
String
51
0
Map
55
4
List
59
32
String
4
0
List
12
4
String
0
0
String
20
0
Array
67
4
Multi Array
79
8
String
99
0
+++++++++++++++++++
0
+++++++++++++++++++

Re: Binary Serializer and Reflection system

Posted: Fri Nov 23, 2018 10:04 pm
by idle
Kwai chang caine wrote:Utility a little bit complicated for me :oops:
But apparently works on W10 X64 / v5.62 x86 :wink:
Thanks for sharing 8)
It'll only get more complicated but the end result will be useful.

you'll be able to save a programs state and load it again
create restore points or session histories to file or DB
send a binary blob across the network without having to fluff around serializing it.
and develop data base applications with the absolute minimal effort. .

Re: Binary Serializer and Reflection system

Posted: Sat Nov 24, 2018 8:46 pm
by Kwai chang caine
Waouuuh !! You have always powerfull idea :shock:
Often so powerfull, than i must return to school for only try to use several of yours projects :lol:

Re: Binary Serializer and Reflection system

Posted: Sat Nov 24, 2018 10:16 pm
by idle
Kwai chang caine wrote:Waouuuh !! You have always powerfull idea :shock:
Often so powerfull, than i must return to school for only try to use several of yours projects :lol:
You're probably not alone there but that's probably because the codes confusing, reflecting the mind that wrote it.
It might take some time to get it done, I'm still going round in circles trying to work it out.
All times are UTC+01:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited