lightning memory mapped DB

Everything else that doesn't fall into one of the other PB categories.
User avatar
idle
Always Here
Always Here
Posts: 5042
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

lightning memory mapped DB

Post by idle »

Anyone interested in LMDB, lightning memory mapped DB
I had a nice day with much gnashing of teeth building it with VS for windows and I'm just about to start getting my head around it
and would like some help testing and wrapping it.

https://symas.com/lmdb/
AN ULTRA-FAST, ULTRA-COMPACT, CRASH-PROOF KEY-VALUE EMBEDDED DATA STORE.
Symas LMDB is an extraordinarily fast, memory-efficient database we developed for the OpenLDAP Project. With memory-mapped files, it has the read performance of a pure in-memory database while retaining the persistence of standard disk-based databases.
Bottom line, with only 32KB of object code, LMDB may seem tiny. But it’s the right 32KB. Compact and efficient are two sides of a coin; that’s part of what makes LMDB so powerful.
bench marks
http://www.lmdb.tech/bench/microbench/

I built v0.9.22 from the openldap project.
https://github.com/openldap/openldap/tr ... es/liblmdb

What's it uses, it replaces the need to mess around with SQL for starters key's can be binary or strings
It's persistent by design with full acid transactions and being memory mapped is fast.
So you can easily use it as a DB for you application or to add persistence for your applications state

1st test
https://www.dropbox.com/s/unz1m8gkphcou ... 2.zip?dl=0
Windows 11, Manjaro, Raspberry Pi OS
Image
User avatar
idle
Always Here
Always Here
Posts: 5042
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

Re: lightning memory mapped DB

Post by idle »

1st test, don't all rush to say anything constructive, like who needs a DB anyway!

https://www.dropbox.com/s/unz1m8gkphcou ... 2.zip?dl=0
make sure you compile with create temporary exe in source directory or you'll have a db in your root
Windows 11, Manjaro, Raspberry Pi OS
Image
User avatar
blueb
Addict
Addict
Posts: 1041
Joined: Sat Apr 26, 2003 2:15 pm
Location: Cuernavaca, Mexico

Re: lightning memory mapped DB

Post by blueb »

idle wrote:1st test, don't all rush to say anything constructive, like who needs a DB anyway!
:mrgreen:

Thanks Idle.

Spent an hour yesterday at the Symas web site reading over the documentation.
Certainly looks usable in PB.

Thanks for converting the library for PureBasic users, now I can dig a little deeper.

Having a multi-CPU machine I'm especially interested in...

Code: Select all

;*	The library is fully thread-aware And supports concurrent Read/write
;*	access from multiple processes And threads. Data pages use a copy-on-
;*	write strategy so no active Data pages are ever overwritten, which
;*	also provides resistance To corruption And eliminates the need of any
;*	special recovery procedures after a system crash. Writes are fully
;*	serialized; only one write transaction may be active at a time, which
;*	guarantees that writers can never deadlock. The database Structure is
;*	multi-versioned so readers run With no locks; writers cannot block
;*	readers, And readers don't block writers.
Thanks again... put me on your list.
- It was too lonely at the top.

System : PB 6.10 Beta 9 (x64) and Win Pro 11 (x64)
Hardware: AMD Ryzen 9 5900X w/64 gigs Ram, AMD RX 6950 XT Graphics w/16gigs Mem
User avatar
Kwai chang caine
Always Here
Always Here
Posts: 5342
Joined: Sun Nov 05, 2006 11:42 pm
Location: Lyon - France

Re: lightning memory mapped DB

Post by Kwai chang caine »

Thanks a lot IDLE for this another great job :shock:
Like usually i'm surely alone to not all understand several of your precious works, so much they are powerfull :oops:
And obviously this one is in the lot :wink:

But i have obviously, when even tested it, for see the effect and apparently that works on W10 X64 / v5.61 x86 :D

See yourself :
LMDB 0.9.22: (March 21, 2018)
1001 duplicates
1001
what foo is this? 0 ok!
what foo is this? 1 ok!
what foo is this? 2 ok!
what foo is this? 3 ok!
what foo is this? 4 ok!
what foo is this? 5 ok!
what foo is this? 6 ok!
what foo is this? 7 ok!
.......
etc .....
......
what foo is this? 997 ok!
what foo is this? 998 ok!
what foo is this? 999 ok!
what foo is this? 1000 ok!
For better understand, if it's possible KCC understand something a day :oops:
The "lmdb.lib" on your zip is created by you for us? from the original C lib found here ?
https://github.com/LMDB/lmdb
is it that ?
ImageThe happiness is a road...
Not a destination
User avatar
idle
Always Here
Always Here
Posts: 5042
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

Re: lightning memory mapped DB

Post by idle »

@blueb
It's a great looking lib and has been around for a while, there are numerous c test files in the git sources
which should be easy enough to convert.

@Kcc
No I used the version from
https://github.com/openldap/openldap/tr ... es/liblmdb
the page you linked to is their official lmdb mirror, but I couldn't get that to compile
and even the version from the openldap, I had to do a few function replacements, windows is a 3rd class OS according to the author.

It should be easy to compile on linux at least.
Windows 11, Manjaro, Raspberry Pi OS
Image
User avatar
Kwai chang caine
Always Here
Always Here
Posts: 5342
Joined: Sun Nov 05, 2006 11:42 pm
Location: Lyon - France

Re: lightning memory mapped DB

Post by Kwai chang caine »

Thanks IDLE for your answer 8)
I'm always impressed when someone can use a LIB create for C and back it in PB :shock:
windows is a 3rd class OS according to the author.
:lol:
ImageThe happiness is a road...
Not a destination
Post Reply