- this is a MAJOR update: the file format changed (new magics $ABAB0004/5);
- now uses
SynCommons unit (avoid too much duplicated code);- buffered writing and reading to file: major speed up of the unit, since Windows file access API are dead slow; for instance, reading uses now memory-mapped files for best possible performance;
- all previous caching (not working in fact) has been disabled (the caching is now implemented more efficiently at OS level, within memory mapped files);
-
TSynBigTableString has no 65535 key length limitation any
more;- values or UTF-8 keys of fixed-size are now stored in the most efficient way;
- new
Update() methods, allowing to change the content of any
record;- new
GetPointer() methods, to retrieve a pointer to the data,
directly in memory mapped buffer (faster than a standar Get() call);- new
GetAsStream() methods, to retrieve a data into an in-memory
stream, pointing into the memory mapped buffer in most cases;- new
GetIterating() method, which will loop into all data items,
calling a call-back with pointers to each data element (very fast
method);-
fDeleted[] array now stored in ascending order, to make whole
unit faster.
New benchmarks
are impressive.
Here for 1,000,000 items of 8 bytes key/values (Y Axis is seconds):
In short: it performs better than any other key/value library, even the Tokyo Cabinet.
Our results (on my laptop, i.e. less powerful than the Xeon quad core of the
reference pdf used for other DBM on this graph) with 1,000,000 items: write
time 320.2 ms, read time (new GetIterating() method) 26.8 ms, file size
18,984,259 (18.1 MB).
The new buffered reading and writing, and usage of memory-mapped files,
improves performance a lot.
File storage has been enhanced a lot: the indexes and offsets are stored in a
very optimized way, reducing the necessary disk space needed.
It's worth adding that our main unit purpose is not to have the fastest access of low sized data (e.g. 8 bytes key/values, as in this test), but to store any amount of data (up to 1 GB) on disk, with no total size limitation (64 bit indexes).
Available from our Source Code repository and from a zip archive (with bundled exe to make your own benchmarks). Compiles with Delphi 6 up to Delphi XE.
