- this is a MAJOR update: the file format changed (new magics $ABAB0004/5);
- now uses
SynCommonsunit (avoid too much duplicated code);
- buffered writing and reading to file: major speed up of the unit, since Windows file access API are dead slow; for instance, reading uses now memory-mapped files for best possible performance;
- all previous caching (not working in fact) has been disabled (the caching is now implemented more efficiently at OS level, within memory mapped files);
TSynBigTableStringhas no 65535 key length limitation any more;
- values or UTF-8 keys of fixed-size are now stored in the most efficient way;
Update()methods, allowing to change the content of any record;
GetPointer()methods, to retrieve a pointer to the data, directly in memory mapped buffer (faster than a standar Get() call);
GetAsStream()methods, to retrieve a data into an in-memory stream, pointing into the memory mapped buffer in most cases;
GetIterating()method, which will loop into all data items, calling a call-back with pointers to each data element (very fast method);
fDeletedarray now stored in ascending order, to make whole unit faster.
New benchmarks are impressive.
Here for 1,000,000 items of 8 bytes key/values (Y Axis is seconds):
In short: it performs better than any other key/value library, even the Tokyo Cabinet.
Our results (on my laptop, i.e. less powerful than the Xeon quad core of the
reference pdf used for other DBM on this graph) with 1,000,000 items: write
time 320.2 ms, read time (new GetIterating() method) 26.8 ms, file size
18,984,259 (18.1 MB).
The new buffered reading and writing, and usage of memory-mapped files, improves performance a lot.
File storage has been enhanced a lot: the indexes and offsets are stored in a very optimized way, reducing the necessary disk space needed.
It's worth adding that our main unit purpose is not to have the fastest access of low sized data (e.g. 8 bytes key/values, as in this test), but to store any amount of data (up to 1 GB) on disk, with no total size limitation (64 bit indexes).