Synopse Big Table
By A.Bouchez on 2010, Tuesday March 16, 07:49 - Synopse BigTable - Permalink
An open source Delphi unit for very fast data storage and access. If you just need to save raw data on disk, and retrieve it with an unique ID number, you can use this unit, which is much faster than any database engine. Work from Delphi 2 to Delphi 2010. Licensed under a MPL/GPL/LGPL tri-license.
TSynBigTable is a class to store huge amount of data, just specified
by an integer ID
- data is stored in an unique file
- data is appended at the end of this file at adding (but use a caching mechanism for immediate adding)
- use a temporary in memory adding, till the UpdateToFile method is called
- retrieval is very fast (can't be faster IMHO)
- data items can be deleted
- file can be packed using the Pack method in order to retrieve free space from deleted entries (sounds like a VACUUM command, but faster)
- total size of file has no limit (but your hard disk, of course)
- limit of one data block depends on RAM (RawByteString is used as storage for data block)
- before Delphi 2007, much faster when using FastMM4 memory manager - see Project1.dpr source file
- after profiling, most of the time is spent in the Windows kernel, waiting from hard disk write of raw data; in all cases, this class is much faster than any SQL engine storing BLOB, and than plain Win32 files.
Source code example (extracted from TestBigTable function):
- data is stored in an unique file
- data is appended at the end of this file at adding (but use a caching mechanism for immediate adding)
- use a temporary in memory adding, till the UpdateToFile method is called
- retrieval is very fast (can't be faster IMHO)
- data items can be deleted
- file can be packed using the Pack method in order to retrieve free space from deleted entries (sounds like a VACUUM command, but faster)
- total size of file has no limit (but your hard disk, of course)
- limit of one data block depends on RAM (RawByteString is used as storage for data block)
- before Delphi 2007, much faster when using FastMM4 memory manager - see Project1.dpr source file
- after profiling, most of the time is spent in the Windows kernel, waiting from hard disk write of raw data; in all cases, this class is much faster than any SQL engine storing BLOB, and than plain Win32 files.
Source code example (extracted from TestBigTable function):
T := TSynBigTable.Create(FN);
try
for i := 1 to n do
if T.Add(CreateString(i))<>i then
exit else
if T.CurrentInMemoryDataSize>10 shl 20 then // write on disk every 10 MB
T.UpdateToFile;
for i := 1 to n do
if not T.Get(i,Data) or not TestString(i,Data) then
exit;
finally
T.Free;
end;
You can download the source code and the unit and the test program from http://synopse.info/files/SynBigTable.zip
Note that the test program, embedding the full database engine, is less than 32 KB in size (without any UPX)...
Comments
I've updated the unit to a 1.1 version (available at the same download address).
After some more automated tested, a problem occurs about saving deletions.
It seems now fixed.
Feel free to improve the unitary test function, and stress the class, in order to find any possible issue!
I am very interested in this BigTable, but I have not had look at the source code. I have a function demand, did not know that can realize, I need to obtain continuous records, like the following SQL sentence:
select id, Str from BigTable where id between @id - 30 and @id + 30
It's very easy: just make a for i := id-30 to id+30 loop!
Response will be immediate.
Take a look at the code!
Try add filename with guid string (using guidtostr), and you'll see that retreived file is not correct in offset (few bytes). try also rewriting with this "guid" name, in hex editor I can see accumulating of these names.
also, finding of the files is not casesensitive. you should use wide string compare function (win. api)