- data is stored in an unique file
- data is appended at the end of this file at adding (but use a caching mechanism for immediate adding)
- use a temporary in memory adding, till the UpdateToFile method is called
- retrieval is very fast (can't be faster IMHO)
- data items can be deleted
- file can be packed using the Pack method in order to retrieve free space from deleted entries (sounds like a VACUUM command, but faster)
- total size of file has no limit (but your hard disk, of course)
- limit of one data block depends on RAM (RawByteString is used as storage for data block)
- before Delphi 2007, much faster when using FastMM4 memory manager - see Project1.dpr source file
- after profiling, most of the time is spent in the Windows kernel, waiting from hard disk write of raw data; in all cases, this class is much faster than any SQL engine storing BLOB, and than plain Win32 files.
Source code example (extracted from TestBigTable function):
T := TSynBigTable.Create(FN);
try
for i := 1 to n do
if T.Add(CreateString(i))<>i then
exit else
if T.CurrentInMemoryDataSize>10 shl 20 then // write on disk every 10 MB
T.UpdateToFile;
for i := 1 to n do
if not T.Get(i,Data) or not TestString(i,Data) then
exit;
finally
T.Free;
end;
You can download the source code and the unit and the test program from http://synopse.info/files/SynBigTable.zip
Note that the test program, embedding the full database engine, is less than 32 KB in size (without any UPX)...
5 reactions
1 From A.Bouchez - 16/03/2010, 14:23
I've updated the unit to a 1.1 version (available at the same download address).
After some more automated tested, a problem occurs about saving deletions.
It seems now fixed.
Feel free to improve the unitary test function, and stress the class, in order to find any possible issue!
2 From Visli - 18/03/2010, 02:38
I am very interested in this BigTable, but I have not had look at the source code. I have a function demand, did not know that can realize, I need to obtain continuous records, like the following SQL sentence:
select id, Str from BigTable where id between @id - 30 and @id + 30
3 From A. Bouchez - 18/03/2010, 07:07
It's very easy: just make a for i := id-30 to id+30 loop!
Response will be immediate.
Take a look at the code!
4 From A.Bouchez - 21/06/2010, 11:13