TSynBigTable is a class to store huge amount of data, just specified by an integer ID
- data is stored in an unique file
- data is appended at the end of this file at adding (but use a caching mechanism for immediate adding)
- use a temporary in memory adding, till the UpdateToFile method is called
- retrieval is very fast (can't be faster IMHO)
- data items can be deleted
- file can be packed using the Pack method in order to retrieve free space from deleted entries (sounds like a VACUUM command, but faster)
- total size of file has no limit (but your hard disk, of course)
- limit of one data block depends on RAM (RawByteString is used as storage for data block)
- before Delphi 2007, much faster when using FastMM4 memory manager - see Project1.dpr source file
- after profiling, most of the time is spent in the Windows kernel, waiting from hard disk write of raw data; in all cases, this class is much faster than any SQL engine storing BLOB, and than plain Win32 files.

Source code example (extracted from TestBigTable function):
  T := TSynBigTable.Create(FN);
  try
    for i := 1 to n do
      if T.Add(CreateString(i))<>i then
        exit else
      if T.CurrentInMemoryDataSize>10 shl 20 then // write on disk every 10 MB
        T.UpdateToFile;
    for i := 1 to n do
      if not T.Get(i,Data) or not TestString(i,Data) then
        exit;
  finally
    T.Free;
  end;

You can download the source code and the unit and the test program from http://synopse.info/files/SynBigTable.zip

Note that the test program, embedding the full database engine, is less than 32 KB in size (without any UPX)...