Here are the main changes to this version:
- new Open() Read() and Seek() methods to read data like in
a TStream
- new Clear method to flush the table and rebuild from
scratch
- don't cache data bigger than 1 MB (to save RAM)
You can freely download the 1.3 updated version from http://synopse.info/files/SynBigTable.zip
The purpose of this unit seems not to be very clear. Here are some typical usage:
What are these classes meant for?
- store thunmbails of pictures which content is not intented to change
- a logging or read/only audit trail mechanism (this kind of data is not often
deleted)
- access to compressed data on a CD-ROM or DVD-ROM (see our Open Source
compression libraries in our web site) - you can even add items to the list,
since they will remain in memory; but they will be lost when you close the
file
- cache a huge quantity of generated HTML or XML pages
- add a simple storage and data persistence to your application, in a
few KB of code
- have an easy way to share data between Delphi 7 and Delphi 2010 applications,
without any Unicode headache
- etc... etc...
What are these classes NOT for?
- replacing NTFS - go to Linux and pick the right file system you want
- storing hierarchical data (like directories) - use B-Tree instead
- store big data items (more than some MB)
- store data elements which change a lot, and which are often deleted
- replace a SQL database engine - use our SQLite3
framework instead
- save your soul or make me rich
13 reactions
1 From A.Bouchez - 26/03/2010, 10:25
It's a python-like style: blocks are viewed via the indentation, and I can see more useful code lines on the same screen height.
Use any code formatter (as the one available in Delphi 2010) if you don't like this style.
The great CnWizard I'm using with my old Delphi 7 IDE handles this code style just as perfect.
2 From SimonB - 30/03/2010, 09:41
Looks very useful indeed. My only complaint is the lack of any real world examples. I have run your demo project and I still don't have a clue how to use SynopseBigTable.
I think there would be a lot of interest in this if you were to provide a simple form-based example showing how to store and retrieve different types of data -- record type data, text documents, HTML files, etc. I think it'll take me an hour or so to work this out myself -- which seems a waste when you could show the user straight away. The size of the data and the SBT's speed isn't important -- we can take it for granted that it's extremely fast.
Simon
3 From SimonB - 30/03/2010, 10:48
The more I think about it, the more it seems to me Synopse BigTable could be a very handy addition to all sorts of Delphi projects -- if only it were a little more accessible.
For example, at the moment I use DiSQLite3 to store graphics configurations -- details of graphic object type, size, position, etc. But SQLite3 is unnecessarily powerful for my purposes. All I really need is a kind of flat file Blob database with rudimentary indexing facilities (or not even those). In other words, just what SBT already provides.
How about providing a simple form-based demo where you show how to store and then retrieve jpg images using a open picture dialog? Or just a few lines of code.
Simon
4 From SimonB - 03/04/2010, 12:04
Working out how to use Synpose Big Table was easier than I thought it would be from looking at the TestBigTable code.
I still think it would be very useful to have a form-based demo giving a variety of applications with different types of data.
5 From A. Bouchez - 03/04/2010, 12:28
Yes ... I'm not good at demo writing!!!!
6 From SimonB - 09/04/2010, 18:16
I have worked out how store most kinds of data. Perhaps it was a good learning experience -- to have to do this myself, unaided by manual or examples.
The one thing I would still like to know, and probably won't be able to discover by studying the code, is how to use the custom header feature. Any chance of a few words of guidance? Perhaps just a few lines of example code. One line? A sentence?
I should add that I think Synapse Big Table is a fantastically handy tool -- possibly the best Delphi utility since .... Mike Litschke's Virtual Treeview.
7 From A. Bouchez - 10/04/2010, 15:14
Adding new file format need some understanding of the file layout on disk.
In short, the TSynBigTable file is just the concatenation of the data blocks, following by some "header" data (yes, I know, header should be at the head, not at the end....). This "header" data is basically the ID and the data offset tables. This "header" data is read in memory, during all the TSynBigTable instance, and written back at the end of the file, just after the last appened data block, during update to file (e.g. TSynBigTable.Free, of course).
Note: there is no potential file corruption if power fails before the TSynBigTable instance is freed. All the newly added data is contained in memory, in what is called a "journal" in most DB, i.e. fInMemory*[] arrays of the class. This content is written, together with the updated "header" data, to the disk.
So the custom header feature is a way of extending the header content. I used this to store a UTF-8 encoded string ID, together with the numerical ID, for the TSynBigTableString class.
See TSynBigTableString.CustomHeader method for an example of how to add data to the header.
Your custom method has to handle actions, defined with the TSynBigTableCustomHeader type:
- if Action is sbtWrite, your CustomHeader method should write all additional header data into fFile, then return the size of data written into fFile as an integer
- if Action is sbtRead, your CustomHeader method should read the custom header data from fFile
- if Action is sbtGetMagic, your CustomHeader methodshould return a magic integer value, which will be used to identify the file format, i.e. the class (e.g. TSynBigTable and TSynBigTableString have two different magic numbers, so that each of them is able to load a true content of its class).
8 From visli - 04/05/2010, 09:42
Is SynBigTable thread safe?
9 From A.Bouchez - 04/05/2010, 11:59
It would be thread safe if you use a Critical Section to protect the access to the class, which must be unique. Or even better by using a TMultiReadExclusiveWriteSynchronizer: Unlike a critical section, which blocks all other threads from reading or writing its associated memory, TMultiReadExclusiveWriteSynchronizer allows multiple threads to read from the protected memory simultaneously, while ensuring that any thread writing to the memory has exclusive access.
I have modified the component to had such a thread-safe behavior. Just define the THREADSAFE conditional to enable it.
10 From SimonB - 14/05/2010, 07:59
Is there any way of modifying SynBigTable for Delphi 7, which doesn't support rawbytestring?
11 From A. Bouchez - 14/05/2010, 12:26
The unit works very well with Delphi 7, in which AnsiString has no CodePage, so is a rawbytestring.
At the beginning of the unit, the following code:
type
{$ifndef UNICODE}
/// compatibility definition for storing raw data before Delphi 2009
RawByteString = AnsiString;
{$endif}
will define a RawByteString for Delphi 7.
It works perfectly. You can transtype RawbyteString and string/AnsiString in any Delphi 7 code.
12 From A.Bouchez - 21/06/2010, 11:10