Here are some modification introduced in this 1.2 version:
- new TSynBigTableString class to store data indexed from
a UTF-8 encoded string ID instead of a numerical ID;
- added caching for last Get() items (may speed up next
Get() a little bit);
- custom Get() method for range retrieval into a dynamic
array;
- TSynBigTable modified in order to handle custom data in
header (used to store string IDs for TSynBigTableString for
instance);
- whole engine more robust against any file corruption or type
mismatch;
- fix: Count property returned an incorrect value
(including deleted values);
- added timing (in 1/10 ms) for test steps.
- string ID are case-sensitive (important warning);
- string ID are of UTF8String type, so you must make explicit conversion in your program to the native generic string type - you can use our Utf8ToString() and StringToUtf8() function provided in the same unit, which work for all version of Delphi (Unicode aware or not);
- because of string ID, this class is a bit slower than TSynBigTable (but bottleneck is in OS disk access, not in our string ID implementation) - but in all cases, the engine is very fast;
- each string ID maximum length is 65535 (to save some disk space);
- string ID are stored in alphabetical order, and can be accessed directly from StringID[] and NumericalID[] properties.
The source files (and the compiled test exe) can be downloaded from http://synopse.info/files/SynBigTable.zip
12 reactions
1 From Harry Rogers - 18/03/2010, 17:59
Fast elegant and informative.
Thank you for this
2 From A. Bouchez - 18/03/2010, 19:04
I've uploaded a 1.2a version, which compiles without warning under Delphi 2009/2010, and which is even faster than 1.1/1.2 when adding data. It should better scale with very large number of entries.
3 From sanyin - 22/03/2010, 11:17
You have some bugs.
First, when put some large file (file1) (6 mb) then try to add another file (file2) and later overwriting file2 by deleting it first then adding, "file2" name istn written nice (filenames are duplicated)...
Second, you should add an option to extract part of file or load it from tstream.
Also, you should avoid non-unicode "fileopen" adn all strings, tstrings...
4 From A.Bouchez - 22/03/2010, 14:19
First: I'll check that, and include it in the test code.
Second: I'm not sure I understand what you mean. The code seems 100% Unicode ready. TFileName is an UnicodeString under Delphi 2009/2010. We don't use any TStrings here, neither generic string, but RawByteString for storage and UTF8String for UTF-8 encoded strings. IMHO it's perfectly correct, safe and cross-version compatible (tested under Delphi 7 and 2009).
5 From A.Bouchez - 22/03/2010, 14:20
To load a file from TStream, just use the TStringStream class.
But that's a good idea: I'll add a method which will create a TMemoryStream containing the data, in complement to the RawByteString version.
6 From sanyin - 22/03/2010, 15:54
I have added "GetPart (id,offset,len) which is needed for partialy loading (you dont have to load whole file always)
also, please try to avoid those RawByteStrings...why not to use bytearray, it uses zero based indexing, this way code looks ugly.
also set some methods as virtual to allow easy expanding.
what about directory/file manager?
try look at
http://gp.17slon.com/gp/gpstructure...
it is good but very slow (uses implemented "FAT system")
7 From A. Bouchez - 22/03/2010, 22:37
Why are RawByteStrings ugly? I like to use the data with pointers. You can overload the method to return some other type, if you want. But RawByteStrings have copy-on-write, which a dynamic array of byte doesn't have, if I'm right. In most cases, it can be MUCH faster.
I was not able to reproduce the bug you spoke about.
Here is what I wrote as test:
TS.Add(CreateString(1),'file1');
TS.Add(CreateString(2),'file2');
TS.Delete('file2');
TS.Add(CreateString(3),'file2');
As a result, 'file1' and 'file2' content are correct. I've also tried the above code with Open/Create and Pack inserted in all positions I could think of, with no error. Please provide some code to reproduce the bug.
I've added new Open() method, together with Read() and Seek() methods, to work more like a standard Stream. See version 1.3 of the unit. See http://blog.synopse.info/post/2010/...
If you want another directory/file system than NTFS, just go to Linux, you will find the one you need.
8 From A.Bouchez - 26/03/2010, 15:31
I've made a lot of tests with GuidToString() as filename. It works perfectly.
I suspect your code is not correct. Can you post the code which don't work?
Since the unit uses internaly UTF-8 encoding, WideStringCompare is pointless here.
9 From A.Bouchez - 26/03/2010, 15:31
I've made a lot of tests with GuidToString() as filename. It works perfectly.
I suspect your code is not correct. Can you post the code which don't work?
Since the unit uses internaly UTF-8 encoding, WideStringCompare is pointless here.
10 From A.Bouchez - 21/06/2010, 11:11