TDynArray and Record compare/load/save using fast RTTI
The SynCommons unit has been enhanced:
- new low-level RTTI functions for handling record types:
RecordSave, RecordSaveLength, RecordLoad;
TDynArray object, which is a wrapper around any dynamic
TDynArray, you can access any dynamic array (like
TIntegerDynArray = array of integer) using
properties and methods, e.g.
Count, Add, Insert, Delete, Clear, IndexOf,
Find, Sort and some new methods like
SaveTo which allow fast binary serialization
of any dynamic array, even containing strings or records - a
CreateOrderedIndex method is also available to create individual
index according to the dynamic array content. You can also serialize the array
content into JSON, if you wish.
What I like with dynamic arrays is that they are reference-counted, don't
Create/try..finally...Free code, and are well handled by
the Delphi compiler.
They are no replacement to a
TCollection nor a
TList (which are the standard and efficient way of storing
class instances), but they are very handy way of having a list of content or a
dictionary at hand, with no class nor properties definition.
You can look at them like Python's list, tuples (via records handling) and dictionaries (via
Find method), in pure Delphi. Our new methods
(about searching and serialization) allow most usage of those script-level
structures in your Delphi code.
The main purpose of this modification was made after a question posted
to store dynamic arrays in a TList?
I also would like to handle dynamic arrays in our ORM, but we didn't have enough RTTI-based structure for this yet.
I've created a wrapper around dynamic array RTTI, in order to
TList-like properties, and even more features.
Since I wanted to handle dynamic array of records, I also created some low-level fast access to the record content, using the common RTTI, which a much faster than the "new" enhanced RTTI available since Delphi 2010.
1. TList-like properties
Here is how you can have method-driven access to the dynamic array:
type TGroup: array of integer;
var Group: TGroup; GroupA: TDynArray; i, v: integer; begin GroupA.Init(TypeInfo(TGroup),Group); // associate GroupA with Group for i := 0 to 1000 do begin v := i+1000; // need argument passed as a const variable GroupA.Add(v); end; v := 1500; if GroupA.IndexOf(v)<0 then // search by content ShowMessage('Error: 1500 not found!'); for i := GroupA.Count-1 downto 0 do if i and 3=0 then GroupA.Delete(i); // delete integer at index i end;
TDynArray wrapper will work also with array of string or
array of records...
Records need only to be packed and have only not reference counted fields (
byte, integer, double...) or
reference-counted fields (no
Yes, you read well: it will handle a dynamic array of records, in which you can put some strings or whatever data you need.
IndexOf() method will search by content. That is e.g. for
array of record, all record fields content (including strings)
TDynArray is just a wrapper around an existing
dynamic array. In the code above,
methods are modifying the content of the
You can therefore initialize a
TDynArray wrapper on need, to
access more efficiently any native Delphi dynamic array.
TDynArray doesn't contain any data: the elements are stored in the
2. Enhanced features
I've added some methods to the
TDynArray record/object, which
are not available in a plain
TList - with those methods, we come
closer to some native generics implementation:
- now you can save and load a dynamic array content to or from a stream or a string (using
LoadFrom/SaveTo methods) - it will use a proprietary but very fast
binary stream layout;
- and you can sort the dynamic array content by two means: either in-place (i.e. the array elements content is exchanged - use the
method in this case) or via an external integer index look-up array (using the
CreateOrderedIndex method - in this case, you can have several
orders to the same data);
- you can specify any custom comparison function, and there is a new
Find method will can use fast binary search if available.
Here is how those new methods work:
var Test: RawByteString; ... Test := GroupA.SaveTo; GroupA.Clear; GroupA.LoadFrom(Test); GroupA.Compare := SortDynArrayInteger; GroupA.Sort; for i := 1 to GroupA.Count-1 do if Group[i]<Group[i-1] then ShowMessage('Error: unsorted!'); v := 1500; if GroupA.Find(v)<0 then // fast binary search ShowMessage('Error: 1500 not found!');
Some unique methods like
Slice, Reverse or
AddArray are also available, and mimic well-known Python
Still closer to the generic paradigm, working for Delphi 6 up to XE, without the need of the slow enhanced RTTI...
3. Capacity handling via an external Count
One common speed issue with the
TDynArray is that the internal
memory buffer is reallocated when you change its length.
That is, whenever you call
an internal call to
SetLength(DynArray) is performed. This could
be slow, because it always executes some extra code, including a call to
In order not to suffer for this, you can define an external
Count value, as an
In this case, the
Length(DynArray) will be the memory capacity of
the dynamic array, and the exact number of stored item will be available from
Count variable. A
Count property is exposed by
TDynArray, and will always reflect the number of items stored in
the dynamic array. It will point either to the external
variable, if defined; or it will reflect the
just as usual.
As a result, adding or deleting items could be much faster.
4. JSON serialization
Take a look at the JSON serialization features:
TDynArray.LoadFromJSON methods are available for UTF-8 JSON
serialization of dynamic arrays.
Most common kind of dynamic arrays (
array of byte, word, integer,
cardinal, Int64, double, currency, RawUTF8, WinAnsiString, string) will
be serialized as a list of JSON elements.
Other not-known dynamic arrays (like any
array of packed record)
will be serialized as binary, then Base64 encoded.
If you have any ideas of standard dynamic arrays which should be handled, feel free to post your proposal in the forum!
These methods were the purpose of adding in SynCommons both
Base64ToBin functions, very optimized
for speed. In fact, we will use the Base64 encoding to load or save
any dynamic array of records from a
property... using SQLite3 blob fields for storage, but Base64
for JSON transmission (much more efficient than hexa).
This JSON serialization will indeed be used in our main ORM to support dynamic arrays as enhanced properties (stored as blob).
5. One step further
TTestLowLevelCommon._TDynArray method implements the
automated unitary tests associated with this wrapper.
You'll find out there samples of array of records and more advanced features, with various kind of data.
Feedback and comments, full interface description of the
type are available in