TDynArray and Record compare/load/save using fast RTTI

Abstract

The main purpose of this modification was made after a question posted in How to store dynamic arrays in a TList?
I also would like to handle dynamic arrays in our ORM, but we didn't have enough RTTI-based structure for this yet.

I've created a wrapper around dynamic array RTTI, in order to provide TList-like properties, and even more features.
Since I wanted to handle dynamic array of records, I also created some low-level fast access to the record content, using the common RTTI, which a much faster than the "new" enhanced RTTI available since Delphi 2010.

1. TList-like properties

Here is how you can have method-driven access to the dynamic array:

type
   TGroup: array of integer;
var
   Group: TGroup;
   GroupA: TDynArray;
   i, v: integer;
begin
  GroupA.Init(TypeInfo(TGroup),Group); // associate GroupA with Group
  for i := 0 to 1000 do begin
    v := i+1000; // need argument passed as a const variable
    GroupA.Add(v);
  end;
  v := 1500;
  if GroupA.IndexOf(v)<0 then // search by content
    ShowMessage('Error: 1500 not found!');
  for i := GroupA.Count-1 downto 0 do
    if i and 3=0 then
      GroupA.Delete(i); // delete integer at index i
end;

This TDynArray wrapper will work also with array of string or array of records...
Records need only to be packed and have only not reference counted fields (byte, integer, double...) or string reference-counted fields (no Variant nor Interface within).

Yes, you read well: it will handle a dynamic array of records, in which you can put some strings or whatever data you need.

The IndexOf() method will search by content. That is e.g. for an array of record, all record fields content (including strings) must match.

Note that TDynArray is just a wrapper around an existing dynamic array. In the code above, Add and Delete methods are modifying the content of the Group variable.
You can therefore initialize a TDynArray wrapper on need, to access more efficiently any native Delphi dynamic array.
TDynArray doesn't contain any data: the elements are stored in the dynamic array.

2. Enhanced features

I've added some methods to the TDynArray record/object, which are not available in a plain TList - with those methods, we come closer to some native generics implementation:
- now you can save and load a dynamic array content to or from a stream or a string (using LoadFromStream/SaveToStream or LoadFrom/SaveTo methods) - it will use a proprietary but very fast binary stream layout;
- and you can sort the dynamic array content by two means: either in-place (i.e. the array elements content is exchanged - use the Sort method in this case) or via an external integer index look-up array (using the CreateOrderedIndex method - in this case, you can have several orders to the same data);
- you can specify any custom comparison function, and there is a new Find method will can use fast binary search if available.

Here is how those new methods work:

var
  Test: RawByteString;
...
  Test := GroupA.SaveTo;
  GroupA.Clear;
  GroupA.LoadFrom(Test);
  GroupA.Compare := SortDynArrayInteger;
  GroupA.Sort;
  for i := 1 to GroupA.Count-1 do
    if Group[i]<Group[i-1] then
      ShowMessage('Error: unsorted!');
  v := 1500;
  if GroupA.Find(v)<0 then // fast binary search
    ShowMessage('Error: 1500 not found!');

Some unique methods like Slice, Reverse or AddArray are also available, and mimic well-known Python methods.

Still closer to the generic paradigm, working for Delphi 6 up to XE, without the need of the slow enhanced RTTI...

3. Capacity handling via an external Count

One common speed issue with the TDynArray is that the internal memory buffer is reallocated when you change its length.

That is, whenever you call Add or Delete methods, an internal call to SetLength(DynArray) is performed. This could be slow, because it always executes some extra code, including a call to ReallocMem.

In order not to suffer for this, you can define an external Count value, as an Integer variable.
In this case, the Length(DynArray) will be the memory capacity of the dynamic array, and the exact number of stored item will be available from this Count variable. A Count property is exposed by TDynArray, and will always reflect the number of items stored in the dynamic array. It will point either to the external Count variable, if defined; or it will reflect the Length(DynArray), just as usual.
As a result, adding or deleting items could be much faster.

4. JSON serialization

Take a look at the JSON serialization features: TTextWriter.AddDynArrayJSON and TDynArray.LoadFromJSON methods are available for UTF-8 JSON serialization of dynamic arrays.

Most common kind of dynamic arrays (array of byte, word, integer, cardinal, Int64, double, currency, RawUTF8, WinAnsiString, string) will be serialized as a list of JSON elements.
Other not-known dynamic arrays (like any array of packed record) will be serialized as binary, then Base64 encoded.
If you have any ideas of standard dynamic arrays which should be handled, feel free to post your proposal in the forum!

These methods were the purpose of adding in SynCommons both BinToBase64 and Base64ToBin functions, very optimized for speed. In fact, we will use the Base64 encoding to load or save any dynamic array of records from a TSQLRecord published property... using SQLite3 blob fields for storage, but Base64 for JSON transmission (much more efficient than hexa).
This JSON serialization will indeed be used in our main ORM to support dynamic arrays as enhanced properties (stored as blob).

5. One step further

The TTestLowLevelCommon._TDynArray method implements the automated unitary tests associated with this wrapper.

You'll find out there samples of array of records and more advanced features, with various kind of data.

Feedback and comments, full interface description of the TDynArray type are available in our forum.

Synopse Open Source