TDocVariant Pros and Cons

Since years, our TDocVariant can store any JSON/BSON document-based content, i.e. either:

  • Name/value pairs, for object-oriented documents - internally identified as dvObject sub-type;
  • An array of values (including nested documents), for array-oriented documents - internally identified as dvArray sub-type;
  • Any combination of the two, by nesting TDocVariant instances.

Every TDocVariant instance is also a custom variant type:

  • So you can just store or convert it to or from variant variables;
  • You can use late binding to access its object properties, which is some kind of magic in the rigid world of modern pascal;
  • The Delphi IDE (and Lazarus 3.x) debuggers have native support of it, so can display its variant content as JSON;
  • If you define variant types in any class or record, our framework will recognize TDocVariant content and (un)serialize it as JSON, e.g. in its ORM, SOA or Mustache/MVC parts.

Several drawbacks come also from this power:

  • Switching between variant and its TDocVariantData record may be tricky, and it sometimes requires some confusing pointer references;
  • Each TDocVariant instance could be used as a weak reference to other data, or maintain its own content - in some corner cases, incorrect use may leak memory or get some GPF issues;
  • A TDocVariant could be either an object/dictionary or an array/list, so finding the right methods may be difficult, or raise exceptions at runtime;
  • It evolved from a simple store to a full in-memory engine, so the advanced features are usually underestimated;
  • The TDocVariantData record is far away from the class system most Delphi/FPC are used to;
  • By default, double values are not parsed - only currency - which makes sense if you don't want to loose any precision, but has been found confusing.

Enough complains.
Just make it better.

Entering IDocList and IDocDict Interfaces

We introduced two high-level wrapper interface types:

  • IDocList (or its alias IDocArray) to store a list of elements;
  • IDocDict (or its alias IDocObject) to store a dictionary of key:value pairs.

The interface methods and naming follows the usual Python List and Dictionaries, and wrap their own TDocVariant storage inside safe and class dedicated IDocList and IDocDict types.

You may be able to write on modern Delphi:

var
  list: IDocList;
  dict: IDocDict;
  v: variant;
  i: integer;
begin  
  // creating a new list/array from items
  list := DocList([1, 2, 3, 'four', 1.0594631]); // double are allowed by default

  // iterating over the list
  for v in list do
    Listbox1.Items.Add(v); // convert from variant to string

  // or a sub-range of the list (with Python-like negative indexes)
  for i in list.Range(0, -3) do
    Listbox2.Items.Add(IntToStr(i)); // [1, 2] as integer

  // search for the existence of some elements
  assert(list.Exists(2));
  assert(list.Exists('four'));

  // a list of objects, from JSON, with an intruder
  list := DocList('[{"a":0,"b":20},{"a":1,"b":21},"to be ignored",{"a":2,"b":22}]');

  // enumerate all objects/dictionaries, ignoring non-objects elements
  for dict in list.Objects do
  begin
    if dict.Exists('b') then
      ListBox2.Items.Add(dict['b']);
    if dict.Get('a', i) then
      ListBox3.Items.Add(IntToStr(i));
  end;

  // delete one element
  list.Del(1);
  assert(list.Json = '[{"a":0,"b":20},"to be ignored",{"a":2,"b":22}]');

  // extract one element
  if list.PopItem(v, 1) then
    assert(v = 'to be ignored');

  // convert to a JSON string
  Label1.Caption := list.ToString;
  // display '[{"a":0,"b":20},{"a":2,"b":22}]'
end;

and even more advanced features, like sorting, searching, and expression filtering:

var
  v: variant;
  f: TDocDictFields;
  list, list2: IDocList;
  dict: IDocDict;
begin
  list := DocList('[{"a":10,"b":20},{"a":1,"b":21},{"a":11,"b":20}]');

  // sort a list/array by the nested objects field(s)
  list.SortByKeyValue(['b', 'a']);
  assert(list.Json = '[{"a":10,"b":20},{"a":11,"b":20},{"a":1,"b":21}]');
  
  // enumerate a list/array with a conditional expression :)
  for dict in list.Objects('b<21') do
    assert(dict.I['b'] < 21);

  // another enumeration with a variable as conditional expression
  for dict in list.Objects('a=', 10) do
    assert(dict.I['a'] = 10);

  // create a new IDocList from a conditional expression
  list2 := list.Filter('b =', 20);
  assert(list2.Json = '[{"a":10,"b":20},{"a":11,"b":20}]');

  // direct access to the internal TDocVariantData storage
  assert(list.Value^.Count = 3);
  assert(list.Value^.Kind = dvArray);
  assert(dict.Value^.Kind = dvObject);
 
  // TDocVariantData from a variant intermediary
  v := list.AsVariant;
  assert(_Safe(v)^.Count = 3);
  v := dict.AsVariant;
  assert(_Safe(v)^.Count = 2);

  // high-level Python-like methods
  if list.Len > 0 then
    while list.PopItem(v) do
    begin
      assert(list.Count(v) = 0); // count the number of appearances
      assert(not list.Exists(v));
      Listbox1.Items.Add(v.a); // late binding 
      dict := DocDictFrom(v); // transtyping from variant to IDocDict
      assert(dict.Exists('a') and dict.Exists('b'));
      // enumerate the key:value elements of this dictionary
      for f in dict do
      begin
        Listbox2.Items.Add(f.Key);
        Listbox3.Items.Add(f.Value);
      end;
    end;

  // create from any complex "compact" JSON
  // (note the key names are not "quoted")
  list := DocList('[{ab:1,cd:{ef:"two"}}]');

  // we still have the late binding magic working
  assert(list[0].ab = 1);
  assert(list[0].cd.ef = 'two');

  // create a dictionary from key:value pairs supplied from code
  dict := DocDict(['one', 1, 'two', 2, 'three', _Arr([5, 6, 7, 'huit'])]);
  assert(dict.Len = 3); // one dictionary with 3 elements
  assert(dict.Json = '{"one":1,"two":2,"three":[5,6,7,"huit"]}');

  // convert to JSON with nice formatting (line feeds and spaces)
  Memo1.Caption := dic.ToString(jsonHumanReadable);

  // sort by key names
  dict.Sort;
  assert(dict.Json = '{"one":1,"three":[5,6,7,"huit"],"two":2}');

  // note that it will ensure faster O(log(n)) key lookup after Sort:
  // (beneficial for performance on objects with a high number of keys)
  assert(dict['two'] = 2); // default lookup as variant value
  assert(dict.I['two'] = 2); // explicit conversion to integer
end;

Since the high-level instances are interface and the internal content is variant, their life time are both safe and usual - and you don't need to write any try..finaly list.Free code.

And performance is still high, because e.g. a huge JSON array would have a single IDocList allocated, and all the nested nodes will be hold as efficient dynamic arrays of variants.

Two last one-liners may show how our mORMot library is quite unique in the forest/jungle of JSON libraries for Delphi and FPC:

assert(DocList('[{ab:1,cd:{ef:"two"}}]')[0].cd.ef = 'two');

assert(DocList('[{ab:1,cd:{ef:"two"}}]').First('ab<>0').cd.ef = 'two');

If you compare e.g. to how the standard Delphi JSON library works, with all its per-node classes, you may find quite a difference!
Note that those both lines compile and run with the antique Delphi 7 compiler - who said the pascal language was not expressive, even back in the day?

We hope we succeeded in forging a new way to work with JSON documents, so that you may consider it for your projects on Delphi or FPC.
Any feedback is welcome in our forum, as usual!

BTW, do you know why I picked up this 1.0594631 number in the code?
Hint: this is something I used when I was a kid programming music on a Z80 CPU... and I still remember this constant. :D