2024-02-01

IDocList/IDocDict JSON for Delphi and FPC

Since years, our Open Source mORMot framework offers several ways to work with any combination of arrays/objects documents defined at runtime, e.g. via JSON, with a lot of features, and very high performance.

Our TDocVariant custom variant type is a powerful way of working with such schema-less data, but it was found confusing by some users.
So we developed a new set of interface definitions around it, to ease its usage, without sacrificing its power. We modelized them around Python Lists and Dictionaries, which is proven ground - with some extensions of course.

TDocVariant Pros and Cons

Since years, our TDocVariant can store any JSON/BSON document-based content, i.e. either:

  • Name/value pairs, for object-oriented documents - internally identified as dvObject sub-type;
  • An array of values (including nested documents), for array-oriented documents - internally identified as dvArray sub-type;
  • Any combination of the two, by nesting TDocVariant instances.

Every TDocVariant instance is also a custom variant type:

  • So you can just store or convert it to or from variant variables;
  • You can use late binding to access its object properties, which is some kind of magic in the rigid world of modern pascal;
  • The Delphi IDE (and Lazarus 3.x) debuggers have native support of it, so can display its variant content as JSON;
  • If you define variant types in any class or record, our framework will recognize TDocVariant content and (un)serialize it as JSON, e.g. in its ORM, SOA or Mustache/MVC parts.

Several drawbacks come also from this power:

  • Switching between variant and its TDocVariantData record may be tricky, and it sometimes requires some confusing pointer references;
  • Each TDocVariant instance could be used as a weak reference to other data, or maintain its own content - in some corner cases, incorrect use may leak memory or get some GPF issues;
  • A TDocVariant could be either an object/dictionary or an array/list, so finding the right methods may be difficult, or raise exceptions at runtime;
  • It evolved from a simple store to a full in-memory engine, so the advanced features are usually underestimated;
  • The TDocVariantData record is far away from the class system most Delphi/FPC are used to;
  • By default, double values are not parsed - only currency - which makes sense if you don't want to loose any precision, but has been found confusing.

Enough complains.
Just make it better.

Entering IDocList and IDocDict Interfaces

We introduced two high-level wrapper interface types:

  • IDocList (or its alias IDocArray) to store a list of elements;
  • IDocDict (or its alias IDocObject) to store a dictionary of key:value pairs.

The interface methods and naming follows the usual Python List and Dictionaries, and wrap their own TDocVariant storage inside safe and class dedicated IDocList and IDocDict types.

You may be able to write on modern Delphi:

var
  list: IDocList;
  dict: IDocDict;
  v: variant;
  i: integer;
begin  
  // creating a new list/array from items
  list := DocList([1, 2, 3, 'four', 1.0594631]); // double are allowed by default

  // iterating over the list
  for v in list do
    Listbox1.Items.Add(v); // convert from variant to string

  // or a sub-range of the list (with Python-like negative indexes)
  for i in list.Range(0, -3) do
    Listbox2.Items.Add(IntToStr(i)); // [1, 2] as integer

  // search for the existence of some elements
  assert(list.Exists(2));
  assert(list.Exists('four'));

  // a list of objects, from JSON, with an intruder
  list := DocList('[{"a":0,"b":20},{"a":1,"b":21},"to be ignored",{"a":2,"b":22}]');

  // enumerate all objects/dictionaries, ignoring non-objects elements
  for dict in list.Objects do
  begin
    if dict.Exists('b') then
      ListBox2.Items.Add(dict['b']);
    if dict.Get('a', i) then
      ListBox3.Items.Add(IntToStr(i));
  end;

  // delete one element
  list.Del(1);
  assert(list.Json = '[{"a":0,"b":20},"to be ignored",{"a":2,"b":22}]');

  // extract one element
  if list.PopItem(v, 1) then
    assert(v = 'to be ignored');

  // convert to a JSON string
  Label1.Caption := list.ToString;
  // display '[{"a":0,"b":20},{"a":2,"b":22}]'
end;

and even more advanced features, like sorting, searching, and expression filtering:

var
  v: variant;
  f: TDocDictFields;
  list, list2: IDocList;
  dict: IDocDict;
begin
  list := DocList('[{"a":10,"b":20},{"a":1,"b":21},{"a":11,"b":20}]');

  // sort a list/array by the nested objects field(s)
  list.SortByKeyValue(['b', 'a']);
  assert(list.Json = '[{"a":10,"b":20},{"a":11,"b":20},{"a":1,"b":21}]');
  
  // enumerate a list/array with a conditional expression :)
  for dict in list.Objects('b<21') do
    assert(dict.I['b'] < 21);

  // another enumeration with a variable as conditional expression
  for dict in list.Objects('a=', 10) do
    assert(dict.I['a'] = 10);

  // create a new IDocList from a conditional expression
  list2 := list.Filter('b =', 20);
  assert(list2.Json = '[{"a":10,"b":20},{"a":11,"b":20}]');

  // direct access to the internal TDocVariantData storage
  assert(list.Value^.Count = 3);
  assert(list.Value^.Kind = dvArray);
  assert(dict.Value^.Kind = dvObject);
 
  // TDocVariantData from a variant intermediary
  v := list.AsVariant;
  assert(_Safe(v)^.Count = 3);
  v := dict.AsVariant;
  assert(_Safe(v)^.Count = 2);

  // high-level Python-like methods
  if list.Len > 0 then
    while list.PopItem(v) do
    begin
      assert(list.Count(v) = 0); // count the number of appearances
      assert(not list.Exists(v));
      Listbox1.Items.Add(v.a); // late binding 
      dict := DocDictFrom(v); // transtyping from variant to IDocDict
      assert(dict.Exists('a') and dict.Exists('b'));
      // enumerate the key:value elements of this dictionary
      for f in dict do
      begin
        Listbox2.Items.Add(f.Key);
        Listbox3.Items.Add(f.Value);
      end;
    end;

  // create from any complex "compact" JSON
  // (note the key names are not "quoted")
  list := DocList('[{ab:1,cd:{ef:"two"}}]');

  // we still have the late binding magic working
  assert(list[0].ab = 1);
  assert(list[0].cd.ef = 'two');

  // create a dictionary from key:value pairs supplied from code
  dict := DocDict(['one', 1, 'two', 2, 'three', _Arr([5, 6, 7, 'huit'])]);
  assert(dict.Len = 3); // one dictionary with 3 elements
  assert(dict.Json = '{"one":1,"two":2,"three":[5,6,7,"huit"]}');

  // convert to JSON with nice formatting (line feeds and spaces)
  Memo1.Caption := dic.ToString(jsonHumanReadable);

  // sort by key names
  dict.Sort;
  assert(dict.Json = '{"one":1,"three":[5,6,7,"huit"],"two":2}');

  // note that it will ensure faster O(log(n)) key lookup after Sort:
  // (beneficial for performance on objects with a high number of keys)
  assert(dict['two'] = 2); // default lookup as variant value
  assert(dict.I['two'] = 2); // explicit conversion to integer
end;

Since the high-level instances are interface and the internal content is variant, their life time are both safe and usual - and you don't need to write any try..finaly list.Free code.

And performance is still high, because e.g. a huge JSON array would have a single IDocList allocated, and all the nested nodes will be hold as efficient dynamic arrays of variants.

Two last one-liners may show how our mORMot library is quite unique in the forest/jungle of JSON libraries for Delphi and FPC:

assert(DocList('[{ab:1,cd:{ef:"two"}}]')[0].cd.ef = 'two');
assert(DocList('[{ab:1,cd:{ef:"two"}}]').First('ab<>0').cd.ef = 'two');

If you compare e.g. to how the standard Delphi JSON library works, with all its per-node classes, you may find quite a difference!
Note that those both lines compile and run with the antique Delphi 7 compiler - who said the pascal language was not expressive, even back in the day?

We hope we succeeded in forging a new way to work with JSON documents, so that you may consider it for your projects on Delphi or FPC.
Any feedback is welcome in our forum, as usual!

BTW, do you know why I picked up this 1.0594631 number in the code?
Hint: this is something I used when I was a kid programming music on a Z80 CPU... and I still remember this constant. :D

2024-01-01

Happy New Year 2024 and Welcome MGET

Last year 2023 was perhaps not the best ever, and, just after Christmas, we think about all people we know still in war or distress.
But in the small mORMot world, 2023 was a fine millesima. A lot of exciting features, a pretty good rank in benchmarks, and a proof of being ready for the next decade.

For this new year, we would like to introduce you to a new mORMot baby: the mget command line tool, a HTTP/HTTPS web client with peer-to-peer caching.
It is just a wrapper around a set of the new PeerCache feature, built-in the framework web client class - so you can use it in your own projects if you need to.

Continue reading

2023-12-09

Native X.509, RSA and HSM Support

Today, almost all computer security relies on asymmetric cryptography and X.509 certificates as file or hardware modules.
And the RSA algorithm is still used to sign the vast majority of those certificates. Even if there are better options (like ECC-256), RSA-2048 seems the actual standard, at least still allowed for a few years.

So we added pure pascal RSA cryptography and X.509 certificates support in mORMot.
Last but not least, we also added Hardware Security Modules support via the PKCS#11 standard.
Until now, we were mostly relying on OpenSSL, but a native embedded solution would be smaller in code size, better for reducing dependencies, and easier to work with (especially for HSM). The main idea is to offer only safe algorithms and methods, so that you can write reliable software, even if you are no cryptographic expert. :)

Continue reading

2023-10-31

Pascal In The Race: TFB Challenge Benchmarks

Round 22 of the TechEmpower Frameworks has just finished.
And this time, there was a pascal framework in the race: our little mORMot!

Numbers are quite good, because we are rated #12 among 302 frameworks over 791 runs of several configurations.

Continue reading

2023-09-08

End Of Live OpenSSL 1.1 vs Slow OpenSSL 3.0

mormotSecurity.jpg, Sep 2023

You may have noticed that the OpenSSL 1.1.1 series will reach End of Life (EOL) next Monday...
Most sensible options are to switch to 3.0 or 3.1 as soon as possible.

mormotSecurity.jpg, Sep 2023

Of course, our mORMot 2 OpenSSL unit runs on 1.1 and 3.x branches, and self-adapt at runtime to the various API incompatibilities existing between each branch.
But we also discovered that switching to OpenSSL 3.0 could led into big performance regressions... so which version do you need to use?

Continue reading

2023-09-06

Meet at EKON 27

EKON27.png, Sep 2023

There is still a bit more than one day left for "very early birds" offer for EKON 27 conference in Germany, and meet us for 3 sessions (including a half-day training/introduction to mORMot 2)!

EKON27.png, Sep 2023

Join us the 6-8th of November in Düsseldorf!

Continue reading

2023-08-24

mORMot 2.1 Released

We are pleased to announce the release of mORMot 2.1.
The download link is available on github.

The mORMot family is growing up. :)

Continue reading

2023-07-20

The LUTI and the mORMot

RegressTests.png, Jul 2023

Since its earliest days, our mORMot framework did offer extensive regression tests. In fact, it is fully test-driven, and almost 78 million individual tests are performed to cover all its abilities:

RegressTests.png, Jul 2023

We just integrated those tests to the TranquilIT build farm, and its great LUTI tool. So we have now continuous integration tests over several versions of Windows, Linux, and even Mac!
LUTI is the best mORMot's friends these days. :)

Continue reading

- page 1 of 50