Synopse Open Source - Tag - TDocVariantmORMot MVC / SOA / ORM and friends2024-02-02T17:08:25+00:00urn:md5:cc547126eb580a9adbec2349d7c65274DotclearIDocList/IDocDict JSON for Delphi and FPCurn:md5:b0a5a6970b7e21d2a5c0ea539316e6b52024-02-01T16:15:00+00:002024-02-02T17:08:25+00:00Arnaud BouchezmORMot FrameworkDelphiFreePascalGoodPracticeIDocDictIDocListJSONmORMotmORMot2performanceTDocVariant<p>Since years, our Open Source <em>mORMot</em> framework offers several ways to work with any combination of arrays/objects documents defined at runtime, e.g. via JSON, with a lot of features, and very high performance.</p>
<p><img src="https://blog.synopse.info?post/public/blog/json.png" alt="" /></p>
<p>Our <code>TDocVariant</code> custom variant type is a powerful way of working with such schema-less data, but it was found confusing by some users.<br />
So we developed a new set of <code>interface</code> definitions around it, to ease its usage, without sacrificing its power. We modelized them around Python <a href="https://www.w3schools.com/python/python_lists.asp">Lists</a> and <a href="https://www.w3schools.com/python/python_dictionaries.asp">Dictionaries</a>, which is proven ground - with some extensions of course.</p> <h4>TDocVariant Pros and Cons</h4>
<p>Since years, our <code>TDocVariant</code> can store any JSON/BSON document-based content, i.e. either:</p>
<ul>
<li>Name/value pairs, for object-oriented documents - internally identified as <code>dvObject</code> sub-type;</li>
<li>An array of values (including nested documents), for array-oriented documents - internally identified as <code>dvArray</code> sub-type;</li>
<li>Any combination of the two, by nesting <code>TDocVariant</code> instances.</li>
</ul>
<p>Every <code>TDocVariant</code> instance is also a custom <code>variant</code> type:</p>
<ul>
<li>So you can just store or convert it to or from <code>variant</code> variables;</li>
<li>You can use <em>late binding</em> to access its object properties, which is some kind of magic in the rigid world of modern pascal;</li>
<li>The Delphi IDE (and Lazarus 3.x) debuggers have native support of it, so can display its <code>variant</code> content as JSON;</li>
<li>If you define <code>variant</code> types in any class or record, our framework will recognize <code>TDocVariant</code> content and (un)serialize it as JSON, e.g. in its ORM, SOA or Mustache/MVC parts.</li>
</ul>
<p>Several drawbacks come also from this power:</p>
<ul>
<li>Switching between <code>variant</code> and its <code>TDocVariantData</code> record may be tricky, and it sometimes requires some confusing pointer references;</li>
<li>Each <code>TDocVariant</code> instance could be used as a weak reference to other data, or maintain its own content - in some corner cases, incorrect use may leak memory or get some GPF issues;</li>
<li>A <code>TDocVariant</code> could be either an object/dictionary or an array/list, so finding the right methods may be difficult, or raise exceptions at runtime;</li>
<li>It evolved from a simple store to a full in-memory engine, so the advanced features are usually underestimated;</li>
<li>The <code>TDocVariantData</code> record is far away from the class system most Delphi/FPC are used to;</li>
<li>By default, <code>double</code> values are not parsed - only <code>currency</code> - which makes sense if you don't want to loose any precision, but has been found confusing.</li>
</ul>
<p>Enough complains. <br />
Just make it better.</p>
<h4>Entering IDocList and IDocDict Interfaces</h4>
<p>We introduced two high-level wrapper <code>interface</code> types:</p>
<ul>
<li><code>IDocList</code> (or its alias <code>IDocArray</code>) to store a list of elements;</li>
<li><code>IDocDict</code> (or its alias <code>IDocObject</code>) to store a dictionary of key:value pairs.</li>
</ul>
<p>The <code>interface</code> methods and naming follows the usual Python List and Dictionaries, and wrap their own <code>TDocVariant</code> storage inside safe and class dedicated <code>IDocList</code> and <code>IDocDict</code> types.</p>
<p>You may be able to write on modern Delphi:</p>
<pre>
var
list: IDocList;
dict: IDocDict;
v: variant;
i: integer;
begin
// creating a new list/array from items
list := DocList([1, 2, 3, 'four', 1.0594631]); // double are allowed by default
// iterating over the list
for v in list do
Listbox1.Items.Add(v); // convert from variant to string
// or a sub-range of the list (with Python-like negative indexes)
for i in list.Range(0, -3) do
Listbox2.Items.Add(IntToStr(i)); // [1, 2] as integer
// search for the existence of some elements
assert(list.Exists(2));
assert(list.Exists('four'));
// a list of objects, from JSON, with an intruder
list := DocList('[{"a":0,"b":20},{"a":1,"b":21},"to be ignored",{"a":2,"b":22}]');
// enumerate all objects/dictionaries, ignoring non-objects elements
for dict in list.Objects do
begin
if dict.Exists('b') then
ListBox2.Items.Add(dict['b']);
if dict.Get('a', i) then
ListBox3.Items.Add(IntToStr(i));
end;
// delete one element
list.Del(1);
assert(list.Json = '[{"a":0,"b":20},"to be ignored",{"a":2,"b":22}]');
// extract one element
if list.PopItem(v, 1) then
assert(v = 'to be ignored');
// convert to a JSON string
Label1.Caption := list.ToString;
// display '[{"a":0,"b":20},{"a":2,"b":22}]'
end;
</pre>
<p>and even more advanced features, like sorting, searching, and expression filtering:</p>
<pre>
var
v: variant;
f: TDocDictFields;
list, list2: IDocList;
dict: IDocDict;
begin
list := DocList('[{"a":10,"b":20},{"a":1,"b":21},{"a":11,"b":20}]');
// sort a list/array by the nested objects field(s)
list.SortByKeyValue(['b', 'a']);
assert(list.Json = '[{"a":10,"b":20},{"a":11,"b":20},{"a":1,"b":21}]');
// enumerate a list/array with a conditional expression <img src="https://blog.synopse.info?pf=smile.svg" alt=":)" class="smiley" />
for dict in list.Objects('b<21') do
assert(dict.I['b'] < 21);
// another enumeration with a variable as conditional expression
for dict in list.Objects('a=', 10) do
assert(dict.I['a'] = 10);
// create a new IDocList from a conditional expression
list2 := list.Filter('b =', 20);
assert(list2.Json = '[{"a":10,"b":20},{"a":11,"b":20}]');
// direct access to the internal TDocVariantData storage
assert(list.Value^.Count = 3);
assert(list.Value^.Kind = dvArray);
assert(dict.Value^.Kind = dvObject);
// TDocVariantData from a variant intermediary
v := list.AsVariant;
assert(_Safe(v)^.Count = 3);
v := dict.AsVariant;
assert(_Safe(v)^.Count = 2);
// high-level Python-like methods
if list.Len > 0 then
while list.PopItem(v) do
begin
assert(list.Count(v) = 0); // count the number of appearances
assert(not list.Exists(v));
Listbox1.Items.Add(v.a); // late binding
dict := DocDictFrom(v); // transtyping from variant to IDocDict
assert(dict.Exists('a') and dict.Exists('b'));
// enumerate the key:value elements of this dictionary
for f in dict do
begin
Listbox2.Items.Add(f.Key);
Listbox3.Items.Add(f.Value);
end;
end;
// create from any complex "compact" JSON
// (note the key names are not "quoted")
list := DocList('[{ab:1,cd:{ef:"two"}}]');
// we still have the late binding magic working
assert(list[0].ab = 1);
assert(list[0].cd.ef = 'two');
// create a dictionary from key:value pairs supplied from code
dict := DocDict(['one', 1, 'two', 2, 'three', _Arr([5, 6, 7, 'huit'])]);
assert(dict.Len = 3); // one dictionary with 3 elements
assert(dict.Json = '{"one":1,"two":2,"three":[5,6,7,"huit"]}');
// convert to JSON with nice formatting (line feeds and spaces)
Memo1.Caption := dic.ToString(jsonHumanReadable);
// sort by key names
dict.Sort;
assert(dict.Json = '{"one":1,"three":[5,6,7,"huit"],"two":2}');
// note that it will ensure faster O(log(n)) key lookup after Sort:
// (beneficial for performance on objects with a high number of keys)
assert(dict['two'] = 2); // default lookup as variant value
assert(dict.I['two'] = 2); // explicit conversion to integer
end;
</pre>
<p>Since the high-level instances are <code>interface</code> and the internal content is <code>variant</code>, their life time are both safe and usual - and you don't need to write any <code>try..finaly list.Free</code> code.</p>
<p>And performance is still high, because e.g. a huge JSON array would have a single <code>IDocList</code> allocated, and all the nested nodes will be hold as efficient dynamic arrays of variants.</p>
<p>Two last one-liners may show how our <em>mORMot</em> library is quite unique in the forest/jungle of JSON libraries for Delphi and FPC:</p>
<pre>
assert(DocList('[{ab:1,cd:{ef:"two"}}]')[0].cd.ef = 'two');
</pre>
<pre>
assert(DocList('[{ab:1,cd:{ef:"two"}}]').First('ab<>0').cd.ef = 'two');
</pre>
<p>If you compare e.g. to how the standard Delphi JSON library works, with all its per-node classes, you may find quite a difference!<br />
Note that those both lines compile and run with the antique Delphi 7 compiler - who said the pascal language was not expressive, even back in the day?</p>
<p>We hope we succeeded in forging a new way to work with JSON documents, so that you may consider it for your projects on Delphi or FPC.<br />
Any feedback is <a href="https://synopse.info/forum/viewtopic.php?id=6805">welcome in our forum</a>, as usual!</p>
<p>BTW, do you know why I picked up this 1.0594631 number in the code?<br />
Hint: this is something I used when I was a kid programming music on a Z80 CPU... and I still remember this constant. :D</p>Delphi 10.4 / Delphi 11 Alexandria Breaking Changesurn:md5:35cc16a39c1a1906bade7a6484e7dcd52021-09-21T08:30:00+01:002021-09-21T10:23:32+01:00Arnaud BouchezmORMot Framework10.410.4.111asmblogbugDelphiFreeFreePascalInlininginterfaceperformanceTDocVariant<p>The latest revision of Delphi, named Delphi 11 Alexandria, is out.<br />
A lot <a href="https://www.embarcadero.com/products/rad-studio/whats-new-in-11-alexandria?aldSet=en-GB">of new features</a>, some enhanced platforms. Nice!<br />
But it is also for us the opportunity to come back to some breaking changes, which appeared in Delphi 10.4 earlier this year, and are now "officially" part of Delphi 11.</p>
<p><img src="https://blog.synopse.info?post/public/blog/Delphi11.png" alt="" /></p>
<p>The main breaking change of Delphi 10.4 and later, as reported by <em>mORMot</em> users, is the new lifetime of local variables.<br />
TL&LR: a local variable which is not explicitly declared, but returned by a function may be released as soon as it is not used any more, whereas in the original implementation, it was allocated as a regular local variable, and we could expect its lifetime to remain active up to the end of the function. With Delphi 10.4, it is not the case any more: the compiler could release/clear the local variable sooner, to reduce the allocation pressure.</p>
<p>Idea behind this change is that it may have better register allocation within the function, so it "may" theoretically result in faster code. Not convinced about it, anyway - we will discuss that.<br />
The main thing is that it could break existing code, because it breaks the Delphi compiler expectation since decades.<br />
Some perfectly fine working code would end to work as expected. We have identified several use cases with <em>mORMot</em> which are affected by this change. Since it seems there will be no coming back from Delphi point of view, it is worth a blog article. <img src="https://blog.synopse.info?pf=wink.svg" alt=";)" class="smiley" /></p> <h4>Early Local Variable Release: What and Why</h4>
<p>The main entry point of this new "feature" is <a href="https://quality.embarcadero.com/browse/RSP-30050">RSP-30050</a>.<br />
This entry describes the new behavior of the compiler. If you don't have access to the Quality portal, here are the highlights.</p>
<p>Since years, if a function returned an <code>interface</code> instance, then this instance would remain active until the end of the current function, and the compiler generated an eventual <code>:= nil</code> statement at its final <code>end;</code>. <br />
It is a very common way of automatic memory management in Delphi code. A lot of API return an <code>interface</code> instance, which would manage automatically its lifetime using proper reference counting. No need of <code>try ... finally Free</code> block, because the compiler will generate it for you.</p>
<p>The fact that the hidden local variable was only released at the function ending was used sometimes, e.g. to create "auto-free" class features, or change the mouse pointer on screen during a process.<br />
This RSP is about a change of lifetime: now the instance is released sooner, before the final <code>end;</code> statement of the function.</p>
<p>In practice, the compiler changed its behavior when compiling the following code:</p>
<pre>
procedure Test(anObject : TObject = nil)
begin
if not Assigned(anObject) then
begin
AutoFree(anObject, TMyClass.Create);
anObject.Init;
end; //Delphi 10.4 destroys IAutoFree here
... some other code
end; //Delphi 10.3 destroys IAutoFree here
</pre>
<p>The final reasoning from Embarcadero, in the RSP, is the following:</p>
<ul>
<li>Q/ "should we change our code to use Delphi 10.4 or newer?"<br /> A/ You should change your code. We have been considering options, but we need a better definition of the lifetime of temporaries (which was undefined in the past) and that's going to be at the most local scope level – like that of an inline variable. This is what most other programming languages do and helps the compiler optimize the generated code.</li>
<li>Sync status from internal system, internal issue closed as " Works As Expected " on Jul 26, 2021 with comment: The lifetime of temporaries (which was undefined in the past) and is at the most local scope level – like that of an inline variable. This is what most other programming languages do and helps the compiler optimize the generated code.</li>
</ul>
<p>So this is the "Delphi 11 Alexandria" way of thinking.</p>
<p>Pretty clear, and making sense - at least from the Embarcadero team point of view.<br />
From the user point of view, the benefit is not so obvious. Changing perfectly working code, on a huge project, with the risk of changed behavior, random GPF, exceptions or memory leaks, just to follow "what most other programming languages do" (tm) does not convince me.<br />
They already did it with ARC or RawByteString... and they came back to common sense, after a few years.</p>
<p>Of course, here the impact is much less than with ARC. But it is the very same logic. Why waste our time and money?<br />
My point is that Embarcadero should be more customer focused.</p>
<h3>Better Performance?</h3>
<p>Theoretically speaking, from better local variable management comes better code.<br />
This is perfectly true for highly optimized compilers like GCC or LLVM. They do wonders when generating code. I push you to consider viewing <a href="https://www.youtube.com/watch?v=bSkpMdDe4g4">this great video of Matt Godbolt about "What has my compiler done for me lately?"</a>. Fun and exciting for sure.</p>
<p>But the Delphi compiler, even with its LLVM backend, is not at this level of integration. For a regular VCL/FMX application using a database, generated code is fast enough. They should better fix inlining issues, which sometimes induce some performance problems - just check how functions returning floats are implemented. What is possible with a full LLVM stack is not possible with a Delphi front-end, because LLVM is so complex and changing, and requires a full compiler stack from front-end to back-end to leverage its full optimizing power.</p>
<p>What we did for years with Delphi, to leverage its performance, is to follow some simple rules, like:</p>
<ul>
<li>Make it right, then make it fast;</li>
<li>Identify the real bottlenecks using a profiler: don't guess;</li>
<li>Avoid unneeded calls;</li>
<li>Use tuned libraries;</li>
<li>Avoid memory allocation;</li>
<li>Avoid copies or reference counting;</li>
<li>Avoid hidden try...finally;</li>
<li>Better register allocation by using a sub-function for loops.</li>
</ul>
<p>Check our <a href="https://blog.synopse.info/?post/2018/11/12/EKON-22-Slides-and-Code">blog article and the slides and code proposed at Ekon 22</a>.</p>
<p>The last point is what interests us.<br />
Local variable allocations don't make a performance difference in normal code. It only makes a difference within a loop of thousands of occurrences. With a very small function, including only a processing loop and a few input parameters, we ease registers allocation, and performance is enhanced. For one-way simple code, stack variable allocations do not matter much in terms of performance.</p>
<p>In practice, writing a <code>SubCall()</code> dedicated function is the way to go for performance:</p>
<pre>
procedure SubCall(p: PIntegerArray; n: integer);
var
i: PtrInt;
begin
for i := 0 to n - 1 do // here every variable will be registers
p[i] := i;
end;
procedure TTestEkon22Performance.BetterRegisterAllocation;
var
ints: TIntegerDynArray;
i, j, n: integer;
timer: TPrecisionTimer;
begin
SetLength(ints, 50000);
n := 1000;
timer.Start;
for j := 1 to n do
for i := 0 to high(ints) do // here some variables will be allocated on stack - even when inlining "for var i ..."
ints[i] := i;
NotifyTestSpeed('regular loop', length(ints) * n, length(ints) * n * SizeOf(Integer), @timer);
timer.Start;
for j := 1 to n do
SubCall(pointer(ints), length(ints));
NotifyTestSpeed('dedicated call', length(ints) * n, length(ints) * n * SizeOf(Integer), @timer);
end;
</pre>
<p>Of course, we may argue that local scope can increase performance, because initialization/finalization are delayed or by-passed.<br />
This was the point of this <a href="https://blog.grijjy.com/2018/11/02/inline-variables-can-increase-performance/">good blog article</a>.<br />
But in practice, the benefit is not so obvious, because on some platforms, creating nested <code>try..finally</code> blocks for each local variable scope actually slows down the execution, or increase the linking time and executable size, because more exception traps are to be generated.</p>
<p>Inlined variables have undoubtful benefits, e.g. within a loop or to reduce the code verbosity when the type is known, and complex - which is the case with generics.<br />
So we will be fair, and read the Grijjy blog article until its conclusion: "Use With Care" and "there are some drawbacks too". No magic bullet.</p>
<h3>Show Me The Code</h3>
<p>We could find micro-benchmarks where this could make a difference.<br />
But I don't like micro-benchmarks. Real code does not lie, and from what I have seen, in real production code, there is almost no performance change since Delphi 2009. Only a few percents more or less depending on the use case. We observed noticeable boost at generics code level for sure. But it comes more from RTL optimization and (iterative) rewrite, than from compiler improvements. The biggest performance boost was in Delphi 2006/2007, back when inlining was introduced in the compiler. Since then, some kind of values (like floats) have troubles being inlined. Also sometimes generated code is incorrect, or just trigger Internal errors - just ask any library maintainer using Delphi generics and inlining...</p>
<p>About code, the main argument is that proper coding should require small functions. It is the truth since early days of programming.<br />
If you have dozen of local variables, and hundredths of lines of code within a function, this is really time for refactoring into a class or a record. Don't hope that the compiler make your code any faster or maintainable.</p>
<p>So I doubt changing the local variable lifetime would make any difference for Delphi end-users.<br />
If only I could be wrong - but don't show me micro-benchmarks, they are pointless. The <em>mORMot</em> regression tests for instance, are more convincing. And they tend to be slower year after year due to Windows itself (background tasks like antivirus, slow NTFS, new CPU security mitigations...). For raw data processing, when OS is not involved, they tend to give almost the same timing since Delphi 2007. The fastest execution is on FPC + Linux, mostly due to the OS itself - and slightly to FPC better inlining abilities.</p>
<h3>In Practice, For mORMot Users</h3>
<p>As a consequence, the behavior of some well used <em>mORMot</em> features did change with 2021's Delphis:</p>
<ul>
<li><code>TSynLog.Enter</code> and automatic Leave generation in the logs;</li>
<li><code>TAutoFree</code> and automatic memory management of classes;</li>
<li><code>_Safe()</code> returning a <code>PDocVariantData</code> on a temporary <code>variant</code>.</li>
</ul>
<p>We discussed this in our forum <a href="https://synopse.info/forum/viewtopic.php?id=5709">here</a> and <a href="https://synopse.info/forum/viewtopic.php?id=5993">here</a>.</p>
<p>About <code>TSynLog.Enter</code> and <code>TAutoFree</code>, it was already the case with FPC. So for cross-compiler code, you should already use a local variable, or an explicit <code>with</code> statement.<br />
So I am fine with that.</p>
<p>My concern is about <code>_Safe()</code> and a temporary <code>variant</code>. It works fine on FPC, but is <a href="https://synopse.info/forum/viewtopic.php?id=5993">broken on latest Delphi</a>. So we have introduced <code>_DV()</code> which returns a <code>TDocVariantData</code> and not a <code>PDocVariantData</code> <a href="https://github.com/synopse/mORMot2/commit/7937b01e7fcd02033a1f70c397e3f8191e9a8aef">which is slower, but safer</a>. For me, it is a regression, and it should be fixed. We will see what would happen on Embarcadero side.</p>
<p>To circumvent these issues, <a href="https://synopse.info/forum/viewtopic.php?pid=34043#p34043">Eugene suggested that we may use Custom Managed Records</a> as replacement on new version of Delphi. But I am not sure we have the warranty that it is not affected.</p>
<p>Feedback is <a href="https://synopse.info/forum/viewtopic.php?id=5995">welcome on our forum, as usual</a>!</p>"SQL and NoSQL", not "SQL vs NoSQL"urn:md5:c22e98fb0648c165bc9ee01f23a34cdd2015-08-23T13:34:00+02:002015-08-23T13:03:13+02:00AB4327-GANDImORMot FrameworkblogDatabaseDelphiJSONMongoDBmORMotNoSQLODMOpenSourceORMSQLSQLite3TDocVariant<p>You know certainly that our <em>mORMot</em> Open Source framework is an ORM,
i.e. mapping objects to a relational / SQL database (<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_13">Object
Relational Mapping</a>).<br />
You may have followed also that it is able to connect to a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_83">
NoSQL database</a>, like <a href="https://www.mongodb.org/">MongoDB</a>, and
that the objects are then mapped via an ODM (<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_82">Object
Document Mapping</a>) - the original SQL SELECT are even <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_207">
translated on the fly to MongoDB queries</a>.</p>
<p>But thanks to <em>mORMot</em>, it is not "<em>SQL vs NoSQL</em>" - but
"<em>SQL and NoSQL</em>".<br />
You are not required to make an exclusive choice.<br />
You can share best of both worlds, depending on your application needs.</p>
<p><img src="http://bodescu.me/wp-content/uploads/2015/02/sql-vs-nosql-450x217.png" alt="" /></p>
<p>In fact, the framework is able to <em>add NoSQL features to a regular
relational / SQL database</em>, by storing JSON documents in TEXT columns.</p>
<p>In your end-user code, you just define a <code>variant</code> field in
the ORM, and store a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_80">
TDocVariant document</a> within.<br />
We also added some dedicated functions at SQL level, so that
<em>SQLite3</em> could be used as embedded fast engine, and provide
advanced WHERE clauses on this JSON content.</p> <h3>Schemaless storage via a variant</h3>
<p>As we just wrote, a first-class candidate for <em>data sharding</em> in a
<code>TSQLRecord</code> is our <em>TDocVariant custom variant type</em>.</p>
<p>You may define:</p>
<pre>
TSQLRecordData = <strong>class</strong>(TSQLRecord)
<strong>private</strong>
fName: RawUTF8;
fData: <strong>variant</strong>;
<strong>public</strong>
<strong>published</strong>
<strong>property</strong> Name: RawUTF8 <strong>read</strong> fTest <strong>write</strong> fTest <strong>stored</strong> AS_UNIQUE;
<span style="background-color:yellow;"><strong>property</strong> Data: <strong>variant read</strong> fData <strong>write</strong> fData;</span>
<strong>end</strong>;
</pre>
<p>Here, we defined two indexed keys, ready to access any data record:</p>
<ul>
<li>Via the <code>ID: TID</code> property defined at <code>TSQLRecord</code>
level, which will map the <em>SQLite3</em> <code>RowID</code> primary key;</li>
<li>Via the <code>Name: RawUTF8</code> property, which will was marked to be
indexed by setting the "<code>stored AS_UNIQUE</code>" attribute.</li>
</ul>
<p>Then, any kind of data may be stored in the <code>Data: variant</code>
published property. In the database, it will be stored as JSON UTF-8 text,
ready to be retrieved from any client, including AJAX / HTML5
applications.<br />
<em>Delphi</em> clients or servers will access those data via
<em>late-binding</em>, from its <code>TDocVariant</code> instance.</p>
<p>You just reproduced the <em>schema-less</em> approach of the NoSQL database
engines, in a few lines of code! Thanks to the <em>mORMot</em>'s <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_6">
JSON RESTful Client-Server</a></em> design, your applications are able to store
any kind of document, and easily access to them via HTTP.</p>
<p>The documents stored in such a database can have varying sets of fields,
with different types for each field. One could have the following objects in a
single collection of our <code>Data: variant</code> rows:</p>
<pre>
<em>{ name : "Joe", x : 3.3, y : [1,2,3] }</em>
<em>{ name : "Kate", x : "abc" }</em>
<em>{ q : 456 }</em>
</pre>
<p>Of course, when using the database for real problems, the data does have a
fairly consistent structure. Something like the following would be more common,
e.g. for a table persisting <em>student</em> objects:</p>
<pre>
<em>{ name : "Joe", age : 30, interests : "football" }</em>
<em>{ name : "Kate", age : 25 }</em>
</pre>
<p>Generally, there is a direct analogy between this <em>schema-less</em> style
and dynamically typed languages. Constructs such as those above are easy to
represent in <em>PHP</em>, <em>Python</em> and <em>Ruby</em>. And, thanks to
our <code>TDocVariant</code> <em>late-binding</em> magic, even our good
<em>Delphi</em> is able to handle those structures in our code. What we are
trying to do here is make this mapping to the database natural, like:</p>
<pre>
<strong>var</strong> aRec: TSQLRecordData;
aID: TID;
<strong>begin</strong>
<em>// initialization of one record</em>
aRec := TSQLRecordData.Create;
aRec.Name := 'Joe'; <em>// one unique key</em>
aRec.data := _JSONFast('{name:"Joe",age:30}'); <em>// create a TDocVariant</em>
<em>// or we can use this overloaded constructor for simple fields</em>
aRec := TSQLRecordData.Create(['Joe',_ObjFast(['name','Joe','age',30])]);
<em>// now we can play with the data, e.g. via late-binding:</em>
writeln(aRec.Name); <em>// will write 'Joe'</em>
writeln(aRec.Data); <em>// will write '{"name":"Joe","age":30}' (auto-converted to JSON string)</em>
aRec.Data.age := aRec.Data.age+1; <em>// one year older</em>
aRec.Data.interests := 'football'; <em>// add a property to the schema</em>
aID := aClient.Add(aRec,true); <em>// will store {"name":"Joe","age":31,"interests":"footbal"}</em>
aRec.Free;
<em>// now we can retrieve the data either via the aID created integer, or via Name='Joe'</em>
<strong>end</strong>;
</pre>
<p>One of the great benefits of these dynamic objects is that schema migrations
become very easy.<br />
With a traditional RDBMS, releases of code might contain data migration
scripts. Further, each release should have a reverse migration script in case a
rollback is necessary.<br />
<code>ALTER TABLE</code> operations can be very slow and result in scheduled
downtime.</p>
<p>With a <em>schema-less</em> organization of the data, 90% of the time
adjustments to the database become transparent and automatic.<br />
For example, if we wish to add GPA to the <em>student</em> objects, we add the
attribute, re-save, and all is well - if we look up an existing student and
reference GPA, we just get back null.<br />
Furthermore, if we roll back our code, the new GPA fields in the existing
objects are unlikely to cause problems if our code was well written.</p>
<p>In fact, <em>SQlite3</em> is so efficient about its indexes B-TREE storage,
that such a structure may be used as a credible alternative to much heavier
<em>NoSQL</em> engines, like <em>MongoDB</em> or <em>CouchDB</em>.<br />
With the possibility to add some "regular" fields, e.g. plain numbers (like
ahead-computed aggregation values), or text (like a summary or description
field), you can still use any needed fast SQL query, without the complexity of
<em>map/reduce</em> algorithm used by the <em>NoSQL</em> paradigm. You could
even use the <em>Full Text Search</em> - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_8">
FTS3/FTS4</a></em> - or RTREE extension advanced features of <em>SQLite3</em>
to perform your queries.<br />
Then, thanks to <em>mORMot</em>'s ability to access any external database
engine, you are able to perform a JOINed query of your <em>schema-less</em>
data with some data stored e.g. in an Oracle, PostgreSQL or MS SQL enterprise
database. Or switch later to a true <em>MongoDB</em> storage, in just one line
of code - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_84">
MongoDB + ORM = ODM</a></em>.</p>
<h3>JSON operations from SQL code</h3>
<p>As we stated, any <code>variant</code> field would be serialized as JSON,
then stored as plain TEXT in the database.<br />
In order to make a complex query on the stored JSON, you could retrieve it in
your end-user code, then use the corresponding <code>TDocVariant</code>
instance to perform the search on its content.<br />
Of course, all this has a noticeable performance cost, especially when the data
tend to grow.</p>
<p>The natural way of solving those performance issue is to add some "regular"
RDBMS fields, with a proper index, then perform the requests on those
fields.<br />
But sometimes, you may need to do some addition query, perhaps in conjunction
with "regular" field lookup, on the JSON data stored itself.<br />
In order to avoid the slowest conversion to the ORM client side, we defined
some SQL functions, dedicated to JSON process.</p>
<p>The first is <code>JsonGet()</code>, and is able to extract any value from
the TEXT field, mapping a <code>variant</code>:</p>
<table>
<tbody>
<tr>
<td><code>JsonGet(ArrColumn,0)</code></td>
<td>returns a property value by index, from a JSON array</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'PropName')</code></td>
<td>returns a property value by name, from a JSON object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Obj1.Obj2.Prop')</code></td>
<td>returns a property value by path, including nested JSON objects</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop1,Prop2')</code></td>
<td>extract properties by name, from a JSON object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop1,Obj1.Prop')</code></td>
<td>extract properties by name (including nested JSON objects), from a JSON
object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop*')</code></td>
<td>extract properties by wildchar name, from a JSON object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop*,Obj1.P*')</code></td>
<td>extract properties by wildchar name (including nested JSON objects), from a
JSON object</td>
</tr>
</tbody>
</table>
<p>If no value does match, this function would return the SQL
<code>NULL</code>. If the matching value is a simple JSON text or number, it
will be returned as a TEXT, INTEGER or DOUBLE value, ready to be passed as a
result column or any WHERE clause. If the returned value is a nested JSON
object or array, it will be returned as TEXT, serialized as JSON; as a
consequence, you may use it as the source of another <code>JsonGet()</code>
function, or even able to gather the results via the <code>CONCAT()</code>
aggregate function.</p>
<p>The comma-separated syntax allowed in the property name parameter (e.g.
<code>'Prop1,Prop2,Prop3'</code>), would search for several properties at once
in a single object, returning a JSON object of all matching values - e.g.
<code>'{"Prop2":"Value2","Prop3":123}'</code> if the <code>Prop1</code>
property did not appear in the stored JSON object.</p>
<p>If you end the property name with a <code>*</code> character, it would
return a JSON object, with all matching properties.<br />
Any nested object would have its property names be flattened as
{<code>"Obj1.Prop":...}</code>, within the returned JSON object.<br />
Note that the comma-separated syntax also allows such wildchar search, so that
e.g.</p>
<pre>
JsonGet(ObjColumn,'owner') = {"login":"smith","id":123456} as TEXT
JsonGet(ObjColumn,'owner.login') = "smith" as TEXT
JsonGet(ObjColumn,'owner.id') = 123456 as INTEGER
JsonGet(ObjColumn,'owner.name') = NULL
JsonGet(ObjColumn,'owner.login,owner.id') = {"owner.login":"smith","owner.id":123456} as TEXT
JsonGet(ObjColumn,'owner.I*') = {"owner.id:123456} as TEXT
JsonGet(ObjColumn,'owner.*') = {"owner.login":"smith","owner.id":123456} as TEXT
JsonGet(ObjColumn,'unknown.*') = NULL
</pre>
<p>Another function, named <code>JsonHas()</code> is similar to
<code>JsonGet()</code>, but will return TRUE or FALSE depending if the supplied
property (specified by name or index) do exist. It may be faster to use
<code>JsonHas()</code> than <code>JsonGet()</code> e.g. in a WHERE clause, when
you do not want to process this property value, but only return data rows
containing needed information.</p>
<pre>
JsonHas(ObjColumn,'owner') = true
JsonHas(ObjColumn,'owner.login') = true
JsonHas(ObjColumn,'owner.name') = false
JsonHas(ObjColumn,'owner.i*') = true
JsonHas(ObjColumn,'owner.n*') = false
</pre>
<p>Since the process would take place within the <em>SQLite3</em> engine
itself, and since they use a SAX-like fast approach (without any temporary
memory allocation during its search), those JSON functions could be pretty
efficient, and proudly compare to some dedicated NoSQL engines.</p>
<h3>The upcoming official SQlite3 extension</h3>
<p>As you may have noticed <a href="http://www.sqlite.org/src/info/178f9a352c6c9e15">in the SQlite3 timeline</a>,
some extension functions for processing JSON are currently being
implemented.</p>
<p>The <em>mORMot</em> native implementation is very proven, since it uses the
<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_2">
JSON core of the framework</a>, and is able to handle the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_39">
"extended" JSON syntax of MongoDB</a>: field names can be serialized
<em>without</em> double quotes, which reduces a lot the storage size, and the
transmission size, when the JSON is stored within a TEXT column, which would
escape all double quote characters.</p>
<p>I guess the performance of our implementation is probably faster than
current <em>SQlite3</em>'s extension.<br />
We use a SAX-like fast approach, without any temporary memory allocation during
its search, whereas AFAIK <em>SQlite3</em>'s extension is doing a lot of
memory allocations, creating all nodes of the JSON documents, like a DOM.</p>
<p>But here again, we are not exclusive guys. If you like this upcoming JSON
extension, you are free to use it! No offense taken!</p>
<p>Feedback <a href="http://synopse.info/forum/viewtopic.php?pid=17448#p17448">is welcome on our
forum</a>, as usual.</p>Introducing mORMot's architecture and design principlesurn:md5:265b0a5927ad96f526d355c5c9aca5322014-04-18T11:40:00+02:002015-01-04T10:10:34+01:00AB4327-GANDImORMot Frameworkcode-firstDatabasedatabase-firstDelphiDependencyInjectionDocumentationDomainDrivenDTOdynamic arrayEventSourcingfactoryFireDACGoodPracticehttp.sysHttpApiHTTPSinterfaceJSONLateBindingmockModelmORMotMSSQLMySQLNextGenNexusDBNoSQLODBCOleDBOpenSourceOracleORMperformancePostgreSQLRepositoryRTTIsessionshardingSOASQLSQLite3SynDBSynopseTDataSetTDocVariantTDynArraytransactionUnicodeUniDACValueObjectVirtualTableWinHTTPWinINetZEOS <p>We have just released a set of slides introducing </p>
<ul>
<li>ORM, SOA, REST, JSON, MVC, MVVM, SOLID, Mocks/Stubs, Domain-Driven Design
concepts with Delphi, </li>
<li>and showing some sample code using our Open Source <em>mORMot</em>
framework.</li>
</ul>
<p>You can follow the <a href="https://drive.google.com/folderview?id=0B0r8u-FwvxWdeVJVZnBhSEpKYkE&usp=sharing">
public link on Google Drive</a>!</p>
<p><img src="http://images.fineartamerica.com/images-medium-large/1-golden-marmot-maureen-ida-farley.jpg" width="450" height="291" alt="" /></p>
<p>This is a great opportunity to discovers some patterns you may not be
familiar with, and find out how <em>mORMot</em> try to implement them.<br />
This set of slides may be less intimidating than our huge documentation - do
not be terrified by our <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html">
Online Documentation</a>!<br />
The <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#SIDE_TITL_40">
first set of pages</a> (presenting architecture and design principles) is worth
reading.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1707">welcome on our forum, as
usual</a>.</p>Enhanced and fixed late-binding of variants for Delphi XE2 and upurn:md5:26711898589d8e5acb9aaac0aee8f5bb2014-03-29T16:45:00+01:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkblogDatabaseDelphiLateBindingmORMotOleAutomationOleDBORMperformanceSourcestringTDocVariantUnicodeXE2<p>For several units of our framework, we allow <em>late-binding</em> of data
values, using a <code>variant</code> and direct named access to
properties:<br />
- In <em>SynCommons</em>, we defined our <code>TDocVariant</code> custom
variant type, able to store any JSON/BSON document-based content;<br />
- In <em>SynBigTable</em>, we use the <code>TSynTableVariantType</code> custom
variant type, as defined in <em>SynCommons</em>;<br />
- In <em>SynDB</em>, we defined a <code>TSQLDBRowVariantType</code>, ready to
access any column of a RDBMS data result set row;<br />
- In <em>mORMot</em>, we allow access to <code>TSQLTableRowVariantType</code>
column values.</p>
<p>It's a very convenient way of accessing result rows values. Code is still
very readable, and safe at the same time.</p>
<p>For instance, we can write:</p>
<pre>
<strong>var</strong> V: <strong>variant</strong>;
...
TDocVariant.New(V); <em>// or slightly slower V := TDocVariant.New;</em>
V.name := 'John';
V.year := 1972;
<em>// now V contains {"name":"john","year":1982}</em>
</pre>
<p>This is just another implementation of KISS design in our framework.</p>
<p><img src="http://www.healthcatalyst.com/wp-content/uploads/2013/08/iStock_000004097533_ExtraSmall.jpg" alt="" /></p>
<p>Since Delphi XE2, some modifications were introduced to the official
<code>DispInvoke()</code> RTL implementation:</p>
<ol>
<li>A new <code>varUStrArg</code> kind of parameter has been defined, which
will allow to transmit <code>UnicodeString</code> property values;</li>
<li>All text property values would be transmitted as BSTR <code>/ WideString /
varOleStr</code> variants to the invoked variant type;</li>
<li>All textual property names were normalized to be in UPPERCASE.</li>
</ol>
<p>Those modifications are worth considering...<br />
And we may have discovered two regressions: one about speed, and the other
about an unexpected logic bug...</p> <h2>The issues</h2>
<p>The first modification does make sense, and was indeed a welcome fix for an
Unicode version of Delphi. It should have been as such since Delphi 2009.</p>
<p>Temporary conversion to <code>WideString</code> does make sense in the COM /
OLE world, but is an awfull performance bottleneck in the pure Delphi realm,
i.e. when using late-binding with custom type of variants (as for all our
custom variant types). This may be a noticeable speed penalty, in comparison to
previous versions of the compiler.</p>
<p>Last but not least, the conversion to uppercase is a bug. For instance, the
following code won't work as expected since Delphi XE2:</p>
<pre>
<strong>var</strong> V: <strong>variant</strong>;
...
TDocVariant.New(V); <em>// or slightly slower V := TDocVariant.New;</em>
V.name := 'John';
V.year := 1972;
<em>// before Delphi XE2, V contains {"name":"john","year":1982} - as expected</em>
<em>// since Delphi XE2, V contains {"NAME":"john","YEAR":1982} - sounds like a bug, doesn't it?</em>
</pre>
<p>This sounds indeed like an awfull regression.</p>
<h2>Fix included in the <em>mORMot</em> framework</h2>
<p>Since revision 1.18 of the framework, the patch described in this <a href="https://blog.synopse.info?post/post/2011/07/01/Faster-variant-late-binding">previous blog article</a> has
been modified for Delphi XE2 and up, as such:</p>
<ul>
<li>It will handle <code>varUStrArg</code> kind of parameter as exepcted;</li>
<li>It will avoid any temporary conversion to <code>WideString</code> for
textual values;</li>
<li>It will by-pass the property name change into uppercase.</li>
</ul>
<p>As soon as you define <em>SynCommons</em> in any of your program's uses
class, <a href="https://blog.synopse.info?post/post/2011/07/01/Faster-variant-late-binding">our hooked
<code>DispInvoke()</code> will take place</a>, and identify any of our
<code>TSynInvokeableVariantType</code> classes.</p>
<p>As a result, it will by-pass the performance bottleneck of the default RTL
implementation, and also fix the uppercase conversion of the property name.</p>
<p>Of course, if this variant is not a <code>TSynInvokeableVariantType</code>
instance (e.g. any <em>Ole Automation</em> call), the regular
<code>TInvokeableVariantType.DispInvoke()</code> method as defined in
<code>Variants.pas</code> will be executed, to maintain the best compatibility
possible.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1680">welcome in our forum</a>, as
usual!</p>TDocVariant custom variant typeurn:md5:7d8dec0eaab0b811075d6a84ed6fb60b2014-02-25T18:41:00+01:002014-02-25T18:41:00+01:00AB4327-GANDImORMot FrameworkAJAXblogDelphiDocumentationDomainDrivenDTOdynamic arrayIDEJSONLateBindingmORMotobjectORMParsingperformanceRADrecordshardingSOAstringTDocVariantTDynArray<p>With revision 1.18 of the framework, we just introduced two new custom types
of <code>variant</code>s:</p>
<ul>
<li><code>TDocVariant</code> kind of <code>variant</code>;</li>
<li><code>TBSONVariant</code> kind of <code>variant</code>.</li>
</ul>
<p>The second custom type (which handles <em>MongoDB</em>-specific extensions -
like <code>ObjectID</code> or other specific types like dates or binary) will
be presented later, when dealing with <em>MongoDB</em> support in
<em>mORMot</em>, together with the BSON kind of content. BSON /
<em>MongoDB</em> support is implemented in the <code>SynMongoDB.pas</code>
unit.</p>
<p>We will now focus on <code>TDocVariant</code> itself, which is a generic
container of JSON-like objects or arrays.<br />
This custom variant type is implemented in <code>SynCommons.pas</code> unit, so
is ready to be used everywhere in your code, even without any link to the
<em>mORMot</em> ORM kernel, or <em>MongoDB</em>.</p>
<p><img src="http://www.kumc.edu/Images/information%20resources/document-management-software.jpg" alt="" /></p>
<h4>TDocVariant documents</h4>
<p><code>TDocVariant</code> implements a custom variant type which can be used
to store any JSON/BSON document-based content, i.e. either:</p>
<ul>
<li>Name/value pairs, for object-oriented documents;</li>
<li>An array of values (including nested documents), for array-oriented
documents;</li>
<li>Any combination of the two, by nesting <code>TDocVariant</code>
instances.</li>
</ul>
<p>Here are the main features of this custom variant type:</p>
<ul>
<li>DOM approach of any <em>object</em> or <em>array</em> documents;</li>
<li>Perfect storage for dynamic value-objects content, with a
<em>schema-less</em> approach (as you may be used to in scripting languages
like Python or JavaScript);</li>
<li>Allow nested documents, with no depth limitation but the available
memory;</li>
<li>Assignment can be either <em>per-value</em> (default, safest but slower
when containing a lot of nested data), or <em>per-reference</em> (immediate
reference-counted assignment);</li>
<li>Very fast JSON serialization / un-serialization with support of
<em>MongoDB</em>-like extended syntax;</li>
<li>Access to properties in code, via late-binding (including almost no speed
penalty due to our VCL hack as <a href="https://blog.synopse.info?post/post/2011/07/01/Faster-variant-late-binding">already detailed</a>);</li>
<li>Direct access to the internal variant <em>names</em> and <em>values</em>
arrays from code, by trans-typing into a <code>TDocVariantData
record</code>;</li>
<li>Instance life-time is managed by the compiler (like any other
<code>variant</code> type), without the need to use <code>interfaces</code> or
explicit <code>try..finally</code> blocks;</li>
<li>Optimized to use as little memory and CPU resource as possible (in contrast
to most other libraries, it does not allocate one <code>class</code> instance
per node, but rely on pre-allocated arrays);</li>
<li>Opened to extension of any content storage - for instance, it will
perfectly integrate with BSON serialization and custom <em>MongoDB</em> types
(<em>ObjectID, RegEx</em>...), to be used in conjunction with <em>MongoDB</em>
servers;</li>
<li>Perfectly integrated with our <a href="https://blog.synopse.info?post/post/2011/03/12/TDynArray-and-Record-compare/load/save-using-fast-RTTI">Dynamic
array wrapper</a> and its JSON serialization as with the <a href="https://blog.synopse.info?post/post/2013/12/10/JSON-record-serialization"><code>record</code>
serialization</a>;</li>
<li>Designed to work with our <em>mORMot</em> ORM: any <code>TSQLRecord</code>
instance containing such <code>variant</code> custom types as published
properties will be recognized by the ORM core, and work as expected with any
database back-end (storing the content as JSON in a TEXT column);</li>
<li>Designed to work with our <em>mORMot</em> SOA: any <a href="https://blog.synopse.info?post/post/2012/03/07/Interface-based-services"><code>interface</code>-based
service</a> is able to consume or publish such kind of content, as
<code>variant</code> kind of parameters;</li>
<li>Fully integrated with the Delphi IDE: any <code>variant</code> instance
will be displayed as JSON in the IDE debugger, making it very convenient to
work with.</li>
</ul>
<p>To create instances of such <code>variant</code>, you can use some
easy-to-remember functions:</p>
<ul>
<li><code>_Obj() _ObjFast()</code> global functions to create a
<code>variant</code> <em>object</em> document;</li>
<li><code>_Arr() _ArrFast()</code> global functions to create a
<code>variant</code> <em>array</em> document;</li>
<li><code>_Json() _JsonFast() _JsonFmt() _JsonFastFmt()</code> global functions
to create any <code>variant</code> <em>object</em> or <em>array</em> document
from JSON, supplied either with standard or <em>MongoDB</em>-extended
syntax.</li>
</ul> <h3>Variant object documents</h3>
<p>With <code>_Obj()</code>, an <em>object</em> <code>variant</code> instance
will be initialized with data supplied two by two, as <em>Name,Value</em>
pairs, e.g.</p>
<pre>
<strong>var</strong> V1,V2: <strong>variant</strong>; <em>// stored as any variant</em>
...
V1 := _Obj(['name','John','year',1972]);
V2 := _Obj(['name','John','doc',_Obj(['one',1,'two',2.5])]); <em>// with nested objects</em>
</pre>
<p>Then you can convert those objects into JSON, by two means:</p>
<ul>
<li>Using the <code>VariantSaveJson()</code> function, which return directly
one UTF-8 content;</li>
<li>Or by trans-typing the <code>variant</code> instance into a string (this
will be slower, but is possible).</li>
</ul>
<pre>
writeln(VariantSaveJson(V1)); <em>// explicit conversion into RawUTF8</em>
writeln(V1); <em>// implicit conversion from variant into string</em>
<em>// both commands will write '{"name":"john","year":1982}'</em>
writeln(VariantSaveJson(V2)); <em>// explicit conversion into RawUTF8</em>
writeln(V2); <em>// implicit conversion from variant into string</em>
<em>// both commands will write '{"name":"john","doc":{"one":1,"two":2.5}}'</em>
</pre>
<p>As a consequence, the Delphi IDE debugger is able to display such variant
values as their JSON representation.<br />
That is, <code>V1</code> will be displayed as
<code>'"name":"john","year":1982'</code> in the IDE debugger <em>Watch
List</em> window, or in the <em>Evaluate/Modify</em> (F7) expression
tool.<br />
This is pretty convenient, and much more user friendly than any class-based
solution (which requires the installation of a specific design-time package in
the IDE).</p>
<p>You can access to the object properties via late-binding, with any depth of
nesting objects, in your code:</p>
<pre>
writeln('name=',V1.name,' year=',V1.year);
<em>// will write 'name=John year=1972'</em>
writeln('name=',V2.name,' doc.one=',V2.doc.one,' doc.two=',doc.two);
<em>// will write 'name=John doc.one=1 doc.two=2.5</em>
V1.name := 'Mark'; <em>// overwrite a property value</em>
writeln(V1.name); <em>// will write 'Mark'</em>
V1.age := 12; <em>// add a property to the object</em>
writeln(V1.age); <em>// will write '12'</em>
</pre>
<p>Note that the property names will be evaluated at runtime only, not at
compile time.<br />
For instance, if you write <code>V1.nome</code> instead of
<code>V1.name</code>, there will be no error at compilation, but an
<code>EDocVariant</code> exception will be raised at execution (unless you set
the <code>dvoReturnNullForUnknownProperty</code> option to
<code>_Obj/_Arr/_Json/_JsonFmt</code> which will return a <code>null</code>
variant for such undefined properties).</p>
<p>In addition to the property names, some pseudo-methods are available for
such <em>object</em> <code>variant</code> instances:</p>
<pre>
writeln(V1._Count); <em>// will write 3 i.e. the number of name/value pairs in the object document</em>
writeln(V1._Kind); <em>// will write 1 i.e. ord(sdkObject)</em>
<strong>for</strong> i := 0 <strong>to</strong> V2._Count-1 <strong>do</strong>
writeln(V2.Name(i),'=',V2.Value(i));
<em>// will write in the console:</em>
<em>// name=John</em>
<em>// doc={"one":1,"two":2.5}</em>
<em>// age=12</em>
<strong>if</strong> V1.Exists('year') <strong>then</strong>
writeln(V1.year);
</pre>
<p>You may also trans-type your <code>variant</code> instance into a
<code>TDocVariantData record</code>, and access directly to its
internals.<br />
For instance:</p>
<pre>
TDocVariantData(V1).AddValue('comment','Nice guy');
<strong>with</strong> TDocVariantData(V1) <strong>do</strong> <em>// direct transtyping</em>
<strong>if</strong> Kind=sdkObject <strong>then</strong> <em>// direct access to the TDocVariantDataKind field</em>
<strong>for</strong> i := 0 <strong>to</strong> Count-1 <strong>do</strong> <em>// direct access to the Count: integer field</em>
writeln(Names[i],'=',Values[i]); <em>// direct access to the internal storage arrays</em>
</pre>
<p>By definition, trans-typing via a <code>TDocVariantData record</code> is
slightly faster than using late-binding.<br />
But you must ensure that the <code>variant</code> instance is really a
<code>TDocVariant</code> kind of data before transtyping e.g. by calling
<code>DocVariantType.IsOfType(aVariant)</code>.</p>
<h3>Variant array documents</h3>
<p>With <code>_Arr()</code>, an <em>array</em> <code>variant</code> instance
will be initialized with data supplied as a list of <em>Value1,Value2,...</em>,
e.g.</p>
<pre>
<strong>var</strong> V1,V2: <strong>variant</strong>; <em>// stored as any variant</em>
...
V1 := _Arr(['John','Mark','Luke']);
V2 := _Obj(['name','John','array',_Arr(['one','two',2.5])]); <em>// as nested array</em>
</pre>
<p>Then you can convert those objects into JSON, by two means:</p>
<ul>
<li>Using the <code>VariantSaveJson()</code> function, which return directly
one UTF-8 content;</li>
<li>Or by trans-typing the <code>variant</code> instance into a string (this
will be slower, but is possible).</li>
</ul>
<pre>
writeln(VariantSaveJson(V1));
writeln(V1); <em>// implicit conversion from variant into string</em>
<em>// both commands will write '["John","Mark","Luke"]'</em>
writeln(VariantSaveJson(V2));
writeln(V2); <em>// implicit conversion from variant into string</em>
<em>// both commands will write '{"name":"john","array":["one","two",2.5]}'</em>
</pre>
<p>As a with any <em>object</em> document, the Delphi IDE debugger is able to
display such <em>array</em> <code>variant</code> values as their JSON
representation.</p>
<p>Late-binding is also available, with a special set of pseudo-methods:</p>
<pre>
writeln(V1._Count); <em>// will write 3 i.e. the number of items in the array document</em>
writeln(V1._Kind); <em>// will write 2 i.e. ord(sdkArray)</em>
<strong>for</strong> i := 0 <strong>to</strong> V1._Count-1 <strong>do</strong>
writeln(V1.Value(i),':',V2._(i));
<em>// will write in the console:</em>
<em>// John John</em>
<em>// Mark Mark</em>
<em>// Luke Luke</em>
<strong>if</strong> V1.Exists('John') <strong>then</strong>
writeln('John found in array');
</pre>
<p>Of course, trans-typing into a <code>TDocVariantData record</code> is
possible, and will be slightly faster than using late-binding.</p>
<h3>Create variant object or array documents from JSON</h3>
<p>With <code>_Json()</code> or <code>_JsonFmt()</code>, either a
<em>document</em> or <em>array</em> <code>variant</code> instance will be
initialized with data supplied as JSON, e.g.</p>
<pre>
<strong>var</strong> V1,V2,V3,V4: <strong>variant</strong>; <em>// stored as any variant</em>
...
V1 := _Json('{"name":"john","year":1982}'); <em>// strict JSON syntax</em>
V2 := _Json('{name:"john",year:1982}'); <em>// with MongoDB extended syntax for names</em>
V3 := _Json('{"name":?,"year":?}',[],['john',1982]);
V4 := _JsonFmt('{%:?,%:?}',['name','year'],['john',1982]);
writeln(VariantSaveJSON(V1));
writeln(VariantSaveJSON(V2));
writeln(VariantSaveJSON(V3));
<em>// all commands will write '{"name":"john","year":1982}'</em>
</pre>
<p>Of course, you can nest objects or arrays as parameters to the
<code>_JsonFmt()</code> function.</p>
<p>The supplied JSON can be either in strict JSON syntax, or with the
<em>MongoDB</em> extended syntax, i.e. with unquoted property names.<br />
It could be pretty convenient and also less error-prone when typing in the
Delphi code to forget about quotes around the property names of your JSON.</p>
<p>Note that <em>TDocVariant</em> implements an open interface for adding any
custom extensions to JSON: for instance, if the <code>SynMongoDB.pas</code>
unit is defined in your application, you will be able to create any MongoDB
specific types in your JSON, like <code>ObjectID()</code>, <code>new
Date()</code> or even <code>/regex/option</code>.</p>
<p>As a with any <em>object</em> or <em>array</em> document, the Delphi IDE
debugger is able to display such <code>variant</code> values as their JSON
representation.</p>
<h3>Per-value or per-reference</h3>
<p>By default, the <code>variant</code> instance created by <code>_Obj() _Arr()
_Json() _JsonFmt()</code> will use a <em>copy-by-value</em> pattern.<br />
It means that when an instance is affected to another variable, a new
<code>variant</code> document will be created, and all internal values will be
copied. Just like a <code>record</code> type.</p>
<p>This will imply that if you modify any item of the copied variable, it won't
change the original variable:</p>
<pre>
<strong>var</strong> V1,V2: <strong>variant</strong>;
...
V1 := _Obj(['name','John','year',1972]);
V2 := V1; <em>// create a new variant, and copy all values</em>
V2.name := 'James'; <em>// modifies V2.name, but not V1.name</em>
writeln(V1.name,' and ',V2.name);
<em>// will write 'John and James'</em>
</pre>
<p>As a result, your code will be perfectly safe to work with, since
<code>V1</code> and <code>V2</code> will be uncoupled.</p>
<p>But one drawback is that passing such a value may be pretty slow, for
instance, when you nest objects:</p>
<pre>
<strong>var</strong> V1,V2: <strong>variant</strong>;
...
V1 := _Obj(['name','John','year',1972]);
V2 := _Arr(['John','Mark','Luke']);
V1.names := V2; <em>// here the whole V2 array will be re-allocated into V1.names</em>
</pre>
<p>Such a behavior could be pretty time and resource consuming, in case of a
huge document.</p>
<p>All <code>_Obj() _Arr() _Json() _JsonFmt()</code> functions have an optional
<code>TDocVariantOptions</code> parameter, which allows to change the behavior
of the created <code>TDocVariant</code> instance, especially setting
<code>dvoValueCopiedByReference</code>.</p>
<p>This particular option will set the <em>copy-by-reference</em> pattern:</p>
<pre>
<strong>var</strong> V1,V2: <strong>variant</strong>;
...
V1 := _Obj(['name','John','year',1972],[dvoValueCopiedByReference]);
V2 := V1; <em>// creates a reference to the V1 instance</em>
V2.name := 'James'; <em>// modifies V2.name, but also V1.name</em>
writeln(V1.name,' and ',V2.name);
<em>// will write 'James and James'</em>
</pre>
<p>You may think this behavior is somewhat weird for a <code>variant</code>
type. But if you forget about <em>per-value</em> objects and consider those
<code>TDocVariant</code> types as a Delphi <code>class</code> instance (which
is a <em>per-reference</em> type), without the need of having a fixed schema
nor handling manually the memory, it will probably start to make sense.</p>
<p>Note that a set of global functions have been defined, which allows direct
creation of documents with <em>per-reference</em> instance lifetime, named
<code>_ObjFast() _ArrFast() _JsonFast() _JsonFmtFast()</code>.<br />
Those are just wrappers around the corresponding <code>_Obj() _Arr() _Json()
_JsonFmt()</code> functions, with the following <code>JSON_OPTIONS[true]</code>
constant passed as options parameter:</p>
<pre>
<strong>const</strong>
<em>/// some convenient TDocVariant options</em>
<em>// - JSON_OPTIONS[false] is _Json() and _JsonFmt() functions default</em>
<em>// - JSON_OPTIONS[true] are used by _JsonFast() and _JsonFastFmt() functions</em>
JSON_OPTIONS: <strong>array</strong>[Boolean] <strong>of</strong> TDocVariantOptions = (
[dvoReturnNullForUnknownProperty],
<span style="background-color:yellow;">[dvoReturnNullForUnknownProperty,dvoValueCopiedByReference]);</span>
</pre>
<p>When working with complex documents, e.g. with BSON / <em>MongoDB</em>
documents, almost all content will be created in "fast" <em>per-reference</em>
mode.</p>
<h3>Advanced TDocVariant process</h3>
<h3>Object or array document creation options</h3>
<p>As stated above, a <code>TDocVariantOptions</code> parameter enables to
define the behavior of a <code>TDocVariant</code> custom type for a given
instance.<br />
Please refer to the documentation of this set of options to find out the
available settings. Some are related to the memory model, other to
case-sensitivity of the property names, other to the behavior expected in case
of non-existing property, and so on...</p>
<p>Note that this setting is <em>local</em> to the given <code>variant</code>
instance.</p>
<p>In fact, <code>TDocVariant</code> does not force you to stick to one memory
model nor a set of global options, but you can use the best pattern depending
on your exact process.<br />
You can even <em>mix</em> the options - i.e. including some objects as
properties in an object created with other options - but in this case, the
initial options of the nested object will remain. So you should better use this
feature with caution.</p>
<p>You can use the <code>_Unique()</code> global function to force a variant
instance to have an unique set of options, and all nested documents to become
<em>by-value</em>, or <code>_UniqueFast()</code> for all nested documents to
become <em>by-reference</em>.</p>
<pre>
<em>// assuming V1='{"name":"James","year":1972}' created by-reference</em>
_Unique(V1); <em>// change options of V1 to be by-value</em>
V2 := V1; <em>// creates a full copy of the V1 instance</em>
V2.name := 'John'; <em>// modifies V2.name, but not V1.name</em>
writeln(V1.name); <em>// write 'James'</em>
writeln(V2.name); <em>// write 'John'</em>
V1 := _Arr(['root',V2]); <em>// created as by-value by default, as V2 was</em>
writeln(V1._Count); <em>// write 2</em>
_UniqueFast(V1); <em>// change options of V1 to be by-reference</em>
V2 := V1;
V1._(1).name := 'Jim';
writeln(V1);
writeln(V2);
<em>// both commands will write '["root",{"name":"Jim","year":1972}]'</em>
</pre>
<p>The easiest is to stick to one set of options in your code, i.e.:</p>
<ul>
<li>Either using the <code>_*()</code> global functions if your business code
does send some <code>TDocVariant</code> instances to any other part of your
logic, for further storage: in this case, the <em>by-value</em> pattern does
make sense;</li>
<li>Or using the <code>_*Fast()</code> global functions if the
<code>TDocVariant</code> instances are local to a small part of your code, e.g.
used as schema-less <em>Data Transfer Objects</em> (<em>DTO</em>).</li>
</ul>
<p>In all cases, be aware that, like any <code>class</code> type, the
<code>const</code>, <code>var</code> and <code>out</code> specifiers of method
parameters does not behave to the <code>TDocVariant</code> value, but to its
reference.</p>
<h3>Integration with other mORMot units</h3>
<p>In fact, whenever a <em>schema-less</em> storage structure is needed, you
may use a <code>TDocVariant</code> instance instead of <code>class</code> or
<code>record</code> strong-typed types:</p>
<ul>
<li>Client-Server ORM will support <code>TDocVariant</code> in any of the
<code>TSQLRecord variant</code> published properties;</li>
<li>Interface-based services will support <code>TDocVariant</code> as
<code>variant</code> parameters of any method, which make them as perfect
<em>DTO</em>;</li>
<li>Since JSON support is implemented with any <code>TDocVariant</code> value
from the ground up, it makes a perfect fit for working with AJAX clients, in a
script-like approach;</li>
<li>If you use our <code>SynMongoDB.pas</code> unit to access a
<em>MongoDB</em> server, <code>TDocVariant</code> will be the native storage to
create or access BSON arrays or objects documents;</li>
<li>Cross-cutting features (like logging or <code>record</code> / <em>dynamic
array</em> enhancements) will also benefit from this <code>TDocVariant</code>
custom type.</li>
</ul>
<p>We are pretty convinced that when you will start playing with
<code>TDocVariant</code>, you won't be able to live without it any more.<br />
It introduces the full power of late-binding and schema-less patterns to your
application, which can be pretty useful for prototyping or in Agile
development.<br />
You do not need to use scripting engines like Python or JavaScript to have this
feature, if you need it.</p>
<p>Feedback and comments are <a href="http://synopse.info/forum/viewtopic.php?id=1631">welcome in our forum, as
usual</a>!</p>