Synopse Open Source - Tag - DatabasemORMot MVC / SOA / ORM and friends2024-02-02T17:08:25+00:00urn:md5:cc547126eb580a9adbec2349d7c65274DotclearModern Pascal is Still in the Raceurn:md5:af5e15cc3f4bde55561318a1bb5dd3bb2022-11-26T10:05:00+00:002022-11-29T13:21:57+00:00Arnaud BouchezPascal ProgrammingblogcollectionsCrossPlatformDatabaseDelphiFPCGarbageCollectorgenericsGoGoodPracticeMetaProgrammingmORMotmORMot2performanceRTTIRust<p>A recent poll <a href="https://forum.lazarus.freepascal.org/index.php/topic,61276.0.html">on the Lazarus/FPC forum</a> highlighted a fact: pascal coders are older than most coders. Usually, at our age, we should be managers, not developers. But we like coding in pascal. It is still fun after decades!<br />
But does it mean that you should not use pascal for any new project? Are the language/compilers/libraries outdated?<br />
In the company I currently work for, we have young coders, just out-of-school or still-in-school, which joined the team and write great code!</p>
<p><img src="https://blog.synopse.info?post/public/blog/performance.jpg" alt="" /></p>
<p>And a recent thread <a href="https://forum.lazarus.freepascal.org/index.php/topic,61035.0.html">in this very same forum</a> was about comparing languages to implement a REST server, in C#, Go, Scala, TypeScript, Elixir and Rust.<br />
Several pascal versions are about to be contributed, one in which <em>mORMot</em> shines.</p> <h4>The Challenge and the Algorithms</h4>
<p>The original challenge is available at <a href="https://github.com/losvedir/transit-lang-cmp">transit-lang-cmp</a> with the original source code, of all those fancy languages and libraries.</p>
<p>In practice, the goal of this test program is to load two big CSVs into memory (80MB + 2MB), then serve over HTTP some JSON generated by route identifiers, joining both CSVs.<br />
The resulting JSON could be of 30KB up to 2MB. And all data is generated on the fly from the CSV in memory.</p>
<p>To be fair, a regular/business coder would have used a database for this. Not silly memory structures. And asked for money to setup a huge set of cloud machines with load balancing. <img src="https://blog.synopse.info?pf=smile.svg" alt=":-)" class="smiley" /></p>
<h4>Reference Implementations in Today Languages</h4>
<p>The "modern" / "school" approach, as implemented in the reference project in Go/Rust/C#/... is using two lists for the CSVs data, then two maps/dictionaries between route ID and lists indexes.</p>
<ul>
<li>The <a href="https://github.com/losvedir/transit-lang-cmp/blob/main/trogsit/app.go">Golang version</a> has a good expressiveness, and is nice to read, even if you don't know the language.</li>
<li>The <a href="https://github.com/losvedir/transit-lang-cmp/tree/main/Trannet">C# version</a> is also readable, but making a webserver is still confusing because it is not built from code, but from config files.</li>
<li><a href="https://github.com/losvedir/transit-lang-cmp/tree/main/trexit">Elixir</a> is a bit over-complicated to my taste.</li>
<li><a href="https://github.com/losvedir/transit-lang-cmp/tree/main/trala">Scala</a> and <a href="https://github.com/losvedir/transit-lang-cmp/tree/main/trypsit">TypeScript/Deno</a> versions, are fine to read, but really slow. You may better use a database instead.</li>
<li>Just for fun, check <a href="https://github.com/losvedir/transit-lang-cmp/blob/main/trustit/src/main.rs">the Rust version</a> - do you think Rust is good for big maintainable projects with junior developers?</li>
</ul>
<p>There was a first attempt to write a FPC version of it, by Leledumbo.<br />
His <a href="https://github.com/leledumbo/transit-lang-cmp/blob/main/trascal/app.pas">Source Code repository</a> is a nice pascal conversion of above code. But performance was disappointing. Especially because the standard JSON library can not work directly with high level structures like collections or arrays.</p>
<p>So is Pascal out of the race?<br />
Let's call the <em>mORMot</em> to the rescue!</p>
<h4>Following the mORMot Way</h4>
<p>For the <em>mORMot</em> version in FPC, I used another approach, with two diverse algorithms:</p>
<ul>
<li>I ensured the lists were sorted in memory, then made a O(log(n)) binary lookup in it;</li>
<li>All stored strings were "interned", i.e. the same text was sharing a single string instance, and FPC reference counting did its magic.</li>
</ul>
<p>There is no low-level tricks like generating the JSON by hand or using complex data structures - data structures are still are high-level, with readable field names and such. The logic and the intent are clearly readable.<br />
We just leveraged the pascal language, and <em>mORMot</em> features. For instance, string interning is part of the framework, if needed.</p>
<p>Please <a href="https://github.com/synopse/mORMot2/tree/master/ex/lang-cmp/LangCmp.dpr">check the source code in our repository</a>.</p>
<p>As a result:</p>
<ul>
<li>Code is still readable, short and efficient (most of the process is done by <em>mORMot</em>, i.e. CSV, searching, JSON);</li>
<li>It uses much less memory - 10 times less memory than Go when holding the data, 5 times less memory than Go when serving the data;</li>
<li>Performance is as fast as Go, and its very tuned/optimized compiler and RTL.</li>
</ul>
<p><img src="https://blog.synopse.info?post/public/blog/mORMot2-small.png" alt="" /></p>
<h4>Algorithms Matters</h4>
<p>Main idea was to let the algorithms match the input data and the expected resultset.<br />
As programmers do when programming games. Not as coders do when pissing out business software. <img src="https://blog.synopse.info?pf=wink.svg" alt=";-)" class="smiley" /></p>
<ul>
<li>The source code is still pretty readable, thanks to using <em>mORMot</em> efficient <code>TDynArray</code> to map the dynamic array storage, and its CSV and JSON abilities.</li>
<li>I guess source is still understandable for out-of-school programmers - much more readable than Rust for instance.</li>
</ul>
<p>To by fair, I used typed pointers in <code>TScheduler.BuildTripResponse</code> but it is not so hard getting their purpose, and FPC compiles this function into very efficient assembly. I could have used regular dynamic array access with indexes, it would have been slightly slower, but not really easier to follow, nor safer (if we compile with no range checking).</p>
<p>Worth noting that we did not make any specific tuning, like pre-allocating the results with constants, as other frameworks did. We just specified the data, then let <em>mORMot</em> play with it - that's all.<br />
The <em>mORMot</em> RTTI level matches what we expect for modern frameworks: not only some classes to store JSON, but convenient serialization/unserialization using structures like class or record.<br />
Using modern Pascal dynamic arrays and records to define the data structures let the compiler leverage the memory for us, with no need to write any <code>try..finally..Free</code> blocks, and use interfaces. "Manual memory management" with Pascal is not mandatory and can easily be bypassed. Only for the WebServer, we have a <code>Free</code>, which is expected to close it.</p>
<h4>Give Me Some Numbers</h4>
<p>Here are a performance comparison with Go (FPC on the left, Go on the right):</p>
<pre>
parsed 1790905 stop times in 968.43ms | parsed 1790905 stop times in 3.245251432s
parsed 71091 trips in 39.54ms | parsed 71091 trips in 85.747852ms
running (0m33.4s), 00/50 VUs, 348 complete and 0 interrupted | running (0m32.3s), 00/50 VUs, 320 complete and 0 interrupted
default ✓ [======================================] 50 VUs 30 default ✓ [======================================] 50 VUs 30
data_received..................: 31 GB 933 MB/s | data_received..................: 31 GB 971 MB/s
data_sent......................: 3.2 MB 97 kB/s | data_sent......................: 3.0 MB 92 kB/s
http_req_blocked...............: avg=9µs min=1.09µs | http_req_blocked...............: avg=6.77µs min=1.09µs
http_req_connecting............: avg=2.95µs min=0s | http_req_connecting............: avg=1.73µs min=0s
http_req_duration..............: avg=47.59ms min=97.28µs | http_req_duration..............: avg=49.02ms min=123.81µ
{ expected_response:true }...: avg=47.59ms min=97.28µs | { expected_response:true }...: avg=49.02ms min=123.81µ
http_req_failed................: 0.00% ✓ 0 ✗ | http_req_failed................: 0.00% ✓ 0 ✗ 3
http_req_receiving.............: avg=9.66ms min=15.35µs | http_req_receiving.............: avg=5.92ms min=14.76µs
http_req_sending...............: avg=87.24µs min=5.2µs | http_req_sending...............: avg=70.71µs min=5.2µs
http_req_tls_handshaking.......: avg=0s min=0s | http_req_tls_handshaking.......: avg=0s min=0s
http_req_waiting...............: avg=37.83ms min=54.74µs | http_req_waiting...............: avg=43.02ms min=91.84µs
http_reqs......................: 34452 1032.205528/s | http_reqs......................: 31680 981.949476/s
iteration_duration.............: avg=4.72s min=3.54s | iteration_duration.............: avg=4.86s min=2.19s
iterations.....................: 348 10.426318/s | iterations.....................: 320 9.918682/s
vus............................: 30 min=30 ma | vus............................: 15 min=15 max
vus_max........................: 50 min=50 ma | vus_max........................: 50 min=50 max
</pre>
<p>So CSV loading was much faster, then the HTTP server performance was almost the same.</p>
<h4>No Alzheimer</h4>
<p>Here are some numbers about memory consumption:</p>
<blockquote><p>Upon finished loading the CSV, mORMot only eats 80MB, heck so little. Sounds a bit magical. But during load test, it fluctuates between 250-350MB, upon which it returns to 80MB at the end.
The Go version eats 925MB upon finished loading the CSV. During load test, it tops at 1.5GB, returning to 925MB afterwards.</p></blockquote>
<p>Nice to read. :)</p>
<h4>Pascal has a Modern and Capable Ecosystem</h4>
<p>This article was not only about Pascal, but about algorithms and libraries.<br />
The challenge was initially about comparing them. Not only as unrealistic micro-benchmarks, or "computer language benchmark games", but as data processing abilities on a real usecase.</p>
<p><strong>And... Pascal is still in the race for sure!</strong><br />
Not only for "old" people like me - I just got 50 years old. ;-)</p>
<p>The more we spread such kind of information, the less people would make jokes about pascal programmers.<br />
Delphi and FPC are as old as Java, so it is time to get the big picture, not following marketing trends.</p>New Client for MongoDB 5.1/6 Supporturn:md5:41f28034cfacffa2bf7fde35e8e64b432022-08-12T20:12:00+01:002022-08-12T20:12:00+01:00Arnaud BouchezmORMot FrameworkCrossPlatformDatabaseDelphiFPCFreePascalJSONMongoDBmORMot2NoSQLODMORMperformanceRESTSource<p>Starting with its version 5.1, <em>MongoDB</em> disabled the legacy protocol used for communication since its beginning.<br />
As a consequence, our <em>mORMot</em> client was not able to communicate any more with the latest versions of <em>MongoDB</em> instances.</p>
<p><img src="https://blog.synopse.info?post/public/mongodb.png" alt="" /></p>
<p>Last week, we made a deep rewrite of <a href="https://github.com/synopse/mORMot2/blob/master/src/db/mormot.db.nosql.mongodb.pas">mormot.db.nosql.mongodb.pas</a>, which changed the default protocol to use the new layout on the wire. Now messages use regular <a href="https://www.mongodb.com/docs/current/reference/command/">MongoDB Database Commands</a>, with automated compression if needed.</p>
<p>No change is needed in your end-user <em>MongoDB</em> or ORM/ODM code. The upgrade is as simple as update your <em>mORMot 2</em> source, then recompile.</p> <h4>The Mongo Wire Protocol</h4>
<p>Since its beginning, <em>MongoDB</em> used a simple protocol over TCP, via several binary opcodes and message, for CRUD operations.</p>
<p>A new alternative protocol was <a href="https://emptysqua.re/blog/driver-features-for-mongodb-3-6/">introduced in version 3.6,</a> and the former protocol was marked as deprecated.<br />
Two new opcodes were introduced, OP_MSG and OP_COMPRESSED, to replace all other frames. They just encapsulate, with or without compression, some abstract BSON content.<br />
The official documentation details those changes <a href="https://www.mongodb.com/docs/manual/reference/mongodb-wire-protocol/">in this web page</a>.</p>
<p>In short (picture extracted from the blog above), the protocol came from this:</p>
<p><img src="https://blog.synopse.info?post/public/ye-olde-wire-protocol.png" alt="" /></p>
<p>to this:</p>
<p><img src="https://blog.synopse.info?post/public/op-msg.png" alt="" /></p>
<p>The main benefit is that the commands and answers are just conventional BSON, so the protocol can change at logical/BSON/JSON level by adding or changing some members, with no need of dealing with low-level binary structures.</p>
<p>With the version 5.1 of <em>MongoDB</em>, the previous protocol was not just deprecated, but disabled.<br />
So we had to update the <em>mORMot 2</em> client code! (yes, the <em>mORMot 1</em> code has not been updated - it may become a good reason to upgrade)</p>
<h4>Deep Rewrite</h4>
<p>In fact, the official MongoDB documentation is somewhat vague. And the official drivers are a bit difficult to reverse-engineer, due to the verbose nature of C, Java or C#. The native/node driver was easiest to dissect, and we used it as reference.<br />
Luckily enough, there are some <a href="https://github.com/mongodb/specifications/blob/master/source/message/OP_MSG.rst">specification document available too</a>, which offers some additional valuable clarifications.</p>
<p>After some testing, we managed to replace all previous OP_QUERY and its brothers to the new OP_MSG frame, which is, as documented in the specification, "One opcode to rule them all". <img src="https://blog.synopse.info?pf=wink.svg" alt=";)" class="smiley" /></p>
<p>Once we had the commands working, we needed to rewrite all CRUD operations using commands, and not opcodes.<br />
Queries are now made with <code><a href="https://www.mongodb.com/docs/current/reference/command/find/">find</a></code> and <code><a href="https://www.mongodb.com/docs/current/reference/command/aggregate/">aggregate</a></code> commands. Their results are now located in a <code>"cursor": firstBatch": ..</code> BSON array within the response. And a new <code><a href="https://www.mongodb.com/docs/current/reference/command/getMore/">getMore</a></code> command is to be used to retrieve the next values within a <code>"cursor": nextBatch": ...</code> resultset.<br />
For writing, <code><a href="https://www.mongodb.com/docs/current/reference/command/insert">insert</a></code>, <code><a href="https://www.mongodb.com/docs/current/reference/command/update">update</a></code> and <code><a href="https://www.mongodb.com/docs/current/reference/command/delete">delete</a></code> commands are called, with their appropriate BSON content.</p>
<p>During the refactoring, we optimized the BSON process, and also enhanced the whole process, mainly the logs and the execution efficiency. The <em>mORMot</em> client side should not be a bottleneck. And it is not, even with this NoSQL database.</p>
<p>Don't expect any performance enhancement, or new features. It is just some low-level protocol change at TCP level.<br />
But if you used the "non acknowledged write mode" of the former protocol, which was unsafe but very fast, you will have lower performance with the new protocol, because the new protocol always acknowledges the commands it receives. So, in some very specific configurations, the new protocol may reduce the performance.</p>
<h4>Backward Compatibility</h4>
<p>All those changes were encapsulated in our revised <a href="https://github.com/synopse/mORMot2/blob/master/src/db/mormot.db.nosql.mongodb.pas">mormot.db.nosql.mongodb.pas</a> unit.</p>
<p>If you have a very old <em>MongoDB</em> instance, and don't want to upgrade, you could just compile your project with the <code>MONGO_OLDPROTOCOL</code> conditional, to use the deprecated opcodes.<br />
If the <em>MongoDB</em> team does not care much with backward compatibility (they could have kept the previous protocol for sure, they still maintain it for the handshake message if needed), we do care about not breaking too much things with <em>mORMot</em>, so we kept the previous code, and tested/validated it too, for legacy systems.</p>
<h4>New Sample</h4>
<p>We translated and introduced the <em>MongoDB</em> benchmark sample to <em>mORMot 2</em> code base.</p>
<p>You could find it, and run it, from <a href="https://github.com/synopse/mORMot2/tree/master/ex/mongodb">our source code repository</a>.</p>
<p>This code is a good entry point for what is possible with this unit in our framework, for both direct access or ORM/ODM access.<br />
And you would be able to guess the performance numbers you may achieve with your project.</p>
<p>Running a <em>MongoDB</em> database in a container is as easy as executing the following command:</p>
<pre>
sudo docker run --name mongodb -d -p 27017:27017 mongo:latest
</pre>
<p>Then you will have a <em>MongoDB</em> server instance accessible on <code>localhost:27017</code>, so you could run the sample straight away.</p>
<h4>Delphi/FPC Open Source Rocks</h4>
<p>We hope you will find the change painless and transparent. We did not modify the high-level client methods, nor break the ORM/ODM: you can still write some SELECT complex statements, and our ORM will translate it into <em>MongoDB</em> aggregate commands.</p>
<p>To my knowledge, there is <a href="https://github.com/stijnsanders/TMongoWire/commit/7f12a64f571e476704bdcb737e1fc087ef792f59">only a single other Delphi/FPC client library</a> which made the upgrade to the new protocol, at today. Once we made our own changes, we notified other library authors, and Stijn made very quickly the needed changes. Congrats! Maybe our code could be used as reference for other library maintainers, because the protocol needs some small tweaks sometimes.<br />
It is important to have some maintenance on the library you use. And our little <em>mORMot</em> is still on the edge: thanks to FPC, it runs very well on Linux and BSD, which makes it perfect for professional services running in the long term! :)</p>
<p>Your feedback is welcome <a href="https://synopse.info/forum/viewtopic.php?id=6318">in the forum thread which initiated these modifications</a>, as usual!<br />
Don't hesitate to notify us any missing or broken feature.<br />
Thanks Daniel for your report and support!</p>mORMot 2 ORM Performanceurn:md5:332da916ca3f905cedf4c82d14fc9b872022-02-15T13:24:00+00:002022-02-15T13:24:00+00:00Arnaud BouchezmORMot Framework64bitAES-CTRasmblogDatabaseDelphiFPCfpcx64mmFreePascalJSONMicroservicesmORMotmORMot2ORMperformanceSQLSQLite3<p>The official release of <em>mORMot 2</em> is around the edge.
It may be the occasion to show some data persistence performance numbers, in respect to <em>mORMot 1</em>.</p>
<p><img src="https://blog.synopse.info?post/public/blog/marmotrunningsnow.jpg" alt="" /></p>
<p>For the version 2 of our framework, its ORM feature has been enhanced and tuned in several aspects: REST routing optimization, ORM/JSON serialization, and in-memory and SQL engines tuning.
Numbers are talking. You could compare with any other solution, and compile and run the tests by yourself for both framework, and see how it goes on your own computer or server.<br />
In a nutshell, we <em>almost reach 1 million inserts per second on SQLite3</em>, and are above the million inserts in our in-memory engine. Reading speed is 1.2 million and 1.7 million respectively. From the object to the storage, and back. And forcing AES-CTR encryption on disk almost don't change anything. Now we are talking. <img src="https://blog.synopse.info?pf=wink.svg" alt=";)" class="smiley" /></p> <h3>Platform Used</h3>
<p>Those numbers were taken from the "external database" sample, which is available on both versions of the framework.<br />
This is the very same benchmark as used in previous benchmarks on this blog or our documentation. So you could compare the numbers.</p>
<p>But we run the tests on a new computer, featuring a Intel(R) Core(TM) i5-7300U cpu, from a good old Thinkpad T470 notebook, with a SATA SSD.<br />
So on a more modern hardware, like a high-end AMD or Xeon server, the million inserts is easily passed. And on a slow VM, you will get pretty good numbers.</p>
<p>We run the tests on Linux x86_64 (Debian 11), compiled from FPC 3.2 stable - which is our target platform for performance.<br />
In fact, performance matters mainly on the server side - clients are usually fast enough to run whatever process they need, and the ORM/database/persistence process is likely to be located on the server side. So in <em>mORMot 2</em>, we focused on a x86_64 Linux server for performance, which is the cheapest and safest solution around.</p>
<p>Both frameworks used our <a href="https://github.com/synopse/mORMot2/blob/master/src/core/mormot.core.fpcx64mm.pas">in-memory heap manager for FPC</a>, written in x86_64 assembly. It has a noticeable performance benefit, especially on multi-thread process (not shown here, but during the main regression tests).</p>
<h3>Insertion Speed</h3>
<p>Here are the <em>mORMot 2</em> insertion numbers:</p>
<p>Running tests using Synopse mORMot framework 2.0.1, compiled with Free Pascal 3.2 64 bit, against SQLite 3.37.2, on Debian GNU/Linux 11 (bullseye) - Linux 5.10.0-10-amd64, at 2022-02-14 21:09:37.</p>
<p><table><tbody><tr align="center"><td> </td><td><strong>Direct</strong></td><td><strong>Batch</strong></td><td><strong>Trans</strong></td><td><strong>Batch Trans</strong></td></tr>
<tr align="center"><td><strong>Sqlite file full</strong></td><td>98</td><td>5908</td><td>74089</td><td>242072</td></tr>
<tr align="center"><td><strong>Sqlite file off</strong></td><td>13534</td><td>474428</td><td>151315</td><td>919624</td></tr>
<tr align="center"><td><strong>Sqlite file off exc</strong></td><td>42961</td><td>691037</td><td>153374</td><td>929281</td></tr>
<tr align="center"><td><strong>Sqlite file off exc aes</strong></td><td>26882</td><td>533788</td><td>152795</td><td>874814</td></tr>
<tr align="center"><td><strong>Sqlite in memory</strong></td><td>114664</td><td>969743</td><td>152190</td><td>972478</td></tr>
<tr align="center"><td><strong>In memory static</strong></td><td>411895</td><td>1086956</td><td>428724</td><td>1301236</td></tr>
<tr align="center"><td><strong>In memory virtual</strong></td><td>385445</td><td>1200480</td><td>412762</td><td>1219660</td></tr>
<tr align="center"><td><strong>External sqlite file full</strong></td><td>107</td><td>5957</td><td>83531</td><td>111043</td></tr>
<tr align="center"><td><strong>External sqlite file off</strong></td><td>16509</td><td>291528</td><td>151890</td><td>390502</td></tr>
<tr align="center"><td><strong>External sqlite file off exc</strong></td><td>58922</td><td>354924</td><td>150179</td><td>392649</td></tr>
<tr align="center"><td><strong>External sqlite in memory</strong></td><td>114476</td><td>991080</td><td>154564</td><td>991375</td></tr>
<tr align="center"><td><strong>Remote sqlite socket</strong></td><td>19789</td><td>155763</td><td>19439</td><td>236406</td></tr>
</tbody></table></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Batch+Trans|Trans|Batch|Direct&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,1301236&chds=0,1301236,0,1301236,0,1301236,0,1301236,0,1301236&chd=t:98,5908,74089,242072|13534,474428,151315,919624|42961,691037,153374,929281|26882,533788,152795,874814|114664,969743,152190,972478|411895,1086956,428724,1301236|385445,1200480,412762,1219660|107,5957,83531,111043|16509,291528,151890,390502|58922,354924,150179,392649|114476,991080,154564,991375|19789,155763,19439,236406&chdl=Sqlite+file+full|Sqlite+file+off|Sqlite+file+off+exc|Sqlite+file+off+exc+aes|Sqlite+in+memory|In+memory+static|In+memory+virtual|External+sqlite+file+full|External+sqlite+file+off|External+sqlite+file+off+exc|External+sqlite+in+memory|Remote+sqlite+socket" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Remote+sqlite+socket|External+sqlite+in+memory|External+sqlite+file+off+exc|External+sqlite+file+off|External+sqlite+file+full|In+memory+virtual|In+memory+static|Sqlite+in+memory|Sqlite+file+off+exc+aes|Sqlite+file+off+exc|Sqlite+file+off|Sqlite+file+full&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,1301236&chds=0,1301236,0,1301236,0,1301236,0,1301236,0,1301236,0,1301236,0,1301236,0,1301236,0,1301236,0,1301236,0,1301236,0,1301236&chd=t:98,13534,42961,26882,114664,411895,385445,107,16509,58922,114476,19789|5908,474428,691037,533788,969743,1086956,1200480,5957,291528,354924,991080,155763|74089,151315,153374,152795,152190,428724,412762,83531,151890,150179,154564,19439|242072,919624,929281,874814,972478,1301236,1219660,111043,390502,392649,991375,236406&chdl=Direct|Batch|Trans|Batch+Trans" /></p>
<p>In comparison, here are the <em>mORMot 1</em> performance - which was already ahead of most other solutions - on the same machine:</p>
<p>Running tests using Synopse mORMot framework 1.18.6365, compiled with Free Pascal 3.2 64 bit, against SQLite 3.37.2, on Debian GNU/Linux 11 (bullseye) - Linux 5.10.0-10-amd64, at 2022-02-15 10:27:28.</p>
<p><table><tbody><tr align="center"><td> </td><td><strong>Direct</strong></td><td><strong>Batch</strong></td><td><strong>Trans</strong></td><td><strong>Batch Trans</strong></td></tr>
<tr align="center"><td><strong>Sqlite file full</strong></td><td>97</td><td>5034</td><td>45603</td><td>90704</td></tr>
<tr align="center"><td><strong>Sqlite file off</strong></td><td>12504</td><td>212350</td><td>96605</td><td>272910</td></tr>
<tr align="center"><td><strong>Sqlite file off exc</strong></td><td>36189</td><td>244702</td><td>94754</td><td>273687</td></tr>
<tr align="center"><td><strong>Sqlite file off exc aes</strong></td><td>22933</td><td>219857</td><td>96513</td><td>267881</td></tr>
<tr align="center"><td><strong>Sqlite in memory</strong></td><td>79953</td><td>275269</td><td>97776</td><td>269963</td></tr>
<tr align="center"><td><strong>In memory static</strong></td><td>208125</td><td>473126</td><td>227821</td><td>505254</td></tr>
<tr align="center"><td><strong>In memory virtual</strong></td><td>202683</td><td>467595</td><td>220031</td><td>472545</td></tr>
<tr align="center"><td><strong>External sqlite file full</strong></td><td>100</td><td>2809</td><td>49603</td><td>102247</td></tr>
<tr align="center"><td><strong>External sqlite file off</strong></td><td>15329</td><td>190701</td><td>109346</td><td>301768</td></tr>
<tr align="center"><td><strong>External sqlite file off exc</strong></td><td>48818</td><td>264760</td><td>109767</td><td>304710</td></tr>
<tr align="center"><td><strong>External sqlite in memory</strong></td><td>93820</td><td>303766</td><td>111437</td><td>311779</td></tr>
<tr align="center"><td><strong>Remote sqlite socket</strong></td><td>17952</td><td>68360</td><td>14915</td><td>97096</td></tr>
</tbody></table></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Batch+Trans|Trans|Batch|Direct&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,505254&chds=0,505254,0,505254,0,505254,0,505254,0,505254&chd=t:97,5034,45603,90704|12504,212350,96605,272910|36189,244702,94754,273687|22933,219857,96513,267881|79953,275269,97776,269963|208125,473126,227821,505254|202683,467595,220031,472545|100,2809,49603,102247|15329,190701,109346,301768|48818,264760,109767,304710|93820,303766,111437,311779|17952,68360,14915,97096&chdl=Sqlite+file+full|Sqlite+file+off|Sqlite+file+off+exc|Sqlite+file+off+exc+aes|Sqlite+in+memory|In+memory+static|In+memory+virtual|External+sqlite+file+full|External+sqlite+file+off|External+sqlite+file+off+exc|External+sqlite+in+memory|Remote+sqlite+socket" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Remote+sqlite+socket|External+sqlite+in+memory|External+sqlite+file+off+exc|External+sqlite+file+off|External+sqlite+file+full|In+memory+virtual|In+memory+static|Sqlite+in+memory|Sqlite+file+off+exc+aes|Sqlite+file+off+exc|Sqlite+file+off|Sqlite+file+full&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,505254&chds=0,505254,0,505254,0,505254,0,505254,0,505254,0,505254,0,505254,0,505254,0,505254,0,505254,0,505254,0,505254&chd=t:97,12504,36189,22933,79953,208125,202683,100,15329,48818,93820,17952|5034,212350,244702,219857,275269,473126,467595,2809,190701,264760,303766,68360|45603,96605,94754,96513,97776,227821,220031,49603,109346,109767,111437,14915|90704,272910,273687,267881,269963,505254,472545,102247,301768,304710,311779,97096&chdl=Direct|Batch|Trans|Batch+Trans" /></p>
<p>As you can see, the performance benefits are noticeable. For a <em>MicroService</em>, an embedded SQlite3 storage may give pretty amazing scalability of your SOA processing.</p>
<h3>Reading Speed</h3>
<p>Here are the <em>mORMot 2</em> reading numbers:</p>
<p><table><tbody><tr align="center"><td> </td><td><strong>By one</strong></td><td><strong>All Virtual</strong></td><td><strong>All Direct</strong></td></tr>
<tr align="center"><td><strong>Sqlite file full</strong></td><td>115775</td><td>1151012</td><td>1121956</td></tr>
<tr align="center"><td><strong>Sqlite file off</strong></td><td>123198</td><td>1157407</td><td>1162925</td></tr>
<tr align="center"><td><strong>Sqlite file off exc</strong></td><td>270504</td><td>1162790</td><td>1174122</td></tr>
<tr align="center"><td><strong>Sqlite file off exc aes</strong></td><td>269978</td><td>1160227</td><td>1171920</td></tr>
<tr align="center"><td><strong>Sqlite in memory</strong></td><td>273950</td><td>1154201</td><td>1150350</td></tr>
<tr align="center"><td><strong>In memory static</strong></td><td>467551</td><td>1803751</td><td>1743071</td></tr>
<tr align="center"><td><strong>In memory virtual</strong></td><td>464209</td><td>771188</td><td>777786</td></tr>
<tr align="center"><td><strong>External sqlite file full</strong></td><td>184836</td><td>522247</td><td>1142204</td></tr>
<tr align="center"><td><strong>External sqlite file off</strong></td><td>180900</td><td>519237</td><td>1155134</td></tr>
<tr align="center"><td><strong>External sqlite file off exc</strong></td><td>186046</td><td>512583</td><td>1153003</td></tr>
<tr align="center"><td><strong>External sqlite in memory</strong></td><td>274559</td><td>1168770</td><td>1182872</td></tr>
<tr align="center"><td><strong>Remote sqlite socket</strong></td><td>22246</td><td>444820</td><td>873133</td></tr>
</tbody></table></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|All+Direct|All+Virtual|By+one&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,1803751&chds=0,1803751,0,1803751,0,1803751&chd=t:115775,1151012,1121956|123198,1157407,1162925|270504,1162790,1174122|269978,1160227,1171920|273950,1154201,1150350|467551,1803751,1743071|464209,771188,777786|184836,522247,1142204|180900,519237,1155134|186046,512583,1153003|274559,1168770,1182872|22246,444820,873133&chdl=Sqlite+file+full|Sqlite+file+off|Sqlite+file+off+exc|Sqlite+file+off+exc+aes|Sqlite+in+memory|In+memory+static|In+memory+virtual|External+sqlite+file+full|External+sqlite+file+off|External+sqlite+file+off+exc|External+sqlite+in+memory|Remote+sqlite+socket" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|Remote+sqlite+socket|External+sqlite+in+memory|External+sqlite+file+off+exc|External+sqlite+file+off|External+sqlite+file+full|In+memory+virtual|In+memory+static|Sqlite+in+memory|Sqlite+file+off+exc+aes|Sqlite+file+off+exc|Sqlite+file+off|Sqlite+file+full&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,1803751&chds=0,1803751,0,1803751,0,1803751,0,1803751,0,1803751,0,1803751,0,1803751,0,1803751,0,1803751,0,1803751,0,1803751,0,1803751&chd=t:115775,123198,270504,269978,273950,467551,464209,184836,180900,186046,274559,22246|1151012,1157407,1162790,1160227,1154201,1803751,771188,522247,519237,512583,1168770,444820|1121956,1162925,1174122,1171920,1150350,1743071,777786,1142204,1155134,1153003,1182872,873133&chdl=By+one|All+Virtual|All+Direct" /></p>
<p>In comparison, here are the <em>mORMot 1</em> performance - which was already ahead of most other solutions - on the same machine:</p>
<p><table><tbody><tr align="center"><td> </td><td><strong>By one</strong></td><td><strong>All Virtual</strong></td><td><strong>All Direct</strong></td></tr>
<tr align="center"><td><strong>Sqlite file full</strong></td><td>72439</td><td>847888</td><td>848176</td></tr>
<tr align="center"><td><strong>Sqlite file off</strong></td><td>73248</td><td>837100</td><td>858811</td></tr>
<tr align="center"><td><strong>Sqlite file off exc</strong></td><td>111037</td><td>845737</td><td>848032</td></tr>
<tr align="center"><td><strong>Sqlite file off exc aes</strong></td><td>112973</td><td>863557</td><td>869716</td></tr>
<tr align="center"><td><strong>Sqlite in memory</strong></td><td>111766</td><td>864154</td><td>879043</td></tr>
<tr align="center"><td><strong>In memory static</strong></td><td>229074</td><td>1395868</td><td>1401738</td></tr>
<tr align="center"><td><strong>In memory virtual</strong></td><td>228081</td><td>625782</td><td>626095</td></tr>
<tr align="center"><td><strong>External sqlite file full</strong></td><td>107587</td><td>412745</td><td>840618</td></tr>
<tr align="center"><td><strong>External sqlite file off</strong></td><td>132890</td><td>393948</td><td>805023</td></tr>
<tr align="center"><td><strong>External sqlite file off exc</strong></td><td>133347</td><td>411082</td><td>821422</td></tr>
<tr align="center"><td><strong>External sqlite in memory</strong></td><td>135197</td><td>411658</td><td>820075</td></tr>
<tr align="center"><td><strong>Remote sqlite socket</strong></td><td>19991</td><td>367161</td><td>643832</td></tr>
</tbody></table></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|All+Direct|All+Virtual|By+one&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,1401738&chds=0,1401738,0,1401738,0,1401738&chd=t:72439,847888,848176|73248,837100,858811|111037,845737,848032|112973,863557,869716|111766,864154,879043|229074,1395868,1401738|228081,625782,626095|107587,412745,840618|132890,393948,805023|133347,411082,821422|135197,411658,820075|19991,367161,643832&chdl=Sqlite+file+full|Sqlite+file+off|Sqlite+file+off+exc|Sqlite+file+off+exc+aes|Sqlite+in+memory|In+memory+static|In+memory+virtual|External+sqlite+file+full|External+sqlite+file+off|External+sqlite+file+off+exc|External+sqlite+in+memory|Remote+sqlite+socket" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|Remote+sqlite+socket|External+sqlite+in+memory|External+sqlite+file+off+exc|External+sqlite+file+off|External+sqlite+file+full|In+memory+virtual|In+memory+static|Sqlite+in+memory|Sqlite+file+off+exc+aes|Sqlite+file+off+exc|Sqlite+file+off|Sqlite+file+full&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,1401738&chds=0,1401738,0,1401738,0,1401738,0,1401738,0,1401738,0,1401738,0,1401738,0,1401738,0,1401738,0,1401738,0,1401738,0,1401738&chd=t:72439,73248,111037,112973,111766,229074,228081,107587,132890,133347,135197,19991|847888,837100,845737,863557,864154,1395868,625782,412745,393948,411082,411658,367161|848176,858811,848032,869716,879043,1401738,626095,840618,805023,821422,820075,643832&chdl=By+one|All+Virtual|All+Direct" /></p>
<h3>Feedback Welcome</h3>
<p>We encourage you to download the full source of both framework, from:</p>
<ul>
<li><a href="https://github.com/synopse/mORMot">https://github.com/synopse/mORMot</a></li>
<li><a href="https://github.com/synopse/mORMot2">https://github.com/synopse/mORMot2</a></li>
</ul>
<p>Then you can compile the "15 - External DB performance" sample on <em>mORMot 1</em>, and "extdb-bench" example on <em>mORMot 2</em>.</p>
<p>Feedback is <a href="https://synopse.info/forum/viewtopic.php?id=6143">welcome in our forum,</a> as usual!</p>SQlite3 Encryption Not Possible Any More Since 3.32.xurn:md5:266d34ae4331f9400def516f185677b32020-06-05T17:23:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkAESAES-NiDatabaseperformancesecuritySQLite3VFS<p>About latest SQlite3 3.32.xxx there is a big problem with codecs.</p>
<p>Critical changes to the public SQLite code were introduced on Feb 7, 2020: “<a href="https://www.sqlite.org/src/timeline?c=5a877221ce90e752">Simplify the code by removing the unsupported and undocumented SQLITE_HAS_CODEC compile-time option</a>”. With the release of SQLite version 3.32.0 on May 22, 2020 these changes finally took officially effect, although they weren't officially announced.</p>
<p><img alt="" src="https://blog.synopse.info?post/public/blog/RugbyTears.jpg" title="RugbyTears.jpg, Jun 2020" /></p>
<p>As a sad and unexpected consequence,<strong> we are NOT ANY MORE able to compile the new SQlite3 amalgamation with our encryption patch</strong>.</p> <p>Reverting the patch is not an option, since it is much more than a few lines, and it messes with SQLite3 VM opcodes, so it will induce a lot of potential errors.</p>
<p>Perhaps a dedicated VFS may be the solution - this is the path chosen by <a href="https://github.com/utelle/SQLite3MultipleCiphers">https://github.com/utelle/SQLite3MultipleCiphers</a> for instance.<br />
But a VFS is a lot of work, error prone, and performance may also decrease.<br />
If we go this VFS path, we may add a compression feature, too...</p>
<p><a href="https://synopse.info/forum/viewtopic.php?pid=32288#p32288">Any feedback is welcome!<br />
If anyone has an idea...</a></p>
<p><strong>Edit/Update</strong>: We have implemented a solution based on VFS.<br />
Please see <a href="https://synopse.info/forum/viewtopic.php?pid=32327#p32327">https://synopse.info/forum/viewtopic.php?pid=32327#p32327</a></p>Audit Trail for Servicesurn:md5:988ba15b873c9c344638de4fbce379222015-12-11T21:31:00+01:002015-12-11T21:31:00+01:00AB4327-GANDImORMot FrameworkAuditTrailblogDatabaseDelphiDocumentationDomainDrivenGoodPracticeJSONMongoDBmORMotORMSOASQLite3<p>We have seen previously how the ORM part of the framework is able to provide
an <em><a href="https://blog.synopse.info?post/post/2014/06/22/Audit-trail-for-ORM-change-tracking">Audit
Trail for change tracking</a></em>.<br />
It is a very convenient way of storing the change of state of the data.</p>
<p>On the other side, in any modern SOA solution, data is not at the center any
more, but services.<br />
Sometimes, the data is not stored within your server, but in a third-party
<em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_17">
Service-Oriented Architecture (SOA)</a></em>.<br />
Being able to monitor the service execution of the whole system becomes sooner
or later mandatory.</p>
<p><img width="300" height="200" src="http://cdn.collider.com/wp-content/image-base/Clubhouse/B/Back_to_the_Future_Delorean/Back%20to%20the%20Future%20Delorean%20Time%20Machine%20(8).jpg" alt="" /></p>
<p>Our framework allows to create an <em>Audit Trail</em> of any incoming or
outgoing service operation, in a secure, efficient and automated way.</p> <h3>When logging is not enough</h3>
<p>By default, any <code>interface</code>-based service process would be logged
by the framework - see <em>Framework log integration</em> - in dedicated
<code>sllServiceCall</code> and <code>sllServiceReturn</code> log levels.<br />
You may see output similar to the following:</p>
<pre>
18:03:18 Enter mORMot.TSQLRestServerFullMemory(024500A0).URI(POST root/DomUserQuery.SelectByLogonName/1 inlen=7)
<span style="background-color:yellow;">18:03:18 Service call mORMot.TSQLRestServerFullMemory(024500A0) DomUserQuery.SelectByLogonName["979"]</span>
18:03:18 Server mORMot.TSQLRestServerFullMemory(024500A0) POST root/DomUserQuery.SelectByLogonName SOA-Interface -> 200 with outlen=21 in 16 us
<span style="background-color:yellow;">18:03:18 Service return mORMot.TSQLRestServerFullMemory(024500A0) {"result":[0],"id":1}</span>
18:03:18 Leave 00.000.017
</pre>
<p>The above lines match the execution of the following method, as defined in
<code>dddDomUserCQRS.pas</code>:</p>
<pre>
IDomUserQuery = <strong>interface</strong>(ICQRSService)
['{198C01D6-5189-4B74-AAF4-C322237D7D53}']
<em>/// would select a single TUser from its logon name</em>
<em>// - then use Get() method to retrieve its content</em>
<strong>function</strong> SelectByLogonName(<strong>const</strong> aLogonName: RawUTF8): TCQRSResult;
...
</pre>
<p>The actual execution was:</p>
<pre>
IDomUserQuery.SelectByLogonName('979') -> cqrsSuccess
</pre>
<p>Here <code>cqrsSuccess</code> is the first item of the enumeration result,
returned as an integer JSON value <code>"result":[0]</code> by the method:</p>
<pre>
TCQRSResult =
(cqrsSuccess, cqrsSuccessWithMoreData,
cqrsUnspecifiedError, cqrsBadRequest, cqrsNotFound,
...
</pre>
<p>This detailed log (including micro-second timing on the "<em>Leave</em>"
rows) is very helpful for support, especially to investigate about any error
occurring on a production server.<br />
But it would not be enough (or on the contrary provide "too much information"
which "kills the information") to monitor the higher level of the process,
especially on a server with a lot of concurrent activity.</p>
<h3>Tracing Service Methods</h3>
<p>The framework allows to optionally store each SOA method execution in a
database, with the input and output parameters, and accurate timing.<br />
You could enable this automated process:</p>
<ul>
<li>Either at service level, using
<code>TServiceFactoryServer.SetServiceLog()</code>;</li>
<li>Or for all services of a <code>TSQLRestServer.ServiceContainer</code>
instance, via <code>TServiceContainerServer.SetServiceLog()</code>.</li>
</ul>
<p>For instance, you may enable it for a whole REST server:</p>
<pre>
(aRestSOAServer.ServiceContainer <strong>as</strong> TServiceContainerServer).SetServiceLog(
aRestLogServer,TSQLRecordServiceLog);
</pre>
<p>This single command would create an Audit Trail with all service calls made
on <code>aRestSOAServer</code> to the <code>TSQLRecordServiceLog</code> ORM
class of <code>aRestLogServer</code>.<br />
Keeping a dedicated REST server for the log entries would reduce the overhead
on the main server, and ease its maintenance.</p>
<p>Actual storage takes place within a class inheriting from
<code>TSQLRecordServiceLog</code>:</p>
<pre>
TSQLRecordServiceLog = <strong>class</strong>(TSQLRecord)
...
<strong>published</strong>
<em>/// the 'interface.method' identifier of this call</em>
<em>// - this column will be indexed, for fast SQL queries, with the MicroSec</em>
<em>// column (for performance tuning)</em>
<strong>property</strong> Method: RawUTF8 <strong>read</strong> fMethod <strong>write</strong> fMethod;
<em>/// the input parameters, as a JSON document</em>
<em>// - will be stored in JSON_OPTIONS_FAST_EXTENDED format, i.e. with</em>
<em>// shortened field names, for smaller TEXT storage</em>
<em>// - content may be searched using JsonGet/JsonHas SQL functions on a</em>
<em>// SQlite3 storage, or with direct document query under MongoDB/PostgreSQL</em>
<strong>property</strong> Input: <strong>variant read</strong> fInput <strong>write</strong> fInput;
<em>/// the output parameters, as a JSON document, including result: for a function</em>
<em>// - will be stored in JSON_OPTIONS_FAST_EXTENDED format, i.e. with</em>
<em>// shortened field names, for smaller TEXT storage</em>
<em>// - content may be searched using JsonGet/JsonHas SQL functions on a</em>
<em>// SQlite3 storage, or with direct document query under MongoDB/PostgreSQL</em>
<strong>property</strong> Output: <strong>variant read</strong> fOutput <strong>write</strong> fOutput;
<em>/// the Session ID, if there is any</em>
<strong>property</strong> Session: integer <strong>read</strong> fSession <strong>write</strong> fSession;
<em>/// the User ID, if there is an identified Session</em>
<strong>property</strong> User: integer <strong>read</strong> fUser <strong>write</strong> fUser;
<em>/// will be filled by the ORM when this record is written in the database</em>
<strong>property</strong> Time: TModTime <strong>read</strong> fTime <strong>write</strong> fTime;
<em>/// execution time of this method, in micro seconds</em>
<strong>property</strong> MicroSec: integer <strong>read</strong> fMicroSec <strong>write</strong> fMicroSec;
<strong>end</strong>;
</pre>
<p>As you can see, all input and output parameters are part of the record, as
two <code><a href="https://blog.synopse.info?post/post/2014/02/25/TDocVariant-custom-variant-type">TDocVariant</a></code>
instances. Since they are stored as JSON/TEXT, you could perform some requests
directly on their content, especially if actual storage take place in a
<em><a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">MongoDB</a></em>
database: you may even use dedicated indexes on the parameter values, and/or
run advanced <em>map/reduce</em> queries.</p>
<p>Since very accurate timing, with a micro-second resolution, is part of the
information, you would be able to make filtering or advanced statistics using
simple SQL clauses. It has never been easier to monitor your SOA system, and
identify potential issues.<br />
You may easily extract this information from your database, and feed a
real-time visual monitoring chart system, for instance. Or identify and spy
unusual execution patterns (e.g. unexpected timing or redounding error codes),
which would match some SQL requests: those SQL statements may be run
automatically on a regular basis, to prevent any problem before it actually
happen.</p>
<h3>Tracing Asynchronous External Calls</h3>
<p>Sometimes, your server may be the client of another process. In an SOA
environment, you may interface with a third-party REST service for an external
process, e.g. sending a real-time notification.</p>
<p>On the REST client instance, you can execute the
<code>TServiceFactoryClient.SendNotifications()</code> method for a given
service:</p>
<pre>
aNotificationClientService.SendNotifications(aServicesLogRest,
TSQLRecordServiceNotifications, fSettings.NotificationsRetrySeconds);
</pre>
<p>This single command would create an Audit Trail with all notification calls
sent to <code>aNotificationClientService</code>, in the
<code>TSQLRecordServiceNotifications</code> ORM class of
<code>aServicesLogRest</code>.</p>
<p>You may use the following <code>TSQLRecordServiceNotifications
class</code>:</p>
<pre>
TSQLRecordServiceNotifications = <strong>class</strong>(TSQLRecordServiceLog)
...
<strong>published</strong>
<em>/// when this notification has been sent</em>
<em>// - equals 0 until it was actually notified</em>
<strong>property</strong> Sent: TTimeLog <strong>read</strong> fSent <strong>write</strong> fSent;
<strong>end</strong>;
</pre>
<p>The additional <code>Sent</code> property would contain the
<code>TTimeLog</code> time-stamp on which the notification would have taken
place.</p>
<p>In fact, all methods executed via this notification service would now be
first stored in this table, then the remote HTTP notifications would take place
asynchronously in the background.<br />
Transmission would be in order (first-in-first-out), and in case of any
connection problem (e.g. the remote server not returning a <code>200 HTTP
SUCCESS</code> status code), it won't move to the next entry, and would retry
after the <code>NotificationsRetrySeconds</code> period, as supplied to the
<code>SendNotifications()</code> method.</p>
<p>Of course, you may define your own sub-class, to customize the destination
Audit Trail table:</p>
<pre>
<strong>type</strong>
TSQLMyNotifications = <strong>class</strong>(TSQLRecordServiceNotifications);
</pre>
<p>Thanks to those <code>TSQLRecordServiceLog</code> classes, high-level
support and analysis has never become easier. The actual implementation of
those features has been tuned to minimize the impact on main performance, by
using e.g. delayed write operations via <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_28">
BATCH sequences for adding/updating/deleting records</a></em>, or a dedicated
background thread for the asynchronous notification process.</p>
<p>For updated information, take a look at <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_183">
our official documentation</a>!</p>ORM TNullable* fields for NULL storageurn:md5:f69498b2be4a78f9a9bc0ca71dc12f672015-09-25T14:51:00+02:002015-09-25T14:06:41+02:00AB4327-GANDImORMot FrameworkblogDatabaseDelphiJSONmORMotNULLnullableORMSQL<p>In <em>Delphi</em> code, NULLable types do not exist as such. There is no
native <code>int?</code> type, as in C#.<br />
But at SQL and JSON levels, the NULL value does exist and should be converted
as expected by the ORM.</p>
<p><img src="http://www.sqlines.com/_media/oracle-to-sql-server/oracle_decode4.jpg" alt="" /></p>
<p>In <em>SQLite3</em> itself, NULL is handled as stated in <a href="http://www.sqlite.org/lang_expr.html">http://www.sqlite.org/lang_expr.html</a>
(see e.g. <code>IS</code> and <code>IS NOT</code> operators).<br />
It is worth noting that NULL handling is not consistent among all existing
database engines, e.g. when you are comparing NULL with non NULL values... so
we recommend using it with care in any database statements, or only with proper
(unit) testing, when you switch from one database engine to another.</p>
<p>By default, in the <em>mORMot</em> ORM/SQL code, NULL will appear only in
case of a BLOB storage with a size of <code>0</code> bytes.<br />
Otherwise, you should not see it as a value, <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_26">
in most kinds of ORM properties</a>.</p>
<p>Null-oriented value types have been implemented in our framework, since the
object pascal language <a href="http://wiert.me/2015/09/25/a-while-ago-allen-bauer-commented-on-the-working-theory-of-nullable-in-delphi/">
does not allow defining a nullable type (yet)</a>. </p>
<p>We choose to store those values as <code>variant</code>, with a set of
<code>TNullable</code> dedicated types, as defined in
<code>mORMot.pas</code>:</p>
<pre>
<strong>type</strong>
TNullableInteger = <strong>type variant</strong>;
TNullableBoolean = <strong>type variant</strong>;
TNullableFloat = <strong>type variant</strong>;
TNullableCurrency = <strong>type variant</strong>;
TNullableDateTime = <strong>type variant</strong>;
TNullableTimeLog = <strong>type variant</strong>;
TNullableUTF8Text = <strong>type variant</strong>;
</pre> <p>In order to define a <code>NULLable</code> column of such types, you could
use them as types for your <code>TSQLRecord</code> class definition:</p>
<pre>
<strong>type</strong>
TSQLNullableRecord = <strong>class</strong>(TSQLRecord)
<strong>protected</strong>
fInt: TNullableInteger;
fBool: TNullableBoolean;
fFlt: TNullableFloat;
fCurr: TNullableCurrency;
fDate: TNullableDateTime;
fTimeStamp: TNullableTimeLog;
fCLOB: TNullableUTF8Text;
fText: TNullableUTF8Text;
<strong>published</strong>
<strong>property</strong> Int: TNullableInteger <strong>read</strong> fInt <strong>write</strong> fInt;
<strong>property</strong> Bool: TNullableBoolean <strong>read</strong> fBool <strong>write</strong> fBool;
<strong>property</strong> Flt: TNullableFloat <strong>read</strong> fFlt <strong>write</strong> fFlt;
<strong>property</strong> Curr: TNullableCurrency <strong>read</strong> fCurr <strong>write</strong> fCurr;
<strong>property</strong> Date: TNullableDateTime <strong>read</strong> fDate <strong>write</strong> fDate;
<strong>property</strong> TimeStamp: TNullableTimeLog <strong>read</strong> fTimeStamp <strong>write</strong> fTimeStamp;
<strong>property</strong> CLOB: TNullableUTF8Text <strong>read</strong> fCLOB <strong>write</strong> fCLOB;
<strong>property</strong> Text: TNullableUTF8Text <strong>index</strong> 32 <strong>read</strong> fText <strong>write</strong> fText;
<strong>end</strong>;
</pre>
<p>Such a class would let the ORM handle SQL NULL values as expected, i.e.
returning a <code>null</code> variant value, or an integer/number/text value if
there is something stored.<br />
Of course, the corresponding column in the database would have the expected
data type, e.g. a <code>NULLABLE INTEGER</code> for
<code>TNullableInteger</code> property.</p>
<p>Note that <code>TNullableUTF8Text</code> is defined as a
<code>RawUTF8</code> usual field.<br />
That is, without any size limitation by default (as for the <code>CLOB</code>
property), or with an explicit size limitation using the <code>index ###</code>
attribute (as for <code>Text</code> property, which would be converted as a
<code>VARCHAR(32)</code> SQL column).</p>
<p>You could use the following wrapper functions to create a
<code>TNullable*</code> value from any non-nullable standard Delphi value:</p>
<pre>
<strong>function</strong> NullableInteger(<strong>const</strong> Value: Int64): TNullableInteger;
<strong>function</strong> NullableBoolean(Value: boolean): TNullableBoolean;
<strong>function</strong> NullableFloat(<strong>const</strong> Value: double): TNullableFloat;
<strong>function</strong> NullableCurrency(<strong>const</strong> Value: currency): TNullableCurrency;
<strong>function</strong> NullableDateTime(<strong>const</strong> Value: TDateTime): TNullableDateTime;
<strong>function</strong> NullableTimeLog(<strong>const</strong> Value: TTimeLog): TNullableTimeLog;
<strong>function</strong> NullableUTF8Text(<strong>const</strong> Value: RawUTF8): TNullableUTF8Text;
</pre>
<p>Some corresponding functions are able to return the expected
<code>null</code> value for each kind, with strong typing (to be used for FPC
compatibility, which does not allow direct assignment to a <code>TNullable* =
type variant</code> property):</p>
<pre>
<strong>function</strong> NullableIntegerNull: TNullableInteger;
<strong>function</strong> NullableIntegerBoolean: TNullableBoolean;
...
</pre>
<p>You could check for a <code>TNullable*</code> value to contain null, using
the following functions:</p>
<pre>
<strong>function</strong> NullableIntegerIsEmptyOrNull(<strong>const</strong> V: TNullableInteger): Boolean;
<strong>function</strong> NullableBooleanIsEmptyOrNull(<strong>const</strong> V: TNullableBoolean): Boolean;
...
</pre>
<p>Or retrieve a Delphi non-nullable value in one step, using the corresponding
wrappers:</p>
<pre>
<strong>function</strong> NullableIntegerToValue(<strong>const</strong> V: TNullableInteger; <strong>out</strong> Value: Int64): Boolean;
<strong>function</strong> NullableBooleanToValue(<strong>const</strong> V: TNullableBoolean; <strong>out</strong> Value: Boolean): Boolean;
...
<strong>function</strong> NullableIntegerToValue(<strong>const</strong> V: TNullableInteger): Int64;
<strong>function</strong> NullableBooleanToValue(<strong>const</strong> V: TNullableBoolean; <strong>out</strong> Value: Boolean): Boolean;
...
</pre>
<p>Thanks to those types, and their corresponding wrapper functions, you have
at hand everything needed to safely store some nullable values into your
application database, with proper handling on Delphi side.</p>
<p>Thanks <strong>hnb</strong> for the idea, feedback, and support!<br />
The <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_177">
documentation has been updated</a>, and should be browsed as reference.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=2668">welcome on our forum, as
usual</a>!</p>Delphi 10 = DX Seattle is out, mORMot supports iturn:md5:ac1f3bb4eb71134abb9c8f08fb724bf22015-08-31T17:50:00+02:002015-08-31T16:57:57+02:00AB4327-GANDIPascal ProgrammingblogDatabaseDelphiDXMongoDBmORMotNoSQLODMORMSeattle <p>We expected Delphi XE9, and now we have <a href="https://plus.google.com/u/0/+MarcoCantu/posts/F6WZDc93iHW">Rad Studio 10
Seattle</a>, with Delphi renamed as <a href="http://www.embarcadero.com/en/products/delphi">Delphi 10 Seattle</a>, or
simply <strong>DX</strong>.</p>
<p><img src="http://www.embarcadero.com/images/banners/seattle/delphi-sidebar.png" alt="" /></p>
<p>No big news for the Delphi compiler itself (we are still waiting for
<em>Linux</em> server support), but a lot of <em>FireMonkey</em> updates,
Windows 10 compatibility enhancements, enhancements to JSON (better performance
using a SAX approach), and <a href="http://synopse.info/forum/viewtopic.php?pid=17517#p17517">NoSQL/MongoDB
support in FireDAC</a>.<br />
The documentation is <a href="http://docwiki.embarcadero.com/Libraries/Seattle/en/FireDAC.Phys.MongoDBWrapper">
rather sparse for the new features</a>, but it goes into the right direction
(we <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">support
MongoDB</a> since a long time, in our ORM/ODM).<br />
See <a href="http://docwiki.embarcadero.com/RADStudio/Seattle/en/What%27s_New">what's new
in details</a>.</p>
<p><strong>Of course, our Open Source</strong> <em><strong>mORMot</strong></em>
<strong>framework supports this version.</strong><br />
Feedback is welcome, as usual!<br />
Enjoy the new DX IDE!</p>"SQL and NoSQL", not "SQL vs NoSQL"urn:md5:c22e98fb0648c165bc9ee01f23a34cdd2015-08-23T13:34:00+02:002015-08-23T13:03:13+02:00AB4327-GANDImORMot FrameworkblogDatabaseDelphiJSONMongoDBmORMotNoSQLODMOpenSourceORMSQLSQLite3TDocVariant<p>You know certainly that our <em>mORMot</em> Open Source framework is an ORM,
i.e. mapping objects to a relational / SQL database (<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_13">Object
Relational Mapping</a>).<br />
You may have followed also that it is able to connect to a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_83">
NoSQL database</a>, like <a href="https://www.mongodb.org/">MongoDB</a>, and
that the objects are then mapped via an ODM (<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_82">Object
Document Mapping</a>) - the original SQL SELECT are even <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_207">
translated on the fly to MongoDB queries</a>.</p>
<p>But thanks to <em>mORMot</em>, it is not "<em>SQL vs NoSQL</em>" - but
"<em>SQL and NoSQL</em>".<br />
You are not required to make an exclusive choice.<br />
You can share best of both worlds, depending on your application needs.</p>
<p><img src="http://bodescu.me/wp-content/uploads/2015/02/sql-vs-nosql-450x217.png" alt="" /></p>
<p>In fact, the framework is able to <em>add NoSQL features to a regular
relational / SQL database</em>, by storing JSON documents in TEXT columns.</p>
<p>In your end-user code, you just define a <code>variant</code> field in
the ORM, and store a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_80">
TDocVariant document</a> within.<br />
We also added some dedicated functions at SQL level, so that
<em>SQLite3</em> could be used as embedded fast engine, and provide
advanced WHERE clauses on this JSON content.</p> <h3>Schemaless storage via a variant</h3>
<p>As we just wrote, a first-class candidate for <em>data sharding</em> in a
<code>TSQLRecord</code> is our <em>TDocVariant custom variant type</em>.</p>
<p>You may define:</p>
<pre>
TSQLRecordData = <strong>class</strong>(TSQLRecord)
<strong>private</strong>
fName: RawUTF8;
fData: <strong>variant</strong>;
<strong>public</strong>
<strong>published</strong>
<strong>property</strong> Name: RawUTF8 <strong>read</strong> fTest <strong>write</strong> fTest <strong>stored</strong> AS_UNIQUE;
<span style="background-color:yellow;"><strong>property</strong> Data: <strong>variant read</strong> fData <strong>write</strong> fData;</span>
<strong>end</strong>;
</pre>
<p>Here, we defined two indexed keys, ready to access any data record:</p>
<ul>
<li>Via the <code>ID: TID</code> property defined at <code>TSQLRecord</code>
level, which will map the <em>SQLite3</em> <code>RowID</code> primary key;</li>
<li>Via the <code>Name: RawUTF8</code> property, which will was marked to be
indexed by setting the "<code>stored AS_UNIQUE</code>" attribute.</li>
</ul>
<p>Then, any kind of data may be stored in the <code>Data: variant</code>
published property. In the database, it will be stored as JSON UTF-8 text,
ready to be retrieved from any client, including AJAX / HTML5
applications.<br />
<em>Delphi</em> clients or servers will access those data via
<em>late-binding</em>, from its <code>TDocVariant</code> instance.</p>
<p>You just reproduced the <em>schema-less</em> approach of the NoSQL database
engines, in a few lines of code! Thanks to the <em>mORMot</em>'s <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_6">
JSON RESTful Client-Server</a></em> design, your applications are able to store
any kind of document, and easily access to them via HTTP.</p>
<p>The documents stored in such a database can have varying sets of fields,
with different types for each field. One could have the following objects in a
single collection of our <code>Data: variant</code> rows:</p>
<pre>
<em>{ name : "Joe", x : 3.3, y : [1,2,3] }</em>
<em>{ name : "Kate", x : "abc" }</em>
<em>{ q : 456 }</em>
</pre>
<p>Of course, when using the database for real problems, the data does have a
fairly consistent structure. Something like the following would be more common,
e.g. for a table persisting <em>student</em> objects:</p>
<pre>
<em>{ name : "Joe", age : 30, interests : "football" }</em>
<em>{ name : "Kate", age : 25 }</em>
</pre>
<p>Generally, there is a direct analogy between this <em>schema-less</em> style
and dynamically typed languages. Constructs such as those above are easy to
represent in <em>PHP</em>, <em>Python</em> and <em>Ruby</em>. And, thanks to
our <code>TDocVariant</code> <em>late-binding</em> magic, even our good
<em>Delphi</em> is able to handle those structures in our code. What we are
trying to do here is make this mapping to the database natural, like:</p>
<pre>
<strong>var</strong> aRec: TSQLRecordData;
aID: TID;
<strong>begin</strong>
<em>// initialization of one record</em>
aRec := TSQLRecordData.Create;
aRec.Name := 'Joe'; <em>// one unique key</em>
aRec.data := _JSONFast('{name:"Joe",age:30}'); <em>// create a TDocVariant</em>
<em>// or we can use this overloaded constructor for simple fields</em>
aRec := TSQLRecordData.Create(['Joe',_ObjFast(['name','Joe','age',30])]);
<em>// now we can play with the data, e.g. via late-binding:</em>
writeln(aRec.Name); <em>// will write 'Joe'</em>
writeln(aRec.Data); <em>// will write '{"name":"Joe","age":30}' (auto-converted to JSON string)</em>
aRec.Data.age := aRec.Data.age+1; <em>// one year older</em>
aRec.Data.interests := 'football'; <em>// add a property to the schema</em>
aID := aClient.Add(aRec,true); <em>// will store {"name":"Joe","age":31,"interests":"footbal"}</em>
aRec.Free;
<em>// now we can retrieve the data either via the aID created integer, or via Name='Joe'</em>
<strong>end</strong>;
</pre>
<p>One of the great benefits of these dynamic objects is that schema migrations
become very easy.<br />
With a traditional RDBMS, releases of code might contain data migration
scripts. Further, each release should have a reverse migration script in case a
rollback is necessary.<br />
<code>ALTER TABLE</code> operations can be very slow and result in scheduled
downtime.</p>
<p>With a <em>schema-less</em> organization of the data, 90% of the time
adjustments to the database become transparent and automatic.<br />
For example, if we wish to add GPA to the <em>student</em> objects, we add the
attribute, re-save, and all is well - if we look up an existing student and
reference GPA, we just get back null.<br />
Furthermore, if we roll back our code, the new GPA fields in the existing
objects are unlikely to cause problems if our code was well written.</p>
<p>In fact, <em>SQlite3</em> is so efficient about its indexes B-TREE storage,
that such a structure may be used as a credible alternative to much heavier
<em>NoSQL</em> engines, like <em>MongoDB</em> or <em>CouchDB</em>.<br />
With the possibility to add some "regular" fields, e.g. plain numbers (like
ahead-computed aggregation values), or text (like a summary or description
field), you can still use any needed fast SQL query, without the complexity of
<em>map/reduce</em> algorithm used by the <em>NoSQL</em> paradigm. You could
even use the <em>Full Text Search</em> - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_8">
FTS3/FTS4</a></em> - or RTREE extension advanced features of <em>SQLite3</em>
to perform your queries.<br />
Then, thanks to <em>mORMot</em>'s ability to access any external database
engine, you are able to perform a JOINed query of your <em>schema-less</em>
data with some data stored e.g. in an Oracle, PostgreSQL or MS SQL enterprise
database. Or switch later to a true <em>MongoDB</em> storage, in just one line
of code - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_84">
MongoDB + ORM = ODM</a></em>.</p>
<h3>JSON operations from SQL code</h3>
<p>As we stated, any <code>variant</code> field would be serialized as JSON,
then stored as plain TEXT in the database.<br />
In order to make a complex query on the stored JSON, you could retrieve it in
your end-user code, then use the corresponding <code>TDocVariant</code>
instance to perform the search on its content.<br />
Of course, all this has a noticeable performance cost, especially when the data
tend to grow.</p>
<p>The natural way of solving those performance issue is to add some "regular"
RDBMS fields, with a proper index, then perform the requests on those
fields.<br />
But sometimes, you may need to do some addition query, perhaps in conjunction
with "regular" field lookup, on the JSON data stored itself.<br />
In order to avoid the slowest conversion to the ORM client side, we defined
some SQL functions, dedicated to JSON process.</p>
<p>The first is <code>JsonGet()</code>, and is able to extract any value from
the TEXT field, mapping a <code>variant</code>:</p>
<table>
<tbody>
<tr>
<td><code>JsonGet(ArrColumn,0)</code></td>
<td>returns a property value by index, from a JSON array</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'PropName')</code></td>
<td>returns a property value by name, from a JSON object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Obj1.Obj2.Prop')</code></td>
<td>returns a property value by path, including nested JSON objects</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop1,Prop2')</code></td>
<td>extract properties by name, from a JSON object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop1,Obj1.Prop')</code></td>
<td>extract properties by name (including nested JSON objects), from a JSON
object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop*')</code></td>
<td>extract properties by wildchar name, from a JSON object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop*,Obj1.P*')</code></td>
<td>extract properties by wildchar name (including nested JSON objects), from a
JSON object</td>
</tr>
</tbody>
</table>
<p>If no value does match, this function would return the SQL
<code>NULL</code>. If the matching value is a simple JSON text or number, it
will be returned as a TEXT, INTEGER or DOUBLE value, ready to be passed as a
result column or any WHERE clause. If the returned value is a nested JSON
object or array, it will be returned as TEXT, serialized as JSON; as a
consequence, you may use it as the source of another <code>JsonGet()</code>
function, or even able to gather the results via the <code>CONCAT()</code>
aggregate function.</p>
<p>The comma-separated syntax allowed in the property name parameter (e.g.
<code>'Prop1,Prop2,Prop3'</code>), would search for several properties at once
in a single object, returning a JSON object of all matching values - e.g.
<code>'{"Prop2":"Value2","Prop3":123}'</code> if the <code>Prop1</code>
property did not appear in the stored JSON object.</p>
<p>If you end the property name with a <code>*</code> character, it would
return a JSON object, with all matching properties.<br />
Any nested object would have its property names be flattened as
{<code>"Obj1.Prop":...}</code>, within the returned JSON object.<br />
Note that the comma-separated syntax also allows such wildchar search, so that
e.g.</p>
<pre>
JsonGet(ObjColumn,'owner') = {"login":"smith","id":123456} as TEXT
JsonGet(ObjColumn,'owner.login') = "smith" as TEXT
JsonGet(ObjColumn,'owner.id') = 123456 as INTEGER
JsonGet(ObjColumn,'owner.name') = NULL
JsonGet(ObjColumn,'owner.login,owner.id') = {"owner.login":"smith","owner.id":123456} as TEXT
JsonGet(ObjColumn,'owner.I*') = {"owner.id:123456} as TEXT
JsonGet(ObjColumn,'owner.*') = {"owner.login":"smith","owner.id":123456} as TEXT
JsonGet(ObjColumn,'unknown.*') = NULL
</pre>
<p>Another function, named <code>JsonHas()</code> is similar to
<code>JsonGet()</code>, but will return TRUE or FALSE depending if the supplied
property (specified by name or index) do exist. It may be faster to use
<code>JsonHas()</code> than <code>JsonGet()</code> e.g. in a WHERE clause, when
you do not want to process this property value, but only return data rows
containing needed information.</p>
<pre>
JsonHas(ObjColumn,'owner') = true
JsonHas(ObjColumn,'owner.login') = true
JsonHas(ObjColumn,'owner.name') = false
JsonHas(ObjColumn,'owner.i*') = true
JsonHas(ObjColumn,'owner.n*') = false
</pre>
<p>Since the process would take place within the <em>SQLite3</em> engine
itself, and since they use a SAX-like fast approach (without any temporary
memory allocation during its search), those JSON functions could be pretty
efficient, and proudly compare to some dedicated NoSQL engines.</p>
<h3>The upcoming official SQlite3 extension</h3>
<p>As you may have noticed <a href="http://www.sqlite.org/src/info/178f9a352c6c9e15">in the SQlite3 timeline</a>,
some extension functions for processing JSON are currently being
implemented.</p>
<p>The <em>mORMot</em> native implementation is very proven, since it uses the
<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_2">
JSON core of the framework</a>, and is able to handle the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_39">
"extended" JSON syntax of MongoDB</a>: field names can be serialized
<em>without</em> double quotes, which reduces a lot the storage size, and the
transmission size, when the JSON is stored within a TEXT column, which would
escape all double quote characters.</p>
<p>I guess the performance of our implementation is probably faster than
current <em>SQlite3</em>'s extension.<br />
We use a SAX-like fast approach, without any temporary memory allocation during
its search, whereas AFAIK <em>SQlite3</em>'s extension is doing a lot of
memory allocations, creating all nodes of the JSON documents, like a DOM.</p>
<p>But here again, we are not exclusive guys. If you like this upcoming JSON
extension, you are free to use it! No offense taken!</p>
<p>Feedback <a href="http://synopse.info/forum/viewtopic.php?pid=17448#p17448">is welcome on our
forum</a>, as usual.</p>GetIt "Spirit" Concernsurn:md5:c8befaf992f7ae8f2053d260241925262015-06-06T10:52:00+02:002015-06-06T12:23:00+02:00AB4327-GANDIPascal ProgrammingblogDatabaseDelphiGetItmORMotOpenSourceORMSOA <p>I'm confused by the <a href="http://community.embarcadero.com/index.php/article/news/16027-submitting-libraries-to-getit">
GetIt Submitting official blog page</a>.<br />
Reminds me the darker ages of <a href="http://www.deltics.co.nz/blog/posts/1097">Delphi licensing change of
XE3</a>.</p>
<p><a href="http://blog.marcocantu.com/blog/2015-june-more-info-getit.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+marcocantublog+%28marcocantu.blog%29">
GetIt</a> is the new XE8 package manager for RAD Studio. Information about how
to submit your libraries to GetIt has just been made available by Embarcadero.
The idea behind GetIt is really to make is easier and faster to discover,
install, and keep updated some of the best open source libraries for Delphi and
C++Builder.</p>
<p>When you <a href="http://community.embarcadero.com/index.php/article/news/16027-submitting-libraries-to-getit">
look at the approval conditions</a>, it sounds like if mORMot would not find
its way in this package manager:</p>
<blockquote>
<p>Replacing key capabilities that are part of the core platforms definitions
such as Client/Server FireDAC pack or the DataSnap/EMS Enterprise middleware,
would be less likely to be accepted.<br />
The different SKUs are meant for different types of developers, and multi-tier
capabilities with strong client/server RDBMS integration
<strong>require</strong> an Enterprise edition license.<br />
We will bias acceptance toward GetIt libraries that respect the <strong>spirit
of our licensing and editions</strong>, not just use the letter of the license
and the technical boundaries. If you are unsure about your submission please
check with us first. </p>
</blockquote>
<p>What is this "<em>spirit of our licensing and editions</em>"?<br />
Why is it not part of the official license terms?<br />
Where does this assumption comes from?<br />
Would the licensing conditions change in the close future, as with the XE3
"episode"?<br />
Would Marco's interpretation become the new rule?</p>
<p>It clearly <a href="https://blog.synopse.info?post/post/2012/09/03/Client-Server-allowed-back-to-XE3-pro">reminds me the XE3
time where there was an attempt from Embarcadero to modify their licence
terms</a>, so that third party vendors or Open Source libraries would not be
allowed to create multi-tier frameworks with Delphi!<br />
Is it true that "<em>strong client/server RDBMS integration
<strong>require</strong> an Enterprise edition</em>" ?<br />
Last time I checked the licence terms, it was not stated.<br />
Why on earth would we have to pay for the <em>Entreprise</em> edition, if the
<em>Professionnal</em> edition is all that you need?</p>
<p><img src="http://static.tvtropes.org/pmwiki/pub/images/suicidalstupidity_6656.jpg" alt="" /></p>
<p>Still the same "closed" spirit.<br />
It is like if they know their own n-Tier solution weaknesses, so they try to
avoid any other possibility, but to use their own.<br />
They clearly do not understand the benefit and dynamic of Open Source.</p>
<p>I guess our little <em>mORMot</em> falls directly into this "unwelcomed"
category.<br />
I did not make the submission yet. But should I?<br />
Perhaps sub-part of the framework may find its way in: <a href="http://synopse.info/fossil/wiki?name=PDF+Engine">SynPdf</a>, <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html#">SynCrypto</a>,
<a href="http://synopse.info/files/html/api-1.18/SynGdiPlus.html#">SynGdiPlus</a>,
<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_45">
SynCommons</a>...<br />
But the main ORM/SOA/REST/MVC/StubMock features would certainly be
rejected.</p>
<p>Our Open Source project is sometimes preferred to DataSnap/EMS (or even
FireDAC), not only for licence cost, but also about features, documentation,
stability, compatibility with older versions of Delphi, performance, and Linux
compatibility.<br />
I have encountered several companies which are still using Delphi <em>because
of <em>mORMot</em></em>: if they did not have found it, they would have moved
to C# or Java, just to be able to use a full SOA/MVC stack, which was not
available, even in the "Enterprise" version of Delphi.</p>
<p>Story repeats itself.<br />
I just wanted to ensure that the licensing terms would not change in that
direction.<br />
I - as many Delphi users - would not let this GetIt "spirit" become the new
rule.<br />
We have to react, as we did for XE3, otherwise we may all suffer!<br />
IMHO Embacardero should better focus on the compiler and IDE, not cutting the
branch on which they are seated...</p>
<p>What do you think? <a href="http://synopse.info/forum/viewtopic.php?id=2623">Comments and feebacks are
welcome</a>!</p>Asynchronous Service - WebSockets, Callbacks and Publish-Subscribeurn:md5:4b0f2fbc0b9085026d4baacbf5ee8ed92015-04-06T21:34:00+02:002015-04-06T20:57:33+02:00AB4327-GANDImORMot FrameworkAESAES-NiAJAXauthenticationblogDatabaseDelphiDocumentationEventCollaborationEventSourcinginterfaceModelORMsecuritySOASOLIDSourceSQLite3SynopseWebSockets<p>When publishing SOA services, most of them are defined as
<em>stateless</em>, in a typical query/answer pattern - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_17">
Service-Oriented Architecture (SOA)</a></em>.<br />
This fits exactly with the <em>RESTful</em> approach of <em>Client-Server
services via interfaces</em>, as proposed by the framework.</p>
<p>But it may happen that a client application (or service) needs to know the
state of a given service. In a pure <em>stateless</em> implementation, it will
have to <em>query</em> the server for any state change, i.e. for any pending
notification - this is called <em>polling</em>.</p>
<p><img src="https://blog.synopse.info?post/public/mORMot/BatNotification.png" alt="" title="RealTime Notification, Apr 2015" /></p>
<p><em>Polling</em> may take place for instance:</p>
<ul>
<li>When a time consuming work is to be processed on the server side. In this
case, the client could not wait for it to be finished, without raising a
timeout on the HTTP connection: as a workaround, the client may start the work,
then ask for its progress status regularly using a timer and a dedicated method
call;</li>
<li>When an unpredictable event is to be notified from the server side. In this
case, the client should ask regularly (using a timer, e.g. every second), for
any pending event, then react on purpose.</li>
</ul>
<p>It may therefore sounds preferred, and in some case necessary, to have the
ability to let the server <em>notify</em> one or several clients without any
prior query, nor having the requirement of a client-side timer:</p>
<ul>
<li><em>Polling</em> may be pretty resource consuming on both client and server
sides, and add some unwanted latency;</li>
<li>If immediate notification is needed, some kind of "long polling" algorithm
may take place, i.e. the server will wait for a long time before returning the
notification state if no event did happen: in this case, a dedicated connection
is required, in addition to the REST one;</li>
<li>In an event-driven systems, a lot of messages are sent to the clients: a
proper publish/subscribe mechanism is preferred, otherwise the complexity of
polling methods may increase and become inefficient and unmaintainable;</li>
<li>Explicit push notifications may be necessary, e.g. when a lot of potential
events, associated with a complex set of parameters, are likely to be sent by
the client.</li>
</ul>
<p>Our <em>mORMot</em> framework is therefore able to easily implement
asynchronous callbacks over <em><a href="http://www.developerfusion.com/article/143158/an-introduction-to-websockets/">WebSockets</a></em>,
defining the callbacks as <code>interface</code> parameters in service method
definitions - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_154">
Available types for methods parameters</a></em>.</p> <h3>1. WebSockets support</h3>
<p>By definition, HTTP connections are stateless and one-way, i.e. a client
sends a request to the server, which replies back with an answer.<br />
There is no way to let the server send a message to the client, without a prior
request from the client side.</p>
<p><em>WebSockets</em> is a communication protocol which is able to
<em>upgrade</em> a regular HTTP connection into a dual-way communication
wire.<br />
After a safe handshake, the underlying TCP/IP socket is able to be accessed
directly, via a set of lightweight <em>frames</em> over an application-defined
<em>protocol</em>, without the HTTP overhead.</p>
<p>The <code><a href="http://synopse.info/files/html/api-1.18/SynBidirSock.html">SynBidirSock.pas</a></code>
unit implements low-level server and client <em>WebSockets</em>
communication.</p>
<p>The <code>TWebSocketProtocol</code> class defines an abstract <em>WebSockets
protocol</em>, currently implemented as several classes.</p>
<p>For our <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#SIDE_TITL_63">
Client-Server services via interfaces</a></em>, we would still need to make
<em>RESTful</em> requests, so the basic <em>WebSockets</em> framing has been
enhanced to support <code>TWebSocketProtocolRest</code> REST-compatible
protocols, able to use the single connection for both REST queries and
asynchronous notifications.</p>
<p>Two classes are available for your SOA applications:</p>
<ul>
<li><code>TWebSocketProtocolJSON</code> as a "pure" JSON light protocol;</li>
<li><code>TWebSocketProtocolBinary</code> as a binary proprietary protocol,
with optional frame compression and AES encryption (using AES-NI hardware
instructions, if available).</li>
</ul>
<p>In practice, on the server side, you would start your
<code>TSQLHttpServer</code> by specifying <code>useBidirSocket</code> as kind
of server:</p>
<pre>
HttpServer := TSQLHttpServer.Create('8888',[Server],'+',useBidirSocket);
</pre>
<p>Under the hood, it will instantiate a <code>TWebSocketServer</code> HTTP
server, as defined in <code>mORMotHttpServer.pas</code>, based on the sockets
API, able to upgrade the HTTP protocol into <em>WebSockets</em>.<br />
Our <em>High-performance http.sys server</em> is not yet able to switch to
<em>WebSockets</em> - and at API level, it would require at least <em>Windows
8</em> or <em>Windows 2012 Server</em>.</p>
<p>Then you enable <em>WebSockets</em> for the
<code>TWebSocketProtocolBinary</code> protocol, with an encryption key:</p>
<pre>
HttpServer.WebSocketsEnable(Server,'encryptionkey');
</pre>
<p>On the client side, you would use a <code>TSQLHttpClientWebsockets</code>
instance, as defined in <code>mORMotHttpClient.pas</code>, then explicitly
upgrade the connection to use <em>WebSockets</em> (since by default, it will
stick to the HTTP protocol):</p>
<pre>
Client := TSQLHttpClientWebsockets.Create('127.0.0.1','8888',TSQLModel.Create([]));
Client.WebSocketsUpgrade('encryptionkey');
</pre>
<p>The expected protocol detail should match the one on the server, i.e.
<code>'encryptionkey'</code> encryption over our binary protocol.</p>
<p>Once upgraded to <em>WebSockets</em>, you may use regular REST commands, as
usual:</p>
<pre>
Client.ServerTimeStampSynchronize;
</pre>
<p>But in addition to regular query/answer commands as defined for
<em>Client-Server services via interfaces</em>, you would be able to define
callbacks using <code>interface</code> parameters to the service methods.</p>
<p>Under the hood, both client and server will communicate using
<em>WebSockets</em> frames, maintaining the connection active using heartbeats
(via ping/pong frames), and with clean connection shutdown, from any side. You
can use the <code>Settings</code> property of the
<code>TWebSocketServerRest</code> instance, as returned by
<code>TSQLHttpServer.WebSocketsEnable()</code>, to customize the low-level
<em>WebSockets</em> protocol (e.g. timeouts or heartbeats) on the server side.
The <code>TSQLHttpClientWebsockets.WebSockets</code>.<code>Settings</code>
property would allow the same, on the client side.</p>
<p>We have observed, from our regression tests and internal benchmarking, that
using our <em>WebSockets</em> may be faster than regular HTTP, since its frames
would be sent as once, whereas HTTP headers and body are not sent in the same
TCP packet, and compression would be available for the whole frame, whereas
HTTP headers are not compressed. The ability to use strong AES encryption would
make this mean of communication even safer than plain HTTP, even with
<em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_151">
AES encryption over HTTP</a></em>.</p>
<h3>2. Using a callback to notify long term end-of-process</h3>
<p>An example is better than 100 talks.<br />
So let's take a look at the <code>Project31LongWorkServer.dpr</code> and
<code>Project31LongWorkClient.dpr</code> samples, from the <code><a href="https://github.com/synopse/mORMot/tree/master/SQLite3/Samples/31%20-%20WebSockets">
SQLite3\Samples\31 - WebSockets</a></code> sub-folder.<br />
They will implement a client/server application, in which the client launches a
long term process on the server side, then is notified when the process is
done, either with success, or failure.</p>
<p>First we define the interfaces to be used, in a shared
<code>Project31LongWorkCallbackInterface.pas</code> unit:</p>
<pre>
<strong>type</strong>
ILongWorkCallback = <strong>interface</strong>(IInvokable)
['{425BF199-19C7-4B2B-B1A4-A5BE7A9A4748}']
<strong>procedure</strong> WorkFinished(<strong>const</strong> workName: <strong>string</strong>; timeTaken: integer);
<strong>procedure</strong> WorkFailed(<strong>const</strong> workName, error: <strong>string</strong>);
<strong>end</strong>;<br /> ILongWorkService = <strong>interface</strong>(IInvokable)
['{09FDFCEF-86E5-4077-80D8-661801A9224A}']
<strong>procedure</strong> StartWork(<strong>const</strong> workName: <strong>string</strong>; <strong>const</strong> onFinish: ILongWorkCallback);
<strong>function</strong> TotalWorkCount: Integer;
<strong>end</strong>;
</pre>
<p>The only specific definition is the <code>const onFinish:
ILongWorkCallback</code> parameter, supplied to the
<code>ILongWorkService.StartWork()</code> method.<br />
The client will create a class implementing <code>ILongWorkCallback</code>,
then specify it as parameter to this method.<br />
On the server side, a "fake" class will implement
<code>ILongWorkCallback</code>, then will call back the client using the very
same <em>WebSockets</em> connection, when any of its methods will be
executed.</p>
<p>As you can see, a single callback <code>interface</code> instance may have
several methods, with their own set of parameters (here
<code>WorkFinished</code> and <code>WorkFailed</code>), so that the callback
may be quite expressive.<br />
Any kind of usual parameters would be transmitted, after serialization:
<code>string</code>, <code>integer</code>, but even <code>record</code>,
<em>dynamic arrays</em>, <code>TSQLRecord</code> or <code>TPersistent</code>
values.</p>
<p>When the <code>ILongWorkCallback</code> instance will be released on the
client side, the server will be notified, so that any further notification
won't create a connection error.<br />
We will see later how to handle those events.</p>
<h3>Client service consumption</h3>
<p>The client may be connected to the server as such (see the
<code>Project31LongWorkClient.dpr</code> sample source code for the full
details, including error handling):</p>
<pre>
<strong>var</strong> Client: TSQLHttpClientWebsockets;
workName: <strong>string</strong>;
Service: ILongWorkService;
callback: ILongWorkCallback;
<strong>begin</strong>
Client := TSQLHttpClientWebsockets.Create('127.0.0.1','8888',TSQLModel.Create([]));
<span style="background-color:yellow;">Client.WebSocketsUpgrade(PROJECT31_TRANSMISSION_KEY);</span>
Client.ServiceDefine([ILongWorkService],sicShared);
Client.Services.Resolve(ILongWorkService,Service);
</pre>
<p>Then we define our callback, using a dedicated class:</p>
<pre>
<strong>type</strong>
TLongWorkCallback = <strong>class</strong>(TInterfacedCallback,ILongWorkCallback)
<strong>protected</strong>
<strong>procedure</strong> WorkFinished(<strong>const</strong> workName: <strong>string</strong>; timeTaken: integer);
<strong>procedure</strong> WorkFailed(<strong>const</strong> workName, error: <strong>string</strong>);
<strong>end</strong>;<br /><strong>procedure</strong> TLongWorkCallback.WorkFailed(<strong>const</strong> workName, error: <strong>string</strong>);
<strong>begin</strong>
writeln(#13'Received callback WorkFailed(',workName,') with message "',error,'"');
<strong>end</strong>;
<br /><strong>procedure</strong> TLongWorkCallback.WorkFinished(<strong>const</strong> workName: <strong>string</strong>;
timeTaken: integer);
<strong>begin</strong>
writeln(#13'Received callback WorkFinished(',workName,') in ',timeTaken,'ms');
<strong>end</strong>;
</pre>
<p>Then we specify this kind of callback as parameter to start a long term
work:</p>
<pre>
<span style="background-color:yellow;">callback := TLongWorkCallback.Create(Client,ILongWorkCallback);</span>
<strong>try</strong>
<strong>repeat</strong>
readln(workName);
<strong>if</strong> workName='' <strong>then</strong>
break;
<span style="background-color:yellow;">Service.StartWork(workName,callback);</span>
<strong>until</strong> false;
<strong>finally</strong>
callback := <strong>nil</strong>; <em>// the server will be notified and release its "fake" class</em>
Service := <strong>nil</strong>; <em>// release the service local instance BEFORE Client.Free</em>
<strong>end</strong>;
</pre>
<p>As you can see, the client is able to start one or several work processes,
then expects to be notified of the process ending on its callback
<code>interface</code> instance, without explicitly polling the server for its
state, since the connection was upgraded to <em>WebSockets</em> via a call to
<code>TSQLHttpClientWebsockets.WebSocketsUpgrade()</code>.</p>
<h3>Server side implementation</h3>
<p>The server would define the working thread as such (see the
<code>Project31LongWorkServer.dpr</code> sample source code for the full
details):</p>
<pre>
<strong>type</strong>
TLongWorkServiceThread = <strong>class</strong>(TThread)
<strong>protected</strong>
<span style="background-color:yellow;">fCallback: ILongWorkCallback;</span>
fWorkName: <strong>string</strong>;
<strong>procedure</strong> Execute; <strong>override</strong>;
<strong>public</strong>
<strong>constructor</strong> Create(<strong>const</strong> workName: <strong>string</strong>; <strong>const</strong> callback: ILongWorkCallback);
<strong>end</strong>;
<br /><strong>constructor</strong> TLongWorkServiceThread.Create(<strong>const</strong> workName: <strong>string</strong>;
<strong>const</strong> callback: ILongWorkCallback);
<strong>begin</strong>
<strong>inherited</strong> Create(false);
<span style="background-color:yellow;">fCallback := Callback;</span>
fWorkName := workName;
FreeOnTerminate := true;
<strong>end</strong>;
<br /><strong>procedure</strong> TLongWorkServiceThread.Execute;
<strong>var</strong> tix: Int64;
<strong>begin</strong>
tix := GetTickCount64;
<span style="background-color:yellow;">Sleep(5000+Random(1000)); <em>// some hard work</em></span>
<strong>if</strong> Random(100)>20 <strong>then</strong>
<span style="background-color:yellow;">fCallback.WorkFinished(fWorkName,GetTickCount64-tix) <strong>else</strong></span>
<span style="background-color:yellow;">fCallback.WorkFailed(fWorkName,'expected random failure');</span>
<strong>end</strong>;
</pre>
<p>The callback is expected to be supplied as a <code>ILongWorkCallback</code>
interface instance, then stored in a <code>fCallback</code> protected field for
further notification.<br />
Some work is done in the <code>TLongWorkServiceThread.Execute</code> method
(here just a <code>Sleep()</code> of more than 5 seconds), and the end-of-work
notification is processed, as success or failure (depending on random in this
fake process class), on either of the <code>ILongWorkCallback</code> interface
methods.</p>
<p>The following <code>class</code> will define, implement and register the
<code>ILongWorkService</code> service on the server side:</p>
<pre>
<strong>type</strong>
TLongWorkService = <strong>class</strong>(TInterfacedObject,ILongWorkService)
<strong>protected</strong>
fTotalWorkCount: Integer;
<strong>public</strong>
<strong>procedure</strong> StartWork(<strong>const</strong> workName: <strong>string</strong>; <strong>const</strong> onFinish: ILongWorkCallback);
<strong>function</strong> TotalWorkCount: Integer;
<strong>end</strong>;
<br /><strong>procedure</strong> TLongWorkService.StartWork(<strong>const</strong> workName: <strong>string</strong>;
<strong>const</strong> onFinish: ILongWorkCallback);
<strong>begin</strong>
InterlockedIncrement(fTotalWorkCount);
<span style="background-color:yellow;">TLongWorkServiceThread.Create(workName,onFinish);</span>
<strong>end</strong>;
<br /><strong>function</strong> TLongWorkService.TotalWorkCount: Integer;
<strong>begin</strong>
result := fTotalWorkCount;
<strong>end</strong>;
<br /><strong>var</strong> HttpServer: TSQLHttpServer;
Server: TSQLRestServerFullMemory;
<strong>begin</strong>
Server := TSQLRestServerFullMemory.CreateWithOwnModel([]);
<span style="background-color:yellow;">Server.ServiceDefine(TLongWorkService,[ILongWorkService],sicShared);</span>
HttpServer := TSQLHttpServer.Create('8888',[Server],'+',useBidirSocket);
<span style="background-color:yellow;">HttpServer.WebSocketsEnable(Server,PROJECT31_TRANSMISSION_KEY);</span>
...
</pre>
<p>Purpose of those methods is just to create and launch the
<code>TLongWorkServiceThread</code> process from a client request, then
maintain a total count of started works, in a <code>sicShared</code> service
instance - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_92">
Instances life time implementation</a></em> - hosted in a
<code>useBidirSocket</code> kind of HTTP server.</p>
<p>We have to explicitly call <code>TSQLHttpServer.WebSocketsEnable()</code> so
that this server would be able to upgrade to our <em>WebSockets</em> protocol,
using our binary framing, and the very same encryption key as on the client
side - shared as a <code>PROJECT31_TRANSMISSION_KEY</code> constant in the
sample, but which may be safely stored on both sides.</p>
<h3>3. Publish-subscribe for events</h3>
<p>In event-driven architectures, the <em>publish-subscribe</em> messaging
pattern is a way of letting senders (called <em>publishers</em>) transmit
messages to their receivers (called <em>subscribers</em>), without any prior
knowledge of who those subscribers are.<br />
In practice, the <em>subscribers</em> will express interest for a set of
messages, which will be sent by the <em>publisher</em> to all the
<em>subscribers</em> of a given message, as soon as it is be notified.</p>
<p>In our <em>Client-Server services via interfaces</em> implementation,
messages are gathered in <code>interface</code> types, and each message defined
as a single method, their content being the methods parameters.<br />
Most of the SOA alternative (in Java or C#) do require class definition for
messages. Our KISS approach will just use method parameters values as message
definition.</p>
<p>To maintain a list of <em>subscribers</em>, the easiest is to store a
<em>dynamic array</em> of <code>interface</code> instances, on the
<em>publisher</em> side.</p>
<h3>Defining the interfaces</h3>
<p>We will now implement a simple <em>chat</em> service, able to let several
clients communicate together, broadcasting any message to all the other
connected instances.<br />
This sample is also located in the the <code><a href="https://github.com/synopse/mORMot/tree/master/SQLite3/Samples/31%20-%20WebSockets">
SQLite3\Samples\31 - WebSockets</a></code> sub-folder, as
<code>Project31ChatServer.dpr</code> and
<code>Project31ChatClient.dpr</code>.</p>
<p>So you first define the callback interface, and the service interface:</p>
<pre>
<strong>type</strong>
IChatCallback = <strong>interface</strong>(IInvokable)
['{EA7EFE51-3EBA-4047-A356-253374518D1D}']
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo, msg: <strong>string</strong>);
<strong>end</strong>;
<br /> IChatService = <strong>interface</strong>(IInvokable)
['{C92DCBEA-C680-40BD-8D9C-3E6F2ED9C9CF}']
<strong>procedure</strong> Join(<strong>const</strong> pseudo: <strong>string</strong>; <strong>const</strong> callback: IChatCallback);
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo,msg: <strong>string</strong>);
<strong>procedure</strong> CallbackReleased(<strong>const</strong> callback: IInvokable);
<strong>end</strong>;
</pre>
<p>Those interface types will be shared by both server and client sides, in the
common <code>Project31ChatCallbackInterface.pas</code> unit. The definition is
pretty close to what we wrote just above to notify long term
end-of-process.<br />
The only additional method is <code>IChatServer.CallbackReleased()</code>,
which, by convention, will be called on the server side when any
<code>callback</code> interface instance is released on the client side.</p>
<p>As such, the <code>IChatService.Join()</code> method will implement the
<em>subscription</em> to the chat service, whereas
<code>IChatServer.CallbackReleased()</code> will be called when the client-side
callback instance will be released (i.e. when its variable will be assigned to
<code>nil</code>), to <em>unsubscribe</em> for the chat service.</p>
<h3>Writing the Publisher</h3>
<p>On the server side, each call to <code>IChatService.Join()</code> would
<em>subscribe</em> to an internal list of connections, simply stored as an
<code>array of IChatCallback</code>:</p>
<pre>
<strong>type</strong>
TChatService = <strong>class</strong>(TInterfacedObject,IChatService)
<strong>protected</strong>
<span style="background-color:yellow;">fConnected: <strong>array of</strong> IChatCallback;</span>
<strong>public</strong>
<strong>procedure</strong> Join(<strong>const</strong> pseudo: <strong>string</strong>; <strong>const</strong> callback: IChatCallback);
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo,msg: <strong>string</strong>);
<strong>procedure</strong> CallbackReleased(<strong>const</strong> callback: IInvokable);
<strong>end</strong>;
<br /><strong>procedure</strong> TChatService.Join(<strong>const</strong> pseudo: <strong>string</strong>;
<strong>const</strong> callback: IChatCallback);
<strong>begin</strong>
<span style="background-color:yellow;">InterfaceArrayAdd(fConnected,callback);</span>
<strong>end</strong>;
</pre>
<p>The <code>InterfaceArrayAdd()</code> function, as defined in
<code>SynCommons.pas</code>, is a simple wrapper around any <em>dynamic
array</em> of <code>interface</code> instances, so that you may use it, or the
associated <code>InterfaceArrayFind()</code> or
<code>InterfaceArrayDelete()</code> functions, to maintain the list of
subscriptions.</p>
<p>Then a remote call to the <code>IChatService.BlaBla()</code> method should
be broadcasted to all connected clients, just by calling the
<code>IChatCallback.BlaBla()</code> method:</p>
<pre>
<strong>procedure</strong> TChatService.BlaBla(<strong>const</strong> pseudo,msg: <strong>string</strong>);
<strong>var</strong> i: integer;
<strong>begin</strong>
<strong>for</strong> i := 0 <strong>to</strong> high(fConnected) <strong>do</strong>
<span style="background-color:yellow;">fConnected[i].BlaBla(pseudo,msg);</span>
<strong>end</strong>;
</pre>
<p>Note that every call to <code>IChatCallback.BlaBla()</code> within the loop
would be made via <em>WebSockets</em>, in an asynchronous and non blocking way,
so that even in case of huge number of clients, the
<code>IChatService.BlaBla()</code> method won't block.<br />
In case of high numbers of messages, the framework is even able to
<em>gather</em> push notification messages into a single bigger message, to
reduce the resource use.</p>
<p>On the server side, the service implementation has been registered as
such:</p>
<pre>
Server.ServiceDefine(TChatService,[IChatService],sicShared).
SetOptions([],[optExecLockedPerInterface]);
</pre>
<p>Here, the <code>optExecLockedPerInterface</code> option has been set, so
that all method calls would be made thread-safe, so that concurrent access to
the internal <code>fConnected[]</code> list would be safe.<br />
Since a global list of connections is to be maintained, the service life time
is defined as <code>sicShared</code> - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_92">
Instances life time implementation</a></em>.</p>
<p>The following method will be called by the server, when a client callback
instance is released (either explicitly, or if the connection is broken), so
could be used to <em>unsubscribe</em> to the notification, simply by deleting
the callback from the internal <code>fConnected[]</code> array:</p>
<pre>
<strong>procedure</strong> TChatService.CallbackReleased(<strong>const</strong> callback: IInvokable);
<strong>begin</strong>
<span style="background-color:yellow;">InterfaceArrayDelete(fConnected,callback);</span>
<strong>end</strong>;
</pre>
<p>The framework will in fact recognize the following method definition in any
<code>interface</code> type for a service:</p>
<pre>
<strong>procedure</strong> CallbackReleased(<strong>const</strong> callback: IInvokable);
</pre>
<p>When a callback <code>interface</code> parameter (in our case,
<code>IChatCallback</code>) will be released on the client side, this method
will be called with the corresponding <code>interface</code> instance as
parameter.<br />
You do not have to call explicitly any method on the client side to
<em>unsubscribe</em> a service: assigning <em>nil</em> to a callback variable,
or feeing the <code>class</code> instance owning it as a field on the
<em>subscriber</em> side, will automatically unregister it on the
<em>publisher</em> side.</p>
<h3>Consuming the service from the Subscriber side</h3>
<p>On the client side, you implement the <code>IChatCallback</code> callback
interface:</p>
<pre>
<strong>type</strong>
TChatCallback = <strong>class</strong>(TInterfacedCallback,IChatCallback)
<strong>protected</strong>
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo, msg: <strong>string</strong>);
<strong>end</strong>;
<br /><strong>procedure</strong> TChatCallback.BlaBla(<strong>const</strong> pseudo, msg: <strong>string</strong>);
<strong>begin</strong>
writeln(#13'@',pseudo,' ',msg);
<strong>end</strong>;
</pre>
<p>Then you subscribe to your remote service as such:</p>
<pre>
<span style="background-color:yellow;"><strong>var</strong> Service: IChatService;</span>
<span style="background-color:yellow;">callback: IChatCallback;</span>
...
Client.ServiceDefine([IChatService],sicShared);
<span style="background-color:yellow;"><strong>if not</strong> Client.Services.Resolve(IChatService,Service) <strong>then</strong></span>
<strong>raise</strong> EServiceException.Create('Service IChatService unavailable');
...
<span style="background-color:yellow;">callback := TChatCallback.Create(Client,IChatCallback);</span>
<span style="background-color:yellow;">Service.Join(pseudo,callback);</span>
...
<strong>try</strong>
<strong>repeat</strong>
readln(msg);
<strong>if</strong> msg='' <strong>then</strong>
break;
<span style="background-color:yellow;">Service.BlaBla(pseudo,msg);</span>
<strong>until</strong> false;
<strong>finally</strong>
<span style="background-color:yellow;">callback := <strong>nil</strong>; <em>// will unsubscribe from the remote publisher</em></span>
Service := <strong>nil</strong>; <em>// release the service local instance BEFORE Client.Free</em>
<strong>end</strong>;
</pre>
<p>You could easily implement more complex <em>publish/subscribe</em>
mechanisms, including filtering, time to live or tuned broadcasting, by storing
some additional information to the <code>interface</code> instance (e.g. some
value to filter, a timestamp). A dynamic array of dedicated
<code>record</code>s, or a list of <code>class</code> instances, may be used to
store the <em>subscribers</em> expectations.</p>
<p>If you compare with existing client/server SOA solutions (in Delphi, Java,
C# or even in Go or other frameworks), this <code>interface</code>-based
callback mechanism sounds pretty unique and easy to work with.<br />
In fact, this is a good way of implementing callbacks conforming to <em>SOLID
design principles</em> on the server side, and let the <em>mORMot</em>
framework publish this mechanism in a client/server way, by using
<em>WebSockets</em>. The very same code could be used on the server side, with
no transmission nor marshaling overhead (via direct <code>interface</code>
calls), and over a network, with optimized use of resource and bandwidth (via
"fake" <code>interface</code> calls, and JSON marshalling over TCP/IP).</p>
<div style="margin-left: 2em">Ensure you <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_149">
take a look at the corresponding online documentation</a>, which would be
updated often than this blog article!<br />
The ORM part of the framework uses this <em>Publish-Subscribe</em> feature to
implement <a href="https://blog.synopse.info?post/post/2015/04/06/Real-Time-ORM-Master/Slave-Replication-via-WebSockets">Real-Time
master/slave replication</a>.<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=15632#p15632">welcome on our
forum</a>, as usual.</div>Real-Time ORM Master/Slave Replication via WebSocketsurn:md5:1709c8e98682a9bc997b21cade0a10352015-04-06T21:01:00+02:002015-04-06T20:09:38+02:00AB4327-GANDImORMot FrameworkAJAXblogDatabaseDelphiDocumentationEventCollaborationEventSourcinginterfaceModelORMreplicationSOLIDSourceSQLite3SynopseWebSockets<p>In a <a href="https://blog.synopse.info?post/post/2015/03/31/ORM-Master/Slave-Replication">previous
article</a>, we presented how Master/Slave replication may be easily
implemented in <em>mORMot</em>'s RESTful ORM.<br />
Do not forget to <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_147">
visit the corresponding paragraphs of our online documentation</a>, which has
been updated, and is more accurate!</p>
<p><img src="http://www.ibm.com/developerworks/library/wa-reverseajax2/fig01.gif" alt="" /></p>
<p>Sometimes, the on-demand synchronization is not enough.<br />
So we have just introduced real-time replication via <em>WebSockets</em>.<br />
For instance, you may need to:</p>
<ul>
<li>Synchronize a short list of always evolving items which should be reflected
as soon as possible;</li>
<li>Involve some kind of ACID-like behavior (e.g. handle money!) in your
replicated data;</li>
<li>Replicate not from a GUI application, but from a service, so use of a
<code>TTimer</code> is not an option;</li>
<li>Combine REST requests (for ORM or services) and master/slave ORM
replication on the same wire, e.g. in a multi-threaded application.</li>
</ul>
<p>In this case, the framework is able to use <em>WebSockets</em> and
asynchronous callbacks to let the master/slave replication - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_149">
Asynchronous callbacks</a></em> - take place without the need to ask explicitly
for pending data.<br />
You would need to use
<code>TSQLRestServer.RecordVersionSynchronizeMasterStart</code>,
<code>TSQLRestServer.RecordVersionSynchronizeSlaveStart</code> and
<code>TSQLRestServer.RecordVersionSynchronizeSlaveStop</code> methods over the
proper kind of bidirectional connection.</p> <p>The first requirement is to allow <em>WebSockets</em> on your
<em>Master</em> HTTP server, so initialize the <code>TSQLHttpServer</code>
class as a <code>useBidirSocket</code> kind of server - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_140">
Network and Internet access via HTTP</a></em>:</p>
<pre>
MasterServer := TSQLRestServerDB.Create(MasterModel,'master.db3');
HttpMasterServer := TSQLHttpServer.Create('8888',[MasterServer],'+',useBidirSocket);
HttpMasterServer.WebSocketsEnable(Server,'PrivateAESEncryptionKey');
</pre>
<p>On the <em>Slave</em> side, the HTTP client should also be upgraded to
support <em>WebSockets</em>:</p>
<pre>
MasterClient := TSQLHttpClientWebsockets.Create('127.0.0.1',HTTP_DEFAULTPORT,MasterModel);
MasterClient.WebSocketsUpgrade('PrivateAESEncryptionKey');
</pre>
<p>Of course, the model should match for both <code>MasterServer</code> and
<code>MasterClient</code> instances.<br />
As the <em>WebSockets</em> protocol definition - here above the same
<code>'PrivateAESEncryptionKey'</code> private key.</p>
<p>Then you enable the real-time replication service on the <em>Master</em>
side:</p>
<pre>
MasterServer.RecordVersionSynchronizeMasterStart;
</pre>
<p>In practice, it will publish a <code>IServiceRecordVersion
interface</code>-based service on the server side - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_63">
Client-Server services via interfaces</a></em>.</p>
<p>Assuming that the <em>slave</em> database has been defined as such:</p>
<pre>
SlaveServer := TSQLRestServerDB.Create(SlaveModel,'slave.db3');
</pre>
<p>(in this case, the <code>SlaveModel</code> may not be the same as the
<code>MasterModel</code>, but <code>TSQLRecordPeopleVersioned</code> should be
part of both models)<br />
Then you can initiate real-time replication from the <em>slave</em> side with a
single line, for a given table:</p>
<pre>
SlaveServer.RecordVersionSynchronizeSlaveStart(TSQLRecordPeopleVersioned,MasterClient);
</pre>
<p>The above command will subscribe to the remote <code>MasterSlave</code>
replication service (i.e. <code>IServiceRecordVersion interface</code>), to
receive any change concerning the <code>TSQLRecordPeopleVersioned</code> ORM
table, using the <code>MasterClient</code> connection via <em>WebSockets</em>,
and persist all updates into the local <code>SlaveServer</code> database.</p>
<p>To stop the real-time notification for this ORM table, you could
execute:</p>
<pre>
SlaveServer.RecordVersionSynchronizeSlaveStop(TSQLRecordPeopleVersioned);
</pre>
<p>Even if you do not call <code>RecordVersionSynchronizeSlaveStop()</code>,
the replication will be stopped when the main <code>SlaveServer</code> instance
will be released, and the <code>MasterServer</code> be <em>unsubscribe</em>
this connection for its internal notification list.</p>
<p>The real-time notification details have been tuned, to consume as minimum
bandwidth and resources as possible.<br />
For instance, if several modifications are to be notified on a slave connection
in a short amount of time, the master is able to gather those modifications as
a single <em>WebSockets</em> frame, which would be applied as a whole to the
slave database, in a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_28">
single BATCH transaction</a>.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=15577#p15577">welcome in our
forum</a>, as usual!</p>ORM Master/Slave Replicationurn:md5:08d531e552853cbab93bd086a1bab2902015-03-31T22:40:00+02:002015-04-05T07:27:38+02:00AB4327-GANDImORMot FrameworkbackupBatchblogDatabaseDelphiDocumentationGoodPracticeMaster-SlavemORMotreplicationRestshardingSourceSQLite3<p>As stated during <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_26">
TSQLRecord fields definition</a></em>, the ORM is able to maintain a revision
number for any <code>TSQLRecord</code> table, so that it the table may be
easily synchronized remotely by another <code>TSQLRestServer</code>
instance.<br />
If you define a <code>TRecordVersion</code> published property, the ORM core
will fill this field just before any write with a monotonically increasing
revision number, and will take care of any deletion, so that those
modifications may be replayed later on any other database.</p>
<p><img src="http://i41.tinypic.com/35lw00x.jpg" alt="" /></p>
<p>This synchronization will work as a strict <em>master/slave replication</em>
scheme, as a one-way on demand refresh of a replicated table.<br />
Each write operation on the master database on a given table may be easily
reflected on one or several slave databases, with almost no speed nor storage
size penalty.</p> <h3>Enable synchronization</h3>
<p>In order to enable this replication mechanism, you should define a
<code>TRecordVersion</code> published property in the <code>TSQLRecord</code>
class type definition:</p>
<pre>
TSQLRecordPeopleVersioned = <strong>class</strong>(TSQLRecordPeople)
<strong>protected</strong>
fFirstName: RawUTF8;
fLastName: RawUTF8;
fVersion: TRecordVersion;
<strong>published</strong>
<strong>property</strong> FirstName: RawUTF8 <strong>read</strong> fFirstName <strong>write</strong> fFirstName;
<strong>property</strong> LastName: RawUTF8 <strong>read</strong> fLastName <strong>write</strong> fLastName;
<strong>property</strong> Version: TRecordVersion <strong>read</strong> fVersion <strong>write</strong> fVersion;
<strong>end</strong>;
</pre>
<p>Only a single <code>TRecordVersion</code> field is allowed per
<code>TSQLRecord</code> class - it would not mean anything to manage more than
one field of this type.</p>
<p>Note that this field will be somewhat "hidden" to most ORM process: a
regular <code>TSQLRest.Retrieve</code> won't fill this <code>Version</code>
property, since it is an internal implementation detail.<br />
If you want to lookup its value, you would have to explicitly state its field
name at retrieval. Any <code>TRecordVersion</code> is indeed considered as a
"non simple field", just like BLOB fields, so would need explicit retrieval of
its value.</p>
<p>In practice, any <code>TSQLRest.Add</code> and <code>TSQLRest.Update</code>
on this <code>TSQLRecordPeopleVersioned</code> class will increase this
<code>Version</code> revision number field, and a <code>TSQLRest.Delete</code>
will populate an external <code>TSQLRecordTableDelete</code> table with the
<code>ID</code> of the deleted record, associated with a
<code>TRecordVersion</code> revision.</p>
<p>As consequences:</p>
<ul>
<li>The monotonic <code>TRecordVersion</code> number is shared at
<code>TSQLRestServer</code> level, among all tables containing a
<code>TRecordVersion</code> published field;</li>
<li>The <code>TSQLRecordTableDelete</code> table should be part of the
<code>TSQLModel</code>, in conjunction with
<code>TSQLRecordPeopleVersioned</code>;</li>
<li>If the <code>TSQLRecordTableDelete</code> table is not part of the
<code>TSQLModel</code>, the <code>TSQLRestServer</code> will add it - but you
should better make it explicitly appearing in the data model;</li>
<li>A single <code>TSQLRecordTableDelete</code> table will maintain the list of
all deleted data rows, of all tables containing a <code>TRecordVersion</code>
published field;</li>
<li>The <code>TSQLRecordPeopleVersioned</code> table appearance order in the
<code>TSQLModel</code> will matter, since <code>TSQLRecordTableDelete.ID</code>
will use this table index order in the database model to identify the table
type of the deleted row - in a similar way to <em>TRecordReference and
TRecordReferenceToBeDeleted</em>.</li>
</ul>
<p>All the synchronization preparation will be taken care by the ORM kernel on
its own, during any write operation. There is nothing particular to maintain or
setup, in addition to this <code>TRecordVersion</code> field definition, and
the global <code>TSQLRecordTableDelete</code> table.</p>
<h3>From master to slave</h3>
<p>To replicate this <code>TSQLRecordPeopleVersioned</code> table from another
<code>TSQLRestServer</code> instance, just call the following method:</p>
<pre>
aServer.RecordVersionSynchronizeSlave(TSQLRecordPeopleVersioned,aClient);
</pre>
<p>This single line will request a remote server via a <code>Client:
TSQLRestClientURI</code> connection (which may be over HTTP) for any pending
modifications since its last call, then will fill the local <code>aServer:
TSQLRestServer</code> database so that the local
<code>TSQLRecordPeopleVersioned</code> table will contain the very same content
as the remote master <code>TSQLRestServer</code>.</p>
<p>You can safely call
<code>TSQLRestServer.RecordVersionSynchronizeSlave</code> from several clients,
to replicate the master data in several databases.</p>
<p>Only the modified data will be transmitted over the wire, as two REST/JSON
queries (one for the insertions/updates, another for the deletions), and all
the local write process will use optimized <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_78">
BATCH writing</a>. This means that the synchronization process will try to use
as minimal bandwidth and resources as possible, on both sides.</p>
<p>Of course, the slaves should be considered as read-only, otherwise the
version numbers may conflict, and the whole synchronization may become a
failure.<br />
But you can safely replicate servers in cascade, if needed: the version numbers
will be propagated from masters to slaves, and the data will always be in a
consistent way.</p>
<h3>Replication use cases</h3>
<p>We may consider a very common corporate infrastructure:</p>
<p><a href="https://blog.synopse.info?post/public/mORMot/mORMotMasterSlaveReplication.png"><img src="https://blog.synopse.info?post/public/mORMot/.mORMotMasterSlaveReplication_m.jpg" alt="Master Slave ORM Replication" title="Master Slave ORM Replication, Mar 2015" /></a></p>
<p>This kind of installation, with a main central office, and a network of
local offices, would benefit from this master/slave replication.<br />
Simple <em>redirection</em> may be used - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_93">
Redirect to an external TSQLRest</a></em> - but it would expect the work to
continue, even in case of <em>Internet</em> network failure.<br />
REST redirection would expect a 100% connection uplink, which may be critical
in some cases.</p>
<p>You could therefore implement replication in several ways:</p>
<ul>
<li>Either the main office is the master, and any write would be push to the
<em>Main Server</em>, whereas local offices would have a replicated copy of the
information - drawback is that in case of network failure, the local office
would be able to only read the data;</li>
<li>Or each local office may host its own data in a dedicated table,
synchronized as a master database; the main office will replicate (as a slave)
the private data of each local servers; in addition, all this data gathered by
the <em>Main Server</em> may be further replication to the other local offices,
and be still accessible in read mode - in case of network failure, all the data
is available on the local servers, and the local private table is still
writable.</li>
</ul>
<p>Of course, the second solution seems preferable, even if a bit more
difficult to implement. The ablity of all local offices to work offline on
their own private data, but still having all the other data accessible as
read-only, would be a huge ROI.</p>
<p>As a benefit of using replication, the central main server would be less
stressed, since most of the process would take place in local servers, and the
main office server would only be used for shared data backup and read-only
gathering of the other local databases. Only a small network bandwith would be
necessary (much less than a pure web solution), and CPU/storage resources would
be minimal.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=11863#p11863">welcome on our
forum</a>, as usual!<br />
See also <a href="http://synopse.info/fossil/info/3453f314d9">the associated
feature request</a>, for upcoming features related to replication.<br />
And the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_147">
corresponding documentation article</a>, which will be updated often, so is
preferred to this static blog article!</p>ODM magic: complex queries over NoSQL / MongoDBurn:md5:795ff9d748ddac411e5d1e63588427d72014-11-28T17:37:00+01:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkblogDatabaseJSONMongoDBmORMotNoSQLODMORMRestSQL<p>You know that our <em>mORMot</em> is <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">able to access directly any
<em>MongoDB</em> database engine</a>, allowing its ORM to become an ODM,
and using NoSQL instead of SQL for the query languages.</p>
<p><img src="https://cdn.tutsplus.com/net/uploads/legacy/1091_mongodb/preview.jpg" alt="" /></p>
<p>But at <em>mORMot</em> level, you could share the same code between your
RDBMS and NoSQL databases.<br />
The ORM/ODM is able to do all the conversions by itself!<br />
Since <a href="http://synopse.info/fossil/info/abbdfa5563">we have just
improved this feature</a>, it is time to enlighten its current status.</p> <p>To perform a query and retrieve the content of several documents, you can
use regular <code>CreateAndFillPrepare</code> or <code>FillPrepare</code>
methods:</p>
<pre>
<span style="background-color:yellow;">R := TSQLORM.CreateAndFillPrepare(Client,WHERE_CLAUSE,[WHERE_PARAMETERS]);</span>
<strong>try</strong>
n := 0;
<span style="background-color:yellow;"><strong>while</strong> R.FillOne <strong>do begin</strong></span>
<em>// here R instance contains all values of one document, excluding BLOBs</em>
inc(n);
<strong>end</strong>;
assert(n=COLL_COUNT);
<strong>finally</strong>
R.Free;
<strong>end</strong>;
</pre>
<p>A WHERE clause can also be defined for <code>CreateAndFillPrepare</code> or
<code>FillPrepare</code> methods.<br />
This WHERE clause could contain several expressions, joined with
<code>AND</code> / <code>OR</code>.<br />
Each of those expressions could use:</p>
<ul>
<li>The simple comparators <code>= < <= <> > >=</code>,</li>
<li>An <code>IN (....)</code> clause,</li>
<li><code>IS NULL</code> / <code>IS NOT NULL</code> tests,</li>
<li>A <code>LIKE</code> operation,</li>
<li>Or even any <code>...DynArrayContains()</code> specific function.</li>
</ul>
<p>The <em>mORMot</em> ODM will convert this SQL-like statement into the
optimized <code>MongoDB</code> query expression, using e.g. a <em>regular
expression</em> for the <code>LIKE</code> operator.</p>
<p>The <code>LIMIT</code>, <code>OFFSET</code> and <code>ORDER BY</code>
clauses will also be handled as expected.<br />
A special care would be taken for an <code>ORDER BY</code> on textual values:
by design, <em>MongoDB</em> will always sort text with case-sensitivity, which
is not what we expect: so our ODM will sort such content on client side, after
having been retrieved from the <em>MongoDB</em> server. For numerical fields,
<em>MongoDB</em> sorting features would be processed on the server side.</p>
<p>The <code>COUNT(*)</code> function would also be converted into the proper
<em>MongoDB</em> API call, so that such operations would be as costless as
possible.<br />
<code>DISTINCT() MAX() MIN() AVG()</code> functions and the <code>GROUP
BY</code> clause would also be converted into optimized <em>MongoDB</em>
aggregation pipelines, on the fly.<br />
You could even set aliases for the columns (e.g. <code>max(RowID) as
first</code>) and perform simple addition/substraction of an integer value.</p>
<p>Here are some typical WHERE clauses, and the corresponding <em>MongoDB</em>
query document as generated by the ODM:</p>
<table>
<tbody>
<tr>
<td><strong>WHERE clause</strong></td>
<td><strong><em>MongoDB</em> Query</strong></td>
</tr>
<tr>
<td><code>'Name=?',['Name 43']</code></td>
<td><code>{Name:"Name 43"}</code></td>
</tr>
<tr>
<td><code>'Age<?',[51]</code></td>
<td><code>{Age:{$lt:51}}</code></td>
</tr>
<tr>
<td><code>'Age in (1,10,20)'</code></td>
<td><code>{Age:{$in:[1,10,20]}}</code></td>
</tr>
<tr>
<td><code>'Age in (1,10,20) and ID=?',[10]</code></td>
<td><code>{Age:{$in:[1,10,20]},_id:10}</code></td>
</tr>
<tr>
<td><code>'Age in (10,20) or ID=?',[30]</code></td>
<td><code>{$or:[{Age:{$in:[10,20]}},{_id:30}]}</code></td>
</tr>
<tr>
<td><code>'Name like ?',['name 1%']</code></td>
<td><code>{Name:/^name 1/i}</code></td>
</tr>
<tr>
<td><code>'Name like ?',['name 1']</code></td>
<td><code>{Name:/^name 1$/i}</code></td>
</tr>
<tr>
<td><code>'Name like ?',['%ame 1%']</code></td>
<td><code>{Name:/ame 1/i}</code></td>
</tr>
<tr>
<td><code>'Data is null'</code></td>
<td><code>{Data:null}</code></td>
</tr>
<tr>
<td><code>'Data is not null'</code></td>
<td><code>{Data:{$ne:null}}</code></td>
</tr>
<tr>
<td><code>'Age<? limit 10',[51]</code></td>
<td><code>{Age:{$lt:51}} + limit 10</code></td>
</tr>
<tr>
<td><code>'Age in (10,20) or ID=? order by ID desc',[30]</code></td>
<td>
<code>{$query:{$or:[{Age:{$in:[10,20]}},{_id:30}]},$orderby:{_id:-1}}</code></td>
</tr>
<tr>
<td><code>'order by Name'</code></td>
<td><code>{} + client side text sort by Name</code></td>
</tr>
<tr>
<td><code>'Age in (1,10,20) and
IntegerDynArrayContains(Ints,?)',[10])</code></td>
<td><code>{Age:{$in:[1,10,20]},Ints:{$in:[10]}}</code></td>
</tr>
<tr>
<td><code>Distinct(Age),max(RowID) as first,count(Age) as count group by
age</code></td>
<td>
<code>{$group:{_id:"$Age",f1:{$max:"$_id"},f2:{$sum:1}}},{$project:{_id:0,"Age":"$_id","first":"$f1","count":"$f2"}}</code></td>
</tr>
<tr>
<td><code>min(RowID),max(RowID),Count(RowID)</code></td>
<td>
<code>{$group:{_id:null,f0:{$min:"$_id"},f1:{$max:"$_id"},f2:{$sum:1}}},{$project:{_id:0,"min(RowID)":"$f0","max(RowID)":"$f1","Count(RowID)":"$f2"}}</code></td>
</tr>
<tr>
<td><code>min(RowID) as a,max(RowID)+1 as b,Count(RowID) as c</code></td>
<td>
<code>{$group:{_id:null,f0:{$min:"$_id"},f1:{$max:"$_id"},f2:{$sum:1}}},{$project:{_id:0,"a":"$f0","b":{$add:["$f1",1]},"c":"$f2"}}</code></td>
</tr>
</tbody>
</table>
<p>Note that parenthesis and mixed <code>AND OR</code> expressions are not
handled yet.<br />
You could always execute any complex query (e.g. aggregations or
<em>Map/Reduce</em>) by using directly the <code>TMongoCollection</code>
methods.</p>
<p>But for most cases, <em>mORMot</em> allows to share the same exact code
between your regular SQL databases or NoSQL engines.<br />
You do not need to learn the <em>MongoDB</em> query syntax: the ODM would
compute the right expression for you, depending on the database engine it runs
on.</p>
<p>Ensure you <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_83">
took a look at the updated documentation</a> to better understand NoSQL/MongoDB
integration with <em>mORMot</em>.<br />
And feel free to <a href="http://synopse.info/forum/">use our forum</a> for
feedback, as usual!</p>
<p>Enjoy the <em>NoSQL</em> world!</p>HTTP remote access for SynDB SQL executionurn:md5:8ac51ed0a6e45f22746c8ba9e6d3e63c2014-11-18T01:08:00+01:002014-11-18T01:11:52+01:00AB4327-GANDIOpen Source librariesarray bindingblogcompressionDatabaseDelphiFirebirdFireDACHTTPhttp.sysHTTPSmORMotMSSQLMySQLOpenSourceOracleORMperformancesecuritySQLSynDBSynDBExplorerTDataSetwebWinHTTPWinINetZEOS<p>For <em>mORMot</em>, we developed a fully feature direct access layer to any
RDBMS, implemented <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_126">
in the SynDB.pas unit</a>.</p>
<p>You can use those <code>SynDB</code> classes to execute any SQL statement,
without any link to the framework ORM.<br />
At reading, the resulting performance is much higher than using the standard
<code>TDataSet</code> component, which is in fact a true performance
bottleneck.<br />
It has genuine features, like <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_136">
column access via late-binding</a>, <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_137">
an innovative <code>ISQLDBRows</code> interface, and ability to directly access
the low-level binary buffers of the database clients.</a></p>
<p><img src="http://vpn-services.bestreviews.net/files/vpn-remote-access.jpg" alt="" /></p>
<p>We just added a nice feature to those classes: the ability to access
remotely, via plain HTTP, to any <code>SynDB</code> supported database!</p> <p>To define a HTTP server, you may write:</p>
<pre>
<strong>uses</strong> SynDB, <em>// RDBMS core</em>
SynDBSQLite3, SynSQLite3Static, <em>// static SQLite3 engine</em>
SynDBRemote; <em>// for HTTP server</em>
...
<strong>var</strong> Props: TSQLDBConnectionProperties;
HttpServer: TSQLDBServerAbstract;
...
Props := TSQLDBSQLite3ConnectionProperties.Create('data.db3','','','');
<span style="background-color:yellow;">HttpServer := TSQLDBServerHttpApi.Create(Props,'remote','8092','user','pass');</span>
</pre>
<p>The above code will initialize a connection to a local <code>data.db3</code>
<em>SQlite3</em> database (in the <code>Props</code> variable), and then
publish it using the <code>http.sys</code> kernel mode HTTP server to the
<code>http://1.2.3.4:8092/remote</code> URI - if the server's IP is
<code>1.2.3.4</code>.</p>
<p>On the client side, you can then write:</p>
<pre>
<strong>uses</strong> SynDB, <em>// RDBMS core</em>
SynDBRemote; <em>// for HTTP server</em>
...
<strong>var</strong> Props: TSQLDBConnectionProperties;
...
Props := TSQLDBWinHTTPConnectionProperties.Create('1.2.3.4:8092','root','user','pass');
</pre>
<p>As you can see, there is no link to <code>SynDBSQLite3.pas</code> nor
<code>SynSQLite3Static.pas</code> on the client side.<br />
Just the HTTP link is needed.<br />
No need to deploy the RDBMS client libraries with your application, nor setup
the local network firewall.</p>
<p>We defined here a single user, with 'user' / 'pass' credentials, but you may
manage more users on the server side, using the <code>Authenticate</code>
property of <code>TSQLDBServerAbstract</code>.<br />
Note that in our remote access, user management does not match the RDBMS user
rights: you should better have your own set of users at application level, for
higher security, and a better integration with your business logic. If creating
a new user on a RDBMS could be painful, it is pretty easy on our
<code>SynDBRemote.pas</code> side.</p>
<p>Then, you execute your favorite SQL using the connection just as usual:</p>
<pre>
<strong>procedure</strong> Test(Props: TSQLDBConnectionProperties);
<strong>var</strong> Stmt: ISQLDBRows;
<strong>begin</strong>
Stmt := Props.Execute('select * from People where YearOfDeath=?',[1519]);
<strong>while</strong> Stmt.Step <strong>do begin</strong>
assert(Stmt.ColumnInt('ID')>0);
assert(Stmt.ColumnInt('YearOfDeath')=1519);
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>You may even use this remote connection e.g. using a stand-alone shared
<em>SQLite3</em> database as high performance but low maintenance client-server
database engine.</p>
<p>The transmission protocol uses an optimized binary format, which is
compressed and digitally signed on both ends, and the remote user
authentication will be performed via a challenge validation scheme.<br />
You could publish your server over HTTPS, if needed.</p>
<p>Even if you may be tempted to use such remote access to implement a
<em>n-Tier architecture</em>, you should rather use <em>mORMot</em>'s
Client-Server ORM instead. Our little <em>mORMot</em> is not an ORM on
which we added a data transmission layer: it is a full RESTful system, with a
true SOA design.<br />
But for integrating some legacy SQL code into a new architecture,
<code>SynDBRemote.pas</code> may have its benefits, used in conjunction with
<em>mORMot</em>'s higher level features.</p>
<p>Feel free to <a href="http://synopse.info/forum/viewtopic.php?id=2178">post
your feedback on our forum</a>, as usual, or <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_131">
browse the updated documentation</a>!</p>BREAKING CHANGE - TSQLRecord.ID primary key changed to TID: Int64urn:md5:efba053f92e0e5a1217d27e7f7d1e0832014-11-14T15:11:00+01:002014-11-14T15:15:13+01:00AB4327-GANDImORMot Framework64bitblogDatabaseDelphiDocumentationmORMotORMPrimaryKeyRestSourceSQLSQLite3<p>Up to now, the <code>TSQLRecord.ID</code> property was defined in
<code>mORMot.pas</code> as a plain
<code>PtrInt</code>/<code>NativeInt</code> (i.e. <code>Integer</code> under
Win32), since it was type-cast as pointer for <code>TSQLRecord</code> published
properties.<br />
We introduced a new <code>TID</code> type, so that the ORM primary key would
now be defined as <code>Int64</code>.</p>
<p><img src="http://blog.codinghorror.com/content/images/uploads/2007/11/6a0120a85dcdae970b012877701d33970c-pi.png" alt="" width="415" height="145/" /></p>
<p>All the framework ORM process relies on the <code>TSQLRecord</code>
class.<br />
This abstract <code>TSQLRecord</code> class features a lot of built-in methods,
convenient to do most of the ORM process in a generic way, at record level.</p>
<p>It first defines a <em>primary key</em> field, defined as <code>ID:
TID</code>, i.e. as <code>Int64</code> in <code>mORMot.pas</code>:</p>
<pre>
<strong>type</strong>
TID = <strong>type</strong> Int64;
...
TSQLRecord = <strong>class</strong>(TObject)
...
<strong>property</strong> ID: TID <strong>read</strong> GetID <strong>write</strong> fID;
...
</pre>
<p>In fact, our ORM relies now on a <code>Int64</code> primary key, matching
the <em>SQLite3</em> <code>ID</code>/<code>RowID</code> primary key.<br />
This primary key will be used as <a href="https://blog.synopse.info?post/post/2014/01/10/REpresentational-State-Transfer-%28REST%29">RESTful resource
identifier</a>, for all CRUD operations.</p> <h3>Limitation or feature?</h3>
<p>You may be disappointed by this limitation, which is needed by the
<em>SQLite3</em>'s implementation of Virtual Tables - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_20">
Virtual Tables magic</a></em>.</p>
<p>We won't debate about a composite primary key (i.e. several fields), which
is not a good idea for an ORM.<br />
In your previous RDBMS data modeling, you may be used to define a TEXT primary
key, or even a GUID primary key: those kinds of keys are somewhat less
efficient than an INTEGER, especially for ORM internals, since they are not
monotonic.<br />
Note that you can always define a secondary key, as <code>string</code> or
<code>TGUID</code> field, if needed - using <code>stored AS_UNIQUE</code>
attribute as explained below.</p>
<p>Having <code>Int64</code> wide primary key will allow to compute huge IDs,
which was found to be almost mandatory to implement <a href="http://synopse.info/fossil/tktview/3453f314d97d">safe multi-node
replication</a>.</p>
<h3>Introducing TID published properties</h3>
<p><code>TSQLRecord</code> published properties do match a class instance
pointer, so are 32 bit (at least for <em>Win32/Linux32</em> executables).<br />
Since the <code>TSQLRecord.ID</code> field is declared as <code>TID =
Int64</code>, we may loose information if the stored <code>ID</code> is greater
than 2,147,483,647 (i.e. a signed 32 bit value).</p>
<p>You can now define a published property as <code>TID</code> to store any
value of our primary key, i.e. up to 9,223,372,036,854,775,808.<br />
Note that in this case, there is no information about the joined table.</p>
<p>As a consequence, the ORM will perform the following optimizations for
<code>TID</code> fields:</p>
<ul>
<li>An <em>index</em> will be created on the database, for the corresponding
column;</li>
<li>When a referenced record is deleted, the ORM <em>won't do anything</em>,
since it has no information about the table to track - this is the main
difference with <code>TSQLRecord</code> published property.</li>
</ul>
<p>See the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_26">
corresponding updated documentation</a>.<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=13588#p13588">welcome on our
forum</a>, as usual.</p>Are "Micro Services" the proper way of writing SOA?urn:md5:74545ff9a5e7f8de80d9b5b918ee16972014-10-25T11:14:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkblogDatabaseDelphiDocumentationGoodPracticeORMRestSOA <p>I just wanted to share a <a href="http://martinfowler.com/articles/microservices.html">great article by Martin
Fowler, about Micro Services</a>.</p>
<p><img src="http://martinfowler.com/articles/microservices/images/sketch.png" alt="" width="425" height="260" /></p>
<p>IMHO such "Micro Services" are the proper way of <a href="https://blog.synopse.info?post/post/2014/01/04/Domain-Driven-Design%3A-part-3">defining a SOA project</a>,
following <a href="https://blog.synopse.info?post/post/2011/11/27/SOLID-design-principles">SOLID
principles</a>.<br />
If we follow the "<em>Single Responsibility</em>" principle, we will define
small uncoupled services, which do one single task. This will fulfill
the "<em>Interface Segregation</em>" principle, and since we rely on
abstractions, the "<em>Liskov Substitution</em>" will also apply. By
definition, the SOA catalog/publish mechanism will perform "<em>Dependency
Inversion</em>".<br />
All this sounds just like the definition of "Micro Services".</p>
<p>Worth reading, and a nice illustration of <a href="https://blog.synopse.info?post/post/2014/05/30/Software-Design%2C-Brook%2C-mORMot%2C-RAD%2C-SOLID-and-OOP">how
to build <em>mORMot</em> applications</a>!</p>MVC/MVVM Web Applicationsurn:md5:0f6dae7b4dfdc4fb48483baae67ae6512014-10-24T19:41:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkblogDatabaseDelphiDocumentationHTMLinterfaceJSONmORMotmultithreadMustacheMVCMVVMOpenSourceORMperformanceSOASource<p>We almost finished implementing a <a href="http://synopse.info/fossil/info/bd94c11ab1">long-standing feature request</a>,
introducing MVC / MVVM for Web Applications (like <a href="http://rubyonrails.org/">RubyOnRails</a>) in <em>mORMot</em>.<br />
This is a huge step forward, opening new perspectives not only to our
framework, but for the Delphi community.<br />
In respect to the <a href="http://stackoverflow.com/q/9853766/458259">existing
MVC frameworks for Delphi</a>, our solution is closer to <a href="https://code.google.com/p/delphionrails/">Delphi On Rails</a> (by the
<a href="http://en.wikipedia.org/wiki/Convention_over_configuration">convention-over-configuration</a>
pattern) than the <a href="https://code.google.com/p/delphimvcframework/">Delphi MVC
Framework</a> or <a href="http://xxm.sourceforge.net/">XMM</a>.<br />
The <em>mORMot</em> point of view is unique, and quite powerful,
since it is integrated with other parts of our framework, as its <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">ORM/ODM</a> or <a href="https://blog.synopse.info?post/post/2012/03/07/Interface-based-services">interface-based services</a>.<br />
Of course, this is a work in progress, so you are welcome to put your feedback,
patches or new features!</p>
<p><img src="http://d2o0t5hpnwv4c1.cloudfront.net/1122_zend2/images/ash-mvc-architecture.gif" alt="" /></p>
<p>We will now explain how to build a <em>MVC/MVVM web application</em> using
<em>mORMot</em>, starting from the "<em><a href="https://github.com/synopse/mORMot/tree/master/SQLite3/Samples/30%20-%20MVC%20Server">30
- MVC Server</a></em>" sample.<br />
First of all, <a href="https://github.com/synopse/mORMot/tree/master/SQLite3/Samples/30%20-%20MVC%20Server">
check the source in our GitHub repository</a>: two <code>.pas</code> files, and
a set of minimalist <a href="https://blog.synopse.info?post/post/2014/04/28/Mustache-Logic-less-templates-for-Delphi-part-3">Mustache
views</a>.</p>
<p>This little web application publishes a simple BLOG, not fully finished yet
(this is a <em>Sample</em>, remember!).<br />
But you can still execute it in your desktop browser, or any mobile device
(thanks to a simple <em>Bootstrap</em>-based responsive design), and see the
articles list, view one article and its comments, view the author information,
log in and out.</p>
<p>This sample is implemented as such:</p>
<table>
<tbody>
<tr>
<td><strong>MVVM</strong></td>
<td><strong>Source</strong></td>
<td><strong>mORMot</strong></td>
</tr>
<tr>
<td><em>Model</em></td>
<td><code>MVCModel.pas</code></td>
<td><code>TSQLRestServerDB</code> ORM over a <em>SQlite3</em> database</td>
</tr>
<tr>
<td><em>View</em></td>
<td><code>*.html</code></td>
<td><em>Mustache template engine</em> in the <em>Views</em> sub-folder</td>
</tr>
<tr>
<td><em>ViewModel</em></td>
<td><code>MVCViewModel.pas</code></td>
<td>Defined as one <code>IBlogApplication</code> interface</td>
</tr>
</tbody>
</table>
<p>For the sake of the simplicity, the sample will create some fake data in its
own local <em>SQlite3</em> database, the first time it is executed.</p> <p>To discover more, the easiest is to read the <a href="https://blog.synopse.info?post/public/mORMot/mORMotMVC.pdf"><em>mORMot</em> MVC Draft
Documentation</a>!<br />
(which is a chapter extracted from today's version of our <a href="http://synopse.info/files/pdf/Synopse%20mORMot%20Framework%20SAD%201.18.pdf">Software
Architecture Document</a> pdf)</p>
<p>As usual, <a href="http://synopse.info/forum/viewtopic.php?id=2119">feedback
is welcome in our forum</a>!</p>Legacy code, mORMot, and database sharingurn:md5:8855247aa1091834083ba2a25e2c16802014-09-12T09:56:00+02:002014-09-12T09:56:00+02:00AB4327-GANDImORMot FrameworkAJAXblogCrossPlatformDatabaseDocumentationFirebirdGoodPracticelegacyORMRADSOASQL<p>It is pretty much possible that you would have to maintain and evolve a
legacy project, based on an existing database, with a lot of already written
SQL statements - see <em><a href="https://blog.synopse.info?post/post/2012/12/31/Enhance-existing-projects-with-mORMot">Legacy code and
existing projects</a></em>.</p>
<p><img src="http://thumbs.dreamstime.com/t/old-stamp-marmot-black-fox-7894949.jpg" /></p>
<p>For instance, you would like to use <em>mORMot</em> for new features, and/or
add mobile or HTML clients - see <em><a href="https://blog.synopse.info?post/post/2014/08/11/CrossPlatform-Clients/Units-Platforms">Cross-Platform
clients</a></em>.<br />
In this case, the ORM advanced features - like <em><a href="https://blog.synopse.info?post/post/2012/02/14/ORM-cache">ORM Cache</a></em> or BATCH process, see
<em><a href="https://blog.synopse.info?post/post/2011/06/03/BATCH-sequences-for-adding/updating/deleting-records">BATCH
sequences for adding/updating/deleting records</a></em> - may conflict with the
legacy code, for the tables which may have to be shared.<br />
Here are some guidelines when working on such a project.</p>
<p>To be exhaustive about your question, we need to consider each ORM CRUD
operation.<br />
We may have to divide them in three kinds: read queries, insertions, and
modifications of existing data.</p> <p>About ORM <code>Retrieve()</code> methods, the ORM cache can be tuned per
table, and you will definitively lack of some cache, but remember :</p>
<ul>
<li>That you can set a "time out" period for this cache, so that you may still
benefit of it in most cases;</li>
<li>That you have a cache at server level and another at client level, so you
can tune it to be less aggressive on the client, for instance;</li>
<li>That you can tune the ORM cache <em>per ID</em>, so some items which are
not likely to change can still be cached.</li>
</ul>
<p>About ORM <code>Add()</code> or <code>BatchAdd()</code> methods, when using
the external engine, if any external process is likely to INSERT new rows,
ensure you set the <code>TSQLRestStorageExternal EngineAddUseSelectMaxID</code>
property to TRUE, so that it will compute the next maximum ID by hand.<br />
But it still may be an issue, since the external process may do an INSERT
<em>during</em> the ORM insertion.<br />
So the best is perhaps to NOT use the ORM <code>Add()</code> or
<code>BatchAdd()</code> methods, but rely on dedicated INSERT SQL statement,
e.g. hosted in an interface-based service on the server side.</p>
<p>About ORM <code>Update() Delete() BatchUpdate() BatchDelete()</code>
methods, they sound safe to be used in conjunction with external process
modifying the DB, as soon as you use transactions to let the modifications be
atomic, and won't conflict any concurrent modifications in the legacy code.</p>
<p>Perhaps the safer pattern, when working with external tables which are to be
modified in the background by some legacy code, may be to use server-side
<em>interface-based services</em> - see <em>Client-Server services via
interfaces</em> - for any process involving external tables which may be
modified by another process, with manual SQL, instead of using the ORM "magic".
But it will depend on your business logic, and you will fail to benefit from
the ORM features of the framework.<br />
Nevertheless, introducing <em>Service-Oriented Architecture (SOA)</em> into
your application would be very beneficial: ORM is not mandatory, especially if
you are "fluent" in SQL queries, know how to make them as standard as possible,
and have a lot of legacy code, perhaps with already tuned SQL statements.</p>
<p>Introducing SOA is mandatory to introduce new kind of clients to your
applications, like mobile apps or AJAX modern sites: you could not access
directly the database any more, as you did with your legacy Delphi application,
and RAD DB components.<br />
All new features, involving new tables to store new data, would still benefit
of the <em>mORMot</em>'s ORM, and could still be hosted in the very same
external database, shared by your existing code.<br />
Then, you will be able to identify <em>seams</em> - see <em><a href="https://blog.synopse.info?post/post/2012/12/31/Enhance-existing-projects-with-mORMot">Legacy code and
existing projects</a></em> - in your legacy code, and move them to your new
<em>mORMot</em> services, then let your application evolve into a newer
<a href="https://blog.synopse.info?post/post/2014/05/30/Software-Design%2C-Brook%2C-mORMot%2C-RAD%2C-SOLID-and-OOP">
SOA/MVC architecture</a>, without breaking anything, nor starting from
scratch.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=2008">welcome on our forum</a>, as
usual.</p>Audit-trail for ORM change trackingurn:md5:97d3f68fe68d0d8fb0b8ede83bedc05a2014-06-22T15:23:00+02:002014-06-22T14:40:31+02:00AB4327-GANDImORMot FrameworkAuditTrailblogDatabaseDelphiDocumentationEntityObjectGoodPracticeJSONModelmORMotORMRestSourceSQL<p>Since most CRUD operations are centered within the scope of our
<em>mORMot</em> server, we implemented in the ORM an integrated mean of
tracking changes (aka Audit Trail) of any <code>TSQLRecord</code>.<br />
In short, our ORM is transformed into a time-machine, just like the good old
<em>DeLorean</em>!</p>
<p><img src="http://evworld.com/images/b2f_delorean_rear.jpg" /></p>
<p>Keeping a track of the history of business objects is one very common need
for software modeling, and a must-have for any <a href="https://blog.synopse.info?post/post/2014/01/04/Domain-Driven-Design%3A-part-2">accurate data modeling</a>,
like <em>Domain-Driven Design</em>.<br />
By default, as expected by the OOP model, any change to an object will forget
any previous state of this object. But thanks to <em>mORMot</em>'s exclusive
change-tracking feature, you can persist the history of your objects.</p>
<h3>Enabling audit-trail</h3>
<p>By default, change-tracking feature will be disabled, saving performance and
disk use.<br />
But you can enable change tracking for any class, by calling the following
method, on server side:</p>
<pre>
aServer.TrackChanges([TSQLInvoice]);
</pre>
<p>This single line will let <code>aServer: TSQLRestServer</code> monitor all
CRUD operations, and store all changes of the <code>TSQLInvoice</code> table
within a <code>TSQLRecordHistory</code> table.</p> <p>Since all content change will be stored in this single table by default
(note that the <code>TrackChanges()</code> method accepts an <em>array of
classes</em> as parameters, and can be called several times), it may be handy
to define several tables for history storage. Later on, an external database
engine may be defined to store history, e.g. on cheap hardware (and big hard
drives), whereas your may database may be powered by high-end hardware (and
small SSDs) - see <em><a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-benchmark">External database
access</a></em>.<br />
To do so, you define your custom class for history storage, then supply it as
parameter:</p>
<pre>
<strong>type</strong>
TSQLRecordSecondaryHistory = <strong>class</strong>(TSQLRecord);
(...)
aServer.TrackChanges([TSQLInvoice],TSQLRecordSecondaryHistory);
</pre>
<p>Then, all history will be stored in this
<code>TSQLRecordSecondaryHistory</code> class (in its own table named
<code>SecondaryHistory</code>), and not the default
<code>TSQLRecordHistory</code> class (in its <code>History</code> table).</p>
<h3>A true Time Machine for your objects</h3>
<p>Once the object changes are tracked, you can later on browse the history of
the object, by using the <code>TSQLRecordHistory.CreateHistory()</code>, then
<code>HistoryGetLast</code>, <code>HistoryCount</code>, and
<code>HistoryGet()</code> methods:</p>
<pre>
<strong>var</strong> aHist: TSQLRecordSecondaryHistory;
aInvoice: TSQLInvoice;
aEvent: TSQLEvent;
aTimeStamp: TModTime;
(...)
aInvoice := TSQLInvoice.Create;
aHist := TSQLRecordSecondaryHistory.CreateHistory(aClient,TSQLRecordPeopleExt,400);
<strong>try</strong>
writeln('Number of items in the record history: ',aHist.HistoryCount);
<strong>for</strong> i := 0 <strong>to</strong> aHist.HistoryCount-1 <strong>do begin</strong>
aHist.HistoryGet(i,aEvent,aTimeStamp,aInvoice);
writeln;
writeln('Event: ',GetEnumName(TypeInfo(TSQLEvent),ord(aEvent))^);
writeln('TimeStamp: ',TTimeLogBits(TimeStamp).ToText);
writeln('Value: ',aInvoice.GetJSONValues(true,true,soSelect));
<strong>end</strong>;
<strong>finally</strong>
aHist.Free;
aInvoice.Free;
<strong>end</strong>;
</pre>
<p>As a result, our ORM is also transformed into a true <em>time machine</em>,
for the objects which need it.</p>
<p>This feature will be available on both client and server sides, via the
<code>TSQLRecordHistory</code> table.</p>
<h3>Automatic history packing</h3>
<p>This <code>TSQLRecordHistory</code> class will in fact create a
<code>History</code> table in the main database, defined as such:</p>
<pre>
TSQLRecordHistory = <strong>class</strong>(TSQLRecord)
(...)
<strong>published</strong>
<em>/// identifies the modified record</em>
<strong>property</strong> ModifiedRecord: PtrInt <strong>read</strong> fModifiedRecord;
<em>/// the kind of modification stored</em>
<strong>property</strong> Event: TSQLEvent <strong>read</strong> fEvent;
<em>/// for seAdd/seUpdate, the data stored as JSON</em>
<strong>property</strong> SentDataJSON: RawUTF8 <strong>index</strong> 4000 <strong>read</strong> fSentData;
<em>/// when the modification was recorded</em>
<strong>property</strong> TimeStamp: TModTime <strong>read</strong> fTimeStamp;
<em>/// after some events are written as individual SentData content, they</em>
<em>// will be gathered and compressed within one BLOB field</em>
<strong>property</strong> History: TSQLRawBlob <strong>read</strong> fHistory;
<strong>end</strong>;
</pre>
<p>In short, any modification via the ORM will be stored in the
<code>TSQLRecordHistory</code> table, as a JSON object of the changed fields,
in <code>TSQLRecordHistory.SentDataJSON</code>.</p>
<p>By design, direct SQL changes are not handled. If you run some SQL
statements like <code>DELETE FROM ...</code> or <code>UPDATE ... SET ...</code>
are executed within your application or from any external program, then the
History table won't be updated.<br />
In fact, the ORM does not set any DB triggers to track low-level changes: it
would slow down the process, and void the <em>persistence agnosticism</em>
paradigm we want to follow, e.g. allowing to use a <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">NoSQL database like MongoDB</a>.</p>
<p>When the history grows, the JSON content may become huge, and fill the disk
space with a lot of duplicated content. In order to save disk space, when a
record reaches a define number of JSON data rows, all this JSON content is
gathered and compressed into a BLOB, in
<code>TSQLRecordHistory.History</code>.<br />
You can force this packing process by calling
<code>TSQLRestServer.TrackChangesFlush()</code> manually in your code. Calling
this method will also have a welcome side effect: it will read the actual
content of the record from the database, then add a fake <code>seUpdate</code>
event in the history if the field values do not match the one computed from
tracked changes, to ensure that the audit trail will be correct. As a
consequence, history will become always synchronized with the actual data
persisted in the database, even if external SQL did by-pass the CRUD methods of
the ORM, via unsafe <code>DELETE FROM ...</code> or <code>UPDATE ... SET
...</code> statements.</p>
<p>You can tune how packing is defined for a given <code>TSQLRecord</code>
table, by using some optional parameters to the registering method:</p>
<pre>
<strong>procedure</strong> TrackChanges(<strong>const</strong> aTable: <strong>array of</strong> TSQLRecordClass;
aTableHistory: TSQLRecordHistoryClass=<strong>nil</strong>; aMaxHistoryRowBeforeBlob: integer=1000;
aMaxHistoryRowPerRecord: integer=10; aMaxUncompressedBlobSize: integer=64*1024); <strong>virtual</strong>;
</pre>
<p>Take a look at the documentation of this method (or the comments in its
declaration code) for further information.<br />
Default options will let <code>TSQLRestServer.TrackChangesFlush()</code> be
called after 1000 individual <code>TSQLRecordHistory.SentDataJSON</code> rows
are stored, then will compress them into a BLOB once 10 JSON rows are available
for a given record, ensuring that the uncompressed BLOB size for a single
record won't use more than 64 KB of memory (but probably much less in the
database, since it is stored with very high compression rate).</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=11164#p11164">welcome on our
forum</a>, as usual.</p>MongoDB + mORMot benchmarkurn:md5:ffe569e390738cbf6244be3f0a64bf512014-05-07T09:31:00+02:002014-05-07T08:57:07+02:00AB4327-GANDImORMot FrameworkACIDAJAXBatchblogDatabaseDelphiDocumentationDomainDrivenGoodPracticeJSONmappingModelMongoDBmORMotNoSQLOpenSourceORMperformanceRestshardingSourceSQLtransactionUnicode<p>Here are some benchmark charts about <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM"><em>MongoDB</em> integration in
<em>mORMot</em>'s ORM</a>.</p>
<p><img src="http://www.ceillac.com/IMAGES%20SITE/animaux/marmot-s2.jpg" alt="" /></p>
<p><em>MongoDB</em> appears as a serious competitor to SQL databases, with the
potential benefit of horizontal scaling and installation/administration ease -
performance is very high, and its document-based storage fits perfectly with
<em>mORMot</em>'s advanced ORM features like <em><a href="https://blog.synopse.info?post/post/2011/07/03/Sharde-nothing-architecture">Shared nothing architecture (or
sharding)</a></em>.</p> <p>Following tests were using Synopse <em>mORMot</em> framework 1.18, compiled
with Delphi XE4, against SQLite 3.8.4.3.</p>
<p>We won't show all database engine, but the most representative ones.<br />
Please refer to <a href="https://blog.synopse.info?post/post/2014/03/07/Support-of-MySQL%2C-DB2-and-PostgreSQL">this other benchmark
article for some more complete information</a>.</p>
<h1>Insertion speed</h1>
<p>'MongoDB ack/no ack' for direct <em>MongoDB</em> access
(<code>SynMongoDB.pas</code>) with or without write acknowledge.</p>
<p>For the testings, we used a local <em>MongoDB</em> 2.6 instance in 64 bit
mode.</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>Direct</strong></td>
<td><strong>Batch</strong></td>
<td><strong>Trans</strong></td>
<td><strong>Batch Trans</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>466</td>
<td>437</td>
<td>81754</td>
<td>108752</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>2012</td>
<td>2044</td>
<td>84550</td>
<td>111731</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off exc)</strong></td>
<td>25079</td>
<td>28192</td>
<td>83943</td>
<td>115159</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>69961</td>
<td>94871</td>
<td>87279</td>
<td>118657</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>232385</td>
<td>400608</td>
<td>252678</td>
<td>402803</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>242812</td>
<td>409131</td>
<td>240003</td>
<td>405712</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>490</td>
<td>11918</td>
<td>87556</td>
<td>151144</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>2141</td>
<td>47266</td>
<td>89249</td>
<td>160616</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>33199</td>
<td>145471</td>
<td>90025</td>
<td>158815</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>76411</td>
<td>184706</td>
<td>89834</td>
<td>192618</td>
</tr>
<tr align="center">
<td><strong>MongoDB (ack)</strong></td>
<td>10081</td>
<td>84585</td>
<td>9800</td>
<td>85232</td>
</tr>
<tr align="center">
<td><strong>MongoDB (no ack)</strong></td>
<td>33223</td>
<td>189186</td>
<td>27974</td>
<td>206355</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>474</td>
<td>11917</td>
<td>36740</td>
<td>55767</td>
</tr>
<tr align="center">
<td><strong>FireDAC SQlite3</strong></td>
<td>20735</td>
<td>40083</td>
<td>40408</td>
<td>121359</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>10056</td>
<td>10155</td>
<td>18769</td>
<td>20335</td>
</tr>
<tr align="center">
<td><strong>FireDAC Firebird</strong></td>
<td>19742</td>
<td>48684</td>
<td>19904</td>
<td>47803</td>
</tr>
<tr align="center">
<td><strong>MSSQL2012 local</strong></td>
<td>3470</td>
<td>35510</td>
<td>10750</td>
<td>47653</td>
</tr>
<tr align="center">
<td><strong>ODBC MSSQL2012</strong></td>
<td>3659</td>
<td>6252</td>
<td>5537</td>
<td>6290</td>
</tr>
<tr align="center">
<td><strong>FireDAC MSSQL2012</strong></td>
<td>3276</td>
<td>5838</td>
<td>9540</td>
<td>40040</td>
</tr>
<tr align="center">
<td><strong>ZEOS PostgreSQL</strong></td>
<td>2953</td>
<td>23740</td>
<td>6913</td>
<td>29780</td>
</tr>
<tr align="center">
<td><strong>ODBC PostgreSQL</strong></td>
<td>2902</td>
<td>25040</td>
<td>3576</td>
<td>28714</td>
</tr>
<tr align="center">
<td><strong>FireDAC PostgreSQL</strong></td>
<td>3054</td>
<td>23329</td>
<td>7149</td>
<td>24844</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Batch+Trans|Trans|Batch|Direct&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,409131&chds=0,409131,0,409131,0,409131,0,409131,0,409131&chd=t:466,437,81754,108752|2012,2044,84550,111731|25079,28192,83943,115159|69961,94871,87279,118657|232385,400608,252678,402803|242812,409131,240003,405712|490,11918,87556,151144|2141,47266,89249,160616|33199,145471,90025,158815|76411,184706,89834,192618|10081,84585,9800,85232|33223,189186,27974,206355|474,11917,36740,55767|20735,40083,40408,121359|10056,10155,18769,20335|19742,48684,19904,47803|3470,35510,10750,47653|3659,6252,5537,6290|3276,5838,9540,40040|2953,23740,6913,29780|2902,25040,3576,28714|3054,23329,7149,24844&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|MongoDB+%28ack%29|MongoDB+%28no+ack%29|ZEOS+SQlite3|FireDAC+SQlite3|ZEOS+Firebird|FireDAC+Firebird|MSSQL2012+local|ODBC+MSSQL2012|FireDAC+MSSQL2012|ZEOS+PostgreSQL|ODBC+PostgreSQL|FireDAC+PostgreSQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|FireDAC+PostgreSQL|ODBC+PostgreSQL|ZEOS+PostgreSQL|FireDAC+MSSQL2012|ODBC+MSSQL2012|MSSQL2012+local|FireDAC+Firebird|ZEOS+Firebird|FireDAC+SQlite3|ZEOS+SQlite3|MongoDB+%28no+ack%29|MongoDB+%28ack%29|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,409131&chds=0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131&chd=t:466,2012,25079,69961,232385,242812,490,2141,33199,76411,10081,33223,474,20735,10056,19742,3470,3659,3276,2953,2902,3054|437,2044,28192,94871,400608,409131,11918,47266,145471,184706,84585,189186,11917,40083,10155,48684,35510,6252,5838,23740,25040,23329|81754,84550,83943,87279,252678,240003,87556,89249,90025,89834,9800,27974,36740,40408,18769,19904,10750,5537,9540,6913,3576,7149|108752,111731,115159,118657,402803,405712,151144,160616,158815,192618,85232,206355,55767,121359,20335,47803,47653,6290,40040,29780,28714,24844&chdl=Direct|Batch|Trans|Batch+Trans" /></p>
<p><em>MongoDB</em> bulk insertion has been implemented, which shows an amazing
speed increase in Batch mode. Depending on the <em>MongoDB write concern</em>
mode, insertion speed can be very high: by default, every write process will be
acknowledge by the server, but you can by-pass this request if you set the
<code>wcUnacknowledged</code> mode - note that in this case, any error (e.g. an
unique field duplicated value) will never be notified, so it should not be used
in production, unless you need this feature to quickly populate a database, or
consolidate some data as fast as possible.</p>
<h1>Read speed</h1>
<p>The same records are now read, either one by one, either as a list:</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>By one</strong></td>
<td><strong>All Virtual</strong></td>
<td><strong>All Direct</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>21607</td>
<td>455083</td>
<td>458757</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>22177</td>
<td>456454</td>
<td>458001</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off exc)</strong></td>
<td>98014</td>
<td>454215</td>
<td>457540</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>99190</td>
<td>461808</td>
<td>464252</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>235504</td>
<td>756773</td>
<td>750300</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>233666</td>
<td>332402</td>
<td>733460</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>103917</td>
<td>210863</td>
<td>458379</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>101498</td>
<td>209634</td>
<td>441033</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>101839</td>
<td>218292</td>
<td>439947</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>102414</td>
<td>185494</td>
<td>438904</td>
</tr>
<tr align="center">
<td><strong>MongoDB (ack)</strong></td>
<td>8002</td>
<td>242353</td>
<td>251268</td>
</tr>
<tr align="center">
<td><strong>MongoDB (no ack)</strong></td>
<td>8234</td>
<td>252079</td>
<td>254582</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>31135</td>
<td>173593</td>
<td>263060</td>
</tr>
<tr align="center">
<td><strong>FireDAC SQlite3</strong></td>
<td>6318</td>
<td>67169</td>
<td>92291</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>12076</td>
<td>67853</td>
<td>85828</td>
</tr>
<tr align="center">
<td><strong>FireDAC Firebird</strong></td>
<td>1918</td>
<td>37113</td>
<td>44894</td>
</tr>
<tr align="center">
<td><strong>MSSQL2012 local</strong></td>
<td>7904</td>
<td>182401</td>
<td>349797</td>
</tr>
<tr align="center">
<td><strong>ODBC MSSQL2012</strong></td>
<td>8693</td>
<td>113973</td>
<td>178526</td>
</tr>
<tr align="center">
<td><strong>FireDAC MSSQL2012</strong></td>
<td>3054</td>
<td>63730</td>
<td>86051</td>
</tr>
<tr align="center">
<td><strong>ZEOS PostgreSQL</strong></td>
<td>7031</td>
<td>122327</td>
<td>176298</td>
</tr>
<tr align="center">
<td><strong>ODBC PostgreSQL</strong></td>
<td>7281</td>
<td>66843</td>
<td>91489</td>
</tr>
<tr align="center">
<td><strong>FireDAC PostgreSQL</strong></td>
<td>1644</td>
<td>45184</td>
<td>61252</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|All+Direct|All+Virtual|By+one&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,756773&chds=0,756773,0,756773,0,756773&chd=t:21607,455083,458757|22177,456454,458001|98014,454215,457540|99190,461808,464252|235504,756773,750300|233666,332402,733460|103917,210863,458379|101498,209634,441033|101839,218292,439947|102414,185494,438904|8002,242353,251268|8234,252079,254582|31135,173593,263060|6318,67169,92291|12076,67853,85828|1918,37113,44894|7904,182401,349797|8693,113973,178526|3054,63730,86051|7031,122327,176298|7281,66843,91489|1644,45184,61252&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|MongoDB+%28ack%29|MongoDB+%28no+ack%29|ZEOS+SQlite3|FireDAC+SQlite3|ZEOS+Firebird|FireDAC+Firebird|MSSQL2012+local|ODBC+MSSQL2012|FireDAC+MSSQL2012|ZEOS+PostgreSQL|ODBC+PostgreSQL|FireDAC+PostgreSQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|FireDAC+PostgreSQL|ODBC+PostgreSQL|ZEOS+PostgreSQL|FireDAC+MSSQL2012|ODBC+MSSQL2012|MSSQL2012+local|FireDAC+Firebird|ZEOS+Firebird|FireDAC+SQlite3|ZEOS+SQlite3|MongoDB+%28no+ack%29|MongoDB+%28ack%29|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,756773&chds=0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773&chd=t:21607,22177,98014,99190,235504,233666,103917,101498,101839,102414,8002,8234,31135,6318,12076,1918,7904,8693,3054,7031,7281,1644|455083,456454,454215,461808,756773,332402,210863,209634,218292,185494,242353,252079,173593,67169,67853,37113,182401,113973,63730,122327,66843,45184|458757,458001,457540,464252,750300,733460,458379,441033,439947,438904,251268,254582,263060,92291,85828,44894,349797,178526,86051,176298,91489,61252&chdl=By+one|All+Virtual|All+Direct" /></p>
<p><em>MongoDB</em> appears as a serious competitor to SQL databases, with the
potential benefit of horizontal scaling and installation/administration ease -
performance is very high, and its document-based storage fits perfectly with
<em>mORMot</em>'s advanced ORM features like <em><a href="https://blog.synopse.info?post/post/2011/07/03/Sharde-nothing-architecture">Shared nothing architecture (or
sharding)</a></em>.</p>
<p>You can get more information about <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-database-access">low-level integration of
<em>MongoDB</em></a> in <em>mORMot</em>, or our integrated <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">ORM/ODM support in the
framework</a>.<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1742">welcome
on our forum</a>, as usual!</p>