Synopse Open Source - Tag - shardingmORMot MVC / SOA / ORM and friends2024-02-02T17:08:25+00:00urn:md5:cc547126eb580a9adbec2349d7c65274DotclearORM Master/Slave Replicationurn:md5:08d531e552853cbab93bd086a1bab2902015-03-31T22:40:00+02:002015-04-05T07:27:38+02:00AB4327-GANDImORMot FrameworkbackupBatchblogDatabaseDelphiDocumentationGoodPracticeMaster-SlavemORMotreplicationRestshardingSourceSQLite3<p>As stated during <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_26">
TSQLRecord fields definition</a></em>, the ORM is able to maintain a revision
number for any <code>TSQLRecord</code> table, so that it the table may be
easily synchronized remotely by another <code>TSQLRestServer</code>
instance.<br />
If you define a <code>TRecordVersion</code> published property, the ORM core
will fill this field just before any write with a monotonically increasing
revision number, and will take care of any deletion, so that those
modifications may be replayed later on any other database.</p>
<p><img src="http://i41.tinypic.com/35lw00x.jpg" alt="" /></p>
<p>This synchronization will work as a strict <em>master/slave replication</em>
scheme, as a one-way on demand refresh of a replicated table.<br />
Each write operation on the master database on a given table may be easily
reflected on one or several slave databases, with almost no speed nor storage
size penalty.</p> <h3>Enable synchronization</h3>
<p>In order to enable this replication mechanism, you should define a
<code>TRecordVersion</code> published property in the <code>TSQLRecord</code>
class type definition:</p>
<pre>
TSQLRecordPeopleVersioned = <strong>class</strong>(TSQLRecordPeople)
<strong>protected</strong>
fFirstName: RawUTF8;
fLastName: RawUTF8;
fVersion: TRecordVersion;
<strong>published</strong>
<strong>property</strong> FirstName: RawUTF8 <strong>read</strong> fFirstName <strong>write</strong> fFirstName;
<strong>property</strong> LastName: RawUTF8 <strong>read</strong> fLastName <strong>write</strong> fLastName;
<strong>property</strong> Version: TRecordVersion <strong>read</strong> fVersion <strong>write</strong> fVersion;
<strong>end</strong>;
</pre>
<p>Only a single <code>TRecordVersion</code> field is allowed per
<code>TSQLRecord</code> class - it would not mean anything to manage more than
one field of this type.</p>
<p>Note that this field will be somewhat "hidden" to most ORM process: a
regular <code>TSQLRest.Retrieve</code> won't fill this <code>Version</code>
property, since it is an internal implementation detail.<br />
If you want to lookup its value, you would have to explicitly state its field
name at retrieval. Any <code>TRecordVersion</code> is indeed considered as a
"non simple field", just like BLOB fields, so would need explicit retrieval of
its value.</p>
<p>In practice, any <code>TSQLRest.Add</code> and <code>TSQLRest.Update</code>
on this <code>TSQLRecordPeopleVersioned</code> class will increase this
<code>Version</code> revision number field, and a <code>TSQLRest.Delete</code>
will populate an external <code>TSQLRecordTableDelete</code> table with the
<code>ID</code> of the deleted record, associated with a
<code>TRecordVersion</code> revision.</p>
<p>As consequences:</p>
<ul>
<li>The monotonic <code>TRecordVersion</code> number is shared at
<code>TSQLRestServer</code> level, among all tables containing a
<code>TRecordVersion</code> published field;</li>
<li>The <code>TSQLRecordTableDelete</code> table should be part of the
<code>TSQLModel</code>, in conjunction with
<code>TSQLRecordPeopleVersioned</code>;</li>
<li>If the <code>TSQLRecordTableDelete</code> table is not part of the
<code>TSQLModel</code>, the <code>TSQLRestServer</code> will add it - but you
should better make it explicitly appearing in the data model;</li>
<li>A single <code>TSQLRecordTableDelete</code> table will maintain the list of
all deleted data rows, of all tables containing a <code>TRecordVersion</code>
published field;</li>
<li>The <code>TSQLRecordPeopleVersioned</code> table appearance order in the
<code>TSQLModel</code> will matter, since <code>TSQLRecordTableDelete.ID</code>
will use this table index order in the database model to identify the table
type of the deleted row - in a similar way to <em>TRecordReference and
TRecordReferenceToBeDeleted</em>.</li>
</ul>
<p>All the synchronization preparation will be taken care by the ORM kernel on
its own, during any write operation. There is nothing particular to maintain or
setup, in addition to this <code>TRecordVersion</code> field definition, and
the global <code>TSQLRecordTableDelete</code> table.</p>
<h3>From master to slave</h3>
<p>To replicate this <code>TSQLRecordPeopleVersioned</code> table from another
<code>TSQLRestServer</code> instance, just call the following method:</p>
<pre>
aServer.RecordVersionSynchronizeSlave(TSQLRecordPeopleVersioned,aClient);
</pre>
<p>This single line will request a remote server via a <code>Client:
TSQLRestClientURI</code> connection (which may be over HTTP) for any pending
modifications since its last call, then will fill the local <code>aServer:
TSQLRestServer</code> database so that the local
<code>TSQLRecordPeopleVersioned</code> table will contain the very same content
as the remote master <code>TSQLRestServer</code>.</p>
<p>You can safely call
<code>TSQLRestServer.RecordVersionSynchronizeSlave</code> from several clients,
to replicate the master data in several databases.</p>
<p>Only the modified data will be transmitted over the wire, as two REST/JSON
queries (one for the insertions/updates, another for the deletions), and all
the local write process will use optimized <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_78">
BATCH writing</a>. This means that the synchronization process will try to use
as minimal bandwidth and resources as possible, on both sides.</p>
<p>Of course, the slaves should be considered as read-only, otherwise the
version numbers may conflict, and the whole synchronization may become a
failure.<br />
But you can safely replicate servers in cascade, if needed: the version numbers
will be propagated from masters to slaves, and the data will always be in a
consistent way.</p>
<h3>Replication use cases</h3>
<p>We may consider a very common corporate infrastructure:</p>
<p><a href="https://blog.synopse.info?post/public/mORMot/mORMotMasterSlaveReplication.png"><img src="https://blog.synopse.info?post/public/mORMot/.mORMotMasterSlaveReplication_m.jpg" alt="Master Slave ORM Replication" title="Master Slave ORM Replication, Mar 2015" /></a></p>
<p>This kind of installation, with a main central office, and a network of
local offices, would benefit from this master/slave replication.<br />
Simple <em>redirection</em> may be used - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_93">
Redirect to an external TSQLRest</a></em> - but it would expect the work to
continue, even in case of <em>Internet</em> network failure.<br />
REST redirection would expect a 100% connection uplink, which may be critical
in some cases.</p>
<p>You could therefore implement replication in several ways:</p>
<ul>
<li>Either the main office is the master, and any write would be push to the
<em>Main Server</em>, whereas local offices would have a replicated copy of the
information - drawback is that in case of network failure, the local office
would be able to only read the data;</li>
<li>Or each local office may host its own data in a dedicated table,
synchronized as a master database; the main office will replicate (as a slave)
the private data of each local servers; in addition, all this data gathered by
the <em>Main Server</em> may be further replication to the other local offices,
and be still accessible in read mode - in case of network failure, all the data
is available on the local servers, and the local private table is still
writable.</li>
</ul>
<p>Of course, the second solution seems preferable, even if a bit more
difficult to implement. The ablity of all local offices to work offline on
their own private data, but still having all the other data accessible as
read-only, would be a huge ROI.</p>
<p>As a benefit of using replication, the central main server would be less
stressed, since most of the process would take place in local servers, and the
main office server would only be used for shared data backup and read-only
gathering of the other local databases. Only a small network bandwith would be
necessary (much less than a pure web solution), and CPU/storage resources would
be minimal.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=11863#p11863">welcome on our
forum</a>, as usual!<br />
See also <a href="http://synopse.info/fossil/info/3453f314d9">the associated
feature request</a>, for upcoming features related to replication.<br />
And the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_147">
corresponding documentation article</a>, which will be updated often, so is
preferred to this static blog article!</p>2015: the future of mORMot is BigDataurn:md5:aad77f14b241587052c36d8614eb16372014-12-31T11:35:00+01:002014-12-31T12:17:12+01:00AB4327-GANDImORMot FrameworkbackupblogCrossPlatformDelphiLinuxmORMotORMperformancereplicationRestsecurityshardingSOASynopse<p>How would be 2015 like for our little rodents?<br />
Due to popular request of several users of <em>mORMot</em>, we identified and
designed some feature requests dedicated to <a href="http://en.wikipedia.org/wiki/Big_data">BigData</a> process.</p>
<p><img src="http://4.bp.blogspot.com/-dFLzN6qwi1I/UoELR41hJXI/AAAAAAAAAB8/9du_nNOdDtE/s1600/Big_data.ashx_.png" alt="" /></p>
<p>In fact, your <em>data</em> is the new value, especially if you propose
<em>SaaS</em> (<a href="http://en.wikipedia.org/wiki/Software_as_a_service">Software As A Service</a>)
hosting to your customers, with a farm of <em><ins>mORMot</ins></em>
servers.<br />
Recent Linux support for <em>mORMot</em> servers, together with the
high performance and installation ease of our executable, open the gate to
cheap cloud-based hosting.<br />
As a consequence, a lot of information would certainly be gathered by
your <em>mORMot</em> servers, and a single monolithic database
is not an option any more.</p>
<p>For <em>mORMot</em> solutions hosted in cloud, a lot of data may be
generated. The default <em>SQLite3</em> storage engine may be less convenient,
once it reaches some GB of file content. Backup becomes to be slow and
inefficient, and hosting this oldest data in the main DB, probably stored on an
expensive SSD, may be a lost of resource. Vertical scaling is limited by
hardware and price factors.</p>
<p>This is were <strong>data sharding</strong> comes into scene.<br />
Note that <em>sharding</em> is not replication/backup, nor clustering, nor just
spreading. We are speaking about application-level data splitting, to ease
maintenance and horizontal scalability of <em>mORMot</em> servers.</p>
<p>Data sharding could already be implemented with <em>mORMot</em> servers,
thanks to <code>TSQLRestStorage</code>:</p>
<ul>
<li>Using <code>TSQLRestStorageExternal</code>: any table may have its own
external SQL database engine, may be in its separated DB server;</li>
<li>Using <code>TSQLRestStorageMongoDB</code>: any table may be stored on a
<em>MongoDB</em> cluster, with its own sharding abilities;</li>
<li>Using <code>TSQLRestStorageRemote</code>: each table may have its own
remote ORM/REST server.</li>
</ul>
<p>But when data stored in a single table tends to grow without limit, this
feature is not enough.<br />
Let's see how the close future of <em>mORMot</em> looks like.</p> <p>The following features are on the road map:</p>
<ul>
<li><a href="http://synopse.info/fossil/info/3453f314d9">Auto-Synch</a> between
<em>mORMot</em> servers (as cache or for branch office cache), and potentially
<em>mORMot</em> clients (offline mode);</li>
<li><a href="http://synopse.info/fossil/tktview/cde2d68eb60463">Automated
sharding</a> for BigData storage;</li>
<li><a href="http://synopse.info/fossil/tktview/17958b22c028">Monitoring</a> of
your farm of <em>mORMot</em> servers;</li>
<li><a href="http://synopse.info/fossil/tktview/9357b49fe2">Workaround to use
interfaces</a> / SOA under FPC (Linux);</li>
<li><a href="http://synopse.info/fossil/tktview/8e4c17a082">Integrate the
Monkey HTTP Server</a> for Linux;</li>
<li>Implements <a href="http://synopse.info/fossil/tktview?name=d71391c2e7">WiredTiger NoSQL
engine</a>.</li>
</ul>
<p>Please refer to each feature request ticket for design patterns, and
implementation proposal.<br />
The <a href="http://synopse.info/fossil/tktview?name=aa230e5299">publish/subscribe
event-based mechanism</a> would be first to be implemented, since it would be
used by the first items.</p>
<p>Your feedback is <a href="http://synopse.info/forum/viewtopic.php?id=2259">welcome on our forum</a>, as
usual!</p>MongoDB + mORMot benchmarkurn:md5:ffe569e390738cbf6244be3f0a64bf512014-05-07T09:31:00+02:002014-05-07T08:57:07+02:00AB4327-GANDImORMot FrameworkACIDAJAXBatchblogDatabaseDelphiDocumentationDomainDrivenGoodPracticeJSONmappingModelMongoDBmORMotNoSQLOpenSourceORMperformanceRestshardingSourceSQLtransactionUnicode<p>Here are some benchmark charts about <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM"><em>MongoDB</em> integration in
<em>mORMot</em>'s ORM</a>.</p>
<p><img src="http://www.ceillac.com/IMAGES%20SITE/animaux/marmot-s2.jpg" alt="" /></p>
<p><em>MongoDB</em> appears as a serious competitor to SQL databases, with the
potential benefit of horizontal scaling and installation/administration ease -
performance is very high, and its document-based storage fits perfectly with
<em>mORMot</em>'s advanced ORM features like <em><a href="https://blog.synopse.info?post/post/2011/07/03/Sharde-nothing-architecture">Shared nothing architecture (or
sharding)</a></em>.</p> <p>Following tests were using Synopse <em>mORMot</em> framework 1.18, compiled
with Delphi XE4, against SQLite 3.8.4.3.</p>
<p>We won't show all database engine, but the most representative ones.<br />
Please refer to <a href="https://blog.synopse.info?post/post/2014/03/07/Support-of-MySQL%2C-DB2-and-PostgreSQL">this other benchmark
article for some more complete information</a>.</p>
<h1>Insertion speed</h1>
<p>'MongoDB ack/no ack' for direct <em>MongoDB</em> access
(<code>SynMongoDB.pas</code>) with or without write acknowledge.</p>
<p>For the testings, we used a local <em>MongoDB</em> 2.6 instance in 64 bit
mode.</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>Direct</strong></td>
<td><strong>Batch</strong></td>
<td><strong>Trans</strong></td>
<td><strong>Batch Trans</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>466</td>
<td>437</td>
<td>81754</td>
<td>108752</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>2012</td>
<td>2044</td>
<td>84550</td>
<td>111731</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off exc)</strong></td>
<td>25079</td>
<td>28192</td>
<td>83943</td>
<td>115159</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>69961</td>
<td>94871</td>
<td>87279</td>
<td>118657</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>232385</td>
<td>400608</td>
<td>252678</td>
<td>402803</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>242812</td>
<td>409131</td>
<td>240003</td>
<td>405712</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>490</td>
<td>11918</td>
<td>87556</td>
<td>151144</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>2141</td>
<td>47266</td>
<td>89249</td>
<td>160616</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>33199</td>
<td>145471</td>
<td>90025</td>
<td>158815</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>76411</td>
<td>184706</td>
<td>89834</td>
<td>192618</td>
</tr>
<tr align="center">
<td><strong>MongoDB (ack)</strong></td>
<td>10081</td>
<td>84585</td>
<td>9800</td>
<td>85232</td>
</tr>
<tr align="center">
<td><strong>MongoDB (no ack)</strong></td>
<td>33223</td>
<td>189186</td>
<td>27974</td>
<td>206355</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>474</td>
<td>11917</td>
<td>36740</td>
<td>55767</td>
</tr>
<tr align="center">
<td><strong>FireDAC SQlite3</strong></td>
<td>20735</td>
<td>40083</td>
<td>40408</td>
<td>121359</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>10056</td>
<td>10155</td>
<td>18769</td>
<td>20335</td>
</tr>
<tr align="center">
<td><strong>FireDAC Firebird</strong></td>
<td>19742</td>
<td>48684</td>
<td>19904</td>
<td>47803</td>
</tr>
<tr align="center">
<td><strong>MSSQL2012 local</strong></td>
<td>3470</td>
<td>35510</td>
<td>10750</td>
<td>47653</td>
</tr>
<tr align="center">
<td><strong>ODBC MSSQL2012</strong></td>
<td>3659</td>
<td>6252</td>
<td>5537</td>
<td>6290</td>
</tr>
<tr align="center">
<td><strong>FireDAC MSSQL2012</strong></td>
<td>3276</td>
<td>5838</td>
<td>9540</td>
<td>40040</td>
</tr>
<tr align="center">
<td><strong>ZEOS PostgreSQL</strong></td>
<td>2953</td>
<td>23740</td>
<td>6913</td>
<td>29780</td>
</tr>
<tr align="center">
<td><strong>ODBC PostgreSQL</strong></td>
<td>2902</td>
<td>25040</td>
<td>3576</td>
<td>28714</td>
</tr>
<tr align="center">
<td><strong>FireDAC PostgreSQL</strong></td>
<td>3054</td>
<td>23329</td>
<td>7149</td>
<td>24844</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Batch+Trans|Trans|Batch|Direct&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,409131&chds=0,409131,0,409131,0,409131,0,409131,0,409131&chd=t:466,437,81754,108752|2012,2044,84550,111731|25079,28192,83943,115159|69961,94871,87279,118657|232385,400608,252678,402803|242812,409131,240003,405712|490,11918,87556,151144|2141,47266,89249,160616|33199,145471,90025,158815|76411,184706,89834,192618|10081,84585,9800,85232|33223,189186,27974,206355|474,11917,36740,55767|20735,40083,40408,121359|10056,10155,18769,20335|19742,48684,19904,47803|3470,35510,10750,47653|3659,6252,5537,6290|3276,5838,9540,40040|2953,23740,6913,29780|2902,25040,3576,28714|3054,23329,7149,24844&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|MongoDB+%28ack%29|MongoDB+%28no+ack%29|ZEOS+SQlite3|FireDAC+SQlite3|ZEOS+Firebird|FireDAC+Firebird|MSSQL2012+local|ODBC+MSSQL2012|FireDAC+MSSQL2012|ZEOS+PostgreSQL|ODBC+PostgreSQL|FireDAC+PostgreSQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|FireDAC+PostgreSQL|ODBC+PostgreSQL|ZEOS+PostgreSQL|FireDAC+MSSQL2012|ODBC+MSSQL2012|MSSQL2012+local|FireDAC+Firebird|ZEOS+Firebird|FireDAC+SQlite3|ZEOS+SQlite3|MongoDB+%28no+ack%29|MongoDB+%28ack%29|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,409131&chds=0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131&chd=t:466,2012,25079,69961,232385,242812,490,2141,33199,76411,10081,33223,474,20735,10056,19742,3470,3659,3276,2953,2902,3054|437,2044,28192,94871,400608,409131,11918,47266,145471,184706,84585,189186,11917,40083,10155,48684,35510,6252,5838,23740,25040,23329|81754,84550,83943,87279,252678,240003,87556,89249,90025,89834,9800,27974,36740,40408,18769,19904,10750,5537,9540,6913,3576,7149|108752,111731,115159,118657,402803,405712,151144,160616,158815,192618,85232,206355,55767,121359,20335,47803,47653,6290,40040,29780,28714,24844&chdl=Direct|Batch|Trans|Batch+Trans" /></p>
<p><em>MongoDB</em> bulk insertion has been implemented, which shows an amazing
speed increase in Batch mode. Depending on the <em>MongoDB write concern</em>
mode, insertion speed can be very high: by default, every write process will be
acknowledge by the server, but you can by-pass this request if you set the
<code>wcUnacknowledged</code> mode - note that in this case, any error (e.g. an
unique field duplicated value) will never be notified, so it should not be used
in production, unless you need this feature to quickly populate a database, or
consolidate some data as fast as possible.</p>
<h1>Read speed</h1>
<p>The same records are now read, either one by one, either as a list:</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>By one</strong></td>
<td><strong>All Virtual</strong></td>
<td><strong>All Direct</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>21607</td>
<td>455083</td>
<td>458757</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>22177</td>
<td>456454</td>
<td>458001</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off exc)</strong></td>
<td>98014</td>
<td>454215</td>
<td>457540</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>99190</td>
<td>461808</td>
<td>464252</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>235504</td>
<td>756773</td>
<td>750300</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>233666</td>
<td>332402</td>
<td>733460</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>103917</td>
<td>210863</td>
<td>458379</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>101498</td>
<td>209634</td>
<td>441033</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>101839</td>
<td>218292</td>
<td>439947</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>102414</td>
<td>185494</td>
<td>438904</td>
</tr>
<tr align="center">
<td><strong>MongoDB (ack)</strong></td>
<td>8002</td>
<td>242353</td>
<td>251268</td>
</tr>
<tr align="center">
<td><strong>MongoDB (no ack)</strong></td>
<td>8234</td>
<td>252079</td>
<td>254582</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>31135</td>
<td>173593</td>
<td>263060</td>
</tr>
<tr align="center">
<td><strong>FireDAC SQlite3</strong></td>
<td>6318</td>
<td>67169</td>
<td>92291</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>12076</td>
<td>67853</td>
<td>85828</td>
</tr>
<tr align="center">
<td><strong>FireDAC Firebird</strong></td>
<td>1918</td>
<td>37113</td>
<td>44894</td>
</tr>
<tr align="center">
<td><strong>MSSQL2012 local</strong></td>
<td>7904</td>
<td>182401</td>
<td>349797</td>
</tr>
<tr align="center">
<td><strong>ODBC MSSQL2012</strong></td>
<td>8693</td>
<td>113973</td>
<td>178526</td>
</tr>
<tr align="center">
<td><strong>FireDAC MSSQL2012</strong></td>
<td>3054</td>
<td>63730</td>
<td>86051</td>
</tr>
<tr align="center">
<td><strong>ZEOS PostgreSQL</strong></td>
<td>7031</td>
<td>122327</td>
<td>176298</td>
</tr>
<tr align="center">
<td><strong>ODBC PostgreSQL</strong></td>
<td>7281</td>
<td>66843</td>
<td>91489</td>
</tr>
<tr align="center">
<td><strong>FireDAC PostgreSQL</strong></td>
<td>1644</td>
<td>45184</td>
<td>61252</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|All+Direct|All+Virtual|By+one&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,756773&chds=0,756773,0,756773,0,756773&chd=t:21607,455083,458757|22177,456454,458001|98014,454215,457540|99190,461808,464252|235504,756773,750300|233666,332402,733460|103917,210863,458379|101498,209634,441033|101839,218292,439947|102414,185494,438904|8002,242353,251268|8234,252079,254582|31135,173593,263060|6318,67169,92291|12076,67853,85828|1918,37113,44894|7904,182401,349797|8693,113973,178526|3054,63730,86051|7031,122327,176298|7281,66843,91489|1644,45184,61252&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|MongoDB+%28ack%29|MongoDB+%28no+ack%29|ZEOS+SQlite3|FireDAC+SQlite3|ZEOS+Firebird|FireDAC+Firebird|MSSQL2012+local|ODBC+MSSQL2012|FireDAC+MSSQL2012|ZEOS+PostgreSQL|ODBC+PostgreSQL|FireDAC+PostgreSQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|FireDAC+PostgreSQL|ODBC+PostgreSQL|ZEOS+PostgreSQL|FireDAC+MSSQL2012|ODBC+MSSQL2012|MSSQL2012+local|FireDAC+Firebird|ZEOS+Firebird|FireDAC+SQlite3|ZEOS+SQlite3|MongoDB+%28no+ack%29|MongoDB+%28ack%29|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,756773&chds=0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773&chd=t:21607,22177,98014,99190,235504,233666,103917,101498,101839,102414,8002,8234,31135,6318,12076,1918,7904,8693,3054,7031,7281,1644|455083,456454,454215,461808,756773,332402,210863,209634,218292,185494,242353,252079,173593,67169,67853,37113,182401,113973,63730,122327,66843,45184|458757,458001,457540,464252,750300,733460,458379,441033,439947,438904,251268,254582,263060,92291,85828,44894,349797,178526,86051,176298,91489,61252&chdl=By+one|All+Virtual|All+Direct" /></p>
<p><em>MongoDB</em> appears as a serious competitor to SQL databases, with the
potential benefit of horizontal scaling and installation/administration ease -
performance is very high, and its document-based storage fits perfectly with
<em>mORMot</em>'s advanced ORM features like <em><a href="https://blog.synopse.info?post/post/2011/07/03/Sharde-nothing-architecture">Shared nothing architecture (or
sharding)</a></em>.</p>
<p>You can get more information about <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-database-access">low-level integration of
<em>MongoDB</em></a> in <em>mORMot</em>, or our integrated <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">ORM/ODM support in the
framework</a>.<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1742">welcome
on our forum</a>, as usual!</p>MongoDB + mORMot ORM = ODMurn:md5:232877d989279d738aab47cacf5358f82014-05-07T09:14:00+02:002014-05-07T13:52:17+02:00AB4327-GANDImORMot FrameworkACIDAJAXBatchblogDatabaseDelphiDocumentationDomainDrivenGoodPracticeJSONmappingModelMongoDBmORMotNoSQLOpenSourceORMperformanceRestshardingSourceSQLtransactionUnicode<p><em><a href="http://www.mongodb.org/">MongoDB</a></em> (from "humongous") is
a cross-platform document-oriented database system, and certainly the best
known NoSQL database.<br />
According to <a href="http://db-engines.com">http://db-engines.com</a> in April
2014, <em>MongoDB</em> is in 5th place of the most popular types of database
management systems, and first place for NoSQL database management
systems.<br />
Our <em>mORMot</em> gives premium access to this database, featuring full
<em>NoSQL and Object-Document Mapping (ODM)</em> abilities to the
framework.</p>
<p><img src="http://upload.wikimedia.org/wikipedia/en/thumb/e/eb/MongoDB_Logo.png/320px-MongoDB_Logo.png" alt="" /></p>
<p>Integration is made at two levels:</p>
<ul>
<li>Direct low-level access to the <em>MongoDB</em> server, in the
<code>SynMongoDB.pas</code> unit;</li>
<li>Close integration with our ORM (which becomes <em>defacto</em> an ODM), in
the <code>mORMotMongoDB.pas</code> unit.</li>
</ul>
<p><em>MongoDB</em> eschews the traditional table-based relational database
structure in favor of JSON-like documents with dynamic schemas
(<em>MongoDB</em> calls the format BSON), which matches perfectly
<em>mORMot</em>'s RESTful approach.</p>
<p>This second article will focus on integration of <em>MongoDB</em> with our
ORM.</p> <h3>MongoDB + ORM = ODM</h3>
<p>The <code>mORMotMongoDB.pas</code> unit is able to let any
<code>TSQLRecord</code> class be persisted on a remote <em>MongoDB</em>
server.</p>
<p>As a result, our ORM is able to be used as a <em>NoSQL and Object-Document
Mapping (ODM)</em> framework, with almost no code change.<br />
Any <em>MongoDB</em> database can be accessed via RESTful commands, using JSON
over HTTP.</p>
<p>This integration benefits from the other parts of the framework (e.g. our
UTF-8 dedicated process, which is also the native encoding for BSON), so you
can easily mix SQL and NoSQL databases with the exact same code, and are still
able to tune any SQL or <em>MongoDB</em> request in your code, if
necessary.</p>
<h3>Register the TSQLRecord class</h3>
<p>In the database model, we define a <code>TSQLRecord</code> class, as
usual:</p>
<pre>
TSQLORM = <strong>class</strong>(TSQLRecord)
<strong>private</strong>
fAge: integer;
fName: RawUTF8;
fDate: TDateTime;
fValue: <strong>variant</strong>;
fInts: TIntegerDynArray;
fCreateTime: TCreateTime;
fData: TSQLRawBlob;
<strong>published</strong>
<strong>property</strong> Name: RawUTF8 <strong>read</strong> fName <strong>write</strong> fName <strong>stored</strong> AS_UNIQUE;
<strong>property</strong> Age: integer <strong>read</strong> fAge <strong>write</strong> fAge;
<strong>property</strong> Date: TDateTime <strong>read</strong> fDate <strong>write</strong> fDate;
<strong>property</strong> Value: <strong>variant read</strong> fValue <strong>write</strong> fValue;
<strong>property</strong> Ints: TIntegerDynArray <strong>index</strong> 1 <strong>read</strong> fInts <strong>write</strong> fInts;
<strong>property</strong> Data: TSQLRawBlob <strong>read</strong> fData <strong>write</strong> fData;
<strong>property</strong> CreateTime: TCreateTime <strong>read</strong> fCreateTime <strong>write</strong> fCreateTime;
<strong>end</strong>;
</pre>
<p>Note that we did not define any <strong><code>index</code></strong>
<code>...</code> values for the <code>RawUTF8</code> property, as we need for
external SQL databases, since <em>MongoDB</em> does not expect any restriction
about text fields length.</p>
<p>The property values will be stored in the native <em>MongoDB</em> layout,
i.e. with a better coverage than the SQL types:</p>
<table>
<tbody>
<tr>
<td><strong>Delphi</strong></td>
<td><strong><em>MongoDB</em></strong></td>
<td><strong>Remarks</strong></td>
</tr>
<tr>
<td><code>byte</code></td>
<td>int32</td>
</tr>
<tr>
<td><code>word</code></td>
<td>int32</td>
</tr>
<tr>
<td><code>integer</code></td>
<td>int32</td>
</tr>
<tr>
<td><code>cardinal</code></td>
<td>N/A</td>
<td>You should use <code>Int64</code> instead</td>
</tr>
<tr>
<td><code>Int64</code></td>
<td>int64</td>
</tr>
<tr>
<td><code>boolean</code></td>
<td>boolean</td>
<td>0 is <code>false</code>, anything else is <code>true</code></td>
</tr>
<tr>
<td>enumeration</td>
<td>int32</td>
<td>store the ordinal value of the enumerated item(i.e. starting at 0 for the
first element)</td>
</tr>
<tr>
<td>set</td>
<td>int32</td>
<td>each bit corresponding to an enumerated item (therefore a set of up to 64
elements can be stored in such a field)</td>
</tr>
<tr>
<td><code>single</code></td>
<td>double</td>
</tr>
<tr>
<td><code>double</code></td>
<td>double</td>
</tr>
<tr>
<td><code>extended</code></td>
<td>double</td>
<td>stored as <code>double</code> (precision lost)</td>
</tr>
<tr>
<td><code>currency</code></td>
<td>double</td>
<td>safely converted to/from <code>currency</code> type with fixed decimals,
without rounding error</td>
</tr>
<tr>
<td><code>RawUTF8</code></td>
<td>UTF-8</td>
<td>this is the <strong>preferred</strong> field type for storing some textual
content in the ORM</td>
</tr>
<tr>
<td><code>WinAnsiString</code></td>
<td>UTF-8</td>
<td><em>WinAnsi</em> char-set (code page 1252) in Delphi</td>
</tr>
<tr>
<td><code>RawUnicode</code></td>
<td>UTF-8</td>
<td><em>UCS2</em> char-set in Delphi, as <code>AnsiString</code></td>
</tr>
<tr>
<td><code>WideString</code></td>
<td>UTF-8</td>
<td><em>UCS2</em> char-set, as COM BSTR type (Unicode in all version of
Delphi)</td>
</tr>
<tr>
<td><code>SynUnicode</code></td>
<td>UTF-8</td>
<td>Will be either <code>WideString</code> before Delphi 2009, or
<code>UnicodeString</code> later</td>
</tr>
<tr>
<td><code>string</code></td>
<td>UTF-8</td>
<td>Not to be used before Delphi 2009 (unless you may loose some data during
conversion) - <code>RawUTF8</code> is preferred in all cases</td>
</tr>
<tr>
<td><code>TDateTime</code></td>
<td>datetime</td>
<td>ISO 8601 encoded date time</td>
</tr>
<tr>
<td><code>TTimeLog</code></td>
<td>int64</td>
<td>as proprietary fast <code>Int64</code> date time</td>
</tr>
<tr>
<td><code>TModTime</code></td>
<td>int64</td>
<td>the server date time will be stored when a record is modified (as
proprietary fast <code>Int64</code>)</td>
</tr>
<tr>
<td><code>TCreateTime</code></td>
<td>int64</td>
<td>the server date time will be stored when a record is created (as
proprietary fast <code>Int64</code>)</td>
</tr>
<tr>
<td><code>TSQLRecord</code></td>
<td>int32</td>
<td><code>RowID</code> pointing to another record (warning: the field value
contains <code>pointer(RowID)</code>, not a valid object instance - the record
content must be retrieved with late-binding via its <code>ID</code> using a
<code>PtrInt(Field)</code> typecast or the <code>Field.ID</code> method), or by
using e.g. <code>CreateJoined()</code></td>
</tr>
<tr>
<td><code>TSQLRecordMany</code></td>
<td>nothing</td>
<td>data is stored in a separate <em>pivot</em> table; for MongoDB, you should
better use <em>data sharding</em>, and an embedded sub-document</td>
</tr>
<tr>
<td><code>TRecordReference</code></td>
<td>int32</td>
<td>store both <code>ID</code> and <code>TSQLRecord</code> type in a
<code>RecordRef</code>-like value (use e.g. <code>TSQLRest.
Retrieve(Reference)</code> to get a record content)</td>
</tr>
<tr>
<td><code>TPersistent</code></td>
<td>object</td>
<td>BSON object (from <code>ObjectToJSON</code>)</td>
</tr>
<tr>
<td><code>TCollection</code></td>
<td>array</td>
<td>BSON array of objects (from <code>ObjectToJSON</code>)</td>
</tr>
<tr>
<td><code>TObjectList</code></td>
<td>array</td>
<td>BSON array of objects (from <code>ObjectToJSON</code>) - see
<code>TJSONSerializer. RegisterClassForJSON</code> <em>TObjectList
serialization</em></td>
</tr>
<tr>
<td><code>TStrings</code></td>
<td>array</td>
<td>BSON array of strings (from <code>ObjectToJSON</code>)</td>
</tr>
<tr>
<td><code>TRawUTF8List</code></td>
<td>array</td>
<td>BSON array of string (from <code>ObjectToJSON</code>)</td>
</tr>
<tr>
<td>any <code>TObject</code></td>
<td>object</td>
<td>See <code>TJSONSerializer. RegisterCustomSerializer</code> <em>TObject
serialization</em></td>
</tr>
<tr>
<td><code>TSQLRawBlob</code></td>
<td>binary</td>
<td>BSON binary for blob storage - note that you need to retrieve explicitly
such fields, which are not part of the default <code>Retrieve()</code> method;
this type is an alias to <code>RawByteString</code></td>
</tr>
<tr>
<td><em>dynamic arrays</em></td>
<td>array<br />
binary</td>
<td>if the dynamic array can be saved as true JSON, will be stored as BSON
array - otherwise, will be stored in the <code>TDynArray.SaveTo</code> binary
format - note that <code>TByteDynArray</code> will always be stored as BSON
binary, so that <code>TSQLRawBlob</code> may be reserved to store content on
<em>GridFS</em> in the future</td>
</tr>
<tr>
<td><code>variant</code></td>
<td>array<br />
object</td>
<td>BSON number, text, object or array, depending on <em>TDocVariant custom
variant type</em> or <code>TBSONVariant</code> stored value</td>
</tr>
<tr>
<td><code>record</code></td>
<td>binary<br />
object</td>
<td>BSON as defined in code by overriding
<code>TSQLRecord.InternalRegisterCustomProperties</code></td>
</tr>
</tbody>
</table>
<p>On the server side (there won't be any difference for the client), you
define a <code>TMongoDBClient</code>, and assign it to a given
<code>TSQLRecord</code> class:</p>
<pre>
MongoClient := TMongoClient.Create('localhost',27017);
DB := MongoClient.Database['dbname'];
Model := TSQLModel.Create([TSQLORM]);
Client := TSQLRestClientDB.Create(Model,<strong>nil</strong>,':memory:',TSQLRestServerDB);
<span style="background-color:yellow;"><strong>if</strong> StaticMongoDBRegister(TSQLORM,fClient.Server,fDB,'collectionname')=<strong>nil then</strong></span>
<strong>raise</strong> Exception.Create('Error');
</pre>
<p>And... that's all!</p>
<p>You can then use any ORM command, as usual:</p>
<pre>
writeln(Client.TableRowCount(TSQLORM)=0);
</pre>
<p>As with external databases, you can specify the field names mapping between
the objects and the <em>MongoDB</em> collection.<br />
By default, the <code>TSQLRecord.ID</code> property is mapped to the
<em>MongoDB</em>'s <code>_id</code> field, and the ORM will populate this
<code>_id</code> field with a sequence of integer values, just like any
<code>TSQLRecord</code> table.<br />
You can specify your own mapping, using for instance:</p>
<pre>
aModel.Props[aClass].ExternalDB.MapField(..)
</pre>
<p>Since the field names are stored within the document itself, it may be a
good idea to use shorter naming for the <em>MongoDB</em> collection. It may
save some storage space, when working with a huge number of documents.</p>
<p>Once the <code>TSQLRecord</code> is mapped to a <em>MongoDB</em> collection,
you can always have direct access to the <code>TMongoCollection</code> instance
later on, by calling:</p>
<pre>
(aServer.StaticDataServer[aClass] <strong>as</strong> TSQLRestServerStaticMongoDB).Collection
</pre>
<p>This may allow any specific task, including any tuned query or process.</p>
<h3>ORM/ODM CRUD methods</h3>
<p>You can add documents with the standard CRUD methods of the ORM, as
usual:</p>
<pre>
R := TSQLORM.Create;
<strong>try</strong>
<strong>for</strong> i := 1 <strong>to</strong> COLL_COUNT <strong>do begin</strong>
R.Name := 'Name '+Int32ToUTF8(i);
R.Age := i;
R.Date := 1.0*(30000+i);
R.Value := _ObjFast(['num',i]);
R.Ints := <strong>nil</strong>;
R.DynArray(1).Add(i);
<span style="background-color:yellow;">assert(Client.Add(R,True)=i);</span>
<strong>end</strong>;
<strong>finally</strong>
R.Free;
<strong>end</strong>;
</pre>
<p>As we already saw, the framework is able to handle any kind of properties,
including complex types like <em>dynamic arrays</em> or
<code>variant</code>.<br />
In the above code, a <code>TDocVariant</code> document has been stored in
<code>R.Value</code>, and a dynamic array of <code>integer</code> values is
accessed via its <code>index 1</code> shortcut and the
<code>TSQLRecord.DynArray()</code> method.</p>
<p>The usual <code>Retrieve</code> / <code>Delete</code> / <code>Update</code>
methods are available:</p>
<pre>
R := TSQLORM.Create;
<strong>try</strong>
<strong>for</strong> i := 1 <strong>to</strong> COLL_COUNT <strong>do begin</strong>
<span style="background-color:yellow;">Check(Client.Retrieve(i,R));</span>
<em>// here R instance contains all values of one document, excluding BLOBs</em>
<strong>end</strong>;
<strong>finally</strong>
R.Free;
<strong>end</strong>;
</pre>
<p>You can define a WHERE clause, as if the back-end where a regular SQL
database:</p>
<pre>
R := TSQLORM.CreateAndFillPrepare(Client,'ID=?',[i]);
<strong>try</strong>
...
</pre>
<p>Current implementation understand one condition over one single field, with
<code>= > >= < <= IN</code> clauses. More advanced queries are
possible, but won't be handled as SQL, but via direct access to the
<code>TMongoDBCollection</code>.</p>
<p>To perform a query and retrieve the content of several documents, you can
use regular <code>CreateAndFillPrepare</code> or <code>FillPrepare</code>
methods:</p>
<pre>
<span style="background-color:yellow;">R := TSQLORM.CreateAndFillPrepare(Client,'');</span>
<strong>try</strong>
n := 0;
<span style="background-color:yellow;"><strong>while</strong> R.FillOne <strong>do begin</strong></span>
<em>// here R instance contains all values of one document, excluding BLOBs</em>
inc(n);
<strong>end</strong>;
assert(n=COLL_COUNT);
<strong>finally</strong>
R.Free;
<strong>end</strong>;
</pre>
<p>A WHERE clause can also be defined for <code>CreateAndFillPrepare</code> or
<code>FillPrepare</code> methods.</p>
<h3>BATCH mode</h3>
<p>In addition to individual CRUD operations, our <em>MongoDB</em> is able to
use BATCH mode for adding or deleting documents.</p>
<p>You can write the exact same code as with any SQL back-end:</p>
<pre>
Client.BatchStart(TSQLORM);
R := TSQLORM.Create;
<strong>try</strong>
<strong>for</strong> i := 1 <strong>to</strong> COLL_COUNT <strong>do begin</strong>
R.Name := 'Name '+Int32ToUTF8(i);
R.Age := i;
R.Date := 1.0*(30000+i);
R.Value := _ObjFast(['num',i]);
R.Ints := <strong>nil</strong>;
R.DynArray(1).Add(i);
assert(Client.BatchAdd(R,True)>=0);
<strong>end</strong>;
<strong>finally</strong>
R.Free;
<strong>end</strong>;
assert(Client.BatchSend(IDs)=HTML_SUCCESS);
</pre>
<p>Or for deletion:</p>
<pre>
Client.BatchStart(TSQLORM);
<strong>for</strong> i := 5 <strong>to</strong> COLL_COUNT <strong>do</strong>
<strong>if</strong> i <strong>mod</strong> 5=0 <strong>then</strong>
assert(fClient.BatchDelete(i)>=0);
assert(Client.BatchSend(IDs)=HTML_SUCCESS);
</pre>
<p>Speed benefit may be huge in regard to individual Add/Delete operations,
even on a local <em>MongoDB</em> server. We will see some benchmark numbers
now.</p>
<h3>ORM/ODM performance</h3>
<p>You can take a look at <em><a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-benchmark">Data access benchmark</a></em> to
compare <em>MongoDB</em> as back-end for our ORM classes.</p>
<p>In respect to external SQL engines, it features very high speed, low CPU
use, and almost no difference in use. We interfaced the <code>BatchAdd()</code>
and <code>BatchDelete()</code> methods to benefit of <em>MongoDB</em> BULK
process, and avoided most memory allocation during the process.</p>
<p>Here are some numbers, extracted from the <code>MongoDBTests.dpr</code>
sample, which reflects the performance of our ORM/ODM, depending on the
<em>Write Concern</em> mode used:</p>
<pre>
2. ORM
<br />2.1. ORM with acknowledge:
- Connect to local server: 6 assertions passed 18.65ms
- Insert: 5,002 assertions passed 521.25ms
<span style="background-color:yellow;"> 5000 rows inserted in 520.65ms i.e. 9603/s, aver. 104us, 2.9 MB/s</span>
- Insert in batch mode: 5,004 assertions passed 65.37ms
<span style="background-color:yellow;"> 5000 rows inserted in 65.07ms i.e. 76836/s, aver. 13us, 8.4 MB/s</span>
- Retrieve: 45,001 assertions passed 640.95ms
5000 rows retrieved in 640.75ms i.e. 7803/s, aver. 128us, 2.1 MB/s
- Retrieve all: 40,001 assertions passed 20.79ms
<span style="background-color:yellow;"> 5000 rows retrieved in 20.33ms i.e. 245941/s, aver. 4us, 27.1 MB/s</span>
- Retrieve one with where clause: 45,410 assertions passed 673.01ms
5000 rows retrieved in 667.17ms i.e. 7494/s, aver. 133us, 2.0 MB/s
- Update: 40,002 assertions passed 681.31ms
5000 rows updated in 660.85ms i.e. 7565/s, aver. 132us, 2.4 MB/s
- Blobs: 125,003 assertions passed 2.16s
5000 rows updated in 525.97ms i.e. 9506/s, aver. 105us, 2.4 MB/s
- Delete: 38,003 assertions passed 175.86ms
1000 rows deleted in 91.37ms i.e. 10944/s, aver. 91us, 2.3 MB/s
- Delete in batch mode: 33,003 assertions passed 34.71ms
<span style="background-color:yellow;"> 1000 rows deleted in 14.90ms i.e. 67078/s, aver. 14us, 597 KB/s</span>
Total failed: 0 / 376,435 - ORM with acknowledge PASSED 5.00s
<br />2.2. ORM without acknowledge:
- Connect to local server: 6 assertions passed 16.83ms
- Insert: 5,002 assertions passed 179.79ms
<span style="background-color:yellow;"> 5000 rows inserted in 179.15ms i.e. 27908/s, aver. 35us, 3.9 MB/s</span>
- Insert in batch mode: 5,004 assertions passed 66.30ms
<span style="background-color:yellow;"> 5000 rows inserted in 31.46ms i.e. 158891/s, aver. 6us, 17.5 MB/s</span>
- Retrieve: 45,001 assertions passed 642.05ms
5000 rows retrieved in 641.85ms i.e. 7789/s, aver. 128us, 2.1 MB/s
- Retrieve all: 40,001 assertions passed 20.68ms
<span style="background-color:yellow;"> 5000 rows retrieved in 20.26ms i.e. 246718/s, aver. 4us, 27.2 MB/s</span>
- Retrieve one with where clause: 45,410 assertions passed 680.99ms
5000 rows retrieved in 675.24ms i.e. 7404/s, aver. 135us, 2.0 MB/s
- Update: 40,002 assertions passed 231.75ms
5000 rows updated in 193.74ms i.e. 25807/s, aver. 38us, 3.6 MB/s
- Blobs: 125,003 assertions passed 1.44s
5000 rows updated in 150.58ms i.e. 33202/s, aver. 30us, 2.6 MB/s
- Delete: 38,003 assertions passed 103.57ms
1000 rows deleted in 19.73ms i.e. 50668/s, aver. 19us, 2.4 MB/s
- Delete in batch mode: 33,003 assertions passed 47.50ms
<span style="background-color:yellow;"> 1000 rows deleted in 364us i.e. 2747252/s, aver. 0us, 23.4 MB/s</span>
Total failed: 0 / 376,435 - ORM without acknowledge PASSED 3.44s
</pre>
<p>As for direct <em>MongoDB</em> access, the <code>wcUnacknowledged</code> is
not to be used on production, but may be very useful in some particular
scenarios.<br />
As expected, the reading process is not impacted by the <em>Write Concern</em>
mode set.</p>
<p>You can take a look at the previous blog article, about <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-database-access">low-level <em>MongoDB</em> direct
access</a>.<br />
Or check out <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-benchmark">more complete
benchmark results</a>, to compare <em>MongoDB</em> with other SQL back-end
handled by our ORM.<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1742">welcome
on our forum</a>, as usual!</p>Direct MongoDB database accessurn:md5:783573bb416f11adc20f4cffa50477cd2014-05-07T09:10:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkACIDAJAXBatchblogDatabaseDelphiDocumentationDomainDrivenGoodPracticeJSONmappingModelMongoDBmORMotNoSQLOpenSourceORMperformanceRestshardingSourceSQLtransactionUnicode<p><em><a href="http://www.mongodb.org/">MongoDB</a></em> (from "humongous") is
a cross-platform document-oriented database system, and certainly the best
known NoSQL database.<br />
According to <a href="http://db-engines.com">http://db-engines.com</a> in April
2014, <em>MongoDB</em> is in 5th place of the most popular types of database
management systems, and first place for NoSQL database management
systems.<br />
Our <em>mORMot</em> framework gives premium access to this database,
featuring full <em>NoSQL and Object-Document Mapping (ODM)</em> abilities to
the framework.</p>
<p><img src="http://upload.wikimedia.org/wikipedia/en/thumb/e/eb/MongoDB_Logo.png/320px-MongoDB_Logo.png" alt="" /></p>
<p>Integration is made at two levels:</p>
<ul>
<li>Direct low-level access to the <em>MongoDB</em> server, in the
<code>SynMongoDB.pas</code> unit;</li>
<li>Close integration with our ORM (which becomes <em>defacto</em> an ODM), in
the <code>mORMotMongoDB.pas</code> unit.</li>
</ul>
<p><em>MongoDB</em> eschews the traditional table-based relational database
structure in favor of JSON-like documents with dynamic schemas
(<em>MongoDB</em> calls the format BSON), which matches perfectly
<em>mORMot</em>'s RESTful approach.</p>
<p>In this first article, we will detail direct low-level access to the
<em>MongoDB</em> server, via the <code>SynMongoDB.pas</code> unit.</p> <h3>MongoDB client</h3>
<p>The <code>SynMongoDB.pas</code> unit features direct optimized access to a
<em>MongoDB</em> server.</p>
<p>It gives access to any BSON data, including documents, arrays, and
<em>MongoDB</em>'s custom types (like ObjectID, dates, binary, regex or
Javascript):</p>
<ul>
<li>For instance, a <code>TBSONObjectID</code> can be used to create some
genuine document identifiers on the client side (<em>MongoDB</em> does not
generate the IDs for you: a common way is to generate unique IDs on the client
side);</li>
<li>Generation of BSON content from any Delphi types (via
<code>TBSONWriter</code>);</li>
<li>Fast in-place parsing of the BSON stream, without any memory allocation
(via <code>TBSONElement</code>);</li>
<li>A <code>TBSONVariant</code> custom variant type, to store
<em>MongoDB</em>'s custom type values;</li>
<li>Interaction with the <code>SynCommons</code>' <em>TDocVariant custom
variant type</em> as document storage and late-binding access;</li>
<li>Marshalling BSON to and from JSON, with the <em>MongoDB</em> extended
syntax for handling its custom types.</li>
</ul>
<p>This unit defines some objects able to connect and manage databases and
collections of documents on any <em>MongoDB</em> servers farm:</p>
<ul>
<li>Connection to one or several servers, including secondary hosts, via the
<code>TMongoClient</code> class;</li>
<li>Access to any database instance, via the <code>TMongoDatabase</code>
class;</li>
<li>Access to any collection, via the <code>TMongoCollection</code> class;</li>
<li>It features some nice abilities about speed, like BULK insert or delete
mode, and explicit <em>Write Concern</em> settings.</li>
</ul>
<p>At collection level, you can have direct access to the data, with high level
structures like <code>TDocVariant</code>/<code>TBSONVariant</code>, with
easy-to-read JSON, or low level BSON content.<br />
You can also tune most aspects of the client process, e.g. about error handling
or <em>write concerns</em> (i.e. how remote data modifications are
acknowledged).</p>
<h3>Connecting to a server</h3>
<p>Here is some sample code, which is able to connect to a <em>MongoDB</em>
server, and returns the server time:</p>
<pre>
<strong>var</strong> Client: TMongoClient;
DB: TMongoDatabase;
serverTime: TDateTime;
res: <strong>variant</strong>; <em>// we will return the command result as TDocVariant</em>
errmsg: RawUTF8;
<strong>begin</strong>
Client := TMongoClient.Create('localhost',27017);
<strong>try</strong>
DB := Client.Database['mydb'];
writeln('Connecting to ',DB.Name); <em>// will write 'mydb'</em>
errmsg := DB.RunCommand('hostInfo',res); <em>// run a command</em>
<strong>if</strong> errmsg<>'' <strong>then</strong>
exit; <em>// quit on any error</em>
serverTime := res.system.currentTime; <em>// direct conversion to TDateTime</em>
writeln('Server time is ',DateTimeToStr(serverTime));
<strong>finally</strong>
Client.Free; <em>// will release the DB instance</em>
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>Note that for this low-level command, we used a <code>TDocVariant</code>,
and its late-binding abilities.</p>
<p>In fact, if you put your mouse over the <code>res</code> variable during
debugging, you will see the following JSON content:</p>
<pre>
{"system":{"currentTime":"2014-05-06T15:24:25","hostname":"Acer","cpuAddrSize":64,"memSizeMB":3934,"numCores":4,"cpuArch":"x86_64","numaEnabled":false},"os":{"type":"Windows","name":"Microsoft Windows 7","version":"6.1 SP1 (build 7601)"},"extra":{"pageSize":4096},"ok":1}
</pre>
<p>And we simply access to the server time by writing
<code>res.system.currentTime</code>.</p>
<h3>Adding some documents to the collection</h3>
<p>We will now explain how to add documents to a given collection.</p>
<p>We assume that we have a <code>DB: TMongoDatabase</code> instance available.
Then we will create the documents with a <code>TDocVariant</code> instance,
which will be filled via late-binding, and via a <code>doc.Clear</code>
pseudo-method used to flush any previous property value:</p>
<pre>
<strong>var</strong> Coll: TMongoCollection;
doc: <strong>variant</strong>;
i: integer;
<strong>begin</strong>
Coll := DB.CollectionOrCreate[COLL_NAME];
TDocVariant.New(doc);
<strong>for</strong> i := 1 <strong>to</strong> 10 <strong>do</strong>
<strong>begin</strong>
doc.Clear;
doc.Name := 'Name '+IntToStr(i+1);
doc.Number := i;
<span style="background-color:yellow;">Coll.Save(doc);</span>
writeln('Inserted with _id=',doc._id);
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>Thanks to <code>TDocVariant</code> late-binding abilities, code is pretty
easy to understand and maintain.</p>
<p>This code will display the following on the console:</p>
<pre>
Inserted with _id=5369029E4F901EE8114799D9
Inserted with _id=5369029E4F901EE8114799DA
Inserted with _id=5369029E4F901EE8114799DB
Inserted with _id=5369029E4F901EE8114799DC
Inserted with _id=5369029E4F901EE8114799DD
Inserted with _id=5369029E4F901EE8114799DE
Inserted with _id=5369029E4F901EE8114799DF
Inserted with _id=5369029E4F901EE8114799E0
Inserted with _id=5369029E4F901EE8114799E1
Inserted with _id=5369029E4F901EE8114799E2
</pre>
<p>It means that the <code>Coll.Save()</code> method was clever enough to
understand that the supplied document does not have any <code>_id</code> field,
so will compute one on the client side before sending the document data to the
<em>MongoDB</em> server.</p>
<p>We may have written:</p>
<pre>
<strong>for</strong> i := 1 <strong>to</strong> 10 <strong>do</strong>
<strong>begin</strong>
doc.Clear;
<span style="background-color:yellow;">doc._id := ObjectID;</span>
doc.Name := 'Name '+IntToStr(i+1);
doc.Number := i;
Coll.Save(doc);
writeln('Inserted with _id=',doc._id);
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>Which will compute the document identifier explicitly before calling
<code>Coll.Save()</code>.<br />
In this case, we may have called directly <code>Coll.Insert()</code>, which is
somewhat faster.</p>
<p>Note that you are not obliged to use a <em>MongoDB</em> ObjectID as
identifier. You can use any value, if you are sure that it will be genuine. For
instance, you can use an integer:</p>
<pre>
<strong>for</strong> i := 1 <strong>to</strong> 10 <strong>do</strong>
<strong>begin</strong>
doc.Clear;
<span style="background-color:yellow;">doc._id := i;</span>
doc.Name := 'Name '+IntToStr(i+1);
doc.Number := i;
<span style="background-color:yellow;">Coll.Insert(doc);</span>
writeln('Inserted with _id=',doc._id);
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>The console will display now:</p>
<pre>
Inserted with _id=1
Inserted with _id=2
Inserted with _id=3
Inserted with _id=4
Inserted with _id=5
Inserted with _id=6
Inserted with _id=7
Inserted with _id=8
Inserted with _id=9
Inserted with _id=10
</pre>
<p>Note that the <em>mORMot</em> ORM will compute a genuine series of integers
in a similar way, which will be used as expected by the
<code>TSQLRecord.ID</code> primary key property.</p>
<p>The <code>TMongoCollection</code> class can also write a list of documents,
and send them at once to the <em>MongoDB</em> server: this BULK insert mode -
close to the <em>Array Binding</em> feature of some SQL providers, and
implemented in our <em>SynDB</em> classes - see <em>BATCH sequences for
adding/updating/deleting records</em> - can increase the insertion by a factor
of 10 times, even when connected to a local instance: imagine how much time it
may save over a physical network!</p>
<p>For instance, you may write:</p>
<pre>
<strong>var</strong> docs: TVariantDynArray;
...
SetLength(docs,COLL_COUNT);
<strong>for</strong> i := 0 <strong>to</strong> COLL_COUNT-1 <strong>do begin</strong>
TDocVariant.New(docs[i]);
docs[i]._id := ObjectID; <em>// compute new ObjectID on the client side</em>
docs[i].Name := 'Name '+IntToStr(i+1);
docs[i].FirstName := 'FirstName '+IntToStr(i+COLL_COUNT);
docs[i].Number := i;
<strong>end</strong>;
Coll.Insert(docs); <em>// insert all values at once</em>
...
</pre>
<p>You will find out later for some numbers about the speed increase due to
such BULK insert.</p>
<h3>Retrieving the documents</h3>
<p>You can retrieve the document as a <code>TDocVariant</code> instance:</p>
<pre>
<strong>var</strong> doc: <strong>variant</strong>;
...
doc := Coll.FindOne(5);
writeln('Name: ',doc.Name);
writeln('Number: ',doc.Number);
</pre>
<p>Which will write on the console:</p>
<pre>
Name: Name 6
Number: 5
</pre>
<p>You have access to the whole <code>Query</code> parameter, if needed:</p>
<pre>
doc := Coll.FindDoc('{_id:?}',[5]);
doc := Coll.FindOne(5); <em>// same as previous</em>
</pre>
<p>This <code>Query</code> filter is similar to a WHERE clause in SQL. You can
write complex search patterns, if needed - see <a href="http://docs.mongodb.org/manual/reference/method/db.collection.find">http://docs.mongodb.org/manual/reference/method/db.collection.find</a>
for reference.</p>
<p>You can retrieve a list of documents, as a dynamic array of
<code>TDocVariant</code>:</p>
<pre>
<strong>var</strong> docs: TVariantDynArray;
...
Coll.FindDocs(docs);
<strong>for</strong> i := 0 <strong>to</strong> high(docs) <strong>do</strong>
writeln('Name: ',docs[i].Name,' Number: ',docs[i].Number);
</pre>
<p>Which will output:</p>
<pre>
Name: Name 2 Number: 1
Name: Name 3 Number: 2
Name: Name 4 Number: 3
Name: Name 5 Number: 4
Name: Name 6 Number: 5
Name: Name 7 Number: 6
Name: Name 8 Number: 7
Name: Name 9 Number: 8
Name: Name 10 Number: 9
Name: Name 11 Number: 10
</pre>
<p>If you want to retrieve the documents directly as JSON, we can write:</p>
<pre>
<strong>var</strong> json: RawUTF8;
...
json := Coll.FindJSON(null,null);
writeln(json);
...
</pre>
<p>This will append the following to the console:</p>
<pre>
[{"_id":1,"Name":"Name 2","Number":1},{"_id":2,"Name":"Name 3","Number":2},{"_id
":3,"Name":"Name 4","Number":3},{"_id":4,"Name":"Name 5","Number":4},{"_id":5,"N
ame":"Name 6","Number":5},{"_id":6,"Name":"Name 7","Number":6},{"_id":7,"Name":"
Name 8","Number":7},{"_id":8,"Name":"Name 9","Number":8},{"_id":9,"Name":"Name 1
0","Number":9},{"_id":10,"Name":"Name 11","Number":10}]
</pre>
<p>You can note that <code>FindJSON()</code> has two properties, which are the
<code>Query</code> filter, and a <code>Projection</code> mapping (similar to
the column names in a <code>SELECT col1,col2</code>).<br />
So we may have written:</p>
<pre>
json := Coll.FindJSON('{_id:?}',[5]);
writeln(json);
</pre>
<p>Which would output:</p>
<pre>
[{"_id":5,"Name":"Name 6","Number":5}]
</pre>
<p>Note here than we used an overloaded <code>FindJSON()</code> method, which
accept the <em>MongoDB</em> extended syntax (here, the field name is unquoted),
and parameters as variables.</p>
<p>We can specify a projection:</p>
<pre>
json := Coll.FindJSON('{_id:?}',[5],'{Name:1}');
writeln(json);
</pre>
<p>Which will only return the "Name" and "_id" fields (since <code>_id</code>
is, by <em>MongoDB</em> convention, always returned:</p>
<pre>
[{"_id":5,"Name":"Name 6"}]
</pre>
<p>To return only the "Name" field, you can specify <code>'_id:0,Name:1'</code>
as extended JSON for the <em>projection</em> parameter.</p>
<pre>
[{"Name":"Name 6"}]
</pre>
<p>There are other methods able to retrieve data, also directly as BSON binary
data. They will be used for best speed e.g. in conjunction with our ORM, but
for most end-user code, using <code>TDocVariant</code> is safer and easier to
maintain.</p>
<h3>Updating or deleting documents</h3>
<p>The <code>TMongoCollection</code> class has some methods dedicated to alter
existing documents.</p>
<p>At first, the <code>Save()</code> method can be used to update a document
which has been first retrieved:</p>
<pre>
doc := Coll.FindOne(5);
doc.Name := 'New!';
Coll.Save(doc);
writeln('Name: ',Coll.FindOne(5).Name);
</pre>
<p>Which will write:</p>
<pre>
Name: New!
</pre>
<p>Note that we used here an integer value (5) as key, but we may use an
<em>ObjectID</em> instead, if needed.</p>
<p>The <code>Coll.Save()</code> method could be changed into
<code>Coll.Update()</code>, which expects an explicit <code>Query</code>
operator, in addition to the updated document content:</p>
<pre>
doc := Coll.FindOne(5);
doc.Name := 'New!';
Coll.Update(BSONVariant(['_id',5]),doc);
writeln('Name: ',Coll.FindOne(5).Name);
</pre>
<p>Note that by <em>MongoDB</em>'s design, any call to <code>Update()</code>
will <em>replace</em> the whole document.</p>
<p>For instance, if you write:</p>
<pre>
writeln('Before: ',Coll.FindOne(3));
Coll.Update('{_id:?}',[3],'{Name:?}',['New Name!']);
writeln('After: ',Coll.FindOne(3));
</pre>
<p>Then the <code>Number</code> field will disappear!</p>
<pre>
Before: {"_id":3,"Name":"Name 4","Number":3}
After: {"_id":3,"Name":"New Name!"}
</pre>
<p>If you need to update only some fields, you will have to use the
<code>$set</code> modifier:</p>
<pre>
writeln('Before: ',Coll.FindOne(4));
Coll.Update('{_id:?}',[4],'{$set:{Name:?}}',['New Name!']);
writeln('After: ',Coll.FindOne(4));
</pre>
<p>Which will write on the console the value as expected:</p>
<pre>
Before: {"_id":4,"Name":"Name 5","Number":4}
After: {"_id":4,"Name":"New Name!","Number":4}
</pre>
<p>Now the <code>Number</code> field remains untouched.</p>
<p>You can also use the <code>Coll.UpdateOne()</code> method, which will update
the supplied fields, and leave the non specified fields untouched:</p>
<pre>
writeln('Before: ',Coll.FindOne(2));
Coll.UpdateOne(2,_Obj(['Name','NEW']));
writeln('After: ',Coll.FindOne(2));
</pre>
<p>Which will output as expected:</p>
<pre>
Before: {"_id":2,"Name":"Name 3","Number":2}
After: {"_id":2,"Name":"NEW","Number":2}
</pre>
<p>You can refer to the documentation of the <code>SynMongoDB.pas</code> unit,
to find out all functions, classes and methods available to work with
<em>MongoDB</em>.</p>
<h3>Write Concern and Performance</h3>
<p>You can take a look at the <code>MongoDBTests.dpr</code> sample - located in
the <code>SQLite3- MongoDB</code> sub-folder of the source code repository, and
the <code>TTestDirect</code> classes, to find out some performance
information.</p>
<p>In fact, this <code>TTestDirect</code> is inherited twice, to run the same
tests with diverse write concern.</p>
<p>The difference between the two classes will take place at client
initialization:</p>
<pre>
<strong>procedure</strong> TTestDirect.ConnectToLocalServer;
...
fClient := TMongoClient.Create('localhost',27017);
<strong>if</strong> ClassType=TTestDirectWithAcknowledge <strong>then</strong>
<span style="background-color:yellow;">fClient.WriteConcern := wcAcknowledged <strong>else</strong></span>
<strong>if</strong> ClassType=TTestDirectWithoutAcknowledge <strong>then</strong>
<span style="background-color:yellow;">fClient.WriteConcern := wcUnacknowledged;</span>
...
</pre>
<p><code>wcAcknowledged</code> is the default safe mode: the <em>MongoDB</em>
server confirms the receipt of the write operation. Acknowledged write concern
allows clients to catch network, duplicate key, and other errors. But it adds
an additional round-trip from the client to the server, and wait for the
command to be finished before returning the error status: so it will slow down
the write process.</p>
<p>With <code>wcUnacknowledged</code>, <em>MongoDB</em> does not acknowledge
the receipt of write operation. Unacknowledged is similar to errors ignored;
however, drivers attempt to receive and handle network errors when possible.
The driver's ability to detect network errors depends on the system's
networking configuration.</p>
<p>The speed difference between the two is worth mentioning, as stated by the
regression tests status, running on a local <em>MongoDB</em> instance:</p>
<pre>
1. Direct access
<br />1.1. Direct with acknowledge:
- Connect to local server: 6 assertions passed 4.72ms
- Drop and prepare collection: 8 assertions passed 9.38ms
<span style="background-color:yellow;"> - Fill collection: 15,003 assertions passed 558.79ms</span>
<span style="background-color:yellow;"> 5000 rows inserted in 548.83ms i.e. 9110/s, aver. 109us, 3.1 MB/s</span>
- Drop collection: no assertion 856us
<span style="background-color:yellow;"> - Fill collection bulk: 2 assertions passed 74.59ms</span>
<span style="background-color:yellow;"> 5000 rows inserted in 64.76ms i.e. 77204/s, aver. 12us, 7.2 MB/s</span>
- Read collection: 30,003 assertions passed 2.75s
5000 rows read at once in 9.66ms i.e. 517330/s, aver. 1us, 39.8 MB/s
<span style="background-color:yellow;"> - Update collection: 7,503 assertions passed 784.26ms</span>
<span style="background-color:yellow;"> 5000 rows updated in 435.30ms i.e. 11486/s, aver. 87us, 3.7 MB/s</span>
<span style="background-color:yellow;"> - Delete some items: 4,002 assertions passed 370.57ms</span>
<span style="background-color:yellow;"> 1000 rows deleted in 96.76ms i.e. 10334/s, aver. 96us, 2.2 MB/s</span>
Total failed: 0 / 56,527 - Direct with acknowledge PASSED 4.56s
<br />1.2. Direct without acknowledge:
- Connect to local server: 6 assertions passed 1.30ms
- Drop and prepare collection: 8 assertions passed 8.59ms
<span style="background-color:yellow;"> - Fill collection: 15,003 assertions passed 192.59ms</span>
<span style="background-color:yellow;"> 5000 rows inserted in 168.50ms i.e. 29673/s, aver. 33us, 4.4 MB/s</span>
- Drop collection: no assertion 845us
<span style="background-color:yellow;"> - Fill collection bulk: 2 assertions passed 68.54ms</span>
<span style="background-color:yellow;"> 5000 rows inserted in 58.67ms i.e. 85215/s, aver. 11us, 7.9 MB/s</span>
- Read collection: 30,003 assertions passed 2.75s
5000 rows read at once in 9.99ms i.e. 500150/s, aver. 1us, 38.5 MB/s
<span style="background-color:yellow;"> - Update collection: 7,503 assertions passed 446.48ms</span>
<span style="background-color:yellow;"> 5000 rows updated in 96.27ms i.e. 51933/s, aver. 19us, 7.7 MB/s</span>
<span style="background-color:yellow;"> - Delete some items: 4,002 assertions passed 297.26ms</span>
<span style="background-color:yellow;"> 1000 rows deleted in 19.16ms i.e. 52186/s, aver. 19us, 2.8 MB/s</span>
Total failed: 0 / 56,527 - Direct without acknowledge PASSED 3.77s
</pre>
<p>As you can see, the reading speed is not affected by the <em>Write
Concern</em> settings.<br />
But data writing can be multiple times faster, when each write command is not
acknowledged.</p>
<p>Since there is no error handling, <code>wcUnacknowledged</code> is not to be
used on production. You may use it for replication, or for data consolidation,
e.g. feeding a database with a lot of existing data as fast as possible.</p>
<p>Stay tuned for the next article, which <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">will detail the <em>MongoDB</em>
integration 's ORM</a>... and the last one of the series, with <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-benchmark">benchmarks of <em>MongoDB</em>
against SQL engines, via our ORM</a>...<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1742">welcome
on our forum</a>, as usual!</p>Introducing mORMot's architecture and design principlesurn:md5:265b0a5927ad96f526d355c5c9aca5322014-04-18T11:40:00+02:002015-01-04T10:10:34+01:00AB4327-GANDImORMot Frameworkcode-firstDatabasedatabase-firstDelphiDependencyInjectionDocumentationDomainDrivenDTOdynamic arrayEventSourcingfactoryFireDACGoodPracticehttp.sysHttpApiHTTPSinterfaceJSONLateBindingmockModelmORMotMSSQLMySQLNextGenNexusDBNoSQLODBCOleDBOpenSourceOracleORMperformancePostgreSQLRepositoryRTTIsessionshardingSOASQLSQLite3SynDBSynopseTDataSetTDocVariantTDynArraytransactionUnicodeUniDACValueObjectVirtualTableWinHTTPWinINetZEOS <p>We have just released a set of slides introducing </p>
<ul>
<li>ORM, SOA, REST, JSON, MVC, MVVM, SOLID, Mocks/Stubs, Domain-Driven Design
concepts with Delphi, </li>
<li>and showing some sample code using our Open Source <em>mORMot</em>
framework.</li>
</ul>
<p>You can follow the <a href="https://drive.google.com/folderview?id=0B0r8u-FwvxWdeVJVZnBhSEpKYkE&usp=sharing">
public link on Google Drive</a>!</p>
<p><img src="http://images.fineartamerica.com/images-medium-large/1-golden-marmot-maureen-ida-farley.jpg" width="450" height="291" alt="" /></p>
<p>This is a great opportunity to discovers some patterns you may not be
familiar with, and find out how <em>mORMot</em> try to implement them.<br />
This set of slides may be less intimidating than our huge documentation - do
not be terrified by our <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html">
Online Documentation</a>!<br />
The <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#SIDE_TITL_40">
first set of pages</a> (presenting architecture and design principles) is worth
reading.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1707">welcome on our forum, as
usual</a>.</p>TDocVariant custom variant typeurn:md5:7d8dec0eaab0b811075d6a84ed6fb60b2014-02-25T18:41:00+01:002014-02-25T18:41:00+01:00AB4327-GANDImORMot FrameworkAJAXblogDelphiDocumentationDomainDrivenDTOdynamic arrayIDEJSONLateBindingmORMotobjectORMParsingperformanceRADrecordshardingSOAstringTDocVariantTDynArray<p>With revision 1.18 of the framework, we just introduced two new custom types
of <code>variant</code>s:</p>
<ul>
<li><code>TDocVariant</code> kind of <code>variant</code>;</li>
<li><code>TBSONVariant</code> kind of <code>variant</code>.</li>
</ul>
<p>The second custom type (which handles <em>MongoDB</em>-specific extensions -
like <code>ObjectID</code> or other specific types like dates or binary) will
be presented later, when dealing with <em>MongoDB</em> support in
<em>mORMot</em>, together with the BSON kind of content. BSON /
<em>MongoDB</em> support is implemented in the <code>SynMongoDB.pas</code>
unit.</p>
<p>We will now focus on <code>TDocVariant</code> itself, which is a generic
container of JSON-like objects or arrays.<br />
This custom variant type is implemented in <code>SynCommons.pas</code> unit, so
is ready to be used everywhere in your code, even without any link to the
<em>mORMot</em> ORM kernel, or <em>MongoDB</em>.</p>
<p><img src="http://www.kumc.edu/Images/information%20resources/document-management-software.jpg" alt="" /></p>
<h4>TDocVariant documents</h4>
<p><code>TDocVariant</code> implements a custom variant type which can be used
to store any JSON/BSON document-based content, i.e. either:</p>
<ul>
<li>Name/value pairs, for object-oriented documents;</li>
<li>An array of values (including nested documents), for array-oriented
documents;</li>
<li>Any combination of the two, by nesting <code>TDocVariant</code>
instances.</li>
</ul>
<p>Here are the main features of this custom variant type:</p>
<ul>
<li>DOM approach of any <em>object</em> or <em>array</em> documents;</li>
<li>Perfect storage for dynamic value-objects content, with a
<em>schema-less</em> approach (as you may be used to in scripting languages
like Python or JavaScript);</li>
<li>Allow nested documents, with no depth limitation but the available
memory;</li>
<li>Assignment can be either <em>per-value</em> (default, safest but slower
when containing a lot of nested data), or <em>per-reference</em> (immediate
reference-counted assignment);</li>
<li>Very fast JSON serialization / un-serialization with support of
<em>MongoDB</em>-like extended syntax;</li>
<li>Access to properties in code, via late-binding (including almost no speed
penalty due to our VCL hack as <a href="https://blog.synopse.info?post/post/2011/07/01/Faster-variant-late-binding">already detailed</a>);</li>
<li>Direct access to the internal variant <em>names</em> and <em>values</em>
arrays from code, by trans-typing into a <code>TDocVariantData
record</code>;</li>
<li>Instance life-time is managed by the compiler (like any other
<code>variant</code> type), without the need to use <code>interfaces</code> or
explicit <code>try..finally</code> blocks;</li>
<li>Optimized to use as little memory and CPU resource as possible (in contrast
to most other libraries, it does not allocate one <code>class</code> instance
per node, but rely on pre-allocated arrays);</li>
<li>Opened to extension of any content storage - for instance, it will
perfectly integrate with BSON serialization and custom <em>MongoDB</em> types
(<em>ObjectID, RegEx</em>...), to be used in conjunction with <em>MongoDB</em>
servers;</li>
<li>Perfectly integrated with our <a href="https://blog.synopse.info?post/post/2011/03/12/TDynArray-and-Record-compare/load/save-using-fast-RTTI">Dynamic
array wrapper</a> and its JSON serialization as with the <a href="https://blog.synopse.info?post/post/2013/12/10/JSON-record-serialization"><code>record</code>
serialization</a>;</li>
<li>Designed to work with our <em>mORMot</em> ORM: any <code>TSQLRecord</code>
instance containing such <code>variant</code> custom types as published
properties will be recognized by the ORM core, and work as expected with any
database back-end (storing the content as JSON in a TEXT column);</li>
<li>Designed to work with our <em>mORMot</em> SOA: any <a href="https://blog.synopse.info?post/post/2012/03/07/Interface-based-services"><code>interface</code>-based
service</a> is able to consume or publish such kind of content, as
<code>variant</code> kind of parameters;</li>
<li>Fully integrated with the Delphi IDE: any <code>variant</code> instance
will be displayed as JSON in the IDE debugger, making it very convenient to
work with.</li>
</ul>
<p>To create instances of such <code>variant</code>, you can use some
easy-to-remember functions:</p>
<ul>
<li><code>_Obj() _ObjFast()</code> global functions to create a
<code>variant</code> <em>object</em> document;</li>
<li><code>_Arr() _ArrFast()</code> global functions to create a
<code>variant</code> <em>array</em> document;</li>
<li><code>_Json() _JsonFast() _JsonFmt() _JsonFastFmt()</code> global functions
to create any <code>variant</code> <em>object</em> or <em>array</em> document
from JSON, supplied either with standard or <em>MongoDB</em>-extended
syntax.</li>
</ul> <h3>Variant object documents</h3>
<p>With <code>_Obj()</code>, an <em>object</em> <code>variant</code> instance
will be initialized with data supplied two by two, as <em>Name,Value</em>
pairs, e.g.</p>
<pre>
<strong>var</strong> V1,V2: <strong>variant</strong>; <em>// stored as any variant</em>
...
V1 := _Obj(['name','John','year',1972]);
V2 := _Obj(['name','John','doc',_Obj(['one',1,'two',2.5])]); <em>// with nested objects</em>
</pre>
<p>Then you can convert those objects into JSON, by two means:</p>
<ul>
<li>Using the <code>VariantSaveJson()</code> function, which return directly
one UTF-8 content;</li>
<li>Or by trans-typing the <code>variant</code> instance into a string (this
will be slower, but is possible).</li>
</ul>
<pre>
writeln(VariantSaveJson(V1)); <em>// explicit conversion into RawUTF8</em>
writeln(V1); <em>// implicit conversion from variant into string</em>
<em>// both commands will write '{"name":"john","year":1982}'</em>
writeln(VariantSaveJson(V2)); <em>// explicit conversion into RawUTF8</em>
writeln(V2); <em>// implicit conversion from variant into string</em>
<em>// both commands will write '{"name":"john","doc":{"one":1,"two":2.5}}'</em>
</pre>
<p>As a consequence, the Delphi IDE debugger is able to display such variant
values as their JSON representation.<br />
That is, <code>V1</code> will be displayed as
<code>'"name":"john","year":1982'</code> in the IDE debugger <em>Watch
List</em> window, or in the <em>Evaluate/Modify</em> (F7) expression
tool.<br />
This is pretty convenient, and much more user friendly than any class-based
solution (which requires the installation of a specific design-time package in
the IDE).</p>
<p>You can access to the object properties via late-binding, with any depth of
nesting objects, in your code:</p>
<pre>
writeln('name=',V1.name,' year=',V1.year);
<em>// will write 'name=John year=1972'</em>
writeln('name=',V2.name,' doc.one=',V2.doc.one,' doc.two=',doc.two);
<em>// will write 'name=John doc.one=1 doc.two=2.5</em>
V1.name := 'Mark'; <em>// overwrite a property value</em>
writeln(V1.name); <em>// will write 'Mark'</em>
V1.age := 12; <em>// add a property to the object</em>
writeln(V1.age); <em>// will write '12'</em>
</pre>
<p>Note that the property names will be evaluated at runtime only, not at
compile time.<br />
For instance, if you write <code>V1.nome</code> instead of
<code>V1.name</code>, there will be no error at compilation, but an
<code>EDocVariant</code> exception will be raised at execution (unless you set
the <code>dvoReturnNullForUnknownProperty</code> option to
<code>_Obj/_Arr/_Json/_JsonFmt</code> which will return a <code>null</code>
variant for such undefined properties).</p>
<p>In addition to the property names, some pseudo-methods are available for
such <em>object</em> <code>variant</code> instances:</p>
<pre>
writeln(V1._Count); <em>// will write 3 i.e. the number of name/value pairs in the object document</em>
writeln(V1._Kind); <em>// will write 1 i.e. ord(sdkObject)</em>
<strong>for</strong> i := 0 <strong>to</strong> V2._Count-1 <strong>do</strong>
writeln(V2.Name(i),'=',V2.Value(i));
<em>// will write in the console:</em>
<em>// name=John</em>
<em>// doc={"one":1,"two":2.5}</em>
<em>// age=12</em>
<strong>if</strong> V1.Exists('year') <strong>then</strong>
writeln(V1.year);
</pre>
<p>You may also trans-type your <code>variant</code> instance into a
<code>TDocVariantData record</code>, and access directly to its
internals.<br />
For instance:</p>
<pre>
TDocVariantData(V1).AddValue('comment','Nice guy');
<strong>with</strong> TDocVariantData(V1) <strong>do</strong> <em>// direct transtyping</em>
<strong>if</strong> Kind=sdkObject <strong>then</strong> <em>// direct access to the TDocVariantDataKind field</em>
<strong>for</strong> i := 0 <strong>to</strong> Count-1 <strong>do</strong> <em>// direct access to the Count: integer field</em>
writeln(Names[i],'=',Values[i]); <em>// direct access to the internal storage arrays</em>
</pre>
<p>By definition, trans-typing via a <code>TDocVariantData record</code> is
slightly faster than using late-binding.<br />
But you must ensure that the <code>variant</code> instance is really a
<code>TDocVariant</code> kind of data before transtyping e.g. by calling
<code>DocVariantType.IsOfType(aVariant)</code>.</p>
<h3>Variant array documents</h3>
<p>With <code>_Arr()</code>, an <em>array</em> <code>variant</code> instance
will be initialized with data supplied as a list of <em>Value1,Value2,...</em>,
e.g.</p>
<pre>
<strong>var</strong> V1,V2: <strong>variant</strong>; <em>// stored as any variant</em>
...
V1 := _Arr(['John','Mark','Luke']);
V2 := _Obj(['name','John','array',_Arr(['one','two',2.5])]); <em>// as nested array</em>
</pre>
<p>Then you can convert those objects into JSON, by two means:</p>
<ul>
<li>Using the <code>VariantSaveJson()</code> function, which return directly
one UTF-8 content;</li>
<li>Or by trans-typing the <code>variant</code> instance into a string (this
will be slower, but is possible).</li>
</ul>
<pre>
writeln(VariantSaveJson(V1));
writeln(V1); <em>// implicit conversion from variant into string</em>
<em>// both commands will write '["John","Mark","Luke"]'</em>
writeln(VariantSaveJson(V2));
writeln(V2); <em>// implicit conversion from variant into string</em>
<em>// both commands will write '{"name":"john","array":["one","two",2.5]}'</em>
</pre>
<p>As a with any <em>object</em> document, the Delphi IDE debugger is able to
display such <em>array</em> <code>variant</code> values as their JSON
representation.</p>
<p>Late-binding is also available, with a special set of pseudo-methods:</p>
<pre>
writeln(V1._Count); <em>// will write 3 i.e. the number of items in the array document</em>
writeln(V1._Kind); <em>// will write 2 i.e. ord(sdkArray)</em>
<strong>for</strong> i := 0 <strong>to</strong> V1._Count-1 <strong>do</strong>
writeln(V1.Value(i),':',V2._(i));
<em>// will write in the console:</em>
<em>// John John</em>
<em>// Mark Mark</em>
<em>// Luke Luke</em>
<strong>if</strong> V1.Exists('John') <strong>then</strong>
writeln('John found in array');
</pre>
<p>Of course, trans-typing into a <code>TDocVariantData record</code> is
possible, and will be slightly faster than using late-binding.</p>
<h3>Create variant object or array documents from JSON</h3>
<p>With <code>_Json()</code> or <code>_JsonFmt()</code>, either a
<em>document</em> or <em>array</em> <code>variant</code> instance will be
initialized with data supplied as JSON, e.g.</p>
<pre>
<strong>var</strong> V1,V2,V3,V4: <strong>variant</strong>; <em>// stored as any variant</em>
...
V1 := _Json('{"name":"john","year":1982}'); <em>// strict JSON syntax</em>
V2 := _Json('{name:"john",year:1982}'); <em>// with MongoDB extended syntax for names</em>
V3 := _Json('{"name":?,"year":?}',[],['john',1982]);
V4 := _JsonFmt('{%:?,%:?}',['name','year'],['john',1982]);
writeln(VariantSaveJSON(V1));
writeln(VariantSaveJSON(V2));
writeln(VariantSaveJSON(V3));
<em>// all commands will write '{"name":"john","year":1982}'</em>
</pre>
<p>Of course, you can nest objects or arrays as parameters to the
<code>_JsonFmt()</code> function.</p>
<p>The supplied JSON can be either in strict JSON syntax, or with the
<em>MongoDB</em> extended syntax, i.e. with unquoted property names.<br />
It could be pretty convenient and also less error-prone when typing in the
Delphi code to forget about quotes around the property names of your JSON.</p>
<p>Note that <em>TDocVariant</em> implements an open interface for adding any
custom extensions to JSON: for instance, if the <code>SynMongoDB.pas</code>
unit is defined in your application, you will be able to create any MongoDB
specific types in your JSON, like <code>ObjectID()</code>, <code>new
Date()</code> or even <code>/regex/option</code>.</p>
<p>As a with any <em>object</em> or <em>array</em> document, the Delphi IDE
debugger is able to display such <code>variant</code> values as their JSON
representation.</p>
<h3>Per-value or per-reference</h3>
<p>By default, the <code>variant</code> instance created by <code>_Obj() _Arr()
_Json() _JsonFmt()</code> will use a <em>copy-by-value</em> pattern.<br />
It means that when an instance is affected to another variable, a new
<code>variant</code> document will be created, and all internal values will be
copied. Just like a <code>record</code> type.</p>
<p>This will imply that if you modify any item of the copied variable, it won't
change the original variable:</p>
<pre>
<strong>var</strong> V1,V2: <strong>variant</strong>;
...
V1 := _Obj(['name','John','year',1972]);
V2 := V1; <em>// create a new variant, and copy all values</em>
V2.name := 'James'; <em>// modifies V2.name, but not V1.name</em>
writeln(V1.name,' and ',V2.name);
<em>// will write 'John and James'</em>
</pre>
<p>As a result, your code will be perfectly safe to work with, since
<code>V1</code> and <code>V2</code> will be uncoupled.</p>
<p>But one drawback is that passing such a value may be pretty slow, for
instance, when you nest objects:</p>
<pre>
<strong>var</strong> V1,V2: <strong>variant</strong>;
...
V1 := _Obj(['name','John','year',1972]);
V2 := _Arr(['John','Mark','Luke']);
V1.names := V2; <em>// here the whole V2 array will be re-allocated into V1.names</em>
</pre>
<p>Such a behavior could be pretty time and resource consuming, in case of a
huge document.</p>
<p>All <code>_Obj() _Arr() _Json() _JsonFmt()</code> functions have an optional
<code>TDocVariantOptions</code> parameter, which allows to change the behavior
of the created <code>TDocVariant</code> instance, especially setting
<code>dvoValueCopiedByReference</code>.</p>
<p>This particular option will set the <em>copy-by-reference</em> pattern:</p>
<pre>
<strong>var</strong> V1,V2: <strong>variant</strong>;
...
V1 := _Obj(['name','John','year',1972],[dvoValueCopiedByReference]);
V2 := V1; <em>// creates a reference to the V1 instance</em>
V2.name := 'James'; <em>// modifies V2.name, but also V1.name</em>
writeln(V1.name,' and ',V2.name);
<em>// will write 'James and James'</em>
</pre>
<p>You may think this behavior is somewhat weird for a <code>variant</code>
type. But if you forget about <em>per-value</em> objects and consider those
<code>TDocVariant</code> types as a Delphi <code>class</code> instance (which
is a <em>per-reference</em> type), without the need of having a fixed schema
nor handling manually the memory, it will probably start to make sense.</p>
<p>Note that a set of global functions have been defined, which allows direct
creation of documents with <em>per-reference</em> instance lifetime, named
<code>_ObjFast() _ArrFast() _JsonFast() _JsonFmtFast()</code>.<br />
Those are just wrappers around the corresponding <code>_Obj() _Arr() _Json()
_JsonFmt()</code> functions, with the following <code>JSON_OPTIONS[true]</code>
constant passed as options parameter:</p>
<pre>
<strong>const</strong>
<em>/// some convenient TDocVariant options</em>
<em>// - JSON_OPTIONS[false] is _Json() and _JsonFmt() functions default</em>
<em>// - JSON_OPTIONS[true] are used by _JsonFast() and _JsonFastFmt() functions</em>
JSON_OPTIONS: <strong>array</strong>[Boolean] <strong>of</strong> TDocVariantOptions = (
[dvoReturnNullForUnknownProperty],
<span style="background-color:yellow;">[dvoReturnNullForUnknownProperty,dvoValueCopiedByReference]);</span>
</pre>
<p>When working with complex documents, e.g. with BSON / <em>MongoDB</em>
documents, almost all content will be created in "fast" <em>per-reference</em>
mode.</p>
<h3>Advanced TDocVariant process</h3>
<h3>Object or array document creation options</h3>
<p>As stated above, a <code>TDocVariantOptions</code> parameter enables to
define the behavior of a <code>TDocVariant</code> custom type for a given
instance.<br />
Please refer to the documentation of this set of options to find out the
available settings. Some are related to the memory model, other to
case-sensitivity of the property names, other to the behavior expected in case
of non-existing property, and so on...</p>
<p>Note that this setting is <em>local</em> to the given <code>variant</code>
instance.</p>
<p>In fact, <code>TDocVariant</code> does not force you to stick to one memory
model nor a set of global options, but you can use the best pattern depending
on your exact process.<br />
You can even <em>mix</em> the options - i.e. including some objects as
properties in an object created with other options - but in this case, the
initial options of the nested object will remain. So you should better use this
feature with caution.</p>
<p>You can use the <code>_Unique()</code> global function to force a variant
instance to have an unique set of options, and all nested documents to become
<em>by-value</em>, or <code>_UniqueFast()</code> for all nested documents to
become <em>by-reference</em>.</p>
<pre>
<em>// assuming V1='{"name":"James","year":1972}' created by-reference</em>
_Unique(V1); <em>// change options of V1 to be by-value</em>
V2 := V1; <em>// creates a full copy of the V1 instance</em>
V2.name := 'John'; <em>// modifies V2.name, but not V1.name</em>
writeln(V1.name); <em>// write 'James'</em>
writeln(V2.name); <em>// write 'John'</em>
V1 := _Arr(['root',V2]); <em>// created as by-value by default, as V2 was</em>
writeln(V1._Count); <em>// write 2</em>
_UniqueFast(V1); <em>// change options of V1 to be by-reference</em>
V2 := V1;
V1._(1).name := 'Jim';
writeln(V1);
writeln(V2);
<em>// both commands will write '["root",{"name":"Jim","year":1972}]'</em>
</pre>
<p>The easiest is to stick to one set of options in your code, i.e.:</p>
<ul>
<li>Either using the <code>_*()</code> global functions if your business code
does send some <code>TDocVariant</code> instances to any other part of your
logic, for further storage: in this case, the <em>by-value</em> pattern does
make sense;</li>
<li>Or using the <code>_*Fast()</code> global functions if the
<code>TDocVariant</code> instances are local to a small part of your code, e.g.
used as schema-less <em>Data Transfer Objects</em> (<em>DTO</em>).</li>
</ul>
<p>In all cases, be aware that, like any <code>class</code> type, the
<code>const</code>, <code>var</code> and <code>out</code> specifiers of method
parameters does not behave to the <code>TDocVariant</code> value, but to its
reference.</p>
<h3>Integration with other mORMot units</h3>
<p>In fact, whenever a <em>schema-less</em> storage structure is needed, you
may use a <code>TDocVariant</code> instance instead of <code>class</code> or
<code>record</code> strong-typed types:</p>
<ul>
<li>Client-Server ORM will support <code>TDocVariant</code> in any of the
<code>TSQLRecord variant</code> published properties;</li>
<li>Interface-based services will support <code>TDocVariant</code> as
<code>variant</code> parameters of any method, which make them as perfect
<em>DTO</em>;</li>
<li>Since JSON support is implemented with any <code>TDocVariant</code> value
from the ground up, it makes a perfect fit for working with AJAX clients, in a
script-like approach;</li>
<li>If you use our <code>SynMongoDB.pas</code> unit to access a
<em>MongoDB</em> server, <code>TDocVariant</code> will be the native storage to
create or access BSON arrays or objects documents;</li>
<li>Cross-cutting features (like logging or <code>record</code> / <em>dynamic
array</em> enhancements) will also benefit from this <code>TDocVariant</code>
custom type.</li>
</ul>
<p>We are pretty convinced that when you will start playing with
<code>TDocVariant</code>, you won't be able to live without it any more.<br />
It introduces the full power of late-binding and schema-less patterns to your
application, which can be pretty useful for prototyping or in Agile
development.<br />
You do not need to use scripting engines like Python or JavaScript to have this
feature, if you need it.</p>
<p>Feedback and comments are <a href="http://synopse.info/forum/viewtopic.php?id=1631">welcome in our forum, as
usual</a>!</p>Domain-Driven-Design and mORMoturn:md5:9b58d020f0f50017b1b1fefbe4b4d3722013-01-05T20:18:00+01:002013-01-05T21:06:05+01:00AB4327-GANDImORMot FrameworkAggregateRootblogDatabaseDelphiDocumentationDomainDrivenDTOEntityObjectEventCollaborationEventSourcingfactoryGoodPracticeinterfacemockmORMotORMRepositoryRTTIsecurityshardingSOASourceValueObject<p>Implementing Domain-Driven-Design (DDD) is one goal of our <em>mORMot</em>
framework.</p>
<p><img src="http://textcreate.files.wordpress.com/2011/09/marmot.jpg" alt="" width="326" height="177/" /></p>
<p>We already <a href="https://blog.synopse.info?post/post/2012/05/25/Domain-Driven-design">presented this
particular n-Tier architecture</a>.</p>
<p>It is now time to enter deeper into the material, provide some definition
and reference.<br />
You can also search the web for reference, or look at <a href="http://domaindrivendesign.org/">the official web site</a>.<br />
<a href="http://msdn.microsoft.com/en-us/magazine/dd419654.aspx">A general
presentation</a> of the corresponding concepts, in the .NET world, was used as
reference of this blog entry.</p>
<p>Stay tuned, and ride the <em>mORMot</em>!</p> <p>For the definition of your objects or internal data structures (what good
programmers care about), you are therefore encouraged to make a difference
between several kind of objects.</p>
<p>Domain-level representation are, generally speaking, rich on behavior,
therefore also of several families/species of objects.<br />
Let us list the most high-level definitions of DDD:</p>
<ul>
<li><em>Value Objects</em> contain attributes (value, size) but no conceptual
identity - e.g. money bills, or seats in a Rock concert, as they are
interchangeable;</li>
<li><em>Entity objects</em> are defined by their attributes, but have a thread
of continuity, signified by an identity - e.g. persons, or seats in most
planes, as each one is unique and identified;</li>
<li><em>Aggregates</em> are collection of objects (Values and/or Entities) that
are bound together by a root entity, otherwise known as an aggregate root.
Typically, aggregates are persisted in the database, and guarantee the
consistency of changes by isolating its members from external objects (i.e. you
can link to an aggregate via its ID, but you can not directly access to its
internal objects). See <a href="https://blog.synopse.info?post/post/2011/07/03/Sharde-nothing-architecture">share-nothing architecture
article</a> which, in <em>mORMot</em>, sounds very much like an <a href="http://martinfowler.com/bliki/AggregateOrientedDatabase.html">Aggregate-Oriented-Database</a>.</li>
</ul>
<p>In addition to these domain-level objects, some cross-cutting types may
appear, especially at the application layer:</p>
<ul>
<li><em>Data Transfer Objects</em> (DTO) are transmission objects, which
purpose is to not send your domain across the wire (i.e. separate your layers,
following the <em>Anti-Corruption Layer</em> pattern). It encourages you
to create gatekeepers that work to prevent non-domain concepts from leaking
into your model. They keep both domain and application layers clean.</li>
<li><em>Commands</em> and <em>Events</em> are some kind of DTO, since they
communicate data about an event and they themselves encapsulate no behavior -
in <em>mORMot</em>, we try to let the framework do all the plumbing, letting
those types be implemented via interfaces, avoiding the need to define them by
hand.</li>
</ul>
<p>DDD then favors some patterns to use those objects efficiently:</p>
<ul>
<li><em>Ubiquitous Language </em>is where everything begins.<br />
In the real outside world, i.e. for <a href="http://www.gnu.org/fun/jokes/10-kinds-of-people.html">the other 10th kind of
people</a> how do not know about binary, domain experts use company- or
industry- standard terminology. As developers, we have to understand this
vocabulary and not only use it when speaking with domain experts but also see
the same terminology reflected in our code. If the terms "class code" or "rate
sets" or "exposure" are frequently used in conversation, we shall find
corresponding class names in the code. In DDD, it is critical that developers
use the business language in code consciously and as a disciplined rule.</li>
<li><em>Factory</em> pattern is used to create object instances.<br />
Main benefit is that alternative implementations may be easily interchanged.
Such abstraction helps testing - see <a href="https://blog.synopse.info?post/post/2012/10/14/Stubs-and-Mocks-for-Delphi-with-mORMot">mocking</a> - but
also introduces <a href="https://blog.synopse.info?post/post/2012/03/07/Interface-based-services">interface-based services</a>.</li>
<li>Value objects are by definition immutable, so should be handled as
read-only.<br />
They are incapable of change once they are created. Why is it important that
they be immutable? With value objects, you're seeking side-effect-free
functions, yet another concept borrowed by DDD. When you add $20 to $20, are
you changing $20? No, you are creating a new money descriptor of $40.</li>
<li><em>Repository</em> pattern is used to save and dispense each Aggregate
root.<br />
It matches the <a href="http://martinfowler.com/eaaCatalog/layerSupertype.html">"Layer Supertype"
pattern</a>, e.g. via our <em>mORMot</em> <code>TSQLRecord</code> and
<code>TSQLRest</code> classes and their Client-Server ORM features, or via
dedicated repository classes - saving data is indeed a concern orthogonal to
the model itself. Some architects claim that persistence is infrastructure, not
domain. You may benefit in defining your own repository interface, if the
standard ORM / CRUD operations are not enough.</li>
<li>Aggregate roots and entities, with all their methods, often end up as
<em>state machines</em>, and the behavior matches accordingly.<br />
In the domain, since aggregate roots are the only kind of entity to which your
software may hold a reference, they tend to be the main access point of all
process.</li>
<li>Domain services pattern is used to model primary operations.<br />
Domain Services give you a tool for modeling processes that do not have an
identity or life-cycle in your domain, that is, that are not linked to one
aggregate root, perhaps none, or several. In this terminology, services are not
tied to a particular person, place, or thing in my application, but tend to
embody processes. They tend to be named after verbs or business activities that
domain experts introduce into the so-called <em>Ubiquitous Language</em>. If
you follow the <a href="https://blog.synopse.info?post/post/2011/11/27/SOLID-design-principles">interface
segregation principle</a>, your domain services should be exposed as dedicated
client-oriented methods. Do not leak your domain! In DDD, you develop your
application layer services directly from the needs of your client applications,
letting the Domain layer focus on the business logic and persistence
tasks.</li>
</ul>
<p>In practice, <em>mORMot</em>'s Client-Server architecture may be used as
such:</p>
<ul>
<li><em>Value Objects</em> are probably meant to be defined as
<code>record</code>, with methods (i.e. in this case as <code>object</code> for
older versions of Delphi).<br />
But you would need to <a href="https://blog.synopse.info?post/post/2012/05/03/Custom-JSON-serialization-of-records">customize the JSON
serialization</a>, when targeting AJAX clients (by default,
<code>record</code>s are serialized as binary + Base64 encoding). You may also
use <code>TComponent</code> or <code>TSQLRecord</code> classes, ensuring the
published properties do not have setters but just <code>read F...</code>
definition, to make them read-only, and, at the same time, directly
serializable.</li>
<li><a href="https://blog.synopse.info?post/post/2012/07/12/One-ORM-to-rule-them-all">Client-Server ORM</a>
can be used to persist your aggregate roots directly, or you can use a
repository service.<br />
In the first case, your aggregates will be defined as <code>TSQLRecord</code>;
otherwise, you should define a dedicated persistence service, then use plain
DTO (like Delphi <code>record</code>) or even publish the
<code>TSQLRecord</code> types, and benefit of their automated
serialization.</li>
<li><a href="https://blog.synopse.info?post/post/2010/07/18/DataSnap-like-Client-Server-JSON-RESTful-Services-in-Delphi-7-2010">
Services via methods</a> - can be used to publish methods corresponding to your
aggregate roots defined as <code>TSQLRecord</code>.<br />
This will make it pretty RESTful compatible.</li>
<li><a href="https://blog.synopse.info?post/post/2012/03/07/Interface-based-services-sample-code">Services
via interfaces</a> can be used to publish all your processes.<br />
Dedicated factories can be used on both Client and Server side, to define your
repositories and/or domain operations.</li>
</ul>
<p>Don't be afraid of writing some translation layers between
<code>TSQLRecord</code> and DTO records. It will be very fast, on the server
side. If your service interfaces are cleaner, don't hesitate. But if it tends
to enforce you writing a lot of wrapping code, forget about it. Or automate the
wrapper coding, using RTTI and code generators. You have to weight the PROs and
the CONs, like always...</p>
<p>Even <a href="https://blog.synopse.info?post/post/2012/12/31/Enhance-existing-projects-with-mORMot">legacy
code and existing projects</a> will benefit for it. <br />
<a href="http://www.infoq.com/articles/Utilizing-Logging">Finding so-called
seams</a>, along with isolating your core domain, can be extremely valuable
when using DDD techniques to refactor and tighten the highest value parts of
your code.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=6088#p6088">welcome on our
forum</a>, as usual.</p>Custom JSON serialization of any classurn:md5:d77361842637fe22d8ea16f4173204942012-04-13T12:07:00+02:002013-01-17T08:40:22+01:00AB4327-GANDImORMot FrameworkDelphiDocumentationJSONmORMotobjectshardingSource<p>By default, our <em>mORMot</em> framework is able to serialize any class
into a true JSON object.</p>
<p>All <code>published</code> properties will be written as JSON object
members.</p>
<p>In some cases, it may be handy to have a custom serialization, for instance
if you want to manage some third-party classes, or to adapt the serialization
scheme to a particular purpose, at runtime.</p> <p>You can add a customized serialization of any <code>class</code>, by calling
the <code>TJSONSerializer. RegisterCustomSerializer</code> class method. Two
callbacks are to be defined for a specific class type, and will be used to
serialize or un-serialize the object instance. The callbacks are class methods
(<code>procedure() of object</code>), and not plain functions (for some evolved
objects, it may have sense to use a context during serialization).</p>
<p>In the current implementation of this feature, callbacks expect low-level
implementation. That is, their implementation code shall follow function
<code>JSONToObject()</code> patterns, i.e. calling low-level
<code>GetJSONField()</code> function to decode the JSON content, and follow
function <code>TJSONSerializer.WriteObject()</code> patterns, i.e.
<code>aSerializer.Add/AddInstanceName/AddJSONEscapeString</code> to encode the
class instance as JSON.</p>
<p>Note that the process is called outside the "{<code>{...}</code>" JSON
object layout, allowing any serialization scheme: even a class content can be
serialized as a JSON string, JSON array or JSON number, on request.</p>
<p>For instance, we'd like to customize the serialization of this class
(defined in <code>SynCommons.pas</code>):</p>
<pre>
TFileVersion = <strong>class</strong>
<strong>protected</strong>
fDetailed: <strong>string</strong>;
fBuildDateTime: TDateTime;
<strong>public</strong>
Major: Integer;
Minor: Integer;
Release: Integer;
Build: Integer;
BuildYear: integer;
Main: <strong>string</strong>;
<strong>published</strong>
<strong>property</strong> Detailed: <strong>string read</strong> fDetailed <strong>write</strong> fDetailed;
<strong>property</strong> BuildDateTime: TDateTime <strong>read</strong> fBuildDateTime <strong>write</strong> fBuildDateTime;
<strong>end</strong>;
</pre>
<p>By default, since it has been defined within <code>$M+ ... $M-</code>
conditionals, RTTI is available for the <code>published</code> properties (just
as if it were inheriting from <code>TPersistent</code>). That is, the default
JSON serialization will be for instance:</p>
<pre>
{"Detailed":"1.2.3.4","BuildDateTime":"1911-03-14T00:00:00"}
</pre>
<p>This is what is expected when serialized within a <code>TSynLog</code>
content, or for main use.</p>
<p>We would like to serialize this <code>class</code> as such:</p>
<pre>
{"Major":1,"Minor":2001,"Release":3001,"Build":4001,"Main":"1","BuildDateTime":"1911-03-14"}
</pre>
<p>We will therefore define the <em>Writer</em> callback, as such:</p>
<pre>
<strong>class procedure</strong> TCollTstDynArray.FVClassWriter(<strong>const</strong> aSerializer: TJSONSerializer;
aValue: TObject; aOptions: TTextWriterWriteObjectOptions);
<strong>var</strong> V: TFileVersion <strong>absolute</strong> aValue;
<strong>begin</strong>
aSerializer.AddJSONEscape(['Major',V.Major,'Minor',V.Minor,'Release',V.Release,
'Build',V.Build,'Main',V.Main,'BuildDateTime',DateTimeToIso8601Text(V.BuildDateTime)]);
<strong>end</strong>;
</pre>
<p>Most of the JSON serialization work will be made within the
<code>AddJSONEscape</code> method, expecting the JSON object description as an
array of name/value pairs.</p>
<p>Then the associated <em>Reader</em> callback could be, for instance:</p>
<pre>
<strong>class function</strong> TCollTstDynArray.FVClassReader(<strong>const</strong> aValue: TObject; aFrom: PUTF8Char;
<strong>var</strong> aValid: Boolean): PUTF8Char;
<strong>var</strong> V: TFileVersion <strong>absolute</strong> aValue;
Values: TPUtf8CharDynArray;
<strong>begin</strong>
aValid := false;
aFrom := JSONDecode(aFrom,['Major','Minor','Release','Build','Main','BuildDateTime'],Values);
<strong>if</strong> aFrom=<strong>nil then</strong>
exit;
V.Major := GetInteger(Values[0]);
V.Minor := GetInteger(Values[1]);
V.Release := GetInteger(Values[2]);
V.Build := GetInteger(Values[3]);
V.Main := UTF8DecodeToString(Values[4],StrLen(Values[4]));
V.BuildDateTime := Iso8601ToDateTimePUTF8Char(Values[5]);
aValid := true;
result := aFrom;
<strong>end</strong>;
</pre>
<p>Here, the <code>JSONDecode</code> function will un-serialize the JSON object
into an array of <code>PUTF8Char</code> values, without any memory allocation
(in fact, <code>Values[]</code> will point to un-escaped and #0 terminated
content within the <code>aFrom</code> memory buffer. So decoding is very
fast.</p>
<p>Then, the registration step will be defined as such:</p>
<pre>
TJSONSerializer.RegisterCustomSerializer(TFileVersion,
TCollTstDynArray.FVClassReader,TCollTstDynArray.FVClassWriter);
</pre>
<p>If you want to disable the custom serialization, you may call the same
method as such:</p>
<pre>
TJSONSerializer.RegisterCustomSerializer(TFileVersion,<strong>nil</strong>,<strong>nil</strong>);
</pre>
<p>This will reset the JSON serialization of the specified class to the default
serializer (i.e. writing of <code>published</code> properties).</p>
<p>The above code uses some low-level functions of the framework (i.e.
<code>AddJSONEscape</code> and <code>JSONDecode</code>) to implement
serialization as a JSON object, but you may use any other serialization scheme,
on need. That is, you may serialize the whole class instance just as one JSON
string or numerical value, or even a JSON array. It will depend of the
implementation of the <em>Reader</em> and <em>Writer</em> registered
callbacks.</p>
<p>Feedback and comments <a href="http://synopse.info/forum/viewtopic.php?id=674">are welcome on our
forum</a>.</p>Custom JSON serialization of any dynamic array contenturn:md5:984c939c3b5a3e6164ecd4408bb9448f2012-04-12T20:33:00+02:002012-04-13T11:02:05+02:00AB4327-GANDImORMot FrameworkDelphidynamic arrayJSONmORMotobjectParsingperformancerecordshardingSourceTDynArrayUnicode<p>One nice feature of the <code>TDynArray</code> wrapper and its associated
methods, as defined and used in our framework, is the ability to <a href="https://blog.synopse.info?post/post/2011/03/12/TDynArray-and-Record-compare/load/save-using-fast-RTTI">serialize
any dynamic array as JSON content</a>.</p>
<p>By default, only "standard" dynamic arrays (like
<code>TIntegerDynArray</code>) are serialized as true JSON array: other not
known kind of arrays are serialized as binary content, within a <em>Base64</em>
encoding.</p>
<p>This is a very efficient solution for a pure Delphi application, since it
will be fast and always works, but won't be easy to deal with from an AJAX
client.</p>
<p>Applications can now supply a custom JSON serialization for any other
dynamic array, via the <code>TTextWriter.RegisterCustomJSONSerializer()</code>
class method.<br />
Two callbacks are to be supplied for a dynamic array type information, in order
to handle proper serialization and un-serialization of the JSON array.</p> <p>For instance, we would like to serialize a dynamic array of the following
record:</p>
<pre>
TFV = <strong>packed record</strong>
Major, Minor, Release, Build: integer;
Main, Detailed: <strong>string</strong>;
<strong>end</strong>;
TFVs = <strong>array of</strong> TFV;
</pre>
<p>With the default serialization, such a dynamic array will be serialized as a
<em>Base64</em> encoded binary buffer. This won't be easy to understand from an
AJAX client, for instance.</p>
<p>In order to add a custom serialization for this kind of record, we need to
implement the two needed callbacks. Our expected format will be a JSON array of
all fields, i.e.:</p>
<pre>
[1,2001,3001,4001,"1","1001"]
</pre>
<p>We may have used another layout, e.g. using <code>JSONEncode()</code>
function and a JSON object layout, or any other valid JSON content.</p>
<p>Here comes the writer:</p>
<pre>
<strong>class procedure</strong> TCollTstDynArray.FVWriter(<strong>const</strong> aWriter: TTextWriter; <strong>const</strong> aValue);
<strong>var</strong> V: TFV <strong>absolute</strong> aValue;
<strong>begin</strong>
aWriter.Add('[%,%,%,%,"%","%"]',
[V.Major,V.Minor,V.Release,V.Build,V.Main,V.Detailed],twJSONEscape);
<strong>end</strong>;
</pre>
<p>This event will write one entry of the dynamic array, without the last ','
(which will be appended by <code>TTextWriter.AddDynArrayJSON</code>).<br />
In this method, <code>twJSONEscape</code> is used to escape the supplied
<code>string</code> content as a valid JSON string (with double quotes and
proper UTF-8 encoding).</p>
<p>Of course, the <em>Writer</em> is easier to code than the <em>Reader</em>
itself:</p>
<pre>
<strong>class function</strong> TCollTstDynArray.FVReader(P: PUTF8Char; <strong>var</strong> aValue;
<strong>out</strong> aValid: Boolean): PUTF8Char;
<strong>var</strong> V: TFV <strong>absolute</strong> aValue;
<strong>begin</strong> <em>// '[1,2001,3001,4001,"1","1001"],[2,2002,3002,4002,"2","1002"],...'</em>
aValid := false;
result := <strong>nil</strong>;
<strong>if</strong> (P=<strong>nil</strong>) <strong>or</strong> (P^<>'[') <strong>then</strong>
exit;
inc(P);
V.Major := GetNextItemCardinal(P);
V.Minor := GetNextItemCardinal(P);
V.Release := GetNextItemCardinal(P);
V.Build := GetNextItemCardinal(P);
V.Main := UTF8ToString(GetJSONField(P,P));
V.Detailed := UTF8ToString(GetJSONField(P,P));
<strong>if</strong> P=<strong>nil then</strong>
exit;
aValid := true;
result := P; <em>// ',' or ']' for last item of array</em>
<strong>end</strong>;
</pre>
<p>The reader method shall return a pointer to the next separator of the JSON
input buffer just after this item (either <code>','</code> or
<code>']'</code>).</p>
<p>The registration process itself is as simple as:</p>
<pre>
TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TFVs),
TCollTstDynArray.FVReader,TCollTstDynArray.FVWriter);
</pre>
<p>Then, from the user code point of view, this dynamic array handling won't
change: once registered, the JSON serializers are used everywhere in the
framework, as soon as this type is registered globally.</p>
<p>Here is a <em>Writer</em> method using a JSON object layout:</p>
<pre>
<strong>class procedure</strong> TCollTstDynArray.FVWriter2(<strong>const</strong> aWriter: TTextWriter; <strong>const</strong> aValue);
<strong>var</strong> V: TFV <strong>absolute</strong> aValue;
<strong>begin</strong>
aWriter.AddJSONEscape(['Major',V.Major,'Minor',V.Minor,'Release',V.Release,
'Build',V.Build,'Main',V.Main,'Detailed',V.Detailed]);
<strong>end</strong>;
</pre>
<p>This will create some JSON content as such:</p>
<pre>
{"Major":1,"Minor":2001,"Release":3001,"Build":4001,"Main":"1","Detailed":"1001"}
</pre>
<p>Then the corresponding <em>Reader</em> callback could be written as:</p>
<pre>
<strong>class function</strong> TCollTstDynArray.FVReader2(P: PUTF8Char; <strong>var</strong> aValue;
<strong>out</strong> aValid: Boolean): PUTF8Char;
<strong>var</strong> V: TFV <strong>absolute</strong> aValue;
Values: TPUtf8CharDynArray;
<strong>begin</strong>
aValid := false;
P := JSONDecode(P,['Major','Minor','Release','Build','Main','Detailed'],Values);
<strong>if</strong> P=<strong>nil then</strong>
exit;
V.Major := GetInteger(Values[0]);
V.Minor := GetInteger(Values[1]);
V.Release := GetInteger(Values[2]);
V.Build := GetInteger(Values[3]);
V.Main := UTF8DecodeToString(Values[4],StrLen(Values[4]));
V.Detailed := UTF8DecodeToString(Values[5],StrLen(Values[5]));
aValid := true;
result := P; <em>// ',' or ']' for last item of array</em>
<strong>end</strong>;
</pre>
<p>Most of the JSON decoding process is performed within the
<code>JSONDecode()</code> function, which will let <code>Values[]</code> point
to null-terminated un-escaped content within the <code>P^</code> buffer. In
fact, such process will do only one memory allocation (for
<code>Values[]</code>), and will therefore be very fast.</p>
<p>You can define now your custom JSON serializers, starting for the above code
as reference.</p>
<p>This JSON serialization will indeed be used in our main ORM to support
<em>dynamic arrays</em> as enhanced properties (stored as BLOB), and in the
<code>interface</code>-based SOA architecture of the framework, for content
transmission.</p>
<p>Feedback and comments are <a href="http://synopse.info/forum/viewtopic.php?pid=4082#p4082">welcome in our
forum</a>, just as usual.</p>"Sharding" or "Share nothing" architectureurn:md5:bf19cfd9ebc4bf748c53eac86d1d5cba2011-07-03T20:12:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkblogDatabaseDelphidynamic arrayhasmanyORMperformanceRestsharding<p>Here is what wikipedia states at <a href="http://en.wikipedia.org/wiki/Shared_nothing_architecture">http://en.wikipedia.org/wiki/Shared_nothing_architecture</a>:</p>
<blockquote>
<p><em>A shared nothing architecture (SN) is a distributed computing
architecture in which each node is independent and self-sufficient, and there
is no single point of contention across the system. People typically contrast
SN with systems that keep a large amount of centrally-stored state information,
whether in a database, an application server, or any other similar single point
of contention.</em></p>
</blockquote>
<p>This is just one approach of "sharding". Sharding is indeed related to a
shared nothing architecture - once sharded, each shard can live in a totally
separate logical schema instance.<br />
"I sharded, therefore it scales"...</p>
<p>You can do this in Delphi... and opens a new world of scaling
opportunities... Just as <em>Google</em>, <em>Facebook</em>, or
<em>eBay</em> do... </p> <p>As you already know, in our ORM, high-level types like <a href="https://blog.synopse.info?post/post/2011/03/12/TDynArray-and-Record-compare/load/save-using-fast-RTTI">dynamic
arrays</a> or <code>TPersistent</code> / <code>TCollection</code> properties
are perfectly handled, and stored as BLOB or TEXT inside the main data
row.<br />
There is no external linked table, no <em>Master/Detail</em> to maintain. In
fact, each <code>TSQLRecord</code> instance content could be made
self-contained in our ORM.</p>
<p>Of course, you have the choice to implement the data in the class
master/detail scheme. You have direct OOP implementation of <a href="https://blog.synopse.info?post/post/2010/10/28/%22has-many%22-and-%22has-many-through%22-relationships">one-to-one,
one-to-many, or many-to-many relationship</a>.</p>
<p>But when the server starts to have an increasing number of clients, such a
"shared nothing" data layout could be a major benefit. In fact, the so-called
<em>sharding</em>, or horizontal partitioning of data, is a proven solution for
web-scale databases, such as those in use by social networking sites. How does
<em>eBay</em> or <em>Facebook</em> scale with so many users? Just by
<em>sharding</em>.</p>
<p>A simple but very efficient <em>sharding</em> mechanism could therefore be
implemented with our ORM. In-memory databases, or our <em>BigTable</em>
component are good candidate for light speed data process. Even <em>SQLite</em>
could scale very well in some cases.</p>
<p>Storing detailed data in BLOB or in TEXT as JSON could first sounds a wrong
idea. It does break one widely accepted principle of the RDBMS architecture.
But even <em>Google</em> had to break this dogma. And when <em>MySQL</em> or
such tries to implement sharding, it does need a lot of effort. Others, like
the NoSQL <em><a href="http://www.google.fr/url?sa=t&source=web&cd=1&ved=0CCkQFjAA&url=http%3A%2F%2Fwww.mongodb.org%2F&ei=srAQTuqtKIzasgbN4KHODg&usg=AFQjCNFkupTrrD2YYX6ydM2wb9IbWll3Vw&sig2=Hw8pU9KTy4M3Iu9i8hJwPA">
MongoDB</a></em>, are better candidates: they are not tight to the SQL flat
scheme.</p>
<p>Therefore, on second thought, having at hand a shared nothing architecture
could be a great advantage. Our ORM is already ready to break the
table-oriented scheme of SQL.<br />
Probably another benefit of the RESTful approach!</p>
<p>Just my Sunday's evening thoughts...<br />
<a href="http://synopse.info/forum/viewtopic.php?pid=2248">Any
feedback?</a></p>