Synopse Open Source - Tag - NoSQLmORMot MVC / SOA / ORM and friends2024-02-02T17:08:25+00:00urn:md5:cc547126eb580a9adbec2349d7c65274DotclearNew Client for MongoDB 5.1/6 Supporturn:md5:41f28034cfacffa2bf7fde35e8e64b432022-08-12T20:12:00+01:002022-08-12T20:12:00+01:00Arnaud BouchezmORMot FrameworkCrossPlatformDatabaseDelphiFPCFreePascalJSONMongoDBmORMot2NoSQLODMORMperformanceRESTSource<p>Starting with its version 5.1, <em>MongoDB</em> disabled the legacy protocol used for communication since its beginning.<br />
As a consequence, our <em>mORMot</em> client was not able to communicate any more with the latest versions of <em>MongoDB</em> instances.</p>
<p><img src="https://blog.synopse.info?post/public/mongodb.png" alt="" /></p>
<p>Last week, we made a deep rewrite of <a href="https://github.com/synopse/mORMot2/blob/master/src/db/mormot.db.nosql.mongodb.pas">mormot.db.nosql.mongodb.pas</a>, which changed the default protocol to use the new layout on the wire. Now messages use regular <a href="https://www.mongodb.com/docs/current/reference/command/">MongoDB Database Commands</a>, with automated compression if needed.</p>
<p>No change is needed in your end-user <em>MongoDB</em> or ORM/ODM code. The upgrade is as simple as update your <em>mORMot 2</em> source, then recompile.</p> <h4>The Mongo Wire Protocol</h4>
<p>Since its beginning, <em>MongoDB</em> used a simple protocol over TCP, via several binary opcodes and message, for CRUD operations.</p>
<p>A new alternative protocol was <a href="https://emptysqua.re/blog/driver-features-for-mongodb-3-6/">introduced in version 3.6,</a> and the former protocol was marked as deprecated.<br />
Two new opcodes were introduced, OP_MSG and OP_COMPRESSED, to replace all other frames. They just encapsulate, with or without compression, some abstract BSON content.<br />
The official documentation details those changes <a href="https://www.mongodb.com/docs/manual/reference/mongodb-wire-protocol/">in this web page</a>.</p>
<p>In short (picture extracted from the blog above), the protocol came from this:</p>
<p><img src="https://blog.synopse.info?post/public/ye-olde-wire-protocol.png" alt="" /></p>
<p>to this:</p>
<p><img src="https://blog.synopse.info?post/public/op-msg.png" alt="" /></p>
<p>The main benefit is that the commands and answers are just conventional BSON, so the protocol can change at logical/BSON/JSON level by adding or changing some members, with no need of dealing with low-level binary structures.</p>
<p>With the version 5.1 of <em>MongoDB</em>, the previous protocol was not just deprecated, but disabled.<br />
So we had to update the <em>mORMot 2</em> client code! (yes, the <em>mORMot 1</em> code has not been updated - it may become a good reason to upgrade)</p>
<h4>Deep Rewrite</h4>
<p>In fact, the official MongoDB documentation is somewhat vague. And the official drivers are a bit difficult to reverse-engineer, due to the verbose nature of C, Java or C#. The native/node driver was easiest to dissect, and we used it as reference.<br />
Luckily enough, there are some <a href="https://github.com/mongodb/specifications/blob/master/source/message/OP_MSG.rst">specification document available too</a>, which offers some additional valuable clarifications.</p>
<p>After some testing, we managed to replace all previous OP_QUERY and its brothers to the new OP_MSG frame, which is, as documented in the specification, "One opcode to rule them all". <img src="https://blog.synopse.info?pf=wink.svg" alt=";)" class="smiley" /></p>
<p>Once we had the commands working, we needed to rewrite all CRUD operations using commands, and not opcodes.<br />
Queries are now made with <code><a href="https://www.mongodb.com/docs/current/reference/command/find/">find</a></code> and <code><a href="https://www.mongodb.com/docs/current/reference/command/aggregate/">aggregate</a></code> commands. Their results are now located in a <code>"cursor": firstBatch": ..</code> BSON array within the response. And a new <code><a href="https://www.mongodb.com/docs/current/reference/command/getMore/">getMore</a></code> command is to be used to retrieve the next values within a <code>"cursor": nextBatch": ...</code> resultset.<br />
For writing, <code><a href="https://www.mongodb.com/docs/current/reference/command/insert">insert</a></code>, <code><a href="https://www.mongodb.com/docs/current/reference/command/update">update</a></code> and <code><a href="https://www.mongodb.com/docs/current/reference/command/delete">delete</a></code> commands are called, with their appropriate BSON content.</p>
<p>During the refactoring, we optimized the BSON process, and also enhanced the whole process, mainly the logs and the execution efficiency. The <em>mORMot</em> client side should not be a bottleneck. And it is not, even with this NoSQL database.</p>
<p>Don't expect any performance enhancement, or new features. It is just some low-level protocol change at TCP level.<br />
But if you used the "non acknowledged write mode" of the former protocol, which was unsafe but very fast, you will have lower performance with the new protocol, because the new protocol always acknowledges the commands it receives. So, in some very specific configurations, the new protocol may reduce the performance.</p>
<h4>Backward Compatibility</h4>
<p>All those changes were encapsulated in our revised <a href="https://github.com/synopse/mORMot2/blob/master/src/db/mormot.db.nosql.mongodb.pas">mormot.db.nosql.mongodb.pas</a> unit.</p>
<p>If you have a very old <em>MongoDB</em> instance, and don't want to upgrade, you could just compile your project with the <code>MONGO_OLDPROTOCOL</code> conditional, to use the deprecated opcodes.<br />
If the <em>MongoDB</em> team does not care much with backward compatibility (they could have kept the previous protocol for sure, they still maintain it for the handshake message if needed), we do care about not breaking too much things with <em>mORMot</em>, so we kept the previous code, and tested/validated it too, for legacy systems.</p>
<h4>New Sample</h4>
<p>We translated and introduced the <em>MongoDB</em> benchmark sample to <em>mORMot 2</em> code base.</p>
<p>You could find it, and run it, from <a href="https://github.com/synopse/mORMot2/tree/master/ex/mongodb">our source code repository</a>.</p>
<p>This code is a good entry point for what is possible with this unit in our framework, for both direct access or ORM/ODM access.<br />
And you would be able to guess the performance numbers you may achieve with your project.</p>
<p>Running a <em>MongoDB</em> database in a container is as easy as executing the following command:</p>
<pre>
sudo docker run --name mongodb -d -p 27017:27017 mongo:latest
</pre>
<p>Then you will have a <em>MongoDB</em> server instance accessible on <code>localhost:27017</code>, so you could run the sample straight away.</p>
<h4>Delphi/FPC Open Source Rocks</h4>
<p>We hope you will find the change painless and transparent. We did not modify the high-level client methods, nor break the ORM/ODM: you can still write some SELECT complex statements, and our ORM will translate it into <em>MongoDB</em> aggregate commands.</p>
<p>To my knowledge, there is <a href="https://github.com/stijnsanders/TMongoWire/commit/7f12a64f571e476704bdcb737e1fc087ef792f59">only a single other Delphi/FPC client library</a> which made the upgrade to the new protocol, at today. Once we made our own changes, we notified other library authors, and Stijn made very quickly the needed changes. Congrats! Maybe our code could be used as reference for other library maintainers, because the protocol needs some small tweaks sometimes.<br />
It is important to have some maintenance on the library you use. And our little <em>mORMot</em> is still on the edge: thanks to FPC, it runs very well on Linux and BSD, which makes it perfect for professional services running in the long term! :)</p>
<p>Your feedback is welcome <a href="https://synopse.info/forum/viewtopic.php?id=6318">in the forum thread which initiated these modifications</a>, as usual!<br />
Don't hesitate to notify us any missing or broken feature.<br />
Thanks Daniel for your report and support!</p>Letters of Hopeurn:md5:e98a568eefb37c7fe997014115e6ed3f2015-10-23T14:19:00+02:002015-10-23T13:22:47+02:00AB4327-GANDIPascal ProgrammingblogCQRSDelphiDomainDrivenEventCollaborationEventSourcingMongoDBmORMotNoSQLSOA <p>As we <a href="https://blog.synopse.info?post/post/2015/09/21/Embarcadero-bought-by">already notified in
this blog</a>, Embarcadero has been finally bought by IDERA.</p>
<p>Delphi users <a href="http://www.yanniel.info/2015/10/ideras-intention-for-embarcadero-development-tools.html">
received a letter from Randy Jacops</a>, IDERA CEO.<br />
Written in my mother language, in perfect French. Nice!</p>
<p><img src="http://borgenproject.org/wp-content/uploads/Demonstrate_Hope.jpg" alt="" /></p>
<p>The letter states that they have 20,000 customers...<br />
It sounds more realistic than the numbers usually given for Delphi
"users".<br />
Even if it counts for all their tools.<br />
<img src="https://blog.synopse.info?pf=smile.svg" alt=":)" class="smiley" /></p>
<p>In our forums, we have 1,384 <a href="http://synopse.info/forum/userlist.php">registered users</a> (real humans: we
do not accept bots via a <a href="https://en.wikipedia.org/wiki/Turing_test">Turing test</a> during
registration).<br />
It sounds like if Open Source projects are able to gather a lot of users.<br />
And certainly because we maintain support from Delphi 6 up to Seattle (and even
Delphi 5 for some part of our libraries)... we have for sure users using
FPC/Lazarus (which we also started to support), and others which did not
upgrade to the latest Delphi version!</p>
<p>In Randy's letter, the community has a special place.<br />
I hope future of Delphi would see Open Source projects brought by the community
as a chance, <a href="http://synopse.info/forum/viewtopic.php?pid=17453#p17453">not as
competition</a>.</p>
<p>I'm currently working on a cloud of <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html">
<em>mORMot</em> servers</a>, serving content coming from high numbers of
connected objects.<br />
Object Pascal powered servers, under Windows or Linux (with FPC), are working
24/7 with very low resource use.<br />
A lot of BigData stream is gathered into <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_84">
MongoDB</a> servers, following the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_167">CQRS
pattern</a>.<br />
It is so easy to deploy those servers (including their high performance
embedded SQlite3 database), that almost everyone in my company did install
their own "cloud", mainly for testing purpose of the objects we are
selling...<br />
Real-time remote monitoring of the servers is very easy and integrated. You
could even see the log changing in real-time, or run your SQL requests on the
databases, with ease.<br />
When I compare to previous projects I had to write or maintain using Java or
.Net, I can tell you that it is "something else".<br />
The IT administrators were speechless when they discovered how it worked: no
need of containers, no need of virtual machines (but for infrastructure
needs)...<br />
The whole stack is <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_17">
SOA oriented</a>, in an <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_173">
Event-Driven design</a> (thanks to WebSockets callbacks). It follows <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_54">
DDD principles</a>, thanks to the perfect readability of the object pascal
language.<br />
Delphi, and Open Source, could be great to create <a href="https://en.wikipedia.org/wiki/Internet_of_Things">Internet Of
Things</a> servers...</p>Delphi 10 = DX Seattle is out, mORMot supports iturn:md5:ac1f3bb4eb71134abb9c8f08fb724bf22015-08-31T17:50:00+02:002015-08-31T16:57:57+02:00AB4327-GANDIPascal ProgrammingblogDatabaseDelphiDXMongoDBmORMotNoSQLODMORMSeattle <p>We expected Delphi XE9, and now we have <a href="https://plus.google.com/u/0/+MarcoCantu/posts/F6WZDc93iHW">Rad Studio 10
Seattle</a>, with Delphi renamed as <a href="http://www.embarcadero.com/en/products/delphi">Delphi 10 Seattle</a>, or
simply <strong>DX</strong>.</p>
<p><img src="http://www.embarcadero.com/images/banners/seattle/delphi-sidebar.png" alt="" /></p>
<p>No big news for the Delphi compiler itself (we are still waiting for
<em>Linux</em> server support), but a lot of <em>FireMonkey</em> updates,
Windows 10 compatibility enhancements, enhancements to JSON (better performance
using a SAX approach), and <a href="http://synopse.info/forum/viewtopic.php?pid=17517#p17517">NoSQL/MongoDB
support in FireDAC</a>.<br />
The documentation is <a href="http://docwiki.embarcadero.com/Libraries/Seattle/en/FireDAC.Phys.MongoDBWrapper">
rather sparse for the new features</a>, but it goes into the right direction
(we <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">support
MongoDB</a> since a long time, in our ORM/ODM).<br />
See <a href="http://docwiki.embarcadero.com/RADStudio/Seattle/en/What%27s_New">what's new
in details</a>.</p>
<p><strong>Of course, our Open Source</strong> <em><strong>mORMot</strong></em>
<strong>framework supports this version.</strong><br />
Feedback is welcome, as usual!<br />
Enjoy the new DX IDE!</p>"SQL and NoSQL", not "SQL vs NoSQL"urn:md5:c22e98fb0648c165bc9ee01f23a34cdd2015-08-23T13:34:00+02:002015-08-23T13:03:13+02:00AB4327-GANDImORMot FrameworkblogDatabaseDelphiJSONMongoDBmORMotNoSQLODMOpenSourceORMSQLSQLite3TDocVariant<p>You know certainly that our <em>mORMot</em> Open Source framework is an ORM,
i.e. mapping objects to a relational / SQL database (<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_13">Object
Relational Mapping</a>).<br />
You may have followed also that it is able to connect to a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_83">
NoSQL database</a>, like <a href="https://www.mongodb.org/">MongoDB</a>, and
that the objects are then mapped via an ODM (<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_82">Object
Document Mapping</a>) - the original SQL SELECT are even <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_207">
translated on the fly to MongoDB queries</a>.</p>
<p>But thanks to <em>mORMot</em>, it is not "<em>SQL vs NoSQL</em>" - but
"<em>SQL and NoSQL</em>".<br />
You are not required to make an exclusive choice.<br />
You can share best of both worlds, depending on your application needs.</p>
<p><img src="http://bodescu.me/wp-content/uploads/2015/02/sql-vs-nosql-450x217.png" alt="" /></p>
<p>In fact, the framework is able to <em>add NoSQL features to a regular
relational / SQL database</em>, by storing JSON documents in TEXT columns.</p>
<p>In your end-user code, you just define a <code>variant</code> field in
the ORM, and store a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_80">
TDocVariant document</a> within.<br />
We also added some dedicated functions at SQL level, so that
<em>SQLite3</em> could be used as embedded fast engine, and provide
advanced WHERE clauses on this JSON content.</p> <h3>Schemaless storage via a variant</h3>
<p>As we just wrote, a first-class candidate for <em>data sharding</em> in a
<code>TSQLRecord</code> is our <em>TDocVariant custom variant type</em>.</p>
<p>You may define:</p>
<pre>
TSQLRecordData = <strong>class</strong>(TSQLRecord)
<strong>private</strong>
fName: RawUTF8;
fData: <strong>variant</strong>;
<strong>public</strong>
<strong>published</strong>
<strong>property</strong> Name: RawUTF8 <strong>read</strong> fTest <strong>write</strong> fTest <strong>stored</strong> AS_UNIQUE;
<span style="background-color:yellow;"><strong>property</strong> Data: <strong>variant read</strong> fData <strong>write</strong> fData;</span>
<strong>end</strong>;
</pre>
<p>Here, we defined two indexed keys, ready to access any data record:</p>
<ul>
<li>Via the <code>ID: TID</code> property defined at <code>TSQLRecord</code>
level, which will map the <em>SQLite3</em> <code>RowID</code> primary key;</li>
<li>Via the <code>Name: RawUTF8</code> property, which will was marked to be
indexed by setting the "<code>stored AS_UNIQUE</code>" attribute.</li>
</ul>
<p>Then, any kind of data may be stored in the <code>Data: variant</code>
published property. In the database, it will be stored as JSON UTF-8 text,
ready to be retrieved from any client, including AJAX / HTML5
applications.<br />
<em>Delphi</em> clients or servers will access those data via
<em>late-binding</em>, from its <code>TDocVariant</code> instance.</p>
<p>You just reproduced the <em>schema-less</em> approach of the NoSQL database
engines, in a few lines of code! Thanks to the <em>mORMot</em>'s <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_6">
JSON RESTful Client-Server</a></em> design, your applications are able to store
any kind of document, and easily access to them via HTTP.</p>
<p>The documents stored in such a database can have varying sets of fields,
with different types for each field. One could have the following objects in a
single collection of our <code>Data: variant</code> rows:</p>
<pre>
<em>{ name : "Joe", x : 3.3, y : [1,2,3] }</em>
<em>{ name : "Kate", x : "abc" }</em>
<em>{ q : 456 }</em>
</pre>
<p>Of course, when using the database for real problems, the data does have a
fairly consistent structure. Something like the following would be more common,
e.g. for a table persisting <em>student</em> objects:</p>
<pre>
<em>{ name : "Joe", age : 30, interests : "football" }</em>
<em>{ name : "Kate", age : 25 }</em>
</pre>
<p>Generally, there is a direct analogy between this <em>schema-less</em> style
and dynamically typed languages. Constructs such as those above are easy to
represent in <em>PHP</em>, <em>Python</em> and <em>Ruby</em>. And, thanks to
our <code>TDocVariant</code> <em>late-binding</em> magic, even our good
<em>Delphi</em> is able to handle those structures in our code. What we are
trying to do here is make this mapping to the database natural, like:</p>
<pre>
<strong>var</strong> aRec: TSQLRecordData;
aID: TID;
<strong>begin</strong>
<em>// initialization of one record</em>
aRec := TSQLRecordData.Create;
aRec.Name := 'Joe'; <em>// one unique key</em>
aRec.data := _JSONFast('{name:"Joe",age:30}'); <em>// create a TDocVariant</em>
<em>// or we can use this overloaded constructor for simple fields</em>
aRec := TSQLRecordData.Create(['Joe',_ObjFast(['name','Joe','age',30])]);
<em>// now we can play with the data, e.g. via late-binding:</em>
writeln(aRec.Name); <em>// will write 'Joe'</em>
writeln(aRec.Data); <em>// will write '{"name":"Joe","age":30}' (auto-converted to JSON string)</em>
aRec.Data.age := aRec.Data.age+1; <em>// one year older</em>
aRec.Data.interests := 'football'; <em>// add a property to the schema</em>
aID := aClient.Add(aRec,true); <em>// will store {"name":"Joe","age":31,"interests":"footbal"}</em>
aRec.Free;
<em>// now we can retrieve the data either via the aID created integer, or via Name='Joe'</em>
<strong>end</strong>;
</pre>
<p>One of the great benefits of these dynamic objects is that schema migrations
become very easy.<br />
With a traditional RDBMS, releases of code might contain data migration
scripts. Further, each release should have a reverse migration script in case a
rollback is necessary.<br />
<code>ALTER TABLE</code> operations can be very slow and result in scheduled
downtime.</p>
<p>With a <em>schema-less</em> organization of the data, 90% of the time
adjustments to the database become transparent and automatic.<br />
For example, if we wish to add GPA to the <em>student</em> objects, we add the
attribute, re-save, and all is well - if we look up an existing student and
reference GPA, we just get back null.<br />
Furthermore, if we roll back our code, the new GPA fields in the existing
objects are unlikely to cause problems if our code was well written.</p>
<p>In fact, <em>SQlite3</em> is so efficient about its indexes B-TREE storage,
that such a structure may be used as a credible alternative to much heavier
<em>NoSQL</em> engines, like <em>MongoDB</em> or <em>CouchDB</em>.<br />
With the possibility to add some "regular" fields, e.g. plain numbers (like
ahead-computed aggregation values), or text (like a summary or description
field), you can still use any needed fast SQL query, without the complexity of
<em>map/reduce</em> algorithm used by the <em>NoSQL</em> paradigm. You could
even use the <em>Full Text Search</em> - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_8">
FTS3/FTS4</a></em> - or RTREE extension advanced features of <em>SQLite3</em>
to perform your queries.<br />
Then, thanks to <em>mORMot</em>'s ability to access any external database
engine, you are able to perform a JOINed query of your <em>schema-less</em>
data with some data stored e.g. in an Oracle, PostgreSQL or MS SQL enterprise
database. Or switch later to a true <em>MongoDB</em> storage, in just one line
of code - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_84">
MongoDB + ORM = ODM</a></em>.</p>
<h3>JSON operations from SQL code</h3>
<p>As we stated, any <code>variant</code> field would be serialized as JSON,
then stored as plain TEXT in the database.<br />
In order to make a complex query on the stored JSON, you could retrieve it in
your end-user code, then use the corresponding <code>TDocVariant</code>
instance to perform the search on its content.<br />
Of course, all this has a noticeable performance cost, especially when the data
tend to grow.</p>
<p>The natural way of solving those performance issue is to add some "regular"
RDBMS fields, with a proper index, then perform the requests on those
fields.<br />
But sometimes, you may need to do some addition query, perhaps in conjunction
with "regular" field lookup, on the JSON data stored itself.<br />
In order to avoid the slowest conversion to the ORM client side, we defined
some SQL functions, dedicated to JSON process.</p>
<p>The first is <code>JsonGet()</code>, and is able to extract any value from
the TEXT field, mapping a <code>variant</code>:</p>
<table>
<tbody>
<tr>
<td><code>JsonGet(ArrColumn,0)</code></td>
<td>returns a property value by index, from a JSON array</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'PropName')</code></td>
<td>returns a property value by name, from a JSON object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Obj1.Obj2.Prop')</code></td>
<td>returns a property value by path, including nested JSON objects</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop1,Prop2')</code></td>
<td>extract properties by name, from a JSON object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop1,Obj1.Prop')</code></td>
<td>extract properties by name (including nested JSON objects), from a JSON
object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop*')</code></td>
<td>extract properties by wildchar name, from a JSON object</td>
</tr>
<tr>
<td><code>JsonGet(ObjColumn,'Prop*,Obj1.P*')</code></td>
<td>extract properties by wildchar name (including nested JSON objects), from a
JSON object</td>
</tr>
</tbody>
</table>
<p>If no value does match, this function would return the SQL
<code>NULL</code>. If the matching value is a simple JSON text or number, it
will be returned as a TEXT, INTEGER or DOUBLE value, ready to be passed as a
result column or any WHERE clause. If the returned value is a nested JSON
object or array, it will be returned as TEXT, serialized as JSON; as a
consequence, you may use it as the source of another <code>JsonGet()</code>
function, or even able to gather the results via the <code>CONCAT()</code>
aggregate function.</p>
<p>The comma-separated syntax allowed in the property name parameter (e.g.
<code>'Prop1,Prop2,Prop3'</code>), would search for several properties at once
in a single object, returning a JSON object of all matching values - e.g.
<code>'{"Prop2":"Value2","Prop3":123}'</code> if the <code>Prop1</code>
property did not appear in the stored JSON object.</p>
<p>If you end the property name with a <code>*</code> character, it would
return a JSON object, with all matching properties.<br />
Any nested object would have its property names be flattened as
{<code>"Obj1.Prop":...}</code>, within the returned JSON object.<br />
Note that the comma-separated syntax also allows such wildchar search, so that
e.g.</p>
<pre>
JsonGet(ObjColumn,'owner') = {"login":"smith","id":123456} as TEXT
JsonGet(ObjColumn,'owner.login') = "smith" as TEXT
JsonGet(ObjColumn,'owner.id') = 123456 as INTEGER
JsonGet(ObjColumn,'owner.name') = NULL
JsonGet(ObjColumn,'owner.login,owner.id') = {"owner.login":"smith","owner.id":123456} as TEXT
JsonGet(ObjColumn,'owner.I*') = {"owner.id:123456} as TEXT
JsonGet(ObjColumn,'owner.*') = {"owner.login":"smith","owner.id":123456} as TEXT
JsonGet(ObjColumn,'unknown.*') = NULL
</pre>
<p>Another function, named <code>JsonHas()</code> is similar to
<code>JsonGet()</code>, but will return TRUE or FALSE depending if the supplied
property (specified by name or index) do exist. It may be faster to use
<code>JsonHas()</code> than <code>JsonGet()</code> e.g. in a WHERE clause, when
you do not want to process this property value, but only return data rows
containing needed information.</p>
<pre>
JsonHas(ObjColumn,'owner') = true
JsonHas(ObjColumn,'owner.login') = true
JsonHas(ObjColumn,'owner.name') = false
JsonHas(ObjColumn,'owner.i*') = true
JsonHas(ObjColumn,'owner.n*') = false
</pre>
<p>Since the process would take place within the <em>SQLite3</em> engine
itself, and since they use a SAX-like fast approach (without any temporary
memory allocation during its search), those JSON functions could be pretty
efficient, and proudly compare to some dedicated NoSQL engines.</p>
<h3>The upcoming official SQlite3 extension</h3>
<p>As you may have noticed <a href="http://www.sqlite.org/src/info/178f9a352c6c9e15">in the SQlite3 timeline</a>,
some extension functions for processing JSON are currently being
implemented.</p>
<p>The <em>mORMot</em> native implementation is very proven, since it uses the
<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_2">
JSON core of the framework</a>, and is able to handle the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_39">
"extended" JSON syntax of MongoDB</a>: field names can be serialized
<em>without</em> double quotes, which reduces a lot the storage size, and the
transmission size, when the JSON is stored within a TEXT column, which would
escape all double quote characters.</p>
<p>I guess the performance of our implementation is probably faster than
current <em>SQlite3</em>'s extension.<br />
We use a SAX-like fast approach, without any temporary memory allocation during
its search, whereas AFAIK <em>SQlite3</em>'s extension is doing a lot of
memory allocations, creating all nodes of the JSON documents, like a DOM.</p>
<p>But here again, we are not exclusive guys. If you like this upcoming JSON
extension, you are free to use it! No offense taken!</p>
<p>Feedback <a href="http://synopse.info/forum/viewtopic.php?pid=17448#p17448">is welcome on our
forum</a>, as usual.</p>ODM magic: complex queries over NoSQL / MongoDBurn:md5:795ff9d748ddac411e5d1e63588427d72014-11-28T17:37:00+01:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkblogDatabaseJSONMongoDBmORMotNoSQLODMORMRestSQL<p>You know that our <em>mORMot</em> is <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">able to access directly any
<em>MongoDB</em> database engine</a>, allowing its ORM to become an ODM,
and using NoSQL instead of SQL for the query languages.</p>
<p><img src="https://cdn.tutsplus.com/net/uploads/legacy/1091_mongodb/preview.jpg" alt="" /></p>
<p>But at <em>mORMot</em> level, you could share the same code between your
RDBMS and NoSQL databases.<br />
The ORM/ODM is able to do all the conversions by itself!<br />
Since <a href="http://synopse.info/fossil/info/abbdfa5563">we have just
improved this feature</a>, it is time to enlighten its current status.</p> <p>To perform a query and retrieve the content of several documents, you can
use regular <code>CreateAndFillPrepare</code> or <code>FillPrepare</code>
methods:</p>
<pre>
<span style="background-color:yellow;">R := TSQLORM.CreateAndFillPrepare(Client,WHERE_CLAUSE,[WHERE_PARAMETERS]);</span>
<strong>try</strong>
n := 0;
<span style="background-color:yellow;"><strong>while</strong> R.FillOne <strong>do begin</strong></span>
<em>// here R instance contains all values of one document, excluding BLOBs</em>
inc(n);
<strong>end</strong>;
assert(n=COLL_COUNT);
<strong>finally</strong>
R.Free;
<strong>end</strong>;
</pre>
<p>A WHERE clause can also be defined for <code>CreateAndFillPrepare</code> or
<code>FillPrepare</code> methods.<br />
This WHERE clause could contain several expressions, joined with
<code>AND</code> / <code>OR</code>.<br />
Each of those expressions could use:</p>
<ul>
<li>The simple comparators <code>= < <= <> > >=</code>,</li>
<li>An <code>IN (....)</code> clause,</li>
<li><code>IS NULL</code> / <code>IS NOT NULL</code> tests,</li>
<li>A <code>LIKE</code> operation,</li>
<li>Or even any <code>...DynArrayContains()</code> specific function.</li>
</ul>
<p>The <em>mORMot</em> ODM will convert this SQL-like statement into the
optimized <code>MongoDB</code> query expression, using e.g. a <em>regular
expression</em> for the <code>LIKE</code> operator.</p>
<p>The <code>LIMIT</code>, <code>OFFSET</code> and <code>ORDER BY</code>
clauses will also be handled as expected.<br />
A special care would be taken for an <code>ORDER BY</code> on textual values:
by design, <em>MongoDB</em> will always sort text with case-sensitivity, which
is not what we expect: so our ODM will sort such content on client side, after
having been retrieved from the <em>MongoDB</em> server. For numerical fields,
<em>MongoDB</em> sorting features would be processed on the server side.</p>
<p>The <code>COUNT(*)</code> function would also be converted into the proper
<em>MongoDB</em> API call, so that such operations would be as costless as
possible.<br />
<code>DISTINCT() MAX() MIN() AVG()</code> functions and the <code>GROUP
BY</code> clause would also be converted into optimized <em>MongoDB</em>
aggregation pipelines, on the fly.<br />
You could even set aliases for the columns (e.g. <code>max(RowID) as
first</code>) and perform simple addition/substraction of an integer value.</p>
<p>Here are some typical WHERE clauses, and the corresponding <em>MongoDB</em>
query document as generated by the ODM:</p>
<table>
<tbody>
<tr>
<td><strong>WHERE clause</strong></td>
<td><strong><em>MongoDB</em> Query</strong></td>
</tr>
<tr>
<td><code>'Name=?',['Name 43']</code></td>
<td><code>{Name:"Name 43"}</code></td>
</tr>
<tr>
<td><code>'Age<?',[51]</code></td>
<td><code>{Age:{$lt:51}}</code></td>
</tr>
<tr>
<td><code>'Age in (1,10,20)'</code></td>
<td><code>{Age:{$in:[1,10,20]}}</code></td>
</tr>
<tr>
<td><code>'Age in (1,10,20) and ID=?',[10]</code></td>
<td><code>{Age:{$in:[1,10,20]},_id:10}</code></td>
</tr>
<tr>
<td><code>'Age in (10,20) or ID=?',[30]</code></td>
<td><code>{$or:[{Age:{$in:[10,20]}},{_id:30}]}</code></td>
</tr>
<tr>
<td><code>'Name like ?',['name 1%']</code></td>
<td><code>{Name:/^name 1/i}</code></td>
</tr>
<tr>
<td><code>'Name like ?',['name 1']</code></td>
<td><code>{Name:/^name 1$/i}</code></td>
</tr>
<tr>
<td><code>'Name like ?',['%ame 1%']</code></td>
<td><code>{Name:/ame 1/i}</code></td>
</tr>
<tr>
<td><code>'Data is null'</code></td>
<td><code>{Data:null}</code></td>
</tr>
<tr>
<td><code>'Data is not null'</code></td>
<td><code>{Data:{$ne:null}}</code></td>
</tr>
<tr>
<td><code>'Age<? limit 10',[51]</code></td>
<td><code>{Age:{$lt:51}} + limit 10</code></td>
</tr>
<tr>
<td><code>'Age in (10,20) or ID=? order by ID desc',[30]</code></td>
<td>
<code>{$query:{$or:[{Age:{$in:[10,20]}},{_id:30}]},$orderby:{_id:-1}}</code></td>
</tr>
<tr>
<td><code>'order by Name'</code></td>
<td><code>{} + client side text sort by Name</code></td>
</tr>
<tr>
<td><code>'Age in (1,10,20) and
IntegerDynArrayContains(Ints,?)',[10])</code></td>
<td><code>{Age:{$in:[1,10,20]},Ints:{$in:[10]}}</code></td>
</tr>
<tr>
<td><code>Distinct(Age),max(RowID) as first,count(Age) as count group by
age</code></td>
<td>
<code>{$group:{_id:"$Age",f1:{$max:"$_id"},f2:{$sum:1}}},{$project:{_id:0,"Age":"$_id","first":"$f1","count":"$f2"}}</code></td>
</tr>
<tr>
<td><code>min(RowID),max(RowID),Count(RowID)</code></td>
<td>
<code>{$group:{_id:null,f0:{$min:"$_id"},f1:{$max:"$_id"},f2:{$sum:1}}},{$project:{_id:0,"min(RowID)":"$f0","max(RowID)":"$f1","Count(RowID)":"$f2"}}</code></td>
</tr>
<tr>
<td><code>min(RowID) as a,max(RowID)+1 as b,Count(RowID) as c</code></td>
<td>
<code>{$group:{_id:null,f0:{$min:"$_id"},f1:{$max:"$_id"},f2:{$sum:1}}},{$project:{_id:0,"a":"$f0","b":{$add:["$f1",1]},"c":"$f2"}}</code></td>
</tr>
</tbody>
</table>
<p>Note that parenthesis and mixed <code>AND OR</code> expressions are not
handled yet.<br />
You could always execute any complex query (e.g. aggregations or
<em>Map/Reduce</em>) by using directly the <code>TMongoCollection</code>
methods.</p>
<p>But for most cases, <em>mORMot</em> allows to share the same exact code
between your regular SQL databases or NoSQL engines.<br />
You do not need to learn the <em>MongoDB</em> query syntax: the ODM would
compute the right expression for you, depending on the database engine it runs
on.</p>
<p>Ensure you <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_83">
took a look at the updated documentation</a> to better understand NoSQL/MongoDB
integration with <em>mORMot</em>.<br />
And feel free to <a href="http://synopse.info/forum/">use our forum</a> for
feedback, as usual!</p>
<p>Enjoy the <em>NoSQL</em> world!</p>BREAKING CHANGE: TSQLRestServerStatic* classes are now renamed as TSQLRestStorage*urn:md5:d4ac2a6610a67b47008cc74f4ddabc642014-05-09T17:32:00+02:002014-05-09T16:35:52+02:00AB4327-GANDImORMot FrameworkblogDelphiDocumentationGoodPracticemappingModelMongoDBmORMotNoSQLODBCOleDBORMRest<p>From the beginning, server-side storage tables which were not store in
a <em>SQLite3</em> database were implemented via some classes
inheriting from <code>TSQLRestServerStatic.<br />
<span style="color: rgb(0, 0, 0); font-family: 'DejaVu Sans', 'Lucida Grande', 'Lucida Sans Unicode', Arial, sans-serif; font-size: 12px; font-weight: normal;">
This <code>TSQLRestServerStatic</code> was inheriting
from </span>TSQLRestServer</code>, which did not make much sense (but was
made for laziness years ago, if I remember well).</p>
<p>Now, a new <code>TSQLRestStorage</code> class, directly inheriting from
<code>TSQLRest</code>, is used for per-table storage.<br />
This <a href="http://synopse.info/fossil/info/e1e58505f3">huge code
refactoring</a> results in a much cleaner design, and will enhance code
maintainability.<br />
Documentation has been updated to reflect the changes.</p>
<p><img src="http://blogs.scientificamerican.com/media/inline/blog/Image/marmot_trap.jpg" alt="" /></p>
<p>Note that this won't change anything when using the framework (but the new
class names): it is an implementation detail, which had to be fixed.</p> <p>In the <em>mORMot</em> units, you may also find those classes also
inheriting from <code>TSQLRestStorage</code>:</p>
<p><a href="https://blog.synopse.info?post/public/mORMot/TSQLRestStorageHierarchy.png"><img src="https://blog.synopse.info?post/public/mORMot/.TSQLRestStorageHierarchy_m.jpg" alt="" title="TSQLRestStorage classes, mai 2014" /></a></p>
<p>In the above class hierarchy, the
<code>TSQLRestStorage[InMemory][External]</code> classes are in fact used to
store some <code>TSQLRecord</code> tables in any non-SQL backend:</p>
<ul>
<li><code>TSQLRestStorageExternal</code> maps tables stored in an external
database (e.g. Oracle, MSSQL, PostgreSQL, FireBird, MySQL or any OleDB/ODBC
provider, via our <em>SynDB</em> <a href="https://blog.synopse.info?post/post/2014/03/07/Support-of-MySQL%2C-DB2-and-PostgreSQL">optimized
classes</a>);</li>
<li><code>TSQLRestStorageInMemory</code> stores the data in a
<code>TObjectList</code>- with amazing performance;</li>
<li><code>TSQLRestStorageMongoDB</code> will connect to a <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">remote <em>MongoDB</em> server</a> to
store the tables as a NoSQL collection of documents.</li>
</ul>
<p>Those classes are used within a main <code>TSQLRestServer</code> to host
some given <code>TSQLRecord</code> classes, either in-memory, or on external
databases.<br />
They do not enter in account in our Client-Server architecture, but are
implementation details, on the server side.<br />
From the client side, you do not have to worry about how the data is stored,
just consume it via <a href="https://blog.synopse.info?post/post/2014/01/10/REpresentational-State-Transfer-%28REST%29">REST</a>.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1752">welcome in our forum</a>, as
usual!</p>MongoDB + mORMot benchmarkurn:md5:ffe569e390738cbf6244be3f0a64bf512014-05-07T09:31:00+02:002014-05-07T08:57:07+02:00AB4327-GANDImORMot FrameworkACIDAJAXBatchblogDatabaseDelphiDocumentationDomainDrivenGoodPracticeJSONmappingModelMongoDBmORMotNoSQLOpenSourceORMperformanceRestshardingSourceSQLtransactionUnicode<p>Here are some benchmark charts about <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM"><em>MongoDB</em> integration in
<em>mORMot</em>'s ORM</a>.</p>
<p><img src="http://www.ceillac.com/IMAGES%20SITE/animaux/marmot-s2.jpg" alt="" /></p>
<p><em>MongoDB</em> appears as a serious competitor to SQL databases, with the
potential benefit of horizontal scaling and installation/administration ease -
performance is very high, and its document-based storage fits perfectly with
<em>mORMot</em>'s advanced ORM features like <em><a href="https://blog.synopse.info?post/post/2011/07/03/Sharde-nothing-architecture">Shared nothing architecture (or
sharding)</a></em>.</p> <p>Following tests were using Synopse <em>mORMot</em> framework 1.18, compiled
with Delphi XE4, against SQLite 3.8.4.3.</p>
<p>We won't show all database engine, but the most representative ones.<br />
Please refer to <a href="https://blog.synopse.info?post/post/2014/03/07/Support-of-MySQL%2C-DB2-and-PostgreSQL">this other benchmark
article for some more complete information</a>.</p>
<h1>Insertion speed</h1>
<p>'MongoDB ack/no ack' for direct <em>MongoDB</em> access
(<code>SynMongoDB.pas</code>) with or without write acknowledge.</p>
<p>For the testings, we used a local <em>MongoDB</em> 2.6 instance in 64 bit
mode.</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>Direct</strong></td>
<td><strong>Batch</strong></td>
<td><strong>Trans</strong></td>
<td><strong>Batch Trans</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>466</td>
<td>437</td>
<td>81754</td>
<td>108752</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>2012</td>
<td>2044</td>
<td>84550</td>
<td>111731</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off exc)</strong></td>
<td>25079</td>
<td>28192</td>
<td>83943</td>
<td>115159</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>69961</td>
<td>94871</td>
<td>87279</td>
<td>118657</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>232385</td>
<td>400608</td>
<td>252678</td>
<td>402803</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>242812</td>
<td>409131</td>
<td>240003</td>
<td>405712</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>490</td>
<td>11918</td>
<td>87556</td>
<td>151144</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>2141</td>
<td>47266</td>
<td>89249</td>
<td>160616</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>33199</td>
<td>145471</td>
<td>90025</td>
<td>158815</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>76411</td>
<td>184706</td>
<td>89834</td>
<td>192618</td>
</tr>
<tr align="center">
<td><strong>MongoDB (ack)</strong></td>
<td>10081</td>
<td>84585</td>
<td>9800</td>
<td>85232</td>
</tr>
<tr align="center">
<td><strong>MongoDB (no ack)</strong></td>
<td>33223</td>
<td>189186</td>
<td>27974</td>
<td>206355</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>474</td>
<td>11917</td>
<td>36740</td>
<td>55767</td>
</tr>
<tr align="center">
<td><strong>FireDAC SQlite3</strong></td>
<td>20735</td>
<td>40083</td>
<td>40408</td>
<td>121359</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>10056</td>
<td>10155</td>
<td>18769</td>
<td>20335</td>
</tr>
<tr align="center">
<td><strong>FireDAC Firebird</strong></td>
<td>19742</td>
<td>48684</td>
<td>19904</td>
<td>47803</td>
</tr>
<tr align="center">
<td><strong>MSSQL2012 local</strong></td>
<td>3470</td>
<td>35510</td>
<td>10750</td>
<td>47653</td>
</tr>
<tr align="center">
<td><strong>ODBC MSSQL2012</strong></td>
<td>3659</td>
<td>6252</td>
<td>5537</td>
<td>6290</td>
</tr>
<tr align="center">
<td><strong>FireDAC MSSQL2012</strong></td>
<td>3276</td>
<td>5838</td>
<td>9540</td>
<td>40040</td>
</tr>
<tr align="center">
<td><strong>ZEOS PostgreSQL</strong></td>
<td>2953</td>
<td>23740</td>
<td>6913</td>
<td>29780</td>
</tr>
<tr align="center">
<td><strong>ODBC PostgreSQL</strong></td>
<td>2902</td>
<td>25040</td>
<td>3576</td>
<td>28714</td>
</tr>
<tr align="center">
<td><strong>FireDAC PostgreSQL</strong></td>
<td>3054</td>
<td>23329</td>
<td>7149</td>
<td>24844</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Batch+Trans|Trans|Batch|Direct&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,409131&chds=0,409131,0,409131,0,409131,0,409131,0,409131&chd=t:466,437,81754,108752|2012,2044,84550,111731|25079,28192,83943,115159|69961,94871,87279,118657|232385,400608,252678,402803|242812,409131,240003,405712|490,11918,87556,151144|2141,47266,89249,160616|33199,145471,90025,158815|76411,184706,89834,192618|10081,84585,9800,85232|33223,189186,27974,206355|474,11917,36740,55767|20735,40083,40408,121359|10056,10155,18769,20335|19742,48684,19904,47803|3470,35510,10750,47653|3659,6252,5537,6290|3276,5838,9540,40040|2953,23740,6913,29780|2902,25040,3576,28714|3054,23329,7149,24844&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|MongoDB+%28ack%29|MongoDB+%28no+ack%29|ZEOS+SQlite3|FireDAC+SQlite3|ZEOS+Firebird|FireDAC+Firebird|MSSQL2012+local|ODBC+MSSQL2012|FireDAC+MSSQL2012|ZEOS+PostgreSQL|ODBC+PostgreSQL|FireDAC+PostgreSQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|FireDAC+PostgreSQL|ODBC+PostgreSQL|ZEOS+PostgreSQL|FireDAC+MSSQL2012|ODBC+MSSQL2012|MSSQL2012+local|FireDAC+Firebird|ZEOS+Firebird|FireDAC+SQlite3|ZEOS+SQlite3|MongoDB+%28no+ack%29|MongoDB+%28ack%29|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,409131&chds=0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131,0,409131&chd=t:466,2012,25079,69961,232385,242812,490,2141,33199,76411,10081,33223,474,20735,10056,19742,3470,3659,3276,2953,2902,3054|437,2044,28192,94871,400608,409131,11918,47266,145471,184706,84585,189186,11917,40083,10155,48684,35510,6252,5838,23740,25040,23329|81754,84550,83943,87279,252678,240003,87556,89249,90025,89834,9800,27974,36740,40408,18769,19904,10750,5537,9540,6913,3576,7149|108752,111731,115159,118657,402803,405712,151144,160616,158815,192618,85232,206355,55767,121359,20335,47803,47653,6290,40040,29780,28714,24844&chdl=Direct|Batch|Trans|Batch+Trans" /></p>
<p><em>MongoDB</em> bulk insertion has been implemented, which shows an amazing
speed increase in Batch mode. Depending on the <em>MongoDB write concern</em>
mode, insertion speed can be very high: by default, every write process will be
acknowledge by the server, but you can by-pass this request if you set the
<code>wcUnacknowledged</code> mode - note that in this case, any error (e.g. an
unique field duplicated value) will never be notified, so it should not be used
in production, unless you need this feature to quickly populate a database, or
consolidate some data as fast as possible.</p>
<h1>Read speed</h1>
<p>The same records are now read, either one by one, either as a list:</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>By one</strong></td>
<td><strong>All Virtual</strong></td>
<td><strong>All Direct</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>21607</td>
<td>455083</td>
<td>458757</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>22177</td>
<td>456454</td>
<td>458001</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off exc)</strong></td>
<td>98014</td>
<td>454215</td>
<td>457540</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>99190</td>
<td>461808</td>
<td>464252</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>235504</td>
<td>756773</td>
<td>750300</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>233666</td>
<td>332402</td>
<td>733460</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>103917</td>
<td>210863</td>
<td>458379</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>101498</td>
<td>209634</td>
<td>441033</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>101839</td>
<td>218292</td>
<td>439947</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>102414</td>
<td>185494</td>
<td>438904</td>
</tr>
<tr align="center">
<td><strong>MongoDB (ack)</strong></td>
<td>8002</td>
<td>242353</td>
<td>251268</td>
</tr>
<tr align="center">
<td><strong>MongoDB (no ack)</strong></td>
<td>8234</td>
<td>252079</td>
<td>254582</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>31135</td>
<td>173593</td>
<td>263060</td>
</tr>
<tr align="center">
<td><strong>FireDAC SQlite3</strong></td>
<td>6318</td>
<td>67169</td>
<td>92291</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>12076</td>
<td>67853</td>
<td>85828</td>
</tr>
<tr align="center">
<td><strong>FireDAC Firebird</strong></td>
<td>1918</td>
<td>37113</td>
<td>44894</td>
</tr>
<tr align="center">
<td><strong>MSSQL2012 local</strong></td>
<td>7904</td>
<td>182401</td>
<td>349797</td>
</tr>
<tr align="center">
<td><strong>ODBC MSSQL2012</strong></td>
<td>8693</td>
<td>113973</td>
<td>178526</td>
</tr>
<tr align="center">
<td><strong>FireDAC MSSQL2012</strong></td>
<td>3054</td>
<td>63730</td>
<td>86051</td>
</tr>
<tr align="center">
<td><strong>ZEOS PostgreSQL</strong></td>
<td>7031</td>
<td>122327</td>
<td>176298</td>
</tr>
<tr align="center">
<td><strong>ODBC PostgreSQL</strong></td>
<td>7281</td>
<td>66843</td>
<td>91489</td>
</tr>
<tr align="center">
<td><strong>FireDAC PostgreSQL</strong></td>
<td>1644</td>
<td>45184</td>
<td>61252</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|All+Direct|All+Virtual|By+one&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,756773&chds=0,756773,0,756773,0,756773&chd=t:21607,455083,458757|22177,456454,458001|98014,454215,457540|99190,461808,464252|235504,756773,750300|233666,332402,733460|103917,210863,458379|101498,209634,441033|101839,218292,439947|102414,185494,438904|8002,242353,251268|8234,252079,254582|31135,173593,263060|6318,67169,92291|12076,67853,85828|1918,37113,44894|7904,182401,349797|8693,113973,178526|3054,63730,86051|7031,122327,176298|7281,66843,91489|1644,45184,61252&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|MongoDB+%28ack%29|MongoDB+%28no+ack%29|ZEOS+SQlite3|FireDAC+SQlite3|ZEOS+Firebird|FireDAC+Firebird|MSSQL2012+local|ODBC+MSSQL2012|FireDAC+MSSQL2012|ZEOS+PostgreSQL|ODBC+PostgreSQL|FireDAC+PostgreSQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|FireDAC+PostgreSQL|ODBC+PostgreSQL|ZEOS+PostgreSQL|FireDAC+MSSQL2012|ODBC+MSSQL2012|MSSQL2012+local|FireDAC+Firebird|ZEOS+Firebird|FireDAC+SQlite3|ZEOS+SQlite3|MongoDB+%28no+ack%29|MongoDB+%28ack%29|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,756773&chds=0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773,0,756773&chd=t:21607,22177,98014,99190,235504,233666,103917,101498,101839,102414,8002,8234,31135,6318,12076,1918,7904,8693,3054,7031,7281,1644|455083,456454,454215,461808,756773,332402,210863,209634,218292,185494,242353,252079,173593,67169,67853,37113,182401,113973,63730,122327,66843,45184|458757,458001,457540,464252,750300,733460,458379,441033,439947,438904,251268,254582,263060,92291,85828,44894,349797,178526,86051,176298,91489,61252&chdl=By+one|All+Virtual|All+Direct" /></p>
<p><em>MongoDB</em> appears as a serious competitor to SQL databases, with the
potential benefit of horizontal scaling and installation/administration ease -
performance is very high, and its document-based storage fits perfectly with
<em>mORMot</em>'s advanced ORM features like <em><a href="https://blog.synopse.info?post/post/2011/07/03/Sharde-nothing-architecture">Shared nothing architecture (or
sharding)</a></em>.</p>
<p>You can get more information about <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-database-access">low-level integration of
<em>MongoDB</em></a> in <em>mORMot</em>, or our integrated <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">ORM/ODM support in the
framework</a>.<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1742">welcome
on our forum</a>, as usual!</p>MongoDB + mORMot ORM = ODMurn:md5:232877d989279d738aab47cacf5358f82014-05-07T09:14:00+02:002014-05-07T13:52:17+02:00AB4327-GANDImORMot FrameworkACIDAJAXBatchblogDatabaseDelphiDocumentationDomainDrivenGoodPracticeJSONmappingModelMongoDBmORMotNoSQLOpenSourceORMperformanceRestshardingSourceSQLtransactionUnicode<p><em><a href="http://www.mongodb.org/">MongoDB</a></em> (from "humongous") is
a cross-platform document-oriented database system, and certainly the best
known NoSQL database.<br />
According to <a href="http://db-engines.com">http://db-engines.com</a> in April
2014, <em>MongoDB</em> is in 5th place of the most popular types of database
management systems, and first place for NoSQL database management
systems.<br />
Our <em>mORMot</em> gives premium access to this database, featuring full
<em>NoSQL and Object-Document Mapping (ODM)</em> abilities to the
framework.</p>
<p><img src="http://upload.wikimedia.org/wikipedia/en/thumb/e/eb/MongoDB_Logo.png/320px-MongoDB_Logo.png" alt="" /></p>
<p>Integration is made at two levels:</p>
<ul>
<li>Direct low-level access to the <em>MongoDB</em> server, in the
<code>SynMongoDB.pas</code> unit;</li>
<li>Close integration with our ORM (which becomes <em>defacto</em> an ODM), in
the <code>mORMotMongoDB.pas</code> unit.</li>
</ul>
<p><em>MongoDB</em> eschews the traditional table-based relational database
structure in favor of JSON-like documents with dynamic schemas
(<em>MongoDB</em> calls the format BSON), which matches perfectly
<em>mORMot</em>'s RESTful approach.</p>
<p>This second article will focus on integration of <em>MongoDB</em> with our
ORM.</p> <h3>MongoDB + ORM = ODM</h3>
<p>The <code>mORMotMongoDB.pas</code> unit is able to let any
<code>TSQLRecord</code> class be persisted on a remote <em>MongoDB</em>
server.</p>
<p>As a result, our ORM is able to be used as a <em>NoSQL and Object-Document
Mapping (ODM)</em> framework, with almost no code change.<br />
Any <em>MongoDB</em> database can be accessed via RESTful commands, using JSON
over HTTP.</p>
<p>This integration benefits from the other parts of the framework (e.g. our
UTF-8 dedicated process, which is also the native encoding for BSON), so you
can easily mix SQL and NoSQL databases with the exact same code, and are still
able to tune any SQL or <em>MongoDB</em> request in your code, if
necessary.</p>
<h3>Register the TSQLRecord class</h3>
<p>In the database model, we define a <code>TSQLRecord</code> class, as
usual:</p>
<pre>
TSQLORM = <strong>class</strong>(TSQLRecord)
<strong>private</strong>
fAge: integer;
fName: RawUTF8;
fDate: TDateTime;
fValue: <strong>variant</strong>;
fInts: TIntegerDynArray;
fCreateTime: TCreateTime;
fData: TSQLRawBlob;
<strong>published</strong>
<strong>property</strong> Name: RawUTF8 <strong>read</strong> fName <strong>write</strong> fName <strong>stored</strong> AS_UNIQUE;
<strong>property</strong> Age: integer <strong>read</strong> fAge <strong>write</strong> fAge;
<strong>property</strong> Date: TDateTime <strong>read</strong> fDate <strong>write</strong> fDate;
<strong>property</strong> Value: <strong>variant read</strong> fValue <strong>write</strong> fValue;
<strong>property</strong> Ints: TIntegerDynArray <strong>index</strong> 1 <strong>read</strong> fInts <strong>write</strong> fInts;
<strong>property</strong> Data: TSQLRawBlob <strong>read</strong> fData <strong>write</strong> fData;
<strong>property</strong> CreateTime: TCreateTime <strong>read</strong> fCreateTime <strong>write</strong> fCreateTime;
<strong>end</strong>;
</pre>
<p>Note that we did not define any <strong><code>index</code></strong>
<code>...</code> values for the <code>RawUTF8</code> property, as we need for
external SQL databases, since <em>MongoDB</em> does not expect any restriction
about text fields length.</p>
<p>The property values will be stored in the native <em>MongoDB</em> layout,
i.e. with a better coverage than the SQL types:</p>
<table>
<tbody>
<tr>
<td><strong>Delphi</strong></td>
<td><strong><em>MongoDB</em></strong></td>
<td><strong>Remarks</strong></td>
</tr>
<tr>
<td><code>byte</code></td>
<td>int32</td>
</tr>
<tr>
<td><code>word</code></td>
<td>int32</td>
</tr>
<tr>
<td><code>integer</code></td>
<td>int32</td>
</tr>
<tr>
<td><code>cardinal</code></td>
<td>N/A</td>
<td>You should use <code>Int64</code> instead</td>
</tr>
<tr>
<td><code>Int64</code></td>
<td>int64</td>
</tr>
<tr>
<td><code>boolean</code></td>
<td>boolean</td>
<td>0 is <code>false</code>, anything else is <code>true</code></td>
</tr>
<tr>
<td>enumeration</td>
<td>int32</td>
<td>store the ordinal value of the enumerated item(i.e. starting at 0 for the
first element)</td>
</tr>
<tr>
<td>set</td>
<td>int32</td>
<td>each bit corresponding to an enumerated item (therefore a set of up to 64
elements can be stored in such a field)</td>
</tr>
<tr>
<td><code>single</code></td>
<td>double</td>
</tr>
<tr>
<td><code>double</code></td>
<td>double</td>
</tr>
<tr>
<td><code>extended</code></td>
<td>double</td>
<td>stored as <code>double</code> (precision lost)</td>
</tr>
<tr>
<td><code>currency</code></td>
<td>double</td>
<td>safely converted to/from <code>currency</code> type with fixed decimals,
without rounding error</td>
</tr>
<tr>
<td><code>RawUTF8</code></td>
<td>UTF-8</td>
<td>this is the <strong>preferred</strong> field type for storing some textual
content in the ORM</td>
</tr>
<tr>
<td><code>WinAnsiString</code></td>
<td>UTF-8</td>
<td><em>WinAnsi</em> char-set (code page 1252) in Delphi</td>
</tr>
<tr>
<td><code>RawUnicode</code></td>
<td>UTF-8</td>
<td><em>UCS2</em> char-set in Delphi, as <code>AnsiString</code></td>
</tr>
<tr>
<td><code>WideString</code></td>
<td>UTF-8</td>
<td><em>UCS2</em> char-set, as COM BSTR type (Unicode in all version of
Delphi)</td>
</tr>
<tr>
<td><code>SynUnicode</code></td>
<td>UTF-8</td>
<td>Will be either <code>WideString</code> before Delphi 2009, or
<code>UnicodeString</code> later</td>
</tr>
<tr>
<td><code>string</code></td>
<td>UTF-8</td>
<td>Not to be used before Delphi 2009 (unless you may loose some data during
conversion) - <code>RawUTF8</code> is preferred in all cases</td>
</tr>
<tr>
<td><code>TDateTime</code></td>
<td>datetime</td>
<td>ISO 8601 encoded date time</td>
</tr>
<tr>
<td><code>TTimeLog</code></td>
<td>int64</td>
<td>as proprietary fast <code>Int64</code> date time</td>
</tr>
<tr>
<td><code>TModTime</code></td>
<td>int64</td>
<td>the server date time will be stored when a record is modified (as
proprietary fast <code>Int64</code>)</td>
</tr>
<tr>
<td><code>TCreateTime</code></td>
<td>int64</td>
<td>the server date time will be stored when a record is created (as
proprietary fast <code>Int64</code>)</td>
</tr>
<tr>
<td><code>TSQLRecord</code></td>
<td>int32</td>
<td><code>RowID</code> pointing to another record (warning: the field value
contains <code>pointer(RowID)</code>, not a valid object instance - the record
content must be retrieved with late-binding via its <code>ID</code> using a
<code>PtrInt(Field)</code> typecast or the <code>Field.ID</code> method), or by
using e.g. <code>CreateJoined()</code></td>
</tr>
<tr>
<td><code>TSQLRecordMany</code></td>
<td>nothing</td>
<td>data is stored in a separate <em>pivot</em> table; for MongoDB, you should
better use <em>data sharding</em>, and an embedded sub-document</td>
</tr>
<tr>
<td><code>TRecordReference</code></td>
<td>int32</td>
<td>store both <code>ID</code> and <code>TSQLRecord</code> type in a
<code>RecordRef</code>-like value (use e.g. <code>TSQLRest.
Retrieve(Reference)</code> to get a record content)</td>
</tr>
<tr>
<td><code>TPersistent</code></td>
<td>object</td>
<td>BSON object (from <code>ObjectToJSON</code>)</td>
</tr>
<tr>
<td><code>TCollection</code></td>
<td>array</td>
<td>BSON array of objects (from <code>ObjectToJSON</code>)</td>
</tr>
<tr>
<td><code>TObjectList</code></td>
<td>array</td>
<td>BSON array of objects (from <code>ObjectToJSON</code>) - see
<code>TJSONSerializer. RegisterClassForJSON</code> <em>TObjectList
serialization</em></td>
</tr>
<tr>
<td><code>TStrings</code></td>
<td>array</td>
<td>BSON array of strings (from <code>ObjectToJSON</code>)</td>
</tr>
<tr>
<td><code>TRawUTF8List</code></td>
<td>array</td>
<td>BSON array of string (from <code>ObjectToJSON</code>)</td>
</tr>
<tr>
<td>any <code>TObject</code></td>
<td>object</td>
<td>See <code>TJSONSerializer. RegisterCustomSerializer</code> <em>TObject
serialization</em></td>
</tr>
<tr>
<td><code>TSQLRawBlob</code></td>
<td>binary</td>
<td>BSON binary for blob storage - note that you need to retrieve explicitly
such fields, which are not part of the default <code>Retrieve()</code> method;
this type is an alias to <code>RawByteString</code></td>
</tr>
<tr>
<td><em>dynamic arrays</em></td>
<td>array<br />
binary</td>
<td>if the dynamic array can be saved as true JSON, will be stored as BSON
array - otherwise, will be stored in the <code>TDynArray.SaveTo</code> binary
format - note that <code>TByteDynArray</code> will always be stored as BSON
binary, so that <code>TSQLRawBlob</code> may be reserved to store content on
<em>GridFS</em> in the future</td>
</tr>
<tr>
<td><code>variant</code></td>
<td>array<br />
object</td>
<td>BSON number, text, object or array, depending on <em>TDocVariant custom
variant type</em> or <code>TBSONVariant</code> stored value</td>
</tr>
<tr>
<td><code>record</code></td>
<td>binary<br />
object</td>
<td>BSON as defined in code by overriding
<code>TSQLRecord.InternalRegisterCustomProperties</code></td>
</tr>
</tbody>
</table>
<p>On the server side (there won't be any difference for the client), you
define a <code>TMongoDBClient</code>, and assign it to a given
<code>TSQLRecord</code> class:</p>
<pre>
MongoClient := TMongoClient.Create('localhost',27017);
DB := MongoClient.Database['dbname'];
Model := TSQLModel.Create([TSQLORM]);
Client := TSQLRestClientDB.Create(Model,<strong>nil</strong>,':memory:',TSQLRestServerDB);
<span style="background-color:yellow;"><strong>if</strong> StaticMongoDBRegister(TSQLORM,fClient.Server,fDB,'collectionname')=<strong>nil then</strong></span>
<strong>raise</strong> Exception.Create('Error');
</pre>
<p>And... that's all!</p>
<p>You can then use any ORM command, as usual:</p>
<pre>
writeln(Client.TableRowCount(TSQLORM)=0);
</pre>
<p>As with external databases, you can specify the field names mapping between
the objects and the <em>MongoDB</em> collection.<br />
By default, the <code>TSQLRecord.ID</code> property is mapped to the
<em>MongoDB</em>'s <code>_id</code> field, and the ORM will populate this
<code>_id</code> field with a sequence of integer values, just like any
<code>TSQLRecord</code> table.<br />
You can specify your own mapping, using for instance:</p>
<pre>
aModel.Props[aClass].ExternalDB.MapField(..)
</pre>
<p>Since the field names are stored within the document itself, it may be a
good idea to use shorter naming for the <em>MongoDB</em> collection. It may
save some storage space, when working with a huge number of documents.</p>
<p>Once the <code>TSQLRecord</code> is mapped to a <em>MongoDB</em> collection,
you can always have direct access to the <code>TMongoCollection</code> instance
later on, by calling:</p>
<pre>
(aServer.StaticDataServer[aClass] <strong>as</strong> TSQLRestServerStaticMongoDB).Collection
</pre>
<p>This may allow any specific task, including any tuned query or process.</p>
<h3>ORM/ODM CRUD methods</h3>
<p>You can add documents with the standard CRUD methods of the ORM, as
usual:</p>
<pre>
R := TSQLORM.Create;
<strong>try</strong>
<strong>for</strong> i := 1 <strong>to</strong> COLL_COUNT <strong>do begin</strong>
R.Name := 'Name '+Int32ToUTF8(i);
R.Age := i;
R.Date := 1.0*(30000+i);
R.Value := _ObjFast(['num',i]);
R.Ints := <strong>nil</strong>;
R.DynArray(1).Add(i);
<span style="background-color:yellow;">assert(Client.Add(R,True)=i);</span>
<strong>end</strong>;
<strong>finally</strong>
R.Free;
<strong>end</strong>;
</pre>
<p>As we already saw, the framework is able to handle any kind of properties,
including complex types like <em>dynamic arrays</em> or
<code>variant</code>.<br />
In the above code, a <code>TDocVariant</code> document has been stored in
<code>R.Value</code>, and a dynamic array of <code>integer</code> values is
accessed via its <code>index 1</code> shortcut and the
<code>TSQLRecord.DynArray()</code> method.</p>
<p>The usual <code>Retrieve</code> / <code>Delete</code> / <code>Update</code>
methods are available:</p>
<pre>
R := TSQLORM.Create;
<strong>try</strong>
<strong>for</strong> i := 1 <strong>to</strong> COLL_COUNT <strong>do begin</strong>
<span style="background-color:yellow;">Check(Client.Retrieve(i,R));</span>
<em>// here R instance contains all values of one document, excluding BLOBs</em>
<strong>end</strong>;
<strong>finally</strong>
R.Free;
<strong>end</strong>;
</pre>
<p>You can define a WHERE clause, as if the back-end where a regular SQL
database:</p>
<pre>
R := TSQLORM.CreateAndFillPrepare(Client,'ID=?',[i]);
<strong>try</strong>
...
</pre>
<p>Current implementation understand one condition over one single field, with
<code>= > >= < <= IN</code> clauses. More advanced queries are
possible, but won't be handled as SQL, but via direct access to the
<code>TMongoDBCollection</code>.</p>
<p>To perform a query and retrieve the content of several documents, you can
use regular <code>CreateAndFillPrepare</code> or <code>FillPrepare</code>
methods:</p>
<pre>
<span style="background-color:yellow;">R := TSQLORM.CreateAndFillPrepare(Client,'');</span>
<strong>try</strong>
n := 0;
<span style="background-color:yellow;"><strong>while</strong> R.FillOne <strong>do begin</strong></span>
<em>// here R instance contains all values of one document, excluding BLOBs</em>
inc(n);
<strong>end</strong>;
assert(n=COLL_COUNT);
<strong>finally</strong>
R.Free;
<strong>end</strong>;
</pre>
<p>A WHERE clause can also be defined for <code>CreateAndFillPrepare</code> or
<code>FillPrepare</code> methods.</p>
<h3>BATCH mode</h3>
<p>In addition to individual CRUD operations, our <em>MongoDB</em> is able to
use BATCH mode for adding or deleting documents.</p>
<p>You can write the exact same code as with any SQL back-end:</p>
<pre>
Client.BatchStart(TSQLORM);
R := TSQLORM.Create;
<strong>try</strong>
<strong>for</strong> i := 1 <strong>to</strong> COLL_COUNT <strong>do begin</strong>
R.Name := 'Name '+Int32ToUTF8(i);
R.Age := i;
R.Date := 1.0*(30000+i);
R.Value := _ObjFast(['num',i]);
R.Ints := <strong>nil</strong>;
R.DynArray(1).Add(i);
assert(Client.BatchAdd(R,True)>=0);
<strong>end</strong>;
<strong>finally</strong>
R.Free;
<strong>end</strong>;
assert(Client.BatchSend(IDs)=HTML_SUCCESS);
</pre>
<p>Or for deletion:</p>
<pre>
Client.BatchStart(TSQLORM);
<strong>for</strong> i := 5 <strong>to</strong> COLL_COUNT <strong>do</strong>
<strong>if</strong> i <strong>mod</strong> 5=0 <strong>then</strong>
assert(fClient.BatchDelete(i)>=0);
assert(Client.BatchSend(IDs)=HTML_SUCCESS);
</pre>
<p>Speed benefit may be huge in regard to individual Add/Delete operations,
even on a local <em>MongoDB</em> server. We will see some benchmark numbers
now.</p>
<h3>ORM/ODM performance</h3>
<p>You can take a look at <em><a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-benchmark">Data access benchmark</a></em> to
compare <em>MongoDB</em> as back-end for our ORM classes.</p>
<p>In respect to external SQL engines, it features very high speed, low CPU
use, and almost no difference in use. We interfaced the <code>BatchAdd()</code>
and <code>BatchDelete()</code> methods to benefit of <em>MongoDB</em> BULK
process, and avoided most memory allocation during the process.</p>
<p>Here are some numbers, extracted from the <code>MongoDBTests.dpr</code>
sample, which reflects the performance of our ORM/ODM, depending on the
<em>Write Concern</em> mode used:</p>
<pre>
2. ORM
<br />2.1. ORM with acknowledge:
- Connect to local server: 6 assertions passed 18.65ms
- Insert: 5,002 assertions passed 521.25ms
<span style="background-color:yellow;"> 5000 rows inserted in 520.65ms i.e. 9603/s, aver. 104us, 2.9 MB/s</span>
- Insert in batch mode: 5,004 assertions passed 65.37ms
<span style="background-color:yellow;"> 5000 rows inserted in 65.07ms i.e. 76836/s, aver. 13us, 8.4 MB/s</span>
- Retrieve: 45,001 assertions passed 640.95ms
5000 rows retrieved in 640.75ms i.e. 7803/s, aver. 128us, 2.1 MB/s
- Retrieve all: 40,001 assertions passed 20.79ms
<span style="background-color:yellow;"> 5000 rows retrieved in 20.33ms i.e. 245941/s, aver. 4us, 27.1 MB/s</span>
- Retrieve one with where clause: 45,410 assertions passed 673.01ms
5000 rows retrieved in 667.17ms i.e. 7494/s, aver. 133us, 2.0 MB/s
- Update: 40,002 assertions passed 681.31ms
5000 rows updated in 660.85ms i.e. 7565/s, aver. 132us, 2.4 MB/s
- Blobs: 125,003 assertions passed 2.16s
5000 rows updated in 525.97ms i.e. 9506/s, aver. 105us, 2.4 MB/s
- Delete: 38,003 assertions passed 175.86ms
1000 rows deleted in 91.37ms i.e. 10944/s, aver. 91us, 2.3 MB/s
- Delete in batch mode: 33,003 assertions passed 34.71ms
<span style="background-color:yellow;"> 1000 rows deleted in 14.90ms i.e. 67078/s, aver. 14us, 597 KB/s</span>
Total failed: 0 / 376,435 - ORM with acknowledge PASSED 5.00s
<br />2.2. ORM without acknowledge:
- Connect to local server: 6 assertions passed 16.83ms
- Insert: 5,002 assertions passed 179.79ms
<span style="background-color:yellow;"> 5000 rows inserted in 179.15ms i.e. 27908/s, aver. 35us, 3.9 MB/s</span>
- Insert in batch mode: 5,004 assertions passed 66.30ms
<span style="background-color:yellow;"> 5000 rows inserted in 31.46ms i.e. 158891/s, aver. 6us, 17.5 MB/s</span>
- Retrieve: 45,001 assertions passed 642.05ms
5000 rows retrieved in 641.85ms i.e. 7789/s, aver. 128us, 2.1 MB/s
- Retrieve all: 40,001 assertions passed 20.68ms
<span style="background-color:yellow;"> 5000 rows retrieved in 20.26ms i.e. 246718/s, aver. 4us, 27.2 MB/s</span>
- Retrieve one with where clause: 45,410 assertions passed 680.99ms
5000 rows retrieved in 675.24ms i.e. 7404/s, aver. 135us, 2.0 MB/s
- Update: 40,002 assertions passed 231.75ms
5000 rows updated in 193.74ms i.e. 25807/s, aver. 38us, 3.6 MB/s
- Blobs: 125,003 assertions passed 1.44s
5000 rows updated in 150.58ms i.e. 33202/s, aver. 30us, 2.6 MB/s
- Delete: 38,003 assertions passed 103.57ms
1000 rows deleted in 19.73ms i.e. 50668/s, aver. 19us, 2.4 MB/s
- Delete in batch mode: 33,003 assertions passed 47.50ms
<span style="background-color:yellow;"> 1000 rows deleted in 364us i.e. 2747252/s, aver. 0us, 23.4 MB/s</span>
Total failed: 0 / 376,435 - ORM without acknowledge PASSED 3.44s
</pre>
<p>As for direct <em>MongoDB</em> access, the <code>wcUnacknowledged</code> is
not to be used on production, but may be very useful in some particular
scenarios.<br />
As expected, the reading process is not impacted by the <em>Write Concern</em>
mode set.</p>
<p>You can take a look at the previous blog article, about <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-database-access">low-level <em>MongoDB</em> direct
access</a>.<br />
Or check out <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-benchmark">more complete
benchmark results</a>, to compare <em>MongoDB</em> with other SQL back-end
handled by our ORM.<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1742">welcome
on our forum</a>, as usual!</p>Direct MongoDB database accessurn:md5:783573bb416f11adc20f4cffa50477cd2014-05-07T09:10:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkACIDAJAXBatchblogDatabaseDelphiDocumentationDomainDrivenGoodPracticeJSONmappingModelMongoDBmORMotNoSQLOpenSourceORMperformanceRestshardingSourceSQLtransactionUnicode<p><em><a href="http://www.mongodb.org/">MongoDB</a></em> (from "humongous") is
a cross-platform document-oriented database system, and certainly the best
known NoSQL database.<br />
According to <a href="http://db-engines.com">http://db-engines.com</a> in April
2014, <em>MongoDB</em> is in 5th place of the most popular types of database
management systems, and first place for NoSQL database management
systems.<br />
Our <em>mORMot</em> framework gives premium access to this database,
featuring full <em>NoSQL and Object-Document Mapping (ODM)</em> abilities to
the framework.</p>
<p><img src="http://upload.wikimedia.org/wikipedia/en/thumb/e/eb/MongoDB_Logo.png/320px-MongoDB_Logo.png" alt="" /></p>
<p>Integration is made at two levels:</p>
<ul>
<li>Direct low-level access to the <em>MongoDB</em> server, in the
<code>SynMongoDB.pas</code> unit;</li>
<li>Close integration with our ORM (which becomes <em>defacto</em> an ODM), in
the <code>mORMotMongoDB.pas</code> unit.</li>
</ul>
<p><em>MongoDB</em> eschews the traditional table-based relational database
structure in favor of JSON-like documents with dynamic schemas
(<em>MongoDB</em> calls the format BSON), which matches perfectly
<em>mORMot</em>'s RESTful approach.</p>
<p>In this first article, we will detail direct low-level access to the
<em>MongoDB</em> server, via the <code>SynMongoDB.pas</code> unit.</p> <h3>MongoDB client</h3>
<p>The <code>SynMongoDB.pas</code> unit features direct optimized access to a
<em>MongoDB</em> server.</p>
<p>It gives access to any BSON data, including documents, arrays, and
<em>MongoDB</em>'s custom types (like ObjectID, dates, binary, regex or
Javascript):</p>
<ul>
<li>For instance, a <code>TBSONObjectID</code> can be used to create some
genuine document identifiers on the client side (<em>MongoDB</em> does not
generate the IDs for you: a common way is to generate unique IDs on the client
side);</li>
<li>Generation of BSON content from any Delphi types (via
<code>TBSONWriter</code>);</li>
<li>Fast in-place parsing of the BSON stream, without any memory allocation
(via <code>TBSONElement</code>);</li>
<li>A <code>TBSONVariant</code> custom variant type, to store
<em>MongoDB</em>'s custom type values;</li>
<li>Interaction with the <code>SynCommons</code>' <em>TDocVariant custom
variant type</em> as document storage and late-binding access;</li>
<li>Marshalling BSON to and from JSON, with the <em>MongoDB</em> extended
syntax for handling its custom types.</li>
</ul>
<p>This unit defines some objects able to connect and manage databases and
collections of documents on any <em>MongoDB</em> servers farm:</p>
<ul>
<li>Connection to one or several servers, including secondary hosts, via the
<code>TMongoClient</code> class;</li>
<li>Access to any database instance, via the <code>TMongoDatabase</code>
class;</li>
<li>Access to any collection, via the <code>TMongoCollection</code> class;</li>
<li>It features some nice abilities about speed, like BULK insert or delete
mode, and explicit <em>Write Concern</em> settings.</li>
</ul>
<p>At collection level, you can have direct access to the data, with high level
structures like <code>TDocVariant</code>/<code>TBSONVariant</code>, with
easy-to-read JSON, or low level BSON content.<br />
You can also tune most aspects of the client process, e.g. about error handling
or <em>write concerns</em> (i.e. how remote data modifications are
acknowledged).</p>
<h3>Connecting to a server</h3>
<p>Here is some sample code, which is able to connect to a <em>MongoDB</em>
server, and returns the server time:</p>
<pre>
<strong>var</strong> Client: TMongoClient;
DB: TMongoDatabase;
serverTime: TDateTime;
res: <strong>variant</strong>; <em>// we will return the command result as TDocVariant</em>
errmsg: RawUTF8;
<strong>begin</strong>
Client := TMongoClient.Create('localhost',27017);
<strong>try</strong>
DB := Client.Database['mydb'];
writeln('Connecting to ',DB.Name); <em>// will write 'mydb'</em>
errmsg := DB.RunCommand('hostInfo',res); <em>// run a command</em>
<strong>if</strong> errmsg<>'' <strong>then</strong>
exit; <em>// quit on any error</em>
serverTime := res.system.currentTime; <em>// direct conversion to TDateTime</em>
writeln('Server time is ',DateTimeToStr(serverTime));
<strong>finally</strong>
Client.Free; <em>// will release the DB instance</em>
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>Note that for this low-level command, we used a <code>TDocVariant</code>,
and its late-binding abilities.</p>
<p>In fact, if you put your mouse over the <code>res</code> variable during
debugging, you will see the following JSON content:</p>
<pre>
{"system":{"currentTime":"2014-05-06T15:24:25","hostname":"Acer","cpuAddrSize":64,"memSizeMB":3934,"numCores":4,"cpuArch":"x86_64","numaEnabled":false},"os":{"type":"Windows","name":"Microsoft Windows 7","version":"6.1 SP1 (build 7601)"},"extra":{"pageSize":4096},"ok":1}
</pre>
<p>And we simply access to the server time by writing
<code>res.system.currentTime</code>.</p>
<h3>Adding some documents to the collection</h3>
<p>We will now explain how to add documents to a given collection.</p>
<p>We assume that we have a <code>DB: TMongoDatabase</code> instance available.
Then we will create the documents with a <code>TDocVariant</code> instance,
which will be filled via late-binding, and via a <code>doc.Clear</code>
pseudo-method used to flush any previous property value:</p>
<pre>
<strong>var</strong> Coll: TMongoCollection;
doc: <strong>variant</strong>;
i: integer;
<strong>begin</strong>
Coll := DB.CollectionOrCreate[COLL_NAME];
TDocVariant.New(doc);
<strong>for</strong> i := 1 <strong>to</strong> 10 <strong>do</strong>
<strong>begin</strong>
doc.Clear;
doc.Name := 'Name '+IntToStr(i+1);
doc.Number := i;
<span style="background-color:yellow;">Coll.Save(doc);</span>
writeln('Inserted with _id=',doc._id);
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>Thanks to <code>TDocVariant</code> late-binding abilities, code is pretty
easy to understand and maintain.</p>
<p>This code will display the following on the console:</p>
<pre>
Inserted with _id=5369029E4F901EE8114799D9
Inserted with _id=5369029E4F901EE8114799DA
Inserted with _id=5369029E4F901EE8114799DB
Inserted with _id=5369029E4F901EE8114799DC
Inserted with _id=5369029E4F901EE8114799DD
Inserted with _id=5369029E4F901EE8114799DE
Inserted with _id=5369029E4F901EE8114799DF
Inserted with _id=5369029E4F901EE8114799E0
Inserted with _id=5369029E4F901EE8114799E1
Inserted with _id=5369029E4F901EE8114799E2
</pre>
<p>It means that the <code>Coll.Save()</code> method was clever enough to
understand that the supplied document does not have any <code>_id</code> field,
so will compute one on the client side before sending the document data to the
<em>MongoDB</em> server.</p>
<p>We may have written:</p>
<pre>
<strong>for</strong> i := 1 <strong>to</strong> 10 <strong>do</strong>
<strong>begin</strong>
doc.Clear;
<span style="background-color:yellow;">doc._id := ObjectID;</span>
doc.Name := 'Name '+IntToStr(i+1);
doc.Number := i;
Coll.Save(doc);
writeln('Inserted with _id=',doc._id);
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>Which will compute the document identifier explicitly before calling
<code>Coll.Save()</code>.<br />
In this case, we may have called directly <code>Coll.Insert()</code>, which is
somewhat faster.</p>
<p>Note that you are not obliged to use a <em>MongoDB</em> ObjectID as
identifier. You can use any value, if you are sure that it will be genuine. For
instance, you can use an integer:</p>
<pre>
<strong>for</strong> i := 1 <strong>to</strong> 10 <strong>do</strong>
<strong>begin</strong>
doc.Clear;
<span style="background-color:yellow;">doc._id := i;</span>
doc.Name := 'Name '+IntToStr(i+1);
doc.Number := i;
<span style="background-color:yellow;">Coll.Insert(doc);</span>
writeln('Inserted with _id=',doc._id);
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>The console will display now:</p>
<pre>
Inserted with _id=1
Inserted with _id=2
Inserted with _id=3
Inserted with _id=4
Inserted with _id=5
Inserted with _id=6
Inserted with _id=7
Inserted with _id=8
Inserted with _id=9
Inserted with _id=10
</pre>
<p>Note that the <em>mORMot</em> ORM will compute a genuine series of integers
in a similar way, which will be used as expected by the
<code>TSQLRecord.ID</code> primary key property.</p>
<p>The <code>TMongoCollection</code> class can also write a list of documents,
and send them at once to the <em>MongoDB</em> server: this BULK insert mode -
close to the <em>Array Binding</em> feature of some SQL providers, and
implemented in our <em>SynDB</em> classes - see <em>BATCH sequences for
adding/updating/deleting records</em> - can increase the insertion by a factor
of 10 times, even when connected to a local instance: imagine how much time it
may save over a physical network!</p>
<p>For instance, you may write:</p>
<pre>
<strong>var</strong> docs: TVariantDynArray;
...
SetLength(docs,COLL_COUNT);
<strong>for</strong> i := 0 <strong>to</strong> COLL_COUNT-1 <strong>do begin</strong>
TDocVariant.New(docs[i]);
docs[i]._id := ObjectID; <em>// compute new ObjectID on the client side</em>
docs[i].Name := 'Name '+IntToStr(i+1);
docs[i].FirstName := 'FirstName '+IntToStr(i+COLL_COUNT);
docs[i].Number := i;
<strong>end</strong>;
Coll.Insert(docs); <em>// insert all values at once</em>
...
</pre>
<p>You will find out later for some numbers about the speed increase due to
such BULK insert.</p>
<h3>Retrieving the documents</h3>
<p>You can retrieve the document as a <code>TDocVariant</code> instance:</p>
<pre>
<strong>var</strong> doc: <strong>variant</strong>;
...
doc := Coll.FindOne(5);
writeln('Name: ',doc.Name);
writeln('Number: ',doc.Number);
</pre>
<p>Which will write on the console:</p>
<pre>
Name: Name 6
Number: 5
</pre>
<p>You have access to the whole <code>Query</code> parameter, if needed:</p>
<pre>
doc := Coll.FindDoc('{_id:?}',[5]);
doc := Coll.FindOne(5); <em>// same as previous</em>
</pre>
<p>This <code>Query</code> filter is similar to a WHERE clause in SQL. You can
write complex search patterns, if needed - see <a href="http://docs.mongodb.org/manual/reference/method/db.collection.find">http://docs.mongodb.org/manual/reference/method/db.collection.find</a>
for reference.</p>
<p>You can retrieve a list of documents, as a dynamic array of
<code>TDocVariant</code>:</p>
<pre>
<strong>var</strong> docs: TVariantDynArray;
...
Coll.FindDocs(docs);
<strong>for</strong> i := 0 <strong>to</strong> high(docs) <strong>do</strong>
writeln('Name: ',docs[i].Name,' Number: ',docs[i].Number);
</pre>
<p>Which will output:</p>
<pre>
Name: Name 2 Number: 1
Name: Name 3 Number: 2
Name: Name 4 Number: 3
Name: Name 5 Number: 4
Name: Name 6 Number: 5
Name: Name 7 Number: 6
Name: Name 8 Number: 7
Name: Name 9 Number: 8
Name: Name 10 Number: 9
Name: Name 11 Number: 10
</pre>
<p>If you want to retrieve the documents directly as JSON, we can write:</p>
<pre>
<strong>var</strong> json: RawUTF8;
...
json := Coll.FindJSON(null,null);
writeln(json);
...
</pre>
<p>This will append the following to the console:</p>
<pre>
[{"_id":1,"Name":"Name 2","Number":1},{"_id":2,"Name":"Name 3","Number":2},{"_id
":3,"Name":"Name 4","Number":3},{"_id":4,"Name":"Name 5","Number":4},{"_id":5,"N
ame":"Name 6","Number":5},{"_id":6,"Name":"Name 7","Number":6},{"_id":7,"Name":"
Name 8","Number":7},{"_id":8,"Name":"Name 9","Number":8},{"_id":9,"Name":"Name 1
0","Number":9},{"_id":10,"Name":"Name 11","Number":10}]
</pre>
<p>You can note that <code>FindJSON()</code> has two properties, which are the
<code>Query</code> filter, and a <code>Projection</code> mapping (similar to
the column names in a <code>SELECT col1,col2</code>).<br />
So we may have written:</p>
<pre>
json := Coll.FindJSON('{_id:?}',[5]);
writeln(json);
</pre>
<p>Which would output:</p>
<pre>
[{"_id":5,"Name":"Name 6","Number":5}]
</pre>
<p>Note here than we used an overloaded <code>FindJSON()</code> method, which
accept the <em>MongoDB</em> extended syntax (here, the field name is unquoted),
and parameters as variables.</p>
<p>We can specify a projection:</p>
<pre>
json := Coll.FindJSON('{_id:?}',[5],'{Name:1}');
writeln(json);
</pre>
<p>Which will only return the "Name" and "_id" fields (since <code>_id</code>
is, by <em>MongoDB</em> convention, always returned:</p>
<pre>
[{"_id":5,"Name":"Name 6"}]
</pre>
<p>To return only the "Name" field, you can specify <code>'_id:0,Name:1'</code>
as extended JSON for the <em>projection</em> parameter.</p>
<pre>
[{"Name":"Name 6"}]
</pre>
<p>There are other methods able to retrieve data, also directly as BSON binary
data. They will be used for best speed e.g. in conjunction with our ORM, but
for most end-user code, using <code>TDocVariant</code> is safer and easier to
maintain.</p>
<h3>Updating or deleting documents</h3>
<p>The <code>TMongoCollection</code> class has some methods dedicated to alter
existing documents.</p>
<p>At first, the <code>Save()</code> method can be used to update a document
which has been first retrieved:</p>
<pre>
doc := Coll.FindOne(5);
doc.Name := 'New!';
Coll.Save(doc);
writeln('Name: ',Coll.FindOne(5).Name);
</pre>
<p>Which will write:</p>
<pre>
Name: New!
</pre>
<p>Note that we used here an integer value (5) as key, but we may use an
<em>ObjectID</em> instead, if needed.</p>
<p>The <code>Coll.Save()</code> method could be changed into
<code>Coll.Update()</code>, which expects an explicit <code>Query</code>
operator, in addition to the updated document content:</p>
<pre>
doc := Coll.FindOne(5);
doc.Name := 'New!';
Coll.Update(BSONVariant(['_id',5]),doc);
writeln('Name: ',Coll.FindOne(5).Name);
</pre>
<p>Note that by <em>MongoDB</em>'s design, any call to <code>Update()</code>
will <em>replace</em> the whole document.</p>
<p>For instance, if you write:</p>
<pre>
writeln('Before: ',Coll.FindOne(3));
Coll.Update('{_id:?}',[3],'{Name:?}',['New Name!']);
writeln('After: ',Coll.FindOne(3));
</pre>
<p>Then the <code>Number</code> field will disappear!</p>
<pre>
Before: {"_id":3,"Name":"Name 4","Number":3}
After: {"_id":3,"Name":"New Name!"}
</pre>
<p>If you need to update only some fields, you will have to use the
<code>$set</code> modifier:</p>
<pre>
writeln('Before: ',Coll.FindOne(4));
Coll.Update('{_id:?}',[4],'{$set:{Name:?}}',['New Name!']);
writeln('After: ',Coll.FindOne(4));
</pre>
<p>Which will write on the console the value as expected:</p>
<pre>
Before: {"_id":4,"Name":"Name 5","Number":4}
After: {"_id":4,"Name":"New Name!","Number":4}
</pre>
<p>Now the <code>Number</code> field remains untouched.</p>
<p>You can also use the <code>Coll.UpdateOne()</code> method, which will update
the supplied fields, and leave the non specified fields untouched:</p>
<pre>
writeln('Before: ',Coll.FindOne(2));
Coll.UpdateOne(2,_Obj(['Name','NEW']));
writeln('After: ',Coll.FindOne(2));
</pre>
<p>Which will output as expected:</p>
<pre>
Before: {"_id":2,"Name":"Name 3","Number":2}
After: {"_id":2,"Name":"NEW","Number":2}
</pre>
<p>You can refer to the documentation of the <code>SynMongoDB.pas</code> unit,
to find out all functions, classes and methods available to work with
<em>MongoDB</em>.</p>
<h3>Write Concern and Performance</h3>
<p>You can take a look at the <code>MongoDBTests.dpr</code> sample - located in
the <code>SQLite3- MongoDB</code> sub-folder of the source code repository, and
the <code>TTestDirect</code> classes, to find out some performance
information.</p>
<p>In fact, this <code>TTestDirect</code> is inherited twice, to run the same
tests with diverse write concern.</p>
<p>The difference between the two classes will take place at client
initialization:</p>
<pre>
<strong>procedure</strong> TTestDirect.ConnectToLocalServer;
...
fClient := TMongoClient.Create('localhost',27017);
<strong>if</strong> ClassType=TTestDirectWithAcknowledge <strong>then</strong>
<span style="background-color:yellow;">fClient.WriteConcern := wcAcknowledged <strong>else</strong></span>
<strong>if</strong> ClassType=TTestDirectWithoutAcknowledge <strong>then</strong>
<span style="background-color:yellow;">fClient.WriteConcern := wcUnacknowledged;</span>
...
</pre>
<p><code>wcAcknowledged</code> is the default safe mode: the <em>MongoDB</em>
server confirms the receipt of the write operation. Acknowledged write concern
allows clients to catch network, duplicate key, and other errors. But it adds
an additional round-trip from the client to the server, and wait for the
command to be finished before returning the error status: so it will slow down
the write process.</p>
<p>With <code>wcUnacknowledged</code>, <em>MongoDB</em> does not acknowledge
the receipt of write operation. Unacknowledged is similar to errors ignored;
however, drivers attempt to receive and handle network errors when possible.
The driver's ability to detect network errors depends on the system's
networking configuration.</p>
<p>The speed difference between the two is worth mentioning, as stated by the
regression tests status, running on a local <em>MongoDB</em> instance:</p>
<pre>
1. Direct access
<br />1.1. Direct with acknowledge:
- Connect to local server: 6 assertions passed 4.72ms
- Drop and prepare collection: 8 assertions passed 9.38ms
<span style="background-color:yellow;"> - Fill collection: 15,003 assertions passed 558.79ms</span>
<span style="background-color:yellow;"> 5000 rows inserted in 548.83ms i.e. 9110/s, aver. 109us, 3.1 MB/s</span>
- Drop collection: no assertion 856us
<span style="background-color:yellow;"> - Fill collection bulk: 2 assertions passed 74.59ms</span>
<span style="background-color:yellow;"> 5000 rows inserted in 64.76ms i.e. 77204/s, aver. 12us, 7.2 MB/s</span>
- Read collection: 30,003 assertions passed 2.75s
5000 rows read at once in 9.66ms i.e. 517330/s, aver. 1us, 39.8 MB/s
<span style="background-color:yellow;"> - Update collection: 7,503 assertions passed 784.26ms</span>
<span style="background-color:yellow;"> 5000 rows updated in 435.30ms i.e. 11486/s, aver. 87us, 3.7 MB/s</span>
<span style="background-color:yellow;"> - Delete some items: 4,002 assertions passed 370.57ms</span>
<span style="background-color:yellow;"> 1000 rows deleted in 96.76ms i.e. 10334/s, aver. 96us, 2.2 MB/s</span>
Total failed: 0 / 56,527 - Direct with acknowledge PASSED 4.56s
<br />1.2. Direct without acknowledge:
- Connect to local server: 6 assertions passed 1.30ms
- Drop and prepare collection: 8 assertions passed 8.59ms
<span style="background-color:yellow;"> - Fill collection: 15,003 assertions passed 192.59ms</span>
<span style="background-color:yellow;"> 5000 rows inserted in 168.50ms i.e. 29673/s, aver. 33us, 4.4 MB/s</span>
- Drop collection: no assertion 845us
<span style="background-color:yellow;"> - Fill collection bulk: 2 assertions passed 68.54ms</span>
<span style="background-color:yellow;"> 5000 rows inserted in 58.67ms i.e. 85215/s, aver. 11us, 7.9 MB/s</span>
- Read collection: 30,003 assertions passed 2.75s
5000 rows read at once in 9.99ms i.e. 500150/s, aver. 1us, 38.5 MB/s
<span style="background-color:yellow;"> - Update collection: 7,503 assertions passed 446.48ms</span>
<span style="background-color:yellow;"> 5000 rows updated in 96.27ms i.e. 51933/s, aver. 19us, 7.7 MB/s</span>
<span style="background-color:yellow;"> - Delete some items: 4,002 assertions passed 297.26ms</span>
<span style="background-color:yellow;"> 1000 rows deleted in 19.16ms i.e. 52186/s, aver. 19us, 2.8 MB/s</span>
Total failed: 0 / 56,527 - Direct without acknowledge PASSED 3.77s
</pre>
<p>As you can see, the reading speed is not affected by the <em>Write
Concern</em> settings.<br />
But data writing can be multiple times faster, when each write command is not
acknowledged.</p>
<p>Since there is no error handling, <code>wcUnacknowledged</code> is not to be
used on production. You may use it for replication, or for data consolidation,
e.g. feeding a database with a lot of existing data as fast as possible.</p>
<p>Stay tuned for the next article, which <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">will detail the <em>MongoDB</em>
integration 's ORM</a>... and the last one of the series, with <a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-benchmark">benchmarks of <em>MongoDB</em>
against SQL engines, via our ORM</a>...<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1742">welcome
on our forum</a>, as usual!</p>Introducing mORMot's architecture and design principlesurn:md5:265b0a5927ad96f526d355c5c9aca5322014-04-18T11:40:00+02:002015-01-04T10:10:34+01:00AB4327-GANDImORMot Frameworkcode-firstDatabasedatabase-firstDelphiDependencyInjectionDocumentationDomainDrivenDTOdynamic arrayEventSourcingfactoryFireDACGoodPracticehttp.sysHttpApiHTTPSinterfaceJSONLateBindingmockModelmORMotMSSQLMySQLNextGenNexusDBNoSQLODBCOleDBOpenSourceOracleORMperformancePostgreSQLRepositoryRTTIsessionshardingSOASQLSQLite3SynDBSynopseTDataSetTDocVariantTDynArraytransactionUnicodeUniDACValueObjectVirtualTableWinHTTPWinINetZEOS <p>We have just released a set of slides introducing </p>
<ul>
<li>ORM, SOA, REST, JSON, MVC, MVVM, SOLID, Mocks/Stubs, Domain-Driven Design
concepts with Delphi, </li>
<li>and showing some sample code using our Open Source <em>mORMot</em>
framework.</li>
</ul>
<p>You can follow the <a href="https://drive.google.com/folderview?id=0B0r8u-FwvxWdeVJVZnBhSEpKYkE&usp=sharing">
public link on Google Drive</a>!</p>
<p><img src="http://images.fineartamerica.com/images-medium-large/1-golden-marmot-maureen-ida-farley.jpg" width="450" height="291" alt="" /></p>
<p>This is a great opportunity to discovers some patterns you may not be
familiar with, and find out how <em>mORMot</em> try to implement them.<br />
This set of slides may be less intimidating than our huge documentation - do
not be terrified by our <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html">
Online Documentation</a>!<br />
The <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#SIDE_TITL_40">
first set of pages</a> (presenting architecture and design principles) is worth
reading.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1707">welcome on our forum, as
usual</a>.</p>Are NoSQL databases ACID?urn:md5:4d2d66071fc63c08ed1794a98d4960fe2014-02-28T22:12:00+01:002014-03-04T09:48:00+01:00AB4327-GANDImORMot FrameworkACIDblogDatabaseDelphiDomainDrivenEntityObjectModelMongoDBmORMotNoSQLobjectORMSourceSQLtransaction<p>One of the main features you may miss when discovering NoSQL ("Not-Only
SQL"?) databases, coming from a RDBMS background, is <a href="http://en.wikipedia.org/wiki/ACID">ACID</a>.</p>
<blockquote>
<p>ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties
that guarantee that database transactions are processed reliably. In the
context of databases, a single logical operation on the data is called a
transaction. For example, a transfer of funds from one bank account to another,
even involving multiple changes such as debiting one account and crediting
another, is a single transaction. (Wikipedia)</p>
</blockquote>
<p>But <a href="http://stackoverflow.com/a/22105320/458259">are there any ACID
NoSQL database</a>?</p>
<p>Please ensure you read <a href="http://martinfowler.com/articles/nosqlKeyPoints.html">the Martin Fowler
introduction about NoSQL databases</a>.<br />
And <a href="https://www.youtube.com/watch?v=qI_g07C_Q5I">the corresponding
video</a>.</p>
<p><img src="http://1.bp.blogspot.com/-TEe81HD9tkQ/Ub5eLpEwBSI/AAAAAAAABNY/52SO5d04rAI/s500/nosql-ecosystem.png" alt="" /></p>
<p>First of all, we can distinguish two types of NoSQL databases:</p>
<ol>
<li>Aggregate-oriented databases;</li>
<li>Graph-oriented databases (e.g. Neo4J).</li>
</ol>
<p>By design, <em>most Graph-oriented databases are ACID</em>!<br />
This is a first good point.</p>
<p>Then, what about the other type?<br />
In Aggregate-oriented databases, we can identify three sub-types:</p>
<ul>
<li>Document-based NoSQL databases (e.g. <a href="https://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CCwQFjAA&url=http%3A%2F%2Fwww.mongodb.org%2F&ei=pPkQU4vnD6Wb0QWdwoHIDA&usg=AFQjCNFkupTrrD2YYX6ydM2wb9IbWll3Vw&sig2=JMyiz8g0GjSeVt1uJeXhNw&bvm=bv.61965928,d.d2k">
MongoDB</a>, <a href="http://couchdb.apache.org/">CouchDB</a>);</li>
<li>Key/Value NoSQL databases (e.g. <a href="http://redis.io/">Redis</a>);</li>
<li>Column family NoSQL databases (e.g. <a href="http://cassandra.apache.org/">Cassandra</a>).</li>
</ul>
<div>Whatever document/key/column oriented they are, they all use some kind of
document storage.<br />
It may be schema-less, blob-stored, column-driven, but it is always some set of
values bound together to be persisted.<br />
This set of values define a particular state of one entity, in a given
model.<br />
Which we may call <em>Aggregate</em>.</div> <p>What we call an <em>Aggregate</em> here, is what <a href="https://blog.synopse.info?post/post/2014/01/04/Domain-Driven-Design%3A-part-3">Eric Evans defined in its
Domain-Driven Design</a> as a self-sufficient of <em>Entities</em> and
<em>Value-Objects</em> in a given <em>Bounded Context</em>.</p>
<blockquote>
<p>As a consequence, an aggregate is a collection of data that we interact with
as a unit.<br />
Aggregates form the boundaries for ACID operations with the database.<br />
(Martin Fowler)</p>
</blockquote>
<p>So, <em>at Aggregate level, we can say that most NoSQL databases can be as
safe as ACID RDBMS</em>, with the proper settings.<br />
Of source, if you tune your server for the best speed, you may come into
something non ACID. But replication will help.</p>
<p>My main point is that you have to use NoSQL databases as they are, not as a
(cheap) alternative to RDBMS.<br />
I have seen too much projects abusing of relations between documents. This
can't be ACID.<br />
If you stay at document level, i.e. at Aggregate boundaries, you do not need
any transaction.<br />
And your data will be as safe as with an ACID database, even if it not truly
ACID, since you do not need those transactions!</p>
<p>If you need transactions and update several "documents" at once, you are not
in the NoSQL world any more - so use a RDBMS engine instead!</p>
<p>We are currently cooking a native direct <em>MongoDB</em> access in our
labs.<br />
See our <a href="http://synopse.info/fossil/finfo?name=SynMongoDB.pas">SynMongoDB.pas
unit</a>...<br />
Still work to do, but I suspect this will be another unit feature of
<em>mORMot</em>, when the corresponding <code>mORMotMongoDB.pas</code> unit
will achieve one highly-optimized bridge between our <a href="https://blog.synopse.info?post/post/2014/01/10/RESTful-mORMot">RESTful ORM</a> and <em>MongoDB</em>.</p>
<p>Stay tuned!</p>