Synopse Open Source - Tag - TDataSetmORMot MVC / SOA / ORM and friends2024-02-02T17:08:25+00:00urn:md5:cc547126eb580a9adbec2349d7c65274DotclearHTTP remote access for SynDB SQL executionurn:md5:8ac51ed0a6e45f22746c8ba9e6d3e63c2014-11-18T01:08:00+01:002014-11-18T01:11:52+01:00AB4327-GANDIOpen Source librariesarray bindingblogcompressionDatabaseDelphiFirebirdFireDACHTTPhttp.sysHTTPSmORMotMSSQLMySQLOpenSourceOracleORMperformancesecuritySQLSynDBSynDBExplorerTDataSetwebWinHTTPWinINetZEOS<p>For <em>mORMot</em>, we developed a fully feature direct access layer to any
RDBMS, implemented <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_126">
in the SynDB.pas unit</a>.</p>
<p>You can use those <code>SynDB</code> classes to execute any SQL statement,
without any link to the framework ORM.<br />
At reading, the resulting performance is much higher than using the standard
<code>TDataSet</code> component, which is in fact a true performance
bottleneck.<br />
It has genuine features, like <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_136">
column access via late-binding</a>, <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_137">
an innovative <code>ISQLDBRows</code> interface, and ability to directly access
the low-level binary buffers of the database clients.</a></p>
<p><img src="http://vpn-services.bestreviews.net/files/vpn-remote-access.jpg" alt="" /></p>
<p>We just added a nice feature to those classes: the ability to access
remotely, via plain HTTP, to any <code>SynDB</code> supported database!</p> <p>To define a HTTP server, you may write:</p>
<pre>
<strong>uses</strong> SynDB, <em>// RDBMS core</em>
SynDBSQLite3, SynSQLite3Static, <em>// static SQLite3 engine</em>
SynDBRemote; <em>// for HTTP server</em>
...
<strong>var</strong> Props: TSQLDBConnectionProperties;
HttpServer: TSQLDBServerAbstract;
...
Props := TSQLDBSQLite3ConnectionProperties.Create('data.db3','','','');
<span style="background-color:yellow;">HttpServer := TSQLDBServerHttpApi.Create(Props,'remote','8092','user','pass');</span>
</pre>
<p>The above code will initialize a connection to a local <code>data.db3</code>
<em>SQlite3</em> database (in the <code>Props</code> variable), and then
publish it using the <code>http.sys</code> kernel mode HTTP server to the
<code>http://1.2.3.4:8092/remote</code> URI - if the server's IP is
<code>1.2.3.4</code>.</p>
<p>On the client side, you can then write:</p>
<pre>
<strong>uses</strong> SynDB, <em>// RDBMS core</em>
SynDBRemote; <em>// for HTTP server</em>
...
<strong>var</strong> Props: TSQLDBConnectionProperties;
...
Props := TSQLDBWinHTTPConnectionProperties.Create('1.2.3.4:8092','root','user','pass');
</pre>
<p>As you can see, there is no link to <code>SynDBSQLite3.pas</code> nor
<code>SynSQLite3Static.pas</code> on the client side.<br />
Just the HTTP link is needed.<br />
No need to deploy the RDBMS client libraries with your application, nor setup
the local network firewall.</p>
<p>We defined here a single user, with 'user' / 'pass' credentials, but you may
manage more users on the server side, using the <code>Authenticate</code>
property of <code>TSQLDBServerAbstract</code>.<br />
Note that in our remote access, user management does not match the RDBMS user
rights: you should better have your own set of users at application level, for
higher security, and a better integration with your business logic. If creating
a new user on a RDBMS could be painful, it is pretty easy on our
<code>SynDBRemote.pas</code> side.</p>
<p>Then, you execute your favorite SQL using the connection just as usual:</p>
<pre>
<strong>procedure</strong> Test(Props: TSQLDBConnectionProperties);
<strong>var</strong> Stmt: ISQLDBRows;
<strong>begin</strong>
Stmt := Props.Execute('select * from People where YearOfDeath=?',[1519]);
<strong>while</strong> Stmt.Step <strong>do begin</strong>
assert(Stmt.ColumnInt('ID')>0);
assert(Stmt.ColumnInt('YearOfDeath')=1519);
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>You may even use this remote connection e.g. using a stand-alone shared
<em>SQLite3</em> database as high performance but low maintenance client-server
database engine.</p>
<p>The transmission protocol uses an optimized binary format, which is
compressed and digitally signed on both ends, and the remote user
authentication will be performed via a challenge validation scheme.<br />
You could publish your server over HTTPS, if needed.</p>
<p>Even if you may be tempted to use such remote access to implement a
<em>n-Tier architecture</em>, you should rather use <em>mORMot</em>'s
Client-Server ORM instead. Our little <em>mORMot</em> is not an ORM on
which we added a data transmission layer: it is a full RESTful system, with a
true SOA design.<br />
But for integrating some legacy SQL code into a new architecture,
<code>SynDBRemote.pas</code> may have its benefits, used in conjunction with
<em>mORMot</em>'s higher level features.</p>
<p>Feel free to <a href="http://synopse.info/forum/viewtopic.php?id=2178">post
your feedback on our forum</a>, as usual, or <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_131">
browse the updated documentation</a>!</p>Introducing mORMot's architecture and design principlesurn:md5:265b0a5927ad96f526d355c5c9aca5322014-04-18T11:40:00+02:002015-01-04T10:10:34+01:00AB4327-GANDImORMot Frameworkcode-firstDatabasedatabase-firstDelphiDependencyInjectionDocumentationDomainDrivenDTOdynamic arrayEventSourcingfactoryFireDACGoodPracticehttp.sysHttpApiHTTPSinterfaceJSONLateBindingmockModelmORMotMSSQLMySQLNextGenNexusDBNoSQLODBCOleDBOpenSourceOracleORMperformancePostgreSQLRepositoryRTTIsessionshardingSOASQLSQLite3SynDBSynopseTDataSetTDocVariantTDynArraytransactionUnicodeUniDACValueObjectVirtualTableWinHTTPWinINetZEOS <p>We have just released a set of slides introducing </p>
<ul>
<li>ORM, SOA, REST, JSON, MVC, MVVM, SOLID, Mocks/Stubs, Domain-Driven Design
concepts with Delphi, </li>
<li>and showing some sample code using our Open Source <em>mORMot</em>
framework.</li>
</ul>
<p>You can follow the <a href="https://drive.google.com/folderview?id=0B0r8u-FwvxWdeVJVZnBhSEpKYkE&usp=sharing">
public link on Google Drive</a>!</p>
<p><img src="http://images.fineartamerica.com/images-medium-large/1-golden-marmot-maureen-ida-farley.jpg" width="450" height="291" alt="" /></p>
<p>This is a great opportunity to discovers some patterns you may not be
familiar with, and find out how <em>mORMot</em> try to implement them.<br />
This set of slides may be less intimidating than our huge documentation - do
not be terrified by our <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html">
Online Documentation</a>!<br />
The <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#SIDE_TITL_40">
first set of pages</a> (presenting architecture and design principles) is worth
reading.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1707">welcome on our forum, as
usual</a>.</p>Support of MySQL, DB2 and PostgreSQLurn:md5:7be2ca38e9a4c0d2c2548de8ed6a46c32014-03-07T10:29:00+01:002014-03-07T10:31:24+01:00AB4327-GANDImORMot FrameworkblogDatabaseDB2DelphiDocumentationFirebirdFireDACmORMotMSSQLmultithreadMySQLNexusDBODBCOleDBOracleORMperformancePostgreSQLRestSQLSQLite3SynDBTDataSetUnicodeUniDACZEOS<p>We just tested, benchmarked and validated <a href="https://dev.mysql.com">Oracle MySQL</a>, <a href="http://www.ibm.com/software/data/db2/express-c/">IBM DB2</a> and <a href="http://www.postgresql.org/">PostgreSQL</a> support for our <code>SynDB</code>
database classes and the <em>mORMot</em>'s ORM core.<br />
This article will also show all updated results, including our <a href="https://blog.synopse.info?post/post/2014/03/03/ORM-enhanced-for-BATCH-insert">newly introduced multi-value
INSERT statement generations</a>, which speed up a lot BATCH insertion.</p>
<p>Stay tuned!</p>
<p><img src="http://fc04.deviantart.net/fs70/f/2012/206/0/7/marmot_can_run_by_jaffa_tamarin-d58lk2i.jpg" alt="" width="450" height="251" /></p>
<p><strong>Purpose here is not to say that one library or database is better or
faster than another, but publish a snapshot of <em>mORMot</em> persistence
layer abilities, depending on each access library.</strong></p>
<p>In this timing, we do not benchmark only the "pure" SQL/DB layer access
(<code>SynDB</code> units), but the whole Client-Server ORM of our
framework.</p>
<p>Process below includes all aspects of our ORM:</p>
<ul>
<li>Access via high level CRUD methods (<em>Add/Update/Delete/Retrieve</em>,
either per-object or in BATCH mode);</li>
<li>Read and write access of <code>TSQLRecord</code> instances, via optimized
RTTI;</li>
<li>JSON marshaling of all values (ready to be transmitted over a
network);</li>
<li>REST routing, with security, logging and statistic;</li>
<li>Virtual cross-database layer using its <em>SQLite3</em> kernel;</li>
<li>SQL on-the-fly generation and translation (in <em>virtual</em> mode);</li>
<li>Access to the database engines via several libraries or providers.</li>
</ul>
<p>In those tests, we just bypassed the communication layer, since
<code>TSQLRestClient</code> and <code>TSQLRestServer</code> are run in-process,
in the same thread - as a <code>TSQLRestServerDB</code> instance. So you have
here some raw performance testimony of our framework's ORM and RESTful core,
and may expect good scaling abilities when running on high-end hardware, over a
network.</p>
<p>On a recent notebook computer (<em>Core i7</em> and SSD drive), depending on
the back-end database interfaced, <em>mORMot</em> excels in speed, as will show
the following benchmark:</p>
<ul>
<li>You can persist up to 570,000 objects per second, or retrieve 870,000
objects per second (for our pure Delphi in-memory engine);</li>
<li>When data is retrieved from server or client 38, you can read more than
900,000 objects per second, whatever the database back-end is;</li>
<li>With a high-performance database like Oracle, and our direct access
classes, you can write 70,000 (via array binding) and read 160,000 objects per
second, over a 100 MB network;</li>
<li>When using alternate database access libraries (e.g. Zeos, or
<code>DB.pas</code> based classes), speed is lower (even if comparable for DB2,
MS SQL, PostgreSQL, MySQL) but still enough for most work, due to some
optimizations in the <em>mORMot</em> code (e.g. caching of prepared statements,
SQL multi-values insertion, direct export to/from JSON, <em>SQlite3</em>
virtual mode design, avoid most temporary memory allocation...).</li>
</ul>
<p>Difficult to find a faster ORM, I suspect.</p> <h3>Software and hardware configuration</h3>
<p>The following tables try to sum up all available possibilities, and give
some benchmark (average objects/second for writing or reading).</p>
<p>In these tables:<br />
- 'SQLite3 (file full/off/exc)' indicates use of the internal <em>SQLite3</em>
engine, with or without <code>Synchronous := smOff</code> and/or
<code>DB.LockingMode := lmExclusive</code>;<br />
- 'SQLite3 (mem)' stands for the internal <em>SQLite3</em> engine running in
memory;<br />
- 'SQLite3 (ext ...)' is about access to a <em>SQLite3</em> engine as external
database , either as file or memory;<br />
- '<code>TObjectList</code>' indicates a
<code>TSQLRestServerStaticInMemory</code> instance, either static (with no SQL
support) or virtual (i.e. SQL featured via <em>SQLite3</em> virtual table
mechanism) which may persist the data on disk as JSON or compressed
binary;<br />
- 'NexusDB' is the free embedded edition, available from official site;<br />
- 'Jet' stands for a <em>MSAccess</em> database engine, accessed via
OleDB.<br />
- 'Oracle' shows the results of our direct OCI access layer
(<code>SynDBOracle.pas</code>);<br />
- 'Zeos *' indicates that the database was accessed directly via the ZDBC
layer;<br />
- 'FireDAC *' stands for <em>FireDAC</em> library;<br />
- 'UniDAC *' stands for <em>UniDAC</em> library;<br />
- 'BDE *' when using a <em>BDE</em> connection;<br />
- 'ODBC *' for a direct access to ODBC.</p>
<p>This list of database providers is to be extended in the future. Any
feedback is welcome!</p>
<p>Numbers are expressed in rows/second (or objects/second). This benchmark was
compiled with Delphi XE4, since newer compilers tends to give better results,
mainly thanks to function in-lining (which was not existing e.g. in Delphi
6-7).</p>
<p>Note that these tests are not about the relative speed of each database
engine, but reflect the current status of the integration of several DB
libraries within the <em>mORMot</em> database access.</p>
<p>Benchmark was run on a <em>Core i7</em> notebook, running <em>Windows
7</em>, with a standard SSD, including anti-virus and background
applications:<br />
- Linked to a shared <em>Oracle</em> 11.2.0.1 database over 100 Mb
Ethernet;<br />
- <em>MS SQL Express 2008 R2</em> running locally in 64 bit mode;<br />
- <em>IBM DB2 Express-C</em> edition 10.5 running locally in 64 bit mode;<br />
- <em>PostgreSQL</em> 9.2.7 running locally in 64 bit mode;<br />
- <em>MySQL</em> 5.6.16 running locally in 64 bit mode;<br />
- <em>Firebird</em> embedded in revision 2.5.2;<br />
- <em>NexusDB</em> 3.11 in Free Embedded Version.</p>
<p>So it was a development environment, very similar to low-cost production
site, not dedicated to give best performance. During the process, CPU was
noticeable used only for <em>SQLite3</em> in-memory and <em>TObjectList</em> -
most of the time, the bottleneck is not the CPU, but the storage or network. As
a result, rates and timing may vary depending on network and server load, but
you get results similar to what could be expected on customer side, with an
average hardware configuration. When using high-head servers and storage,
running on a tuned <em>Linux</em> configuration, you can expect even better
numbers.</p>
<p>Tests were compiled with the Delphi XE4 32 bit mode target platform. Most of
the tests do pass when compiled as a 64 bit executable, with the exception of
some providers (like Jet), not available on this platform. Speed results are
almost the same, only slightly slower; so we won't show them here.</p>
<p>You can compile the "<code>15 - External DB performance</code>" supplied
sample code, and run the very same benchmark on your own configuration.<br />
Feedback is welcome!</p>
<p>From our tests, the UniDAC version we were using had huge stability issues
when used with DB2: the tests did not pass, and the DB2 server just hang
processing the queries, whereas there was no problem with other libraries. It
may have been fixed since, but you won't find any "UniDAC DB2" results in the
benchmark below in the meanwhile.</p>
<h3>Insertion speed</h3>
<p>Here we insert 5,000 rows of data, with diverse scenarios:<br />
- 'Direct' stands for a individual <code>Client.Add()</code> insertion;<br />
- 'Batch' mode will be described 28;<br />
- 'Trans' indicates that all insertion is nested within a transaction - which
makes a great difference, e.g. with a <em>SQlite3</em> database.</p>
<table>
<tbody>
<tr>
<td> </td>
<td><strong>Direct</strong></td>
<td><strong>Batch</strong></td>
<td><strong>Trans</strong></td>
<td><strong>Batch Trans</strong></td>
</tr>
<tr>
<td><strong>SQLite3 (file full)</strong></td>
<td>462</td>
<td>356</td>
<td>95377</td>
<td>130086</td>
</tr>
<tr>
<td><strong>SQLite3 (file off)</strong></td>
<td>844</td>
<td>821</td>
<td>100389</td>
<td>136675</td>
</tr>
<tr>
<td><strong>SQLite3 (file off exc)</strong></td>
<td>28847</td>
<td>35316</td>
<td>102599</td>
<td>144258</td>
</tr>
<tr>
<td><strong>SQLite3 (mem)</strong></td>
<td>89456</td>
<td>120513</td>
<td>104249</td>
<td>146933</td>
</tr>
<tr>
<td><strong>TObjectList (static)</strong></td>
<td>314465</td>
<td>543892</td>
<td>326370</td>
<td>542652</td>
</tr>
<tr>
<td><strong>TObjectList (virtual)</strong></td>
<td>325393</td>
<td>545672</td>
<td>298846</td>
<td>545018</td>
</tr>
<tr>
<td><strong>SQLite3 (ext full)</strong></td>
<td>424</td>
<td>11297</td>
<td>102049</td>
<td>164636</td>
</tr>
<tr>
<td><strong>SQLite3 (ext off)</strong></td>
<td>830</td>
<td>21406</td>
<td>109706</td>
<td>189250</td>
</tr>
<tr>
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>41589</td>
<td>180759</td>
<td>108481</td>
<td>192071</td>
</tr>
<tr>
<td><strong>SQLite3 (ext mem)</strong></td>
<td>101440</td>
<td>234576</td>
<td>113530</td>
<td>190142</td>
</tr>
<tr>
<td><strong>ODBC SQLite3</strong></td>
<td>492</td>
<td>11746</td>
<td>35367</td>
<td>82425</td>
</tr>
<tr>
<td><strong>ZEOS SQlite3</strong></td>
<td>494</td>
<td>11851</td>
<td>56206</td>
<td>85705</td>
</tr>
<tr>
<td><strong>FireDAC SQlite3</strong></td>
<td>26369</td>
<td>50306</td>
<td>49755</td>
<td>155115</td>
</tr>
<tr>
<td><strong>UniDAC SQlite3</strong></td>
<td>477</td>
<td>8725</td>
<td>26552</td>
<td>38756</td>
</tr>
<tr>
<td><strong>ODBC Firebird</strong></td>
<td>1495</td>
<td>18056</td>
<td>13485</td>
<td>17731</td>
</tr>
<tr>
<td><strong>ZEOS Firebird</strong></td>
<td>9733</td>
<td>13429</td>
<td>26348</td>
<td>30616</td>
</tr>
<tr>
<td><strong>FireDAC Firebird</strong></td>
<td>24233</td>
<td>52021</td>
<td>24791</td>
<td>52111</td>
</tr>
<tr>
<td><strong>UniDAC Firebird</strong></td>
<td>5986</td>
<td>14809</td>
<td>6522</td>
<td>14948</td>
</tr>
<tr>
<td><strong>Jet</strong></td>
<td>4235</td>
<td>4424</td>
<td>4954</td>
<td>5094</td>
</tr>
<tr>
<td><strong>NexusDB</strong></td>
<td>5998</td>
<td>15494</td>
<td>7687</td>
<td>18619</td>
</tr>
<tr>
<td><strong>Oracle</strong></td>
<td>226</td>
<td>56112</td>
<td>1133</td>
<td>52367</td>
</tr>
<tr>
<td><strong>ODBC Oracle</strong></td>
<td>236</td>
<td>1664</td>
<td>1515</td>
<td>7709</td>
</tr>
<tr>
<td><strong>FireDAC Oracle</strong></td>
<td>118</td>
<td>48575</td>
<td>1519</td>
<td>12566</td>
</tr>
<tr>
<td><strong>UniDAC Oracle</strong></td>
<td>164</td>
<td>5701</td>
<td>1215</td>
<td>2884</td>
</tr>
<tr>
<td><strong>BDE Oracle</strong></td>
<td>489</td>
<td>927</td>
<td>839</td>
<td>1022</td>
</tr>
<tr>
<td><strong>MSSQL local</strong></td>
<td>5246</td>
<td>54360</td>
<td>12988</td>
<td>62453</td>
</tr>
<tr>
<td><strong>ODBC MSSQL</strong></td>
<td>4911</td>
<td>18652</td>
<td>11541</td>
<td>20976</td>
</tr>
<tr>
<td><strong>FireDAC MSSQL</strong></td>
<td>5016</td>
<td>7341</td>
<td>11686</td>
<td>51242</td>
</tr>
<tr>
<td><strong>UniDAC MSSQL</strong></td>
<td>4392</td>
<td>29768</td>
<td>8649</td>
<td>33464</td>
</tr>
<tr>
<td><strong>ODBC DB2</strong></td>
<td>4792</td>
<td>48387</td>
<td>14085</td>
<td>70104</td>
</tr>
<tr>
<td><strong>FireDAC DB2</strong></td>
<td>4452</td>
<td>48635</td>
<td>11014</td>
<td>52781</td>
</tr>
<tr>
<td><strong>ZEOS PostgreSQL</strong></td>
<td>4196</td>
<td>26663</td>
<td>9689</td>
<td>38735</td>
</tr>
<tr>
<td><strong>ODBC PostgreSQL</strong></td>
<td>4068</td>
<td>19515</td>
<td>5130</td>
<td>27843</td>
</tr>
<tr>
<td><strong>FireDAC PostgreSQL</strong></td>
<td>4181</td>
<td>37000</td>
<td>10111</td>
<td>36483</td>
</tr>
<tr>
<td><strong>UniDAC PostgreSQL</strong></td>
<td>2705</td>
<td>18563</td>
<td>4442</td>
<td>22317</td>
</tr>
<tr>
<td><strong>ODBC MySQL</strong></td>
<td>3160</td>
<td>38309</td>
<td>10856</td>
<td>47630</td>
</tr>
<tr>
<td><strong>ZEOS MySQL</strong></td>
<td>3426</td>
<td>34037</td>
<td>12217</td>
<td>40186</td>
</tr>
<tr>
<td><strong>FireDAC MySQL</strong></td>
<td>3078</td>
<td>43053</td>
<td>10955</td>
<td>45781</td>
</tr>
<tr>
<td><strong>UniDAC MySQL</strong></td>
<td>3119</td>
<td>27772</td>
<td>11246</td>
<td>33288</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Batch+Trans|Trans|Batch|Direct&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,545672&chds=0,545672,0,545672,0,545672,0,545672,0,545672&chd=t:462,356,95377,130086|844,821,100389,136675|28847,35316,102599,144258|89456,120513,104249,146933|314465,543892,326370,542652|325393,545672,298846,545018|424,11297,102049,164636|830,21406,109706,189250|41589,180759,108481,192071|101440,234576,113530,190142|492,11746,35367,82425|494,11851,56206,85705|26369,50306,49755,155115|477,8725,26552,38756|1495,18056,13485,17731|9733,13429,26348,30616|24233,52021,24791,52111|5986,14809,6522,14948|4235,4424,4954,5094|226,56112,1133,52367|236,1664,1515,7709|118,48575,1519,12566|164,5701,1215,2884|5246,54360,12988,62453|4911,18652,11541,20976|5016,7341,11686,51242|4392,29768,8649,33464|4792,48387,14085,70104|4452,48635,11014,52781|4196,26663,9689,38735|4068,19515,5130,27843|4181,37000,10111,36483|2705,18563,4442,22317|3160,38309,10856,47630|3426,34037,12217,40186|3078,43053,10955,45781|3119,27772,11246,33288&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|ODBC+SQLite3|ZEOS+SQlite3|FireDAC+SQlite3|UniDAC+SQlite3|ODBC+Firebird|ZEOS+Firebird|FireDAC+Firebird|UniDAC+Firebird|Jet|Oracle|ODBC+Oracle|FireDAC+Oracle|UniDAC+Oracle|MSSQL+local|ODBC+MSSQL|FireDAC+MSSQL|UniDAC+MSSQL|ODBC+DB2|FireDAC+DB2|ZEOS+PostgreSQL|ODBC+PostgreSQL|FireDAC+PostgreSQL|UniDAC+PostgreSQL|ODBC+MySQL|ZEOS+MySQL|FireDAC+MySQL|UniDAC+MySQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|UniDAC+MySQL|FireDAC+MySQL|ZEOS+MySQL|ODBC+MySQL|UniDAC+PostgreSQL|FireDAC+PostgreSQL|ODBC+PostgreSQL|ZEOS+PostgreSQL|FireDAC+DB2|ODBC+DB2|UniDAC+MSSQL|FireDAC+MSSQL|ODBC+MSSQL|MSSQL+local|UniDAC+Oracle|FireDAC+Oracle|ODBC+Oracle|Oracle|Jet|UniDAC+Firebird|FireDAC+Firebird|ZEOS+Firebird|ODBC+Firebird|UniDAC+SQlite3|FireDAC+SQlite3|ZEOS+SQlite3|ODBC+SQLite3|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,545672&chds=0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672,0,545672&chd=t:462,844,28847,89456,314465,325393,424,830,41589,101440,492,494,26369,477,1495,9733,24233,5986,4235,226,236,118,164,5246,4911,5016,4392,4792,4452,4196,4068,4181,2705,3160,3426,3078,3119|356,821,35316,120513,543892,545672,11297,21406,180759,234576,11746,11851,50306,8725,18056,13429,52021,14809,4424,56112,1664,48575,5701,54360,18652,7341,29768,48387,48635,26663,19515,37000,18563,38309,34037,43053,27772|95377,100389,102599,104249,326370,298846,102049,109706,108481,113530,35367,56206,49755,26552,13485,26348,24791,6522,4954,1133,1515,1519,1215,12988,11541,11686,8649,14085,11014,9689,5130,10111,4442,10856,12217,10955,11246|130086,136675,144258,146933,542652,545018,164636,189250,192071,190142,82425,85705,155115,38756,17731,30616,52111,14948,5094,52367,7709,12566,2884,62453,20976,51242,33464,70104,52781,38735,27843,36483,22317,47630,40186,45781,33288&chdl=Direct|Batch|Trans|Batch+Trans" /></p>
<p>Due to its ACID implementation, <em>SQLite3</em> process on file waits for
the hard-disk to have finished flushing its data, therefore it is the reason
why it is slower than other engines at individual row insertion (less than 10
objects per second with a mechanical hardrive instead of a SDD) outside the
scope of a transaction.</p>
<p>So if you want to reach the best writing performance in your application
with the default engine, you should better use transactions and regroup all
writing into services or a BATCH process. Another possibility could be to
execute <code>DB.Synchronous := smOff</code> and/or <code>DB.LockingMode :=
lmExclusive</code> at <em>SQLite3</em> engine level before process: in case of
power loss at wrong time it may corrupt the database file, but it will increase
the rate by a factor of 50 (with hard drive), <a href="https://blog.synopse.info?post/post/2013/06/14/SQLite3-performance-in-Exclusive-file-locking-mode">as stated
by the "<em>off</em>" and "<em>off exc</em>" rows of the table</a>. Note that
by default, the <em>FireDAC</em> library set both options, so results above are
to be compared with "<em>SQLite3 off exc</em>" rows.</p>
<p>For both our direct <em>Oracle</em> access <code>SynDBOracle.pas</code> unit
and <em>FireDAC</em> library , BATCH process benefits of the array binding
feature a lot (known as <em>Array DML</em> in <em>FireDAC/AnyDAC</em>).</p>
<p>For most engines, our ORM kernel is able to generate the appropriate SQL
statement for speeding up bulk insertion. For instance:<br />
- <em>SQlite3, MySQL, PostgreSQL, MSSQL 2008, DB2, MySQL</em> or
<em>NexusDB</em> handle <code>INSERT</code> statements with multiple
<code>INSERT INTO .. VALUES (..),(..),(..)..</code>;<br />
- <em>Oracle</em> handles <code>INSERT INTO .. INTO .. SELECT 1 FROM
DUAL</code> (weird syntax, isn't it?);<br />
- <em>Firebird</em> implements <code>EXECUTE BLOCK</code>.</p>
<p>As a result, some engines show a nice speed boost when
<code>BatchAdd()</code> is used. Even <em>SQLite3</em> is faster when used as
external engine, in respect to direct execution! This feature is at ORM/SQL
level, so it benefits to any external database library. Of course, if a given
library has a better implementation pattern (e.g. our direct <em>Oracle</em> or
<em>FireDAC</em> with native array binding), it is used instead.</p>
<h3>Reading speed</h3>
<p>Now the same data is retrieved via the ORM layer:<br />
- 'By one' states that one object is read per call (ORM generates a
<code>SELECT * FROM table WHERE ID=?</code> for <code>Client.Retrieve()</code>
method);<br />
- 'All *' is when all 5000 objects are read in a single call (i.e. running
<code>SELECT * FROM table</code> from a <code>FillPrepare()</code> method
call), either forced to use the virtual table layer, or with direct static
call.</p>
<p>Here are some reading speed values, in objects/second:</p>
<table>
<tbody>
<tr>
<td> </td>
<td><strong>By one</strong></td>
<td><strong>All Virtual</strong></td>
<td><strong>All Direct</strong></td>
</tr>
<tr>
<td><strong>SQLite3 (file full)</strong></td>
<td>27284</td>
<td>558721</td>
<td>550842</td>
</tr>
<tr>
<td><strong>SQLite3 (file off)</strong></td>
<td>26896</td>
<td>549450</td>
<td>526149</td>
</tr>
<tr>
<td><strong>SQLite3 (file off exc)</strong></td>
<td>128077</td>
<td>557537</td>
<td>535905</td>
</tr>
<tr>
<td><strong>SQLite3 (mem)</strong></td>
<td>127106</td>
<td>557537</td>
<td>563316</td>
</tr>
<tr>
<td><strong>TObjectList (static)</strong></td>
<td>300012</td>
<td>912408</td>
<td>913742</td>
</tr>
<tr>
<td><strong>TObjectList (virtual)</strong></td>
<td>303287</td>
<td>402706</td>
<td>866551</td>
</tr>
<tr>
<td><strong>SQLite3 (ext full)</strong></td>
<td>135380</td>
<td>267436</td>
<td>553158</td>
</tr>
<tr>
<td><strong>SQLite3 (ext off)</strong></td>
<td>133696</td>
<td>262977</td>
<td>543065</td>
</tr>
<tr>
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>134698</td>
<td>264186</td>
<td>558596</td>
</tr>
<tr>
<td><strong>SQLite3 (ext mem)</strong></td>
<td>137487</td>
<td>259713</td>
<td>557475</td>
</tr>
<tr>
<td><strong>ODBC SQLite3</strong></td>
<td>19461</td>
<td>136600</td>
<td>201280</td>
</tr>
<tr>
<td><strong>ZEOS SQlite3</strong></td>
<td>33541</td>
<td>200835</td>
<td>306955</td>
</tr>
<tr>
<td><strong>FireDAC SQlite3</strong></td>
<td>7683</td>
<td>83532</td>
<td>112470</td>
</tr>
<tr>
<td><strong>UniDAC SQlite3</strong></td>
<td>2522</td>
<td>74030</td>
<td>96420</td>
</tr>
<tr>
<td><strong>ODBC Firebird</strong></td>
<td>3446</td>
<td>69607</td>
<td>97585</td>
</tr>
<tr>
<td><strong>ZEOS Firebird</strong></td>
<td>20296</td>
<td>91974</td>
<td>107229</td>
</tr>
<tr>
<td><strong>FireDAC Firebird</strong></td>
<td>2376</td>
<td>46276</td>
<td>56269</td>
</tr>
<tr>
<td><strong>UniDAC Firebird</strong></td>
<td>2189</td>
<td>66886</td>
<td>88102</td>
</tr>
<tr>
<td><strong>Jet</strong></td>
<td>2640</td>
<td>166112</td>
<td>258277</td>
</tr>
<tr>
<td><strong>NexusDB</strong></td>
<td>1413</td>
<td>120845</td>
<td>208246</td>
</tr>
<tr>
<td><strong>Oracle</strong></td>
<td>1558</td>
<td>120977</td>
<td>159861</td>
</tr>
<tr>
<td><strong>ODBC Oracle</strong></td>
<td>1620</td>
<td>43441</td>
<td>45764</td>
</tr>
<tr>
<td><strong>FireDAC Oracle</strong></td>
<td>1231</td>
<td>42149</td>
<td>54795</td>
</tr>
<tr>
<td><strong>UniDAC Oracle</strong></td>
<td>688</td>
<td>27083</td>
<td>30093</td>
</tr>
<tr>
<td><strong>BDE Oracle</strong></td>
<td>860</td>
<td>3870</td>
<td>4036</td>
</tr>
<tr>
<td><strong>MSSQL local</strong></td>
<td>10135</td>
<td>210837</td>
<td>437905</td>
</tr>
<tr>
<td><strong>ODBC MSSQL</strong></td>
<td>12458</td>
<td>147544</td>
<td>256502</td>
</tr>
<tr>
<td><strong>FireDAC MSSQL</strong></td>
<td>3776</td>
<td>72123</td>
<td>94091</td>
</tr>
<tr>
<td><strong>UniDAC MSSQL</strong></td>
<td>2505</td>
<td>93231</td>
<td>135932</td>
</tr>
<tr>
<td><strong>ODBC DB2</strong></td>
<td>7649</td>
<td>84880</td>
<td>124486</td>
</tr>
<tr>
<td><strong>FireDAC DB2</strong></td>
<td>3155</td>
<td>71456</td>
<td>88264</td>
</tr>
<tr>
<td><strong>ZEOS PostgreSQL</strong></td>
<td>8833</td>
<td>158760</td>
<td>223583</td>
</tr>
<tr>
<td><strong>ODBC PostgreSQL</strong></td>
<td>10361</td>
<td>85680</td>
<td>120913</td>
</tr>
<tr>
<td><strong>FireDAC PostgreSQL</strong></td>
<td>2261</td>
<td>58252</td>
<td>79002</td>
</tr>
<tr>
<td><strong>UniDAC PostgreSQL</strong></td>
<td>864</td>
<td>86900</td>
<td>122856</td>
</tr>
<tr>
<td><strong>ODBC MySQL</strong></td>
<td>10143</td>
<td>65538</td>
<td>82447</td>
</tr>
<tr>
<td><strong>ZEOS MySQL</strong></td>
<td>2052</td>
<td>171803</td>
<td>245772</td>
</tr>
<tr>
<td><strong>FireDAC MySQL</strong></td>
<td>3636</td>
<td>75081</td>
<td>105028</td>
</tr>
<tr>
<td><strong>UniDAC MySQL</strong></td>
<td>4798</td>
<td>99940</td>
<td>146968</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|All+Direct|All+Virtual|By+one&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,913742&chds=0,913742,0,913742,0,913742&chd=t:27284,558721,550842|26896,549450,526149|128077,557537,535905|127106,557537,563316|300012,912408,913742|303287,402706,866551|135380,267436,553158|133696,262977,543065|134698,264186,558596|137487,259713,557475|19461,136600,201280|33541,200835,306955|7683,83532,112470|2522,74030,96420|3446,69607,97585|20296,91974,107229|2376,46276,56269|2189,66886,88102|2640,166112,258277|1558,120977,159861|1620,43441,45764|1231,42149,54795|688,27083,30093|10135,210837,437905|12458,147544,256502|3776,72123,94091|2505,93231,135932|7649,84880,124486|3155,71456,88264|8833,158760,223583|10361,85680,120913|2261,58252,79002|864,86900,122856|10143,65538,82447|2052,171803,245772|3636,75081,105028|4798,99940,146968&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|ODBC+SQLite3|ZEOS+SQlite3|FireDAC+SQlite3|UniDAC+SQlite3|ODBC+Firebird|ZEOS+Firebird|FireDAC+Firebird|UniDAC+Firebird|Jet|Oracle|ODBC+Oracle|FireDAC+Oracle|UniDAC+Oracle|MSSQL+local|ODBC+MSSQL|FireDAC+MSSQL|UniDAC+MSSQL|ODBC+DB2|FireDAC+DB2|ZEOS+PostgreSQL|ODBC+PostgreSQL|FireDAC+PostgreSQL|UniDAC+PostgreSQL|ODBC+MySQL|ZEOS+MySQL|FireDAC+MySQL|UniDAC+MySQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|UniDAC+MySQL|FireDAC+MySQL|ZEOS+MySQL|ODBC+MySQL|UniDAC+PostgreSQL|FireDAC+PostgreSQL|ODBC+PostgreSQL|ZEOS+PostgreSQL|FireDAC+DB2|ODBC+DB2|UniDAC+MSSQL|FireDAC+MSSQL|ODBC+MSSQL|MSSQL+local|UniDAC+Oracle|FireDAC+Oracle|ODBC+Oracle|Oracle|Jet|UniDAC+Firebird|FireDAC+Firebird|ZEOS+Firebird|ODBC+Firebird|UniDAC+SQlite3|FireDAC+SQlite3|ZEOS+SQlite3|ODBC+SQLite3|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,913742&chds=0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742,0,913742&chd=t:27284,26896,128077,127106,300012,303287,135380,133696,134698,137487,19461,33541,7683,2522,3446,20296,2376,2189,2640,1558,1620,1231,688,10135,12458,3776,2505,7649,3155,8833,10361,2261,864,10143,2052,3636,4798|558721,549450,557537,557537,912408,402706,267436,262977,264186,259713,136600,200835,83532,74030,69607,91974,46276,66886,166112,120977,43441,42149,27083,210837,147544,72123,93231,84880,71456,158760,85680,58252,86900,65538,171803,75081,99940|550842,526149,535905,563316,913742,866551,553158,543065,558596,557475,201280,306955,112470,96420,97585,107229,56269,88102,258277,159861,45764,54795,30093,437905,256502,94091,135932,124486,88264,223583,120913,79002,122856,82447,245772,105028,146968&chdl=By+one|All+Virtual|All+Direct" /></p>
<p>The <em>SQLite3</em> layer gives amazing reading results, which makes it a
perfect fit for most typical ORM use. When running with <code>DB.LockingMode :=
lmExclusive</code> defined (i.e. "off exc" rows), reading speed is very high,
and benefits from exclusive access to the database file. External database
access is only required when data is expected to be shared with other
processes.</p>
<p>In the above table, it appears that all libraries based on
<code>DB.pas</code> are slower than the others for reading speed. In fact,
<code>TDataSet</code> sounds to be a real bottleneck, due to its internal data
marshalling. Even <em>FireDAC</em>, which is known to be very optimized for
speed, is limited by the <code>TDataSet</code> structure. Our direct classes,
or even ZEOS/ZDBC performs better, since they are able to output JSON content
with no additional marshalling.</p>
<p>For both writing and reading, <code>TObjectList</code> /
<code>TSQLRestServerStaticInMemory</code> engine gives impressive results, but
has the weakness of being in-memory, so it is not ACID by design, and the data
has to fit in memory. Note that indexes are available for IDs and <code>stored
AS_UNIQUE</code> properties.</p>
<p>As a consequence, search of non-unique values may be slow: the engine has to
loop through all rows of data. But for unique values (defined as <code>stored
AS_UNIQUE</code>), both insertion and search speed is awesome, due to its
optimized O(1) hash algorithm - see the following benchmark, especially the
"<em>By name</em>" row for "<em>TObjectList</em>" columns, which correspond to
a search of an unique <code>RawUTF8</code> property value via this hashing
method.</p>
<table>
<tbody>
<tr>
<td><strong>SQLite3 (file full)</strong></td>
<td><strong>SQLite3 (file off)</strong></td>
<td><strong>SQLite3 (mem)</strong></td>
<td><strong>TObjectList (static)</strong></td>
<td><strong>TObjectList (virt.)</strong></td>
<td><strong>SQLite3 (ext file full)</strong></td>
<td><strong>SQLite3 (ext file off)</strong></td>
<td><strong>SQLite3 (ext mem)</strong></td>
<td><strong>Oracle</strong></td>
<td><strong>Jet</strong></td>
</tr>
<tr>
<td><strong>By one</strong></td>
<td>10461</td>
<td>10549</td>
<td>44737</td>
<td>103577</td>
<td>103553</td>
<td>43367</td>
<td>44099</td>
<td>45220</td>
<td>901</td>
<td>1074</td>
</tr>
<tr>
<td><strong>By name</strong></td>
<td>9694</td>
<td>9651</td>
<td>32350</td>
<td>70534</td>
<td>60153</td>
<td>22785</td>
<td>22240</td>
<td>23055</td>
<td>889</td>
<td>1071</td>
</tr>
<tr>
<td><strong>All Virt.</strong></td>
<td>167095</td>
<td>162956</td>
<td>168651</td>
<td>253292</td>
<td>118203</td>
<td>97083</td>
<td>90592</td>
<td>94688</td>
<td>56639</td>
<td>52764</td>
</tr>
<tr>
<td><strong>All Direct</strong></td>
<td>167123</td>
<td>144250</td>
<td>168577</td>
<td>254284</td>
<td>256383</td>
<td>170794</td>
<td>165601</td>
<td>168856</td>
<td>88342</td>
<td>75999</td>
</tr>
</tbody>
</table>
<p>Above table results were run on a Core 2 duo laptop, so numbers are lower
than with the previous tables.</p>
<p>During the tests, <a href="https://blog.synopse.info?post/post/2012/02/14/ORM-cache">internal caching</a>
was disabled, so you may expect speed enhancements for real applications, when
data is more read than written: for instance, when an object is retrieved from
the cache, you achieve more than 1,00,000 read requests per second, whatever
database is used.</p>
<h3>Analysis and use case proposal</h3>
<p>When declared as virtual table (via a <code>VirtualTableRegister</code>
call), you have the full power of SQL (including JOINs) at hand, with
incredibly fast CRUD operations: 100,000 requests per second for objects read
and write, including serialization and Client-Server communication!</p>
<p>Some providers are first-class citizens to <em>mORMot</em>, like
<em>SQLite3</em>, <em>Oracle</em>, <em>MS SQL</em>, PostgreSQL, MySQL or IBM
DB2. You can connect to them without the bottleneck of the <code>DB.pas</code>
unit, nor any restriction of your Delphi license (a <em>Starter edition</em> is
enough).</p>
<p>First of all, <em>SQLite3</em> is still to be considered, even for a
production server. Thanks to <em>mORMot</em>'s architecture and design, this
"embedded" database could be used as main database engine for a client-server
application with heavy concurrent access - if you have doubts about its scaling
abilities, <a href="https://blog.synopse.info?post/post/2013/09/10/Thread-safety-of-mORMot">see this blog
article</a>. Here, "embedded" is not restricted to "mobile", but sounds like a
self-contained, zero-configuration proven engine.</p>
<p>Most recognized <em>closed source</em> databases are available:<br />
- Direct access to <em>Oracle</em> gives impressive results in BATCH mode (aka
array binding). It may be an obligation if your end-customer stores already its
data in such a server, for instance, and want to leverage the licensing cost of
its own IT solution. <em>Oracle Express</em> edition is free, but somewhat
heavy and limited in terms of data/hardware size (see its licensing
terms);<br />
- <em>MS SQL Server</em>, directly accessed via <em>OleDB</em> (or
<em>ODBC</em>) gives pretty good timing. A <em>MS SQL Server 2008 R2
Express</em> instance is pretty well integrated with the <em>Windows</em>
environment, for a very affordable price (i.e. for free) - the <em>LocalDB</em>
(MSI installer) edition is enough to start with, but also with data/hardware
size limitation, just like <em>Oracle Express</em>;<br />
- IBM <em>DB2</em> is another good candidate, and the <em>Express-C</em> ("C"
standing for Community) offers a no-charge opportunity to run an industry
standard engine, with no restriction on the data size, and somewhat high
hardware limitations (16 GB of RAM and 2 CPU cores for the latest 10.5 release)
or enterprise-level features;<br />
- <em>NexusDB</em> may be considered, if you have existing Delphi code and data
- but it is less known and recognized as the its commercial competitors.</p>
<p><em>Open Source</em> databases are worth considering, especially in
conjunction with an Open Source framework like <em>mORMot</em>:<br />
- <em>MySQL</em> is the well-known engine used by a lot of web sites, mainly
with <em>LAMP</em> (<em>Linux MySQL Apache PHP</em>) configurations. Windows is
not the best platform to run it, but it could be a fairly good candidate,
especially in its <em>MariaDB</em> fork, which sounds more attractive those
days than the official main version, owned by Oracle;<br />
- <em>PostgreSQL</em> is an Enterprise class database, with amazing features
among its Open Source alternatives, and really competes with commercial
solutions. Even under Windows, we think it is easy to install and administrate,
and uses less resource than the other commercial engines.<br />
- <em>Firebird</em> gave pretty consistent timing, when accessed via Zeos/ZDBC.
We show here the embedded version, but the server edition is worth considering,
even if it has a less</p>
<p>To access those databases, OleDB, ODBC or ZDBC providers may also be used,
with direct access. But <em>mORMot</em> is very open-minded: you can use any
<code>DB.pas</code> provider, e.g. <em>FireDAC</em>, <em>UniDAC</em>,
<em>DBExpress</em>, <em>NexusDB</em> or even the <em>BDE</em>, but with the
additional layer introduced by using a <code>TDataSet</code> instance, at
reading.</p>
<p>Therefore, the typical use may be the following (<em>int.</em> meaning
"internal", <em>ext.</em> for "external", <em>mem</em> for "in-memory"):</p>
<table>
<tbody>
<tr>
<td><strong>Database</strong></td>
<td><strong>Use case</strong></td>
</tr>
<tr>
<td>int. SQLite3 file</td>
<td>Created by default.<br />
General safe data handling, with amazing speed in "off exc" mode</td>
</tr>
<tr>
<td>int. SQLite3 mem</td>
<td>Created with <code>:memory:</code> file name.<br />
Fast data handling with no persistence (e.g. for testing or temporary
storage)</td>
</tr>
<tr>
<td><code>TObjectList</code> static</td>
<td>Created with <code>StaticDataCreate</code>.<br />
Best possible performance for small amount of data, without ACID nor SQL</td>
</tr>
<tr>
<td><code>TObjectList</code> virtual</td>
<td>Created with <code>VirtualTableRegister</code>.<br />
Best possible performance for SQL over small amount of data (or even unlimited
amount under Win64), if ACID is not required nor complex SQL</td>
</tr>
<tr>
<td>ext. SQLite3 file</td>
<td>Created with <code>VirtualTableExternalRegister<br /></code>External
back-end, e.g. for disk spanning</td>
</tr>
<tr>
<td>ext. SQLite3 mem</td>
<td>Created with <code>VirtualTableExternalRegister<br /></code>Fast external
back-end (e.g. for testing)</td>
</tr>
<tr>
<td>ext. Oracle / MS SQL / DB2 / PostgreSQL / MySQL / Firebird</td>
<td>Created with <code>VirtualTableExternalRegister<br /></code>Fast, secure
and industry standard back-ends; data can be shared outside
<em>mORMot</em></td>
</tr>
<tr>
<td>ext. NexusDB</td>
<td>Created with <code>VirtualTableExternalRegister<br /></code>The free
embedded version let the whole engine be included within your executable, and
use any existing code, but <em>SQlite3</em> sounds like a better option</td>
</tr>
<tr>
<td>ext. Jet</td>
<td>Created with <code>VirtualTableExternalRegister<br /></code>Could be used
as a data exchange format (e.g. with Office applications)</td>
</tr>
<tr>
<td>ext. Zeos</td>
<td>Created with <code>VirtualTableExternalRegister<br /></code>Allow access to
several external engines, with direct Zeos/ZDBC access which will by-pass the
<code>DB.pas</code> unit and its <code>TDataSet</code> bottleneck - and we will
also prefer an active Open Source project!</td>
</tr>
<tr>
<td>ext. FireDAC/UniDAC</td>
<td>Created with <code>VirtualTableExternalRegister<br /></code>Allow access to
several external engines, including the <code>DB.pas</code> unit and its
<code>TDataSet</code> bottleneck</td>
</tr>
</tbody>
</table>
<p>Whatever database back-end is used, don't forget that <em>mORMot</em> design
will allow you to switch from one library to another, just by changing a
<code>TSQLDBConnectionProperties</code> class type. And note that you can
<em>mix</em> external engines, on purpose: you are not tied to one single
engine, but the database access can be tuned for each ORM table, according to
your project needs.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1642">welcome in our forum, as
usual</a>.</p>ORM enhanced for BATCH inserturn:md5:74e984f2ac0ba51f3b2e7680d5d15f502014-03-03T16:37:00+01:002014-03-04T13:23:02+01:00AB4327-GANDImORMot Frameworkarray bindingblogDatabaseDelphiFireDACmORMotMSSQLNexusDBODBCOleDBOracleORMperformanceRepositorySourceSQLSQLite3SynDBTDataSetUniDACZEOS<p>We just committed some nice features to the ORM kernel, and <em>SynDB*</em>
classes of our <em>mORMot</em> framework.</p>
<p>During <a href="https://blog.synopse.info?post/post/2011/06/03/BATCH-sequences-for-adding/updating/deleting-records">BATCH
insertion</a>, the ORM is able to generate some optimized SQL statements,
depending on the target database, to send several rows of data at once.<br />
It induces a noticeable speed increase when saving several objects into an
external database.</p>
<p><img src="http://fc04.deviantart.net/fs70/f/2012/206/0/7/marmot_can_run_by_jaffa_tamarin-d58lk2i.jpg" alt="" width="450" height="251" /></p>
<p>This feature is available for <em>SQlite3</em> (3.7.11 and later),
<em>MySQL</em>, <em>PostgreSQL</em>, <em>MS SQL</em> <em>Server</em> (2008
and up), <em>Oracle</em>, <em>Firebird</em> and <em>NexusDB</em>.<br />
Since it is working at SQL level, it is available for all supported access
libraries, e.g. <em>ODBC</em>, <em>OleDB</em>, <em>Zeos/ZDBC, UniDAC,
FireDAC</em>.<br />
It means that even properties not implementing array binding (like <em>OleDB,
Zeos</em> or <em>UniDAC</em>) are able to have a huge boost at data insertion,
ready to compete with <a href="http://docwiki.embarcadero.com/RADStudio/XE5/en/Array_DML_(FireDAC)">the
(until now) more optimized libraries</a>.</p> <p>To be more specific:</p>
<p><em>SQlite3, MySQL, PostgreSQL, MSSQL 2008</em> or <em>NexusDB</em> handle
<code>INSERT</code> statements with multiple <code>VALUES</code>, in the
following <a href="http://en.wikipedia.org/wiki/Insert_(SQL)#Multirow_inserts">SQL-92 standard
syntax</a>, using parameters:</p>
<pre>
INSERT INTO TABLE (column-a, [column-b, ...])
VALUES ('value-1a', ['value-1b', ...]),
('value-2a', ['value-2b', ...]),
...
</pre>
<p><em>Oracle</em> implements <a href="http://stackoverflow.com/a/93724/458259">the weird but similar syntax</a>
(note the mandatory <code>SELECT</code> at the end):</p>
<pre>
INSERT ALL
INTO phone_book VALUES ('John Doe', '555-1212')
INTO phone_book VALUES ('Peter Doe', '555-2323')
SELECT * FROM DUAL;
</pre>
<p><em>Firebird</em> implements <a href="http://www.firebirdsql.org/refdocs/langrefupd25-execblock.html">its own
syntax</a>:</p>
<pre>
execute block
as
begin
INSERT INTO phone_book VALUES ('John Doe', '555-1212');
INSERT INTO phone_book VALUES ('Peter Doe', '555-2323');
end
</pre>
<p>As a result, some engines show a nice speed boost when used in BATCH.<br />
Even <em>SQLite3</em> is faster when used as external engine, in respect to
direct execution, in respect to direct execution of individual prepared
statements in loop! </p>
<p>Here are some insertion results, to compare <a href="https://blog.synopse.info?post/post/2013/11/04/Updated-mORMot-database-benchmark-including-MS-SQL">with the
previous benchmark</a>, which did not include these enhancements:</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>Direct</strong></td>
<td><strong>Batch</strong></td>
<td><strong>Trans</strong></td>
<td><strong>Batch Trans</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>488</td>
<td>463</td>
<td>97498</td>
<td>126256</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>789</td>
<td>815</td>
<td>101010</td>
<td>130561</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off exc)</strong></td>
<td>31376</td>
<td>35785</td>
<td>104410</td>
<td>136328</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>88070</td>
<td>106981</td>
<td>106215</td>
<td>144270</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>308584</td>
<td>545732</td>
<td>311837</td>
<td>535733</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>308413</td>
<td>539548</td>
<td>316997</td>
<td>527537</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>308</td>
<td>12151</td>
<td>107469</td>
<td>170636</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>776</td>
<td>22404</td>
<td>111819</td>
<td>188316</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>42213</td>
<td>182561</td>
<td>111642</td>
<td>197464</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>98531</td>
<td>228634</td>
<td>112004</td>
<td>227489</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>497</td>
<td>12071</td>
<td>56489</td>
<td>72720</td>
</tr>
<tr align="center">
<td><strong>ODBC SQlite3</strong></td>
<td>509</td>
<td>12480</td>
<td>38996</td>
<td>82581</td>
</tr>
<tr align="center">
<td><strong>FireDAC SQlite3</strong></td>
<td>24992</td>
<td>50065</td>
<td>21985</td>
<td>156887</td>
</tr>
<tr align="center">
<td><strong>UniDAC SQlite3</strong></td>
<td>469</td>
<td>8981</td>
<td>27667</td>
<td>39239</td>
</tr>
<tr align="center">
<td><strong>NexusDB</strong></td>
<td>5996</td>
<td>15494</td>
<td>7687</td>
<td>18619</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>12732</td>
<td>13848</td>
<td>27456</td>
<td>30724</td>
</tr>
<tr align="center">
<td><strong>ODBC Firebird</strong></td>
<td>1745</td>
<td>18366</td>
<td>14419</td>
<td>18993</td>
</tr>
<tr align="center">
<td><strong>FireDAC Firebird</strong></td>
<td>24000</td>
<td>50329</td>
<td>24050</td>
<td>51423</td>
</tr>
<tr align="center">
<td><strong>UniDAC Firebird</strong></td>
<td>6373</td>
<td>14801</td>
<td>6474</td>
<td>14675</td>
</tr>
<tr align="center">
<td><strong>Jet</strong></td>
<td>4252</td>
<td>4561</td>
<td>5016</td>
<td>5208</td>
</tr>
<tr align="center">
<td><strong>Oracle</strong></td>
<td>310</td>
<td>42327</td>
<td>1046</td>
<td>61661</td>
</tr>
<tr align="center">
<td><strong>ODBC Oracle</strong></td>
<td>337</td>
<td>3962</td>
<td>1356</td>
<td>5197</td>
</tr>
<tr align="center">
<td><strong>FireDAC Oracle</strong></td>
<td>458</td>
<td>35160</td>
<td>1451</td>
<td>37204</td>
</tr>
<tr align="center">
<td><strong>UniDAC Oracle</strong></td>
<td>289</td>
<td>3065</td>
<td>1140</td>
<td>5747</td>
</tr>
<tr align="center">
<td><strong>BDE Oracle</strong></td>
<td>489</td>
<td>927</td>
<td>839</td>
<td>1022</td>
</tr>
<tr align="center">
<td><strong>MSSQL local</strong></td>
<td>5266</td>
<td>54417</td>
<td>13659</td>
<td>62706</td>
</tr>
<tr align="center">
<td><strong>ODBC MSSQL</strong></td>
<td>5050</td>
<td>18739</td>
<td>11804</td>
<td>20796</td>
</tr>
<tr align="center">
<td><strong>FireDAC MSSQL</strong></td>
<td>4989</td>
<td>7315</td>
<td>11267</td>
<td>50520</td>
</tr>
<tr align="center">
<td><strong>UniDAC MSSQL</strong></td>
<td>4404</td>
<td>30845</td>
<td>8879</td>
<td>34933</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Batch+Trans|Trans|Batch|Direct&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,545732&chds=0,545732,0,545732,0,545732,0,545732,0,545732&chd=t:488,463,97498,126256|789,815,101010,130561|31376,35785,104410,136328|88070,106981,106215,144270|308584,545732,311837,535733|308413,539548,316997,527537|308,12151,107469,170636|776,22404,111819,188316|42213,182561,111642,197464|98531,228634,112004,227489|497,12071,56489,72720|24992,50065,21985,156887|469,8981,27667,39239|12732,4976,27456,30724|24000,50329,24050,51423|6373,14801,6474,14675|4252,4561,5016,5208|310,42327,1046,61661|337,3962,1356,5197|458,35160,1451,37204|289,3065,1140,5747|5246,5533,13003,14327|5033,5153,11724,12043|5076,7259,11562,49771|4422,4647,8638,9900&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|ZEOS+SQlite3|FireDAC+SQlite3|UniDAC+SQlite3|ZEOS+Firebird|FireDAC+Firebird|UniDAC+Firebird|Jet|Oracle|ODBC+Oracle|FireDAC+Oracle|UniDAC+Oracle|MSSQL+local|ODBC+MSSQL|FireDAC+MSSQL|UniDAC+MSSQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|UniDAC+MSSQL|FireDAC+MSSQL|ODBC+MSSQL|MSSQL+local|UniDAC+Oracle|FireDAC+Oracle|ODBC+Oracle|Oracle|Jet|UniDAC+Firebird|FireDAC+Firebird|ZEOS+Firebird|UniDAC+SQlite3|FireDAC+SQlite3|ZEOS+SQlite3|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,545732&chds=0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732,0,545732&chd=t:488,789,31376,88070,308584,308413,308,776,42213,98531,497,24992,469,12732,24000,6373,4252,310,337,458,289,5246,5033,5076,4422|463,815,35785,106981,545732,539548,12151,22404,182561,228634,12071,50065,8981,4976,50329,14801,4561,42327,3962,35160,3065,5533,5153,7259,4647|97498,101010,104410,106215,311837,316997,107469,111819,111642,112004,56489,21985,27667,27456,24050,6474,5016,1046,1356,1451,1140,13003,11724,11562,8638|126256,130561,136328,144270,535733,527537,170636,188316,197464,227489,72720,156887,39239,30724,51423,14675,5208,61661,5197,37204,5747,14327,12043,49771,9900&chdl=Direct|Batch|Trans|Batch+Trans" /></p>
<p>This feature is at ORM level, so it benefits to any external database
library.<br />
Of course, if a given library has a better option (e.g. our direct
<em>Oracle</em> or <em>FireDAC</em> array binding), it is used instead.</p>
<p>You can note that we included access to <em>Firebird embedded</em> via
<em>ODBC</em>, using <a href="http://www.firebirdsql.org/en/odbc-driver/">the
official driver</a>.<br />
And also <em>SQLite3</em> access via ODBC, using <a href="http://www.ch-werner.de/sqliteodbc/">this nice full-featured BSD licensed
driver</a>.<br />
Sounds like a not so optimized solution, e.g. in respect to <em>ZDBC/ZEOS</em>
direct connection.<br />
But nice show case of <em>ODBC</em> connection with <em>mORMot</em>.</p>
<p>Reading speed is not affected by this modification, so we won't publish new
data here.<br />
Note that now our native access to external databases outperforms any
third-party drivers, with the only exception of <em>Firebird</em>, which is
still most efficiently accessed via <em>FireDAC</em>.<br />
The <a href="http://synopse.info/files/pdf/Synopse%20mORMot%20Framework%20SAD%201.18.pdf">SAD
1.18 pdf</a> includes the latest benchmark.</p>
<p>If you want to use a <em>map/reduce</em> algorithm in your application, or
the <a href="https://blog.synopse.info?post/post/2014/01/04/Domain-Driven-Design%3A-part-4">DDD's <em>Unit Of
Work</em> pattern</a>, in addition to ORM data access, all those enhancements
may speed up a lot your process. Reading and writing huge amount of data has
never been so fast and easy: you may even be tempted to replace
stored-procedure process by high-level code implemented in your Domain service.
N-tier separation would benefit from it.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1642">welcome on our forum, as
usual</a>.</p>Updated mORMot database benchmark - including MS SQL and PostgreSQLurn:md5:a486d891f9eee1b7508aa972a6e827802013-11-04T18:46:00+01:002014-01-13T09:56:13+01:00AB4327-GANDImORMot FrameworkblogDatabaseDelphiDocumentationFireDACmORMotMSSQLmultithreadNexusDBODBCOleDBOracleORMperformanceRestSQLSQLite3SynDBTDataSetUnicodeUniDACZEOS<p>On an recent notebook computer (<em>Core i7</em> and SSD drive), depending
on the back-end database interfaced, <em>mORMot</em> excels in speed:</p>
<ul>
<li>You can persist up to 570,000 objects per second, or retrieve more than
900,000 objects per second (for our pure Delphi in-memory engine);</li>
<li>When data is retrieved <a href="https://blog.synopse.info?post/post/2012/02/14/ORM-cache">from server or
client cache</a>, you can read more than 900,000 objects per second, whatever
the database back-end is;</li>
<li>With a high-performance database like Oracle and our direct access classes,
you can write 65,000 (via array binding) and read 160,000 objects per second,
over a 100 MB network;</li>
<li>When using alternate database access libraries (e.g. Zeos, or
<code>DB.pas</code> based classes), speed is lower, but still enough for most
work.</li>
</ul>
<p>Difficult to find a faster ORM, I suspect.</p>
<p><img src="http://fc04.deviantart.net/fs70/f/2012/206/0/7/marmot_can_run_by_jaffa_tamarin-d58lk2i.jpg" alt="" width="450" height="251" /></p>
<p>The following tables try to sum up all available possibilities, and give
some benchmark (average objects/second for writing or read).</p>
<p>In these tables:</p>
<ul>
<li>'SQLite3 (file full/off/exc)' indicates use of the internal
<em>SQLite3</em> engine, <a href="https://blog.synopse.info?post/post/2013/06/14/SQLite3-performance-in-Exclusive-file-locking-mode">with or
without</a> <code>Synchronous := smOff</code> and/or <code>DB.LockingMode :=
lmExclusive</code>;</li>
<li>'SQLite3 (mem)' stands for the internal <em>SQLite3</em> engine running in
memory;</li>
<li>'SQLite3 (ext ...)' is about access to a <em>SQLite3</em> engine as
external database - either as file or memory;</li>
<li>'<code>TObjectList</code>' indicates a
<code>TSQLRestServerStaticInMemory</code> instance, either static (with no SQL
support) or virtual (i.e. SQL featured via <em>SQLite3</em> virtual table
mechanism) which may persist the data on disk as JSON or compressed
binary;</li>
<li>'Oracle' shows the results of our direct OCI access layer
(<code>SynDBOracle.pas</code>);</li>
<li>'NexusDB' is the free embedded edition, available from official site;</li>
<li>'Zeos *' indicates that the database was accessed directly via the ZDBC
layer;</li>
<li>'FireDAC *' stands for <em>FireDAC</em> library;</li>
<li>'UniDAC *' stands for <em>UniDAC</em> library;</li>
<li>'BDE *' when using a <em>BDE</em> connection;</li>
<li>'ODBC *' for a direct access to ODBC;</li>
<li>'Jet' stands for a <em>MSAccess</em> database engine, accessed via
OleDB;</li>
<li>'MSSQL local' for a local connection to a <em>MS SQL Express 2008 R2</em>
running instance (this was the version installed with <em>Visual Studio
2010</em>), accessed via OleDB.</li>
</ul>
<p>This list of database providers is to be extended in the future. Any
feedback is welcome!</p>
<p>Numbers are expressed in rows/second (or objects/second). This benchmark was
compiled with Delphi 7, so newer compilers may give even better results, with
in-lining and advanced optimizations.</p>
<p>Note that these tests are not about the relative speed of each database
engine, but reflect the current status of the integration of several DB
libraries within the <em>mORMot</em> database access.</p>
<p><strong>Purpose here is not to say that one library or database is better or
faster than another, but publish a snapshot of current <em>mORMot</em>
persistence layer abilities.</strong></p>
<p>In this timing, we do not benchmark only the "pure" SQL/DB layer access
(<code>SynDB</code> units), but the whole Client-Server ORM of our framework:
process below includes read and write RTTI access of a <code>TSQLRecord</code>,
JSON marshaling, CRUD/REST routing, virtual cross-database layer, SQL
on-the-fly translation. We just bypass the communication layer, since
<code>TSQLRestClient</code> and <code>TSQLRestServer</code> are run in-process,
in the same thread - as a <code>TSQLRestServerDB</code> instance. So you have
here some raw performance testimony of our framework's ORM and RESTful
core.</p>
<p>You can compile the "<code>15 - External DB performance</code>" supplied
sample code, and run the very same benchmark on your own configuration.</p> <h3>Insertion speed</h3>
<p>Here we insert 5,000 rows of data, with diverse scenarios:</p>
<ul>
<li>'Direct' stands for a individual <code>Client.Add()</code> insertion;</li>
<li>'Batch' mode, has <a href="https://blog.synopse.info?post/post/2011/06/03/BATCH-sequences-for-adding/updating/deleting-records">already
been described in this blog</a>;</li>
<li>'Trans' indicates that all insertion is nested within a transaction - which
makes a great difference, e.g. with a <em>SQlite3</em> database.</li>
</ul>
<p>Benchmark was run on a <em>Core i7</em> notebook, with standard SSD,
including anti-virus and background applications, over a 100 Mb corporate
network, linked to a shared <em>Oracle</em> 11g database. A local instance of
<em>MSSQLExpress 2008 R2</em> was running locally. So it was a development
environment, very similar to low-cost production site, not dedicated to give
best performance. During the process, CPU was noticeable used only for
<em>SQLite3</em> in-memory and <em>TObjectList</em> - most of the time, the
bottleneck is not the CPU, but the storage or network. As a result, rates and
timing may vary depending on network and server load, but you get results
similar to what could be expected on customer side, with an average hardware
configuration.</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>Direct</strong></td>
<td><strong>Batch</strong></td>
<td><strong>Trans</strong></td>
<td><strong>Batch Trans</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>483</td>
<td>473</td>
<td>78206</td>
<td>121592</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>879</td>
<td>843</td>
<td>100851</td>
<td>111167</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off exc)</strong></td>
<td>31703</td>
<td>35022</td>
<td>100276</td>
<td>133109</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>86961</td>
<td>114678</td>
<td>104177</td>
<td>138354</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>336700</td>
<td>562239</td>
<td>330425</td>
<td>547405</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>324296</td>
<td>565995</td>
<td>319795</td>
<td>554754</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>503</td>
<td>467</td>
<td>104574</td>
<td>136892</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>890</td>
<td>335</td>
<td>96420</td>
<td>135744</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>44634</td>
<td>50908</td>
<td>114064</td>
<td>154221</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>101578</td>
<td>137540</td>
<td>116563</td>
<td>155322</td>
</tr>
<tr align="center">
<td><strong>FireDAC SQlite3</strong></td>
<td>26660</td>
<td>53033</td>
<td>53320</td>
<td>151754</td>
</tr>
<tr align="center">
<td><strong>UniDAC SQlite3</strong></td>
<td>345</td>
<td>365</td>
<td>27390</td>
<td>37913</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>499</td>
<td>493</td>
<td>56790</td>
<td>66956</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>6535</td>
<td>13545</td>
<td>26531</td>
<td>30136</td>
</tr>
<tr align="center">
<td><strong>FireDAC Firebird</strong></td>
<td>25177</td>
<td>51917</td>
<td>24522</td>
<td>51880</td>
</tr>
<tr align="center">
<td><strong>UniDAC Firebird</strong></td>
<td>6081</td>
<td>7061</td>
<td>6338</td>
<td>7096</td>
</tr>
<tr align="center">
<td><strong>Jet</strong></td>
<td>4224</td>
<td>4390</td>
<td>4844</td>
<td>4984</td>
</tr>
<tr align="center">
<td><strong>NexusDB</strong></td>
<td>5998</td>
<td>6549</td>
<td>7668</td>
<td>8491</td>
</tr>
<tr align="center">
<td><strong>Oracle</strong></td>
<td>566</td>
<td>65183</td>
<td>1223</td>
<td>64413</td>
</tr>
<tr align="center">
<td><strong>ODBC Oracle</strong></td>
<td>589</td>
<td>482</td>
<td>1085</td>
<td>1328</td>
</tr>
<tr align="center">
<td><strong>BDE Oracle</strong></td>
<td>489</td>
<td>511</td>
<td>1024</td>
<td>1003</td>
</tr>
<tr align="center">
<td><strong>ZEOS Oracle</strong></td>
<td>599</td>
<td>657</td>
<td>1330</td>
<td>1464</td>
</tr>
<tr align="center">
<td><strong>FireDAC Oracle</strong></td>
<td>570</td>
<td>45405</td>
<td>1287</td>
<td>48686</td>
</tr>
<tr align="center">
<td><strong>UniDAC Oracle</strong></td>
<td>391</td>
<td>533</td>
<td>1135</td>
<td>1159</td>
</tr>
<tr align="center">
<td><strong>MSSQL local</strong></td>
<td>4619</td>
<td>5472</td>
<td>13373</td>
<td>14364</td>
</tr>
<tr align="center">
<td><strong>ODBC MSSQL</strong></td>
<td>5067</td>
<td>5200</td>
<td>11752</td>
<td>12258</td>
</tr>
<tr align="center">
<td><strong>FireDAC MSSQL</strong></td>
<td>4979</td>
<td>7521</td>
<td>11925</td>
<td>51100</td>
</tr>
<tr align="center">
<td><strong>UniDAC MSSQL</strong></td>
<td>4407</td>
<td>4467</td>
<td>8867</td>
<td>9936</td>
</tr>
<tr align="center">
<td><strong>ZEOS PostgreSQL</strong></td>
<td>2982</td>
<td>2994</td>
<td>6223</td>
<td>6028</td>
</tr>
<tr align="center">
<td><strong>FireDAC PostgreSQL</strong></td>
<td>2796</td>
<td>27638</td>
<td>1466</td>
<td>27171</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Batch+Trans|Trans|Batch|Direct&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,565995&chds=0,565995,0,565995,0,565995,0,565995,0,565995&chd=t:483,473,78206,121592|879,843,100851,111167|31703,35022,100276,133109|86961,114678,104177,138354|336700,562239,330425,547405|324296,565995,319795,554754|503,467,104574,136892|890,335,96420,135744|44634,50908,114064,154221|101578,137540,116563,155322|26660,53033,53320,151754|345,365,27390,37913|499,493,56790,66956|6535,13545,26531,30136|25177,51917,24522,51880|6081,7061,6338,7096|4224,4390,4844,4984|5998,6549,7668,8491|566,65183,1223,64413|589,482,1085,1328|489,511,1024,1003|599,657,1330,1464|570,45405,1287,48686|391,533,1135,1159|4619,5472,13373,14364|5067,5200,11752,12258|4979,7521,11925,51100|4407,4467,8867,9936&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|FireDAC+SQlite3|UniDAC+SQlite3|ZEOS+SQlite3|ZEOS+Firebird|FireDAC+Firebird|UniDAC+Firebird|Jet|NexusDB|Oracle|ODBC+Oracle|BDE+Oracle|ZEOS+Oracle|FireDAC+Oracle|UniDAC+Oracle|MSSQL+local|ODBC+MSSQL|FireDAC+MSSQL|UniDAC+MSSQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|UniDAC+MSSQL|FireDAC+MSSQL|ODBC+MSSQL|MSSQL+local|UniDAC+Oracle|FireDAC+Oracle|ZEOS+Oracle|BDE+Oracle|ODBC+Oracle|Oracle|NexusDB|Jet|UniDAC+Firebird|FireDAC+Firebird|ZEOS+Firebird|ZEOS+SQlite3|UniDAC+SQlite3|FireDAC+SQlite3|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,565995&chds=0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995,0,565995&chd=t:483,879,31703,86961,336700,324296,503,890,44634,101578,26660,345,499,6535,25177,6081,4224,5998,566,589,489,599,570,391,4619,5067,4979,4407|473,843,35022,114678,562239,565995,467,335,50908,137540,53033,365,493,13545,51917,7061,4390,6549,65183,482,511,657,45405,533,5472,5200,7521,4467|78206,100851,100276,104177,330425,319795,104574,96420,114064,116563,53320,27390,56790,26531,24522,6338,4844,7668,1223,1085,1024,1330,1287,1135,13373,11752,11925,8867|121592,111167,133109,138354,547405,554754,136892,135744,154221,155322,151754,37913,66956,30136,51880,7096,4984,8491,64413,1328,1003,1464,48686,1159,14364,12258,51100,9936&chdl=Direct|Batch|Trans|Batch+Trans" /></p>
<p>Due to its ACID implementation, <em>SQLite3</em> process on file waits for
the hard-disk to have finished flushing its data, therefore it is the reason
why it is slower than other engines at individual row insertion (less than 10
objects per second with a mechanical hardrive instead of a SDD) outside the
scope of a transaction.</p>
<p>So if you want to reach the best writing performance in your application
with the default engine, you should better use transactions and regroup all
writing into services or a BATCH process. Another possibility could be to
execute <code>DB.Synchronous := smOff</code> and/or <code>DB.LockingMode :=
lmExclusive</code> at <em>SQLite3</em> engine level before process: in case of
power loss at wrong time it may corrupt the database file, but it will increase
the rate by a factor of 50 (with hard drive), as stated by the "<em>off</em>"
and "<em>off exc</em>" rows of the table. Note that by default, the
<em>FireDAC</em> library set both options, so results above are to be compared
with "<em>SQLite3 off exc</em>" rows.</p>
<p>For both our direct Oracle access <code>SynDBOracle.pas</code> library and
<em>FireDAC</em>, Batch process benefit of the array binding feature a lot
(known as <em>Array DML</em> in <em>FireDAC/AnyDAC</em>).</p>
<h3>Reading speed</h3>
<p>Now the same data is retrieved via the ORM layer:</p>
<ul>
<li>'By one' states that one object is read per call (ORM generates a
<code>SELECT * FROM table WHERE ID=?</code> for <code>Client.Retrieve()</code>
method);</li>
<li>'All *' is when all 5000 objects are read in a single call (i.e. running
<code>SELECT * FROM table</code> from a <code>FillPrepare()</code> method
call), either forced to use the virtual table layer, or with direct static
call.</li>
</ul>
<p>Here are some reading speed values, in objects/second:</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>By one</strong></td>
<td><strong>All Virtual</strong></td>
<td><strong>All Direct</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>27403</td>
<td>551571</td>
<td>559346</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>26840</td>
<td>555062</td>
<td>557537</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off exc)</strong></td>
<td>122537</td>
<td>555185</td>
<td>541418</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>124750</td>
<td>551328</td>
<td>564206</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>299868</td>
<td>945537</td>
<td>935803</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>302114</td>
<td>433463</td>
<td>891583</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>131960</td>
<td>262660</td>
<td>541887</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>132268</td>
<td>264200</td>
<td>548666</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off exc)</strong></td>
<td>132886</td>
<td>265561</td>
<td>504693</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>133911</td>
<td>255036</td>
<td>534302</td>
</tr>
<tr align="center">
<td><strong>FireDAC SQlite3</strong></td>
<td>7749</td>
<td>79058</td>
<td>110338</td>
</tr>
<tr align="center">
<td><strong>UniDAC SQlite3</strong></td>
<td>2581</td>
<td>74839</td>
<td>98460</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>42336</td>
<td>208489</td>
<td>312617</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>20626</td>
<td>95496</td>
<td>129954</td>
</tr>
<tr align="center">
<td><strong>FireDAC Firebird</strong></td>
<td>2370</td>
<td>48066</td>
<td>57449</td>
</tr>
<tr align="center">
<td><strong>UniDAC Firebird</strong></td>
<td>2229</td>
<td>69845</td>
<td>90064</td>
</tr>
<tr align="center">
<td><strong>Jet</strong></td>
<td>2643</td>
<td>159576</td>
<td>258678</td>
</tr>
<tr align="center">
<td><strong>NexusDB</strong></td>
<td>1413</td>
<td>120845</td>
<td>208246</td>
</tr>
<tr align="center">
<td><strong>Oracle</strong></td>
<td>1464</td>
<td>123031</td>
<td>159459</td>
</tr>
<tr align="center">
<td><strong>ODBC Oracle</strong></td>
<td>1269</td>
<td>42528</td>
<td>49029</td>
</tr>
<tr align="center">
<td><strong>BDE Oracle</strong></td>
<td>860</td>
<td>3870</td>
<td>4036</td>
</tr>
<tr align="center">
<td><strong>ZEOS Oracle</strong></td>
<td>913</td>
<td>33167</td>
<td>34806</td>
</tr>
<tr align="center">
<td><strong>FireDAC Oracle</strong></td>
<td>1195</td>
<td>44039</td>
<td>53335</td>
</tr>
<tr align="center">
<td><strong>UniDAC Oracle</strong></td>
<td>657</td>
<td>26212</td>
<td>25770</td>
</tr>
<tr align="center">
<td><strong>MSSQL local</strong></td>
<td>10398</td>
<td>218818</td>
<td>425785</td>
</tr>
<tr align="center">
<td><strong>ODBC MSSQL</strong></td>
<td>13062</td>
<td>137241</td>
<td>256581</td>
</tr>
<tr align="center">
<td><strong>FireDAC MSSQL</strong></td>
<td>4119</td>
<td>76747</td>
<td>102984</td>
</tr>
<tr align="center">
<td><strong>UniDAC MSSQL</strong></td>
<td>2770</td>
<td>91431</td>
<td>135394</td>
</tr>
<tr align="center">
<td><strong>ZEOS PostgreSQL</strong></td>
<td>8005</td>
<td>6896</td>
<td>159933</td>
</tr>
<tr align="center">
<td><strong>FireDAC PostgreSQL</strong></td>
<td>2222</td>
<td>61156</td>
<td>81284</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|All+Direct|All+Virtual|By+one&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,945537&chds=0,945537,0,945537,0,945537&chd=t:27403,551571,559346|26840,555062,557537|122537,555185,541418|124750,551328,564206|299868,945537,935803|302114,433463,891583|131960,262660,541887|132268,264200,548666|132886,265561,504693|133911,255036,534302|7749,79058,110338|2581,74839,98460|42336,208489,312617|20626,95496,129954|2370,48066,57449|2229,69845,90064|2643,159576,258678|1413,120845,208246|1464,123031,159459|1269,42528,49029|860,3870,4036|913,33167,34806|1195,44039,53335|657,26212,25770|10398,218818,425785|13062,137241,256581|4119,76747,102984|2770,91431,135394&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28file+off+exc%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+mem%29|FireDAC+SQlite3|UniDAC+SQlite3|ZEOS+SQlite3|ZEOS+Firebird|FireDAC+Firebird|UniDAC+Firebird|Jet|NexusDB|Oracle|ODBC+Oracle|BDE+Oracle|ZEOS+Oracle|FireDAC+Oracle|UniDAC+Oracle|MSSQL+local|ODBC+MSSQL|FireDAC+MSSQL|UniDAC+MSSQL" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|UniDAC+MSSQL|FireDAC+MSSQL|ODBC+MSSQL|MSSQL+local|UniDAC+Oracle|FireDAC+Oracle|ZEOS+Oracle|BDE+Oracle|ODBC+Oracle|Oracle|NexusDB|Jet|UniDAC+Firebird|FireDAC+Firebird|ZEOS+Firebird|ZEOS+SQlite3|UniDAC+SQlite3|FireDAC+SQlite3|SQLite3+%28ext+mem%29|SQLite3+%28ext+off+exc%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off+exc%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x500&cht=bhg&chco=3D7930,3D8930,309F30,40C355&chxr=0,0,945537&chds=0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537,0,945537&chd=t:27403,26840,122537,124750,299868,302114,131960,132268,132886,133911,7749,2581,42336,20626,2370,2229,2643,1413,1464,1269,860,913,1195,657,10398,13062,4119,2770|551571,555062,555185,551328,945537,433463,262660,264200,265561,255036,79058,74839,208489,95496,48066,69845,159576,120845,123031,42528,3870,33167,44039,26212,218818,137241,76747,91431|559346,557537,541418,564206,935803,891583,541887,548666,504693,534302,110338,98460,312617,129954,57449,90064,258678,208246,159459,49029,4036,34806,53335,25770,425785,256581,102984,135394&chdl=By+one|All+Virtual|All+Direct" /></p>
<p>The <em>SQLite3</em> layer gives amazing reading results, which makes it a
perfect fit for most typical ORM use. When running with <code>DB.LockingMode :=
lmExclusive</code> defined (i.e. "off exc" rows), reading speed is very high,
and benefits from exclusive access to the database file. External database
access is only required when data is expected to be shared with other
processes.</p>
<p>In the above table, it appears that all libraries based on
<code>DB.pas</code> are slower than the others for reading speed. In fact,
<code>TDataSet</code> sounds to be a real bottleneck, due to its internal data
marshalling. Even <em>FireDAC</em>, which is known to be very optimized for
speed, is limited by the <code>TDataSet</code> structure. Our direct classes,
or even ZEOS/ZDBC performs better, since they are able to output JSON content
with no additional marshalling.</p>
<p>For both writing and reading, <code>TObjectList</code> /
<code>TSQLRestServerStaticInMemory</code> engine gives impressive results, but
has the weakness of being in-memory, so it is not ACID by design, and the data
has to fit in memory. Note that indexes are available for IDs and <code>stored
AS_UNIQUE</code> properties.</p>
<p>As a consequence, search of non-unique values may be slow: the engine has to
loop through all rows of data. But for unique values (defined as <code>stored
AS_UNIQUE</code>), both insertion and search speed is awesome, due to its
optimized O(1) hash algorithm - see the following benchmark, especially the
"<em>By name</em>" row for "<em>TObjectList</em>" columns, which correspond to
a search of an unique <code>RawUTF8</code> property value via this hashing
method.</p>
<table>
<tbody>
<tr>
<td><strong>SQLite3 (file full)</strong></td>
<td><strong>SQLite3 (file off)</strong></td>
<td><strong>SQLite3 (mem)</strong></td>
<td><strong>TObjectList (static)</strong></td>
<td><strong>TObjectList (virt.)</strong></td>
<td><strong>SQLite3 (ext file full)</strong></td>
<td><strong>SQLite3 (ext file off)</strong></td>
<td><strong>SQLite3 (ext mem)</strong></td>
<td><strong>Oracle</strong></td>
<td><strong>Jet</strong></td>
</tr>
<tr>
<td><strong>By one</strong></td>
<td>10461</td>
<td>10549</td>
<td>44737</td>
<td>103577</td>
<td>103553</td>
<td>43367</td>
<td>44099</td>
<td>45220</td>
<td>901</td>
<td>1074</td>
</tr>
<tr>
<td><strong>By name</strong></td>
<td>9694</td>
<td>9651</td>
<td>32350</td>
<td>70534</td>
<td>60153</td>
<td>22785</td>
<td>22240</td>
<td>23055</td>
<td>889</td>
<td>1071</td>
</tr>
<tr>
<td><strong>All Virt.</strong></td>
<td>167095</td>
<td>162956</td>
<td>168651</td>
<td>253292</td>
<td>118203</td>
<td>97083</td>
<td>90592</td>
<td>94688</td>
<td>56639</td>
<td>52764</td>
</tr>
<tr>
<td><strong>All Direct</strong></td>
<td>167123</td>
<td>144250</td>
<td>168577</td>
<td>254284</td>
<td>256383</td>
<td>170794</td>
<td>165601</td>
<td>168856</td>
<td>88342</td>
<td>75999</td>
</tr>
</tbody>
</table>
<p>Above table results were run on a Core 2 duo laptop, so numbers are lower
than with the previous tables.</p>
<h3>Analysis and use case proposal</h3>
<p>When declared as virtual table (via a <code>VirtualTableRegister</code>
call), you have the full power of SQL (including JOINs) at hand, with
incredibly fast CRUD operations: 100,000 requests per second for objects read
and write, including serialization and Client-Server communication!</p>
<p>Some providers are first-class citizens to <em>mORMot</em>, like
<em>SQLite3</em>, <em>Oracle</em>, or <em>MS SQL</em>. You can connect to them
without the bottleneck of the <code>DB.pas</code> unit, nor any restriction of
your Delphi license (a <em>Starter edition</em> is enough). For instance,
<em>SQLite3</em> could be used as main database engine for a client-server
application with heavy concurrent access - if you have doubts about its scaling
abilities, <a href="https://blog.synopse.info?post/post/2013/09/10/Thread-safety-of-mORMot">see this blog
article</a>. Direct access to <em>Oracle</em> is also available, with
impressive results in BATCH mode (aka array binding). <em>MS SQL Server</em>,
directly accessed via <em>OleDB</em> (or <em>ODBC</em>) gives pretty good
timing, and a <em>MS SQL Server 2008 R2 Express</em> instance is a convincing
option, for a very offerdable price (i.e. for free) - the <em>LocalDB</em> (MSI
installer) edition is enough to start with. Any other OleDB, ODBC or ZDBC
providers may also be used, with direct access. For instance, <em>Firebird</em>
embedded gives pretty consistent timing, when accessed via Zeos/ZDBC.</p>
<p>But <em>mORMot</em> is very open-minded: you can use any <code>DB.pas</code>
provider, e.g. <em>FireDAC</em>, <em>UniDAC</em>, <em>DBExpress</em>,
<em>NexusDB</em> or even the <em>BDE</em>, but with the additional layer
introduced by using a <code>TDataSet</code> instance, at reading.</p>
<p>Note that all those tests were performed locally and in-process, via a
<code>TSQLRestClientDB</code> instance. For both insertion and reading, a
Client-Server architecture (e.g. using HTTP/1.1 for <em>mORMot</em> clients)
will give even better results for BATCH and retrieve all modes.</p>
<p>During the tests, internal caching was disabled, so you may expect speed
enhancements for real applications, when data is more read than written: for
instance, when an object is retrieved from the cache, you achieve more than
700,000 read requests per second, whatever database is used.</p>
<p>Therefore, the typical use may be the following:</p>
<table>
<tbody>
<tr>
<td><strong>Database</strong></td>
<td><strong>Created by</strong></td>
<td><strong>Use</strong></td>
</tr>
<tr>
<td>int. SQLite3 file</td>
<td>default</td>
<td>General safe data handling, with amazing speed in "off exc" mode</td>
</tr>
<tr>
<td>int. SQLite3 mem</td>
<td><code>:memory:</code></td>
<td>Fast data handling with no persistence (e.g. for testing or temporary
storage)</td>
</tr>
<tr>
<td><code>TObjectList</code> static</td>
<td><code>StaticDataCreate</code></td>
<td>Best possible performance for small amount of data, without ACID nor
SQL</td>
</tr>
<tr>
<td><code>TObjectList</code> virtual</td>
<td><code>VirtualTableRegister</code></td>
<td>Best possible performance for small amount of data, if ACID is not required
nor complex SQL</td>
</tr>
<tr>
<td>ext. SQLite3 file</td>
<td><code>VirtualTableExternalRegister</code></td>
<td>External back-end, e.g. for disk spanning</td>
</tr>
<tr>
<td>ext. SQLite3 mem</td>
<td><code>VirtualTableExternalRegister</code></td>
<td>Fast external back-end (e.g. for testing)</td>
</tr>
<tr>
<td>ext. Oracle / MS SQL / Firebird</td>
<td><code>VirtualTableExternalRegister</code></td>
<td>Fast, secure and industry standard; data can be shared outside
<em>mORMot</em></td>
</tr>
<tr>
<td>ext. NexusDB</td>
<td><code>VirtualTableExternalRegister</code></td>
<td>The free embedded version let the whole engine be included within your
executable, and insertion speed is higher than <em>SQLite3</em>, so it may be a
good alternative if your project mostly insert individual objects - using a
batch within a transaction let <em>SQlite3</em> be the faster engine</td>
</tr>
<tr>
<td>ext. Jet</td>
<td><code>VirtualTableExternalRegister</code></td>
<td>Could be used as a data exchange format (e.g. with Office
applications)</td>
</tr>
<tr>
<td>ext. Zeos/FireDAC/UniDAC</td>
<td><code>VirtualTableExternalRegister</code></td>
<td>Allow access to several external engines, with some advantages for Zeos,
since direct ZDBC access will by-pass the <code>DB.pas</code> unit and its
<code>TDataSet</code> bottleneck - and we will also prefer an active Open
Source project!</td>
</tr>
</tbody>
</table>
<p>Whatever database back-end is used, don't forget that <em>mORMot</em> design
will allow you to switch from one library to another, just by changing a
<code>TSQLDBConnectionProperties</code> class type. And note that you can
<em>mix</em> external engines, on purpose: you are not tied to one single
engine, but the database access can be tuned for each ORM table, according to
your project needs.</p>TDataSet... now I'm confusedurn:md5:2b1db7fa4b9a342600161a4150d23d7a2013-04-22T07:34:00+02:002013-04-22T07:13:54+02:00AB4327-GANDIPascal ProgrammingblogDatabaseDelphiRADTDataSetUnicodeUserInterface<p>You perhaps know that I'm not a big fan of the <code>TDataSet</code> /
RAD DB approach for end-user applications.<br />
They are easy to define, almost no code to write, and you are able to publish a
working solution very fast.</p>
<p>But it is a nightmare to debug and maintain. I prefer the new
<em>DataBinding</em> feature, or... of course... ORM!<br />
In <em>mORMot</em>, we have some auto-generated screens, and in our roadmap, we
forcast to use some auto-binding features, using a KISS by-convention MVC
pattern.</p>
<p>For some users, we made a ORM / <code>TDataSet</code> conversion unit.<br />
And we discovered that <code>TDataSet</code> has a weird, and very
misleading definition of its <code>AsString</code> property, for Unicode
versions of Delphi.</p> <p>In the Delphi 2009+ implementation, you have to use <code>AsString</code>
property for <code>AnsiString</code> and <code>AsWideString</code> for
<code>string=UnicodeString</code>.</p>
<p>In fact, the <code>As*String</code> properties are defined as such:</p>
<pre>
<code>property AsString: string read GetAsString write SetAsString;
property AsWideString: UnicodeString read GetAsWideString write SetAsWideString;
property AsAnsiString: AnsiString read GetAsAnsiString write SetAsAnsiString;
</code>
</pre>
<p>How on earth <a href="http://stackoverflow.com/q/9459186/458259">may we be
able to find out</a> that <code>AsString: string </code>returns in fact an
<code>AnsiString</code>?<br />
It just does not make sense at all, when compared to the rest of the
VCL/RTL.</p>
<p>The implementation, which uses <code>TStringField</code> class for
<code>AnsiString</code> and <code>TWideStringField</code> for
<code>string=UnicodeString</code> just appear to be broken.</p>
<p>Furthermore, <a href="http://docwiki.embarcadero.com/Libraries/XE2/en/Data.DB.TField.AsString">the
documentation is also broken</a>:</p>
<blockquote>
<p><em>Data.DB.TField.AsString<br />
Represents the field's value as a string (Delphi) or an AnsiString
(C++).</em></p>
</blockquote>
<p>This does not represent a <code>string</code> in Delphi, but an
<code>AnsiString</code>!<br />
The fact that the property uses a plain <code>string=UnicodeString</code> type
is perfectly misleading.</p>
<p>From the database point of view, it is up to the DB driver to handle Unicode
or work with a specific charset.<br />
But from the VCL point of view, in Delphi 2009+ you should only know about one
<code>string</code> type, and be confident that using <code>AsString:
String</code> will be Unicode-ready.</p>
<p>If you use our <code>mORMotVCL.pas</code> unit, <a href="http://synopse.info/fossil/info/c6ef9c1d0f">it will behave as
expected</a>.<br />
Thanks you <a href="http://synopse.info/forum/profile.php?id=858">sjerinic</a>
for your input and patience about this issue!<br />
Feedback <a href="http://synopse.info/forum/viewtopic.php?pid=7452#p7452">is
welcome</a>!</p>Introducing ZEOS, UniDAC, NexusDB, BDE, any TDataset to SynDB and mORMot's ORMurn:md5:13965203686258c4c2f7ebb6effed3c52013-02-12T17:42:00+01:002013-05-17T09:10:56+02:00AB4327-GANDImORMot FrameworkDatabaseDelphiJSONmORMotMSSQLNexusDBODBCOleDBOracleORMperformanceRepositoryRestSourceSQLSQLite3SynDBTDataSettransactionUniDACZEOS<p>Up to now, our <code>SynDB</code> database classes were handling
<em>ODBC</em>, <em>OleDB</em> providers and direct <em>Oracle</em> or
<em>SQLite3</em> connection.</p>
<p>We have added a <code>DB.pas</code> based layer, ready to be used with
<em><a href="http://www.devart.com/unidac/">UniDAC</a></em>, <em><a href="http://www.nexusdb.com">NexusDB</a></em>, or the <em>BDE</em>.<br />
Any other <code>TDataset</code> based component is ready to be interfaced,
including <em>UIB</em>, <em>AnyDAC</em> or <em>DBExpress</em>.</p>
<p><img src="https://blog.synopse.info?post/public/mORMot/mORMotDatabase.png" alt="" title="mORMot Database, avr. 2013" /></p>
<p>The <em>ZEOS</em> library (in its latest <a href="http://sourceforge.net/projects/zeoslib/files/Zeos%20Database%20Objects/zeosdbo-7.0.3-stable/">7.0.3
stable version</a>, which works from Delphi 7 up to XE3) has also been
interfaced, but without the <code>TDataset</code>/<code>DB.pas</code> layer:
our <code>SynDBZEOS.pas</code> unit calls the ZDBC layer, which is not
tied to <code>DB.pas</code> nor its RAD components, and is therefore
faster. By the way, it will work also with the <em>Starter edition</em> of
Delphi (which does not include the DB components) - just like the other
"regular" <code>SynDB</code> classes.</p>
<p><img src="https://blog.synopse.info?post/public/mORMot/.641-03461407w_m.jpg" alt="" title="mORMot photo, fév. 2013" /></p>
<p>This is a work in progress, any testing and feedback is welcome!<br />
We had to circumvent some particularities of the libraries, but I guess we have
something interesting.</p>
<p>A dedicated "<em>SynDBDataset</em>" sub-folder has been created <a href="http://synopse.info/fossil/dir?ci=tip">in the repository</a>, to contain all
<code>SynDBDataset.pas</code>-based database providers.<br />
<code>SynDBNexusDB.pas</code> unit has been moved within this sub-folder,
as <code>SynDBUniDAC.pas</code> + <code>SynDBBDE.pas</code> units have
been added.<br />
<code>SynDBZeos.pas</code> has a direct access to the <em>ZDBC</em> layer, so
is not part of the "<em>SynDBDataset</em>" sub-folder.</p>
<p>Here is some benchmark, mainly about <em>Oracle</em> and
<em>SQlite3</em> database access.<br />
Of course, our direct <code>SynDBOracle</code> / <code>SynDBSQLite3</code>
layers are the fastest around, and we can see that ZDBC layer is sometimes more
efficient than the <code>TDataset</code> components.</p> <p>This preliminary results came from the <a href="https://blog.synopse.info?post/post/2013/01/28/External-database-speed-improvements">same hardware than
previously used on this blog</a>.<br />
In short: notebook with a SSD and Core i7 mobile.</p>
<p>Note that these tests are not about the relative speed of each database
engine, but reflect the current status of the integration of several DB
libraries within the <em>mORMot</em> database layer.<br />
Purpose here is not to say that one library is better or faster than another,
but publish a snapshot of <em>mORMot</em> persistence layer abilities.<br />
In this timing, we do not benchmark only the "pure" SQL-DB layer access
(<code>SynDB*</code> units), but the whole Client-Server ORM of our framework:
process below includes read and write RTTI access of a <code>TSQLRecord</code>,
JSON marshalling, CRUD/REST routing, virtual cross-database layer, SQL
on-the-fly translation. We just bypass the communication layer, since
<code>TSQLRestClient</code> and <code>TSQLRestServer</code> are run in-process,
in the same thread - as a <code>TSQLRestServerDB</code> instance. So you have
here some raw performance testimony of our framework's ORM and RESTful
core.</p>
<p><em>Jet/MSAccess</em> database engine is accessed via OleDB.<br />
<em>NexusDB</em> is the <a href="http://www.nexusdb.com/support/index.php?q=FreeEmbedded">free embedded
edition</a>, which is a perfect match for a Client-Server ORM framework like
<em>mORMot</em>.<br />
<em>Oracle</em> server is a remote 11g instance, accessed with a "slow" 100 MB
network.<br />
<em>SQLite3</em> is our static embedded engine, either as internal or external
database, or the <a href="http://www.sqlite.org/download.html">latest 3.7.15.2
official</a> <code>sqlite3.dll</code> library, for <em>ZEOS</em> and
<em>UniDAC</em>.<br />
<em>TObjectList</em> is our in-memory optimized engine, with hashed indexes,
and able to persist its data as JSON file or an optimized and compressed binary
content.</p>
<h3>Insertion speed</h3>
<p>We use the sample provided within the source code tree, i.e. "15 - External
DB performance".</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>Direct</strong></td>
<td><strong>Batch</strong></td>
<td><strong>Trans</strong></td>
<td><strong>Batch Trans</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>536</td>
<td>525</td>
<td>93687</td>
<td>113527</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>941</td>
<td>610</td>
<td>94222</td>
<td>98590</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>84722</td>
<td>104727</td>
<td>100425</td>
<td>126974</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>298560</td>
<td>423657</td>
<td>300951</td>
<td>385089</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>298721</td>
<td>445473</td>
<td>299293</td>
<td>444207</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>262</td>
<td>259</td>
<td>94916</td>
<td>120293</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>337</td>
<td>330</td>
<td>103941</td>
<td>112336</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>95418</td>
<td>119442</td>
<td>110238</td>
<td>143135</td>
</tr>
<tr align="center">
<td><strong>UniDAC SQlite3</strong></td>
<td>239</td>
<td>253</td>
<td>7916</td>
<td>38696</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>372</td>
<td>374</td>
<td>362</td>
<td>467</td>
</tr>
<tr align="center">
<td><strong>Oracle</strong></td>
<td>538</td>
<td>62295</td>
<td>1228</td>
<td>62156</td>
</tr>
<tr align="center">
<td><strong>ODBC Oracle</strong></td>
<td>639</td>
<td>605</td>
<td>1743</td>
<td>1606</td>
</tr>
<tr align="center">
<td><strong>ZEOS Oracle</strong></td>
<td>441</td>
<td>424</td>
<td>1109</td>
<td>1140</td>
</tr>
<tr align="center">
<td><strong>UniDAC Oracle</strong></td>
<td>528</td>
<td>557</td>
<td>1061</td>
<td>1237</td>
</tr>
<tr align="center">
<td><strong>BDE Oracle</strong></td>
<td>489</td>
<td>511</td>
<td>1024</td>
<td>1003</td>
</tr>
<tr align="center">
<td><strong>Jet</strong></td>
<td>4243</td>
<td>4442</td>
<td>4924</td>
<td>4982</td>
</tr>
<tr align="center">
<td><strong>NexusDB</strong></td>
<td>4697</td>
<td>6711</td>
<td>5723</td>
<td>8786</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>1791</td>
<td>1870</td>
<td>6522</td>
<td>7213</td>
</tr>
<tr align="center">
<td><strong>UniDAC Firebird</strong></td>
<td>3670</td>
<td>7135</td>
<td>4751</td>
<td>10125</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|Batch+Trans|Trans|Batch|Direct&chxt=x,y&chbh=a&chs=600x400&cht=bhg&chco=3D7930,3D8930,309F30,6070F0,5070E0,40C355,65D055,80C1A2,3D7930,3D8930,F05050,F04050,F04040,F01040,F0A280&chxr=0,0,445473&chds=0,445473,0,445473,0,445473,0,445473,0,445473&chd=t:536,525,93687,113527|941,610,94222,98590|84722,104727,100425,126974|298560,423657,300951,385089|298721,445473,299293,444207|262,259,94916,120293|337,330,103941,112336|95418,119442,110238,143135|239,253,7916,38696|372,374,362,467|538,62295,1228,62156|639,605,1743,1606|441,424,1109,1140|528,557,1061,1237|489,511,1024,1003|4243,4442,4924,4982|4697,6711,5723,8786|1791,1870,6522,7213|3670,7135,4751,10125&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+mem%29|UniDAC+SQlite3|ZEOS+SQlite3|Oracle|ODBC+Oracle|ZEOS+Oracle|UniDAC+Oracle|BDE+Oracle|Jet|NexusDB|ZEOS+Firebird|UniDAC+Firebird" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Insertion+speed+%28rows%2Fsecond%29&chxl=1:|UniDAC+Firebird|ZEOS+Firebird|NexusDB|Jet|BDE+Oracle|UniDAC+Oracle|ZEOS+Oracle|ODBC+Oracle|Oracle|ZEOS+SQlite3|UniDAC+SQlite3|SQLite3+%28ext+mem%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x400&cht=bhg&chco=3D7930,3D8930,309F30,6070F0,5070E0,40C355,65D055,80C1A2,3D7930,3D8930,F05050,F04050,F04040,F01040,F0A280&chxr=0,0,445473&chds=0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473,0,445473&chd=t:536,941,84722,298560,298721,262,337,95418,239,372,538,639,441,528,489,4243,4697,1791,3670|525,610,104727,423657,445473,259,330,119442,253,374,62295,605,424,557,511,4442,6711,1870,7135|93687,94222,100425,300951,299293,94916,103941,110238,7916,362,1228,1743,1109,1061,1024,4924,5723,6522,4751|113527,98590,126974,385089,444207,120293,112336,143135,38696,467,62156,1606,1140,1237,1003,4982,8786,7213,10125&chdl=Direct|Batch|Trans|Batch+Trans" /></p>
<p><em>NexusDB</em> has a pretty good insertion speed, but no "BATCH" mode, nor
prepared statement re-use. But not bad for such a small unit.<br />
<em>ZEOS</em> also lacks of some optimizations like prepared statement cache,
but does work well for an Open Source free solution.</p>
<h3>Read speed</h3>
<p>Read speed is comparable to insertion speed.<br />
Depending on the design of each library, results are similar.</p>
<p><em>BDE</em> performance is not so good, indeed. When working with Oracle,
it is pretty slow, especially when retrieving data.<br />
But it (still) works!</p>
<p>Both <em>Jet/MSAccess</em> and <em>NexusDB</em> are pretty good, as embedded
engines, for reading.<br />
Of course, our direct <em>SQlite3</em> or in-memory <code>TObjectList</code>
engines are incredibly fast. Just as usual.</p>
<table>
<tbody>
<tr align="center">
<td> </td>
<td><strong>By one</strong></td>
<td><strong>All Virtual</strong></td>
<td><strong>All Direct</strong></td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file full)</strong></td>
<td>26755</td>
<td>435995</td>
<td>440683</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (file off)</strong></td>
<td>26527</td>
<td>435995</td>
<td>438519</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (mem)</strong></td>
<td>126253</td>
<td>438711</td>
<td>423155</td>
</tr>
<tr align="center">
<td><strong>TObjectList (static)</strong></td>
<td>289955</td>
<td>689369</td>
<td>681477</td>
</tr>
<tr align="center">
<td><strong>TObjectList (virtual)</strong></td>
<td>299383</td>
<td>234126</td>
<td>707714</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext full)</strong></td>
<td>122524</td>
<td>202208</td>
<td>356760</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext off)</strong></td>
<td>137358</td>
<td>228102</td>
<td>432376</td>
</tr>
<tr align="center">
<td><strong>SQLite3 (ext mem)</strong></td>
<td>139209</td>
<td>228738</td>
<td>432226</td>
</tr>
<tr align="center">
<td><strong>UniDAC SQlite3</strong></td>
<td>2001</td>
<td>74155</td>
<td>96140</td>
</tr>
<tr align="center">
<td><strong>ZEOS SQlite3</strong></td>
<td>3486</td>
<td>81959</td>
<td>115856</td>
</tr>
<tr align="center">
<td><strong>Oracle</strong></td>
<td>1753</td>
<td>77429</td>
<td>92423</td>
</tr>
<tr align="center">
<td><strong>ODBC Oracle</strong></td>
<td>1909</td>
<td>36789</td>
<td>42530</td>
</tr>
<tr align="center">
<td><strong>ZEOS Oracle</strong></td>
<td>601</td>
<td>47028</td>
<td>55233</td>
</tr>
<tr align="center">
<td><strong>UniDAC Oracle</strong></td>
<td>653</td>
<td>25960</td>
<td>28083</td>
</tr>
<tr align="center">
<td><strong>BDE Oracle</strong></td>
<td>860</td>
<td>3870</td>
<td>4036</td>
</tr>
<tr align="center">
<td><strong>Jet</strong></td>
<td>2605</td>
<td>150051</td>
<td>231792</td>
</tr>
<tr align="center">
<td><strong>NexusDB</strong></td>
<td>1360</td>
<td>128816</td>
<td>204976</td>
</tr>
<tr align="center">
<td><strong>ZEOS Firebird</strong></td>
<td>2508</td>
<td>56771</td>
<td>71822</td>
</tr>
<tr align="center">
<td><strong>UniDAC Firebird</strong></td>
<td>1850</td>
<td>66882</td>
<td>88021</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|All+Direct|All+Virtual|By+one&chxt=x,y&chbh=a&chs=600x400&cht=bhg&chco=3D7930,3D8930,309F30,6070F0,5070E0,40C355,65D055,80C1A2,3D7930,3D8930,F05050,F04050,F04040,F01040,F0A280&chxr=0,0,707714&chds=0,707714,0,707714,0,707714&chd=t:26755,435995,440683|26527,435995,438519|126253,438711,423155|289955,689369,681477|299383,234126,707714|122524,202208,356760|137358,228102,432376|139209,228738,432226|2001,74155,96140|3486,81959,115856|1753,77429,92423|1909,36789,42530|601,47028,55233|653,25960,28083|860,3870,4036|2605,150051,231792|1360,128816,204976|2508,56771,71822|1850,66882,88021&chdl=SQLite3+%28file+full%29|SQLite3+%28file+off%29|SQLite3+%28mem%29|TObjectList+%28static%29|TObjectList+%28virtual%29|SQLite3+%28ext+full%29|SQLite3+%28ext+off%29|SQLite3+%28ext+mem%29|UniDAC+SQlite3|ZEOS+SQlite3|Oracle|ODBC+Oracle|ZEOS+Oracle|UniDAC+Oracle|BDE+Oracle|Jet|NexusDB|ZEOS+Firebird|UniDAC+Firebird" /></p>
<p><img src="http://chart.apis.google.com/chart?chtt=Read+speed+%28rows%2Fsecond%29&chxl=1:|UniDAC+Firebird|ZEOS+Firebird|NexusDB|Jet|BDE+Oracle|UniDAC+Oracle|ZEOS+Oracle|ODBC+Oracle|Oracle|ZEOS+SQlite3|UniDAC+SQlite3|SQLite3+%28ext+mem%29|SQLite3+%28ext+off%29|SQLite3+%28ext+full%29|TObjectList+%28virtual%29|TObjectList+%28static%29|SQLite3+%28mem%29|SQLite3+%28file+off%29|SQLite3+%28file+full%29&chxt=x,y&chbh=a&chs=600x400&cht=bhg&chco=3D7930,3D8930,309F30,6070F0,5070E0,40C355,65D055,80C1A2,3D7930,3D8930,F05050,F04050,F04040,F01040,F0A280&chxr=0,0,707714&chds=0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714,0,707714&chd=t:26755,26527,126253,289955,299383,122524,137358,139209,2001,3486,1753,1909,601,653,860,2605,1360,2508,1850|435995,435995,438711,689369,234126,202208,228102,228738,74155,81959,77429,36789,47028,25960,3870,150051,128816,56771,66882|440683,438519,423155,681477,707714,356760,432376,432226,96140,115856,92423,42530,55233,28083,4036,231792,204976,71822,88021&chdl=By+one|All+Virtual|All+Direct" /></p>
<p>We are working on direct integration of <em>Firebird</em> (either directly
via an unfinished <code>SynDBFirebird.pas</code> unit, or via <em>ZEOS</em> /
<em>Unidac</em> / any <code>TDataset</code> based component), but we are still
fighting with some configuration issues.<br />
Any help is welcome. Don't be shy!</p>
<p>The SAD <em>pdf</em> (<a href="http://synopse.info/files/pdf/Synopse%20mORMot%20Framework%20SAD%201.18.pdf">in
its 1.18 revision</a>) has been updated to include information about external
database connection via those libraries.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1094">welcome on our forum</a>.</p>SQLite3-powered, not SQLite3-limitedurn:md5:5ba60f94b97a08886170fe535b366e3e2012-07-24T09:46:00+02:002013-04-28T08:46:48+02:00AB4327-GANDImORMot FrameworkDatabaseDelphiDocumentationmORMotMSSQLNexusDBODBCOleDBOracleORMperformanceRestSQLSQLite3SynDBTDataSetTObjectListUniDACVirtualTableZEOS<p>Our <a href="http://synopse.info/fossil/wiki?name=Downloads">downloadable
documentation has been enhanced</a>, and contains now a description about the
main feature of 1.15 version, i.e. "database agnosticism".</p>
<p><img src="http://asd.gsfc.nasa.gov/archive/arcade/images/marmot_soldering_iron_small.jpg" alt="" width="380" height="285" /></p>
<p>The core database of our <em>mORMot</em> framework uses the <em>SQLite3</em>
library, which is a Free, Secure, Zero-Configuration, Server-less, Single
Stable Cross-Platform Database File database engine.</p>
<p>As stated below, you can use any other database access layer, if you
wish.<br />
A fast in-memory engine (<code>TObjectList</code>-based) is included, and can
be used instead or together with the <em>SQLite3</em> engine.<br />
Or you may be able to access any remote database, and use one or more
<em>OleDB</em>, <em>ODBC</em>, <em>ZDBC</em>, <code>TDataSet</code>, (or
direct <em>Oracle</em>) connections to store your precious ORM
objects.<br />
The <em>SQlite3</em> will be used as the main SQL engine, able to JOIN all
those tables, thanks to its Virtual Table unique feature.</p>
<p><strong>(article updated after removal of the
<code>TSQLRecordExternal</code> class type for revision 1.17 - note also that
BATCH process is now directly supported by the framework and converted to
<a href="https://blog.synopse.info?post/post/2012/07/19/Oracle-Array-Binding-and-BATCH-performance">bound
array parameters</a> if available)</strong></p> <h3>Database agnosticism</h3>
<p>Since revision 1.15, our ORM RESTful framework is able to access any
available database engine, via a set of generic units and classes.</p>
<p>The framework still relies on <em>SQLite3</em> as its SQL core on the
server, but a dedicated mechanism allows access to any remote database, and mix
those tables content with the native ORM tables of the framework.<br />
Thanks to the unique <a href="http://www.sqlite.org/vtab.html">Virtual Tables
mechanism of <em>SQLite3</em></a>, those external tables may be accessed as
native <em>SQLite3</em> tables in our SQL statements.</p>
<p><img src="https://blog.synopse.info?post/public/mORMot/mORMotDatabase.png" alt="" title="mORMot Database, avr. 2013" /></p>
<p>The current list of available external database classes is:<br />
- Any <em>OleDB</em> provider (including <em>MS SQL Server, Jet</em> or
others);<br />
- Any <em>ODBC</em> provider (including <em>FireBird, MySQL</em>, or
others);<br />
- Any <em>ZeosLib</em> provider (direct <code>ZDBC</code> access);<br />
- Any <code>DB.pas</code> / <code>TDataset</code> based provider (including
NexusDB, DBExpress, AnyDac, UniDAC, BDE...);<br />
- <em>Oracle</em> direct access (via OCI);<br />
- <em>SQLite3</em> database file.</p>
<p>This list is not closed, and may be completed in the near future. Any help
is welcome here: it is not difficult to implement a new unit, following the
patterns already existing. You may start from an existing driver (e.g.
<em>Zeos</em> or <em>Alcinoe</em> libraries). Open Source contribution are
always welcome!</p>
<p><a>In fact, OleDB is a good candidate for database access with good
performance, Unicode native, with a lot of available providers. Thanks to
OleDB, we are already able to access to almost any existing database. The code
overhead in the server executable will also be much less than with adding any
other third-party Delphi library. And we will let Microsoft or the OleDB
provider perform all the testing and debugging for each driver.<br />
ODBC is the new standard, since</a> <a href="https://blog.synopse.info?post/post/2012/02/29/Microsoft-states%3A-OleDB-out-enjoy-ODBC%21">Microsoft is
deprecating OleDB</a>.</p>
<p>Since revision 1.18, any <em>ZeosLib</em> / <em>ZDBC</em> driver can be used
and <code>DB.pas</code> can be used with our <code>SynDB</code> classes. Of
course, using <code>TDataset</code> as intermediate layer will be slower than
the <code>SynDB</code> direct access pattern. But it will allow you to re-use
any existing (third-party) database connection driver, which could make sense
in case of evolution of an existing application, or to use an unsupported
database engine.</p>
<p>An <em>Oracle</em> dedicated direct access was added, because all available
OleDB providers for Oracle (i.e. both Microsoft's and Oracle's) do have
problems with handling BLOB, and we wanted our Clients to have a light-weight
and as fast as possible access to this great database.</p>
<p>Thanks to the design of our classes, it was very easy (and convenient) to
implement <em>SQLite3</em> direct access. It is even used for our regression
tests, in order to implement stand-alone unitary testing.</p>
<h3>ORM Integration</h3>
<p>An <em>external</em> record can be defined as such:</p>
<pre>
<strong>type</strong>
TSQLRecordPeopleExt = <strong>class</strong>(TSQLRecord)
<strong>private</strong>
fData: TSQLRawBlob;
fFirstName: RawUTF8;
fLastName: RawUTF8;
fYearOfBirth: integer;
fYearOfDeath: word;
fLastChange: TModTime;
fCreatedAt: TCreateTime;
<strong>published</strong>
<strong>property</strong> FirstName: RawUTF8 <strong>index</strong> 40 <strong>read</strong> fFirstName <strong>write</strong> fFirstName;
<strong>property</strong> LastName: RawUTF8 <strong>index</strong> 40 <strong>read</strong> fLastName <strong>write</strong> fLastName;
<strong>property</strong> Data: TSQLRawBlob <strong>read</strong> fData <strong>write</strong> fData;
<strong>property</strong> YearOfBirth: integer <strong>read</strong> fYearOfBirth <strong>write</strong> fYearOfBirth;
<strong>property</strong> YearOfDeath: word <strong>read</strong> fYearOfDeath <strong>write</strong> fYearOfDeath;
<strong>property</strong> LastChange: TModTime <strong>read</strong> fLastChange <strong>write</strong> fLastChange;
<strong>property</strong> CreatedAt: TCreateTime <strong>read</strong> fCreatedAt <strong>write</strong> fCreatedAt;
<strong>end</strong>;
</pre>
<p>As you can see, there is no difference with an <em>internal</em> ORM class:
it inherits from <code>TSQLRecord</code>, but it may want it to inherit from
<code>TSQLRecordMany</code> to use <a href="https://blog.synopse.info?post/post/2011/12/06/Automatic-JOIN-query">many-to-many relationship</a>.</p>
<p>The only difference is this <code>index 40</code> attribute in the
definition of <code>FirstName</code> and <code>LastName</code> published
properties: this will define the length (in <code>WideChar</code>) to be used
when creating the external field for TEXT column. In fact, <em>SQLite3</em>
does not care about textual field length, but almost all other database engines
expect a maximum length to be specified when defining a <code>VARCHAR</code>
column in a table. If you do not specify any length in your field definition
(i.e. if there is no <code>index ???</code> attribute), the ORM will create a
column with an unlimited length (e.g. <code>varchar(max)</code> for <em>MS SQL
Server</em> in this case, code will work, but performance and disk usage may be
degraded.</p>
<p>Here is an extract of the regression test corresponding to external
databases:</p>
<pre>
<strong>var</strong> RInt: TSQLRecordPeople;
RExt: TSQLRecordPeopleExt;
(...)
fConnection := TSQLDBSQLite3ConnectionProperties.Create(':memory:','','','');
<span style="background-color:yellow;">VirtualTableExternalRegister(fModel,TSQLRecordPeopleExt,fConnection,'PeopleExternal');</span>
fClient := TSQLRestClientDB.Create(fModel,<strong>nil</strong>,'test.db3',TSQLRestServerDB);
<span style="background-color:yellow;">fClient.Server.StaticVirtualTableDirect := StaticVirtualTableDirect;</span>
<span style="background-color:yellow;">fClient.Server.CreateMissingTables;</span>
(...)
<strong>while</strong> RInt.FillOne <strong>do begin</strong>
RExt.Data := RInt.Data;
(...)
<span style="background-color:yellow;">aID := fClient.Add(RExt,true);</span>
(...)
<span style="background-color:yellow;">Check(fClient.Retrieve(aID,RExt));</span>
(...)
<strong>end</strong>;
<span style="background-color:yellow;">Check(fClient.Server.CreateSQLMultiIndex(</span>
<span style="background-color:yellow;">TSQLRecordPeopleExt,['FirstName','LastName'],false));</span>
Check(RInt.FillRewind);
<strong>while</strong> RInt.FillOne <strong>do begin</strong>
<span style="background-color:yellow;">RExt.FillPrepare(fClient,'FirstName=? and LastName=?',[],</span>
<span style="background-color:yellow;">[RInt.FirstName,RInt.LastName]); <em>// query will use index -> fast <img src="https://blog.synopse.info?pf=smile.svg" alt=":)" class="smiley" /></em></span>
<span style="background-color:yellow;"><strong>while</strong> RExt.FillOne <strong>do begin</strong></span>
<span style="background-color:yellow;">Check(RExt.FirstName=RInt.FirstName);</span>
(...)
<strong>end</strong>;
<strong>end</strong>;
Now := fClient.ServerTimeStamp;
<strong>for</strong> i := 1 <strong>to</strong> aID <strong>do</strong>
<strong>if</strong> i <strong>mod</strong> 100=0 <strong>then begin</strong>
<span style="background-color:yellow;">Check(fClient.Retrieve(i,RExt,true),'for update');</span>
RExt.YearOfBirth := RExt.YearOfDeath;
<span style="background-color:yellow;">Check(fClient.Update(RExt),'Update 1/100 rows');</span>
<span style="background-color:yellow;">Check(fClient.UnLock(RExt));</span>
<strong>end</strong>;
<strong>for</strong> i := 1 <strong>to</strong> aID <strong>do</strong>
<strong>if</strong> i <strong>and</strong> 127=0 <strong>then</strong>
<span style="background-color:yellow;">Check(fClient.Delete(TSQLRecordPeopleExt,i),'Delete 1/128 rows');</span>
<strong>for</strong> i := 1 <strong>to</strong> aID <strong>do begin</strong>
<span style="background-color:yellow;">ok := fClient.Retrieve(i,RExt,false);</span>
Check(ok=(i <strong>and</strong> 127<>0),'deletion');
<strong>if</strong> ok <strong>then begin</strong>
Check(RExt.CreatedAt<=Now);
<strong>if</strong> i <strong>mod</strong> 100=0 <strong>then begin</strong>
Check(RExt.YearOfBirth=RExt.YearOfDeath,'Updated');
Check(RExt.LastChange>=Now);
<strong>end else begin</strong>
Check(RExt.YearOfBirth<>RExt.YearOfDeath,'Not Updated');
Check(RExt.LastChange<=Now);
<strong>end</strong>;
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>As you can see, there is no difference with using the local <em>SQLite3</em>
engine or a remote database engine.<br />
From the Client point of view, you just call the usual RESTful methods, i.e.
<code>Add / Retrieve / Update / UnLock / Delete</code>, and you can even handle
advanced methods like a <code>FillPrepare</code> with a complex WHERE clause,
or <code>CreateSQLMultiIndex / CreateMissingTables</code> on the server side.
Even the creation of the table in the remote database (the <code>'CREATE
TABLE...'</code> SQL statement) is performed by the framework, with the
appropriate column properties according to the database expectations (e.g. a
TEXT for <em>SQLite3</em> will be a NVARCHAR2 field for <em>Oracle</em>).</p>
<p>The only specific instruction is the global
<code>VirtualTableExternalRegister</code> function, which has to be run on the
server side (it does not make any sense to run it on the client side, since for
the client there is no difference between any tables - in short, the client do
not care about storage; the server does).</p>
<p>Note that the <code>LastChange</code> field was defined as a
<code>TModTime</code>: in fact, the current date and time will be stored each
time the record is updated, i.e. for each <code>fClient.Add</code> or
<code>fClient.Update</code> calls. This is tested by both
<code>RExt.LastChange>=Now</code> and <code>RExt.LastChange<=Now</code>
checks in the latest loop. The time used is the "server-time", i.e. the current
time and date on the server (not on the client), and, in the case of external
databases, the time of the remote server (it will execute e.g. a <code>select
getdate()</code> under MS SQL to synchronize the date to be inserted for
<code>LastChange</code>). In order to retrieve this server-side time stamp, we
use <code>Now := fClient.ServerTimeStamp</code> instead of the local
<code>Iso8601Now</code> function.</p>
<p>A similar feature is tested for the <code>CreatedAt</code> published field,
which was defined as <code>TCreateTime</code>: it will be set automatically to
the current server time at record creation (and not changed on modifications).
This is the purpose of the <code>RExt.CreatedAt<=Now</code> check in the
above code.</p>
<p>It's worth noting a genuine capability of our ORM architecture:</p>
<p>The Virtual Table feature of <em>SQLite3</em> will allow those remote tables
to be accessed just like "native" <em>SQLite3</em> tables - in fact, you may be
able e.g. to write a valid SQL query with a <code>JOIN</code> between
<em>SQlite3</em> tables, <em>Microsoft SQL Server</em>, <em>MySQL</em> and
<em>Oracle</em> databases, even with multiple connections and several remote
servers.<br />
Think as an ORM-based <em>Business Intelligence</em> from any database source.
Added to our code-based reporting engine (able to generate pdf), it could be a
very powerful way of consolidating any kind of data.</p>
<p>For a new project, you define regular <code>TSQLRecord</code> classes and
use <em>SQLite3</em> as your main storage by default, and create
<code>TSQLRecordMany</code> external classes if needed (new since revision
1.17); but you could also call <code>VirtualTableExternalRegister</code>,
to use a dedicated server for better response time or additional features (like
data sharing with other applications or languages).</p>
<h3>RTFM</h3>
<p>The SAD documents contains also some new diagrams, which will help
understand how the Client-Server ORM architecture works.</p>
<p>For instance, a typical Client-Server RESTful POST / Add request over
HTTP/1.1 will be implemented as such, on Server side, to handle both "normal"
SQLite3 requests and "Virtual Tables" requests:</p>
<div><a href="https://blog.synopse.info?post/public/ClientServerVirtualTables.png"><img src="https://blog.synopse.info?post/public/.ClientServerVirtualTables_m.jpg" alt="" title="ClientServerVirtualTables.png, août 2011" /></a></div>
<p>In fact, the above diagram corresponds to a database model with only
external virtual tables, and with <code>StaticVirtualTableDirect=false</code>,
i.e. calling the Virtual Table mechanism of SQlite3 for each request.</p>
<p>Most of the time, i.e. for RESTful commands, the execution is more
direct: the static <code>TSQLRestServerStaticExternal</code> instance is
called for most RESTful access, for an overhead reduced by 50%. In practice,
this design will induce no speed penalty, when compared to a direct database
access. It could be even faster, if your mORMot / SQLite3 server is located on
the same computer than the database: in this case, use of JSON and REST could
be faster - even faster when using <a href="https://blog.synopse.info?post/post/2011/06/03/BATCH-sequences-for-adding/updating/deleting-records">batch
process</a>!</p>
<p>More details available <a href="http://synopse.info/fossil/wiki?name=Downloads">in the documentation</a>!</p>
<p><a href="http://synopse.info/forum/viewtopic.php?pid=2422#p2422">Feedback
and comments are welcome on our forum</a>.</p>Use TDataSet in mORMot or SynDBurn:md5:b02d6a9d2164ec4afdb20e2f8a937dc52012-06-24T17:53:00+02:002012-06-24T17:01:01+02:00AB4327-GANDIOpen Source librariesblogDatabaseDelphiDocumentationJSONmORMotMSSQLODBCOleDBOracleORMRADRestSOASourceSQLSQLite3SynDBTDataSet<p>In our documentation, and in all our code source, we avoid using the VCL
<code>DB.pas</code> related units, and all the associated RAD
components.</p>
<p>This is by design, since our experiment encouraged us to "think ORM, forget
anything about RAD (and even SQL in most cases)" in <em>mORMot</em>.<br />
And it introduced some nice border-side effect to Delphi users, e.g. that even
a "Delphi Starter Edition" is able to use <em>mORMot</em>, have access to
<em>SQLite3</em>, <em>MS SQL</em> or <em>Oracle</em> or any other DB, add
<a href="https://blog.synopse.info?post/post/2012/03/07/Interface-based-services">interface-based RESTful
JSON services</a> over it, just for free...</p>
<p>But in the real world, you may need to upgrade some existing application,
get rid of the BDE, or add a SOA layer over an existing business
intelligence.<br />
And <em>mORMot</em> is able to serve you well in those scenarios.<br />
That's why we just added a first attempt to expose <code>SynDB</code> results
and <em>mORMOt</em> <code>TSQLTableJSON</code> content into a
<code>TDataSet</code>.</p> <p>In order to avoid mixing the ORM and DB features with RAD units, we decided
to add two new units:</p>
<ul>
<li><a href="http://synopse.info/fossil/finfo?name=SynDBVCL.pas">SynDBVCL
unit</a> to handle DB VCL dataset using <code>SynDB</code> data access.<br />
This is for direct connection to the DB using our <code>SynDB</code> classes.
May be handy.</li>
<li><a href="http://synopse.info/fossil/finfo?name=SQLite3/SQLite3VCL.pas">SQLite3VCL
unit</a> to fill a DB VCL dataset using <code>TSQLTable/TSQLTableJSON</code>
data access, as returned from our <em>RESTful JSON</em> server.</li>
</ul>
<p>This first draft uses a <code>TClientDataSet</code> in memory to set the
data.<br />
Not perfect, but working. It was not meant for huge amount of data, so you have
an optional parameter to stop filling the <code>TDataSet</code> when a
specified number of data rows is reached.<br />
And, for performance reasons, you would better install the <a href="http://andy.jgknet.de/blog/bugfix-units/midas-speed-fix-12">nice Midas speed
up patch by Andy</a>, with Delphi older than Delphi 2009 Update 3.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=752">welcome
on our forum</a>.</p>