Synopse Open Source - Tag - SourcemORMot MVC / SOA / ORM and friends2024-02-02T17:08:25+00:00urn:md5:cc547126eb580a9adbec2349d7c65274DotclearEnd Of Live OpenSSL 1.1 vs Slow OpenSSL 3.0urn:md5:f20e1a3a1c96e8f65f1fc8ef5a04498c2023-09-08T11:59:00+01:002023-09-08T14:13:49+01:00Arnaud BouchezOpen SourceAESCertificatesDelphiFreePascalGoodPracticeLateBindingLazarusMaxOSXmORMotmORMot2OpenSSLperformancesecuritySource<p>You may have noticed that the OpenSSL 1.1.1 series will reach End of Life (EOL) next Monday...<br />
Most sensible options are to switch to 3.0 or 3.1 as soon as possible.</p>
<p><img src="https://blog.synopse.info?post/public/blog/mormotSecurity.jpg" alt="mormotSecurity.jpg, Sep 2023" /></p>
<p>Of course, our <a href="https://github.com/synopse/mORMot2/blob/master/src/lib/mormot.lib.openssl11.pas"><em>mORMot 2</em> OpenSSL unit</a> runs on 1.1 and 3.x branches, and self-adapt at runtime to the various API incompatibilities existing between each branch.<br />
But we also discovered that switching to OpenSSL 3.0 could led into big performance regressions... so which version do you need to use?</p> <h4>OpenSSL 1.1 End Of Live</h4>
<p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/6/6a/OpenSSL_logo.svg/320px-OpenSSL_logo.svg.png" alt="OpenSSL logo" /></p>
<p>The well known and well established OpenSSL 1.1.1 series will reach End of Life (EOL) on 11th September 2023. So next Monday! <img src="https://blog.synopse.info?pf=sad.svg" alt=":(" class="smiley" /> <br />
Users of OpenSSL 1.1.1 should consider their options and plan any actions they might need to take.</p>
<p>Note that Indy users are <a href="https://github.com/IndySockets/Indy/issues/183">still stuck to the OpenSSL 1.0 branch</a>, even 1.1 is not yet officially supported. Some <a href="https://github.com/IndySockets/Indy/pull/299">alternate IO handlers</a> are able to use newest releases - to some extend.<br />
Indy users should rather move to a better supported library, like our little <em>mORMot</em>.</p>
<p>Also note that there are some API incompatibilities between 1.1 and 3.x. Functions have been renamed, or even removed; new context constructors appeared; some parameters types even changed!<br />
Our unit tries to address all those problems at runtime, and is tested against several version of the OpenSSL library, to ensure you do not have to worry about those low-level issues.</p>
<h4>OpenSSL 3.x Benefits</h4>
<p>With OpenSSL 3.0, the developpers did a huge refactoring of the library internals.<br />
To be fair, the 1.x source code of OpenSSL was kind of a mess, and difficult to maintain. The biggest IT companies did even made their own forks or switched to other libraries. The best known is <a href="https://boringssl.googlesource.com/boringssl/">BoringSSL</a>, maintained by Google, and used e.g. in Chrome and Android.<br />
So it was time for a refactoring, especially for a library as critical as OpenSSL for so many projects.</p>
<p>With the new 3.x branch, a lot of low-level API functions have been deprecated.<br />
In practice, you don't have direct access any more to the internal structures of the library, and should now always use the high-level API to access a context property, or execute the processing methods. For instance, the low-level <code>AES_encrypt</code> function is not available any more: from now on, you need to use the high-level <code>EVP_Encrypt*</code> API.<br />
The official <a href="https://www.openssl.org/docs/man3.0/man7/migration_guide.html">Migration Guide page</a> is clearly huge, and worth reading if you want to prepare yourself to the upcoming years with OpenSSL.</p>
<h4>OpenSSL 3.0 Performance Regression</h4>
<p>The 3.0 branch new code may seem more beautiful and more maintainable, but it had its drawbacks. Newer is not always better.<br />
Most users of this new release <a href="https://github.com/openssl/openssl/issues/17064">observed a huge performance regression</a> when switching from 1.x to 3.0. It affected a lot of projects, from various languages, even script languages which were not already shining about performance. Time regression from 3x up to 10x were reported. On our side, X509 certificates manipulation was really slower than before - the worse being about X509 stores.</p>
<p>Some slowdown were expected and documented (like RSA key generation, which now uses 64 rounds). But the regression was much deeper.<br />
The culprit seems not to be the core cryptographic code, like AES buffer encoding (which asm claims to have been optimized even further on 3.x branch), but the OpenSSL context structures themselves. They were rewritten for future maintainability, but not focusing on their actual performance.</p>
<h4>OpenSSL 3.1 Numbers</h4>
<p>The 3.1 branch claims to have addressed most of these problems.</p>
<p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/ea/The_Tortoise_and_the_Hare_-_Project_Gutenberg_etext_19994.jpg/334px-The_Tortoise_and_the_Hare_-_Project_Gutenberg_etext_19994.jpg" alt="The Tortoise and the Hare" /></p>
<p>To be sure, we run the <em>mORMot</em> cryptographic regression tests with several versions of OpenSSL. And in fact, OpenSSL 3.1 was much faster than OpenSSL 3.0, but still behind OpenSSL 1.1.<br />
Here are the numbers we observed for the whole <code>TTestCoreCrypto</code> method execution, executed on Win32:</p>
<ul>
<li>OpenSSL 1.1 = 15 sec</li>
<li>OpenSSL 3.0 = 33 sec</li>
<li>OpenSSL 3.1 = 18 sec</li>
</ul>
<p>There are several aspects to emphasize:</p>
<ul>
<li>Those tests runs also <em>mORMot</em> engine cryptography, so you don't only test OpenSSL: the "pure mORMot" tests take around 4.5 seconds in the above numbers;</li>
<li>Any serious project should consider compiling on Win64, and running a server on a x86_64 Linux - on this platform, the regression does exist, but only slightly better;</li>
<li>The slowdown was less affecting <code>TTestCoreCrypto.Benchmark</code> (i.e. raw buffer encryption) than <code>TTestCoreCrypto.Catalog</code> (i.e. certificates process);</li>
<li>Our tests were mono-threaded, and worse slow down were reported on heavily threaded process (up to x10).</li>
</ul>
<p>Within the <em>mORMot</em> OpenSSL wrapper, we try to cache as many context as possible. For instance, we don't lookup the OpenSSL algorithm by name for each call, but we cache it at runtime to avoid any slowdown.<br />
But it seems not enough with OpenSSL 3.0, which may affect your application performance.</p>
<h4>To Support or Not Support</h4>
<p>So OpenSSL 3.1 seems to be the way to go.</p>
<p>On Linux (or other POSIX systems), you are likely to use the library shipped with the system.<br />
So you would not worry about which version to use. And, sadly, it is very likely that your distribution provides OpenSSL 3.0 and not OpenSSL 3.1.</p>
<p>On Windows (or Mac), you could (should?) use your "own" dll/so files, so you have to take into account the support level of the library.<br /></p>
<ul>
<li>OpenSSL 3.0 is a Long Term Support (LTS) version, which will be maintained until 7th September 2026.<br /></li>
<li>OpenSSL 3.1 will be supported only until 14th March 2025.</li>
</ul>
<p>These support end dates could appear counter-intuitive, but this is an usual way in Open Source projects, the best known being perhaps <a href="https://ubuntu.com/blog/what-is-an-ubuntu-lts-release">Ubuntu LTS versions</a>.<br />
For more information about OpenSSL support lifetime, look at the <a href="https://www.openssl.org/source/">official OpenSSL Downloads page</a>.</p>
<p>So, for most projects, especially on Windows where you are likely to publish OpenSSL dll with your own executable, switching to OpenSSL 3.1 is likely to be the way to go.<br />
If you need to gather some security certification for your product, you may consider using OpenSSL 3.0 LTS version, which may help your certification remain active for a longer period.</p>
<p>Any feedback is <a href="https://synopse.info/forum/viewtopic.php?id=6697">welcome on our forum</a>, as usual!</p>New Client for MongoDB 5.1/6 Supporturn:md5:41f28034cfacffa2bf7fde35e8e64b432022-08-12T20:12:00+01:002022-08-12T20:12:00+01:00Arnaud BouchezmORMot FrameworkCrossPlatformDatabaseDelphiFPCFreePascalJSONMongoDBmORMot2NoSQLODMORMperformanceRESTSource<p>Starting with its version 5.1, <em>MongoDB</em> disabled the legacy protocol used for communication since its beginning.<br />
As a consequence, our <em>mORMot</em> client was not able to communicate any more with the latest versions of <em>MongoDB</em> instances.</p>
<p><img src="https://blog.synopse.info?post/public/mongodb.png" alt="" /></p>
<p>Last week, we made a deep rewrite of <a href="https://github.com/synopse/mORMot2/blob/master/src/db/mormot.db.nosql.mongodb.pas">mormot.db.nosql.mongodb.pas</a>, which changed the default protocol to use the new layout on the wire. Now messages use regular <a href="https://www.mongodb.com/docs/current/reference/command/">MongoDB Database Commands</a>, with automated compression if needed.</p>
<p>No change is needed in your end-user <em>MongoDB</em> or ORM/ODM code. The upgrade is as simple as update your <em>mORMot 2</em> source, then recompile.</p> <h4>The Mongo Wire Protocol</h4>
<p>Since its beginning, <em>MongoDB</em> used a simple protocol over TCP, via several binary opcodes and message, for CRUD operations.</p>
<p>A new alternative protocol was <a href="https://emptysqua.re/blog/driver-features-for-mongodb-3-6/">introduced in version 3.6,</a> and the former protocol was marked as deprecated.<br />
Two new opcodes were introduced, OP_MSG and OP_COMPRESSED, to replace all other frames. They just encapsulate, with or without compression, some abstract BSON content.<br />
The official documentation details those changes <a href="https://www.mongodb.com/docs/manual/reference/mongodb-wire-protocol/">in this web page</a>.</p>
<p>In short (picture extracted from the blog above), the protocol came from this:</p>
<p><img src="https://blog.synopse.info?post/public/ye-olde-wire-protocol.png" alt="" /></p>
<p>to this:</p>
<p><img src="https://blog.synopse.info?post/public/op-msg.png" alt="" /></p>
<p>The main benefit is that the commands and answers are just conventional BSON, so the protocol can change at logical/BSON/JSON level by adding or changing some members, with no need of dealing with low-level binary structures.</p>
<p>With the version 5.1 of <em>MongoDB</em>, the previous protocol was not just deprecated, but disabled.<br />
So we had to update the <em>mORMot 2</em> client code! (yes, the <em>mORMot 1</em> code has not been updated - it may become a good reason to upgrade)</p>
<h4>Deep Rewrite</h4>
<p>In fact, the official MongoDB documentation is somewhat vague. And the official drivers are a bit difficult to reverse-engineer, due to the verbose nature of C, Java or C#. The native/node driver was easiest to dissect, and we used it as reference.<br />
Luckily enough, there are some <a href="https://github.com/mongodb/specifications/blob/master/source/message/OP_MSG.rst">specification document available too</a>, which offers some additional valuable clarifications.</p>
<p>After some testing, we managed to replace all previous OP_QUERY and its brothers to the new OP_MSG frame, which is, as documented in the specification, "One opcode to rule them all". <img src="https://blog.synopse.info?pf=wink.svg" alt=";)" class="smiley" /></p>
<p>Once we had the commands working, we needed to rewrite all CRUD operations using commands, and not opcodes.<br />
Queries are now made with <code><a href="https://www.mongodb.com/docs/current/reference/command/find/">find</a></code> and <code><a href="https://www.mongodb.com/docs/current/reference/command/aggregate/">aggregate</a></code> commands. Their results are now located in a <code>"cursor": firstBatch": ..</code> BSON array within the response. And a new <code><a href="https://www.mongodb.com/docs/current/reference/command/getMore/">getMore</a></code> command is to be used to retrieve the next values within a <code>"cursor": nextBatch": ...</code> resultset.<br />
For writing, <code><a href="https://www.mongodb.com/docs/current/reference/command/insert">insert</a></code>, <code><a href="https://www.mongodb.com/docs/current/reference/command/update">update</a></code> and <code><a href="https://www.mongodb.com/docs/current/reference/command/delete">delete</a></code> commands are called, with their appropriate BSON content.</p>
<p>During the refactoring, we optimized the BSON process, and also enhanced the whole process, mainly the logs and the execution efficiency. The <em>mORMot</em> client side should not be a bottleneck. And it is not, even with this NoSQL database.</p>
<p>Don't expect any performance enhancement, or new features. It is just some low-level protocol change at TCP level.<br />
But if you used the "non acknowledged write mode" of the former protocol, which was unsafe but very fast, you will have lower performance with the new protocol, because the new protocol always acknowledges the commands it receives. So, in some very specific configurations, the new protocol may reduce the performance.</p>
<h4>Backward Compatibility</h4>
<p>All those changes were encapsulated in our revised <a href="https://github.com/synopse/mORMot2/blob/master/src/db/mormot.db.nosql.mongodb.pas">mormot.db.nosql.mongodb.pas</a> unit.</p>
<p>If you have a very old <em>MongoDB</em> instance, and don't want to upgrade, you could just compile your project with the <code>MONGO_OLDPROTOCOL</code> conditional, to use the deprecated opcodes.<br />
If the <em>MongoDB</em> team does not care much with backward compatibility (they could have kept the previous protocol for sure, they still maintain it for the handshake message if needed), we do care about not breaking too much things with <em>mORMot</em>, so we kept the previous code, and tested/validated it too, for legacy systems.</p>
<h4>New Sample</h4>
<p>We translated and introduced the <em>MongoDB</em> benchmark sample to <em>mORMot 2</em> code base.</p>
<p>You could find it, and run it, from <a href="https://github.com/synopse/mORMot2/tree/master/ex/mongodb">our source code repository</a>.</p>
<p>This code is a good entry point for what is possible with this unit in our framework, for both direct access or ORM/ODM access.<br />
And you would be able to guess the performance numbers you may achieve with your project.</p>
<p>Running a <em>MongoDB</em> database in a container is as easy as executing the following command:</p>
<pre>
sudo docker run --name mongodb -d -p 27017:27017 mongo:latest
</pre>
<p>Then you will have a <em>MongoDB</em> server instance accessible on <code>localhost:27017</code>, so you could run the sample straight away.</p>
<h4>Delphi/FPC Open Source Rocks</h4>
<p>We hope you will find the change painless and transparent. We did not modify the high-level client methods, nor break the ORM/ODM: you can still write some SELECT complex statements, and our ORM will translate it into <em>MongoDB</em> aggregate commands.</p>
<p>To my knowledge, there is <a href="https://github.com/stijnsanders/TMongoWire/commit/7f12a64f571e476704bdcb737e1fc087ef792f59">only a single other Delphi/FPC client library</a> which made the upgrade to the new protocol, at today. Once we made our own changes, we notified other library authors, and Stijn made very quickly the needed changes. Congrats! Maybe our code could be used as reference for other library maintainers, because the protocol needs some small tweaks sometimes.<br />
It is important to have some maintenance on the library you use. And our little <em>mORMot</em> is still on the edge: thanks to FPC, it runs very well on Linux and BSD, which makes it perfect for professional services running in the long term! :)</p>
<p>Your feedback is welcome <a href="https://synopse.info/forum/viewtopic.php?id=6318">in the forum thread which initiated these modifications</a>, as usual!<br />
Don't hesitate to notify us any missing or broken feature.<br />
Thanks Daniel for your report and support!</p>EKON 25 Slidesurn:md5:aed86aeed11190901cf050e9842ec0bc2021-11-16T12:34:00+00:002021-11-16T12:34:00+00:00Arnaud BouchezmORMot Framework64bitAESAES-CTRAES-GCMAES-NiauthenticationCertificatesCrossPlatformDDDDelphiECCECDHECIESECSDAed25519EKONFreePascalinterfacelibdeflatemORMotmORMot2multithreadOpenSSLperformancerandomSOASourceWebSockets<p><a href="https://entwickler-konferenz.de/">EKON 25 at Düsseldorf</a> was a great conference (konference?).</p>
<p>At last, a <strong>physical</strong> gathering of Delphi developers, mostly from Germany, but also from Europe - and even some from USA! No more virtual meetings, which may trigger the well known 'Abstract Error' on modern pascal coders.<br />
There were some happy FPC users too - as I am now. <img src="https://blog.synopse.info?pf=smile.svg" alt=":)" class="smiley" /></p>
<p><img src="https://blog.synopse.info?post/public/blog/Ekon25.png" alt="" /></p>
<p>I have published the slides of my conferences, mostly about mORMot 2.<br />
By the way, I wish we would be able to release officially mORMot 2 in December, before Christmas. I think it starts to be stabilized and already known to be used on production. We expect no more breaking change in the next weeks.</p> <p>Here are the slides of my two 1-hour sessions.</p>
<h5>mORMot Cryptography</h5>
<p>The OpenSource mORMot framework has a strong set of cryptography features. It offers symmetric cryptography with hashing and encryption, together with asymmetric cryptography via private/public key pairs. Its optimized pascal and assembly engines can be embedded into your executable, but you could also call an external OpenSSL library if needed. This session will present mormot.crypt.* units, and apply them to some use cases, from low-level algorithms to high-level JWT or file encryption and signing.</p>
<p><a href="https://www.slideshare.net/ArnaudBouchez1/ekon25-mormot-2-cryptography">mORMot 2 Cryptography on SlideShare</a></p>
<p>I just had an interesting discussion with Michael on <a href="https://gitlab.com/freepascal.org/fpc/source/-/commit/3229cb712e33374b85258aed43726058be633bed#note_734398698">FPC new gitlab platform</a>: the FPC RTL is gaining some official cryptography functions, and I proposed to use mORMot code base as reference, and to introduce some RTL wrapper functions which can redirect to a plain pascal FPC RTL version, or use another engines, like OpenSSL or mORMot, if available.</p>
<h5>Server-Side REST Notifications with mORMot</h5>
<p>The most powerful way of writing REST services is to define them via interfaces, then let the SOA/REST framework do all the routing, data marshalling and communication behind the scenes. One distinctive feature of mORMot is to define a method parameter as a notification interface, and let the server call back the client when needed, as with regular Delphi code. This session will present the benefit of defining REST services using interfaces, and how WebSockets can offer real-time notifications into your rich Delphi client applications.</p>
<p><a href="https://www.slideshare.net/ArnaudBouchez1/ekon25-mormot-2-serverside-notifications">mORMot 2 Server-Side Notifications on SlideShare</a></p>
<p>Feedback is <a href="https://synopse.info/forum/viewtopic.php?id=6051">welcome on our forum, as usual.</a></p>Special Care of Delphi 10.4urn:md5:c3192633bbab96b889db9095f247c94d2020-07-20T09:17:00+01:002020-09-03T06:37:23+01:00Arnaud BouchezmORMot Framework10.4bugDelphiSource<p>A regression in the Delphi 10.4 compiler was identified. Its optimizer wrongly deletes some code, in one very specific part of the framework.</p>
<p><img src="https://blog.synopse.info?post/public/blog/sillybug.jpg" alt="sillybug.jpg, Jul 2020" /></p>
<p>As a result a GPF (Access violation) may be triggered with Delphi 10.4 in release mode - the debug mode (when optimization is disabled) has no problem.
Thanks to great user feedback, we were able to circumvent it. But we should better stay in alert, like any mORMot, until Delphi 10.4 officially release a patch.</p> <p>Sadly, I don't have any Delphi 10.4 licence so I can't reproduce it.
I don't see any issue with Delphi 10.3.3 Community Edition, and most older versions (up to Delphi 6) I could try.</p>
<p>But Sakura found the root cause and <a href="https://synopse.info/forum/viewtopic.php?pid=32673#p32673">reported his findings in our forum</a>.
He created a <a href="https://quality.embarcadero.com/browse/RSP-30088">QC entry in the Embarcadero Quality Portal</a>, with some reproducing code.
We are waiting for a fix from Embarcadero side.</p>
<p>Sounds like if that with the latest commits, <a href="https://synopse.info/forum/viewtopic.php?pid=32689#p32689">we were able to circumvent the problem</a>, while waiting for an official fix in the Delphi 10.4 compiler.</p>
<p>Thanks again Sakura, and <a href="https://synopse.info/forum/viewtopic.php?id=5520">feedback from any 10.4 user is welcome</a>!</p>
<p><strong>Update:</strong> Sakura reported that latest Delphi 10.4.1 release fixed the regression (even if it was wrongly set to "Expected behavior" in EMB Jira). Good news!</p>Preparing Revision 2.x of the mORMot Frameworkurn:md5:a69f3101f568a4660f5bc36e69d1adc42020-03-03T20:14:00+01:002020-03-06T14:54:33+01:00AB4327-GANDImORMot FrameworkblogDelphiFreePascalGoodPracticemORMotmORMot2RepositoryRESTSourceSQLite3Synopse<p>The more I think of it, the more I am convinced it is time to change how the
framework is versioned.<br />
We have version 1.18 since years... difficult to follow... time to upgrade!</p>
<p><img src="https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSsiYRtu9W1D-p2scisYFSumx8FPoJlk5cHU0cmy-en6XpEgzPXMg&s" alt="" /><br />
I would like to upgrade <em>mORMot</em> to version 2 - with a major
refactoring.</p> <p><br class="Apple-interchange-newline" />
So, in practice, two main points to discuss:</p>
<p>1. Switch to <a href="https://semver.org">Semantic Versioning</a></p>
<p><em>Given a version number MAJOR.MINOR.PATCH, increment the:<br />
- MAJOR version when you make incompatible API changes,<br />
- MINOR version when you add functionality in a backwards compatible manner,
and<br />
- PATCH version when you make backwards compatible bug fixes.<br />
Additional labels for pre-release and build metadata are available as
extensions to the MAJOR.MINOR.PATCH format.</em></p>
<p>The idea would be to have 2.# minor version every month, with patches
in-between. A monthly minor upgrade allow to have new features included quick
enough to be interresting.</p>
<p>2. <em>mORMot</em> 2.0.0 would induce incompatible API changes.</p>
<p>I would like:</p>
<p>2.1. To remove all reminicence of the "SQLite3 framework", e.g. the SQLite3
sub-folder.</p>
<p>2.2. Try to split <em>SynCommons.pas</em> and <em>mORMot.pas</em> units into
smaller units. Putting e.g. the raw JSON process or RTTI in some external
unit.</p>
<p>2.3. Introduce some high-level classes and units, perhaps as (why-not?) some
non-visual <em>TComponent</em> version of our classes. Keeping existing
low-level classes, which are stable and full of features - but difficult to
adopt.</p>
<p>2.4. Get rid of oldest Delphi (e.g. Delphi 5 - which is currently broken
BTW) official compatibility, only support mainstream Delphi revisions, and
focus on FPC trunk as main target -and officially maintaining only a few
version of Delphi (2007/XE7/10.3?) - with other version maintained by
contributors.</p>
<p>Any feedback is <a href="https://synopse.info/forum/viewtopic.php?id=5313">welcome in our forum, as
usual</a>!</p>EKON 22 Slides and Codeurn:md5:98df618381dd2402ddfafc2ba65562572018-11-12T11:15:00+01:002018-11-12T19:38:24+01:00AB4327-GANDIPascal ProgrammingasmblogCrossPlatformDelphiDocumentationdynamic arrayexceptionGoodPracticeinterfaceMicroservicesmORMotperformancerecordsessionSource <p><img src="https://image.slidesharecdn.com/ekon222highperformancepascalcodeonservers-181112090032/95/high-performance-object-pascal-code-on-servers-at-ekon-22-1-638.jpg" alt="" width="319" height="240" /></p>
<p>I've uploaded two sets of slides from my presentations at <a href="https://entwickler-konferenz.de/program-en/">EKON 22</a> :</p>
<ul>
<li><a href="https://www.slideshare.net/ArnaudBouchez1/object-pascal-clean-code-guidelines-proposal-at-ekon-22">
Object Pascal Clean Code Guidelines Proposal</a></li>
<li><a href="https://www.slideshare.net/ArnaudBouchez1/high-performance-object-pascal-code-on-servers-at-ekon-22">
High Performance Object Pascal Code on Servers</a><br />
with the <a href="https://synopse.info/files/slides/EKON22_2_High_Performance_Pascal_Code_On_Servers.zip">
associated source code</a></li>
</ul>
<p>The WorkShop about "Getting REST with mORMot" has a <a href="https://github.com/synopse/mORMot/tree/master/SQLite3/Samples/37%20-%20FishShop%20Service">
corresponding new Samples folder in our repository</a>.</p>
<p>And don't forget about one example of <a href="https://blog.synopse.info?post/post/2018/11/12/Win-100GB-of-log-space-from-a-Real-Life-mORMot-Project">high-performance
object pascal code: what we do at LiveMon - and get 100GB of log space for
free</a>!</p>
<p>Feedback is <a href="https://synopse.info/forum/viewtopic.php?id=4774">welcome in our forum</a> as
usual!</p>Status of mORMot ORM SOA MVC with FPCurn:md5:713e7f794fdb05475b36cadd4cde47f02018-02-07T10:31:00+01:002020-07-03T09:29:59+02:00AB4327-GANDImORMot Framework64bitasmblogBSDcrc32cCrossPlatformDelphiECCexceptionFreePascalLazarusLinuxMaxOSXmORMotORMperformanceRestRTTISOASourceSQLite3sse42<p>In the last weeks/months, we worked a lot with FPC.<br />
Delphi is still our main IDE, due to its better debugging experience under
Windows, but we target to have premium support of FPC, on all platforms,
especially Linux.</p>
<p><img src="https://blog.synopse.info?post/public/ScreenShots/lazarusaboutbox.png" alt="" title="Lazarus FPC About, Feb 2018" /></p>
<p>The new Delphi Linux compiler is out of scope, since it is heavily priced,
its performance is not so good, and ARC broke memory management so would need a
deep review/rewrite of our source code, which we can't afford - since we have
FPC which is, <a href="https://synopse.info/forum/viewtopic.php?pid=25984#p25984">from our
opinion</a>, a much better compiler for Linux.<br />
Of course, you can create clients for Delphi Linux and FMX, as usual, using
the <a href="https://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_86">cross-platform
client parts of mORMot</a>. But for server side, this compiler is not
supported, and will probably never be.</p> <p>First of all, since FPC - and Lazarus, its sibling IDE - are Open Source and
free, we can focus on mainly support a single version of the compiler.<br />
Since some missing RTTI for interfaces were recently merged into the trunk, we
start from the current FPC trunk as our main version. Easier than maintaining
Delphi 5 - 10.2 compability!</p>
<p>To install it, we usually use the <a href="https://github.com/newpascal/fpcupdeluxe">fpcupdeluxe tool</a>: you <a href="https://github.com/newpascal/fpcupdeluxe/releases">download a single binary
for your platform</a>, then you run the executable, pickup the compiler (or
cross-compiler) versions you need, and everything is downloaded and compiled
from git/svn on your own computer. Then click on the desktop link, and the IDE
launches in seconds. Nice fresh air in respect to Delphi setup experience!</p>
<p style="margin-top: 0;">As I wrote, in the last weeks/months, we worked a lot
to improve FPC support.</p>
<p style="margin-top: 0;">Just a few commits:</p>
<ul>
<li><a href="https://github.com/synopse/mORMot/commit/7bf5951e322f0164d8e3852f98f9f48b1bd7a6d2">
crc32c 2x/4x speeup by using SSE4.2+pclmulqdq opcodes on x64</a> - speed is now
around 21GB/s</li>
<li><a href="https://github.com/synopse/mORMot/commit/5b569db11e47fd65e2e0f792e1383895e50352b6">
TSynDaemon fork/run support on Linux/Posix</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/4e019e1b41f6759487e7479c194fcb28c25314c9">
fixed vtQWord proper support for FPC</a> - this doesn't exist in Delphi,
but it should!</li>
<li><a href="https://github.com/synopse/mORMot/commit/bc405cffb7bfb79646b46a6afc13968a0d804f32">
added pure pascal version of SynECC.pas</a> to run on all FPC platforms,
including ARM</li>
<li>BSD/OSX <a href="https://github.com/synopse/mORMot/commit/af8b33720d0da12a8476cd5bc34eeeab417cb096">
enhanced</a> <a href="https://github.com/synopse/mORMot/commit/16332915f87f5dcc7de881c9b72b09d05b609eba">
support</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/a5352309f41d8ef70561d819e57732ac0aaf43e3">
added scripts to use fpcupdeluxe's gcc cross-compilers for FPC static linking
on Windows, Linux, BSD and OSX</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/eaddf84b04b7eaf647adb598be140276b9b42046">
updated SQLite3 engine to latest version 3.22.0</a> - statically linked under
FPC Linux (no external dependency to the system libsqlite3.so)</li>
<li><a href="https://github.com/synopse/mORMot/commit/7c6d09df6caf3d8865e488d9147b5340c1a15fca">
HTTP server enhancements and fixes for high performance and stability under
Linux behind a local nginx proxy for production servers</a></li>
<li style="list-style: none"><a href="https://github.com/synopse/mORMot/commit/f40a379cbd3fb0e5076ac2cf664acb8b03ab86e2">
</a></li>
<li>better Linux compatibility</li>
<li><a href="https://github.com/synopse/mORMot/commit/ec4755a19722e744029244e7a2d1bfa985887f3e">
TSQLRecord QWord property fix</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/4b7786c3be0d1aec5abaad26f9de90e58edcce16">
tuned/enhanced logging content</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/d9fe9a5bf9305a0eb05c9f5a80d97a1b49f3bcb8">
deep refactoring of FPC RTTI access to have the same level than Delphi</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/95c5d56edbcb641f716c36d78792853eae67c689">
implemented Exceptions interception and logging for FPC</a><br />
with call stack trace (if available) - includes source code lines if compiled
using -g or -gl switches - tested on Win32, Win64, Linux i386 et x86_64 but
should work on other OS <img src="https://blog.synopse.info?pf=smile.svg" alt=":)" class="smiley" /></li>
<li>SyNode JavaScript engine support under FPC / Linux x86_64 by <a href="https://github.com/synopse/mORMot/commits?author=ssoftpro">ssoftpro</a> and
<a href="https://github.com/synopse/mORMot/commits?author=pavelmash">pavelmash</a> -
including a lot of fixes and tuning for this platform <a href="https://synopse.info/forum/viewtopic.php?pid=25985#p25985">to be heavily used
on production</a></li>
<li>and a lot of smaller enhancements (just search for FPC <a href="https://synopse.info/fossil/timeline?n=500&y=ci&t=&ms=exact">in
the commit timeline</a>), especially <a href="https://github.com/synopse/mORMot/commit/c5ad9b1d1f57177a8fe686271370cb12fc29d3d9">
tuning</a> the pascal code to better compile and execute under FPC, which can
generate very efficient assembly!</li>
</ul>
<div>As you can see, exciting times!</div>
<div>To be honest, the more we work with FPC as a compiler, the more we like
it.<br />
Staty tuned, and we encourage you to discover FPC/Lazarus!</div>AES-256 based Cryptographically Secure Pseudo-Random Number Generator (CSPRNG)urn:md5:d99c0c7b643507d7c78be9a1b13ce23b2016-04-09T11:37:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkAESAES-NiblogCSPRNGDelphiperformancerandomsecuritySource<p>Everyone knows about the pascal <a href="http://docwiki.embarcadero.com/Libraries/en/System.Random">random()
function</a>.<br />
It returns some numbers, using a <a href="http://en.wikipedia.org/wiki/Linear_congruential_generator">linear
congruential generator</a>, with a multiplier of <a href="http://forum.lazarus.freepascal.org/index.php?topic=23536.msg140722#msg140722">134775813</a>,
in its Delphi implementation.<br />
It is fast, but not really secure. Output is very predictable, especially if
you forgot to execute the <a href="http://docwiki.embarcadero.com/Libraries/en/System.RandSeed">RandSeed()
procedure</a>.</p>
<p><img src="http://resources.infosecinstitute.com/wp-content/uploads/121411_1611_SecureRando1.png" alt="" width="500" height="148" /></p>
<p>In real world scenarios, safety always requires random numbers, e.g. for
key/nonce/IV/salt/challenge generation.<br />
The less predictable, the better.<br />
We just included a <a href="https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator">Cryptographically
Secure Pseudo-Random Number Generator</a> (CSPRNG) into our <a href="https://github.com/synopse/mORMot/commit/92135fcdf7503bb54479643e926668532ca357d3">
SynCrypto.pas</a> unit.<br />
The <code>TAESPRNG</code> class would use real system entropy to generate
a sequence of pseudorandom bytes, using AES-256, so returning highly
unpredictable content.</p> <p>The <code>TAESPRNG</code> class is implemented as such:</p>
<ul>
<li>It would gather entropy using dedicated OS API, i.e. the <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa379942">CryptGenRandom
API</a> under Windows or <a href="http://www.2uo.de/myths-about-urandom/">/dev/urandom - /dev/random</a> on
Linux/POSIX systems; and since we are paranoid, we XOR some entropy hash
(retrieved e.g. directly from RDTSC and other system variables) to those OS
values - so even if the API is compromised, we try to have some changing
input;</li>
<li>This entropy (256 bytes of password, 16 bytes of salt) would be hashed
using safe PBKDF2-HMAC-SHA256 key derivation function (256 iterations by
default) to setup the secret key of an AES-256 cypher, and set a counter (CTR)
initial value by applying this AES-256 to the salt;</li>
<li>Each time some output is to be generated, AES-256 is applied to the CTR to
produce 16 bytes (2^128 bits) of pseudorandom data;</li>
<li>The CTR is incremented after each 16-byte block; AES being a block
cipher, it is a permutation of the space of block values: as such, it won't
ever output twice the same 16-byte block, so you can generate 2^128 blocks,
i.e. 2^132 bytes before the CTR overflows;</li>
<li>The AES-256 cypher is re-seeded from entropy on a regular basis (after
some bytes are generated), to avoid potential attacks on backward or
forward security;</li>
<li>The implementation is thread-safe, a shared <code>TAESPRNG.Main</code>
instance is available, but you can create your
own <code>TAESPRNG</code> instance, with tuned parameters (number
of PBKDF2 counts or automatic re-seeding number of bytes);</li>
<li>The level of security would be the same on all platforms, since OS is used
only for entropy, but an unique CSPRNG algorithm would actually generate
the data - even a compromised system (older CryptGenRandomAPI had <a href="http://eprint.iacr.org/2007/419.pdf">known weaknesses</a>, or you could
imagine some plot-based undocumented backdoor) may produce safe enough
output;</li>
<li>It would use <a href="https://blog.synopse.info?post/post/2015/01/15/AES-NI-enabled-for-SynCrypto">AES-NI</a> or Padlock hardware
acceleration, if available.</li>
</ul>
<p>You may use it in your projects by calling <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html#TAESPRNG_FILLRANDOM">TAESPRNG.Main.FillRandom()</a>
overloaded methods or <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html#TAESPRNG_FILLRANDOMBYTES">
TAESPRNG.Main.FillRandomBytes()</a>.</p>
<p>See the <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html#TAESPRNG">class
documentation</a>.</p>
<p>The main idea of a CSPRNG is that its output is as safe as the cypher it is
based on.<br />
Using AES-256, and initial PBKDF2 key derivation of OS-gathered entropy,
implements a <a href="https://en.wikipedia.org/wiki/Randomness_extractor">very
good randomness extractor</a>.<br />
Last but not least, especially if your CPU supports AES-NI (which is very
likely on a server), performance would be very high.<br />
Just use <code>TAESPRNG</code> when you need random input. Then go back to
your own code.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=3263">welcome on our forum</a>, as
usual!</p>New blog about mORMoturn:md5:afad33c1756c8047d6effb4f45bc7bd62015-07-14T17:25:00+02:002015-07-14T16:27:50+02:00AB4327-GANDImORMot FrameworkblogDelphiDocumentationmORMotORMSource<p>An enthusiastic <em>mORMot</em> user, named <em>willo</em> in the forum,
just <a href="https://tamingthemormot.wordpress.com/">started a blog about his
experiments</a> with our framework.</p>
<p><img src="http://www.uebergossenealm.at/tl_files/images/ZTN/murmeltier-dienten-hochkoenig.jpg" alt="" width="300" height="224" /></p>
<p>The information there is clear, simple, and right to the point.<br />
If you are a little lost in our huge documentation, it is a good place to
start!</p> <p>Extracted from <a href="https://tamingthemormot.wordpress.com/about">https://tamingthemormot.wordpress.com/about</a>
:</p>
<blockquote>
<p>While looking around for ideas and libraries to base our new project on, I
stumbled across Roberto Schneider’s excellent article <a href="https://robertocschneiders.wordpress.com/2012/11/22/datasnap-analysis-based-on-speed-stability-tests/">
DataSnap analysis based on Speed & Stability tests</a>. Before reading
this, I had no idea mORMot even existed and had I was happy and excited to have
discovered it. The one part missing from my ideal architectural stack was the
Linux based server, and here mORMot could leverage FPC to provide that missing
piece of the puzzle.</p>
<p>I quickly learned that this library was nor for the fainthearted.
Documentation was complete, but very hard to follow and some parts can only be
described as “left as an exercise for the reader…”. But I was determined. I
will not walk away. I’ve done this before, with TurboVision, and I’ll do it
again. With TurboVision, I had the help of a book, walking me through the
concepts and implementation quirks, with mORMot I have no such luxury. The
benefits promised by mORMot, as with TurboVision, is just too juicy to walk
away from – and that is the reason for this blog.</p>
<p>I will share my experiments and experiences here, and, hopefully, some souls
will find it useful and maybe it will help them flatten the enormously steep
mORMot learning curve.</p>
</blockquote>
<p>Thanks a lot, willo, for sharing!<br />
Feedback is welcome in willo's blog, or <a href="http://synopse.info/forum/viewtopic.php?id=2734">in our forum</a>, as
usual.</p>Faster String process using SSE 4.2 Text Processing Instructions STTNIurn:md5:1d4be4047c104e1087fe12e1242919672015-06-30T19:01:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkasmblogDelphiJSONmORMotperformanceSourcesse42sttni <p>A lot of our code, and probably yours, is highly relying on text
process.<br />
In our <em>mORMot</em> framework, most of its features use <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_2">JSON
text, encoded as UTF-8</a>.<br />
Profiling shows that a lot of time is spent computing the end of a text buffer,
or comparing text content.</p>
<p>You may know that In its SSE4.2 feature set, Intel added STTNI (<a href="https://software.intel.com/en-us/articles/xml-parsing-accelerator-with-intel-streaming-simd-extensions-4-intel-sse4">String
and Text New Instructions</a>) opcodes.<br />
They are several new instructions that perform character searches and
comparison on two operands of 16 bytes at a time.</p>
<p><a href="http://www.x86-secret.com/img/dossier/cpu/core_i7/slide-06b.png"><img src="http://www.x86-secret.com/img/dossier/cpu/core_i7/slide-06b.png" width="400" height="300/" /></a></p>
<p>I've just committed <a href="http://synopse.info/fossil/info/5447d35e1afd">optimized version of StrComp()
and StrLen()</a>, also used for our <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_48">
TDynArrayHashed</a> wrapper.<br />
The patch works from Delphi 5 up to XE8, and with FPC - unknown SSE4.2 opcodes
have been entered as hexadecimal bytes, for compatibility with the last century
compilers!<br />
The resulting speed up may be worth it!</p>
<p>Next logical step would be to use those instruction in the JSON process
itself.<br />
It may speed up the parsing speed of our core functions (which is already very
optimized, but written in a classical one-char-at-a-time reading).<br />
Main benefit would be to read the incoming UTF-8 text buffer by blocks of 16
bytes, and performing several characters comparison in a few CPU cycles, with
no branching.<br />
Also JSON writing would benefit for it, since escaping could be speed up thanks
to STTNI instructions.</p>
<p>Any <a href="http://synopse.info/forum">feedback is welcome</a>, as
usual!</p>CQRS Persistence Service of any DDD object with mORMoturn:md5:dd692ff755eabef2a773a43eea2cd1ec2015-05-18T14:07:00+02:002015-05-18T13:42:16+02:00AB4327-GANDImORMot FrameworkAggregateRootblogCQRSDelphiDocumentationDomainDrivenDTOEntityObjectinterfacemockmORMotORMperformanceRestSourceSQLstubTDDUnitTestingValueObject <p>We introduced DDD concepts some time ago, in a <a href="https://blog.synopse.info?post/post/2014/01/04/Domain-Driven-Design%3A-part-1">series of articles in this
blog</a>.<br />
At that time, we <a href="https://blog.synopse.info?post/post/2014/01/04/Domain-Driven-Design%3A-part-4">proposed a simple way of
using <em>mORMot</em> types</a> to implement DDD in your applications.<br />
But all Domain <em>Entitities</em> being tied to the framework
<code>TSQLRecord</code> class did appear as a limitation, breaking the
<em>Persistence Ignorance</em> principle, since it couples the DDD objects to
the framework implementation details.</p>
<p><img src="http://www.nomadier.com/images/packages/tibet/himalayan-marmot.jpg" width="450" height="150/" /></p>
<p>We introduced a new <a href="https://github.com/synopse/mORMot/blob/master/SQLite3/mORMotDDD.pas">mORMotDDD.pas</a>
unit, which is able to easily create <em>CQRS Persistence
services</em> for any plain Delphi class (the famous <em>PODO</em>s -
<em>Plain Old Delphi Objects</em>).<br />
No need to inherit from <code>TSQLRecord</code>, or pollute your class
definition with attributes!</p>
<p>For instance, a <code>TUser</code> class may be persisted via such a
service:</p>
<pre>
<strong>type</strong>
IDomUserCommand = <strong>interface</strong>(IDomUserQuery)
['{D345854F-7337-4006-B324-5D635FBED312}']
<strong>function</strong> Add(<strong>const</strong> aAggregate: TUser): TCQRSResult;
<strong>function</strong> Update(<strong>const</strong> aUpdatedAggregate: TUser): TCQRSResult;
<strong>function</strong> Delete: TCQRSResult;
<strong>function</strong> Commit: TCQRSResult;
<strong>end</strong>;
</pre>
<p>Here, the write operations are defined in a <code>IDomUserCommand</code>
service, which is separated (but inherits) from <code>IDomUserQuery</code>,
which is used for read operations.<br />
Separating reads and writes is a powerful pattern also known as CQRS,
i.e. <em>Command Query Responsibility Segregation</em>, which we followed
when defining our persistence services.<br />
The framework make it pretty easy to create such services for storing any kind
of <code>class</code> type in any SQL or NoSQL engine, with almost no code to
write.<br />
Last but not least, using such <code>interface</code>-based services for data
persistence will allow to <a href="https://blog.synopse.info?post/post/2012/10/14/Stubs-and-Mocks-for-Delphi-with-mORMot">stub or mock the data
access layer</a>, making unit testing straightforward: you would not fear to
write TDD code any more!</p>
<p>Please <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_169">
refer to our updated documentation for this unique and powerful
feature</a>.<br />
You may take a look at the corresponding <a href="https://github.com/synopse/mORMot/blob/master/SQLite3/DDD/dom/dddDomUserTypes.pas">
dddDomUserTypes.pas</a>, <a href="https://github.com/synopse/mORMot/blob/master/SQLite3/DDD/dom/dddDomUserCQRS.pas">
dddDomUserCQRS.pas</a>, and <a href="https://github.com/synopse/mORMot/blob/master/SQLite3/DDD/infra/dddInfraRepoUser.pas">
dddInfraRepoUser.pas</a> units, detailed as sample reference.<br />
Feedback is <a href="http://synopse.info/forum/viewforum.php?id=2">welcome in
our forum</a>, as usual!</p>Asynchronous Service - WebSockets, Callbacks and Publish-Subscribeurn:md5:4b0f2fbc0b9085026d4baacbf5ee8ed92015-04-06T21:34:00+02:002015-04-06T20:57:33+02:00AB4327-GANDImORMot FrameworkAESAES-NiAJAXauthenticationblogDatabaseDelphiDocumentationEventCollaborationEventSourcinginterfaceModelORMsecuritySOASOLIDSourceSQLite3SynopseWebSockets<p>When publishing SOA services, most of them are defined as
<em>stateless</em>, in a typical query/answer pattern - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_17">
Service-Oriented Architecture (SOA)</a></em>.<br />
This fits exactly with the <em>RESTful</em> approach of <em>Client-Server
services via interfaces</em>, as proposed by the framework.</p>
<p>But it may happen that a client application (or service) needs to know the
state of a given service. In a pure <em>stateless</em> implementation, it will
have to <em>query</em> the server for any state change, i.e. for any pending
notification - this is called <em>polling</em>.</p>
<p><img src="https://blog.synopse.info?post/public/mORMot/BatNotification.png" alt="" title="RealTime Notification, Apr 2015" /></p>
<p><em>Polling</em> may take place for instance:</p>
<ul>
<li>When a time consuming work is to be processed on the server side. In this
case, the client could not wait for it to be finished, without raising a
timeout on the HTTP connection: as a workaround, the client may start the work,
then ask for its progress status regularly using a timer and a dedicated method
call;</li>
<li>When an unpredictable event is to be notified from the server side. In this
case, the client should ask regularly (using a timer, e.g. every second), for
any pending event, then react on purpose.</li>
</ul>
<p>It may therefore sounds preferred, and in some case necessary, to have the
ability to let the server <em>notify</em> one or several clients without any
prior query, nor having the requirement of a client-side timer:</p>
<ul>
<li><em>Polling</em> may be pretty resource consuming on both client and server
sides, and add some unwanted latency;</li>
<li>If immediate notification is needed, some kind of "long polling" algorithm
may take place, i.e. the server will wait for a long time before returning the
notification state if no event did happen: in this case, a dedicated connection
is required, in addition to the REST one;</li>
<li>In an event-driven systems, a lot of messages are sent to the clients: a
proper publish/subscribe mechanism is preferred, otherwise the complexity of
polling methods may increase and become inefficient and unmaintainable;</li>
<li>Explicit push notifications may be necessary, e.g. when a lot of potential
events, associated with a complex set of parameters, are likely to be sent by
the client.</li>
</ul>
<p>Our <em>mORMot</em> framework is therefore able to easily implement
asynchronous callbacks over <em><a href="http://www.developerfusion.com/article/143158/an-introduction-to-websockets/">WebSockets</a></em>,
defining the callbacks as <code>interface</code> parameters in service method
definitions - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_154">
Available types for methods parameters</a></em>.</p> <h3>1. WebSockets support</h3>
<p>By definition, HTTP connections are stateless and one-way, i.e. a client
sends a request to the server, which replies back with an answer.<br />
There is no way to let the server send a message to the client, without a prior
request from the client side.</p>
<p><em>WebSockets</em> is a communication protocol which is able to
<em>upgrade</em> a regular HTTP connection into a dual-way communication
wire.<br />
After a safe handshake, the underlying TCP/IP socket is able to be accessed
directly, via a set of lightweight <em>frames</em> over an application-defined
<em>protocol</em>, without the HTTP overhead.</p>
<p>The <code><a href="http://synopse.info/files/html/api-1.18/SynBidirSock.html">SynBidirSock.pas</a></code>
unit implements low-level server and client <em>WebSockets</em>
communication.</p>
<p>The <code>TWebSocketProtocol</code> class defines an abstract <em>WebSockets
protocol</em>, currently implemented as several classes.</p>
<p>For our <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#SIDE_TITL_63">
Client-Server services via interfaces</a></em>, we would still need to make
<em>RESTful</em> requests, so the basic <em>WebSockets</em> framing has been
enhanced to support <code>TWebSocketProtocolRest</code> REST-compatible
protocols, able to use the single connection for both REST queries and
asynchronous notifications.</p>
<p>Two classes are available for your SOA applications:</p>
<ul>
<li><code>TWebSocketProtocolJSON</code> as a "pure" JSON light protocol;</li>
<li><code>TWebSocketProtocolBinary</code> as a binary proprietary protocol,
with optional frame compression and AES encryption (using AES-NI hardware
instructions, if available).</li>
</ul>
<p>In practice, on the server side, you would start your
<code>TSQLHttpServer</code> by specifying <code>useBidirSocket</code> as kind
of server:</p>
<pre>
HttpServer := TSQLHttpServer.Create('8888',[Server],'+',useBidirSocket);
</pre>
<p>Under the hood, it will instantiate a <code>TWebSocketServer</code> HTTP
server, as defined in <code>mORMotHttpServer.pas</code>, based on the sockets
API, able to upgrade the HTTP protocol into <em>WebSockets</em>.<br />
Our <em>High-performance http.sys server</em> is not yet able to switch to
<em>WebSockets</em> - and at API level, it would require at least <em>Windows
8</em> or <em>Windows 2012 Server</em>.</p>
<p>Then you enable <em>WebSockets</em> for the
<code>TWebSocketProtocolBinary</code> protocol, with an encryption key:</p>
<pre>
HttpServer.WebSocketsEnable(Server,'encryptionkey');
</pre>
<p>On the client side, you would use a <code>TSQLHttpClientWebsockets</code>
instance, as defined in <code>mORMotHttpClient.pas</code>, then explicitly
upgrade the connection to use <em>WebSockets</em> (since by default, it will
stick to the HTTP protocol):</p>
<pre>
Client := TSQLHttpClientWebsockets.Create('127.0.0.1','8888',TSQLModel.Create([]));
Client.WebSocketsUpgrade('encryptionkey');
</pre>
<p>The expected protocol detail should match the one on the server, i.e.
<code>'encryptionkey'</code> encryption over our binary protocol.</p>
<p>Once upgraded to <em>WebSockets</em>, you may use regular REST commands, as
usual:</p>
<pre>
Client.ServerTimeStampSynchronize;
</pre>
<p>But in addition to regular query/answer commands as defined for
<em>Client-Server services via interfaces</em>, you would be able to define
callbacks using <code>interface</code> parameters to the service methods.</p>
<p>Under the hood, both client and server will communicate using
<em>WebSockets</em> frames, maintaining the connection active using heartbeats
(via ping/pong frames), and with clean connection shutdown, from any side. You
can use the <code>Settings</code> property of the
<code>TWebSocketServerRest</code> instance, as returned by
<code>TSQLHttpServer.WebSocketsEnable()</code>, to customize the low-level
<em>WebSockets</em> protocol (e.g. timeouts or heartbeats) on the server side.
The <code>TSQLHttpClientWebsockets.WebSockets</code>.<code>Settings</code>
property would allow the same, on the client side.</p>
<p>We have observed, from our regression tests and internal benchmarking, that
using our <em>WebSockets</em> may be faster than regular HTTP, since its frames
would be sent as once, whereas HTTP headers and body are not sent in the same
TCP packet, and compression would be available for the whole frame, whereas
HTTP headers are not compressed. The ability to use strong AES encryption would
make this mean of communication even safer than plain HTTP, even with
<em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_151">
AES encryption over HTTP</a></em>.</p>
<h3>2. Using a callback to notify long term end-of-process</h3>
<p>An example is better than 100 talks.<br />
So let's take a look at the <code>Project31LongWorkServer.dpr</code> and
<code>Project31LongWorkClient.dpr</code> samples, from the <code><a href="https://github.com/synopse/mORMot/tree/master/SQLite3/Samples/31%20-%20WebSockets">
SQLite3\Samples\31 - WebSockets</a></code> sub-folder.<br />
They will implement a client/server application, in which the client launches a
long term process on the server side, then is notified when the process is
done, either with success, or failure.</p>
<p>First we define the interfaces to be used, in a shared
<code>Project31LongWorkCallbackInterface.pas</code> unit:</p>
<pre>
<strong>type</strong>
ILongWorkCallback = <strong>interface</strong>(IInvokable)
['{425BF199-19C7-4B2B-B1A4-A5BE7A9A4748}']
<strong>procedure</strong> WorkFinished(<strong>const</strong> workName: <strong>string</strong>; timeTaken: integer);
<strong>procedure</strong> WorkFailed(<strong>const</strong> workName, error: <strong>string</strong>);
<strong>end</strong>;<br /> ILongWorkService = <strong>interface</strong>(IInvokable)
['{09FDFCEF-86E5-4077-80D8-661801A9224A}']
<strong>procedure</strong> StartWork(<strong>const</strong> workName: <strong>string</strong>; <strong>const</strong> onFinish: ILongWorkCallback);
<strong>function</strong> TotalWorkCount: Integer;
<strong>end</strong>;
</pre>
<p>The only specific definition is the <code>const onFinish:
ILongWorkCallback</code> parameter, supplied to the
<code>ILongWorkService.StartWork()</code> method.<br />
The client will create a class implementing <code>ILongWorkCallback</code>,
then specify it as parameter to this method.<br />
On the server side, a "fake" class will implement
<code>ILongWorkCallback</code>, then will call back the client using the very
same <em>WebSockets</em> connection, when any of its methods will be
executed.</p>
<p>As you can see, a single callback <code>interface</code> instance may have
several methods, with their own set of parameters (here
<code>WorkFinished</code> and <code>WorkFailed</code>), so that the callback
may be quite expressive.<br />
Any kind of usual parameters would be transmitted, after serialization:
<code>string</code>, <code>integer</code>, but even <code>record</code>,
<em>dynamic arrays</em>, <code>TSQLRecord</code> or <code>TPersistent</code>
values.</p>
<p>When the <code>ILongWorkCallback</code> instance will be released on the
client side, the server will be notified, so that any further notification
won't create a connection error.<br />
We will see later how to handle those events.</p>
<h3>Client service consumption</h3>
<p>The client may be connected to the server as such (see the
<code>Project31LongWorkClient.dpr</code> sample source code for the full
details, including error handling):</p>
<pre>
<strong>var</strong> Client: TSQLHttpClientWebsockets;
workName: <strong>string</strong>;
Service: ILongWorkService;
callback: ILongWorkCallback;
<strong>begin</strong>
Client := TSQLHttpClientWebsockets.Create('127.0.0.1','8888',TSQLModel.Create([]));
<span style="background-color:yellow;">Client.WebSocketsUpgrade(PROJECT31_TRANSMISSION_KEY);</span>
Client.ServiceDefine([ILongWorkService],sicShared);
Client.Services.Resolve(ILongWorkService,Service);
</pre>
<p>Then we define our callback, using a dedicated class:</p>
<pre>
<strong>type</strong>
TLongWorkCallback = <strong>class</strong>(TInterfacedCallback,ILongWorkCallback)
<strong>protected</strong>
<strong>procedure</strong> WorkFinished(<strong>const</strong> workName: <strong>string</strong>; timeTaken: integer);
<strong>procedure</strong> WorkFailed(<strong>const</strong> workName, error: <strong>string</strong>);
<strong>end</strong>;<br /><strong>procedure</strong> TLongWorkCallback.WorkFailed(<strong>const</strong> workName, error: <strong>string</strong>);
<strong>begin</strong>
writeln(#13'Received callback WorkFailed(',workName,') with message "',error,'"');
<strong>end</strong>;
<br /><strong>procedure</strong> TLongWorkCallback.WorkFinished(<strong>const</strong> workName: <strong>string</strong>;
timeTaken: integer);
<strong>begin</strong>
writeln(#13'Received callback WorkFinished(',workName,') in ',timeTaken,'ms');
<strong>end</strong>;
</pre>
<p>Then we specify this kind of callback as parameter to start a long term
work:</p>
<pre>
<span style="background-color:yellow;">callback := TLongWorkCallback.Create(Client,ILongWorkCallback);</span>
<strong>try</strong>
<strong>repeat</strong>
readln(workName);
<strong>if</strong> workName='' <strong>then</strong>
break;
<span style="background-color:yellow;">Service.StartWork(workName,callback);</span>
<strong>until</strong> false;
<strong>finally</strong>
callback := <strong>nil</strong>; <em>// the server will be notified and release its "fake" class</em>
Service := <strong>nil</strong>; <em>// release the service local instance BEFORE Client.Free</em>
<strong>end</strong>;
</pre>
<p>As you can see, the client is able to start one or several work processes,
then expects to be notified of the process ending on its callback
<code>interface</code> instance, without explicitly polling the server for its
state, since the connection was upgraded to <em>WebSockets</em> via a call to
<code>TSQLHttpClientWebsockets.WebSocketsUpgrade()</code>.</p>
<h3>Server side implementation</h3>
<p>The server would define the working thread as such (see the
<code>Project31LongWorkServer.dpr</code> sample source code for the full
details):</p>
<pre>
<strong>type</strong>
TLongWorkServiceThread = <strong>class</strong>(TThread)
<strong>protected</strong>
<span style="background-color:yellow;">fCallback: ILongWorkCallback;</span>
fWorkName: <strong>string</strong>;
<strong>procedure</strong> Execute; <strong>override</strong>;
<strong>public</strong>
<strong>constructor</strong> Create(<strong>const</strong> workName: <strong>string</strong>; <strong>const</strong> callback: ILongWorkCallback);
<strong>end</strong>;
<br /><strong>constructor</strong> TLongWorkServiceThread.Create(<strong>const</strong> workName: <strong>string</strong>;
<strong>const</strong> callback: ILongWorkCallback);
<strong>begin</strong>
<strong>inherited</strong> Create(false);
<span style="background-color:yellow;">fCallback := Callback;</span>
fWorkName := workName;
FreeOnTerminate := true;
<strong>end</strong>;
<br /><strong>procedure</strong> TLongWorkServiceThread.Execute;
<strong>var</strong> tix: Int64;
<strong>begin</strong>
tix := GetTickCount64;
<span style="background-color:yellow;">Sleep(5000+Random(1000)); <em>// some hard work</em></span>
<strong>if</strong> Random(100)>20 <strong>then</strong>
<span style="background-color:yellow;">fCallback.WorkFinished(fWorkName,GetTickCount64-tix) <strong>else</strong></span>
<span style="background-color:yellow;">fCallback.WorkFailed(fWorkName,'expected random failure');</span>
<strong>end</strong>;
</pre>
<p>The callback is expected to be supplied as a <code>ILongWorkCallback</code>
interface instance, then stored in a <code>fCallback</code> protected field for
further notification.<br />
Some work is done in the <code>TLongWorkServiceThread.Execute</code> method
(here just a <code>Sleep()</code> of more than 5 seconds), and the end-of-work
notification is processed, as success or failure (depending on random in this
fake process class), on either of the <code>ILongWorkCallback</code> interface
methods.</p>
<p>The following <code>class</code> will define, implement and register the
<code>ILongWorkService</code> service on the server side:</p>
<pre>
<strong>type</strong>
TLongWorkService = <strong>class</strong>(TInterfacedObject,ILongWorkService)
<strong>protected</strong>
fTotalWorkCount: Integer;
<strong>public</strong>
<strong>procedure</strong> StartWork(<strong>const</strong> workName: <strong>string</strong>; <strong>const</strong> onFinish: ILongWorkCallback);
<strong>function</strong> TotalWorkCount: Integer;
<strong>end</strong>;
<br /><strong>procedure</strong> TLongWorkService.StartWork(<strong>const</strong> workName: <strong>string</strong>;
<strong>const</strong> onFinish: ILongWorkCallback);
<strong>begin</strong>
InterlockedIncrement(fTotalWorkCount);
<span style="background-color:yellow;">TLongWorkServiceThread.Create(workName,onFinish);</span>
<strong>end</strong>;
<br /><strong>function</strong> TLongWorkService.TotalWorkCount: Integer;
<strong>begin</strong>
result := fTotalWorkCount;
<strong>end</strong>;
<br /><strong>var</strong> HttpServer: TSQLHttpServer;
Server: TSQLRestServerFullMemory;
<strong>begin</strong>
Server := TSQLRestServerFullMemory.CreateWithOwnModel([]);
<span style="background-color:yellow;">Server.ServiceDefine(TLongWorkService,[ILongWorkService],sicShared);</span>
HttpServer := TSQLHttpServer.Create('8888',[Server],'+',useBidirSocket);
<span style="background-color:yellow;">HttpServer.WebSocketsEnable(Server,PROJECT31_TRANSMISSION_KEY);</span>
...
</pre>
<p>Purpose of those methods is just to create and launch the
<code>TLongWorkServiceThread</code> process from a client request, then
maintain a total count of started works, in a <code>sicShared</code> service
instance - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_92">
Instances life time implementation</a></em> - hosted in a
<code>useBidirSocket</code> kind of HTTP server.</p>
<p>We have to explicitly call <code>TSQLHttpServer.WebSocketsEnable()</code> so
that this server would be able to upgrade to our <em>WebSockets</em> protocol,
using our binary framing, and the very same encryption key as on the client
side - shared as a <code>PROJECT31_TRANSMISSION_KEY</code> constant in the
sample, but which may be safely stored on both sides.</p>
<h3>3. Publish-subscribe for events</h3>
<p>In event-driven architectures, the <em>publish-subscribe</em> messaging
pattern is a way of letting senders (called <em>publishers</em>) transmit
messages to their receivers (called <em>subscribers</em>), without any prior
knowledge of who those subscribers are.<br />
In practice, the <em>subscribers</em> will express interest for a set of
messages, which will be sent by the <em>publisher</em> to all the
<em>subscribers</em> of a given message, as soon as it is be notified.</p>
<p>In our <em>Client-Server services via interfaces</em> implementation,
messages are gathered in <code>interface</code> types, and each message defined
as a single method, their content being the methods parameters.<br />
Most of the SOA alternative (in Java or C#) do require class definition for
messages. Our KISS approach will just use method parameters values as message
definition.</p>
<p>To maintain a list of <em>subscribers</em>, the easiest is to store a
<em>dynamic array</em> of <code>interface</code> instances, on the
<em>publisher</em> side.</p>
<h3>Defining the interfaces</h3>
<p>We will now implement a simple <em>chat</em> service, able to let several
clients communicate together, broadcasting any message to all the other
connected instances.<br />
This sample is also located in the the <code><a href="https://github.com/synopse/mORMot/tree/master/SQLite3/Samples/31%20-%20WebSockets">
SQLite3\Samples\31 - WebSockets</a></code> sub-folder, as
<code>Project31ChatServer.dpr</code> and
<code>Project31ChatClient.dpr</code>.</p>
<p>So you first define the callback interface, and the service interface:</p>
<pre>
<strong>type</strong>
IChatCallback = <strong>interface</strong>(IInvokable)
['{EA7EFE51-3EBA-4047-A356-253374518D1D}']
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo, msg: <strong>string</strong>);
<strong>end</strong>;
<br /> IChatService = <strong>interface</strong>(IInvokable)
['{C92DCBEA-C680-40BD-8D9C-3E6F2ED9C9CF}']
<strong>procedure</strong> Join(<strong>const</strong> pseudo: <strong>string</strong>; <strong>const</strong> callback: IChatCallback);
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo,msg: <strong>string</strong>);
<strong>procedure</strong> CallbackReleased(<strong>const</strong> callback: IInvokable);
<strong>end</strong>;
</pre>
<p>Those interface types will be shared by both server and client sides, in the
common <code>Project31ChatCallbackInterface.pas</code> unit. The definition is
pretty close to what we wrote just above to notify long term
end-of-process.<br />
The only additional method is <code>IChatServer.CallbackReleased()</code>,
which, by convention, will be called on the server side when any
<code>callback</code> interface instance is released on the client side.</p>
<p>As such, the <code>IChatService.Join()</code> method will implement the
<em>subscription</em> to the chat service, whereas
<code>IChatServer.CallbackReleased()</code> will be called when the client-side
callback instance will be released (i.e. when its variable will be assigned to
<code>nil</code>), to <em>unsubscribe</em> for the chat service.</p>
<h3>Writing the Publisher</h3>
<p>On the server side, each call to <code>IChatService.Join()</code> would
<em>subscribe</em> to an internal list of connections, simply stored as an
<code>array of IChatCallback</code>:</p>
<pre>
<strong>type</strong>
TChatService = <strong>class</strong>(TInterfacedObject,IChatService)
<strong>protected</strong>
<span style="background-color:yellow;">fConnected: <strong>array of</strong> IChatCallback;</span>
<strong>public</strong>
<strong>procedure</strong> Join(<strong>const</strong> pseudo: <strong>string</strong>; <strong>const</strong> callback: IChatCallback);
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo,msg: <strong>string</strong>);
<strong>procedure</strong> CallbackReleased(<strong>const</strong> callback: IInvokable);
<strong>end</strong>;
<br /><strong>procedure</strong> TChatService.Join(<strong>const</strong> pseudo: <strong>string</strong>;
<strong>const</strong> callback: IChatCallback);
<strong>begin</strong>
<span style="background-color:yellow;">InterfaceArrayAdd(fConnected,callback);</span>
<strong>end</strong>;
</pre>
<p>The <code>InterfaceArrayAdd()</code> function, as defined in
<code>SynCommons.pas</code>, is a simple wrapper around any <em>dynamic
array</em> of <code>interface</code> instances, so that you may use it, or the
associated <code>InterfaceArrayFind()</code> or
<code>InterfaceArrayDelete()</code> functions, to maintain the list of
subscriptions.</p>
<p>Then a remote call to the <code>IChatService.BlaBla()</code> method should
be broadcasted to all connected clients, just by calling the
<code>IChatCallback.BlaBla()</code> method:</p>
<pre>
<strong>procedure</strong> TChatService.BlaBla(<strong>const</strong> pseudo,msg: <strong>string</strong>);
<strong>var</strong> i: integer;
<strong>begin</strong>
<strong>for</strong> i := 0 <strong>to</strong> high(fConnected) <strong>do</strong>
<span style="background-color:yellow;">fConnected[i].BlaBla(pseudo,msg);</span>
<strong>end</strong>;
</pre>
<p>Note that every call to <code>IChatCallback.BlaBla()</code> within the loop
would be made via <em>WebSockets</em>, in an asynchronous and non blocking way,
so that even in case of huge number of clients, the
<code>IChatService.BlaBla()</code> method won't block.<br />
In case of high numbers of messages, the framework is even able to
<em>gather</em> push notification messages into a single bigger message, to
reduce the resource use.</p>
<p>On the server side, the service implementation has been registered as
such:</p>
<pre>
Server.ServiceDefine(TChatService,[IChatService],sicShared).
SetOptions([],[optExecLockedPerInterface]);
</pre>
<p>Here, the <code>optExecLockedPerInterface</code> option has been set, so
that all method calls would be made thread-safe, so that concurrent access to
the internal <code>fConnected[]</code> list would be safe.<br />
Since a global list of connections is to be maintained, the service life time
is defined as <code>sicShared</code> - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_92">
Instances life time implementation</a></em>.</p>
<p>The following method will be called by the server, when a client callback
instance is released (either explicitly, or if the connection is broken), so
could be used to <em>unsubscribe</em> to the notification, simply by deleting
the callback from the internal <code>fConnected[]</code> array:</p>
<pre>
<strong>procedure</strong> TChatService.CallbackReleased(<strong>const</strong> callback: IInvokable);
<strong>begin</strong>
<span style="background-color:yellow;">InterfaceArrayDelete(fConnected,callback);</span>
<strong>end</strong>;
</pre>
<p>The framework will in fact recognize the following method definition in any
<code>interface</code> type for a service:</p>
<pre>
<strong>procedure</strong> CallbackReleased(<strong>const</strong> callback: IInvokable);
</pre>
<p>When a callback <code>interface</code> parameter (in our case,
<code>IChatCallback</code>) will be released on the client side, this method
will be called with the corresponding <code>interface</code> instance as
parameter.<br />
You do not have to call explicitly any method on the client side to
<em>unsubscribe</em> a service: assigning <em>nil</em> to a callback variable,
or feeing the <code>class</code> instance owning it as a field on the
<em>subscriber</em> side, will automatically unregister it on the
<em>publisher</em> side.</p>
<h3>Consuming the service from the Subscriber side</h3>
<p>On the client side, you implement the <code>IChatCallback</code> callback
interface:</p>
<pre>
<strong>type</strong>
TChatCallback = <strong>class</strong>(TInterfacedCallback,IChatCallback)
<strong>protected</strong>
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo, msg: <strong>string</strong>);
<strong>end</strong>;
<br /><strong>procedure</strong> TChatCallback.BlaBla(<strong>const</strong> pseudo, msg: <strong>string</strong>);
<strong>begin</strong>
writeln(#13'@',pseudo,' ',msg);
<strong>end</strong>;
</pre>
<p>Then you subscribe to your remote service as such:</p>
<pre>
<span style="background-color:yellow;"><strong>var</strong> Service: IChatService;</span>
<span style="background-color:yellow;">callback: IChatCallback;</span>
...
Client.ServiceDefine([IChatService],sicShared);
<span style="background-color:yellow;"><strong>if not</strong> Client.Services.Resolve(IChatService,Service) <strong>then</strong></span>
<strong>raise</strong> EServiceException.Create('Service IChatService unavailable');
...
<span style="background-color:yellow;">callback := TChatCallback.Create(Client,IChatCallback);</span>
<span style="background-color:yellow;">Service.Join(pseudo,callback);</span>
...
<strong>try</strong>
<strong>repeat</strong>
readln(msg);
<strong>if</strong> msg='' <strong>then</strong>
break;
<span style="background-color:yellow;">Service.BlaBla(pseudo,msg);</span>
<strong>until</strong> false;
<strong>finally</strong>
<span style="background-color:yellow;">callback := <strong>nil</strong>; <em>// will unsubscribe from the remote publisher</em></span>
Service := <strong>nil</strong>; <em>// release the service local instance BEFORE Client.Free</em>
<strong>end</strong>;
</pre>
<p>You could easily implement more complex <em>publish/subscribe</em>
mechanisms, including filtering, time to live or tuned broadcasting, by storing
some additional information to the <code>interface</code> instance (e.g. some
value to filter, a timestamp). A dynamic array of dedicated
<code>record</code>s, or a list of <code>class</code> instances, may be used to
store the <em>subscribers</em> expectations.</p>
<p>If you compare with existing client/server SOA solutions (in Delphi, Java,
C# or even in Go or other frameworks), this <code>interface</code>-based
callback mechanism sounds pretty unique and easy to work with.<br />
In fact, this is a good way of implementing callbacks conforming to <em>SOLID
design principles</em> on the server side, and let the <em>mORMot</em>
framework publish this mechanism in a client/server way, by using
<em>WebSockets</em>. The very same code could be used on the server side, with
no transmission nor marshaling overhead (via direct <code>interface</code>
calls), and over a network, with optimized use of resource and bandwidth (via
"fake" <code>interface</code> calls, and JSON marshalling over TCP/IP).</p>
<div style="margin-left: 2em">Ensure you <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_149">
take a look at the corresponding online documentation</a>, which would be
updated often than this blog article!<br />
The ORM part of the framework uses this <em>Publish-Subscribe</em> feature to
implement <a href="https://blog.synopse.info?post/post/2015/04/06/Real-Time-ORM-Master/Slave-Replication-via-WebSockets">Real-Time
master/slave replication</a>.<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=15632#p15632">welcome on our
forum</a>, as usual.</div>Real-Time ORM Master/Slave Replication via WebSocketsurn:md5:1709c8e98682a9bc997b21cade0a10352015-04-06T21:01:00+02:002015-04-06T20:09:38+02:00AB4327-GANDImORMot FrameworkAJAXblogDatabaseDelphiDocumentationEventCollaborationEventSourcinginterfaceModelORMreplicationSOLIDSourceSQLite3SynopseWebSockets<p>In a <a href="https://blog.synopse.info?post/post/2015/03/31/ORM-Master/Slave-Replication">previous
article</a>, we presented how Master/Slave replication may be easily
implemented in <em>mORMot</em>'s RESTful ORM.<br />
Do not forget to <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_147">
visit the corresponding paragraphs of our online documentation</a>, which has
been updated, and is more accurate!</p>
<p><img src="http://www.ibm.com/developerworks/library/wa-reverseajax2/fig01.gif" alt="" /></p>
<p>Sometimes, the on-demand synchronization is not enough.<br />
So we have just introduced real-time replication via <em>WebSockets</em>.<br />
For instance, you may need to:</p>
<ul>
<li>Synchronize a short list of always evolving items which should be reflected
as soon as possible;</li>
<li>Involve some kind of ACID-like behavior (e.g. handle money!) in your
replicated data;</li>
<li>Replicate not from a GUI application, but from a service, so use of a
<code>TTimer</code> is not an option;</li>
<li>Combine REST requests (for ORM or services) and master/slave ORM
replication on the same wire, e.g. in a multi-threaded application.</li>
</ul>
<p>In this case, the framework is able to use <em>WebSockets</em> and
asynchronous callbacks to let the master/slave replication - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_149">
Asynchronous callbacks</a></em> - take place without the need to ask explicitly
for pending data.<br />
You would need to use
<code>TSQLRestServer.RecordVersionSynchronizeMasterStart</code>,
<code>TSQLRestServer.RecordVersionSynchronizeSlaveStart</code> and
<code>TSQLRestServer.RecordVersionSynchronizeSlaveStop</code> methods over the
proper kind of bidirectional connection.</p> <p>The first requirement is to allow <em>WebSockets</em> on your
<em>Master</em> HTTP server, so initialize the <code>TSQLHttpServer</code>
class as a <code>useBidirSocket</code> kind of server - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_140">
Network and Internet access via HTTP</a></em>:</p>
<pre>
MasterServer := TSQLRestServerDB.Create(MasterModel,'master.db3');
HttpMasterServer := TSQLHttpServer.Create('8888',[MasterServer],'+',useBidirSocket);
HttpMasterServer.WebSocketsEnable(Server,'PrivateAESEncryptionKey');
</pre>
<p>On the <em>Slave</em> side, the HTTP client should also be upgraded to
support <em>WebSockets</em>:</p>
<pre>
MasterClient := TSQLHttpClientWebsockets.Create('127.0.0.1',HTTP_DEFAULTPORT,MasterModel);
MasterClient.WebSocketsUpgrade('PrivateAESEncryptionKey');
</pre>
<p>Of course, the model should match for both <code>MasterServer</code> and
<code>MasterClient</code> instances.<br />
As the <em>WebSockets</em> protocol definition - here above the same
<code>'PrivateAESEncryptionKey'</code> private key.</p>
<p>Then you enable the real-time replication service on the <em>Master</em>
side:</p>
<pre>
MasterServer.RecordVersionSynchronizeMasterStart;
</pre>
<p>In practice, it will publish a <code>IServiceRecordVersion
interface</code>-based service on the server side - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_63">
Client-Server services via interfaces</a></em>.</p>
<p>Assuming that the <em>slave</em> database has been defined as such:</p>
<pre>
SlaveServer := TSQLRestServerDB.Create(SlaveModel,'slave.db3');
</pre>
<p>(in this case, the <code>SlaveModel</code> may not be the same as the
<code>MasterModel</code>, but <code>TSQLRecordPeopleVersioned</code> should be
part of both models)<br />
Then you can initiate real-time replication from the <em>slave</em> side with a
single line, for a given table:</p>
<pre>
SlaveServer.RecordVersionSynchronizeSlaveStart(TSQLRecordPeopleVersioned,MasterClient);
</pre>
<p>The above command will subscribe to the remote <code>MasterSlave</code>
replication service (i.e. <code>IServiceRecordVersion interface</code>), to
receive any change concerning the <code>TSQLRecordPeopleVersioned</code> ORM
table, using the <code>MasterClient</code> connection via <em>WebSockets</em>,
and persist all updates into the local <code>SlaveServer</code> database.</p>
<p>To stop the real-time notification for this ORM table, you could
execute:</p>
<pre>
SlaveServer.RecordVersionSynchronizeSlaveStop(TSQLRecordPeopleVersioned);
</pre>
<p>Even if you do not call <code>RecordVersionSynchronizeSlaveStop()</code>,
the replication will be stopped when the main <code>SlaveServer</code> instance
will be released, and the <code>MasterServer</code> be <em>unsubscribe</em>
this connection for its internal notification list.</p>
<p>The real-time notification details have been tuned, to consume as minimum
bandwidth and resources as possible.<br />
For instance, if several modifications are to be notified on a slave connection
in a short amount of time, the master is able to gather those modifications as
a single <em>WebSockets</em> frame, which would be applied as a whole to the
slave database, in a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_28">
single BATCH transaction</a>.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=15577#p15577">welcome in our
forum</a>, as usual!</p>ORM Master/Slave Replicationurn:md5:08d531e552853cbab93bd086a1bab2902015-03-31T22:40:00+02:002015-04-05T07:27:38+02:00AB4327-GANDImORMot FrameworkbackupBatchblogDatabaseDelphiDocumentationGoodPracticeMaster-SlavemORMotreplicationRestshardingSourceSQLite3<p>As stated during <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_26">
TSQLRecord fields definition</a></em>, the ORM is able to maintain a revision
number for any <code>TSQLRecord</code> table, so that it the table may be
easily synchronized remotely by another <code>TSQLRestServer</code>
instance.<br />
If you define a <code>TRecordVersion</code> published property, the ORM core
will fill this field just before any write with a monotonically increasing
revision number, and will take care of any deletion, so that those
modifications may be replayed later on any other database.</p>
<p><img src="http://i41.tinypic.com/35lw00x.jpg" alt="" /></p>
<p>This synchronization will work as a strict <em>master/slave replication</em>
scheme, as a one-way on demand refresh of a replicated table.<br />
Each write operation on the master database on a given table may be easily
reflected on one or several slave databases, with almost no speed nor storage
size penalty.</p> <h3>Enable synchronization</h3>
<p>In order to enable this replication mechanism, you should define a
<code>TRecordVersion</code> published property in the <code>TSQLRecord</code>
class type definition:</p>
<pre>
TSQLRecordPeopleVersioned = <strong>class</strong>(TSQLRecordPeople)
<strong>protected</strong>
fFirstName: RawUTF8;
fLastName: RawUTF8;
fVersion: TRecordVersion;
<strong>published</strong>
<strong>property</strong> FirstName: RawUTF8 <strong>read</strong> fFirstName <strong>write</strong> fFirstName;
<strong>property</strong> LastName: RawUTF8 <strong>read</strong> fLastName <strong>write</strong> fLastName;
<strong>property</strong> Version: TRecordVersion <strong>read</strong> fVersion <strong>write</strong> fVersion;
<strong>end</strong>;
</pre>
<p>Only a single <code>TRecordVersion</code> field is allowed per
<code>TSQLRecord</code> class - it would not mean anything to manage more than
one field of this type.</p>
<p>Note that this field will be somewhat "hidden" to most ORM process: a
regular <code>TSQLRest.Retrieve</code> won't fill this <code>Version</code>
property, since it is an internal implementation detail.<br />
If you want to lookup its value, you would have to explicitly state its field
name at retrieval. Any <code>TRecordVersion</code> is indeed considered as a
"non simple field", just like BLOB fields, so would need explicit retrieval of
its value.</p>
<p>In practice, any <code>TSQLRest.Add</code> and <code>TSQLRest.Update</code>
on this <code>TSQLRecordPeopleVersioned</code> class will increase this
<code>Version</code> revision number field, and a <code>TSQLRest.Delete</code>
will populate an external <code>TSQLRecordTableDelete</code> table with the
<code>ID</code> of the deleted record, associated with a
<code>TRecordVersion</code> revision.</p>
<p>As consequences:</p>
<ul>
<li>The monotonic <code>TRecordVersion</code> number is shared at
<code>TSQLRestServer</code> level, among all tables containing a
<code>TRecordVersion</code> published field;</li>
<li>The <code>TSQLRecordTableDelete</code> table should be part of the
<code>TSQLModel</code>, in conjunction with
<code>TSQLRecordPeopleVersioned</code>;</li>
<li>If the <code>TSQLRecordTableDelete</code> table is not part of the
<code>TSQLModel</code>, the <code>TSQLRestServer</code> will add it - but you
should better make it explicitly appearing in the data model;</li>
<li>A single <code>TSQLRecordTableDelete</code> table will maintain the list of
all deleted data rows, of all tables containing a <code>TRecordVersion</code>
published field;</li>
<li>The <code>TSQLRecordPeopleVersioned</code> table appearance order in the
<code>TSQLModel</code> will matter, since <code>TSQLRecordTableDelete.ID</code>
will use this table index order in the database model to identify the table
type of the deleted row - in a similar way to <em>TRecordReference and
TRecordReferenceToBeDeleted</em>.</li>
</ul>
<p>All the synchronization preparation will be taken care by the ORM kernel on
its own, during any write operation. There is nothing particular to maintain or
setup, in addition to this <code>TRecordVersion</code> field definition, and
the global <code>TSQLRecordTableDelete</code> table.</p>
<h3>From master to slave</h3>
<p>To replicate this <code>TSQLRecordPeopleVersioned</code> table from another
<code>TSQLRestServer</code> instance, just call the following method:</p>
<pre>
aServer.RecordVersionSynchronizeSlave(TSQLRecordPeopleVersioned,aClient);
</pre>
<p>This single line will request a remote server via a <code>Client:
TSQLRestClientURI</code> connection (which may be over HTTP) for any pending
modifications since its last call, then will fill the local <code>aServer:
TSQLRestServer</code> database so that the local
<code>TSQLRecordPeopleVersioned</code> table will contain the very same content
as the remote master <code>TSQLRestServer</code>.</p>
<p>You can safely call
<code>TSQLRestServer.RecordVersionSynchronizeSlave</code> from several clients,
to replicate the master data in several databases.</p>
<p>Only the modified data will be transmitted over the wire, as two REST/JSON
queries (one for the insertions/updates, another for the deletions), and all
the local write process will use optimized <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_78">
BATCH writing</a>. This means that the synchronization process will try to use
as minimal bandwidth and resources as possible, on both sides.</p>
<p>Of course, the slaves should be considered as read-only, otherwise the
version numbers may conflict, and the whole synchronization may become a
failure.<br />
But you can safely replicate servers in cascade, if needed: the version numbers
will be propagated from masters to slaves, and the data will always be in a
consistent way.</p>
<h3>Replication use cases</h3>
<p>We may consider a very common corporate infrastructure:</p>
<p><a href="https://blog.synopse.info?post/public/mORMot/mORMotMasterSlaveReplication.png"><img src="https://blog.synopse.info?post/public/mORMot/.mORMotMasterSlaveReplication_m.jpg" alt="Master Slave ORM Replication" title="Master Slave ORM Replication, Mar 2015" /></a></p>
<p>This kind of installation, with a main central office, and a network of
local offices, would benefit from this master/slave replication.<br />
Simple <em>redirection</em> may be used - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_93">
Redirect to an external TSQLRest</a></em> - but it would expect the work to
continue, even in case of <em>Internet</em> network failure.<br />
REST redirection would expect a 100% connection uplink, which may be critical
in some cases.</p>
<p>You could therefore implement replication in several ways:</p>
<ul>
<li>Either the main office is the master, and any write would be push to the
<em>Main Server</em>, whereas local offices would have a replicated copy of the
information - drawback is that in case of network failure, the local office
would be able to only read the data;</li>
<li>Or each local office may host its own data in a dedicated table,
synchronized as a master database; the main office will replicate (as a slave)
the private data of each local servers; in addition, all this data gathered by
the <em>Main Server</em> may be further replication to the other local offices,
and be still accessible in read mode - in case of network failure, all the data
is available on the local servers, and the local private table is still
writable.</li>
</ul>
<p>Of course, the second solution seems preferable, even if a bit more
difficult to implement. The ablity of all local offices to work offline on
their own private data, but still having all the other data accessible as
read-only, would be a huge ROI.</p>
<p>As a benefit of using replication, the central main server would be less
stressed, since most of the process would take place in local servers, and the
main office server would only be used for shared data backup and read-only
gathering of the other local databases. Only a small network bandwith would be
necessary (much less than a pure web solution), and CPU/storage resources would
be minimal.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=11863#p11863">welcome on our
forum</a>, as usual!<br />
See also <a href="http://synopse.info/fossil/info/3453f314d9">the associated
feature request</a>, for upcoming features related to replication.<br />
And the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_147">
corresponding documentation article</a>, which will be updated often, so is
preferred to this static blog article!</p>GitHub temporary unavailable from comand line?urn:md5:8bc580cd68d4482d95fbbab44fa110502015-03-30T09:08:00+02:002015-03-30T11:46:48+02:00AB4327-GANDIOpen SourceblogDelphiFossilgitgithubSource<p>Perhaps it is due to a <a href="http://www.delphitools.info/2015/03/19/migrating-from-google-code-to-bitbucket/">
lot of projects moving from closing Google Code to GitHub</a>, but I
experimented some <em>GitHub</em> random communication errors since last
week.</p>
<p><img src="http://i.stack.imgur.com/oQMiy.png" alt="" /></p> <p>In fact, this was not at web interface level, but at <code>git</code>
command line level.</p>
<p>I have to repeat a simple <code>git push</code> commands several times
(up to 10 times), until the communication is accepted, and the source code
change is stored on the server...<br />
Pretty disappointing for a source code hosting service...<br />
Thanks to god, <code>git</code> is a distributed source code management system,
so I can keep the commits on my side until their server is up...</p>
<p><a href="http://synopse.info/fossil/timeline">Self hosting with our little
fossil tool</a>, in addition to using public hosting sites as mirror, sounded
like the right approach.<br />
By experiment, I do not trust outside services for storing my most precious
data, even more if the data - and the hosting service - are free.<br />
Using fossil on our web server is free, fast, safe, and in fact, full of
features.</p>
<p><strong>Update:<br /></strong><a href="https://plus.google.com/+ABouchez/posts/Encmpb1viXH">As reported by
Martin</a>, there is <a href="https://status.github.com">a DDOS attack going
on</a>, maybe this is the reason why this is not available...</p>Framework Documentation Enhanced By Linksurn:md5:1ca7755f719a93121470b7c0014d94152015-03-17T17:10:00+01:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkblogDelphiDocumentationHTMLmORMotSourceSynProject <p>The <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html">
<em>mORMot</em> framework documentation</a>, in its HTML online form, has been
enhanced to include links to almost of the code symbols.</p>
<p><img src="http://images.gadmin.st.s3.amazonaws.com/n18953/images/buehne/murmeltier_1-1.jpg" alt="" width="325" height="182" /></p>
<p>In fact, the latest version of our <a href="http://synopse.info/fossil/wiki?name=SynProject">SynProject tool</a> will
search for code symbols (types, methods, constants, functions): </p>
<ul>
<li>Everywhere in the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#SIDE_MORMOT_FRAMEWORK">
API Reference</a> pages;</li>
<li>In the text <code>formatted as code</code> in the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html">
main documentation</a> content.</li>
</ul>
<div>Some minor cosmetic changes did also occur, especially in the API
Reference.<br />
We hope it would help you discover and work with out little
<em>mORMot</em>!</div>SynCrypto: SSE4 x64 optimized asm for SHA-256urn:md5:17886bc69f9d62ab276f383dfa8bbd822015-02-21T12:58:00+01:002015-02-21T13:24:17+01:00AB4327-GANDIOpen Source libraries64bitasmblogDelphiperformanceshaSourcesse4<p>We have just included some optimized x64 assembler to our Open
Source <a href="http://synopse.info/fossil/info/b7ba18e68252b76c0fe">SynCrypto.pas</a> unit
so that SHA-256 hashing will perform at best speed.<br />
It is an adaptation from <a href="http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/sha-256-implementations-paper.html">
tuned Intel's assembly macros</a>, which makes use of the SSE4 instruction set,
if available.</p>
<p><img src="http://www.xbitlabs.com/images/cpu/core2extreme-qx9650/sse-1.jpg" alt="" width="275" height="182/" /></p> <p>Numbers are talking:</p>
<ul>
<li>under Win32, with a Core i7 CPU: pure pascal: 152ms - x86: 112ms</li>
<li>under Win64, with a Core i7 CPU: pure pascal: 202ms - SSE4: 78ms</li>
</ul>
<p>When executing the following test code:</p>
<pre>
for i := 1 to 100000 do begin
s := SHA256('123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890');
assert(s='f816ca413da6f2881c0cf16cb6d5bbc5d4189f5a9f185855c8bfd6423e099e52');
end;
</pre>
<p>Your <a href="http://synopse.info/forum/viewtopic.php?id=2374">feedback is
welcome</a>, as usual!</p>AES-NI enabled for SynCryptourn:md5:69688bfde7b83f67979567b29bcf85422015-01-15T20:49:00+01:002015-01-15T20:53:02+01:00AB4327-GANDIOpen SourceAESAES-NiasmblogDelphimORMotperformancesecuritySource <p>Today, we <a href="https://github.com/synopse/mORMot/commit/1b119af809a513">committed a new patch
to enable AES-NI hardware acceleration</a> to our <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html">SynCrypto.pas</a>
unit.</p>
<p><a href="http://www.intel.com/content/www/us/en/architecture-and-technology/advanced-encryption-standard--aes-/data-protection-aes-general-technology.html">
Intel® AES-NI</a> is a new encryption instruction set that improves on the
<em>Advanced Encryption Standard</em> (AES) algorithm and accelerates the
encryption of data on newer processors.</p>
<p><img src="https://pbs.twimg.com/media/BqkJVF7CMAA8Aww.jpg" alt="" width="300" height="202/" /></p>
<p>Of course, all this is available in the Delphi unit, from Delphi 6 to XE7:
no external <em>dll</em> nor OS update is needed.<br />
And it will <a href="https://blog.synopse.info?post/post/2015/01/10/mORMot-under-Linux-thanks-to-FPC">work
also on Linux</a>, so could help <a href="https://blog.synopse.info?post/post/2014/01/05/AES-encryption-over-HTTP">encrypting the mORMot
transmission</a> with no power loss.</p>
<p>You have nothing to do: just upgrade your <em>mORMot</em> source code, then
AES-NI instructions will be used, if the CPU offers it.<br />
We have seen performance boost of more than 5x, depending on the size of the
data to be encrypted.</p>
<p>Enjoy!</p>mORMot under Linux thanks to FPCurn:md5:a93fc22b55453318d6aef9fbb4a126f12015-01-10T13:48:00+01:002015-01-10T14:23:06+01:00AB4327-GANDImORMot FrameworkblogCrossPlatformDelphiDocumentationFreePascalinterfaceLinuxmockmORMotOpenSourceORMperformanceSOASourceSQLite3<p>You can use the <em><a href="http://freepascal.org/">FreePascal
Compiler</a></em> (FPC) to compile the <em>mORMot</em> framework source code,
targetting <em>Windows</em> and <em>Linux</em>.</p>
<p><img src="http://www.cloudreviews.com/blog/wp-content/uploads/2012/10/linux.jpg" /></p>
<p><em>Linux</em> is a premium target for cheap and efficient server
<em>Hosting</em>. Since <em>mORMot</em> has no dependency, installing a new
<em>mORMot</em> server is as easy as copying its executable on a blank
<em>Linux</em> host, then run it. No need to install any framework nor runtime.
You could even use diverse operating systems (several <em>Linux</em> or
<em>Windows Server</em> versions) in your <em>mORMot</em> servers farm, with
minimal system requirements, and updates.</p>
<p>We will now see how to write your software with Linux-compiling in mind, and
also give some notes about how to install a Linux Virtual Machine with
<em>Lazarus</em> on your <em>Windows</em> computer, compiling both FPC and
Lazarus from their SVN latest sources!</p> <h3>Compiler expectations</h3>
<p>You should better use the latest SVN trunk version of the FPC 2.7.1 / 3.1.1
compiler, and the corresponding <em>Lazarus</em> IDE.</p>
<p>If you want to use <em>TDocVariant custom variant type</em>, ensure that
your revision includes the fix for <a href="http://mantis.freepascal.org/view.php?id=26773">http://mantis.freepascal.org/view.php?id=26773</a>
bug, i.e. newer than revision 28995 from 2014-11-05T22:17:54. This bug has not
been fixed in 2.6.4 branch.</p>
<p>We recommend using the fpcup tool, as published at <a href="http://wiki.freepascal.org/fpcup">http://wiki.freepascal.org/fpcup</a><br />
To compile the latest svn version of the trunk, just write:</p>
<pre>
fpcup.exe --fpcURL="trunk" --lazURL="trunk"
</pre>
<p>Then ensure you set the static <em>SQlite3</em> .o files for Windows or
Linux in the right folder, as stated about the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_113">
Installation</a>.</p>
<h3>Creating the missing RTTI for interfaces</h3>
<p>Sadly, we have to face some unresolved FPC compiler-level issue, which does
not supply the needed <code>interface</code> RTTI - see <a href="http://bugs.freepascal.org/view.php?id=26774">http://bugs.freepascal.org/view.php?id=26774</a></p>
<p>As a result, SOA, mock/stub and MVC features would not working
directly.<br />
We propose a workaround to compile such applications with FPC. You could use
Delphi to generate one unit containing the needed information.</p>
<p>The <code>mORMotWrappers.pas</code> unit proposes a
<code>ComputeFPCInterfacesUnit()</code> function, which could be used on Delphi
to generate the RTTI unit for FPC, as such:</p>
<ul>
<li>Ensure that the application will use all its needed <code>interface</code>:
for instance, run all your regression tests, and/or use all its SOA/MVC
features if you are not confident about your test coverage;</li>
<li>Just before the application exists, add a call to
<code>ComputeFPCInterfacesUnit()</code> with the proper folders, e.g. at the
very end of your <code>.dpr</code> code.</li>
</ul>
<p>For instance, here is how <code>TestSQL3.dpr</code> has been modified:</p>
<pre>
<strong>program</strong> TestSQL3;
...
<strong>uses</strong>
...
<span style="background-color:yellow;">mORMotWrappers.pas,</span>
...
<strong>begin</strong>
SQLite3ConsoleTests;
<em>{$ifdef COMPUTEFPCINTERFACES}</em>
ChDir(ExtractFilePath(ParamStr(0)));
<span style="background-color:yellow;">ComputeFPCInterfacesUnit(</span>
<span style="background-color:yellow;">['..\CrossPlatform\templates','..\..\CrossPlatform\templates'],</span>
<span style="background-color:yellow;">'\..\..\SQlite3\TestSQL3FPCInterfaces.pas');</span>
<em>{$endif}</em>
<strong>end</strong>.
</pre>
<p>If you define the <code>COMPUTEFPCINTERFACES</code> conditional, the
<code>TestSQL3FPCInterfaces.pas</code> unit will be generated.<br />
You can take a look at <a href="https://github.com/synopse/mORMot/blob/master/SQLite3/TestSQL3FPCInterfaces.pas">
the committed version of this generated file</a>.</p>
<p>Of course, for your own application, you may use absolute path names: here
we used relative naming, via <code>..\</code>, so that it would work on any
development folder configuration.</p>
<p>Then, add it to any of your uses clause, as such:</p>
<pre>
<strong>uses</strong>
...
<span style="background-color:yellow;">TestSQL3FPCInterfaces, <em>// will register RTTI for interfaces under FPC</em></span>
...
</pre>
<p>This unit will do nothing when compiled under <em>Delphi</em>: it will
register the RTTI only when compiled with <em>FPC</em>.<br />
The rest of your code would be untouched, and could be shared between Delphi
and FPC.</p>
<p>If you do not modify the <code>interface</code> methods definition, this
generation step could be safely bypassed.</p>
<p>We hope that in a close future, the FPC team would fix the <a href="http://bugs.freepascal.org/view.php?id=26774">http://bugs.freepascal.org/view.php?id=26774</a>
issue, but the ticket seems pretty inactive since its creation.</p>
<h3>Writing your project for FPC</h3>
<p>If you want your application to compile with FPC, some little patterns
should be followed.</p>
<p>In all your source code file, the easiest is to including the following
<code>mORMot</code> file, which will define all compiler options and
conditionals as expected:</p>
<pre>
<em>{$I Synopse.inc} // define HASINLINE USETYPEINFO CPU32 CPU64 OWNNORMTOUPPER</em>
</pre>
<p>Then in your <code>.dpr</code> file, you should write:</p>
<pre>
<strong>uses</strong>
<em>{$ifdef Linux}</em>
<em>// if you use threads</em>
cthreads,
<em>// widestring manager for Linux if needed !!</em>
<em>// could also be put in another unit ... but doc states: as early as possible</em>
cwstring, <em>// optional</em>
<em>{$endif}</em>
</pre>
<p>For instance a minimal FPC project to run the regression tests may be:</p>
<pre>
<strong>program</strong> LinuxSynTestFPCLinuxi386;
<br /><em>{$I Synopse.inc}
{$APPTYPE CONSOLE}</em>
<br /><strong>uses</strong> <em>{$ifdef Linux}</em> cthreads, cwstring, <em>{$endif}</em> mORMotSelfTests;
<br /><strong>begin</strong>
SQLite3ConsoleTests;
<strong>end</strong>.
</pre>
<p>In your user code, ensure you do not directly link to the
<code>Windows</code> unit, but rely on the cross-platform classes and functions
as defined in <code>SysUtils.pas</code>, <code>Classes.pas</code> and
<code>SynCommons.pas</code>.<br />
You could find in <code>SynFPCTypInfo.pas</code> and
<code>SynFPCLinux.pas</code> some low-level functions dedicated to FPC and
Linux compilation, to be used with legacy units - your new code should better
rely on higher level functions and classes.</p>
<p>If you rely on <em>mORMot</em> classes and types, e.g. use
<code>RawUTF8</code> for all your <code>string</code> process in the business
logic, and do not use Delphi-specific features (like generics, or new syntax
sugar), it would be very easy to let your application compile with FPC.</p>
<p>In practice, we use Delphi as our main IDE, then switch to Lazarus under a
small integrated <em>Linux VirtualBox</em>, running a low resource
<em>XFCE</em> desktop. We defined the <em>Windows</em> folder containing the
source as a <em>VirtualBox</em> shared folder, so that we are able to compile,
debug and test the <em>Linux</em> version of any executable in native
<em>Linux</em>, on the same computer, from the very same sources. We found out
that <em>Lazarus</em> debugging was pretty smooth on <em>Linux</em> - GDB is
smoother on <em>Linux</em> than <em>Windows</em>, by the way. Then switching
from <em>Delphi/Windows</em> to <em>Lazarus/Linux</em> is direct and natural,
especially when the <em>VirtualBox</em> "Integrated Desktop" feature is
enabled.</p>
<h3>Linux installation tips</h3>
<p>Here are a few informal notes about getting running a FPC/Lazarus virtual
machine running <em>XUbuntu</em>, on a <em>Windows</em> host.<br />
They are published as a general guideline, and we would not provide any
reference procedure, nor support it.</p>
<ul>
<li>Install the latest <em>VirtualBox</em> version from <a href="http://www.virtualbox.org/">http://www.virtualbox.org/</a> to Windows;</li>
<li>Download the latest <code>.iso</code> version published at <a href="http://xubuntu.org/">http://xubuntu.org/</a> or any other place - we use XFCE
since it is a very lightweight desktop, perfect to run <em>Lazarus</em>, and we
selected an Ubuntu LTS revision (14.04 at the time of this writing), which
would be the same used on Internet servers;</li>
<li>Create a new virtual machine (VM) in <em>VirtualBox</em>, with 1 or 2 CPUs,
more than 512 MB of RAM (we use 777 MB), and an automatic-growing disk storage,
with a maximal size of 15 GB; ensure that the disk storage is marked as SSD if
your real host storage is a SSD;</li>
<li>Let the CDROM storage point to the <code>.iso</code> you downloaded;</li>
<li>Start the VM and install <em>Linux</em> locally, as usual - you may select
to download the updated packages during the installation, for safety;</li>
<li>When the system restarts, if it asks for software updates, accept and wait
for the update installation to finish - it is a good idea to have the latest
version of the kernel and libraries before installing the <em>VirtualBox</em>
drivers;</li>
<li>Restart your VM when asked to;</li>
<li>Under a Ubuntu/Debian terminal, write the following commands:</li>
</ul>
<pre>
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install dkms
</pre>
<ul>
<li>Restart the VM, then select "Insert Guest Additions CD image" from the VM
"Devices" menu: a virtual CD would be mounted on your system and appear on your
desktop;</li>
<li>Run the following command, according to your current user name and
<em>VirtualBox</em> version:</li>
</ul>
<pre>
sudo sh /media/...user.../VBOXADDITIONS_..../VBoxLinuxAdditions.run
</pre>
<ul>
<li>Restart the VM, then add a permanent shared folder in the VM configuration,
named <code>Lib</code>, and pointing to your local <em>mORMot</em> installation
(e.g. <code>d:\dev\lib</code>;</li>
<li>Create a void folder, e.g. in your home:</li>
</ul>
<pre>
mkdir lib
</pre>
<ul>
<li>Create a launcher for the following command, to mount the shared folder as
expected:</li>
</ul>
<pre>
sudo mount -t vboxsf lib /home/...user.../lib
</pre>
<ul>
<li>Download the version of <code>fpcup</code> for your system, e.g. <a href="https://bitbucket.org/reiniero/fpcup/downloads/fpcup_linux_x86">https://bitbucket.org/reiniero/fpcup/downloads/fpcup_linux_x86</a></li>
<li>Execute the following commands:</li>
</ul>
<pre>
sudo apt-get install build-essential mingw32-binutils subversion libgtk2.0-dev
sudo ln -s /usr/bin/i586-mingw32msvc-windres /usr/bin/windres
./fpcup_linux_x86 --fpcURL="trunk" --lazURL="trunk"
</pre>
<ul>
<li>Now wait for all source to be retrieved by SVN, then the whole build to
take places;</li>
<li>If you have issues during SVN retrieval, go the
<code>development/fpc</code> folder, then run the following before trying again
the <code>fpcup_linux_x86</code> command:</li>
</ul>
<pre>
svn cleanup
svn update
</pre>
<ul>
<li>On success, you can create a launcher pointing to
<code>development/lazarus/startlazarus</code>.</li>
</ul>
<p>If you followed the above steps, you should now have at least a Lazarus IDE
v1.3 and the corresponding FPC 3.1.1 compiler. It is amazing seeing the whole
compiler + IDE being compiled from the official sources, for free, and in a few
minutes.</p>
<p>In fact, this article has been extracted from our official documentation,
which <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_125">
should be checked for its latest version</a>!</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=2279">welcome on our forum</a>, as
usual!</p>Breaking change: New SynLog and SynTests unit extracted from SynCommons.pasurn:md5:dde99923fc21f7edb121e51ed86df8682014-11-23T19:22:00+01:002014-11-23T19:24:48+01:00AB4327-GANDIOpen Source librariesblogDelphiDocumentationlogmORMotSourceTesting <p>In order to enhance code modularity, we extracted logging and testing
features from <a href="http://synopse.info/files/html/api-1.18/SynCommons.html">SynCommons.pas</a>.<br />
Discover the new <a href="http://synopse.info/files/html/api-1.18/SynLog.html">SynLog.pas</a> and
<a href="http://synopse.info/files/html/api-1.18/SynTests.html">SynTests.pas</a>
units!<br />
Documentation has been updated to reflect the changes.</p>
<p>This is a breaking change...<br />
Ensure you add <code>SynLog</code> and/or <code>SynTests</code> to your uses
clauses, just after <code>SynCommons</code>, if you use those logging or
testing features in your programs.</p>
<p><img alt="" src="http://ecx.images-amazon.com/images/I/51RAR1O55SL._SY355_.jpg" /></p>
<p>The corresponding commit is <a href="http://synopse.info/fossil/info/881797779c">this one</a>.</p>
<p>Next code refactoring would probably be about extracting some features from
<a href="http://synopse.info/files/html/api-1.18/mORMot.html">mORMot.pas</a>.<br />
Stay tuned!</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=13808#p13808">welcome on our
forum</a>, as usual.</p>