Synopse Open Source - Tag - SOAmORMot MVC / SOA / ORM and friends2024-02-02T17:08:25+00:00urn:md5:cc547126eb580a9adbec2349d7c65274DotclearEfficient Routing for Christmasurn:md5:a0b8f306951aec1c0a62023f50b1d8d22022-12-28T13:52:00+00:002022-12-28T13:58:58+00:00Arnaud BouchezmORMot FrameworkblogGoodPracticeHTTPmORMotmORMot2ORMperformanceRadixTreeRestroutingSOA<p>This is perhaps the last new feature of <em>mORMot 2</em> before its first stable release: a very efficient custom URI routing for our HTTP/HTTPS servers.</p>
<p><img src="https://blog.synopse.info?post/public/blog/SantaRouting.jpeg" alt="" /></p>
<p>At ORM and SOA level, there is by-convention routing of the URI, depending on the ORM table, SOA interface and method, and <code>TOrmModel.Root</code> value. Even for our MVC web part, we rely on a <code>/root/</code> URI prefix, which may not be always needed.<br />
Relying on convention is perfect between <em>mORMot</em> clients and servers, but in some cases, it may be handy to have something smoother, e.g. to publish a truly REST scheme.</p>
<p>We introduced two routing abilities to <em>mORMot 2</em>, with amazing performance (6-12 million parsings per CPU core), via a new <em><code>THttpServerGeneric.Route</code></em><em></em> property:</p>
<ul>
<li>Internal URI rewrite, to redirect internally from a human/REST-friendly request e.g. to a SOA <code>/root/interface.method</code> layout, or to a MVC web page;</li>
<li>Direct callback execution, with optional parameter parsing.</li>
</ul>
<p><strong>Article edited on 28th December:</strong><br />
Fixed performance numbers (much higher than reported), and introduced latest source changes.</p> <h4>The New TUriRouter Class</h4>
<p><img src="https://blog.synopse.info?post/public/blog/Routing.jpg" alt="" /></p>
<p>From <code>THttpServerGeneric.Route</code>, or from <code>TRestHttpServer.Route</code>, you can access a new <code>TUriRouter</code> class.<br />
It is the class responsible of the core registration process of all custom URI parsing.</p>
<p>By default, it is disabled. The classical <em>mORMot</em> routing applies.<br />
But once you call <code>THttpServerGeneric.Route</code> or <code>TRestHttpServer.Route</code>, you can register URIs and how the HTTP server should process it.</p>
<h4>Internal URI rewrite</h4>
<p>Here, we are not talking about HTTP redirection, i.e. returning a <code>30x</code> HTTP status code to let the client make a new request to another URI.<br />
We allow URI rewriting on the fly, within the server, just before the incoming request is identified to the ORM, MVC or SOA <em>mORMot</em> router.</p>
<p>It offers for instance an alternative URI path to the method-based services or the interface based services, at HTTP/HTTPS server level.</p>
<p>Now, we could write:</p>
<pre>
Server.Route.Get('/info', 'root/timestamp/info');
</pre>
<p>So that any GET on <code>/info</code> will redirect to the internal <code>TRestServer.TimeStamp</code> method-based services, and its hidden <code>/info</code> sub-method, which displays some general statistics about the server.</p>
<p>Or we could write:</p>
<pre>
Server.Route.Get('/user/<id>', '/root/userservice/new?id=<id>', urmPost);
</pre>
<p>to rewrite internally e.g. the GET <code>'/user/1234'</code> URI into a POST at <code>'/root/userservice/new?id=1234'</code>, as published by a <code>IUserService.New(id: Int64)</code> interface-based service method.</p>
<p>As such, you could have the best of both worlds.</p>
<p>This URI redirection may also have a very high benefit for a <em>mORMot</em> MVC web application. You could easily redirect some human-friendly URIs into the MVC routing convention, as expected by the MVC interface definition.</p>
<p>Last but not least, if the redirected URI is an integer in range 200..599, then the server will abort the request immediately with an HTTP status error matching the integer:</p>
<pre>
Server.Route.Get('/admin.php', '403');
</pre>
<p>This could help to avoid calling the main REST engine, or write a callback, just to return an error code.</p>
<h4>Direct Callbacks Execution</h4>
<p>As an alternative, you can assign a <code>TOnHttpServerRequest</code> callback with a given URI, optionally with <code><parameters></code>:</p>
<pre>
TOnHttpServerRequest = function(Ctxt: THttpServerRequestAbstract): cardinal of object;
</pre>
<p>The <code>Ctxt</code> instance is the low-level structure holding the HTTP/HTTPS request, with all input and output context.<br />
It even has a property to retrieve the named parameters within the URI, i.e. an <code><id></code> place holder in the URI registration will be recognized, and available within the callback from the <code>Ctxt['id']</code> property.</p>
<p>For instance, it could be used to publish a standard REST process as:</p>
<pre>
// retrieve a list of picture IDs
Server.Route.Get('/user/<user>/pic', DoUserPic);
// support CRUD access of a given picture by ID
Server.Router.Run([urmGet, urmPost, urmPut, urmDelete], '/user/<user>/pic/<id>', DoUserPic)
</pre>
<p>Then the callback could be something like this:</p>
<pre>
function TMyClass.DoUserPic(Ctxt: THttpServerRequestAbstract): cardinal;
var
user: RawUtf8;
id: Int64;
ids: TInt64DynArray;
begin
user := Ctxt['user'];
if Ctxt.RouteInt64('id', id) then
// manage /user/<user>/pic/<id>
if CRUDUserPictureFromDatabase(Ctxt, user, id) then
result := HTTP_SUCCESS
else
result := HTTP_NOTFOUND;
else if RetrieveUserPictureIDListFromDatabase(Ctxt, user, ids) then
// returned /user/<user>/pic
result := HTTP_SUCCESS
else
result := HTTP_NOTFOUND;
end;
</pre>
<p>Of course, URI redirection to an interface-based service may be more convenient, but if you want to reuse some existing code, and have the best performance possible, you could follow this pattern.</p>
<p>It also may be handy for some low-level tasks of the HTTP server, like proxying to internal sub-servers, or quickly return some 30x redirection, or generate some HTML pages.</p>
<p>If the callback returns a <code>result</code> of 0, then execution will continue as usual, but you can change some <code>Ctxt</code> fields. It may allow for very efficient and tuned client authorization, even before you execute some regular interface-based services.</p>
<p>You could also redirect all published methods of a class instance using RTTI, via <code>TRouterUri.RunMethods()</code>.<br />
Just like <code>TRestServer</code> method-based services, but here at HTTP server level, with even higher performance, since the <em>mORMot</em> REST engine is not involved.</p>
<p>Another use may be to handle some very tuned HTTP OPTIONS requests, if the default CORS feature is too broad for your case.<br />
Or quickly return a standard response for HTTP HEAD requests, without any Content-Length header as it is allowed, to leverage the server process.</p>
<h4>Why Not Make It Fast ?</h4>
<p><img src="https://blog.synopse.info?post/public/blog/marmotblackwhite.png" alt="" /></p>
<p>About performance, its <code>TUriRouter.Process()</code> method is done with no memory allocation for a static route, using a <a href="https://en.wikipedia.org/wiki/Radix_tree">very efficient Radix Tree algorithm</a> for path lookup, over a thread-safe non-blocking URI parsing with values extractions for rewrite or execution.</p>
<p>Here are some numbers from <code>TNetworkProtocols._TUriTree</code> on my old Core i5 laptop, on a single thread/core:</p>
<pre>
1000 URI lookups in 37us i.e. 25.7M/s, aver. 37ns
1000 URI static rewrites in 80us i.e. 11.9M/s, aver. 80ns
1000 URI parametrized rewrites in 117us i.e. 8.1M/s, aver. 117ns
1000 URI static execute in 91us i.e. 10.4M/s, aver. 91ns
1000 URI parametrized execute in 162us i.e. 5.8M/s, aver. 162ns
</pre>
<p>As you can see, this routing won't be the bottleneck in your server process. It has a non blocking O(1) complexity, with no unneeded memory allocation during its process.<br />
I know no other Delphi or FPC web or REST framework using custom Radix Tree data structures. Other libraries parse the URI as parts, then check the parts against registered routed (using hash maps if possible). It is clearly less efficient, and has some known disadvantages we will discuss about.</p>
<h4>How Does It work?</h4>
<p>If we run the following code (from our regression tests):</p>
<pre>
router.Get('/plaintext', DoPlainText);
router.Get('/', DoRequestRoot);
router.Get('/do/<one>/pic/<two>', DoRequest0);
router.Get('/do/<one>', DoRequest1);
router.Get('/do/<one>/pic', DoRequest2);
router.Get('/do/<one>/pic/<two>/', DoRequest3);
router.Get('/da/<one>/<two>/<three>/<four>/', DoRequest4);
writeln(router.Tree[urmGet].ToText);
</pre>
<p>We get the following output on the console:</p>
<pre>
/
d
a/
<one>
/
<two>
/
<three>
/
<four>
/
o/
<one>
/pic
/
<two>
/
plaintext
</pre>
<p>The Radix Tree is expanded as spaces. Above lines make it easy to understand how the URI parsing is done: for the top <code>/</code> node to the last exact matching node.<br />
A node can be some static text like <code>/pic</code> or a parameter like <code><one></code>. The registered routes (either URI rewrite, or a callback) are assigned to one node.</p>
<p>This data structure has several benefits, for our parametrized routing scheme.</p>
<p>1. The nodes form a memory structure very easy to parse an URI, one character per character, even with thousands of registered routes.</p>
<p>2. Unlike hash-maps, a tree structure also allows us to use dynamic parts like the <code><one></code> parameter, since we actually match against the routing patterns instead of just comparing hashes. Hashes map static values, while we need to map URI parameters as dynamic values. Our tree is perfect for this purpose.</p>
<p>3. It greatly reduces the classical routing problem of regular registration based on lists and maps, which can suffer from unexpected behavior, just due to the order of the URI registration calls. With our data structure, we know that there is a single node per path, before even starting the look-up in the prefix-tree. Thanks to the tree structure, we know by design that nodes won't overlap.</p>
<p>4. For even better scalability, the child nodes on each tree level are sorted by depth/priority/usage after each registration. The depth/priority/usage is just the number of sub nodes: children, grandchildren, and so on.. Nodes which are part of the most routing paths are evaluated first. This helps to make as much routes as possible to be reachable as fast as possible. It is also some sort of cost compensation. The longest reachable path (highest cost) can always be evaluated first. You can see this sorting in the above tree sample.</p>
<p>5. As you can see, there is one Radix Tree per supported HTTP method, which are GET, POST, PUT, DELETE, HEAD and OPTIONS. For one thing it is more efficient than holding a per-method registration in every single node, for another thing it greatly reduces any routing problem on path overlapping between methods.</p>
<p>In practice, it is very fast and efficient: from 6 to 12 million of URI parsed per CPU core, depending on the process (static URI or parametrized URI).</p>
<p>If you are curious, you could look at the source code in our repository:<br />
<a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.server.pas">https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.server.pas</a><br />
Some of the code may be a bit difficult to follow, since we use low-level pointers over chars, to avoid intermediate <code>string</code> allocations, and to offer the best performance. Even the values are parsed and stored as integer indexes and lengths, not as pre-allocated <code>string</code> instance... But you could see the classes hierarchy, and how the registration is done - registration is less performance sensitive, so the code is more high-level here. ;-)</p>
<h4>What Does The Marmot Say?</h4>
<p><img src="https://blog.synopse.info?post/public/blog/marmotinflowers.jpg" alt="marmotinflowers.jpg, Dec 2022" /></p>
<p><strong>Merry Christmas again, and enjoy!</strong></p>New Async HTTP/WebSocket Server on mORMot 2urn:md5:b5d5687573d19a81f6d6dadfbc68461d2022-05-21T13:35:00+01:002022-05-21T18:05:45+01:00Arnaud BouchezmORMot FrameworkDelphiFPCfpcx64mmHTTPhttp.sysHTTPSLinuxmORMotmORMot2multithreadperformanceRESTRestsecuritySOA<p>The HTTP server is one main part of any SOA/REST service, by design.<br />
It is the main entry point of all incoming requests. So it should better be stable and efficient. And should be able to scale in the future, if needed.</p>
<p><img src="https://blog.synopse.info?post/public/blog/server.jpg" alt="" /></p>
<p>There have always been several HTTP servers in <em>mORMot</em>. You can use the HTTP server class you need.<br />
In <em>mORMot</em> 2, we added two new server classes, one for publishing over HTTP, another able to upgrade to WebSockets. The main difference is that they are fully event-driven, so their thread pool is able to scale with thousands of concurrent connections, with a fixed number of threads. They are a response to the limitations of our previous socket server.</p> <h4>HTTP Is Not REST</h4>
<p>In <em>mORMot</em>, the HTTP server does not match the REST server. They are two concepts, with diverse units, classes and even... source code folders.<br />
The HTTP server can publish its own process, using a callback, or can leverage one or several REST servers. Or the REST server could be with no communication at all, e.g. be run in the service process within the current thread, executing ORM or SOA requests without any HTTP or WebSockets involved. No difference in the user code: just some interface methods to call, and they will return their answer, whatever it is over HTTP, locally or remotely, on the server side or the client side, in a thread pool or in-process... pascal code just runs its magic for you.</p>
<p>For instance, take a look at <a href="https://github.com/synopse/mORMot2/blob/master/ex/http-server-raw/httpServerRaw.dpr">httpServerRaw</a> as a low-level HTTP server sample, using a callback for every incoming request.<br />
Here you don't have any automatic routing: you just parse the input URI, then return the proper HTTP response, with its status code, headers and body.<br />
Those low-level HTTP servers are implemented in the <em>server</em> units of folder <a href="https://github.com/synopse/mORMot2/tree/master/src/net">src/net</a> - as we will detail below.</p>
<p>Then, the REST process is abstracted from HTTP. Even if both HTTP and REST historically share the <a href="https://en.wikipedia.org/wiki/Roy_Fielding">same father/author/initiator</a>, you could have a REST approach without HTTP. For instance, you could use WebSockets, or direct in-process call.<br />
So in <em>mORMot</em>, the REST process is implemented in the <em>server</em> units of folder <a href="https://github.com/synopse/mORMot2/tree/master/src/rest">src/rest</a>, abstracted from any communication protocol. It does not know nor assume anything about TCP, WebSockets or TLS. Just about URI, headers and body texts, and follow the routing as defined.</p>
<p>Two folders, two uncoupled feature sets. Perhaps a bit confusing when you discover it first. But for the best maintainability and code design.</p>
<h4>Several Servers To Rule Them All</h4>
<p><em>mORMot</em> 2 adopted all HTTP server classes from <em>mORMot</em> 1 source code. Then include some new "asynchronous" servers.<br />
They all inherit from a <code>THttpServerGeneric</code> parent class, so you can follow the Liskow Substitution Principle, and change the class at runtime or compilation, as needed, without altering your actual logic.</p>
<p>HTTP servers are implemented in several units:</p>
<ul>
<li><a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.server.pas"><em>mORMot</em>.net.server.pas</a> offers the <code>THttpServerSocket</code>/<code>THttpServer</code> HTTP/1.1 server, the <code>THttpApiServer</code> HTTP/1.1 server over Windows http.sys module, and <code>THttpApiWebSocketServer</code> over Windows http.sys module;</li>
<li><a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.ws.server.pas"><em>mORMot</em>.net.ws.server.pas</a> offers the <code>TWebSocketServerRest</code> server, which uses WebSockets as mean of transmission, but enable a REST-like blocking request/answer protocol on top of it, with optional bi-directional notifications, using a one-thread-per-connection server;</li>
<li><a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.async.pas"><em>mORMot</em>.net.async.pas</a> offers the new <code>THttpAsyncServer</code> event-driven HTTP server;</li>
<li><a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.ws.async.pas"><em>mORMot</em>.net.ws.async.pas</a> offers the new <code>TWebSocketAsyncServerRest</code> server, which uses WebSockets as mean of transmission, but enable a REST-like blocking request/answer protocol on top of it, with optional bi-directional notifications, using an event-driven server.</li>
</ul>
<p>On Windows, the <a href="https://docs.microsoft.com/en-us/iis/get-started/introduction-to-iis/introduction-to-iis-architecture#hypertext-transfer-protocol-stack-httpsys">http.sys</a> module gives you very good stability, and uses the same Windows-centric way of publishing servers as used by IIS and DotNet. You could even share the same port between several services, if needed.</p>
<p>Our socket-based servers are cross-plaform, and compile and run on both Windows and POSIX (Linux, BSD, MacOS). They use a thread pool for HTTP/1.0 short living requests, and one thread per connection on HTTP/1.1. So they are meant to be used behind a reverse proxy like nginx, which could transmit over HTTP/1.0 with <em>mORMot</em>, but keep efficient HTTP/1.1 or HTTP/2.0 to communicate with the clients.</p>
<p>Both <a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.async.pas"><em>mORMot</em>.net.async.pas</a> and <a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.ws.async.pas"><em>mORMot</em>.net.ws.async.pas</a> are new to <em>mORMot</em> 2. They use an event-driven model, i.e. the opened connections are tracked using a fast API (like epoll on Linux), and the thread pool is used only when there is actually new data pending.</p>
<h4>Events Forever</h4>
<p>Asynchronous socket access, and event loops are the key for best server scalability. In respect to our regular <code>THttpServerSocket</code> class which uses one thread per HTTP/1.1 or WebSockets connection, our asynchronous classes (e.g. <code>THttpAsyncServer</code>) can have thousands of concurrent clients, with minimal CPU and RAM resource consumption.</p>
<p>Here is a typical event-driven socket access:</p>
<p><img src="https://blog.synopse.info?post/public/blog/epoll.png" alt="" /></p>
<p>In <em>mORMot</em> 2 network core, i.e. in unit <a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.sock.pas"><em>mORMot</em>.net.sock.pas</a>, we define an abstract event-driven class:</p>
<pre>
/// implements efficient polling of multiple sockets
// - will maintain a pool of TPollSocketAbstract instances, to monitor
// incoming data or outgoing availability for a set of active connections
// - call Subscribe/Unsubscribe to setup the monitored sockets
// - call GetOne from a main thread, optionally GetOnePending from sub-threads
TPollSockets = class(TPollAbstract)
...
/// initialize the sockets polling
constructor Create(aPollClass: TPollSocketClass = nil);
/// finalize the sockets polling, and release all used memory
destructor Destroy; override;
/// track modifications on one specified TSocket and tag
function Subscribe(socket: TNetSocket; events: TPollSocketEvents;
tag: TPollSocketTag): boolean; override;
/// stop status modifications tracking on one specified TSocket and tag
procedure Unsubscribe(socket: TNetSocket; tag: TPollSocketTag); virtual;
/// retrieve the next pending notification, or let the poll wait for new
function GetOne(timeoutMS: integer; const call: RawUtf8;
out notif: TPollSocketResult): boolean; virtual;
/// retrieve the next pending notification
function GetOnePending(out notif: TPollSocketResult; const call: RawUtf8): boolean;
/// let the poll check for pending events and apend them to fPending results
function PollForPendingEvents(timeoutMS: integer): integer; virtual;
/// manually append one event to the pending nodifications
procedure AddOnePending(aTag: TPollSocketTag; aEvents: TPollSocketEvents;
aNoSearch: boolean);
/// notify any GetOne waiting method to stop its polling loop
procedure Terminate; override;
...
</pre>
<p>Depending on the operating system, it will mimic the <a href="https://man7.org/linux/man-pages/man7/epoll.7.html">epoll api</a> with the underlying low-level system calls.</p>
<p>The events are very abstract, and are in fact just the basic R/W operations on each connection, associated with a "tag", which is likely to be a class pointer associated with a socket/connection:</p>
<pre>
TPollSocketEvent = (
pseRead,
pseWrite,
pseError,
pseClosed);
TPollSocketEvents = set of TPollSocketEvent;
TPollSocketResult = record
tag: TPollSocketTag;
events: TPollSocketEvents;
end;
TPollSocketResults = record
Events: array of TPollSocketResult;
Count: PtrInt;
end;
</pre>
<p>Then whole new HTTP/1.0, HTTP/1.1 and WebSockets stacks have been written on top of those basic socket-driven events. Instead of blocking threads, they use internal state machines, which are much lighter than a thread, and even lighter than a coroutine/goroutine. Each connection is just a class instance, which maintains the state of each client/server communication, and accesses its own socket.<br />
The <em>mORMot</em> asynchronous TCP server has by default one thread to accept the connection, one thread to poll for pending events (calling the <code>GetOne</code> method), then a dedicated number of threads to consume the Read/Write/Close events (via the <code>GetOnePending</code> method). We used as many non-blocking structures as possible, we minimized memory allocation by reusing the same buffers e.g. for the headers or small responses, we pickup the <a href="https://blog.synopse.info?post/2022/05/21/New-Async-HTTP/post/2022/01/22/Three-Locks-To-Rule-Them-All">best locks possible</a> in each case, so that this server could scales nice and smoothly. And simple to use, because handling a new protocol is as easy as inheriting and writing a new connection class.</p>
<p>Our asynchronous servers classes seem now stable, and fast (reported to be twice faster than nginx and six time faster than nodejs!).<br />
But of course, as any new complex code, they may be caveats. And some of them have already be identified and fixed - as <a href="https://synopse.info/forum/viewtopic.php?pid=36546#p36546">reported in our forum</a>. Therefore, feedback is welcome, and a nginx, haproxy or caddy reverse proxy frontend is always a good idea on production.</p>
<p>Writing those servers took more time than previewed, and was sometimes painful. Because debugging multi-thread process is not easy, and especially on several operating systems. There are some subtle differences between the OS, which could lead to unexpected blocking or degraded performance. But we are proud of the result, which compares to the best-in-class servers. Still in modern pascal code, and Open Source software.</p>
<p>Don't hesitate to take a look at the source, and try some samples.<br />
Feedback is <a href="https://synopse.info/forum/viewtopic.php?id=6253">welcome in our forum</a>, as usual.</p>EKON 25 Slidesurn:md5:aed86aeed11190901cf050e9842ec0bc2021-11-16T12:34:00+00:002021-11-16T12:34:00+00:00Arnaud BouchezmORMot Framework64bitAESAES-CTRAES-GCMAES-NiauthenticationCertificatesCrossPlatformDDDDelphiECCECDHECIESECSDAed25519EKONFreePascalinterfacelibdeflatemORMotmORMot2multithreadOpenSSLperformancerandomSOASourceWebSockets<p><a href="https://entwickler-konferenz.de/">EKON 25 at Düsseldorf</a> was a great conference (konference?).</p>
<p>At last, a <strong>physical</strong> gathering of Delphi developers, mostly from Germany, but also from Europe - and even some from USA! No more virtual meetings, which may trigger the well known 'Abstract Error' on modern pascal coders.<br />
There were some happy FPC users too - as I am now. <img src="https://blog.synopse.info?pf=smile.svg" alt=":)" class="smiley" /></p>
<p><img src="https://blog.synopse.info?post/public/blog/Ekon25.png" alt="" /></p>
<p>I have published the slides of my conferences, mostly about mORMot 2.<br />
By the way, I wish we would be able to release officially mORMot 2 in December, before Christmas. I think it starts to be stabilized and already known to be used on production. We expect no more breaking change in the next weeks.</p> <p>Here are the slides of my two 1-hour sessions.</p>
<h5>mORMot Cryptography</h5>
<p>The OpenSource mORMot framework has a strong set of cryptography features. It offers symmetric cryptography with hashing and encryption, together with asymmetric cryptography via private/public key pairs. Its optimized pascal and assembly engines can be embedded into your executable, but you could also call an external OpenSSL library if needed. This session will present mormot.crypt.* units, and apply them to some use cases, from low-level algorithms to high-level JWT or file encryption and signing.</p>
<p><a href="https://www.slideshare.net/ArnaudBouchez1/ekon25-mormot-2-cryptography">mORMot 2 Cryptography on SlideShare</a></p>
<p>I just had an interesting discussion with Michael on <a href="https://gitlab.com/freepascal.org/fpc/source/-/commit/3229cb712e33374b85258aed43726058be633bed#note_734398698">FPC new gitlab platform</a>: the FPC RTL is gaining some official cryptography functions, and I proposed to use mORMot code base as reference, and to introduce some RTL wrapper functions which can redirect to a plain pascal FPC RTL version, or use another engines, like OpenSSL or mORMot, if available.</p>
<h5>Server-Side REST Notifications with mORMot</h5>
<p>The most powerful way of writing REST services is to define them via interfaces, then let the SOA/REST framework do all the routing, data marshalling and communication behind the scenes. One distinctive feature of mORMot is to define a method parameter as a notification interface, and let the server call back the client when needed, as with regular Delphi code. This session will present the benefit of defining REST services using interfaces, and how WebSockets can offer real-time notifications into your rich Delphi client applications.</p>
<p><a href="https://www.slideshare.net/ArnaudBouchez1/ekon25-mormot-2-serverside-notifications">mORMot 2 Server-Side Notifications on SlideShare</a></p>
<p>Feedback is <a href="https://synopse.info/forum/viewtopic.php?id=6051">welcome on our forum, as usual.</a></p>mORMot 2 on Ampere AARM64 CPUurn:md5:01baca710d9e6371f285e77a90accdcd2021-08-17T13:16:00+01:002021-08-17T16:46:59+01:00Arnaud BouchezmORMot Framework64bitaarch64AESAES-GCMAES-NiampereAndroidasmavxblogCcompressioncrccrc32cFPCFreePascalLazarusLinuxMicroservicesmORMotmORMot2multithreadoraclecloudperformanceRESTSOASQLite3<p>Last weeks, we have enhanced mORMot support to one of the more powerful AARM64 CPU available: the <a href="https://amperecomputing.com/">Ampere Altra CPU</a>, as made available on the <a href="https://www.oracle.com/cloud/compute/arm/">Oracle Cloud Infrastructure</a>.</p>
<p><img src="https://blog.synopse.info/public/blog/AmpereCPU.jpg" alt="" /></p>
<p>Long story short, this is an amazing hardware to run on server side, with performance close to what Intel/AMD offers, but with <a href="https://www.oracle.com/cloud/compute/arm/why-arm-processors/">almost linear multi-core scalability</a>. The FPC compiler is able to run good code on it, and our mORMot 2 library is able to use the hardware accelerated opcodes for AES, SHA2, and crc32/crc32c.</p> <h3>Always Free Ampere VM</h3>
<p>Back to the beginning. Tom, one mORMot user, reported on our forum that he successfully <a href="https://synopse.info/forum/viewtopic.php?id=5945">installed FPC and Lazarus on the Oracle Cloud platform</a>, and accessed it via SSH/XRDP:</p>
<blockquote><p>Just open account on Oracle Cloud and create new compute VM: 4 ARMv8.2 CPU 3GHz, 24GB Ram (yes 24GB).
This is always free VM (you can combine this 4 cores and 24GB Ram to 1 or many (4) VM).
Install Ubuntu 20.04 server, then install LXDE and XRdp for remote access.
Now I have nice speed workstation. Install fpcupdeluxe then fpc 3.2.2/laz 2.0.12, all OK. fpcup build is faster then my local pc build <img src="https://blog.synopse.info?pf=smile.svg" alt=":-)" class="smiley" />
This OCI VM can be great mormot application server for some projects. I don't have any connection to Oracle - just test their product.</p></blockquote>
<p>I did the same, and in fact, this platform is really easy to work with, once you have paid a 1€ credit card fee to validate your account. Then you will get an "Always Free VM", with 4 Ampere cores, and 24GB of Ram. Amazing. The Oracle people really like to break into the cloud market, and they make it wide open for developers, so that they consider their Cloud instead of Microsoft's or Amazon's.</p>
<h3>FPC and Lazarus on Linux/AArch64</h3>
<p>The Lazarus experiment is very good on this platform, even remotely. The only issue is the debugger. Gdb was pretty unstable for me - almost as unstable as on Windows. But somewhat usable, until it crashes. <img src="https://blog.synopse.info?pf=sad.svg" alt=":(" class="smiley" /></p>
<p>Finally, and thanks to Alfred - our great friend behind <a href="https://github.com/LongDirtyAnimAlf/fpcupdeluxe">fpcupdeluxe</a> - we identified a problem with our asm stubs when calling mORMot interface-based services. It was in fact a FPC "feature" (it is documented as such in the compiler so it is not a bug), in how arguments are passed as result in the AARCH64 calling ABI. Once identified, we made an explicit exception to help circumvent the problem.</p>
<p>The FPC code quality seems good. At least at the level of the x86_64 Intel/AMD platform. Not as good as gcc for sure, but good enough for production code, and good speed. The only big limitation is that the inlined assembly is very limited: only a few AARCH64 opcodes are available - only what was mandatory for the basic FPC RTL needs.</p>
<h3>Tuning mORMot 2 for AArch64</h3>
<p>To enhance performance, we replaced the basic FPC RTL Move/FillChar functions by the libc memmov/memset. And performance is amazing:</p>
<pre>
FPC RTL
FillChar in 19.06ms, 20.3 GB/s
Move in 9.95ms, 1.5 GB/s
small Move in 15.85ms, 1.3 GB/s
big Move in 222.41ms, 1.7 GB/s
mORMot functions calling the Ubuntu Gnu libc:
FillCharFast in 8.84ms, 43.9 GB/s
MoveFast in 1.25ms, 12.4 GB/s
small MoveFast in 4.98ms, 4.3 GB/s
big MoveFast in 34.32ms, 11.3 GB/s
</pre>
<p>In comparison, here are the numbers on my Core i5 7200U CPU, of mORMot tuned x86_64 asm (faster than the FPC RTL), using SSE2 or AVX instructions:</p>
<pre>
FillCharFast [] in 21.43ms, 18.1 GB/s
MoveFast [] in 2.29ms, 6.8 GB/s
small MoveFast [] in 4.29ms, 5.1 GB/s
big MoveFast [] in 68.33ms, 5.7 GB/s
FillCharFast [cpuAVX] in 20.28ms, 19.1 GB/s
MoveFast [cpuAVX] in 2.26ms, 6.8 GB/s
small MoveFast [cpuAVX] in 4.25ms, 5.1 GB/s
big MoveFast [cpuAVX] in 69.93ms, 5.5 GB/s
</pre>
<p>So we can see that the Ampere CPU memory design is pretty efficient. It is up to twice faster than a Core i5 7200U CPU.</p>
<p>We had to go further, and get some fun with one bottleneck of every server operation: encryption and hashes. So we wrote some C code to be able to use the efficient HW acceleration we wanted for encryption and hashes. You could find the source code in <a href="https://github.com/synopse/mORMot2/tree/master/res/static/armv8">the /res/static/armv8 sub-folder of our repository</a>. Now we have tremendous performance for AES, GCM, SHA2 and CRC32/CRC32C computation.</p>
<pre>
2500 crc32c in 259us i.e. 9.2M/s or 20 GB/s
2500 xxhash32 in 1.47ms i.e. 1.6M/s or 3.5 GB/s
2500 crc32 in 259us i.e. 9.2M/s or 20 GB/s
2500 adler32 in 469us i.e. 5M/s or 11 GB/s
2500 hash32 in 584us i.e. 4M/s or 8.9 GB/s
2500 md5 in 12.12ms i.e. 201.3K/s or 438.7 MB/s
2500 sha1 in 21.75ms i.e. 112.2K/s or 244.5 MB/s
2500 hmacsha1 in 23.81ms i.e. 102.5K/s or 223.4 MB/s
2500 sha256 in 3.41ms i.e. 714.7K/s or 1.5 GB/s
2500 hmacsha256 in 4.12ms i.e. 591.2K/s or 1.2 GB/s
2500 sha384 in 27.71ms i.e. 88K/s or 191.9 MB/s
2500 hmacsha384 in 32.69ms i.e. 74.6K/s or 162.7 MB/s
2500 sha512 in 27.73ms i.e. 88K/s or 191.8 MB/s
2500 hmacsha512 in 32.77ms i.e. 74.4K/s or 162.3 MB/s
2500 sha3_256 in 35.82ms i.e. 68.1K/s or 148.5 MB/s
2500 sha3_512 in 65.48ms i.e. 37.2K/s or 81.2 MB/s
2500 rc4 in 12.98ms i.e. 188K/s or 409.8 MB/s
2500 mormot aes-128-cfb in 8.84ms i.e. 276.1K/s or 601.7 MB/s
2500 mormot aes-128-ofb in 3.78ms i.e. 645K/s or 1.3 GB/s
2500 mormot aes-128-c64 in 4.39ms i.e. 555.6K/s or 1.1 GB/s
2500 mormot aes-128-ctr in 4.52ms i.e. 539.7K/s or 1.1 GB/s
2500 mormot aes-128-cfc in 9.16ms i.e. 266.3K/s or 580.4 MB/s
2500 mormot aes-128-ofc in 5.25ms i.e. 465K/s or 0.9 GB/s
2500 mormot aes-128-ctc in 5.74ms i.e. 425.1K/s or 0.9 GB/s
2500 mormot aes-128-gcm in 7.52ms i.e. 324.5K/s or 707.2 MB/s
2500 mormot aes-256-cfb in 9.52ms i.e. 256.2K/s or 558.4 MB/s
2500 mormot aes-256-ofb in 4.71ms i.e. 517.6K/s or 1.1 GB/s
2500 mormot aes-256-c64 in 5.30ms i.e. 460.5K/s or 0.9 GB/s
2500 mormot aes-256-ctr in 5.33ms i.e. 457.5K/s or 0.9 GB/s
2500 mormot aes-256-cfc in 10.04ms i.e. 243K/s or 529.5 MB/s
2500 mormot aes-256-ofc in 6.11ms i.e. 399K/s or 869.6 MB/s
2500 mormot aes-256-ctc in 6.77ms i.e. 360.4K/s or 785.5 MB/s
2500 mormot aes-256-gcm in 8.38ms i.e. 291.1K/s or 634.4 MB/s
2500 openssl aes-128-cfb in 4.94ms i.e. 493.4K/s or 1 GB/s
2500 openssl aes-128-ofb in 4.12ms i.e. 591.2K/s or 1.2 GB/s
2500 openssl aes-128-ctr in 1.94ms i.e. 1.2M/s or 2.6 GB/s
2500 openssl aes-128-gcm in 3.18ms i.e. 767K/s or 1.6 GB/s
2500 openssl aes-256-cfb in 5.83ms i.e. 418.5K/s or 912.1 MB/s
2500 openssl aes-256-ofb in 5.04ms i.e. 484.1K/s or 1 GB/s
2500 openssl aes-256-ctr in 2.42ms i.e. 0.9M/s or 2.1 GB/s
2500 openssl aes-256-gcm in 3.66ms i.e. 667K/s or 1.4 GB/s
2500 shake128 in 29.63ms i.e. 82.3K/s or 179.5 MB/s
2500 shake256 in 35.07ms i.e. 69.5K/s or 151.6 MB/s
</pre>
<p>Here are the numbers on my Core i5 7200U CPU, with optimized asm, and the last OpenSSL calls:</p>
<pre>
2500 crc32c in 224us i.e. 10.6M/s or 23.1 GB/s
2500 xxhash32 in 817us i.e. 2.9M/s or 6.3 GB/s
2500 crc32 in 341us i.e. 6.9M/s or 15.2 GB/s
2500 adler32 in 241us i.e. 9.8M/s or 21.5 GB/s
2500 hash32 in 441us i.e. 5.4M/s or 11.7 GB/s
2500 aesnihash in 218us i.e. 10.9M/s or 23.8 GB/s
2500 md5 in 8.29ms i.e. 294.1K/s or 641.1 MB/s
2500 sha1 in 13.72ms i.e. 177.8K/s or 387.5 MB/s
2500 hmacsha1 in 15.05ms i.e. 162.1K/s or 353.3 MB/s
2500 sha256 in 17.40ms i.e. 140.2K/s or 305.6 MB/s
2500 hmacsha256 in 18.71ms i.e. 130.4K/s or 284.2 MB/s
2500 sha384 in 11.59ms i.e. 210.5K/s or 458.9 MB/s
2500 hmacsha384 in 13.84ms i.e. 176.3K/s or 384.2 MB/s
2500 sha512 in 11.59ms i.e. 210.5K/s or 458.8 MB/s
2500 hmacsha512 in 13.89ms i.e. 175.7K/s or 382.9 MB/s
2500 sha3_256 in 26.66ms i.e. 91.5K/s or 199.5 MB/s
2500 sha3_512 in 47.96ms i.e. 50.9K/s or 110.9 MB/s
2500 rc4 in 14.05ms i.e. 173.7K/s or 378.6 MB/s
2500 mormot aes-128-cfb in 4.59ms i.e. 530.9K/s or 1.1 GB/s
2500 mormot aes-128-ofb in 4.52ms i.e. 539.4K/s or 1.1 GB/s
2500 mormot aes-128-c64 in 6.23ms i.e. 391.7K/s or 853.7 MB/s
2500 mormot aes-128-ctr in 1.40ms i.e. 1.6M/s or 3.6 GB/s
2500 mormot aes-128-cfc in 4.75ms i.e. 513.2K/s or 1 GB/s
2500 mormot aes-128-ofc in 5.22ms i.e. 467.7K/s or 0.9 GB/s
2500 mormot aes-128-ctc in 1.72ms i.e. 1.3M/s or 3 GB/s
2500 mormot aes-128-gcm in 2.28ms i.e. 1M/s or 2.2 GB/s
2500 mormot aes-256-cfb in 6.12ms i.e. 398.4K/s or 868.3 MB/s
2500 mormot aes-256-ofb in 6.10ms i.e. 400K/s or 871.7 MB/s
2500 mormot aes-256-c64 in 7.86ms i.e. 310.6K/s or 676.9 MB/s
2500 mormot aes-256-ctr in 1.82ms i.e. 1.3M/s or 2.8 GB/s
2500 mormot aes-256-cfc in 6.36ms i.e. 383.5K/s or 835.9 MB/s
2500 mormot aes-256-ofc in 6.77ms i.e. 360.1K/s or 784.8 MB/s
2500 mormot aes-256-ctc in 2.02ms i.e. 1.1M/s or 2.5 GB/s
2500 mormot aes-256-gcm in 2.68ms i.e. 909.2K/s or 1.9 GB/s
2500 openssl aes-128-cfb in 7.11ms i.e. 342.9K/s or 747.3 MB/s
2500 openssl aes-128-ofb in 5.21ms i.e. 468K/s or 1 GB/s
2500 openssl aes-128-ctr in 1.54ms i.e. 1.5M/s or 3.3 GB/s
2500 openssl aes-128-gcm in 1.85ms i.e. 1.2M/s or 2.8 GB/s
2500 openssl aes-256-cfb in 8.65ms i.e. 282.2K/s or 615 MB/s
2500 openssl aes-256-ofb in 6.82ms i.e. 357.6K/s or 779.3 MB/s
2500 openssl aes-256-ctr in 1.93ms i.e. 1.2M/s or 2.6 GB/s
2500 openssl aes-256-gcm in 2.27ms i.e. 1M/s or 2.2 GB/s
2500 shake128 in 23.47ms i.e. 104K/s or 226.6 MB/s
2500 shake256 in 29.64ms i.e. 82.3K/s or 179.5 MB/s
</pre>
<p>The mORMot plain pascal code is used for MD5, SHA1, or shake/SHA3. So it is slower than our optimized asm for Intel/AMD. But not so slow. And those algorithms are either deprecated or not widely used - therefore they are not a bottleneck. OpenSSL numbers are pretty good too on this platform. As a result, AES, GCM, SHA-2 and crc32/crc32c performance is comparable between AARCH64 and Intel/AMD. With amazing SHA-2 numbers.</p>
<p>Then, we compiled the latest SQLite3, Lizard and libdeflate as static libraries, so that you could use them with your executable with no external dependency. Performance is very good:</p>
<pre>
TAlgoSynLZ 3.8 MB->2 MB: comp 287:151MB/s decomp 215:409MB/s
TAlgoLizard 3.8 MB->1.9 MB: comp 18:9MB/s decomp 857:1667MB/s
TAlgoLizardFast 3.8 MB->2.3 MB: comp 193:116MB/s decomp 1282:2135MB/s
TAlgoLizardHuffman 3.8 MB->1.8 MB: comp 84:40MB/s decomp 394:827MB/s
TAlgoDeflate 3.8 MB->1.5 MB: comp 30:12MB/s decomp 78:196MB/s
TAlgoDeflateFast 3.8 MB->1.6 MB: comp 48:20MB/s decomp 73:174MB/s
</pre>
<p>I was a bit surprised by how well the pure pascal version of SynLZ algorithm was running, once compiled with FPC 3.2, on AARCH64. Also the Deflate compression has a small advantage of using our statically linked libdeflate in respect to the plain zlib. But the very good news is that Lizard is really fast on AARCH64: even if it is written in plain C with no manual SIMD/asm code, it is really fast on non Intel/AMD platforms. More than 2GB/s for decompression is very high. I was told that Lizard may be a bit behind ZStandard on Intel/AMD, but its code is simpler, and much more CPU agnostic.</p>
<pre>
2.4. Sqlite file memory map:
- Database direct access: 22,264 assertions passed 55.40ms
- Virtual table direct access: 12 assertions passed 347us
- TOrmTableJson: 144,083 assertions passed 60.25ms
- TRestClientDB: 608,196 assertions passed 783.02ms
- Regexp function: 6,015 assertions passed 11.07ms
- TRecordVersion: 20,060 assertions passed 51.28ms
Total failed: 0 / 800,630 - Sqlite file memory map PASSED 961.45ms
</pre>
<p>Here SQLite3 numbers are similar to what I have on Intel/AMD. So I guess we could really consider using this database as storage back-end for mORMot MicroServices with their stand-alone persistence layer.</p>
<h3>Ampere and Beyond - Apple M1?</h3>
<p>We also tried to support as much as possible the ARM/AARCH64 CPUs with mORMot 2. So now we detect the CPU type and HW platform it runs on, especially on Linux or Android - which is also an AARCH64 platform. Here is what our regression tests report at their ending:</p>
<pre>
Ubuntu 20.04.2 LTS - Linux 5.8.0-1037-oracle (cp utf8)
2 x ARM Neoverse-N1 (aarch64)
on QEMU KVM Virtual Machine virt-4.2
Using mORMot 2.0.1
TSqlite3LibraryStatic 3.36.0 with internal MM
Generated with: Free Pascal 3.2 64 bit Linux compiler
Time elapsed for all tests: 44.38s
Performed 2021-08-17 13:44:09 by ubuntu on lxde
Total assertions failed for all test suits: 0 / 66,050,607
</pre>
<p>As you can see, the CPU was properly identified as <a href="https://www.arm.com/products/silicon-ip-cpu/neoverse/neoverse-n1">ARM Neoverse-N1</a>.</p>
<p>We could consider with good faith using <em>mORMot</em> code on an Apple M1/M1X/M2 CPU, thanks to the FPC (cross-)compiler. If we have access to this hardware. Any feedback is welcome.</p>
<h3>Server Process Performance</h3>
<p>All regression tests do pass whole green, with pretty consistent performance among all its various tasks. JSON process, ORM, SOA or encryption: everything flies on the Ampere CPU. You can check <a href="https://gist.github.com/synopse/0e7275684a2e2bbd2206940c3827055c">the detailed regression tests console output</a>.</p>
<p>Here are some numbers about UTF-8 or JSON process:</p>
<pre>
StrLen() in 1.43ms, 13.3 GB/s
IsValidUtf8(RawUtf8) in 11.75ms, 1.6 GB/s
IsValidUtf8(PUtf8Char) in 13.08ms, 1.4 GB/s
IsValidJson(RawUtf8) in 22.84ms, 858.2 MB/s
IsValidJson(PUtf8Char) in 22.93ms, 854.7 MB/s
JsonArrayCount(P) in 22.97ms, 853.1 MB/s
JsonArrayCount(P,PMax) in 22.89ms, 856.4 MB/s
JsonObjectPropCount() in 11.90ms, 0.9 GB/s
jsonUnquotedPropNameCompact in 72.35ms, 240.6 MB/s
jsonHumanReadable in 119.06ms, 209.4 MB/s
TDocVariant in 245.99ms, 79.7 MB/s
TDocVariant no guess in 260.57ms, 75.2 MB/s
TDocVariant dvoInternNames in 247.56ms, 79.1 MB/s
TOrmTableJson GetJsonValues in 34.88ms, 247.1 MB/s
TOrmTableJson expanded in 42.70ms, 459 MB/s
TOrmTableJson not expanded in 21.42ms, 402.4 MB/s
DynArrayLoadJson in 87.96ms, 222.8 MB/s
TOrmPeopleObjArray in 131.10ms, 149.5 MB/s
fpjson in 115.09ms, 17 MB/s
</pre>
<p>It is nice to see that our pascal code, which has been deeply tuned to let FPC generate the best x86_64 assembly possible, is also able to give very good performance on AARCH64. No need to write some dedicated code, and pollute the source with plenty of <em>$ifdef/$endif</em>: x86_64 is already some kind of RISC-like architecture, with a bigger number of registers, and 64-bit efficient processing. No need to rewrite everything. Optimized pascal code, with tuned pointer arithmetic is platform neutral. I like the quote of SQLite3 author saying that <a href="https://www.sqlite.org/whyc.html">C is a "portable assembly"</a>, and that we could also use tuned pascal code, as we try to do in the mORMot core units, to leverage modern CPU hardware, without the need of fighting against any hype/versatile language.</p>
<h3>Asm is Fun Again</h3>
<p>So we are pretty excited to see how this platform will go in the future. mORMot has invested a lot of time, refactoring and asm tuning to leverage the Intel/AMD platform, focusing on the server side performance. But this AARCH64 technology is really promising, and I can tell you that its RISC instruction set was very cleverly designed. It is very rich and powerful, almost perfect in its balance between power and expressiveness, in respect to the x86_64 platform, which has a lot of inconsistencies and seems outdated when you compare both asm. After decades playing with i386 or x86_64 asm, I had fun again with the ARM v8 assembly. It tastes like "assembly as it should be" (tm). Linking some static C code is a good balance between leveraging the hardware when needed, and keeping platform-independent pascal source. And FPC, as a compiler, is amazing by being open and well done on so many CPUs and platforms. Open Source rocks!</p>
<p>As usual, <a href="https://synopse.info/forum/viewtopic.php?pid=35602#p35602">feedback is welcome on our forum</a>.</p>mORMot 2 Entering Testing Phaseurn:md5:27770631af02b791754acbee20b609682020-11-16T10:30:00+00:002021-02-22T08:39:22+00:00Arnaud BouchezmORMot FrameworkblogDelphiFreePascalmORMotmORMot2ORMRestSOATesting<p>After a lot of work, our <a href="https://blog.synopse.info/?tag/mORMot2">mORMot 2 fork</a> is entering its testing phase.</p>
<p>The main <code>/src/core /src/lib /src/net /src/db /src/orm /src/soa /src/app</code> folders of our Source Code repository have been implemented.</p>
<p><img src="https://blog.synopse.info?post/public/blog/mormot2test.jpg" alt="mormot2test.jpg, Nov 2020" /></p>
<p>Please check <a href="https://github.com/synopse/mORMot2">https://github.com/synopse/mORMot2</a> for the latest version of the source code. The <code>README.md</code> files on each folder would help you discover the new framework design, and the content of each unit.</p> <h4>mORMot 2 Current State</h4>
<p>Take a look <a href="https://blog.synopse.info/?post/2020/11/04/EKON-24-Presentation-Slides">at the mORMot 2 Slides</a> we published last weeks.</p>
<p>The main refactoring points tried to better follow SOLID principles:</p>
<ul>
<li>Switch to Semantic Versioning - see https://semver.org</li>
<li>Split main big units (<code>SynCommons.pas</code>, <code>mORMot.pas</code>) into smaller scope-refined units;</li>
<li>OS- or compiler- specific code separated to ease evolution;</li>
<li>Rename confusing types (e.g. <code>TSQLRecord</code> into <code>TOrm</code>, <code>TSQLRest</code> into <code>TRest</code>...);</li>
<li>Favor composition over inheritance (e.g. <code>TRest</code> class split into proper REST/ORM/SOA classes - and folders);</li>
<li>Rewrite some code to avoid internal errors on Delphi (e.g. untyped const/var changed into pointers);</li>
<li>Tools to leverage MicroServices coding and administration; - see <a href="https://synopse.info/forum/viewtopic.php?pid=33420#p33420">this forum post</a></li>
<li>Add MVC support for SOA rich clients on Delphi VCL and Lazarus LCL;</li>
<li>Introduce high-level non-visual TComponent version of our classes for a more RAD approach.</li>
</ul>
<p>Some mORMot 1.18 features won't be available in mORMot 2:</p>
<ul>
<li>Delphi 5-6 and Kylix compatibility was removed;</li>
<li>BigTable, LVCL, RTTI-UI are now deprecated.</li>
</ul>
<p>On purpose, those old and not used features won't be missed. If you really need them, stick to mORMot 1.18, or propose your own refactoring as mORMot 2 pull requests!</p>
<h4>The Test Phase</h4>
<p>Now, since we have the main code at hand, we are entering the testing/validation phase.</p>
<p>We will populate the <code>/test/</code> sub-folder of the repository with all existing regression tests from mORMot 1.18:</p>
<ul>
<li>In practice, the huge <code>SynSelfTests.pas</code> unit will be split into smaller units.</li>
<li>We will keep all the existing tests, which are already proven;</li>
<li>It will also ensure we have the expected backward compatibility from mORMot 1.18;</li>
<li>Some new tests will emerge, to enhance the test coverage.</li>
</ul>
<p>We kept the testing to the last phase of our mORMot 2 conversion. It helped focusing on the refactoring. We will rely on the huge set of regression tests from mORMot 1.18 to validate our work.</p>
<p>Most errors (there will be) are expected to be found in the new part of the code, i.e. JSON and RTTI. Those two have been deeply rewritten, optimized, and redesigned, so we will see how they behave with our new code base!</p>EKON 24 Presentation Slidesurn:md5:82fd8ba85a11870241692103f54e32962020-11-04T18:28:00+00:002021-02-22T08:39:41+00:00Arnaud BouchezmORMot FrameworkasmAVX2blogEKONmORMotmORMot2ORMperformanceSOASOLIDSSE2 <p>EKON 24 just finished.</p>
<p>"The conference for Delphi & more" was <strong>fully online</strong> this year, due to the viral context...</p>
<p><img src="https://blog.synopse.info?post/public/blog/EKON_24.png" alt="EKON_24.png, Nov 2020" /></p>
<p>But this was a great event, and I am very happy to have been part of it.</p>
<p>Please find the slides on my two sessions:</p>
<ul>
<li><a href="https://www.slideshare.net/ArnaudBouchez1/ekon24-mormot-2">mORMot 2</a></li>
<li><a href="https://www.slideshare.net/ArnaudBouchez1/ekon24-from-delphi-to-avx2">Performance: from Delphi to AVX2</a></li>
</ul>
<p>Of course, nothing replace meeting the nice attendees in person... and share some chat with a nice German beer!
We hope next year!</p>mORMot2 Renamingurn:md5:deb4de72482a42c02ec3fb956d243a612020-10-26T16:24:00+00:002021-02-22T08:39:57+00:00Arnaud BouchezmORMot FrameworkblogmORMotmORMot2ORMRESTRoadMapSOASOLIDTSQLRecordTSQLRest<p>Last weeks, we introduced REST, ORM and SOA process in <a href="https://github.com/synopse/mORMot2">the mORMot2 repository</a>.</p>
<p>During this phase, we split the huge <code>mORMot.pas</code> unit into several <code>mormot.rest.*.pas</code>, <code>mormot.orm.*.pas</code> and <code>mormot.soa.*.pas</code> units, to follow SOLID principles.</p>
<p><img src="https://blog.synopse.info?post/public/blog/mormotkiss.jpg" alt="" /></p>
<p>But we also renamed the base types into something more consistent and easier to work with.
Forget about <code>TSQLRecord</code> or <code>TSQLRest</code>, discover <code>TORM</code> and <code>TRest</code>!</p> <p>The idea is that instead of writing:</p>
<pre>type
TSQLRecordBlogArticle = class(TSQLRecord)
...
</pre>
<p>we write with mORMot 2:</p>
<pre>type
TOrmBlogArticle = class(TOrm)
...
</pre>
<p>I don't find this new code too confusing.
<code>TOrm</code> for the base class to inherit from an ORM persisted class doesn't seem ground braking.</p>
<p>Then we have <code>TRest</code> and <code>TRestORM</code> for the client or server process side (instead of a single bloated <code>TSQLRest</code> class), and <code>TRest*</code> associated types.</p>
<p>To access the ORM methods, you won't call <code>TSQLRest</code> any more, but a new <code>IRestORM</code> interface, which will be the main entry point in your end-user code.
It is available throught a new <code>TRest.ORM</code> property, and will induce a better segregation in your business code: the REST and ORM features will indeed be uncoupled, so the resulting framework code is more SOLID, therefore more maintainable.</p>
<p>About the <code>TSQLRecord</code> new naming convention into a plain and simple <code>TOrm</code>, <code>TOrmRecord</code> and <code>TOrmObject</code> were also proposed. But they won't add anything to the idea, and may be confusing because we define a class, not a record or an object type. Similarly <code>TOrmClass</code> is not an option, because usually <code>T*Class</code> defines a metaclass, like <code>TOrmClass = class of TOrm</code>.</p>
<p>We get rid of all <code>TSQL*</code> names, and keep SQL only for <code>mormot.db.sql.*.pas</code> units, which are actually SQL processing units.
Our ORM is not SQL-only, it was in the beginning, but since years it is NoSQL too, as it can use for instance MongoDB.</p>
<p>Of course, there are <code>TSQL*</code> types definitions available by default, so existing code will continue to work with almost no modification.
There is even a <code>PUREMORMOT2 conditional</code>, which gets rid of the backward compatible type definitions, and use only the new names.
Eventually this new naming convention will become the common ground to build your applications above.</p>
<p>Any <a href="https://synopse.info/forum/viewtopic.php?id=5611">feedback and ideas are welcome on our forum</a>!</p>EKON 23 Presentation Slides and Codeurn:md5:37c503e876bd99defc25a3456ef98bcd2019-10-30T14:26:00+01:002019-10-30T14:39:32+01:00AB4327-GANDImORMot FrameworkblogDocumentationDomainDrivenFreePascalGoodPracticeKDDKingdomDrivenDesignLinuxmORMotMVCORMSOA<p>I just finished my workshop at EKON 23.<br />
Like every year, it was a great event to attempt to, and I enjoyed presenting 2
sessions and 1 workshop.</p>
<p><a href="https://blog.synopse.info?post/public/mORMot/IamLost.png"><img src="https://blog.synopse.info?post/public/mORMot/.IamLost_s.jpg" alt="" title="I am lost, Mar 2013" /></a></p>
<p>Sessions were about "<em>Kingdom Driven Design</em>" (KDD), which is the
name I used to define a cut-down version of "Domain Driven Design" (DDD).<br />
Less paranoid, a bit less isolation, but perhaps more common sense for the less
sensitive projects.<br />
Some presentations and code are now available!</p> <p>"The better is the worst enemy of the good" - may sum-up the moto behind
KDD.<br />
Idea is to leverage the Object-Pascal language and <em>mORMot</em> framework
abilities, for (hopefully) faster time-to-market than DDD, with enough
isolation for most projects.</p>
<p>The workshop was mainly about practical use of the MVC Web part of
<em>mORMot</em>, i.e. the Mustache-template web server rendering system
available.<br />
Perhaps a least known part of <em>mORMot</em>, but very powerful and which may
be good for some kind of projects.</p>
<p>The presentations are available from <em>SlideShare</em>:</p>
<ul>
<li><a href="https://www.slideshare.net/ArnaudBouchez1/ekon23-1-kingdomdrivendesign">https://www.slideshare.net/ArnaudBouchez1/ekon23-1-kingdomdrivendesign</a></li>
<li><a href="https://www.slideshare.net/ArnaudBouchez1/ekon23-2-kingdomdrivendesign-applied-to-social-media-with-mormot">
https://www.slideshare.net/ArnaudBouchez1/ekon23-2-kingdomdrivendesign-applied-to-social-media-with-mormot</a></li>
<li><a href="https://www.slideshare.net/ArnaudBouchez1/ekon23-3-mvc-web-with-mormot-on-linux">
https://www.slideshare.net/ArnaudBouchez1/ekon23-3-mvc-web-with-mormot-on-linux</a></li>
</ul>
<div>And I just started a new GPL Open Source project, using <em>mORMot</em>
and Kingdom-Driven-Design.<br />
You can check <a href="https://github.com/synopse/basilique">https://github.com/synopse/basilique</a><br />
This is a work-in-progress I will build on, for some side projects of mine,
mainly a Social-Media around texts and indexes I created some years ago for
<a href="http://ictuswin.com">IctusWin</a> and <a href="http://www.clerus.org/bibliaclerus/index_eng.html">BibliaClerus</a>.</div>Status of mORMot ORM SOA MVC with FPCurn:md5:713e7f794fdb05475b36cadd4cde47f02018-02-07T10:31:00+01:002020-07-03T09:29:59+02:00AB4327-GANDImORMot Framework64bitasmblogBSDcrc32cCrossPlatformDelphiECCexceptionFreePascalLazarusLinuxMaxOSXmORMotORMperformanceRestRTTISOASourceSQLite3sse42<p>In the last weeks/months, we worked a lot with FPC.<br />
Delphi is still our main IDE, due to its better debugging experience under
Windows, but we target to have premium support of FPC, on all platforms,
especially Linux.</p>
<p><img src="https://blog.synopse.info?post/public/ScreenShots/lazarusaboutbox.png" alt="" title="Lazarus FPC About, Feb 2018" /></p>
<p>The new Delphi Linux compiler is out of scope, since it is heavily priced,
its performance is not so good, and ARC broke memory management so would need a
deep review/rewrite of our source code, which we can't afford - since we have
FPC which is, <a href="https://synopse.info/forum/viewtopic.php?pid=25984#p25984">from our
opinion</a>, a much better compiler for Linux.<br />
Of course, you can create clients for Delphi Linux and FMX, as usual, using
the <a href="https://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_86">cross-platform
client parts of mORMot</a>. But for server side, this compiler is not
supported, and will probably never be.</p> <p>First of all, since FPC - and Lazarus, its sibling IDE - are Open Source and
free, we can focus on mainly support a single version of the compiler.<br />
Since some missing RTTI for interfaces were recently merged into the trunk, we
start from the current FPC trunk as our main version. Easier than maintaining
Delphi 5 - 10.2 compability!</p>
<p>To install it, we usually use the <a href="https://github.com/newpascal/fpcupdeluxe">fpcupdeluxe tool</a>: you <a href="https://github.com/newpascal/fpcupdeluxe/releases">download a single binary
for your platform</a>, then you run the executable, pickup the compiler (or
cross-compiler) versions you need, and everything is downloaded and compiled
from git/svn on your own computer. Then click on the desktop link, and the IDE
launches in seconds. Nice fresh air in respect to Delphi setup experience!</p>
<p style="margin-top: 0;">As I wrote, in the last weeks/months, we worked a lot
to improve FPC support.</p>
<p style="margin-top: 0;">Just a few commits:</p>
<ul>
<li><a href="https://github.com/synopse/mORMot/commit/7bf5951e322f0164d8e3852f98f9f48b1bd7a6d2">
crc32c 2x/4x speeup by using SSE4.2+pclmulqdq opcodes on x64</a> - speed is now
around 21GB/s</li>
<li><a href="https://github.com/synopse/mORMot/commit/5b569db11e47fd65e2e0f792e1383895e50352b6">
TSynDaemon fork/run support on Linux/Posix</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/4e019e1b41f6759487e7479c194fcb28c25314c9">
fixed vtQWord proper support for FPC</a> - this doesn't exist in Delphi,
but it should!</li>
<li><a href="https://github.com/synopse/mORMot/commit/bc405cffb7bfb79646b46a6afc13968a0d804f32">
added pure pascal version of SynECC.pas</a> to run on all FPC platforms,
including ARM</li>
<li>BSD/OSX <a href="https://github.com/synopse/mORMot/commit/af8b33720d0da12a8476cd5bc34eeeab417cb096">
enhanced</a> <a href="https://github.com/synopse/mORMot/commit/16332915f87f5dcc7de881c9b72b09d05b609eba">
support</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/a5352309f41d8ef70561d819e57732ac0aaf43e3">
added scripts to use fpcupdeluxe's gcc cross-compilers for FPC static linking
on Windows, Linux, BSD and OSX</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/eaddf84b04b7eaf647adb598be140276b9b42046">
updated SQLite3 engine to latest version 3.22.0</a> - statically linked under
FPC Linux (no external dependency to the system libsqlite3.so)</li>
<li><a href="https://github.com/synopse/mORMot/commit/7c6d09df6caf3d8865e488d9147b5340c1a15fca">
HTTP server enhancements and fixes for high performance and stability under
Linux behind a local nginx proxy for production servers</a></li>
<li style="list-style: none"><a href="https://github.com/synopse/mORMot/commit/f40a379cbd3fb0e5076ac2cf664acb8b03ab86e2">
</a></li>
<li>better Linux compatibility</li>
<li><a href="https://github.com/synopse/mORMot/commit/ec4755a19722e744029244e7a2d1bfa985887f3e">
TSQLRecord QWord property fix</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/4b7786c3be0d1aec5abaad26f9de90e58edcce16">
tuned/enhanced logging content</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/d9fe9a5bf9305a0eb05c9f5a80d97a1b49f3bcb8">
deep refactoring of FPC RTTI access to have the same level than Delphi</a></li>
<li><a href="https://github.com/synopse/mORMot/commit/95c5d56edbcb641f716c36d78792853eae67c689">
implemented Exceptions interception and logging for FPC</a><br />
with call stack trace (if available) - includes source code lines if compiled
using -g or -gl switches - tested on Win32, Win64, Linux i386 et x86_64 but
should work on other OS <img src="https://blog.synopse.info?pf=smile.svg" alt=":)" class="smiley" /></li>
<li>SyNode JavaScript engine support under FPC / Linux x86_64 by <a href="https://github.com/synopse/mORMot/commits?author=ssoftpro">ssoftpro</a> and
<a href="https://github.com/synopse/mORMot/commits?author=pavelmash">pavelmash</a> -
including a lot of fixes and tuning for this platform <a href="https://synopse.info/forum/viewtopic.php?pid=25985#p25985">to be heavily used
on production</a></li>
<li>and a lot of smaller enhancements (just search for FPC <a href="https://synopse.info/fossil/timeline?n=500&y=ci&t=&ms=exact">in
the commit timeline</a>), especially <a href="https://github.com/synopse/mORMot/commit/c5ad9b1d1f57177a8fe686271370cb12fc29d3d9">
tuning</a> the pascal code to better compile and execute under FPC, which can
generate very efficient assembly!</li>
</ul>
<div>As you can see, exciting times!</div>
<div>To be honest, the more we work with FPC as a compiler, the more we like
it.<br />
Staty tuned, and we encourage you to discover FPC/Lazarus!</div>EKON 21 Slidesurn:md5:5212ca57d269c94d396b78e3ca136f232017-10-24T10:19:00+02:002017-10-28T14:50:43+02:00AB4327-GANDImORMot FrameworkblogDDDDelphiEKONMicroservicesmORMotSOASOLID <p>After having <a href="https://entwickler-konferenz.de/speaker/arnaud-bouchez/">enjoyed EKON 21
conferences</a> in Köln, some quick post to share material about my
presentations.</p>
<p><img src="https://entwickler-konferenz.de/wp-content/uploads/2017/03/EKON21_Logo_Web_140x40_40146_v1.jpg" alt="" /></p>
<ul>
<li><a href="https://synopse.info/files/ekon21/EKON21_1_MicroServices-SOLIDMeetsSOA.pdf">MicroServices:
SOLID Meets SOA</a></li>
<li><a href="https://synopse.info/files/ekon21/EKON21_2_MicroServices-EventDrivenSystems.pdf">
MicroServices: Event-Driven Design</a></li>
<li><a href="https://synopse.info/files/ekon21/EKON21_3_PracticalDDD.pdf">Practical
Domain-Driven Design</a></li>
</ul>
<p>I also included the "Practical DDD" source code in a <a href="https://synopse.info/fossil/dir?ci=tip&name=SQLite3/Samples/35+-+Practical+DDD">
new sample folder of the mORMot repository</a>.</p>
<p>This sample was long-awaited from new <em>mORMot</em> users, and anyone
willing to see how DDD may be implemented in Delphi.<br />
I choose to store the sample in enumerated sub-folders (01, 02, 03...), to show
the typical steps of building such kind of services.</p>
<p>Of course, it follows Test-Driven Design, so we write the test as soon as we
add a new feature.<br />
Stubing/mocking is also used for the Domain regression tests.<br />
And we will eventually add new sub-folders, to include new features and
services.</p>
<p>About EKON, sad I can't share some fresh beer as pdf or in a git/fossil
repository...</p>EKON20 mORMot Conferencesurn:md5:0527c67a4abc7b21c3b79121626a828b2016-11-10T19:36:00+01:002016-11-10T19:50:52+01:00AB4327-GANDImORMot FrameworkblogDDDDelphiEKONmORMotMVCORMRESTSOASOLID <p><a href="https://entwickler-konferenz.de/">EKON20 is now over</a>, and there
was a lot of people, great speakers, beautiful T-Shirt, and fresh beer!</p>
<p><img src="https://entwickler-konferenz.de/wp-content/uploads/2016/02/EKON20_Logo_270pxBreite.png" alt="" /></p>
<p>I've published the slides of my <em>mORMot</em> conferences on
SlideShare...</p>
<ul>
<li><a href="http://www.slideshare.net/ArnaudBouchez1/ekon20-mormot-soa-delphi-conference">EKON20
From RAD to SOA with mORMot</a></li>
<li><a href="http://www.slideshare.net/ArnaudBouchez1/ekon20-mormot-legacy-code-technical-debt-delphi-conference">
EKON20 mORMot Legacy Code Technical Debt Delphi Conference</a></li>
<li><a href="http://www.slideshare.net/ArnaudBouchez1/ekon20-mormot-workshop-delphi">EKON20
Ride a mORMot</a></li>
<li><a href="http://www.slideshare.net/ArnaudBouchez1/2016-mormot">EKON20 2016
mORMot</a></li>
</ul>
<p>The "classic" Synopse/mORMot slides have also been uploaded to their latest
revision, so don't hesitate to check what's new!</p>
<ul>
<li><a href="http://www.slideshare.net/ArnaudBouchez1/a1-from-n-tier-to-soa-48837834">A1
from N-Tier to S</a>OA</li>
<li><a href="http://www.slideshare.net/ArnaudBouchez1/a2-from-soap-to-rest-48837831">A2
from SOAP to REST</a></li>
<li><a href="http://www.slideshare.net/ArnaudBouchez1/a3-from-sql-to-orm">A3
from SQL to ORM</a></li>
<li><a href="http://www.slideshare.net/ArnaudBouchez1/a4-from-rad-to-mvc">A4
from RAD to MVC</a></li>
<li><a href="http://www.slideshare.net/ArnaudBouchez1/d1-from-interfaces-to-solid-48837833">
D1 from interfaces to SOLID</a></li>
<li><a href="http://www.slideshare.net/ArnaudBouchez1/d2-domain-drivendesign-48837830">D2
Domain-Driven-Design</a></li>
</ul>
<p>The <a href="http://synopse.info/files/slides/">PDF are also available for
direct download</a> from our server.</p>
<div>Enjoy!</div>Ride the mORMot at EKON 20 in Dusseldorf!urn:md5:7942473e53d9fe651dd6ff3bfcc1161b2016-09-06T07:35:00+02:002016-09-06T06:37:35+02:00AB4327-GANDImORMot FrameworkblogDelphiEKONFreePascalmORMotORMSOA <p>There are still a few days for <a href="http://entwickler-konferenz.de/">"very early birds" offer for EKON 20
conference</a>, and meet us for 3 sessions (including a half-day
training/introduction to <em>mORMot</em>)!<br />
Join us the 7-9th of November in Düsseldorf!</p>
<p><img src="https://blog.synopse.info?post/public/blowup-images/EKON20_Webbanner_Fullsize_34089_v2.png" alt="" title="EKON20, Sep 2016" /></p>
<p><a href="http://entwickler-konferenz.de/speaker/arnaud-bouchez/">Our
sessions</a> are not restricted to <em>mORMot</em>, but will use
<em>mORMot</em> to illustrate some concepts and design ideas:</p>
<ul>
<li><a href="http://entwickler-konferenz.de/session/reduce-our-technical-debt/">Reduce our
technical debt</a></li>
<li><a href="http://entwickler-konferenz.de/session/from-rad-to-soa-with-mormot/">From RAD
to SOA with mORMot</a></li>
<li><a href="http://entwickler-konferenz.de/session/ride-a-mormot/">Ride a
mORMot</a></li>
</ul>
<p>You can contact me if you want to join, so that I may give you an additional
discount password!</p>
<p>Hope we can meet for a chat and (a few) beers!</p>Audit Trail for Servicesurn:md5:988ba15b873c9c344638de4fbce379222015-12-11T21:31:00+01:002015-12-11T21:31:00+01:00AB4327-GANDImORMot FrameworkAuditTrailblogDatabaseDelphiDocumentationDomainDrivenGoodPracticeJSONMongoDBmORMotORMSOASQLite3<p>We have seen previously how the ORM part of the framework is able to provide
an <em><a href="https://blog.synopse.info?post/post/2014/06/22/Audit-trail-for-ORM-change-tracking">Audit
Trail for change tracking</a></em>.<br />
It is a very convenient way of storing the change of state of the data.</p>
<p>On the other side, in any modern SOA solution, data is not at the center any
more, but services.<br />
Sometimes, the data is not stored within your server, but in a third-party
<em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_17">
Service-Oriented Architecture (SOA)</a></em>.<br />
Being able to monitor the service execution of the whole system becomes sooner
or later mandatory.</p>
<p><img width="300" height="200" src="http://cdn.collider.com/wp-content/image-base/Clubhouse/B/Back_to_the_Future_Delorean/Back%20to%20the%20Future%20Delorean%20Time%20Machine%20(8).jpg" alt="" /></p>
<p>Our framework allows to create an <em>Audit Trail</em> of any incoming or
outgoing service operation, in a secure, efficient and automated way.</p> <h3>When logging is not enough</h3>
<p>By default, any <code>interface</code>-based service process would be logged
by the framework - see <em>Framework log integration</em> - in dedicated
<code>sllServiceCall</code> and <code>sllServiceReturn</code> log levels.<br />
You may see output similar to the following:</p>
<pre>
18:03:18 Enter mORMot.TSQLRestServerFullMemory(024500A0).URI(POST root/DomUserQuery.SelectByLogonName/1 inlen=7)
<span style="background-color:yellow;">18:03:18 Service call mORMot.TSQLRestServerFullMemory(024500A0) DomUserQuery.SelectByLogonName["979"]</span>
18:03:18 Server mORMot.TSQLRestServerFullMemory(024500A0) POST root/DomUserQuery.SelectByLogonName SOA-Interface -> 200 with outlen=21 in 16 us
<span style="background-color:yellow;">18:03:18 Service return mORMot.TSQLRestServerFullMemory(024500A0) {"result":[0],"id":1}</span>
18:03:18 Leave 00.000.017
</pre>
<p>The above lines match the execution of the following method, as defined in
<code>dddDomUserCQRS.pas</code>:</p>
<pre>
IDomUserQuery = <strong>interface</strong>(ICQRSService)
['{198C01D6-5189-4B74-AAF4-C322237D7D53}']
<em>/// would select a single TUser from its logon name</em>
<em>// - then use Get() method to retrieve its content</em>
<strong>function</strong> SelectByLogonName(<strong>const</strong> aLogonName: RawUTF8): TCQRSResult;
...
</pre>
<p>The actual execution was:</p>
<pre>
IDomUserQuery.SelectByLogonName('979') -> cqrsSuccess
</pre>
<p>Here <code>cqrsSuccess</code> is the first item of the enumeration result,
returned as an integer JSON value <code>"result":[0]</code> by the method:</p>
<pre>
TCQRSResult =
(cqrsSuccess, cqrsSuccessWithMoreData,
cqrsUnspecifiedError, cqrsBadRequest, cqrsNotFound,
...
</pre>
<p>This detailed log (including micro-second timing on the "<em>Leave</em>"
rows) is very helpful for support, especially to investigate about any error
occurring on a production server.<br />
But it would not be enough (or on the contrary provide "too much information"
which "kills the information") to monitor the higher level of the process,
especially on a server with a lot of concurrent activity.</p>
<h3>Tracing Service Methods</h3>
<p>The framework allows to optionally store each SOA method execution in a
database, with the input and output parameters, and accurate timing.<br />
You could enable this automated process:</p>
<ul>
<li>Either at service level, using
<code>TServiceFactoryServer.SetServiceLog()</code>;</li>
<li>Or for all services of a <code>TSQLRestServer.ServiceContainer</code>
instance, via <code>TServiceContainerServer.SetServiceLog()</code>.</li>
</ul>
<p>For instance, you may enable it for a whole REST server:</p>
<pre>
(aRestSOAServer.ServiceContainer <strong>as</strong> TServiceContainerServer).SetServiceLog(
aRestLogServer,TSQLRecordServiceLog);
</pre>
<p>This single command would create an Audit Trail with all service calls made
on <code>aRestSOAServer</code> to the <code>TSQLRecordServiceLog</code> ORM
class of <code>aRestLogServer</code>.<br />
Keeping a dedicated REST server for the log entries would reduce the overhead
on the main server, and ease its maintenance.</p>
<p>Actual storage takes place within a class inheriting from
<code>TSQLRecordServiceLog</code>:</p>
<pre>
TSQLRecordServiceLog = <strong>class</strong>(TSQLRecord)
...
<strong>published</strong>
<em>/// the 'interface.method' identifier of this call</em>
<em>// - this column will be indexed, for fast SQL queries, with the MicroSec</em>
<em>// column (for performance tuning)</em>
<strong>property</strong> Method: RawUTF8 <strong>read</strong> fMethod <strong>write</strong> fMethod;
<em>/// the input parameters, as a JSON document</em>
<em>// - will be stored in JSON_OPTIONS_FAST_EXTENDED format, i.e. with</em>
<em>// shortened field names, for smaller TEXT storage</em>
<em>// - content may be searched using JsonGet/JsonHas SQL functions on a</em>
<em>// SQlite3 storage, or with direct document query under MongoDB/PostgreSQL</em>
<strong>property</strong> Input: <strong>variant read</strong> fInput <strong>write</strong> fInput;
<em>/// the output parameters, as a JSON document, including result: for a function</em>
<em>// - will be stored in JSON_OPTIONS_FAST_EXTENDED format, i.e. with</em>
<em>// shortened field names, for smaller TEXT storage</em>
<em>// - content may be searched using JsonGet/JsonHas SQL functions on a</em>
<em>// SQlite3 storage, or with direct document query under MongoDB/PostgreSQL</em>
<strong>property</strong> Output: <strong>variant read</strong> fOutput <strong>write</strong> fOutput;
<em>/// the Session ID, if there is any</em>
<strong>property</strong> Session: integer <strong>read</strong> fSession <strong>write</strong> fSession;
<em>/// the User ID, if there is an identified Session</em>
<strong>property</strong> User: integer <strong>read</strong> fUser <strong>write</strong> fUser;
<em>/// will be filled by the ORM when this record is written in the database</em>
<strong>property</strong> Time: TModTime <strong>read</strong> fTime <strong>write</strong> fTime;
<em>/// execution time of this method, in micro seconds</em>
<strong>property</strong> MicroSec: integer <strong>read</strong> fMicroSec <strong>write</strong> fMicroSec;
<strong>end</strong>;
</pre>
<p>As you can see, all input and output parameters are part of the record, as
two <code><a href="https://blog.synopse.info?post/post/2014/02/25/TDocVariant-custom-variant-type">TDocVariant</a></code>
instances. Since they are stored as JSON/TEXT, you could perform some requests
directly on their content, especially if actual storage take place in a
<em><a href="https://blog.synopse.info?post/post/2014/05/07/MongoDB-mORMot-ORM-ODM">MongoDB</a></em>
database: you may even use dedicated indexes on the parameter values, and/or
run advanced <em>map/reduce</em> queries.</p>
<p>Since very accurate timing, with a micro-second resolution, is part of the
information, you would be able to make filtering or advanced statistics using
simple SQL clauses. It has never been easier to monitor your SOA system, and
identify potential issues.<br />
You may easily extract this information from your database, and feed a
real-time visual monitoring chart system, for instance. Or identify and spy
unusual execution patterns (e.g. unexpected timing or redounding error codes),
which would match some SQL requests: those SQL statements may be run
automatically on a regular basis, to prevent any problem before it actually
happen.</p>
<h3>Tracing Asynchronous External Calls</h3>
<p>Sometimes, your server may be the client of another process. In an SOA
environment, you may interface with a third-party REST service for an external
process, e.g. sending a real-time notification.</p>
<p>On the REST client instance, you can execute the
<code>TServiceFactoryClient.SendNotifications()</code> method for a given
service:</p>
<pre>
aNotificationClientService.SendNotifications(aServicesLogRest,
TSQLRecordServiceNotifications, fSettings.NotificationsRetrySeconds);
</pre>
<p>This single command would create an Audit Trail with all notification calls
sent to <code>aNotificationClientService</code>, in the
<code>TSQLRecordServiceNotifications</code> ORM class of
<code>aServicesLogRest</code>.</p>
<p>You may use the following <code>TSQLRecordServiceNotifications
class</code>:</p>
<pre>
TSQLRecordServiceNotifications = <strong>class</strong>(TSQLRecordServiceLog)
...
<strong>published</strong>
<em>/// when this notification has been sent</em>
<em>// - equals 0 until it was actually notified</em>
<strong>property</strong> Sent: TTimeLog <strong>read</strong> fSent <strong>write</strong> fSent;
<strong>end</strong>;
</pre>
<p>The additional <code>Sent</code> property would contain the
<code>TTimeLog</code> time-stamp on which the notification would have taken
place.</p>
<p>In fact, all methods executed via this notification service would now be
first stored in this table, then the remote HTTP notifications would take place
asynchronously in the background.<br />
Transmission would be in order (first-in-first-out), and in case of any
connection problem (e.g. the remote server not returning a <code>200 HTTP
SUCCESS</code> status code), it won't move to the next entry, and would retry
after the <code>NotificationsRetrySeconds</code> period, as supplied to the
<code>SendNotifications()</code> method.</p>
<p>Of course, you may define your own sub-class, to customize the destination
Audit Trail table:</p>
<pre>
<strong>type</strong>
TSQLMyNotifications = <strong>class</strong>(TSQLRecordServiceNotifications);
</pre>
<p>Thanks to those <code>TSQLRecordServiceLog</code> classes, high-level
support and analysis has never become easier. The actual implementation of
those features has been tuned to minimize the impact on main performance, by
using e.g. delayed write operations via <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_28">
BATCH sequences for adding/updating/deleting records</a></em>, or a dedicated
background thread for the asynchronous notification process.</p>
<p>For updated information, take a look at <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_183">
our official documentation</a>!</p>Try to avoid RTTI (ab)useurn:md5:d296cf380ab8ae68e214ecaab44a8ef22015-11-21T14:30:00+01:002015-11-22T13:06:03+01:00AB4327-GANDIPascal ProgrammingblogDelphiGoodPracticeMetaDataMetaProgrammingmORMotMVCMVVMORMRTTISOAUserInterface<p>There is a very trendy move, since a few years, to value so called "<a href="https://en.wikipedia.org/wiki/Metaprogramming">meta-programming</a>".<br />
In short, it is about the ability to treat programs as their data.<br />
It is a very powerful paradigm in functional languages, and it was also
introduced to OOP languages, <a href="http://csl.ensm-douai.fr/MetaclassTalk">even in SmallTalk</a> a long time
before this concept was trendy in Ruby, C# or Java.</p>
<p><a href="https://pragprog.com/book/cmelixir/metaprogramming-elixir"><img src="https://imagery.pragprog.com/products/430/cmelixir_xlargecover.jpg?1415371472" alt="" /></a></p>
<p>In OOP compiled languages, reflection is used to achieve a similar behavior
at run-time, mainly via RTTI (<a href="https://en.wikipedia.org/wiki/Run-time_type_information">Run-Time Type
Information</a>).<br />
Delphi <a href="http://www.blong.com/Conferences/BorConUK98/DelphiRTTI/CB140.htm">supports
RTTI since its version 1</a>, as it was heavily used e.g. for all UI
streaming.<br />
In our framework, we rely on RTTI for its main features: <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_3">
ORM</a>, <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_63">SOA</a>
and <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_465">
MVC</a> - and even in some other parts, like <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_5">
Desktop UI generation</a>.</p>
<p>But RTTI could easily be abused.<br />
Here are some thoughts, started as a comment in a <a href="http://tech.turbu-rpg.com/473/the-next-rtti-bottleneck/comment-page-1#comment-197674">
good old Mason's blog article</a> about how RTTI performance may be a
bottleneck.<br />
My comment was to get rid of RTTI, and follow a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_47">SOLID</a>
implementation with explicit OOP code, like use of <code><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_46">
interface</a></code>.</p> <p>To be fair, our UI generation from RTTI was more a proof of concept.<br />
It works on some existing projects, with a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#SYNFILETMS_PNG">
nice rendering and user experience</a>.<br />
But the auto-generated UI result has a fixed layout. To customize it, we had to
include additional parameters, and introduce complexity in the generator.<br />
Good old RAD, with a SOA backend, or a MVC/MVVM design approach, as <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_465">
we propose for web servers</a>, would have been more productive.</p>
<p>We admit that some patterns like <a href="https://en.wikipedia.org/wiki/Aspect-oriented_programming">Aspect Oriented
Programing</a> could easily add some behavior to an existing code
base.<br />
The most obvious example is to write some attributes in code and enable method
execution logging.<br />
Sounds like a magical way of writing code.</p>
<p>Just like with our UI generation trick, adding logging could be just as easy
as writing an attribute to the class. Magic! Feature implemented in 5 minutes.
Now you can take a coffee for the whole day - task was budgeted for a
day.<br />
But logging content is poor (only method name/parameters). So you start adding
some attributes at the method level, to customize the log message. And you
start polluting your class. And, sadly, at attribute level, you do not have
access to all needed information (e.g. values or functions in other classes).
So you add an explicit log method call within the method code. And find it
convenient, finally.</p>
<p>In <em>mORMot</em>, you can have auto-generated logging (e.g. all SOA
interface calls, including input parameters and output values), in the log
system, or even in a log database (very convenient, in practice).<br />
This uses RTTI, and is indeed magic. Just a parameter, in the settings file,
and you obtain some additional tracking information. And you did not change the
business logic!<br />
But this automated logging is not enough. You should now have to write a lot of
manual logging, even at business layer, to track real execution context.</p>
<p>But all this run-time injected behavior make it harder to maintain and
debug.<br />
By looking at the code, you could not guess what is really executed.<br />
There is a lot of hidden mechanisms, which are not explicit any more.<br />
When you start to put some business logic using such injection/reflection
techniques, it eventually add complexity.<br />
We have seen unexpected rebounds in <a href="http://martinfowler.com/eaaDev/EventSourcing.html">Event-Sourcing</a>
architectures, using such hidden code injection to propagate events.<br />
I've seen recently servers becoming unstable because of real-time logging
issues: sending the log information remotely did create some kind of recursion,
due to the remote sending logging itself its own process!</p>
<p>Maintainable code should always try to make the implicit explicit.<br />
We would not make explicit all the low-level plumbing - otherwise we <a href="http://www.menuetos.net/">would code everything in assembler</a>.<br />
But we should better make all <em>logical process</em> explicit.<br />
This is where <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_54">
DDD excels</a>.</p>
<p>Using RTTI is a tricky business... which should be exceptional.<br />
Relying on RTTI at runtime, for usual code, smells like a break of <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_159">
Liskov Substitution Principle</a> (and the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_158">
Open/Closed Principle</a>).<br />
Most of the time, code would be much cleaner when using proper OOP and
<code>interface</code> types definition, instead of relying on RTTI and
<code>TValue</code>.<br />
In <em>mORMot</em>, we achieve amazing results without the <a href="http://www.tindex.net/Language/EnhancedRTTI.html">enhanced RTTI</a>, as
introducing in Delphi 10. If it is available, we would use it. But it is not,
we would be able to <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_219">
find workarounds</a>, even for <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_624">
compilation with FPC</a>.</p>
<p>And it is not my own opinion.<br />
In fact, Microsoft, in its newest orientation to "Native" compilation in .Net,
does advice to avoid RTTI (ab)use.<br />
Even at technical level, all this run-time "injection" and "inspection" stuff
is <a href="https://msdn.microsoft.com/en-us/library/dn600640">not so trendy
any more</a>...</p>
<p>RTTI is great, but it may be abused.<br />
There is good laziness, and unsafe laziness.<br />
If you add cross-cutting features (like logging, or authorization) via RTTI -
fine.<br />
But if you put some business logic via code injection - beware.<br />
This blog post just aims to state that, as always, RTTI use is a matter of
balance: design principles should rule, not code magic!</p>
<p>As <a href="https://plus.google.com/107629505609094966321">Markus</a>
commented, as one of the <a href="http://www.amazon.com/dp/0132350882">clean code principles</a>: "Someone
should be able to read the source code like a book and be able to figure out
what it is doing without comments."</p>
<p>Feedback may be <a href="https://plus.google.com/+ABouchez/posts/RVMdYpn4j2b">shared on
Google+</a>.<br />
<em>Edit</em>: I've added one more example, about logging via AOP.</p>Benefits of interface callbacks instead of class messagesurn:md5:5b6d3e9963cfddc9617e774e4a9345e92015-11-17T21:01:00+01:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkblogDelphiDocumentationDomainDrivenEventCollaborationEventSourcinginterfacemockmORMotperformancePublish-SubscribeSOASOLIDstubTestingWebSockets<p>If you compare with existing client/server SOA solutions (in Delphi, Java,
C# or even in Go or other frameworks), <em>mORMot</em>'s
<code>interface</code>-based <a href="https://blog.synopse.info?post/post/2015/04/06/Asynchronous-Service-WebSockets%2C-Callbacks-and-Publish-Subscribe">
callback mechanism</a> sounds pretty unique and easy to work with.</p>
<p><img src="http://www.geo-cc.com/images/soa_middleware.png" alt="" /></p>
<p>Most <em>Events Oriented</em> solutions do use a set of dedicated
<em>messages</em> to propagate the events, with a centralized <em>Message
Bus</em> (like <a href="https://msdn.microsoft.com/en-us/library/ms711472"><em>MSMQ</em></a> or
<em><a href="https://docs.oracle.com/javaee/6/tutorial/doc/bncdr.html">JMS</a></em>), or a
P2P/decentralized approach (see e.g. <a href="http://zeromq.org/"><em>ZeroMQ</em></a> or <em><a href="http://nanomsg.org/">NanoMsg</a></em>). In practice, you are expected to
define one <code>class</code> per message, the <code>class</code> fields being
the message values. You would define e.g. one <code>class</code> to notify a
successful process, and another <code>class</code> to notify an error. SOA
services would eventually tend to be defined by a huge number of individual
classes, with the temptation of re-using existing classes in several
contexts.</p>
<p>Our <code>interface</code>-based approach allows to gather all events:</p>
<ul>
<li>In a single <code>interface</code> type per <em>notification</em>, i.e.
probably per <em>service operation</em>;</li>
<li>With one <em>method</em> per <em>event</em>;</li>
<li>Using method <em>parameters</em> defining the event <em>values</em>.</li>
</ul>
<p>Since asynchronous notifications are needed most of the time, method
parameters would be one-way, i.e. defined only
as <code>const</code> - in such case, an evolved algorithm would
transparently gather those outgoing messages, to enhance scalability when
processing such asynchronous events. Blocking request may also be defined
as <code>var/out</code>, as we will see below, in<em>Workflow
adaptation</em>.</p>
<p>Behind the scene, the framework would still transmit raw messages over IP
sockets (currently over a <a href="https://blog.synopse.info?post/post/2015/04/06/Asynchronous-Service-WebSockets%2C-Callbacks-and-Publish-Subscribe">
WebSockets</a> connection), like other systems, but events notification would
benefit from using interfaces, on both server and client sides.<br />
We will now see how...</p> <h3>Using service and callback interfaces</h3>
<p>For instance, you may define the following generic service and callback to
retrieve a picture from a remote camera, using <em>mORMot</em>'s
<code>interface</code>-based approach:</p>
<pre>
<strong>type</strong>
<em>// define some custom types to make the implicit explicit</em>
TCameraID = RawUTF8;
TPictureID = RawUTF8;
<em>// mORMot notifications using a callback interface definition</em>
IMyCameraCallback = <strong>interface</strong>(IInvokable)
['{445F967F-79C0-4735-A972-0BED6CC63D1D}']
<strong>procedure</strong> Started(<strong>const</strong> Camera: TCameraID; <strong>const</strong> Picture: TPictureID);
<strong>procedure</strong> Progressed(<strong>const</strong> Camera: TCameraID; <strong>const</strong> Picture: TPictureID;
CurrentSize,TotalSize: cardinal);
<strong>procedure</strong> Finished(<strong>const</strong> Camera: TCameraID; <strong>const</strong> Picture: TPictureID;
<strong>const</strong> PublicURI: RawUTF8; TotalSize: cardinal);
<strong>procedure</strong> ErrorOccured(<strong>const</strong> Camera: TCameraID; <strong>const</strong> Picture: TPictureID;
<strong>const</strong> MessageText: RawUTF8);
<strong>end</strong>;
<em>// mORMot main service, also defined as an interface</em>
IMyCameraService = <strong>interface</strong>(IInvokable)
['{3CE61E74-A01D-41F5-A414-94F204F140E1}']
<strong>function</strong> TakePicture(<strong>const</strong> Camera: TCameraID; <strong>const</strong> Callback: IMyCameraCallback): TPictureID;
<strong>end</strong>;
</pre>
<p>In a single look, I guess you did get the expectation of the "Camera
Service".<br />
Take a deep breath, and keep in mind those two type definitions as
reference.<br />
We will now compare with a classical message-based pattern.</p>
<h3>Classical message(s) event</h3>
<p>With a <code>class</code>-based message kind of implementation, you may
either have to define a single class, containing all potential information:</p>
<pre>
<strong>type</strong>
<em>// a single class message would need a status</em>
TMyCameraCallbackState = (
ccsStarted, ccsProgressed, ccsFinished, ccsErrorOccured);
<em>// the single class message</em>
TMyCameraCallbackMessage = <strong>class</strong>
<strong>private</strong>
fCamera: TCameraID;
fPicture: TPictureID;
fTotalSize: cardinal;
fMessageText: RawUTF8;
fState: TMyCameraCallbackState;
<strong>published</strong>
<strong>property</strong> State: TMyCameraCallbackState <strong>read</strong> fState <strong>write</strong> fState;
<strong>property</strong> Camera: TCameraID <strong>read</strong> fCamera <strong>write</strong> fCamera;
<strong>property</strong> Picture: TPictureID <strong>read</strong> fPicture <strong>write</strong> fPicture;
<strong>property</strong> TotalSize: cardinal <strong>read</strong> fTotalSize <strong>write</strong> fTotalSize;
<strong>property</strong> MessageText: RawUTF8 <strong>read</strong> fMessageText <strong>write</strong> fMessageText;
<strong>end</strong>;
</pre>
<p>This single class is easy to write, but makes it a bit confusing to consume
the notification. Which field comes with which state? The client-side code
would eventually consist of a huge <code>case aMessage.State of</code> ...
block, with potential issues. The business logic does not appear in this type
definition. Easy to write, difficult to read - and maintain...</p>
<p>In order to have an implementation closer to <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_47">
SOLID design principles</a></em>, you may define a set of classes, as such:</p>
<pre>
<strong>type</strong>
<em>// all classes would inherit from this one, to have common properties</em>
TMyCameraCallbackAbstract = <strong>class</strong>
<strong>private</strong>
fCamera: TCameraID;
fPicture: TPictureID;
<strong>published</strong>
<strong>property</strong> Camera: TCameraID <strong>read</strong> fCamera <strong>write</strong> fCamera;
<strong>property</strong> Picture: TPictureID <strong>read</strong> fPicture <strong>write</strong> fPicture;
<strong>end</strong>;
<em>// message class when the picture acquisition starts</em>
TMyCameraCallbackStarted = <strong>class</strong>(TMyCameraCallbackAbstract);
<em>// message class when the picture is acquired</em>
TMyCameraCallbackFinished = <strong>class</strong>(TMyCameraCallbackAbstract)
<strong>private</strong>
fPublicURI: RawUTF8;
fTotalSize: cardinal;
<strong>published</strong>
<strong>property</strong> TotalSize: cardinal <strong>read</strong> fTotalSize <strong>write</strong> fTotalSize;
<strong>property</strong> PublicURI: RawUTF8 <strong>read</strong> fPublicURI <strong>write</strong> fPublicURI;
<strong>end</strong>;
<em>// message during picture download</em>
TMyCameraCallbackProgressed = <strong>class</strong>(TMyCameraCallbackFinished)
<strong>private</strong>
fCurrentSize: cardinal;
<strong>published</strong>
<strong>property</strong> CurrentSize: cardinal <strong>read</strong> fCurrentSize <strong>write</strong> fCurrentSize;
<strong>end</strong>;
<em>// error message</em>
TMyCameraCallbackErrorOccured = <strong>class</strong>(TMyCameraCallbackAbstract)
<strong>private</strong>
fMessageText: RawUTF8;
<strong>published</strong>
<strong>property</strong> MessageText: RawUTF8 <strong>read</strong> fMessageText <strong>write</strong> fMessageText;
<strong>end</strong>;
</pre>
<p>Inheritance makes this class hierarchy not as verbose as it may have been
with plain "flat" classes, but it is still much less readable than the
<code>IMyCameraCallback</code> type definition.</p>
<p>In both cases, such <code>class</code> definitions make it difficult to
guess which message does match with a given service. You must be very careful
and consistent about your naming conventions, and uncouple your service
definitions in clear name spaces.</p>
<p>When implementing SOA services, DDD's <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_545">
Ubiquitous Language</a></em> tends to be polluted by the <code>class</code>
definition (getters and setters), and implementation details of the
messages-based notification: your <em>Domain</em> code would be tied to the
message oriented nature of the <em>Infrastructure</em> layer.
<code>interface</code> callbacks would therefore help implementing DDD's
<em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_570">
Event-Driven</a></em> pattern, in a cleaner way.</p>
<h3>Workflow adaptation</h3>
<p>Sometimes, it may be necessary to react to some unexpected event. The
consumer may be able to change the workflow of the producer, depending on some
business rules, or user expectations. By definition, all message-based
implementation are asynchronous: as a result, implementing "reverse" messaging
tends to be difficult to write and debug.</p>
<p>A common implementation is to have a dedicated set of "answer" messages, to
notify the service providers of a state change - it comes with potential race
conditions, or unexpected rebound phenomenons, for instance when you add a node
to an existing event-driven system.</p>
<p>Another solution may be to define explicit <em>rules</em> for service
providers, e.g. when the service is called. You may define a set of workflows,
injected to the provider service at runtime. It will definitively tend to break
the <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_333">
Single Responsibility Principle</a></em>.</p>
<p>On the other hand, since <em>mORMot</em>'s callbacks are true
<code>interface</code> methods, they may return some values (as a
<code>function</code> result or a <code>var/out</code> parameter). On the
server side, such callbacks would block and wait for the client end to
respond.</p>
<p>So by writing an additional method like:</p>
<pre>
IMyCameraCallback = <strong>interface</strong>(IInvokable)
...
<strong>function</strong> ShouldRetryIfBusy(<strong>const</strong> Camera: TCameraID; <strong>const</strong> Picture: TPictureID): boolean;
...
</pre>
<p>... you would be able to implement any needed complex workflow adaptation,
in real time.<br />
The server side code would still be very readable and efficient, with no
complex plumbing, wait queue or state machine to set up.</p>
<h3>From interfaces come abstraction and ease</h3>
<p>As an additional benefit, integration with the Delphi language is clearly
implementation agnostic: you are not even tied to use the framework, when
working with such <code>interface</code> type definitions. In fact, this is a
good way of implementing callbacks conforming to <em>SOLID design
principles</em> on the server side, and let the <em>mORMot</em> framework
publish this mechanism in a client/server way, by using <em>WebSockets</em>,
only if necessary.</p>
<p>The very same code could be used on the server side, with no transmission
nor marshalling overhead (via direct <code>interface</code> instance calls),
and over a network, with optimized use of resource and bandwidth (via "fake"
<code>interface</code> calls, and binary/JSON marshalling over TCP/IP).</p>
<p>On the server side, your code - especially your <em>Domain</em> code - may
interact directly with the lower level services, defined in the <em>Domain</em>
as <code>interface</code> types, and implemented in the <em>infrastructure</em>
layer. You may host both <em>Domain</em> and <em>Infrastructure</em> code in a
single server executable, with direct assignment of local <code>class</code>
instance as callbacks. This will minimize the program resources, in both CPU
and memory terms - which is always a very valuable goal, for any business
system.</p>
<p>Last but not least, using an <code>interface</code> would help implementing
the whole callback mechanism using <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_166">
Stubs and mocks</a></em>, e.g. for easy unit testing via <em>Calls
tracing</em>.<br />
You may also write your unit tests with real local callback <code>class</code>
instances, which would be much easier to debug than over the whole
client/server stack. Once you identified a scenario which fails the system, you
could reproduce it with a dedicated test, even in an aggressive multi-threaded
way, then use the debugger to trace the execution and identify the root cause
of the issue.</p>Letters of Hopeurn:md5:e98a568eefb37c7fe997014115e6ed3f2015-10-23T14:19:00+02:002015-10-23T13:22:47+02:00AB4327-GANDIPascal ProgrammingblogCQRSDelphiDomainDrivenEventCollaborationEventSourcingMongoDBmORMotNoSQLSOA <p>As we <a href="https://blog.synopse.info?post/post/2015/09/21/Embarcadero-bought-by">already notified in
this blog</a>, Embarcadero has been finally bought by IDERA.</p>
<p>Delphi users <a href="http://www.yanniel.info/2015/10/ideras-intention-for-embarcadero-development-tools.html">
received a letter from Randy Jacops</a>, IDERA CEO.<br />
Written in my mother language, in perfect French. Nice!</p>
<p><img src="http://borgenproject.org/wp-content/uploads/Demonstrate_Hope.jpg" alt="" /></p>
<p>The letter states that they have 20,000 customers...<br />
It sounds more realistic than the numbers usually given for Delphi
"users".<br />
Even if it counts for all their tools.<br />
<img src="https://blog.synopse.info?pf=smile.svg" alt=":)" class="smiley" /></p>
<p>In our forums, we have 1,384 <a href="http://synopse.info/forum/userlist.php">registered users</a> (real humans: we
do not accept bots via a <a href="https://en.wikipedia.org/wiki/Turing_test">Turing test</a> during
registration).<br />
It sounds like if Open Source projects are able to gather a lot of users.<br />
And certainly because we maintain support from Delphi 6 up to Seattle (and even
Delphi 5 for some part of our libraries)... we have for sure users using
FPC/Lazarus (which we also started to support), and others which did not
upgrade to the latest Delphi version!</p>
<p>In Randy's letter, the community has a special place.<br />
I hope future of Delphi would see Open Source projects brought by the community
as a chance, <a href="http://synopse.info/forum/viewtopic.php?pid=17453#p17453">not as
competition</a>.</p>
<p>I'm currently working on a cloud of <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html">
<em>mORMot</em> servers</a>, serving content coming from high numbers of
connected objects.<br />
Object Pascal powered servers, under Windows or Linux (with FPC), are working
24/7 with very low resource use.<br />
A lot of BigData stream is gathered into <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_84">
MongoDB</a> servers, following the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_167">CQRS
pattern</a>.<br />
It is so easy to deploy those servers (including their high performance
embedded SQlite3 database), that almost everyone in my company did install
their own "cloud", mainly for testing purpose of the objects we are
selling...<br />
Real-time remote monitoring of the servers is very easy and integrated. You
could even see the log changing in real-time, or run your SQL requests on the
databases, with ease.<br />
When I compare to previous projects I had to write or maintain using Java or
.Net, I can tell you that it is "something else".<br />
The IT administrators were speechless when they discovered how it worked: no
need of containers, no need of virtual machines (but for infrastructure
needs)...<br />
The whole stack is <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_17">
SOA oriented</a>, in an <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_173">
Event-Driven design</a> (thanks to WebSockets callbacks). It follows <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_54">
DDD principles</a>, thanks to the perfect readability of the object pascal
language.<br />
Delphi, and Open Source, could be great to create <a href="https://en.wikipedia.org/wiki/Internet_of_Things">Internet Of
Things</a> servers...</p>Breaking Change in mORMot WebSockets binary protocolurn:md5:ed7c76eba0dd733889df8f53604d3f092015-08-15T14:25:00+02:002015-08-15T13:38:09+02:00AB4327-GANDImORMot FrameworkblogDelphiinterfaceJSONmORMotperformancePublish-SubscribeSOAWebSockets<p>Among all its means of transmission, our <em>mORMot</em> framework
features <a href="https://blog.synopse.info?post/post/2015/04/06/Asynchronous-Service-WebSockets%2C-Callbacks-and-Publish-Subscribe">
WebSockets</a>, allowing bidirectional communications, and interface-based
callbacks for real time notification of SOA events.<br />
After several months of use in production, we identified some needed changes
for this just emerged feature.</p>
<p><img src="http://blog.netclean.com/wp-content/uploads/2014/09/Screen-Shot-2015-01-15-at-14.36.14-314x160.png" alt="" /></p>
<p>We <a href="http://synopse.info/fossil/info/f14b1b2be544f0f42">committed
today a breaking change of the data layout</a> used for our proprietary
<em>WebSockets</em> binary protocol.<br />
From our tests, it would increase the performance and decrease the resource
consumption, especially in case of high number of messages.</p> <p>As <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_150">
our documentation states</a>, the <a href="http://synopse.info/files/html/api-1.18/SynBidirSock.html#TWEBSOCKETPROTOCOLBINARY">
TWebSocketProtocolBinary</a> class implements a binary proprietary protocol,
with optional frame compression and AES encryption (using <a href="https://blog.synopse.info?post/post/2015/01/15/AES-NI-enabled-for-SynCrypto">AES-NI hardware
instructions</a>, if available).</p>
<p>This protocol, even if it is implemented as REST compatible, as several
unique features:</p>
<ul>
<li>The requests are converted into binary frames, optionally compressed (using
<a href="https://blog.synopse.info?post/post/2010/06/26/Pure-pascal-LZO-and-SynLZ-compression-units">SynLZ</a>) and
encrypted;</li>
<li>Since encryption is at frame level, you would avoid the need to use an
additional TLS (aka <code>wss://</code>) layer, which is resource
consuming and not obvious to configure;</li>
<li>If the SOA method does not require any answer (i.e. if the Delphi interface
method is a <code>procedure</code> without any <code>out</code> parameter), the
corresponding frame won't be blocking, and would be added in an in-memory list
- of course, you can disable this feature on request;</li>
<li>The framework is able to <em>gather</em> several pending frames into a
single bigger message (a kind of "jumbo frame"), to reduce the resource use -
this makes a huge difference over a slow network, e.g. over the Internet, where
the latency could be noticeable.</li>
</ul>
<p>Our latest commit ensures that the gathered "jumbo frame" is compressed and
encrypted as one.<br />
Initial implementation did concatenate the already encrypted/compressed smaller
frames before transmission.<br />
This new behavior will reduce the CPU cost, and also the message size, since
compressing a big frame is much more efficient than compressing several smaller
frames, by definition.</p>
<p>But the previous transmitted data is not compatible with this new
layout.<br />
As a consequence, you should upgrade both client and server sides of any
application using our <em>WebSockets</em> binary protocol.<br />
Once we would release a stable version of the framework (the current unstable
1.18 branch would probably become stable 2.0), this format should not
change.</p>
<p>Of course, the JSON protocol (<a href="http://synopse.info/files/html/api-1.18/SynBidirSock.html#TWEBSOCKETPROTOCOLJSON">TWebSocketProtocolJSON</a>)
is not impacted by this change.</p>GetIt "Spirit" Concernsurn:md5:c8befaf992f7ae8f2053d260241925262015-06-06T10:52:00+02:002015-06-06T12:23:00+02:00AB4327-GANDIPascal ProgrammingblogDatabaseDelphiGetItmORMotOpenSourceORMSOA <p>I'm confused by the <a href="http://community.embarcadero.com/index.php/article/news/16027-submitting-libraries-to-getit">
GetIt Submitting official blog page</a>.<br />
Reminds me the darker ages of <a href="http://www.deltics.co.nz/blog/posts/1097">Delphi licensing change of
XE3</a>.</p>
<p><a href="http://blog.marcocantu.com/blog/2015-june-more-info-getit.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+marcocantublog+%28marcocantu.blog%29">
GetIt</a> is the new XE8 package manager for RAD Studio. Information about how
to submit your libraries to GetIt has just been made available by Embarcadero.
The idea behind GetIt is really to make is easier and faster to discover,
install, and keep updated some of the best open source libraries for Delphi and
C++Builder.</p>
<p>When you <a href="http://community.embarcadero.com/index.php/article/news/16027-submitting-libraries-to-getit">
look at the approval conditions</a>, it sounds like if mORMot would not find
its way in this package manager:</p>
<blockquote>
<p>Replacing key capabilities that are part of the core platforms definitions
such as Client/Server FireDAC pack or the DataSnap/EMS Enterprise middleware,
would be less likely to be accepted.<br />
The different SKUs are meant for different types of developers, and multi-tier
capabilities with strong client/server RDBMS integration
<strong>require</strong> an Enterprise edition license.<br />
We will bias acceptance toward GetIt libraries that respect the <strong>spirit
of our licensing and editions</strong>, not just use the letter of the license
and the technical boundaries. If you are unsure about your submission please
check with us first. </p>
</blockquote>
<p>What is this "<em>spirit of our licensing and editions</em>"?<br />
Why is it not part of the official license terms?<br />
Where does this assumption comes from?<br />
Would the licensing conditions change in the close future, as with the XE3
"episode"?<br />
Would Marco's interpretation become the new rule?</p>
<p>It clearly <a href="https://blog.synopse.info?post/post/2012/09/03/Client-Server-allowed-back-to-XE3-pro">reminds me the XE3
time where there was an attempt from Embarcadero to modify their licence
terms</a>, so that third party vendors or Open Source libraries would not be
allowed to create multi-tier frameworks with Delphi!<br />
Is it true that "<em>strong client/server RDBMS integration
<strong>require</strong> an Enterprise edition</em>" ?<br />
Last time I checked the licence terms, it was not stated.<br />
Why on earth would we have to pay for the <em>Entreprise</em> edition, if the
<em>Professionnal</em> edition is all that you need?</p>
<p><img src="http://static.tvtropes.org/pmwiki/pub/images/suicidalstupidity_6656.jpg" alt="" /></p>
<p>Still the same "closed" spirit.<br />
It is like if they know their own n-Tier solution weaknesses, so they try to
avoid any other possibility, but to use their own.<br />
They clearly do not understand the benefit and dynamic of Open Source.</p>
<p>I guess our little <em>mORMot</em> falls directly into this "unwelcomed"
category.<br />
I did not make the submission yet. But should I?<br />
Perhaps sub-part of the framework may find its way in: <a href="http://synopse.info/fossil/wiki?name=PDF+Engine">SynPdf</a>, <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html#">SynCrypto</a>,
<a href="http://synopse.info/files/html/api-1.18/SynGdiPlus.html#">SynGdiPlus</a>,
<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_45">
SynCommons</a>...<br />
But the main ORM/SOA/REST/MVC/StubMock features would certainly be
rejected.</p>
<p>Our Open Source project is sometimes preferred to DataSnap/EMS (or even
FireDAC), not only for licence cost, but also about features, documentation,
stability, compatibility with older versions of Delphi, performance, and Linux
compatibility.<br />
I have encountered several companies which are still using Delphi <em>because
of <em>mORMot</em></em>: if they did not have found it, they would have moved
to C# or Java, just to be able to use a full SOA/MVC stack, which was not
available, even in the "Enterprise" version of Delphi.</p>
<p>Story repeats itself.<br />
I just wanted to ensure that the licensing terms would not change in that
direction.<br />
I - as many Delphi users - would not let this GetIt "spirit" become the new
rule.<br />
We have to react, as we did for XE3, otherwise we may all suffer!<br />
IMHO Embacardero should better focus on the compiler and IDE, not cutting the
branch on which they are seated...</p>
<p>What do you think? <a href="http://synopse.info/forum/viewtopic.php?id=2623">Comments and feebacks are
welcome</a>!</p>Updated Slides about ORM SOA MVC SOLID DDDurn:md5:919d4c1d2e91e2a1aa13c784489587f22015-06-01T10:34:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkblogCQRSDelphiDocumentationDomainDrivenEventCollaborationEventSourcingGoodPracticemockmORMotORMSOASOLIDstub <p>One year ago, we <a href="https://blog.synopse.info?post/post/2014/04/18/Introducing-mORMot-s-architecture-and-design-principles">published
a set of slides</a> about the main concepts implemented by our framework.<br />
Mainly about ORM (and ODM), NoSQL, JSON, SOA, MVC (and MVVM), SOLID, DDD, CQRS
and some patterns like Stubs, Mocks, Factory, Repository, Unit-Of-Work.<br />
Worth a look, if you want to find out the benefits of the latest software
development techniques.<br />
They try to open the landscape of any Delphi developer (probably with a mostly
RAD and OOP background) to some new areas.</p>
<p><img src="http://img5.visualizeus.com/thumbs/ae/db/animal,lol,toocute,%D0%B7%D0%B2%D0%B5%D1%80%D0%B8,funny,marmot-aedb017085cd5d6d93b242e8bed32ff9_h.jpg?ts=93246" alt="" /></p>
<p>I just updated the slides from <a href="https://drive.google.com/folderview?id=0B0r8u-FwvxWdeVJVZnBhSEpKYkE&usp=sharing">
our public <em>GoogleDrive</em> folder</a>.<br />
They now reflect the latest state of the framework (e.g. ORM real-time
synchronization, asynchronous callbacks, DDD CQRS services...).<br />
They have also been polished after several public presentations, since I used
them as base for trainings I made for some European companies.</p>
<p>If you want to go further, or have some more enlightenment, ensure you
<a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html">
took a look at our framework Documentation</a>, which would detail all those
patterns, and how <em>mORMot</em> may help implementing them for your
projects!</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=16326#p16326">welcome on our
forum</a>, as usual!</p>SOLID Design Principlesurn:md5:d7cab0d0591daf63392c0f5efabde8522015-05-03T17:43:00+02:002015-05-03T17:06:16+02:00AB4327-GANDIPascal ProgrammingblogCrossPlatformDelphiDependencyInjectionDocumentationfactoryGoodPracticeinterfacemockRepositorySOASOLIDUnitTestingUserInterfaceweak pointers<p>I've just updated the documentation part about the SOLID Design
Principles.<br />
The <a href="https://blog.synopse.info?post/post/2011/11/27/SOLID-design-principles">former blog article</a>
(almost 4 years old!) sounds like a bit deprecated now...<br />
This is why I would extract here an updated version of this material.</p>
<p>Ensure you checked the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_47">
corresponding part of the mORMot documentation</a>, which is the updated
reference, and probably the easiest to read - including links to all the other
documentation.</p>
<p><img src="http://cre8ivethought.s3.amazonaws.com/images/solid/SOLID-small.jpg" alt="" /></p>
<p>The acronym SOLID is derived from the following OOP principles (quoted from
the corresponding <em>Wikipedia</em> article):</p>
<ul>
<li><em>Single responsibility principle</em>: the notion that an object should
have only a single responsibility;</li>
<li><em>Open/closed principle</em>: the notion that "software entities ...
should be open for extension, but closed for modification";</li>
<li><em>Liskov substitution principle</em>: the notion that "objects in a
program should be replaceable with instances of their subtypes without altering
the correctness of that program” - also named as "<em>design by
contract</em>";</li>
<li><em>Interface segregation principle</em>: the notion that "many client
specific interfaces are better than one general purpose interface.";</li>
<li><em>Dependency inversion principle</em>: the notion that one should "Depend
upon Abstractions. Do not depend upon concretions.". <em>Dependency
injection</em> is one method of following this principle, which is also called
<em>Inversion Of Control</em> (aka IoC).</li>
</ul>
<p>If you have some programming skills, those principles are general statements
you may already found out by yourself. If you start doing serious
object-oriented coding, those principles are best-practice guidelines you would
gain following.</p>
<p>They certainly help to fight the three main code weaknesses:</p>
<ul>
<li><em>Rigidity</em>: Hard to change something because every change affects
too many other parts of the system;</li>
<li><em>Fragility</em>: When you make a change, unexpected parts of the system
break;</li>
<li><em>Immobility</em>: Hard to reuse in another application because it cannot
be disentangled from the current application.</li>
</ul> <h3>1. Single Responsibility Principle</h3>
<p>When you define a class, it shall be designed to implement only one feature.
The so-called feature can be seen as an "<em>axis of change</em>" or a "<em>a
reason for change</em>".</p>
<p>Therefore:</p>
<ul>
<li>One class shall have only one reason that justifies changing its
implementation;</li>
<li>Classes shall have few dependencies on other classes;</li>
<li>Classes shall be abstract from the particular layer they are running - see
<em>Multi-tier architecture</em>.</li>
</ul>
<p>For instance, a <code>TRectangle</code> object should not have both
<code>ComputeArea</code> and <code>Draw</code> methods defined at once - they
would define two responsibilities or axis of change: the first responsibility
is to provide a mathematical model of a rectangle, and the second is to render
it on GUI.</p>
<h3>Splitting classes</h3>
<p>To take an example from real coding, imagine you define a communication
component. You want to communicate, say, with a bar-code scanner peripheral.
You may define a single class, e.g. <code>TBarcodeScanner</code>, supporting
such device connected over a serial port. Later on, the manufacturer deprecates
the serial port support, since no computer still have it, and offer only USB
models in its catalog. You may inherit from <code>TBarcodeScanner</code>, and
add USB support.</p>
<p><img src="https://blog.synopse.info?post/public/mORMot/SOLID1.png" alt="Single-to-rule-them-all class" title="Single-to-rule-them-all class, May 2015" /></p>
<p>But in practice, this new <code>TUsbBarCodeScanner</code> class is difficult
to maintain, since it will inherit from serial-related communication.<br />
So you start splitting the class hierarchy, using an <em>abstract</em> parent
class:</p>
<p><img src="https://blog.synopse.info?post/public/mORMot/SOLID3.png" alt="" title="Single Responsibility: Abstract parent class, May 2015" /></p>
<p>We may define some <code>virtual abstract</code> methods, which would be
overridden in inherited classes:</p>
<pre>
<strong>type</strong>
TAbstractBarcodeScanner = <strong>class</strong>(TComponent)
<strong>protected</strong>
<strong>function</strong> ReadChar: byte; <strong>virtual</strong>; <strong>abstract</strong>;
<strong>function</strong> ReadFrame: TProtocolFrame; <strong>virtual</strong>; <strong>abstract</strong>;
<strong>procedure</strong> WriteChar(aChar: byte); <strong>virtual</strong>; <strong>abstract</strong>;
<strong>procedure</strong> WriteFrame(<strong>const</strong> aFrame: TProtocolFrame); <strong>virtual</strong>; <strong>abstract</strong>;
...
</pre>
<p>Then, <code>TSerialBarCodeScanner</code> and <code>TUsbBarCodeScanner</code>
classes would override those classes, according to the final
implementation.</p>
<p>In fact, this approach is cleaner. But it is not perfect either, since it
may be hard to maintain and extend. Imagine the manufacturer is using a
standard protocol for communication, whatever USB or Serial connection is used.
You would put this communication protocol (e.g. its state machine, its stream
computation, its delaying settings) in the <code>TAbstractBarcodeScanner</code>
class. But perhaps they would be diverse flavors, in
<code>TSerialBarCodeScanner</code> or <code>TUsbBarCodeScanner</code>, or even
due to diverse models and features (e.g. if it supports 2D or 3D
bar-codes).</p>
<p>It appears that putting everything in a single class is not a good idea.
Splitting protocol and communication appears to be preferred. Each "<em>axis of
change</em>" - i.e. every aspect which may need modifications - requires its
own class. Then the <code>T*BarcodeScanner</code> classes would
<em>compose</em> protocols and communication classes within a single
component.</p>
<p>Imagine we have two identified protocols (named <code>BCP1</code> and
<code>BCP2</code>), and two means of communication (serial and USB). So we
would define the following classes:</p>
<p><img src="https://blog.synopse.info?post/public/mORMot/SOLID2.png" alt="Single Responsibility: Spliting protocol and communication" title="Single Responsibility: Spliting protocol and communication, May 2015" /></p>
<p>Then, we may define our final classes and components as such:</p>
<pre>
<strong>type</strong>
TAbstractBarcodeConnection = <strong>class</strong>
<strong>protected</strong>
<strong>function</strong> ReadChar: byte; <strong>override</strong>;
<strong>procedure</strong> WriteChar(aChar: byte); <strong>override</strong>;
...
TAbstractBarcodeProtocol = <strong>class</strong>
<strong>protected</strong>
fConnection: TAbstractBarcodeConnection;
<strong>function</strong> ReadFrame: TProtocolFrame; <strong>override</strong>;
<strong>procedure</strong> WriteFrame(<strong>const</strong> aFrame: TProtocolFrame); <strong>override</strong>;
...
TAbstractBarcodeScanner = <strong>class</strong>(TComponent)
<strong>protected</strong>
fProtocol: TAbstractBarcodeProtocol;
fConnection: AbstractBarcodeConnection;
...
</pre>
<p>And each actual inherited <code>class</code> would initialize the protocol
and connection according to the expected model:</p>
<pre>
<strong>constructor</strong> TSerialBarCodeScanner.Create(<strong>const</strong> aComPort: <strong>string</strong>; aBitRate: integer);
<strong>begin</strong>
fConnection := TSerialBarcodeConnection(aComPort,aBitRate);
fProtocol := TBCP1BarcodeProtocol.Create(fConnection);
<strong>end</strong>;
</pre>
<p>Here, we inject the connection instance to the protocol, since the later may
need to read or write some bytes on the wire, when needed.</p>
<p>Another example is how our database classes are defined in
<code>SynDB.pas</code> - see <em>External SQL database access</em>:</p>
<ul>
<li>The <em>connection properties</em> feature is handled by
<code>TSQLDBConnectionProperties</code> classes;</li>
<li>The actual <em>living connection</em> feature is handled by
<code>TSQLDBConnection</code> classes;</li>
<li>And <em>database requests</em> feature is handled by
<code>TSQLDBStatement</code> instances using dedicated
<code>NewConnection</code> / <code>ThreadSafeConnection</code> /
<code>NewStatement</code> methods.</li>
</ul>
<p>Therefore, you may change how a database connection is defined (e.g. add a
property to a <code>TSQLDBConnectionProperties</code> child), and you won't
have to change the statement implementation itself.</p>
<h3>Do not mix UI and logic</h3>
<p>Another practical "<em>Single Responsibility Principle</em>" smell may
appear in your uses clause.</p>
<p>If your data-only or peripheral-only unit starts like this:</p>
<pre>
<strong>unit</strong> MyDataModel;
<br /><strong>uses</strong>
<span style="background-color:yellow;">Winapi.Windows,</span>
mORMot,
...
</pre>
<p>It would induce a dependency about the <em>Windows</em> Operating System,
whereas your data would certainly benefit from being OS-agnostic. Our todays
compiler (Delphi or FPC) targets several OS, so coupling our data to the actual
<em>Windows</em> unit does show a bad design.</p>
<p>Similarly, you may add a dependency to the VCL, via a reference to the
<code>Forms</code> unit.<br />
If your data-only or peripheral-only unit starts like the following,
beware!</p>
<pre>
<strong>unit</strong> MyDataModel;
<br /><strong>uses</strong>
<span style="background-color:yellow;">Winapi.Messages,</span>
<span style="background-color:yellow;">Vcl.Forms,</span>
mORMot,
...
</pre>
<p>If you later want to use <code>FMX</code>, or <code>LCL</code> (from
Lazarus) in your application, or want to use your <code>MyDataModel</code> unit
on a pure server application without any GUI, you are stuck. The upcoming
<em>Windows Nano Server</em> architecture, which targets the cloud and won't
offer any GUI to the server applications, would even be very sensitive to the
dependency chain of the executable.</p>
<p>Note that if you are used to developed in RAD mode, the units generated by
the IDE wizards come with some default references in the <code>uses</code>
clause of the generated <code>.pas</code> file. So take care of not introducing
any coupling to your own business code!</p>
<p>As a general rule, our ORM/SOA framework source code tries to avoid such
dependencies. All OS-specificities are centralized in our
<code>SynCommons.pas</code> unit, and there is no dependency to the VCL when it
is not mandatory, e.g. in <code>mORMot.pas</code>.</p>
<p>Following the RAD approach, you may start from your UI, i.e. defining the
needed classes in the unit where you visual form (may be VCL or FMX) is
defined. Don't follow this tempting, but dangerous path!</p>
<p>Code like the following may be accepted for a small example (e.g. the one
supplied in the <code>SQlite3\Samples</code> sub-folder of our repository
source code tree), but is to be absolutely avoided for any production ready
<em>mORMot</em>-based application:</p>
<pre>
<strong>interface</strong>
<br /><strong>uses</strong>
Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
Vcl.Controls, Vcl.Forms, Vcl.Dialogs, mORMot, mORMotSQLite3;
<br /><strong>type</strong>
TForm1 = <strong>class</strong>(TForm)
<strong>procedure</strong> FormCreate(Sender: TObject);
<strong>private</strong>
<span style="background-color:yellow;">fModel: TSQLModel;</span>
<span style="background-color:yellow;">fDatabase: TSQLRestServerDB;</span>
<strong>public</strong>
<em>{ Public declarations }</em>
<strong>end</strong>;
<br /> <strong>implementation</strong>
<br /><strong>procedure</strong> TForm1.FormCreate(Sender: TObject);
<strong>begin</strong>
<span style="background-color:yellow;">fModel := TSQLModel.Create([TSQLMyOwnRecord],'root');</span>
<span style="background-color:yellow;">fDatabase := TSQLRestServerDB.Create(fModel,ChangeFileExt(paramstr(0),'.db'));</span>
<strong>end</strong>;
</pre>
<p>In your actual project units, when you define an ORM or SOA
<code>class</code>, never include GUI methods within. In fact, the fact that
our <code>TSQLRecord</code> class definitions are common to both Client and
Server sides makes this principle mandatory. You should not have any GUI
related method on the Server side, and the Client side could use the objects
instances with several GUI implementations (<em>Delphi</em> Client, AJAX
Client...).</p>
<p>Therefore, if you want to change the GUI, you won't have to recompile the
<code>TSQLRecord</code> class and the associated database model. If you want to
deploy your server on a <em>Linux</em> box (using e.g. <em>CrossKylix</em> or
FPC as compiler), you could reuse you very same code, since you do not have
reference to the VCL in your business code.</p>
<p>This <em>Single responsibility principle</em> may sound simple and easy to
follow (even obvious), but in fact, it is one of the hardest principles to get
right. Naturally, we tend to join responsibilities in our class definitions.
Our framework architecture will enforce you, by its Client-Server nature and
all its high-level methods involving <code>interface</code>, to follow this
principle, but it is always up to the end coder to design properly his/her
types.</p>
<h3>2. Open/Closed Principle</h3>
<p>When you define a class or a unit, at the same time:</p>
<ul>
<li>They shall be <em>open for extension</em>;</li>
<li>But <em>closed for modification</em>.</li>
</ul>
<p>It means that you may be able to extend your existing code, without breaking
its initial behavior.<br />
Some other guidelines may be added, but you got the main idea.</p>
<p>Conformance to this open/closed principle is what yields the greatest
benefit of OOP, i.e.:</p>
<ul>
<li>Code re-usability;</li>
<li>Code maintainability;</li>
<li>Code extendibility.</li>
</ul>
<p>Following this principle will make your code far away from a regular RAD
style. But benefits will be huge.</p>
<h3>Applied to our framework units</h3>
<p>When designing our ORM/SOA set of units, we tried to follow this principle.
In fact, you should not have to modify its implementation. You should define
your own units and classes, without the need to <em>hack</em> the framework
source code.</p>
<p>Even if <em>Open Source</em> paradigm allows you to modify the supplied
code, this shall not be done unless you are either fixing a bug or adding a new
common feature. This is in fact the purpose of our <a href="http://synopse.info">http://synopse.info</a> web site, and most of the
framework enhancements have come from user requests.</p>
<p>The framework Open Source license - see <em>License</em> - may encourage
user contributions in order to fulfill the Open/closed design principle:</p>
<ul>
<li>Your application code extends the <em>Synopse mORMot Framework</em> by
defining your own classes or event handlers - this is how it is <em>open for
extension</em>;</li>
<li>The main framework units shall remain inviolate, and common to all users -
this illustrates the <em>closed for modification</em> design.</li>
</ul>
<p>As a beneficial side effect, this principle will ensure that your code would
be ready to follow the framework updates (which are quite regular). When a new
version of <em>mORMot</em> is available, you should be able to retrieve it for
free from our web site, replace your files locally, then build a new enhanced
version of your application, with the benefit of all included fixes and
optimizations. Even the source code repository is available - at <a href="http://synopse.info/fossil">http://synopse.info/fossil</a> or from <a href="https://github.com/synopse/mORMot">https://github.com/synopse/mORMot</a> - and
allows you to follow the current step of evolvment of the framework.</p>
<p>In short, abstraction is the key to peace of mind. All your code shall not
depend on a particular implementation.</p>
<h3>Open/Close in practice</h3>
<p>In order to implement this principle, several conventions could be
envisaged:</p>
<ul>
<li>You shall better define some abstract classes, then use specific overridden
classes for each and every implementation: this is for instance how
Client-Server classes were implemented - see <em>Client-Server
process</em>;</li>
<li>All object members shall be declared <code>private</code> or
<code>protected</code> - this is a good idea to use <em>Service-Oriented
Architecture (SOA)</em> for defining server-side process, and/or make the
<code>TSQLRecord</code> published properties read-only and using some
client-side <code>constructor</code> with parameters;</li>
<li>No singleton nor global variable - <em>ever</em>;</li>
<li>RTTI is dangerous - that is, let our framework use RTTI functions for its
own cooking, but do not use it in your code.</li>
</ul>
<p>In our previous bar-code scanner class hierarchy, we would therefore define
the</p>
<pre>
<strong>type</strong>
TAbstractBarcodeScanner = <strong>class</strong>(TComponent)
<strong>protected</strong>
fProtocol: TAbstractBarcodeProtocol;
fConnection: AbstractBarcodeConnection;
...
<strong>public</strong>
<span style="background-color:yellow;"><strong>property</strong> Protocol: TAbstractBarcodeProtocol <strong>read</strong> fProtocol;</span>
<span style="background-color:yellow;"><strong>property</strong> Connection: AbstractBarcodeConnection <strong>read</strong> fConnection;</span>
...
</pre>
<p>In this code, the actual variables are stored as <code>protected</code>
fields, with only getters (i.e. <strong><code>read</code></strong>) in the
<code>public</code> section. There is no setter (i.e.
<strong><code>write</code></strong>) attribute, which may allow to change the
<code>fProtocol/fConnection</code> instances in user code. You can still access
those fields (it is mandatory in your inherited constructors), but user code
should not use it.</p>
<p>As stated above, for our bar-code reader design, having dedicated classes
for defining protocol and connection will also help implementing the
<em>open/close</em> principle. You would be able to define a new class,
combining its own protocol and connection class instances, so it will be
<em>Open for extension</em>. But you would not change the behavior of a class,
by inheriting it: since protocol and connection are uncoupled, and used via
<em>composition</em> in a dedicated class, it will be <em>Closed for
modification</em>.</p>
<p>Using the newest <code>sealed</code> directive for a class may ensure that
your <code>class</code> definition would follow this principle. If the class
method or property is <code>sealed</code>, you would not be able to change its
behavior in its inherited types, even if you are tempted to.</p>
<h3>No Singleton nor global variables</h3>
<p>About the singleton pattern, you should better always avoid it in your code.
In fact, a singleton was a C++ (and Java) hack invented to implement some kind
of global variables, hidden behind a static class definition. They were
historically introduced to support mixed mode of application-wide
initialization (mainly allocate the <code>stdio</code> objects needed to manage
the console), and were abused in business logic.</p>
<p>Once you use a singleton, or a global variable, you would miss most of the
benefit of OOP. A typical use of singleton is to register some class instances
globally for the application. You may see some framework - or some part of the
RTL - which would allow such global registration. But it would eventually void
most benefits of proper dependency injection - see <em>Dependency Inversion
Principle</em> - since you would not be able to have diverse resolution of the
same class.</p>
<p>For instance, if your database properties, or your application configuration
are stored within a singleton, or a global variable, you would certainly not be
able to use several database at once, or convert your single-user application
with its GUI into a modern multi-user AJAX application:</p>
<pre>
<strong>var</strong>
DBServer: <strong>string</strong> = 'localhost';
DBPort: integer = 1426;
<br /> UITextColor: TColor = clNavy;
UITextSize: integer = 12;
</pre>
<p>Such global variables are a smell of a broken <em>Open/Closed
Principle</em>, since your project would definitively won't be open for
extension. Using a <code>static class</code> variable (as allowed in newer
version of Delphi), is just another way of defining a global variable, just
adding the named scope of the <code>class</code> type.</p>
<p>Even if you do not define some global variable in your code, you may couple
your code from an existing global variable. For instance, defining some
variables with your <code>TMainForm = class(TForm)</code> class defined in the
IDE, then using its global <code>MainForm: TMainForm</code> variable, or the
<code>Application.MainForm</code> property, in your code. You will start to
feel not right, when the unit where your <code>TMainForm</code> is defined
would start to appear in your business code <code>uses</code> clause... just
another global variable in disguise!</p>
<p>In our framework, we tried to never use global registration, but for the
cases where it has been found safe to be implemented, e.g. when RTTI is cached,
or JSON serialization is customized for a given type. All those informations
would be orthogonal to the proper classes using them, so you may find some
global variables in the framework units, only when it is worth it. For
instance, we split <code>TSQLRecord</code>'s information into a
<code>TSQLRecordProperties</code> for the shared intangible RTTI values, and
<code>TSQLModelRecordProperties</code> instances, one per
<code>TSQLModel</code>, for all the <code>TSQLModel/TSQLRest</code> specific
settings - see <em>Several Models</em>.</p>
<h3>3. Liskov Substitution Principle</h3>
<p>Even if her name is barely unmemorable, <em>Barbara Liskov</em> is a great
computer scientist, we should better learn from. It is worth taking a look at
her presentation at <a href="https://www.youtube.com/watch?v=GDVAHA0oyJU">https://www.youtube.com/watch?v=GDVAHA0oyJU</a></p>
<p>The "<em>Liskov substitution principle</em>" states that, if
<code>TChild</code> is a subtype of <code>TParent</code>, then objects of type
<code>TParent</code> may be replaced with objects of type <code>TChild</code>
(i.e., objects of type <code>TChild</code> may be substitutes for objects of
type <code>TParent</code>) without altering any of the desirable properties of
that program (correctness, task performed, etc.).</p>
<p>The example given by <em>Barbara Liskov</em> was about stacks and queues: if
both have <code>Push</code> and <code>Pop</code> methods, they should not
inherit from a single parent type, since the storage behavior of a stack is
quite the contrary of a queue. In your program, if you start to replace a stack
by a queue, you will meet strange behaviors, for sure. According to proper
<em>top-bottom</em> design flow, both types should be uncoupled. You may
implement a <em>stack</em> class using an in-memory list for storage, or
another <em>stack</em> class using a remote SQL engine, but both would have to
behave like a <em>stack</em>, i.e. according to the last-in first-out (LIFO)
principle. On the other hand, any class implementing a <em>queue</em> type
should follow the the first-in first-out (FIFO) order, whatever kind of storage
is used.</p>
<p>In practical <em>Delphi</em> code, relying on abstractions may be
implemented by two means:</p>
<ul>
<li>Using only <code>abstract</code> parent <code>class</code> variables when
consuming objects;</li>
<li>Using <code>interface</code> variable instead of <code>class</code>
implementations.</li>
</ul>
<p>Here, we do not use inheritance for sharing implementation code, but for
defining an expected behavior. Sometimes, you may break the <em>Liskov
Substitution</em> principle in some implementation methods which would be coded
just to gather some reusable pieces of code, preparing some behavior which may
be used only by some of the subtypes. Such "internal" virtual methods of a
subtype may change the behavior of its inherited method, for the sake of
efficiency and maintainability. But with this kind of implementation
inheritance, which is closer to plumbing than designing, methods should be
declared as <code>protected</code>, and not published as part of the type
definition.<br />
By the way, this is exactly what <code>interface</code> type definitions have
to offer. You can inherit from another interface, and this kind of polymorphism
should strictly follow the <em>Liskov Substitution</em> principle. Whereas the
<code>class</code> types, implementing the interfaces, may use some protected
methods which may break the principle, for the sake of code efficiency.</p>
<p>In order to fulfill this principle, you should:</p>
<ul>
<li>Properly name (and comment) your <code>class</code> or
<code>interface</code> definition: having <code>Push</code> and
<code>Pop</code> methods may be not enough to define a contract, so in this
case type inheritance would define the expected expectation - as a consequence,
you should better stay away from "duck typing" patterns, and dynamic languages,
but rely on strong typing;</li>
<li>Use the "behavior" design pattern, when defining your objects hierarchy -
for instance, if a square may be a rectangle, a <code>TSquare</code> object is
definitively <em>not</em> a <code>TRectangle</code> object, since the behavior
of a <code>TSquare</code> object is not consistent with the behavior of a
<code>TRectangle</code> object (square width always equals its height, whereas
it is not the case for most rectangles);</li>
<li>Write your tests using abstract local variables (and this will allow test
code reuse for all children classes);</li>
<li>Follow the concept of <em>Design by Contract</em>, i.e. the Meyer's rule
defined as "<em>when redefining a routine [in a derivative], you may only
replace its precondition by a weaker one, and its postcondition by a stronger
one</em>" - use of preconditions and postconditions also enforce testing
model;</li>
<li>Separate your classes hierarchy: typically, you may consider using
separated object types for implementing persistence and object creation (this
is the common separation between <em>Factory</em> and <em>Repository</em>
patterns).</li>
</ul>
<h3>Use parent classes</h3>
<p>For our framework, it would signify that <code>TSQLRestServer</code> or
<code>TSQLRestClient</code> instances can be substituted to a
<code>TSQLRest</code> object. Most ORM methods expect a <code>TSQLRest</code>
parameter to be supplied.</p>
<p>For instance, you may write:</p>
<pre>
<strong>var</strong> anyRest: TSQLRest;
ID: TID;
rec1,rec2: TSQLMyRecord;
...
ID := anyRest.Add(rec1,true);
rec2 := TSQLMyRecord.Create(anyRest,ID);
...
</pre>
<p>And you may set any kind of actual class instance to <code>anyRest</code>,
either a local stored database engine, or a HTTP remote access:</p>
<pre>
anyRest := TSQLRestServerDB.Create(aModel,'mydatabase.db');
anyRest := TSQLHttpClient.Create('1.2.3.4','8888',aModel,false);
</pre>
<p>You may even found in the <code>dddInfraSettings.pas</code> unit a powerful
<code>TRestSettings.NewRestInstance()</code> method which is able to
instantiate the needed <code>TSQLRest</code> inherited class from a set of JSON
settings, i.e. either a <code>TSQLHttpClient</code>, or a local
<code>TSQLRestServerFullMemory</code>, or a <code>TSQLRestServerDB</code> - the
later either with a local <em>SQlite3</em> database, an external SQL engine, or
an external NoSQL/MongoDB database.</p>
<p>Your code shall refer to abstractions, not to implementations. By using only
methods and properties available at classes parent level, your code won't need
to change because of a specific implementation.</p>
<h3>I'm your father, Luke</h3>
<p>You should note that, in the <em>Liskov substitution principle</em>
definition, "parent" and "child" are no absolute. Which actual
<code>class</code> is considered as "parent" may depend on the context use.</p>
<p>Most of the time, the parent may be the highest class in the hierarchy. For
instance, in the context of a GUI application, you may use the most abstract
class to access the application data, may it be stored locally, or remotely
accessed over HTTP.</p>
<p>But when you initialize the <code>class</code> instance of a local stored
server, you may need to setup the actual data storage, e.g. the file name or
the remote SQL/NoSQL settings. In this context, you would need to access the
"child" properties, regardless of the "parent" abstract use which would take
care later on in the GUI part of the application.</p>
<p>Furthermore, in the context of data replication, server side or client side
would have diverse behavior. In fact, they may be used as master or slave
database, so in this case, you may explicitly define server or client
<code>class</code> in your code. This is what our ORM does for its master/slave
replication - see <em>Master/slave replication</em>.</p>
<p>If we come back to our bar-code scanner sample, most of your GUI code may
rely on <code>TAbstractBarcodeScanner</code> components. But in the context of
the application options, you may define the internal properties of each "child"
class - e.g. the serial or USB port name, so in this case, your new "parent"
class may be either <code>TSerialBarCodeScanner</code> or
<code>TUsbCodeScanner</code>, or even better the
<code>TSerialBarcodeConnection</code> or <code>TUsbBarcodeConnection</code>
properties, to fulfill <em>Single Responsibility principle</em>.</p>
<h3>Don't check the type at runtime</h3>
<p>Some patterns shall never appear in your code. Otherwise, code refactoring
should be done as soon as possible, to let your project be maintainable in the
future.</p>
<p>Statements like the following are to be avoided, in either the parents or
the child's methods:</p>
<pre>
<strong>procedure</strong> TAbstractBarcodeScanner.SomeMethod;
<strong>begin</strong>
<strong>if</strong> self <strong>is</strong> TSerialBarcodeScanner <strong>then</strong>
<strong>begin</strong>
....
<strong>end</strong>
<strong>else</strong>
<strong>if</strong> self <strong>is</strong> TUsbBarcodeScanner <strong>then</strong>
...
</pre>
<p>Or, in its disguised variation, using an enumerated item:</p>
<pre>
<strong>case</strong> fProtocol.MeanOfCommunication <strong>of</strong>
meanSerial: <strong>begin</strong>
...
<strong>end</strong>;
meantUsb:
...
</pre>
<p>This later piece of code does not check <code>self</code>, but the
<code>fProtocol</code> protected field. So even if you try to implement the
<em>Single Responsibility principle</em>, you may still be able to break
<em>Liskov Substitution</em>!</p>
<p>Note that both patterns will eventually break the <em>Single Responsibility
principle</em>: each behavior shall be defined in its own child
<code>class</code> methods. As the <em>Open/Close principle</em> would also be
broken, since the class won't be open for extension, without touching the
parent class, and modify the nested <code>if self is T* then ...</code> or
<code>case fProtocol.* of ...</code> expressions.</p>
<h3>Partially abstract classes</h3>
<p>Another code smell may appear when you define a method which will stay
<code>abstract</code> for some children, instantiated in the project. It would
imply that some of the parent <code>class</code> behavior is not implemented at
this particular hierarchy level. So you would not be able to use all the
parent's methods, as would be expected by the <em>Liskov Substitution
principle</em>.<br />
Note that the compiler will complain for it, hinting that you are creating a
class with abstract methods. Never ignore such hints - which may benefit for
being handled as errors at compilation time, IMHO. The (in)famous
"<code>Abstract Error</code>" error dialog, which may appear at runtime, would
reflect of this bad code implementation.</p>
<p>A more subtle violation of <em>Liskov</em> may appear if you break the
expectation of the parent class. The following code, which emulates a bar-code
reader peripheral by sending the frame by email for debugging purpose (why
not?), clearly fails the <em>Design by Contract</em> approach:</p>
<pre>
TEMailEmulatedBarcodeProtocol = <strong>class</strong>(TAbstractBarcodeProtocol)
<strong>protected</strong>
<strong>function</strong> ReadFrame: TProtocolFrame; <strong>override</strong>;
<strong>procedure</strong> WriteFrame(<strong>const</strong> aFrame: TProtocolFrame); <strong>override</strong>;
...
<strong>function</strong> TEMailEmulatedBarcodeProtocol.ReadFrame: TProtocolFrame;
<strong>begin</strong>
<strong>raise</strong> EBarcodeException.CreateUTF8('%.ReadFrame is not implemented!',[self]);
<strong>end</strong>;
<br /><strong>procedure</strong> TEMailEmulatedBarcodeProtocol.WriteFrame(<strong>const</strong> aFrame: TProtocolFrame);
<strong>begin</strong>
SendEmail(fEmailNotificationAddress,aFrame.AsString);
<strong>end</strong>;
</pre>
<p>We expected this class to fully implement the
<code>TAbstractBarcodeProtocol</code> contract, whereas calling
<code>TEMailEmulatedBarcodeProtocol.ReadFrame</code> would not be able to read
any data frame, but would raise an exception. So we can not use this
<code>TEMailEmulatedBarcodeProtocol class</code> as replacement to any other
<code>TAbstractBarcodeProtocol</code> class, otherwise it would fail at
runtime.<br />
A correct implementation may perhaps to define a
<code>TFakeBarcodeProtocol</code> class, implementing all the parent methods
via a set of events or some text-based scenario, so that it would behave just
like a correct <code>TAbstractBarcodeProtocol</code> class, in the full extend
of its expectations.</p>
<h3>Messing units dependencies</h3>
<p>Last but not least, if you need to explicitly add child classes units to the
parent class unit <code>uses</code> clause, it looks like if you just broke the
<em>Liskov Substitution principle</em>.</p>
<pre>
<strong>unit</strong> AbstractBarcodeScanner;
<br /><strong>uses</strong>
SysUtils,
Classes,
<span style="background-color:yellow;">SerialBarcodeScanner; <em>// Liskov says: "it smells"!</em></span>
<span style="background-color:yellow;">UsbBarcodeScanner; <em>// Liskov says: "it smells"!</em></span>
...
</pre>
<p>If your code is like this, you would have to remove the reference to the
inherited classes, for sure.</p>
<p>Even a dependency to one of the low-level implementation detail is to be
avoided:</p>
<pre>
<strong>unit</strong> AbstractBarcodeScanner;
<br /><strong>uses</strong>
<span style="background-color:yellow;">Windows,</span>
SysUtils,
Classes,
<span style="background-color:yellow;">ComPort;</span>
...
</pre>
<p>Your abstract parent <code>class</code> should <strong>not</strong> be
coupled to a particular <em>Operating System</em>, or a mean of communication,
which may not be needed. Why would you add a dependency to raw <em>RS-232</em>
communication protocol, which is very likely to be deprecated.</p>
<p>One way of getting rid of this dependency is to define some abstract types
(e.g. enumerations or simple structures like <code>record</code>), which would
then be translated into the final types as expected by the
<code>ComPort.pas</code> or <code>Windows.pas</code> units. Consider putting
all the child classes dependencies at <code>constructor</code> level, and/or
use <code>class</code> composition via the <em>Single Responsibility
principle</em> so that the parent <code>class</code> definition would not be
polluted by implementation details of its children.</p>
<p>You my also use a <em>registration list</em>, maintained by the parent unit,
which may be able to register the classes implementing a particular behavior at
runtime. Thanks to <em>Liskov</em>, you would be able to <em>substitute</em>
any parent class by any of its inherited implementation, so defining the types
at runtime only should not be an issue.</p>
<h3>Practical advantages</h3>
<p>The main advantages of this coding pattern are the following:</p>
<ul>
<li>Thanks to this principle, you will be able to <em>stub</em> or
<em>mock</em> an interface or a <code>class</code> - see <em>Interfaces in
practice: dependency injection, stubs and mocks</em> - e.g. uncouple your
object persistence to the actual database it runs on: this principle is
therefore mandatory for implementing unitary testing to your project;</li>
<li>Furthermore, testing would be available not only at isolation level
(testing each child class), but also at abstracted level, i.e. from the client
point of view - you can have implementation which behave correctly when tested
individually, but which failed when tested at higher level if the
<em>Liskov</em> principle was broken;</li>
<li>As we have seen, if this principle is violated, the other principles are
very likely to be also broken - e.g. the parent class would need to be modified
whenever a new derivative of the base class is defined (violation of the
<em>Open/Close</em> principle), or your <code>class</code> types may implement
more than one behavior at a time (violation of the <em>Single
Responsibility</em> principle);</li>
<li>Code re-usability is enhanced by method re-usability: a method defined at a
parent level does not require to be implemented for each child.</li>
</ul>
<p>The SOA and ORM concepts, as implemented by our framework, try to be
compliant with the <em>Liskov substitution principle</em>. It is true at
<code>class</code> level for the ORM, but a more direct <em>Design by
Contract</em> implementation pattern is also available, since the whole SOA
stack involves a wider usage of <code>interfaces</code> in your projects.</p>
<h3>4. Interface Segregation Principle</h3>
<p>This principle states that once an interface has become too 'fat' it shall
be split into smaller and more specific interfaces so that any clients of the
interface will only know about the methods that pertain to them. In a nutshell,
no client should be forced to depend on methods it does not use.</p>
<p>As a result, it will help a system stay decoupled and thus easier to
re-factor, change, and redeploy.</p>
<h3>Consequence of the other principles</h3>
<p><em>Interface segregation</em> should first appear at <code>class</code>
level. Following the <em>Single Responsibility</em> principle, you are very
likely to define several smaller classes, with a small extent of methods. Then
use dedicated types of class, relying on composition to expose its own higher
level set of methods.</p>
<p>The bar-code class hierarchy illustrates this concept. Each
<code>T*BarcodeProtocol</code> and <code>T*BarcodeConnection</code> class will
have its own set of methods, dedicated either to protocol handling, or data
transmission. Then the <code>T*BarCodeScanner</code> classes will
<em>compose</em> those smaller classes into a new class, with a single event
handler:</p>
<pre>
<strong>type</strong>
TOnBarcodeScanned = <strong>procedure</strong>(Sender: TAbstractBarcodeScanner; <strong>const</strong> Barcode: <strong>string</strong>) <strong>of object</strong>;
<br /> TAbstractBarcodeScanner = <strong>class</strong>(TComponent)
...
<span style="background-color:yellow;"><strong>property</strong> OnBarcodeScanned: TOnBarcodeScanned <strong>read</strong> fOnBarcodeScanned <strong>write</strong> fOnBarcodeScanned;</span>
...
</pre>
<p>This single <code>OnBarcodeScanned</code> event will be the published
property of the component. Both protocol and connection details would be hidden
within the internal classes. The final application would use this event, and
react as expected, without actually knowing anything about the implementation
details.</p>
<h3>Using interfaces</h3>
<p>The SOA part of the framework allows direct use of <code>interface</code>
types to implement services. This great Client-Server SOA implementation
pattern - see <em>Server side Services</em> - helps decoupling all services to
individual small methods. In this case also, the stateless used design will
also reduce the use of 'fat' session-related processes: an object life time can
be safely driven by the <code>interface</code> scope.</p>
<p>By defining <em>Delphi</em> <code>interface</code> instead of plain
<code>class</code>, it helps creating small and business-specific contracts,
which can be executed on both client and server side, with the same exact
code.</p>
<p>Since the framework makes interface consumption and publication very easy,
you won't be afraid of exposing your implementation classes as small pertinent
interface.<br />
For instance, if you want to publish a third-party API, you may consider
publishing dedicated interfaces, each depending on every API consumer
expectations. So your main implementation logic won't be polluted by how the
API is consumed, and, as correlative, the published API may be closer to each
particular client needs, without been polluted by the other client needs. DDD
would definitively benefit for <em>Interface Segregation</em>, since this
principle is the golden path to avoid <em>domain leaking</em> - see <em>DTO and
Events to avoid domain leaking</em>.</p>
<h3>5. Dependency Inversion Principle</h3>
<p>Another form of decoupling is to invert the dependency between high and low
level of a software design:</p>
<ul>
<li>High-level modules should not depend on low-level modules. Both should
depend on abstractions;</li>
<li>Abstractions should not depend upon details. Details should depend upon
abstractions.</li>
</ul>
<p>The goal of the <em>dependency inversion principle</em> is to decouple
high-level components from low-level components such that reuse with different
low-level component implementations becomes possible. A simple implementation
pattern could be to use only interfaces owned by, and existing only with the
high-level component package.</p>
<p>This principle results in <em>Inversion Of Control</em> (aka IoC): since you
rely on the abstractions, and try not to depend upon concretions (i.e. on
implementation details), you should first concern by defining your
interfaces.</p>
<h3>Upside Down Development</h3>
<p>In conventional application architecture, lower-level components are
designed to be consumed by higher-level components which enable increasingly
complex systems to be built. This design limits the reuse opportunities of the
higher-level components, and certainly breaks the <em>Liskov substitution
principle</em>.</p>
<p>For our bar-code reader sample, we may be tempted to start from the final
<code>TSerialBarcodeScanner</code> we need in our application. We were asked by
our project leader to allow bar-code scanning in our flagship application, and
the extent of the development has been reduced to support a single model of
device, in RS-232 mode - this may be the device already owned by our end
customer.</p>
<p>This particular customer may have found some RS-232 bar-code relics from the
90s in its closets, but, as an experience programmer, you know that the next
step would be to support USB, in a very close future. All this bar-code reading
stuff will be marketized by your company, so it is very likely than another
customer would very soon ask for using its own brand new bar-code scanners...
which would support only USB.</p>
<p>So you would modelize your classes as above.<br />
Even if the <code>TUsbBarCodeScanner</code> - and its correlative
<code>TUsbBarcodeConnection class</code> - is not written, nor tested (you do
not even have an actual USB bar-code scanner to do proper testing yet!), you
are prepared for it.</p>
<p>When you would eventually add USB support, the UI part of the application
won't have to be touched. Just implementing your new inherited class,
leveraging all previous coding. Following <em>Dependency Inversion</em> from
the beginning would definitively save your time. Even in an <em>Agile</em> kind
of process - where "<em>Responding to change</em>" is most valuable - the small
amount of work on implementing first from the abstraction with the initial
implementation would be very beneficial.</p>
<p>In fact, this <em>Dependency Inversion</em> principle is a prerequisite for
proper <em>Test-Driven</em> Design. Following this TDD pattern, you first write
your test, then fail your test, then write the implementation. In order to
write the test, you need the abstracted interface of the feature to be
available. So you would start from the abstraction, then write the
concretion.</p>
<h3>Injection patterns</h3>
<p>In other languages (like Java or .Net), various patterns such as
<em>Plug-in, Service Locator</em>, or <em>Dependency Injection</em> are then
employed to facilitate the run-time provisioning of the chosen low-level
component implementation to the high-level component.</p>
<p>Our Client-Server architecture facilitates this decoupling pattern for its
ORM part, and allows the use of native <em>Delphi</em> <code>interface</code>
to call services from an abstract factory, for its SOA part.</p>
<p>A set of dedicated classes, defined in <code>mORMot.pas</code>, allows to
leverage <code>IoC</code>: see e.g. <code>TInjectableObject</code>,
<code>TInterfaceResolver</code>,
<code>TInterfaceResolverForSingleInterface</code> and
<code>TInterfaceResolverInjected</code>, which may be used in conjunction with
<code>TInterfaceStub</code> or <code>TServiceContainer</code> high-level
mocking and SOA features of the framework - see <em>Interfaces in practice:
dependency injection, stubs and mocks</em> and <em>Client-Server services via
interfaces</em>.</p>
<h3>Feedback needed</h3>
<p>Feel free to <a href="http://synopse.info/forum/viewtopic.php?id=526">comment this article in our
forum</a>, as usual!</p>