Synopse Open Source - Tag - securitymORMot MVC / SOA / ORM and friends2024-02-02T17:08:25+00:00urn:md5:cc547126eb580a9adbec2349d7c65274DotclearNative X.509, RSA and HSM Supporturn:md5:6404121d596d3c5cee045e29cf93e02b2023-12-09T11:01:00+00:002023-12-10T18:13:06+00:00Arnaud BouchezmORMot FrameworkAsymmetricCrossPlatformCSPRNGDelphiECCed25519forensicFPCFreePascalGoodPracticeinterfacemORMotmORMot2OpenSourceOpenSSLperformancePKCS11RSAsecurityX509<p>Today, almost all computer security relies on asymmetric cryptography and X.509 certificates as file or hardware modules.<br />
And the RSA algorithm is still used to sign the vast majority of those certificates. Even if there are better options (like ECC-256), RSA-2048 seems the actual standard, at least still allowed for a few years.</p>
<p><img src="https://blog.synopse.info?post/public/blog/mormotSecurity.jpg" alt="" /></p>
<p>So we added pure pascal RSA cryptography and X.509 certificates support in <em>mORMot</em>.<br />
Last but not least, we also added Hardware Security Modules support via the PKCS#11 standard.<br />
Until now, we were mostly relying on OpenSSL, but a native embedded solution would be smaller in code size, better for reducing dependencies, and easier to work with (especially for HSM). The main idea is to offer only safe algorithms and methods, so that you can write reliable software, even if you are no cryptographic expert. <img src="https://blog.synopse.info?pf=smile.svg" alt=":)" class="smiley" /></p> <h4>Rivest-Shamir-Adleman (RSA) Public-Key Cryptography</h4>
<p>The RSA public-key algorithm was designed back in 1977, and is still the most widely used. In order to fully implement it, we need to generate new key pairs (public and private keys), then sign and verify (or encrypt or decrypt) data with the key. For instance, a private key is kept secret, and used for an Authority to sign a certificate, and the public key is published, and able to verify a certificate. It is based on large prime numbers, so we needed to develop a Big Integer library, which is not part of Delphi or FPC RTL.</p>
<p><img src="https://blog.synopse.info?post/public/blog/RSAalgo.png" alt="" /></p>
<p>Here as some notes about our implementation in <a href="https://github.com/synopse/mORMot2/blob/master/src/crypt/mormot.crypt.rsa.pas">mormot.crypt.rsa.pas</a>:</p>
<ul>
<li>new pure pascal OOP design of BigInt computation optimized for RSA process;</li>
<li>dedicated x86_64/i386 asm for core computation routines (noticeable speedup);</li>
<li>use half-registers (HalfUInt) for efficient computation on all CPUs (arm and aarch64 numbers are good);</li>
<li>slower than OpenSSL, but likely to be the fastest FPC or Delphi native RSA library, thanks to our optimized asm: for instance, you can generate a new RSA-2048 keypair in less than a second;</li>
<li>internal garbage collection of BigInt instances, to minimize heap pressure during computation, and ensure all values are wiped once used during the process - as proven anti-forensic measure;</li>
<li>includes FIPS-level RSA keypair validation and generation, using a safe random source, with efficient prime number detection, and minimal code size;</li>
<li>features both RSASSA-PKCS1-v1_5 and RSASSA-PSS signature schemes;</li>
<li>started as a fcl-hash fork, but full rewrite inspired by Mbed TLS source because this initial code is slow and incomplete;</li>
<li>references: we followed <a href="https://github.com/Mbed-TLS/mbedtls">the Mbded TLS</a> implementation (which is much easier to follow than OpenSSL), and the well known <a href="https://cacr.uwaterloo.ca/hac/about/chap4.pdf">Handbook of Applied Cryptography (HAC)</a> recommendations;</li>
<li>includes full coverage of unit tests to avoid any regression, validated against the OpenSSL library as audited reference;</li>
<li>this unit will register as <code>Asym</code> 'RS256','RS384','RS512' algorithms (if not overridden by the faster <code>mormot.crypt.openssl</code>), keeping 'RS256-int' and 'PS256-int' available to use our unit;</li>
<li>as used by <code>mormot.crypt.x509</code> (see below) to handle RSA signatures of its X.509 Certificates.</li>
</ul>
<p>For instance, if you want to access a <code>TCryptAsym</code> digital signature instance with RSA-2048 and SHA-256 hashing, you can just use the <code>CryptAsym</code> global variable with <code>caaRS256</code> algorithm as factory.<br />
If you need just public/private key support, you can use <code>CryptPublicKey</code> or <code>CryptPrivateKey</code> factories with <code>ckaRsa</code> algorithm.</p>
<p>About RSA security:</p>
<ul>
<li>RSA-512 or RSA-1024 are considered unsafe and should not be used.</li>
<li>RSA-2048 confers 112-bit of security, and is the usual choice today when this algorithm is to be used.</li>
<li>RSA-3072 could confer 128-bit of security, at the expense of being slower and 50% bigger - so switching to ECC-256 may be a better option, for the same level of security.</li>
<li>RSA-4096 is not worth it in respect to RSA-3072, and RSA-7680 is very big and slow, and only gives 192-bit of security, so should be avoided.</li>
</ul>
<p>Anyway, our library is able to support all those key sizes, up to RSA-7680 is you really need it.<br />
See <a href="https://stackoverflow.com/a/589850/458259">this SO response</a> as reference about RSA keysizes.</p>
<h4>X.509 Certificates</h4>
<p>As we wrote in introduction, X.509 certificates are the base of most computer security.<br />
The whole TLS/HTTPS stack makes use of it, and the whole Internet would collapse without it.</p>
<p>We developed our <a href="https://github.com/synopse/mORMot2/blob/master/src/crypt/mormot.crypt.x509.pas">mormot.crypt.x509.pas</a> unit from scratch, featuring:</p>
<ul>
<li>X.509 Certificates Fields Logic (e.g. X.501 Type Names);</li>
<li>X.509 Certificates and Certificate Signing Request (CSR);</li>
<li>X509 Certificate Revocation List (CRL);</li>
<li>X509 Private Key Infrastructure (PKI);</li>
<li>Registration of our X.509 Engine to the <code>TCryptCert</code>/<code>TCryptStore</code> Factories.</li>
</ul>
<p><img src="https://blog.synopse.info?post/public/blog/X509certificate.png" alt="" /></p>
<p>The raw binary encoding is using the (weird) ASN.1 syntax, which is now implemented as part of the <a href="https://github.com/synopse/mORMot2/blob/master/src/crypt/mormot.crypt.secure.pas">mormot.crypt.secure.pas</a> unit.<br />
We followed the RFC 5280 specifications, and mapped latest X.509 Certificates / CSR / CRL extensions, with some low-level but very readable pascal code using classes, records and enumerates. It features perfect compatibility with our <code>ICryptCert</code> high-level interface wrappers, ready to be used in a very convenient way. We support all basic functions, but also advanced features like open/sealing or text peer information in a human readable format.<br />
When using our unit, your end-user code should not be lost within the complex details and notions of the X.509 format (like OIDs, versions or extensions), but use high-level pascal code, with no possibility to use a weak or invalid configuration.</p>
<p>Of course, it can support not only our new RSA keys, but also ECC-256 as implemented by our native <a href="https://github.com/synopse/mORMot2/blob/master/src/crypt/mormot.crypt.ecc.pas">mormot.crypt.ecc.pas</a>, or any other algorithm, e.g. available from OpenSSL.</p>
<h4>X.509 Private Key Infrastructure (PKI)</h4>
<p>Our unit features a full Private Key Infrastructure (PKI) implementation.<br />
In fact, X.509 certificates are as weak as the PKI they are used on. You can have strong certificates, but a weak verification pattern. In end-user applications, it is typical to see all the security being lost by a poor (e.g. naive) implementation of the keys interaction.</p>
<p><img src="https://blog.synopse.info?post/public/blog/PKI.png" alt="" /></p>
<p>This is why our unit publishes a 'x509-pki' <code>ICryptStore </code> as a full featured PKI:</p>
<ul>
<li>using our <code>TX509</code> and <code>TX509Crl</code> classes for actual certificates process;</li>
<li>clean verification of the chain of trust, with customized depth and proper root Certificate Authority (CA) support, following the RFC 5280 section 6 requirements of a clean "Certification Path Validation";</li>
<li>maintaining a cache of <code>ICryptCert</code> instances, which makes a huge performance benefit in the context of a PKI (e.g. you don't need to parse the X.509 binary, or verify the chain of trust each time).</li>
</ul>
<p>We tried to make performance and usability in the highest possible standards, to let you focus on your business logic, and keep the hard cryptography work done in the <em>mORMot</em> library code.</p>
<h4>Hardware Security Modules (HSM) via PKCS#11</h4>
<p>The PKCS#11 standard is a way to define some software access to Hardware Security Modules, via a set of defined API calls.<br />
We just published the <a href="https://github.com/synopse/mORMot2/blob/master/src/crypt/mormot.crypt.pkcs11.pas">mormot.crypt.pkcs11.pas</a> unit to interface those devices with the other <em>mORMot</em> PKI.</p>
<p><img src="https://blog.synopse.info?post/public/blog/HSM.png" alt="" /></p>
<p>Once you have loaded the library of your actual hardware (typically a <code>.dll</code> or <code>.so</code>) using a <code>TCryptCertAlgoPkcs11</code> instance, you can see all stored certificates and keys, as high-level regular <code>ICryptCert</code> instances, and sign or verify any kind of data (some binary or some other certificates), using the private key safely stored on in the hardware device.<br />
This is usually slower than a pure software verification, but it is much safer, because the private key is sealed within the hardware token, and never leave it. So it can't be intercepted and stolen.</p>
<h4>You are Welcome!</h4>
<p>With those <em>mORMot</em> cryptography units, you now have anything at hand to use standard and proven public-key cryptography in your applications, on both Delphi or FPC, with no external dll deployment issue, and minimal code size increase.<br />
We can thank a lot <a href="https://www.tranquil.it/en">my employer</a> for needing those nice features, therefore letting me work on them.<br />
Open Source rocks! :)</p>End Of Live OpenSSL 1.1 vs Slow OpenSSL 3.0urn:md5:f20e1a3a1c96e8f65f1fc8ef5a04498c2023-09-08T11:59:00+01:002023-09-08T14:13:49+01:00Arnaud BouchezOpen SourceAESCertificatesDelphiFreePascalGoodPracticeLateBindingLazarusMaxOSXmORMotmORMot2OpenSSLperformancesecuritySource<p>You may have noticed that the OpenSSL 1.1.1 series will reach End of Life (EOL) next Monday...<br />
Most sensible options are to switch to 3.0 or 3.1 as soon as possible.</p>
<p><img src="https://blog.synopse.info?post/public/blog/mormotSecurity.jpg" alt="mormotSecurity.jpg, Sep 2023" /></p>
<p>Of course, our <a href="https://github.com/synopse/mORMot2/blob/master/src/lib/mormot.lib.openssl11.pas"><em>mORMot 2</em> OpenSSL unit</a> runs on 1.1 and 3.x branches, and self-adapt at runtime to the various API incompatibilities existing between each branch.<br />
But we also discovered that switching to OpenSSL 3.0 could led into big performance regressions... so which version do you need to use?</p> <h4>OpenSSL 1.1 End Of Live</h4>
<p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/6/6a/OpenSSL_logo.svg/320px-OpenSSL_logo.svg.png" alt="OpenSSL logo" /></p>
<p>The well known and well established OpenSSL 1.1.1 series will reach End of Life (EOL) on 11th September 2023. So next Monday! <img src="https://blog.synopse.info?pf=sad.svg" alt=":(" class="smiley" /> <br />
Users of OpenSSL 1.1.1 should consider their options and plan any actions they might need to take.</p>
<p>Note that Indy users are <a href="https://github.com/IndySockets/Indy/issues/183">still stuck to the OpenSSL 1.0 branch</a>, even 1.1 is not yet officially supported. Some <a href="https://github.com/IndySockets/Indy/pull/299">alternate IO handlers</a> are able to use newest releases - to some extend.<br />
Indy users should rather move to a better supported library, like our little <em>mORMot</em>.</p>
<p>Also note that there are some API incompatibilities between 1.1 and 3.x. Functions have been renamed, or even removed; new context constructors appeared; some parameters types even changed!<br />
Our unit tries to address all those problems at runtime, and is tested against several version of the OpenSSL library, to ensure you do not have to worry about those low-level issues.</p>
<h4>OpenSSL 3.x Benefits</h4>
<p>With OpenSSL 3.0, the developpers did a huge refactoring of the library internals.<br />
To be fair, the 1.x source code of OpenSSL was kind of a mess, and difficult to maintain. The biggest IT companies did even made their own forks or switched to other libraries. The best known is <a href="https://boringssl.googlesource.com/boringssl/">BoringSSL</a>, maintained by Google, and used e.g. in Chrome and Android.<br />
So it was time for a refactoring, especially for a library as critical as OpenSSL for so many projects.</p>
<p>With the new 3.x branch, a lot of low-level API functions have been deprecated.<br />
In practice, you don't have direct access any more to the internal structures of the library, and should now always use the high-level API to access a context property, or execute the processing methods. For instance, the low-level <code>AES_encrypt</code> function is not available any more: from now on, you need to use the high-level <code>EVP_Encrypt*</code> API.<br />
The official <a href="https://www.openssl.org/docs/man3.0/man7/migration_guide.html">Migration Guide page</a> is clearly huge, and worth reading if you want to prepare yourself to the upcoming years with OpenSSL.</p>
<h4>OpenSSL 3.0 Performance Regression</h4>
<p>The 3.0 branch new code may seem more beautiful and more maintainable, but it had its drawbacks. Newer is not always better.<br />
Most users of this new release <a href="https://github.com/openssl/openssl/issues/17064">observed a huge performance regression</a> when switching from 1.x to 3.0. It affected a lot of projects, from various languages, even script languages which were not already shining about performance. Time regression from 3x up to 10x were reported. On our side, X509 certificates manipulation was really slower than before - the worse being about X509 stores.</p>
<p>Some slowdown were expected and documented (like RSA key generation, which now uses 64 rounds). But the regression was much deeper.<br />
The culprit seems not to be the core cryptographic code, like AES buffer encoding (which asm claims to have been optimized even further on 3.x branch), but the OpenSSL context structures themselves. They were rewritten for future maintainability, but not focusing on their actual performance.</p>
<h4>OpenSSL 3.1 Numbers</h4>
<p>The 3.1 branch claims to have addressed most of these problems.</p>
<p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/ea/The_Tortoise_and_the_Hare_-_Project_Gutenberg_etext_19994.jpg/334px-The_Tortoise_and_the_Hare_-_Project_Gutenberg_etext_19994.jpg" alt="The Tortoise and the Hare" /></p>
<p>To be sure, we run the <em>mORMot</em> cryptographic regression tests with several versions of OpenSSL. And in fact, OpenSSL 3.1 was much faster than OpenSSL 3.0, but still behind OpenSSL 1.1.<br />
Here are the numbers we observed for the whole <code>TTestCoreCrypto</code> method execution, executed on Win32:</p>
<ul>
<li>OpenSSL 1.1 = 15 sec</li>
<li>OpenSSL 3.0 = 33 sec</li>
<li>OpenSSL 3.1 = 18 sec</li>
</ul>
<p>There are several aspects to emphasize:</p>
<ul>
<li>Those tests runs also <em>mORMot</em> engine cryptography, so you don't only test OpenSSL: the "pure mORMot" tests take around 4.5 seconds in the above numbers;</li>
<li>Any serious project should consider compiling on Win64, and running a server on a x86_64 Linux - on this platform, the regression does exist, but only slightly better;</li>
<li>The slowdown was less affecting <code>TTestCoreCrypto.Benchmark</code> (i.e. raw buffer encryption) than <code>TTestCoreCrypto.Catalog</code> (i.e. certificates process);</li>
<li>Our tests were mono-threaded, and worse slow down were reported on heavily threaded process (up to x10).</li>
</ul>
<p>Within the <em>mORMot</em> OpenSSL wrapper, we try to cache as many context as possible. For instance, we don't lookup the OpenSSL algorithm by name for each call, but we cache it at runtime to avoid any slowdown.<br />
But it seems not enough with OpenSSL 3.0, which may affect your application performance.</p>
<h4>To Support or Not Support</h4>
<p>So OpenSSL 3.1 seems to be the way to go.</p>
<p>On Linux (or other POSIX systems), you are likely to use the library shipped with the system.<br />
So you would not worry about which version to use. And, sadly, it is very likely that your distribution provides OpenSSL 3.0 and not OpenSSL 3.1.</p>
<p>On Windows (or Mac), you could (should?) use your "own" dll/so files, so you have to take into account the support level of the library.<br /></p>
<ul>
<li>OpenSSL 3.0 is a Long Term Support (LTS) version, which will be maintained until 7th September 2026.<br /></li>
<li>OpenSSL 3.1 will be supported only until 14th March 2025.</li>
</ul>
<p>These support end dates could appear counter-intuitive, but this is an usual way in Open Source projects, the best known being perhaps <a href="https://ubuntu.com/blog/what-is-an-ubuntu-lts-release">Ubuntu LTS versions</a>.<br />
For more information about OpenSSL support lifetime, look at the <a href="https://www.openssl.org/source/">official OpenSSL Downloads page</a>.</p>
<p>So, for most projects, especially on Windows where you are likely to publish OpenSSL dll with your own executable, switching to OpenSSL 3.1 is likely to be the way to go.<br />
If you need to gather some security certification for your product, you may consider using OpenSSL 3.0 LTS version, which may help your certification remain active for a longer period.</p>
<p>Any feedback is <a href="https://synopse.info/forum/viewtopic.php?id=6697">welcome on our forum</a>, as usual!</p>New Async HTTP/WebSocket Server on mORMot 2urn:md5:b5d5687573d19a81f6d6dadfbc68461d2022-05-21T13:35:00+01:002022-05-21T18:05:45+01:00Arnaud BouchezmORMot FrameworkDelphiFPCfpcx64mmHTTPhttp.sysHTTPSLinuxmORMotmORMot2multithreadperformanceRESTRestsecuritySOA<p>The HTTP server is one main part of any SOA/REST service, by design.<br />
It is the main entry point of all incoming requests. So it should better be stable and efficient. And should be able to scale in the future, if needed.</p>
<p><img src="https://blog.synopse.info?post/public/blog/server.jpg" alt="" /></p>
<p>There have always been several HTTP servers in <em>mORMot</em>. You can use the HTTP server class you need.<br />
In <em>mORMot</em> 2, we added two new server classes, one for publishing over HTTP, another able to upgrade to WebSockets. The main difference is that they are fully event-driven, so their thread pool is able to scale with thousands of concurrent connections, with a fixed number of threads. They are a response to the limitations of our previous socket server.</p> <h4>HTTP Is Not REST</h4>
<p>In <em>mORMot</em>, the HTTP server does not match the REST server. They are two concepts, with diverse units, classes and even... source code folders.<br />
The HTTP server can publish its own process, using a callback, or can leverage one or several REST servers. Or the REST server could be with no communication at all, e.g. be run in the service process within the current thread, executing ORM or SOA requests without any HTTP or WebSockets involved. No difference in the user code: just some interface methods to call, and they will return their answer, whatever it is over HTTP, locally or remotely, on the server side or the client side, in a thread pool or in-process... pascal code just runs its magic for you.</p>
<p>For instance, take a look at <a href="https://github.com/synopse/mORMot2/blob/master/ex/http-server-raw/httpServerRaw.dpr">httpServerRaw</a> as a low-level HTTP server sample, using a callback for every incoming request.<br />
Here you don't have any automatic routing: you just parse the input URI, then return the proper HTTP response, with its status code, headers and body.<br />
Those low-level HTTP servers are implemented in the <em>server</em> units of folder <a href="https://github.com/synopse/mORMot2/tree/master/src/net">src/net</a> - as we will detail below.</p>
<p>Then, the REST process is abstracted from HTTP. Even if both HTTP and REST historically share the <a href="https://en.wikipedia.org/wiki/Roy_Fielding">same father/author/initiator</a>, you could have a REST approach without HTTP. For instance, you could use WebSockets, or direct in-process call.<br />
So in <em>mORMot</em>, the REST process is implemented in the <em>server</em> units of folder <a href="https://github.com/synopse/mORMot2/tree/master/src/rest">src/rest</a>, abstracted from any communication protocol. It does not know nor assume anything about TCP, WebSockets or TLS. Just about URI, headers and body texts, and follow the routing as defined.</p>
<p>Two folders, two uncoupled feature sets. Perhaps a bit confusing when you discover it first. But for the best maintainability and code design.</p>
<h4>Several Servers To Rule Them All</h4>
<p><em>mORMot</em> 2 adopted all HTTP server classes from <em>mORMot</em> 1 source code. Then include some new "asynchronous" servers.<br />
They all inherit from a <code>THttpServerGeneric</code> parent class, so you can follow the Liskow Substitution Principle, and change the class at runtime or compilation, as needed, without altering your actual logic.</p>
<p>HTTP servers are implemented in several units:</p>
<ul>
<li><a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.server.pas"><em>mORMot</em>.net.server.pas</a> offers the <code>THttpServerSocket</code>/<code>THttpServer</code> HTTP/1.1 server, the <code>THttpApiServer</code> HTTP/1.1 server over Windows http.sys module, and <code>THttpApiWebSocketServer</code> over Windows http.sys module;</li>
<li><a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.ws.server.pas"><em>mORMot</em>.net.ws.server.pas</a> offers the <code>TWebSocketServerRest</code> server, which uses WebSockets as mean of transmission, but enable a REST-like blocking request/answer protocol on top of it, with optional bi-directional notifications, using a one-thread-per-connection server;</li>
<li><a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.async.pas"><em>mORMot</em>.net.async.pas</a> offers the new <code>THttpAsyncServer</code> event-driven HTTP server;</li>
<li><a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.ws.async.pas"><em>mORMot</em>.net.ws.async.pas</a> offers the new <code>TWebSocketAsyncServerRest</code> server, which uses WebSockets as mean of transmission, but enable a REST-like blocking request/answer protocol on top of it, with optional bi-directional notifications, using an event-driven server.</li>
</ul>
<p>On Windows, the <a href="https://docs.microsoft.com/en-us/iis/get-started/introduction-to-iis/introduction-to-iis-architecture#hypertext-transfer-protocol-stack-httpsys">http.sys</a> module gives you very good stability, and uses the same Windows-centric way of publishing servers as used by IIS and DotNet. You could even share the same port between several services, if needed.</p>
<p>Our socket-based servers are cross-plaform, and compile and run on both Windows and POSIX (Linux, BSD, MacOS). They use a thread pool for HTTP/1.0 short living requests, and one thread per connection on HTTP/1.1. So they are meant to be used behind a reverse proxy like nginx, which could transmit over HTTP/1.0 with <em>mORMot</em>, but keep efficient HTTP/1.1 or HTTP/2.0 to communicate with the clients.</p>
<p>Both <a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.async.pas"><em>mORMot</em>.net.async.pas</a> and <a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.ws.async.pas"><em>mORMot</em>.net.ws.async.pas</a> are new to <em>mORMot</em> 2. They use an event-driven model, i.e. the opened connections are tracked using a fast API (like epoll on Linux), and the thread pool is used only when there is actually new data pending.</p>
<h4>Events Forever</h4>
<p>Asynchronous socket access, and event loops are the key for best server scalability. In respect to our regular <code>THttpServerSocket</code> class which uses one thread per HTTP/1.1 or WebSockets connection, our asynchronous classes (e.g. <code>THttpAsyncServer</code>) can have thousands of concurrent clients, with minimal CPU and RAM resource consumption.</p>
<p>Here is a typical event-driven socket access:</p>
<p><img src="https://blog.synopse.info?post/public/blog/epoll.png" alt="" /></p>
<p>In <em>mORMot</em> 2 network core, i.e. in unit <a href="https://github.com/synopse/mORMot2/blob/master/src/net/mormot.net.sock.pas"><em>mORMot</em>.net.sock.pas</a>, we define an abstract event-driven class:</p>
<pre>
/// implements efficient polling of multiple sockets
// - will maintain a pool of TPollSocketAbstract instances, to monitor
// incoming data or outgoing availability for a set of active connections
// - call Subscribe/Unsubscribe to setup the monitored sockets
// - call GetOne from a main thread, optionally GetOnePending from sub-threads
TPollSockets = class(TPollAbstract)
...
/// initialize the sockets polling
constructor Create(aPollClass: TPollSocketClass = nil);
/// finalize the sockets polling, and release all used memory
destructor Destroy; override;
/// track modifications on one specified TSocket and tag
function Subscribe(socket: TNetSocket; events: TPollSocketEvents;
tag: TPollSocketTag): boolean; override;
/// stop status modifications tracking on one specified TSocket and tag
procedure Unsubscribe(socket: TNetSocket; tag: TPollSocketTag); virtual;
/// retrieve the next pending notification, or let the poll wait for new
function GetOne(timeoutMS: integer; const call: RawUtf8;
out notif: TPollSocketResult): boolean; virtual;
/// retrieve the next pending notification
function GetOnePending(out notif: TPollSocketResult; const call: RawUtf8): boolean;
/// let the poll check for pending events and apend them to fPending results
function PollForPendingEvents(timeoutMS: integer): integer; virtual;
/// manually append one event to the pending nodifications
procedure AddOnePending(aTag: TPollSocketTag; aEvents: TPollSocketEvents;
aNoSearch: boolean);
/// notify any GetOne waiting method to stop its polling loop
procedure Terminate; override;
...
</pre>
<p>Depending on the operating system, it will mimic the <a href="https://man7.org/linux/man-pages/man7/epoll.7.html">epoll api</a> with the underlying low-level system calls.</p>
<p>The events are very abstract, and are in fact just the basic R/W operations on each connection, associated with a "tag", which is likely to be a class pointer associated with a socket/connection:</p>
<pre>
TPollSocketEvent = (
pseRead,
pseWrite,
pseError,
pseClosed);
TPollSocketEvents = set of TPollSocketEvent;
TPollSocketResult = record
tag: TPollSocketTag;
events: TPollSocketEvents;
end;
TPollSocketResults = record
Events: array of TPollSocketResult;
Count: PtrInt;
end;
</pre>
<p>Then whole new HTTP/1.0, HTTP/1.1 and WebSockets stacks have been written on top of those basic socket-driven events. Instead of blocking threads, they use internal state machines, which are much lighter than a thread, and even lighter than a coroutine/goroutine. Each connection is just a class instance, which maintains the state of each client/server communication, and accesses its own socket.<br />
The <em>mORMot</em> asynchronous TCP server has by default one thread to accept the connection, one thread to poll for pending events (calling the <code>GetOne</code> method), then a dedicated number of threads to consume the Read/Write/Close events (via the <code>GetOnePending</code> method). We used as many non-blocking structures as possible, we minimized memory allocation by reusing the same buffers e.g. for the headers or small responses, we pickup the <a href="https://blog.synopse.info?post/2022/05/21/New-Async-HTTP/post/2022/01/22/Three-Locks-To-Rule-Them-All">best locks possible</a> in each case, so that this server could scales nice and smoothly. And simple to use, because handling a new protocol is as easy as inheriting and writing a new connection class.</p>
<p>Our asynchronous servers classes seem now stable, and fast (reported to be twice faster than nginx and six time faster than nodejs!).<br />
But of course, as any new complex code, they may be caveats. And some of them have already be identified and fixed - as <a href="https://synopse.info/forum/viewtopic.php?pid=36546#p36546">reported in our forum</a>. Therefore, feedback is welcome, and a nginx, haproxy or caddy reverse proxy frontend is always a good idea on production.</p>
<p>Writing those servers took more time than previewed, and was sometimes painful. Because debugging multi-thread process is not easy, and especially on several operating systems. There are some subtle differences between the OS, which could lead to unexpected blocking or degraded performance. But we are proud of the result, which compares to the best-in-class servers. Still in modern pascal code, and Open Source software.</p>
<p>Don't hesitate to take a look at the source, and try some samples.<br />
Feedback is <a href="https://synopse.info/forum/viewtopic.php?id=6253">welcome in our forum</a>, as usual.</p>Enhanced HTTP/HTTPS Support in mORMot 2urn:md5:fa0f372038d4e743aa1f81b28cdea9c32021-05-14T12:29:00+01:002021-05-14T12:29:00+01:00Arnaud BouchezmORMot FrameworkDelphiFreePascalhashHTTPHTTPSmORMotmORMot2OpenSSLpasswordperformanceRESTsecuritySSPIAuthWGet<p>HTTP(S) is the main protocol of the Internet.<br />
We enhanced the mORMot 2 socket client to push its implementation into more use cases. The main new feature is perhaps WGET-like processing, with hashing, resuming, console feedback, and direct file download.</p>
<p><img src="https://blog.synopse.info?post/public/blog/wget.png" alt="" /></p> <h3>Several Clients to Rule Them All</h3>
<p>Our framework supports several HTTP clients implementations/classes:</p>
<ul>
<li>Socket-based <code>THttpClientSocket</code>;</li>
<li>Windows WinHTTP API via <code>TWinHttp</code>;</li>
<li>Windows WinINET API via <code>TWinINet</code>;</li>
<li>LibCurl API via <code>TCurlHttp</code>.</li>
</ul>
<p>Depending on the system, you may want to use either of the classes.</p>
<p>In practice, they are not equivalent. For instance, our <code>THttpClientSocket</code> is the fastest over all - especially on UNIX, when using Unix Sockets instead of TCP to communicate (even it is not true HTTP any more, but it is supported by mORMot classes and also by reverse proxies like <code>nginx</code>).</p>
<p>This blog article will detail the latest modifications in mORMot 2 <code>THttpClientSocket</code> class.</p>
<h3>OpenSSL TLS Fixes</h3>
<p>During an intensive test phase, some fixes have been applied to the TLS/HTTPS layer. On Windows, it is now easier to load the OpenSSL dll, and the direct SSPI layer did have some fixes applied.</p>
<h3>Authentication and Proxy Support</h3>
<p><code>THttpClientSocket</code> has now callbacks for HTTP authentication, using any kind of custom scheme, with built-in support of regular Basic algorithm, or a cross-platform Kerberos algorithm, using SSPI on Windows or the GSSAPI on UNIX.</p>
<p>This class can now connect via a HTTP Proxy tunnel, with proper password authentication if needed.</p>
<h3>MultiPart Support</h3>
<p>One of the most complex use of HTTP is the ability to cut a stream as multiple parts. Up to now, there was limited support of the MultiPart encoding.</p>
<p>As consequences:</p>
<ul>
<li>The <code>THttpClientSocket</code> class now accepts not only memory buffers (encoded as <code>RawByteString</code>) but also <code>TStream</code>, so that data of any size could be downloaded (GET) or uploaded (POST);</li>
<li>We added the <code>THttpMultiPartStream</code> class for multipart/formdata HTTP POST.</li>
</ul>
<h3>WGet Support</h3>
<p><a href="https://www.gnu.org/software/wget/">GNU Wget</a> is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols.</p>
<p>We have implemented its basic features set - with extensions - in <code>THttpClientSocket.WGet()</code> method, as a convenient high-level way of downloading files.</p>
<ul>
<li>Direct file download with low resource consumption;</li>
<li>Can resume any previously aborted file, without restarting from the start - it would be very welcome for huge files;</li>
<li>Optional redirection from the original URI following the 30x status codes from the server;</li>
<li>Optional on-the-fly hashing of the content - supporting all <code>mormot.crypt.core</code> algorithms;</li>
<li>Optional progress event, via a callback method or built-in console display;</li>
<li>Optional folder to maintain a list of locally cached files;</li>
<li>Optional limitation of the bandwidth consumed during download, to reduce the network consumption e.g. on slow connections;</li>
<li>Optional timeout check for the whole download process (not only at connection level).</li>
</ul>
<p>We hope these new features will help you work on HTTP, and perhaps switch from other HTTP client libraries if you want to focus on our framework. We are open to any missing feature you may need to implement.</p>SQlite3 Encryption Not Possible Any More Since 3.32.xurn:md5:266d34ae4331f9400def516f185677b32020-06-05T17:23:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkAESAES-NiDatabaseperformancesecuritySQLite3VFS<p>About latest SQlite3 3.32.xxx there is a big problem with codecs.</p>
<p>Critical changes to the public SQLite code were introduced on Feb 7, 2020: “<a href="https://www.sqlite.org/src/timeline?c=5a877221ce90e752">Simplify the code by removing the unsupported and undocumented SQLITE_HAS_CODEC compile-time option</a>”. With the release of SQLite version 3.32.0 on May 22, 2020 these changes finally took officially effect, although they weren't officially announced.</p>
<p><img alt="" src="https://blog.synopse.info?post/public/blog/RugbyTears.jpg" title="RugbyTears.jpg, Jun 2020" /></p>
<p>As a sad and unexpected consequence,<strong> we are NOT ANY MORE able to compile the new SQlite3 amalgamation with our encryption patch</strong>.</p> <p>Reverting the patch is not an option, since it is much more than a few lines, and it messes with SQLite3 VM opcodes, so it will induce a lot of potential errors.</p>
<p>Perhaps a dedicated VFS may be the solution - this is the path chosen by <a href="https://github.com/utelle/SQLite3MultipleCiphers">https://github.com/utelle/SQLite3MultipleCiphers</a> for instance.<br />
But a VFS is a lot of work, error prone, and performance may also decrease.<br />
If we go this VFS path, we may add a compression feature, too...</p>
<p><a href="https://synopse.info/forum/viewtopic.php?pid=32288#p32288">Any feedback is welcome!<br />
If anyone has an idea...</a></p>
<p><strong>Edit/Update</strong>: We have implemented a solution based on VFS.<br />
Please see <a href="https://synopse.info/forum/viewtopic.php?pid=32327#p32327">https://synopse.info/forum/viewtopic.php?pid=32327#p32327</a></p>Application Locking using Asymmetric Encryptionurn:md5:bff923d47f36c2fb0216d2b1bd091e7e2017-03-18T17:54:00+01:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkAsymmetricblogDelphiECCECDHECIESECSDAsecurity<p>A common feature request for professional software is to prevent abuse of
published applications.<br />
For licensing or security reasons, you may be requested to "lock" the execution
of programs, maybe tools or services.</p>
<p><img src="https://lh5.ggpht.com/YAU4DdvU_Tj40BtrB__z-ZaIpBMgRJI4An_TnbmAGshWWSQJmqF9BFpGWmKUx7jVIQ=w300" /></p>
<p>Our Open-Souce <em>mORMot</em> framework can leverage <a href="https://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_187">
Asymmetric Cryptography</a> to ensure that only allowed users could run some
executables, optionally with dedicated settings, on a given computer.<br />
It offers the first brick on which you may build your own system upon.</p>
<p>From the User point of view, he/she will transmit
a <code>user@host.public</code> file, then receives a corresponding
<code>user@host.unlock</code> file, which will unlock the application.<br />
Pretty easy to understand - even if some complex asymmetric encryption is
involved behind the scene.</p> <p>The <code>dddInfraApps.pas</code> unit publishes the following
<code>ECCAuthorize</code> function and type:</p>
<pre>
<strong>type</strong>
TECCAuthorize = (eaSuccess, eaInvalidSecret, eaMissingUnlockFile,
eaInvalidUnlockFile, eaInvalidJson);
<br /><strong>function</strong> ECCAuthorize(aContent: TObject; aSecretDays: integer; <strong>const</strong> aSecretPass,
aDPAPI, aDecryptSalt, aAppLockPublic64: RawUTF8; <strong>const</strong> aSearchFolder: TFileName = '';
aSecretInfo: PECCCertificateSigned = <strong>nil</strong>; aLocalFile: PFileName = <strong>nil</strong>): TECCAuthorize;
</pre>
<p>This function will use several asymmetric key sets:</p>
<ul>
<li>A <em>main key set</em>, named e.g. <code>applock.public</code> and
<code>applock.private</code>, shared for all users of the system;</li>
<li>Several <em>user-specific key sets</em>, named e.g.
<code>userhost.public</code> and <code>userhost.secret</code>, one for each
<code>user</code> and associated computer <code>host</code> name.</li>
</ul>
<p>When the <code>ECCAuthorize</code> function is executed, it will search for
a local <code>userhost.unlock</code> file, named after the current logged user
and the computer host name. Of course, the first time the application is
launched for this user, there will be no such file. It will create two local
<code>userhost.public</code> and <code>userhost.secret</code> files and return
<code>eaMissingUnlockFile</code>.</p>
<p>The <em>main key set</em> will be used to digitally <em>sign</em> the
<code>unlock</code> file.<br />
<em>User-specific key sets</em> will be used to <em>encrypt</em> the
<code>unlock</code> file.</p>
<p>Please check for <a href="https://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_193">
updated information in our online documentation</a>!</p>
<p>Feedback is <a href="https://synopse.info/forum/viewtopic.php?id=3875">welcome in our forum</a>, as
usual.</p>Cheat Mode for Private Keysurn:md5:8232f82e6fbe4eee584ebd5a1afdaebc2016-10-18T13:21:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkAESAsymmetricblogDocumentationECCECDHECIESECSDAGoodPracticemORMotpasswordsecuritySynEcc<p>In order to follow best practice, our <code>.private</code> key files are
always protected by a password. A random value with enough length and
entropy is always proposed by the <code>ECC</code> tool when a key pair is
generated, and could be used directly.<br />
It is always preferred to trust a computer to create true randomness (and
<code>SynCrypto.pas</code>'s secure <code>TAESPRNG</code> was designed to be
the best possible seed, using hardware entropy if available), than using our
human brain, which could be defeated by dictionary-based password
attacks.<br />
Brute force cracking would be almost impossible, since
<code>PBKDF2_HMAC_SHA256</code> Password-Based Key Derivation Function with
60,000 rounds is used, so rainbow tables (i.e. pre-computed passwords list)
will be inoperative, and each password trial would take more time than with a
regular Key Derivation Function.</p>
<p><a href="http://xkcd.com/936/"><img src="http://imgs.xkcd.com/comics/password_strength.png" width="400" height="325" /></a></p>
<p>The issue with strong passwords is that they are difficult to remember. If
you use not pure random passwords, but some easier to remember values with good
entropy, you may try some tools like <a href="https://xkpasswd.net/s">https://xkpasswd.net/s</a> which returns values like
<code>$$19*wrong*DRIVE*read*61$$</code>.<br />
But even then, you will be able to remember only a dozen of such passwords. In
a typical public key infrastructure, you may create hundredths of keys, so
remembering all passwords is no option for an average human being as (you and)
me.</p>
<p>At the end, you end up with using a tool to store all your passwords (last
trend is to use an online service with browser integration), or - admit it -
store them in an <code>Excel</code> document protected by a password. Most IT
people - and even security specialists - end with using such a mean of storage,
just because they need it.<br />
The weaknesses of such solutions can be listed:</p>
<ul>
<li>How could we trust closed source software and third-party online
services?</li>
<li>Even open source like <a href="http://keepass.info/help/base/security.html">http://keepass.info/help/base/security.html</a>
may appear weak (no PBKDF, no AFSplit, managed C#, SHA as PRNG);</li>
<li>The storage is as safe as the "master password" is safe;</li>
<li>If the "master password" is compromised, all your passwords are
published;</li>
<li>You need to know the master password to add a new item to the store.</li>
</ul>
<p>The <code>ECC</code> tool is able to work in "cheat mode", storing all
<code>.private</code> key files generated passwords in an associated
<code>.cheat</code> local file, encrypted using a <code>cheat.public</code>
key.</p>
<p>As a result:</p>
<ul>
<li>Each key pair will have its own associated <code>.cheat</code> file, so you
only unleash one key at a time;</li>
<li>The <code>.cheat</code> file content is meaningless without the
<code>cheat.private</code> key and its master password, so you can manage and
store them together with your <code>.private</code> files;</li>
<li>Only the <code>cheat.public</code> key is needed when creating a key pair,
so you won't leak your master password, and even could generate keys in an
automated way, on a distant server;</li>
<li>The <code>cheat.private</code> key will be safely stored in a separated
place, only needed when you need to recover a password;</li>
<li>It uses strong <em>File Encryption</em>, with proven PBKDF, AFSplit,
AES-PRNG, and ECDH/ECIES algorithms.</li>
</ul> <p>By default, no <code>.cheat</code> files are created. You need to explicitly
initialize the "cheat mode", by creating master <code>cheat.public</code> and
<code>cheat.private</code> key files:</p>
<pre>
>ecc cheatinit
Enter Issuer identifier text of the master cheat keys.
Will be truncated to 15-20 ascii-7 chars.
Issuer [arbou] :
<br />Enter a private PassPhrase for the master cheat.private key (at least 8 chars).
Save this in a safe place: if you forget it, the key will be useless!
NewPass [uQHH*am39LLj] : verysafelongpassword
<br />Enter iteration rounds for the mastercheat.private key (at least 100000).
NewRounds [100000] :
<br />cheat.public/.private file created.
</pre>
<p>As you can see, the default number of PBKDF rounds is high (100000), and
local files have been created:</p>
<pre>
>dir cheat.*
<br />18/10/2016 11:12 4 368 cheat.private
18/10/2016 11:12 568 cheat.public
</pre>
<p>Now we will create a new key pair (in a single command line, with no console
interaction):</p>
<pre>
>ecc new -newpass NewKeyP@ssw0rd -noprompt
<br />Corresponding TSynPersistentWithPassword.ComputePassword:
encryption HeOyjDUAsOhvLZkMA0Y=
authMutual lO0mv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
authServer lO0nv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
authClient lO0kv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
<br />D1045FCBAA1382EE44ED2C212596E9E1.public/.private file created.
</pre>
<p>An associated <code>.cheat</code> file has been created:</p>
<pre>
>dir D10*
<br />18/10/2016 11:15 1 668 D1045FCBAA1382EE44ED2C212596E9E1.cheat
18/10/2016 11:15 2 320 D1045FCBAA1382EE44ED2C212596E9E1.private
18/10/2016 11:15 588 D1045FCBAA1382EE44ED2C212596E9E1.public
</pre>
<p>Imagine you forgot about the <code>NewKeyPssw0rd</code> value. You could use
the following command to retrieve it:</p>
<pre>
>ecc cheat
<br />Enter the first chars of the .private certificate file name.
Auth: D10
<br />Will use: D1045FCBAA1382EE44ED2C212596E9E1.private
<br />Enter the PassPhrase of the master cheat.private file.
AuthPass: verysafelongpassword
<br />Enter the PassPhrase iteration rounds of the cheat.private file.
AuthRounds [100000] :
<br />{
"pass": "NewKeyP@ssw0rd",
"rounds": 60000
}
Corresponding TSynPersistentWithPassword.ComputePassword:
encryption HeOyjDUAsOhvLZkMA0Y=
authMutual lO0mv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
authServer lO0nv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
authClient lO0kv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
</pre>
<p>If your <code>.private</code> key does not have its associated
<code>.cheat</code> file, you won't be able to recover your password:</p>
<pre>
>ecc cheat
<br />Enter the first chars of the .private certificate file name.
Auth: 8BC9
<br />Will use: 8BC90201EF55EE34F62DBA8FE8CF14DC.private
<br />Enter the PassPhrase of the master cheat.private file.
AuthPass: verysafelongpassword
<br />Enter the PassPhrase iteration rounds of the cheat.private file.
AuthRounds [100000] :
<br />Fatal exception EECCException raised with message:
Unknown file 8BC90201EF55EE34F62DBA8FE8CF14DC.cheat
</pre>
<p>In practice, this "cheat mode" will help you implement a safe public key
infrastructure of any size. It will be as secure as the main
<code>cheat.private</code> key file and its associated password remain hidden
and only wisely spread, of course. Don't forget to use the <code>ecc
rekey</code> command on a regular basis, so that you change the master password
of <code>cheat.private</code>. The main benefit of this implementation is that
for all key generation process, only the <code>cheat.public</code> key file is
needed.</p>
<p>The <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_189">
updated documentation</a> is the place to find latest information about this
feature.</p>Public-key Asymmetric Cryptography via SynECCurn:md5:5df692905316ee52e4e3b6dfc834aaeb2016-09-24T20:10:00+02:002016-09-25T19:38:07+02:00AB4327-GANDImORMot FrameworkAESAsymmetricblogCACertificatesDelphiDocumentationECCECDHECIESECSDAmORMotPKIPublicKeysecurityshaSynEcc<p>After weeks of implementation and testing, we introduce today a new feature
of our <em>mORMot</em> Open-Source Framework.</p>
<p><a href="https://xkcd.com/1323/"><img src="https://imgs.xkcd.com/comics/protocol.png" /></a></p>
<p>Asymmetric encryption, also known as <a href="https://en.wikipedia.org/wiki/Public-key_cryptography">public-key
cryptography</a>, uses pairs of keys:</p>
<ul>
<li><em>Public</em> keys that may be disseminated widely;</li>
<li>Paired with <em>private</em> keys which are known only to the owner.</li>
</ul>
<p>The framework <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_187">
SynEcc</a> unit features a full asymmetric encryption system, based on
<em><a href="https://en.wikipedia.org/wiki/Elliptic_curve_cryptography">Elliptic curve
cryptography</a></em> (ECC), which may be used at application level (i.e. to
protect your application data, by signing or encrypting it), or at transmission
level (to enhance communication safety).<br />
A full set of high-level features, including certificates and command line
tool, offers a stand-alone but complete <a href="https://en.wikipedia.org/wiki/Public_key_infrastructure">public-key
infrastructure</a> (PKI).</p> <p>As we have seen when dealing about <em>Security</em>, the framework offers
built-in encryption of the content transmitted between its REST client and
server sides, especially via <em>Custom Encodings</em>, or HTTPS.<br />
The later, when using TLS 1.2 and proven patterns, implements state-of-the-art
security.<br />
But default <em>mORMot</em> encryption, even if using proven algorithms like
AES-CFB-256 and SHA-256, uses symmetric keys, that is the same secret key is
shared on both client and server sides.</p>
<p>Once you have <em>generated</em> a public/private pairs of keys, you can
perform two functions:</p>
<ul>
<li><em>Authenticate</em> a message originated with a holder of the private
key; a <em>certification</em> system should be used to maintain a trust chain
of authority;</li>
<li><em>Encrypt</em> a message with a public key to ensure that only the holder
of the paired private key can decrypt it.</li>
</ul>
<p>There will no doubt be criticism of our decision to re-implement a whole
public-key cryptography stack from scratch, with its own small choice of
algorithms, instead of using an existing library (like OpenSSL), and existing
standards (like X509).<br />
Here are some reasons:</p>
<ul>
<li>We did not start from scratch, since we used another proven Open Source
library for the raw ECC computation, which was the most sensitive part;</li>
<li>Existing librairies have to deal with a lot of algorithms, options and old
features: we wanted a reduced scope, to ease risk assessment - only well-known
and future-proof algorithms were selected (AES-256 excluding ECB, HMAC_SHA256,
PBKDF2_HMAC_SHA256, ECDSA, ECIES...) and default values are very aggressive
(password strength, 60,000 PBKDF2 iterations...);</li>
<li>Existing libraries are so complex that interfacing with them makes the
consuming code complex to write and maintain - <code>SynEcc</code> logic is
implemented in a few dozen lines of code: most of the unit source is about
wrapper methods and documentation, and an average programmer can understand and
review it, even if he/she is no Delphi expert;</li>
<li>A new implementation may benefits from past issues: we followed all
identified best practices, and tried to avoid, from the beginning, buffer
overflows, weak protocols, low entropy, low default values, serial collision,
forensic vulnerabilities, hidden memory copies, evil optimizations;</li>
<li>It integrates nicely with other <em>mORMot</em> features, and re-uses the
<code>SynCrypto.pas</code> unit for actual cryptography on all supported
platforms, so the development effort was not big, and the resulting executables
size did not increase;</li>
<li>As always, we started by writing tests, and we have pretty good automated
tests coverage, from low-level ECC functions up to the highest level (we even
validate the ECC command line tool);</li>
<li>We forbid file stamping, preferred JSON to any other text format, and used
fixed sized binary buffers (e.g. for identifiers), with all-inclusive
information, to avoid memory copies of sensitive data and logic flows depending
on the feature set;</li>
<li>Some unique features were introduced (like AFSpliting or enforcing
passwords for private keys), and in doubt, we always did choose the paranoid
solution;</li>
<li>We are proud that <em>mORMot</em> applications are stand-alone executables,
so the last thing we want to do is to start mandating DLLs, or be coupled to a
specific Operating System;</li>
<li>Having our own embedded code is safer than using the old/unsafe already
installed libraries, especially on an existing server (what is the OpenSSL
version in your good old Debian VM?);</li>
<li>It was fun, we learned a lot, and we hope you will enjoy using it, and
contribute to it!</li>
</ul>
<p>Read the full story <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_187">
in the new dedicated chapter of our documentation</a>.<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=3550">welcome
in our forum</a>, as usual.</p>Anti-forensic, safe storage of private keysurn:md5:82e4334ef5e88e6e4f5ff3efd47087bf2016-05-14T11:14:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkAESblogDelphiforensicmORMotsecuritysha<p>In any modern application, especially on <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_7">
Client/Server nTier architecture</a> as our little <em>mORMot</em> offers, we
often have to persist some private keys in a safe way.<br />
Problem with such keys is that they consist in small amount of bytes (typically
16 or 32 bytes), easy to be left somewhere in disk or memory.<br />
Given the abilities of recent <a href="https://en.wikipedia.org/wiki/Computer_forensics">forensic data recovery
methods</a>, data can't be destroyed on magnetic or flash storage media
reliably.</p>
<p><img src="https://www.kent.ac.uk/careers/pics/cryptography.jpg" alt="" /></p>
<p>We have just added to our <em>SynCrypto</em> OpenSource library
the <em>Anti-forensic Information Splitter</em> algorithm, as <a href="https://gitlab.com/cryptsetup/cryptsetup/wikis/TKS1-draft.pdf">proposed in
TKS1</a>, and implemented in the <a href="https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup">LUKS
standard</a>.<br />
LUKS is the de-facto standard of platform-independent standard on-disk
format for use in various tools.</p> <p>Storing any encrypted master key on disk, must take extra precaution, since
data is not guaranteed to be ever erased.</p>
<ul>
<li>Hard disk have a long long memory. Even if you think data is gone, even if
you have overwritten the whole disk with zeros, even if you invoked the
security erase SATA command of your hard disk, data can be easily recovered, if
not special care is taken to destroy it properly. Bad block remapping of
modern firmwares supports data safety but weakens the opposite, data
destruction.</li>
<li>Modern SSD controllers are also affected by this issue, and are now so
complex that <a href="https://articles.forensicfocus.com/2014/09/23/recovering-evidence-from-ssd-drives-in-2014-understanding-trim-garbage-collection-and-exclusions/">
even the TRIM command may leave some traces</a> of the previously stored
data.</li>
<li>Some bytes may be stored on disk whereas they were supposed to stay in RAM,
e.g. when the <a href="http://www.rekall-forensic.com/posts/2014-10-25-pagefile.html">OS starts
paging its memory</a>, or when you put your computer in hibernate mode.</li>
</ul>
<p>This <em>Anti-Forensic Splitter</em> algorithm supports secure data
destruction crucial for secure on-disk key management.<br />
The key idea is to bloat information and therefor improving the chance of
destroying a single bit of it.<br />
The information is bloated in such a way, that a single missing bit causes the
original information become unrecoverable.</p>
<p>The <a href="https://gitlab.com/cryptsetup/cryptsetup/wikis/TKS1-draft.pdf">TKS1 theory</a>
is presented as such:</p>
<blockquote>
<p><em>An easy approach is to create the inter-dependency for a data set S, S =
s1, s2, . . . sn, is by generating s1 . . . sn−1 random data items and
computing sn so that s1 X s2 X s3 X . . .X sn = D (X denotes the XOR
operation).<br />
The reconstruction is done by carring out the left-side of the equation,
XOR-ing all data items together.<br />
If one item s# is missing, D can’t be reconstructed, since an arbitrary s#
effects the entire D.</em></p>
<p><em>This scheme can be enhanced to include diffusion of information, so that
the k-th bit of an arbitrary si does not only affect the k-th bit of D but the
entire D.<br />
To achieve this diffusion, we insert a diffusion element in the chain of
XORs.<br />
A crytographic hash function can be used as such element, but since it might
not output sufficiently large data, it will be processed a few times with an
increasing number, similar to an initial vector, prepended to the complete
dataset to produce enough data. As a hash function is usually required to be
non-invertible, we can not choose it’s output. Therefore, the last diffusion
will be omitted. This will degrade security slightly, so when computing
destruction probabilities the last element shall never be taken into
account.</em></p>
</blockquote>
<p><a href="https://blog.synopse.info?post/public/mORMot/AFSpliter.png"><img src="https://blog.synopse.info?post/public/mORMot/.AFSpliter_m.jpg" alt="" title="AFSpliter, May 2016" /></a></p>
<p>Our <em>SynCrypto</em> implementation follows this algorithm, and uses
SHA-256 as diffusion element (whereas LUKS used SHA-1).</p>
<p><span class="u5">The <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html#TAESPRNG">TAESPRNG</a>
class now offers those two new methods:</span></p>
<pre>
<span class="u5">function</span> <span class="u4">AFSplit</span><span class="u9">(</span><span class="u5">const</span> <span class="u4">Buffer</span><span class="u9">;</span> <span class="u4">BufferBytes</span><span class="u9">,</span> <span class="u4">StripesCount</span><span class="u9">:</span> <span class="u4">integer</span><span class="u9">):</span> <span class="u4">RawByteString</span><span class="u9">;<br /></span>class function AFUnsplit(const Split: RawByteString; out Buffer; BufferBytes: integer): boolean;
</pre>
<p>Please <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html#TAESPRNG_AFSPLIT">check
the documentation</a> for more details about how to use them.<br />
The implementation source (in today's state) can be found on <a href="http://synopse.info/fossil/info/41bc882fae2b">our source code
repository</a>.</p>AES-256 based Cryptographically Secure Pseudo-Random Number Generator (CSPRNG)urn:md5:d99c0c7b643507d7c78be9a1b13ce23b2016-04-09T11:37:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkAESAES-NiblogCSPRNGDelphiperformancerandomsecuritySource<p>Everyone knows about the pascal <a href="http://docwiki.embarcadero.com/Libraries/en/System.Random">random()
function</a>.<br />
It returns some numbers, using a <a href="http://en.wikipedia.org/wiki/Linear_congruential_generator">linear
congruential generator</a>, with a multiplier of <a href="http://forum.lazarus.freepascal.org/index.php?topic=23536.msg140722#msg140722">134775813</a>,
in its Delphi implementation.<br />
It is fast, but not really secure. Output is very predictable, especially if
you forgot to execute the <a href="http://docwiki.embarcadero.com/Libraries/en/System.RandSeed">RandSeed()
procedure</a>.</p>
<p><img src="http://resources.infosecinstitute.com/wp-content/uploads/121411_1611_SecureRando1.png" alt="" width="500" height="148" /></p>
<p>In real world scenarios, safety always requires random numbers, e.g. for
key/nonce/IV/salt/challenge generation.<br />
The less predictable, the better.<br />
We just included a <a href="https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator">Cryptographically
Secure Pseudo-Random Number Generator</a> (CSPRNG) into our <a href="https://github.com/synopse/mORMot/commit/92135fcdf7503bb54479643e926668532ca357d3">
SynCrypto.pas</a> unit.<br />
The <code>TAESPRNG</code> class would use real system entropy to generate
a sequence of pseudorandom bytes, using AES-256, so returning highly
unpredictable content.</p> <p>The <code>TAESPRNG</code> class is implemented as such:</p>
<ul>
<li>It would gather entropy using dedicated OS API, i.e. the <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa379942">CryptGenRandom
API</a> under Windows or <a href="http://www.2uo.de/myths-about-urandom/">/dev/urandom - /dev/random</a> on
Linux/POSIX systems; and since we are paranoid, we XOR some entropy hash
(retrieved e.g. directly from RDTSC and other system variables) to those OS
values - so even if the API is compromised, we try to have some changing
input;</li>
<li>This entropy (256 bytes of password, 16 bytes of salt) would be hashed
using safe PBKDF2-HMAC-SHA256 key derivation function (256 iterations by
default) to setup the secret key of an AES-256 cypher, and set a counter (CTR)
initial value by applying this AES-256 to the salt;</li>
<li>Each time some output is to be generated, AES-256 is applied to the CTR to
produce 16 bytes (2^128 bits) of pseudorandom data;</li>
<li>The CTR is incremented after each 16-byte block; AES being a block
cipher, it is a permutation of the space of block values: as such, it won't
ever output twice the same 16-byte block, so you can generate 2^128 blocks,
i.e. 2^132 bytes before the CTR overflows;</li>
<li>The AES-256 cypher is re-seeded from entropy on a regular basis (after
some bytes are generated), to avoid potential attacks on backward or
forward security;</li>
<li>The implementation is thread-safe, a shared <code>TAESPRNG.Main</code>
instance is available, but you can create your
own <code>TAESPRNG</code> instance, with tuned parameters (number
of PBKDF2 counts or automatic re-seeding number of bytes);</li>
<li>The level of security would be the same on all platforms, since OS is used
only for entropy, but an unique CSPRNG algorithm would actually generate
the data - even a compromised system (older CryptGenRandomAPI had <a href="http://eprint.iacr.org/2007/419.pdf">known weaknesses</a>, or you could
imagine some plot-based undocumented backdoor) may produce safe enough
output;</li>
<li>It would use <a href="https://blog.synopse.info?post/post/2015/01/15/AES-NI-enabled-for-SynCrypto">AES-NI</a> or Padlock hardware
acceleration, if available.</li>
</ul>
<p>You may use it in your projects by calling <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html#TAESPRNG_FILLRANDOM">TAESPRNG.Main.FillRandom()</a>
overloaded methods or <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html#TAESPRNG_FILLRANDOMBYTES">
TAESPRNG.Main.FillRandomBytes()</a>.</p>
<p>See the <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html#TAESPRNG">class
documentation</a>.</p>
<p>The main idea of a CSPRNG is that its output is as safe as the cypher it is
based on.<br />
Using AES-256, and initial PBKDF2 key derivation of OS-gathered entropy,
implements a <a href="https://en.wikipedia.org/wiki/Randomness_extractor">very
good randomness extractor</a>.<br />
Last but not least, especially if your CPU supports AES-NI (which is very
likely on a server), performance would be very high.<br />
Just use <code>TAESPRNG</code> when you need random input. Then go back to
your own code.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=3263">welcome on our forum</a>, as
usual!</p>AVAST did detect ALL Delphi programs as dangerousurn:md5:c682c0d40c91f1e82626b1d7adfbc0982015-09-17T16:03:00+02:002015-09-17T15:08:22+02:00AB4327-GANDIPascal ProgrammingantivirusAVASTblogDelphifalse-positivesecurity<p>Today, an avalanche of "<a href="https://en.wikipedia.org/wiki/False_positives_and_false_negatives">false
postitive detection</a>" of AVAST heuristic engine did occur.<br />
Any executable built with Delphi XE8 or Delphi 10 Seattle was identified as a
<em>Win32:Banker-MGC [Trj]</em> threat!</p>
<p><img src="http://spudcomics.com/comics/2015-01-14-pirate.png" width="271" height="276" /></p>
<p><a href="https://en.wikipedia.org/wiki/Heuristic_analysis">Heuristic
analysis</a> is a method employed by many computer antivirus programs designed
to detect previously unknown computer viruses, as well as new variants of
viruses already in the "wild".</p> <p>AVAST "experts" introduced some detection rules which identified all Delphi
executables as potentially dangerous.</p>
<p>If you make a small Delphi program without any link to the VCL, with some
access to the Internet, a lot of "cheap" AV programs would identify this
program as a danger.<br />
This is the symptom of poorly maintained heuristic rules.</p>
<p>AFAIK it is the first time their "rules" have been defined so poorly that
even ANY Delphi program is detected as dangerous.<br />
AVAST team should not be proud.<br />
A simple test with a fixed void Delphi application would be enough to detect
such regressions.</p>
<p>At least, they reacted promptly.<br />
There are still a lot of Delphi programs in the wild! ;)<br />
They <a href="https://forum.avast.com/index.php?topic=176583.msg1252118#msg1252118">claimed
this has been fixed by now</a>.</p>
<p>It certainly did cost a lot of money of IT professionals, using Delphi
applications.<br />
But you can ask for your money back, when something is free, right?</p>
<p>In all cases, what is needed is to check any suspicious executable using
meta-scanners.<br />
See for instance <a href="https://www.virustotal.com/en/file/4d94146d1a50a24c32e6158c414f3f7078912b7f0f46c63e94426d1ede9a9303/analysis/1442480149/">
how a valid Delphi executable was checked today, using VirusTotal online
service</a>.</p>Asynchronous Service - WebSockets, Callbacks and Publish-Subscribeurn:md5:4b0f2fbc0b9085026d4baacbf5ee8ed92015-04-06T21:34:00+02:002015-04-06T20:57:33+02:00AB4327-GANDImORMot FrameworkAESAES-NiAJAXauthenticationblogDatabaseDelphiDocumentationEventCollaborationEventSourcinginterfaceModelORMsecuritySOASOLIDSourceSQLite3SynopseWebSockets<p>When publishing SOA services, most of them are defined as
<em>stateless</em>, in a typical query/answer pattern - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_17">
Service-Oriented Architecture (SOA)</a></em>.<br />
This fits exactly with the <em>RESTful</em> approach of <em>Client-Server
services via interfaces</em>, as proposed by the framework.</p>
<p>But it may happen that a client application (or service) needs to know the
state of a given service. In a pure <em>stateless</em> implementation, it will
have to <em>query</em> the server for any state change, i.e. for any pending
notification - this is called <em>polling</em>.</p>
<p><img src="https://blog.synopse.info?post/public/mORMot/BatNotification.png" alt="" title="RealTime Notification, Apr 2015" /></p>
<p><em>Polling</em> may take place for instance:</p>
<ul>
<li>When a time consuming work is to be processed on the server side. In this
case, the client could not wait for it to be finished, without raising a
timeout on the HTTP connection: as a workaround, the client may start the work,
then ask for its progress status regularly using a timer and a dedicated method
call;</li>
<li>When an unpredictable event is to be notified from the server side. In this
case, the client should ask regularly (using a timer, e.g. every second), for
any pending event, then react on purpose.</li>
</ul>
<p>It may therefore sounds preferred, and in some case necessary, to have the
ability to let the server <em>notify</em> one or several clients without any
prior query, nor having the requirement of a client-side timer:</p>
<ul>
<li><em>Polling</em> may be pretty resource consuming on both client and server
sides, and add some unwanted latency;</li>
<li>If immediate notification is needed, some kind of "long polling" algorithm
may take place, i.e. the server will wait for a long time before returning the
notification state if no event did happen: in this case, a dedicated connection
is required, in addition to the REST one;</li>
<li>In an event-driven systems, a lot of messages are sent to the clients: a
proper publish/subscribe mechanism is preferred, otherwise the complexity of
polling methods may increase and become inefficient and unmaintainable;</li>
<li>Explicit push notifications may be necessary, e.g. when a lot of potential
events, associated with a complex set of parameters, are likely to be sent by
the client.</li>
</ul>
<p>Our <em>mORMot</em> framework is therefore able to easily implement
asynchronous callbacks over <em><a href="http://www.developerfusion.com/article/143158/an-introduction-to-websockets/">WebSockets</a></em>,
defining the callbacks as <code>interface</code> parameters in service method
definitions - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_154">
Available types for methods parameters</a></em>.</p> <h3>1. WebSockets support</h3>
<p>By definition, HTTP connections are stateless and one-way, i.e. a client
sends a request to the server, which replies back with an answer.<br />
There is no way to let the server send a message to the client, without a prior
request from the client side.</p>
<p><em>WebSockets</em> is a communication protocol which is able to
<em>upgrade</em> a regular HTTP connection into a dual-way communication
wire.<br />
After a safe handshake, the underlying TCP/IP socket is able to be accessed
directly, via a set of lightweight <em>frames</em> over an application-defined
<em>protocol</em>, without the HTTP overhead.</p>
<p>The <code><a href="http://synopse.info/files/html/api-1.18/SynBidirSock.html">SynBidirSock.pas</a></code>
unit implements low-level server and client <em>WebSockets</em>
communication.</p>
<p>The <code>TWebSocketProtocol</code> class defines an abstract <em>WebSockets
protocol</em>, currently implemented as several classes.</p>
<p>For our <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#SIDE_TITL_63">
Client-Server services via interfaces</a></em>, we would still need to make
<em>RESTful</em> requests, so the basic <em>WebSockets</em> framing has been
enhanced to support <code>TWebSocketProtocolRest</code> REST-compatible
protocols, able to use the single connection for both REST queries and
asynchronous notifications.</p>
<p>Two classes are available for your SOA applications:</p>
<ul>
<li><code>TWebSocketProtocolJSON</code> as a "pure" JSON light protocol;</li>
<li><code>TWebSocketProtocolBinary</code> as a binary proprietary protocol,
with optional frame compression and AES encryption (using AES-NI hardware
instructions, if available).</li>
</ul>
<p>In practice, on the server side, you would start your
<code>TSQLHttpServer</code> by specifying <code>useBidirSocket</code> as kind
of server:</p>
<pre>
HttpServer := TSQLHttpServer.Create('8888',[Server],'+',useBidirSocket);
</pre>
<p>Under the hood, it will instantiate a <code>TWebSocketServer</code> HTTP
server, as defined in <code>mORMotHttpServer.pas</code>, based on the sockets
API, able to upgrade the HTTP protocol into <em>WebSockets</em>.<br />
Our <em>High-performance http.sys server</em> is not yet able to switch to
<em>WebSockets</em> - and at API level, it would require at least <em>Windows
8</em> or <em>Windows 2012 Server</em>.</p>
<p>Then you enable <em>WebSockets</em> for the
<code>TWebSocketProtocolBinary</code> protocol, with an encryption key:</p>
<pre>
HttpServer.WebSocketsEnable(Server,'encryptionkey');
</pre>
<p>On the client side, you would use a <code>TSQLHttpClientWebsockets</code>
instance, as defined in <code>mORMotHttpClient.pas</code>, then explicitly
upgrade the connection to use <em>WebSockets</em> (since by default, it will
stick to the HTTP protocol):</p>
<pre>
Client := TSQLHttpClientWebsockets.Create('127.0.0.1','8888',TSQLModel.Create([]));
Client.WebSocketsUpgrade('encryptionkey');
</pre>
<p>The expected protocol detail should match the one on the server, i.e.
<code>'encryptionkey'</code> encryption over our binary protocol.</p>
<p>Once upgraded to <em>WebSockets</em>, you may use regular REST commands, as
usual:</p>
<pre>
Client.ServerTimeStampSynchronize;
</pre>
<p>But in addition to regular query/answer commands as defined for
<em>Client-Server services via interfaces</em>, you would be able to define
callbacks using <code>interface</code> parameters to the service methods.</p>
<p>Under the hood, both client and server will communicate using
<em>WebSockets</em> frames, maintaining the connection active using heartbeats
(via ping/pong frames), and with clean connection shutdown, from any side. You
can use the <code>Settings</code> property of the
<code>TWebSocketServerRest</code> instance, as returned by
<code>TSQLHttpServer.WebSocketsEnable()</code>, to customize the low-level
<em>WebSockets</em> protocol (e.g. timeouts or heartbeats) on the server side.
The <code>TSQLHttpClientWebsockets.WebSockets</code>.<code>Settings</code>
property would allow the same, on the client side.</p>
<p>We have observed, from our regression tests and internal benchmarking, that
using our <em>WebSockets</em> may be faster than regular HTTP, since its frames
would be sent as once, whereas HTTP headers and body are not sent in the same
TCP packet, and compression would be available for the whole frame, whereas
HTTP headers are not compressed. The ability to use strong AES encryption would
make this mean of communication even safer than plain HTTP, even with
<em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_151">
AES encryption over HTTP</a></em>.</p>
<h3>2. Using a callback to notify long term end-of-process</h3>
<p>An example is better than 100 talks.<br />
So let's take a look at the <code>Project31LongWorkServer.dpr</code> and
<code>Project31LongWorkClient.dpr</code> samples, from the <code><a href="https://github.com/synopse/mORMot/tree/master/SQLite3/Samples/31%20-%20WebSockets">
SQLite3\Samples\31 - WebSockets</a></code> sub-folder.<br />
They will implement a client/server application, in which the client launches a
long term process on the server side, then is notified when the process is
done, either with success, or failure.</p>
<p>First we define the interfaces to be used, in a shared
<code>Project31LongWorkCallbackInterface.pas</code> unit:</p>
<pre>
<strong>type</strong>
ILongWorkCallback = <strong>interface</strong>(IInvokable)
['{425BF199-19C7-4B2B-B1A4-A5BE7A9A4748}']
<strong>procedure</strong> WorkFinished(<strong>const</strong> workName: <strong>string</strong>; timeTaken: integer);
<strong>procedure</strong> WorkFailed(<strong>const</strong> workName, error: <strong>string</strong>);
<strong>end</strong>;<br /> ILongWorkService = <strong>interface</strong>(IInvokable)
['{09FDFCEF-86E5-4077-80D8-661801A9224A}']
<strong>procedure</strong> StartWork(<strong>const</strong> workName: <strong>string</strong>; <strong>const</strong> onFinish: ILongWorkCallback);
<strong>function</strong> TotalWorkCount: Integer;
<strong>end</strong>;
</pre>
<p>The only specific definition is the <code>const onFinish:
ILongWorkCallback</code> parameter, supplied to the
<code>ILongWorkService.StartWork()</code> method.<br />
The client will create a class implementing <code>ILongWorkCallback</code>,
then specify it as parameter to this method.<br />
On the server side, a "fake" class will implement
<code>ILongWorkCallback</code>, then will call back the client using the very
same <em>WebSockets</em> connection, when any of its methods will be
executed.</p>
<p>As you can see, a single callback <code>interface</code> instance may have
several methods, with their own set of parameters (here
<code>WorkFinished</code> and <code>WorkFailed</code>), so that the callback
may be quite expressive.<br />
Any kind of usual parameters would be transmitted, after serialization:
<code>string</code>, <code>integer</code>, but even <code>record</code>,
<em>dynamic arrays</em>, <code>TSQLRecord</code> or <code>TPersistent</code>
values.</p>
<p>When the <code>ILongWorkCallback</code> instance will be released on the
client side, the server will be notified, so that any further notification
won't create a connection error.<br />
We will see later how to handle those events.</p>
<h3>Client service consumption</h3>
<p>The client may be connected to the server as such (see the
<code>Project31LongWorkClient.dpr</code> sample source code for the full
details, including error handling):</p>
<pre>
<strong>var</strong> Client: TSQLHttpClientWebsockets;
workName: <strong>string</strong>;
Service: ILongWorkService;
callback: ILongWorkCallback;
<strong>begin</strong>
Client := TSQLHttpClientWebsockets.Create('127.0.0.1','8888',TSQLModel.Create([]));
<span style="background-color:yellow;">Client.WebSocketsUpgrade(PROJECT31_TRANSMISSION_KEY);</span>
Client.ServiceDefine([ILongWorkService],sicShared);
Client.Services.Resolve(ILongWorkService,Service);
</pre>
<p>Then we define our callback, using a dedicated class:</p>
<pre>
<strong>type</strong>
TLongWorkCallback = <strong>class</strong>(TInterfacedCallback,ILongWorkCallback)
<strong>protected</strong>
<strong>procedure</strong> WorkFinished(<strong>const</strong> workName: <strong>string</strong>; timeTaken: integer);
<strong>procedure</strong> WorkFailed(<strong>const</strong> workName, error: <strong>string</strong>);
<strong>end</strong>;<br /><strong>procedure</strong> TLongWorkCallback.WorkFailed(<strong>const</strong> workName, error: <strong>string</strong>);
<strong>begin</strong>
writeln(#13'Received callback WorkFailed(',workName,') with message "',error,'"');
<strong>end</strong>;
<br /><strong>procedure</strong> TLongWorkCallback.WorkFinished(<strong>const</strong> workName: <strong>string</strong>;
timeTaken: integer);
<strong>begin</strong>
writeln(#13'Received callback WorkFinished(',workName,') in ',timeTaken,'ms');
<strong>end</strong>;
</pre>
<p>Then we specify this kind of callback as parameter to start a long term
work:</p>
<pre>
<span style="background-color:yellow;">callback := TLongWorkCallback.Create(Client,ILongWorkCallback);</span>
<strong>try</strong>
<strong>repeat</strong>
readln(workName);
<strong>if</strong> workName='' <strong>then</strong>
break;
<span style="background-color:yellow;">Service.StartWork(workName,callback);</span>
<strong>until</strong> false;
<strong>finally</strong>
callback := <strong>nil</strong>; <em>// the server will be notified and release its "fake" class</em>
Service := <strong>nil</strong>; <em>// release the service local instance BEFORE Client.Free</em>
<strong>end</strong>;
</pre>
<p>As you can see, the client is able to start one or several work processes,
then expects to be notified of the process ending on its callback
<code>interface</code> instance, without explicitly polling the server for its
state, since the connection was upgraded to <em>WebSockets</em> via a call to
<code>TSQLHttpClientWebsockets.WebSocketsUpgrade()</code>.</p>
<h3>Server side implementation</h3>
<p>The server would define the working thread as such (see the
<code>Project31LongWorkServer.dpr</code> sample source code for the full
details):</p>
<pre>
<strong>type</strong>
TLongWorkServiceThread = <strong>class</strong>(TThread)
<strong>protected</strong>
<span style="background-color:yellow;">fCallback: ILongWorkCallback;</span>
fWorkName: <strong>string</strong>;
<strong>procedure</strong> Execute; <strong>override</strong>;
<strong>public</strong>
<strong>constructor</strong> Create(<strong>const</strong> workName: <strong>string</strong>; <strong>const</strong> callback: ILongWorkCallback);
<strong>end</strong>;
<br /><strong>constructor</strong> TLongWorkServiceThread.Create(<strong>const</strong> workName: <strong>string</strong>;
<strong>const</strong> callback: ILongWorkCallback);
<strong>begin</strong>
<strong>inherited</strong> Create(false);
<span style="background-color:yellow;">fCallback := Callback;</span>
fWorkName := workName;
FreeOnTerminate := true;
<strong>end</strong>;
<br /><strong>procedure</strong> TLongWorkServiceThread.Execute;
<strong>var</strong> tix: Int64;
<strong>begin</strong>
tix := GetTickCount64;
<span style="background-color:yellow;">Sleep(5000+Random(1000)); <em>// some hard work</em></span>
<strong>if</strong> Random(100)>20 <strong>then</strong>
<span style="background-color:yellow;">fCallback.WorkFinished(fWorkName,GetTickCount64-tix) <strong>else</strong></span>
<span style="background-color:yellow;">fCallback.WorkFailed(fWorkName,'expected random failure');</span>
<strong>end</strong>;
</pre>
<p>The callback is expected to be supplied as a <code>ILongWorkCallback</code>
interface instance, then stored in a <code>fCallback</code> protected field for
further notification.<br />
Some work is done in the <code>TLongWorkServiceThread.Execute</code> method
(here just a <code>Sleep()</code> of more than 5 seconds), and the end-of-work
notification is processed, as success or failure (depending on random in this
fake process class), on either of the <code>ILongWorkCallback</code> interface
methods.</p>
<p>The following <code>class</code> will define, implement and register the
<code>ILongWorkService</code> service on the server side:</p>
<pre>
<strong>type</strong>
TLongWorkService = <strong>class</strong>(TInterfacedObject,ILongWorkService)
<strong>protected</strong>
fTotalWorkCount: Integer;
<strong>public</strong>
<strong>procedure</strong> StartWork(<strong>const</strong> workName: <strong>string</strong>; <strong>const</strong> onFinish: ILongWorkCallback);
<strong>function</strong> TotalWorkCount: Integer;
<strong>end</strong>;
<br /><strong>procedure</strong> TLongWorkService.StartWork(<strong>const</strong> workName: <strong>string</strong>;
<strong>const</strong> onFinish: ILongWorkCallback);
<strong>begin</strong>
InterlockedIncrement(fTotalWorkCount);
<span style="background-color:yellow;">TLongWorkServiceThread.Create(workName,onFinish);</span>
<strong>end</strong>;
<br /><strong>function</strong> TLongWorkService.TotalWorkCount: Integer;
<strong>begin</strong>
result := fTotalWorkCount;
<strong>end</strong>;
<br /><strong>var</strong> HttpServer: TSQLHttpServer;
Server: TSQLRestServerFullMemory;
<strong>begin</strong>
Server := TSQLRestServerFullMemory.CreateWithOwnModel([]);
<span style="background-color:yellow;">Server.ServiceDefine(TLongWorkService,[ILongWorkService],sicShared);</span>
HttpServer := TSQLHttpServer.Create('8888',[Server],'+',useBidirSocket);
<span style="background-color:yellow;">HttpServer.WebSocketsEnable(Server,PROJECT31_TRANSMISSION_KEY);</span>
...
</pre>
<p>Purpose of those methods is just to create and launch the
<code>TLongWorkServiceThread</code> process from a client request, then
maintain a total count of started works, in a <code>sicShared</code> service
instance - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_92">
Instances life time implementation</a></em> - hosted in a
<code>useBidirSocket</code> kind of HTTP server.</p>
<p>We have to explicitly call <code>TSQLHttpServer.WebSocketsEnable()</code> so
that this server would be able to upgrade to our <em>WebSockets</em> protocol,
using our binary framing, and the very same encryption key as on the client
side - shared as a <code>PROJECT31_TRANSMISSION_KEY</code> constant in the
sample, but which may be safely stored on both sides.</p>
<h3>3. Publish-subscribe for events</h3>
<p>In event-driven architectures, the <em>publish-subscribe</em> messaging
pattern is a way of letting senders (called <em>publishers</em>) transmit
messages to their receivers (called <em>subscribers</em>), without any prior
knowledge of who those subscribers are.<br />
In practice, the <em>subscribers</em> will express interest for a set of
messages, which will be sent by the <em>publisher</em> to all the
<em>subscribers</em> of a given message, as soon as it is be notified.</p>
<p>In our <em>Client-Server services via interfaces</em> implementation,
messages are gathered in <code>interface</code> types, and each message defined
as a single method, their content being the methods parameters.<br />
Most of the SOA alternative (in Java or C#) do require class definition for
messages. Our KISS approach will just use method parameters values as message
definition.</p>
<p>To maintain a list of <em>subscribers</em>, the easiest is to store a
<em>dynamic array</em> of <code>interface</code> instances, on the
<em>publisher</em> side.</p>
<h3>Defining the interfaces</h3>
<p>We will now implement a simple <em>chat</em> service, able to let several
clients communicate together, broadcasting any message to all the other
connected instances.<br />
This sample is also located in the the <code><a href="https://github.com/synopse/mORMot/tree/master/SQLite3/Samples/31%20-%20WebSockets">
SQLite3\Samples\31 - WebSockets</a></code> sub-folder, as
<code>Project31ChatServer.dpr</code> and
<code>Project31ChatClient.dpr</code>.</p>
<p>So you first define the callback interface, and the service interface:</p>
<pre>
<strong>type</strong>
IChatCallback = <strong>interface</strong>(IInvokable)
['{EA7EFE51-3EBA-4047-A356-253374518D1D}']
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo, msg: <strong>string</strong>);
<strong>end</strong>;
<br /> IChatService = <strong>interface</strong>(IInvokable)
['{C92DCBEA-C680-40BD-8D9C-3E6F2ED9C9CF}']
<strong>procedure</strong> Join(<strong>const</strong> pseudo: <strong>string</strong>; <strong>const</strong> callback: IChatCallback);
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo,msg: <strong>string</strong>);
<strong>procedure</strong> CallbackReleased(<strong>const</strong> callback: IInvokable);
<strong>end</strong>;
</pre>
<p>Those interface types will be shared by both server and client sides, in the
common <code>Project31ChatCallbackInterface.pas</code> unit. The definition is
pretty close to what we wrote just above to notify long term
end-of-process.<br />
The only additional method is <code>IChatServer.CallbackReleased()</code>,
which, by convention, will be called on the server side when any
<code>callback</code> interface instance is released on the client side.</p>
<p>As such, the <code>IChatService.Join()</code> method will implement the
<em>subscription</em> to the chat service, whereas
<code>IChatServer.CallbackReleased()</code> will be called when the client-side
callback instance will be released (i.e. when its variable will be assigned to
<code>nil</code>), to <em>unsubscribe</em> for the chat service.</p>
<h3>Writing the Publisher</h3>
<p>On the server side, each call to <code>IChatService.Join()</code> would
<em>subscribe</em> to an internal list of connections, simply stored as an
<code>array of IChatCallback</code>:</p>
<pre>
<strong>type</strong>
TChatService = <strong>class</strong>(TInterfacedObject,IChatService)
<strong>protected</strong>
<span style="background-color:yellow;">fConnected: <strong>array of</strong> IChatCallback;</span>
<strong>public</strong>
<strong>procedure</strong> Join(<strong>const</strong> pseudo: <strong>string</strong>; <strong>const</strong> callback: IChatCallback);
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo,msg: <strong>string</strong>);
<strong>procedure</strong> CallbackReleased(<strong>const</strong> callback: IInvokable);
<strong>end</strong>;
<br /><strong>procedure</strong> TChatService.Join(<strong>const</strong> pseudo: <strong>string</strong>;
<strong>const</strong> callback: IChatCallback);
<strong>begin</strong>
<span style="background-color:yellow;">InterfaceArrayAdd(fConnected,callback);</span>
<strong>end</strong>;
</pre>
<p>The <code>InterfaceArrayAdd()</code> function, as defined in
<code>SynCommons.pas</code>, is a simple wrapper around any <em>dynamic
array</em> of <code>interface</code> instances, so that you may use it, or the
associated <code>InterfaceArrayFind()</code> or
<code>InterfaceArrayDelete()</code> functions, to maintain the list of
subscriptions.</p>
<p>Then a remote call to the <code>IChatService.BlaBla()</code> method should
be broadcasted to all connected clients, just by calling the
<code>IChatCallback.BlaBla()</code> method:</p>
<pre>
<strong>procedure</strong> TChatService.BlaBla(<strong>const</strong> pseudo,msg: <strong>string</strong>);
<strong>var</strong> i: integer;
<strong>begin</strong>
<strong>for</strong> i := 0 <strong>to</strong> high(fConnected) <strong>do</strong>
<span style="background-color:yellow;">fConnected[i].BlaBla(pseudo,msg);</span>
<strong>end</strong>;
</pre>
<p>Note that every call to <code>IChatCallback.BlaBla()</code> within the loop
would be made via <em>WebSockets</em>, in an asynchronous and non blocking way,
so that even in case of huge number of clients, the
<code>IChatService.BlaBla()</code> method won't block.<br />
In case of high numbers of messages, the framework is even able to
<em>gather</em> push notification messages into a single bigger message, to
reduce the resource use.</p>
<p>On the server side, the service implementation has been registered as
such:</p>
<pre>
Server.ServiceDefine(TChatService,[IChatService],sicShared).
SetOptions([],[optExecLockedPerInterface]);
</pre>
<p>Here, the <code>optExecLockedPerInterface</code> option has been set, so
that all method calls would be made thread-safe, so that concurrent access to
the internal <code>fConnected[]</code> list would be safe.<br />
Since a global list of connections is to be maintained, the service life time
is defined as <code>sicShared</code> - see <em><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_92">
Instances life time implementation</a></em>.</p>
<p>The following method will be called by the server, when a client callback
instance is released (either explicitly, or if the connection is broken), so
could be used to <em>unsubscribe</em> to the notification, simply by deleting
the callback from the internal <code>fConnected[]</code> array:</p>
<pre>
<strong>procedure</strong> TChatService.CallbackReleased(<strong>const</strong> callback: IInvokable);
<strong>begin</strong>
<span style="background-color:yellow;">InterfaceArrayDelete(fConnected,callback);</span>
<strong>end</strong>;
</pre>
<p>The framework will in fact recognize the following method definition in any
<code>interface</code> type for a service:</p>
<pre>
<strong>procedure</strong> CallbackReleased(<strong>const</strong> callback: IInvokable);
</pre>
<p>When a callback <code>interface</code> parameter (in our case,
<code>IChatCallback</code>) will be released on the client side, this method
will be called with the corresponding <code>interface</code> instance as
parameter.<br />
You do not have to call explicitly any method on the client side to
<em>unsubscribe</em> a service: assigning <em>nil</em> to a callback variable,
or feeing the <code>class</code> instance owning it as a field on the
<em>subscriber</em> side, will automatically unregister it on the
<em>publisher</em> side.</p>
<h3>Consuming the service from the Subscriber side</h3>
<p>On the client side, you implement the <code>IChatCallback</code> callback
interface:</p>
<pre>
<strong>type</strong>
TChatCallback = <strong>class</strong>(TInterfacedCallback,IChatCallback)
<strong>protected</strong>
<strong>procedure</strong> BlaBla(<strong>const</strong> pseudo, msg: <strong>string</strong>);
<strong>end</strong>;
<br /><strong>procedure</strong> TChatCallback.BlaBla(<strong>const</strong> pseudo, msg: <strong>string</strong>);
<strong>begin</strong>
writeln(#13'@',pseudo,' ',msg);
<strong>end</strong>;
</pre>
<p>Then you subscribe to your remote service as such:</p>
<pre>
<span style="background-color:yellow;"><strong>var</strong> Service: IChatService;</span>
<span style="background-color:yellow;">callback: IChatCallback;</span>
...
Client.ServiceDefine([IChatService],sicShared);
<span style="background-color:yellow;"><strong>if not</strong> Client.Services.Resolve(IChatService,Service) <strong>then</strong></span>
<strong>raise</strong> EServiceException.Create('Service IChatService unavailable');
...
<span style="background-color:yellow;">callback := TChatCallback.Create(Client,IChatCallback);</span>
<span style="background-color:yellow;">Service.Join(pseudo,callback);</span>
...
<strong>try</strong>
<strong>repeat</strong>
readln(msg);
<strong>if</strong> msg='' <strong>then</strong>
break;
<span style="background-color:yellow;">Service.BlaBla(pseudo,msg);</span>
<strong>until</strong> false;
<strong>finally</strong>
<span style="background-color:yellow;">callback := <strong>nil</strong>; <em>// will unsubscribe from the remote publisher</em></span>
Service := <strong>nil</strong>; <em>// release the service local instance BEFORE Client.Free</em>
<strong>end</strong>;
</pre>
<p>You could easily implement more complex <em>publish/subscribe</em>
mechanisms, including filtering, time to live or tuned broadcasting, by storing
some additional information to the <code>interface</code> instance (e.g. some
value to filter, a timestamp). A dynamic array of dedicated
<code>record</code>s, or a list of <code>class</code> instances, may be used to
store the <em>subscribers</em> expectations.</p>
<p>If you compare with existing client/server SOA solutions (in Delphi, Java,
C# or even in Go or other frameworks), this <code>interface</code>-based
callback mechanism sounds pretty unique and easy to work with.<br />
In fact, this is a good way of implementing callbacks conforming to <em>SOLID
design principles</em> on the server side, and let the <em>mORMot</em>
framework publish this mechanism in a client/server way, by using
<em>WebSockets</em>. The very same code could be used on the server side, with
no transmission nor marshaling overhead (via direct <code>interface</code>
calls), and over a network, with optimized use of resource and bandwidth (via
"fake" <code>interface</code> calls, and JSON marshalling over TCP/IP).</p>
<div style="margin-left: 2em">Ensure you <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_149">
take a look at the corresponding online documentation</a>, which would be
updated often than this blog article!<br />
The ORM part of the framework uses this <em>Publish-Subscribe</em> feature to
implement <a href="https://blog.synopse.info?post/post/2015/04/06/Real-Time-ORM-Master/Slave-Replication-via-WebSockets">Real-Time
master/slave replication</a>.<br />
Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=15632#p15632">welcome on our
forum</a>, as usual.</div>AES-NI enabled for SynCryptourn:md5:69688bfde7b83f67979567b29bcf85422015-01-15T20:49:00+01:002015-01-15T20:53:02+01:00AB4327-GANDIOpen SourceAESAES-NiasmblogDelphimORMotperformancesecuritySource <p>Today, we <a href="https://github.com/synopse/mORMot/commit/1b119af809a513">committed a new patch
to enable AES-NI hardware acceleration</a> to our <a href="http://synopse.info/files/html/api-1.18/SynCrypto.html">SynCrypto.pas</a>
unit.</p>
<p><a href="http://www.intel.com/content/www/us/en/architecture-and-technology/advanced-encryption-standard--aes-/data-protection-aes-general-technology.html">
Intel® AES-NI</a> is a new encryption instruction set that improves on the
<em>Advanced Encryption Standard</em> (AES) algorithm and accelerates the
encryption of data on newer processors.</p>
<p><img src="https://pbs.twimg.com/media/BqkJVF7CMAA8Aww.jpg" alt="" width="300" height="202/" /></p>
<p>Of course, all this is available in the Delphi unit, from Delphi 6 to XE7:
no external <em>dll</em> nor OS update is needed.<br />
And it will <a href="https://blog.synopse.info?post/post/2015/01/10/mORMot-under-Linux-thanks-to-FPC">work
also on Linux</a>, so could help <a href="https://blog.synopse.info?post/post/2014/01/05/AES-encryption-over-HTTP">encrypting the mORMot
transmission</a> with no power loss.</p>
<p>You have nothing to do: just upgrade your <em>mORMot</em> source code, then
AES-NI instructions will be used, if the CPU offers it.<br />
We have seen performance boost of more than 5x, depending on the size of the
data to be encrypted.</p>
<p>Enjoy!</p>2015: the future of mORMot is BigDataurn:md5:aad77f14b241587052c36d8614eb16372014-12-31T11:35:00+01:002014-12-31T12:17:12+01:00AB4327-GANDImORMot FrameworkbackupblogCrossPlatformDelphiLinuxmORMotORMperformancereplicationRestsecurityshardingSOASynopse<p>How would be 2015 like for our little rodents?<br />
Due to popular request of several users of <em>mORMot</em>, we identified and
designed some feature requests dedicated to <a href="http://en.wikipedia.org/wiki/Big_data">BigData</a> process.</p>
<p><img src="http://4.bp.blogspot.com/-dFLzN6qwi1I/UoELR41hJXI/AAAAAAAAAB8/9du_nNOdDtE/s1600/Big_data.ashx_.png" alt="" /></p>
<p>In fact, your <em>data</em> is the new value, especially if you propose
<em>SaaS</em> (<a href="http://en.wikipedia.org/wiki/Software_as_a_service">Software As A Service</a>)
hosting to your customers, with a farm of <em><ins>mORMot</ins></em>
servers.<br />
Recent Linux support for <em>mORMot</em> servers, together with the
high performance and installation ease of our executable, open the gate to
cheap cloud-based hosting.<br />
As a consequence, a lot of information would certainly be gathered by
your <em>mORMot</em> servers, and a single monolithic database
is not an option any more.</p>
<p>For <em>mORMot</em> solutions hosted in cloud, a lot of data may be
generated. The default <em>SQLite3</em> storage engine may be less convenient,
once it reaches some GB of file content. Backup becomes to be slow and
inefficient, and hosting this oldest data in the main DB, probably stored on an
expensive SSD, may be a lost of resource. Vertical scaling is limited by
hardware and price factors.</p>
<p>This is were <strong>data sharding</strong> comes into scene.<br />
Note that <em>sharding</em> is not replication/backup, nor clustering, nor just
spreading. We are speaking about application-level data splitting, to ease
maintenance and horizontal scalability of <em>mORMot</em> servers.</p>
<p>Data sharding could already be implemented with <em>mORMot</em> servers,
thanks to <code>TSQLRestStorage</code>:</p>
<ul>
<li>Using <code>TSQLRestStorageExternal</code>: any table may have its own
external SQL database engine, may be in its separated DB server;</li>
<li>Using <code>TSQLRestStorageMongoDB</code>: any table may be stored on a
<em>MongoDB</em> cluster, with its own sharding abilities;</li>
<li>Using <code>TSQLRestStorageRemote</code>: each table may have its own
remote ORM/REST server.</li>
</ul>
<p>But when data stored in a single table tends to grow without limit, this
feature is not enough.<br />
Let's see how the close future of <em>mORMot</em> looks like.</p> <p>The following features are on the road map:</p>
<ul>
<li><a href="http://synopse.info/fossil/info/3453f314d9">Auto-Synch</a> between
<em>mORMot</em> servers (as cache or for branch office cache), and potentially
<em>mORMot</em> clients (offline mode);</li>
<li><a href="http://synopse.info/fossil/tktview/cde2d68eb60463">Automated
sharding</a> for BigData storage;</li>
<li><a href="http://synopse.info/fossil/tktview/17958b22c028">Monitoring</a> of
your farm of <em>mORMot</em> servers;</li>
<li><a href="http://synopse.info/fossil/tktview/9357b49fe2">Workaround to use
interfaces</a> / SOA under FPC (Linux);</li>
<li><a href="http://synopse.info/fossil/tktview/8e4c17a082">Integrate the
Monkey HTTP Server</a> for Linux;</li>
<li>Implements <a href="http://synopse.info/fossil/tktview?name=d71391c2e7">WiredTiger NoSQL
engine</a>.</li>
</ul>
<p>Please refer to each feature request ticket for design patterns, and
implementation proposal.<br />
The <a href="http://synopse.info/fossil/tktview?name=aa230e5299">publish/subscribe
event-based mechanism</a> would be first to be implemented, since it would be
used by the first items.</p>
<p>Your feedback is <a href="http://synopse.info/forum/viewtopic.php?id=2259">welcome on our forum</a>, as
usual!</p>HTTP remote access for SynDB SQL executionurn:md5:8ac51ed0a6e45f22746c8ba9e6d3e63c2014-11-18T01:08:00+01:002014-11-18T01:11:52+01:00AB4327-GANDIOpen Source librariesarray bindingblogcompressionDatabaseDelphiFirebirdFireDACHTTPhttp.sysHTTPSmORMotMSSQLMySQLOpenSourceOracleORMperformancesecuritySQLSynDBSynDBExplorerTDataSetwebWinHTTPWinINetZEOS<p>For <em>mORMot</em>, we developed a fully feature direct access layer to any
RDBMS, implemented <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_126">
in the SynDB.pas unit</a>.</p>
<p>You can use those <code>SynDB</code> classes to execute any SQL statement,
without any link to the framework ORM.<br />
At reading, the resulting performance is much higher than using the standard
<code>TDataSet</code> component, which is in fact a true performance
bottleneck.<br />
It has genuine features, like <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_136">
column access via late-binding</a>, <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_137">
an innovative <code>ISQLDBRows</code> interface, and ability to directly access
the low-level binary buffers of the database clients.</a></p>
<p><img src="http://vpn-services.bestreviews.net/files/vpn-remote-access.jpg" alt="" /></p>
<p>We just added a nice feature to those classes: the ability to access
remotely, via plain HTTP, to any <code>SynDB</code> supported database!</p> <p>To define a HTTP server, you may write:</p>
<pre>
<strong>uses</strong> SynDB, <em>// RDBMS core</em>
SynDBSQLite3, SynSQLite3Static, <em>// static SQLite3 engine</em>
SynDBRemote; <em>// for HTTP server</em>
...
<strong>var</strong> Props: TSQLDBConnectionProperties;
HttpServer: TSQLDBServerAbstract;
...
Props := TSQLDBSQLite3ConnectionProperties.Create('data.db3','','','');
<span style="background-color:yellow;">HttpServer := TSQLDBServerHttpApi.Create(Props,'remote','8092','user','pass');</span>
</pre>
<p>The above code will initialize a connection to a local <code>data.db3</code>
<em>SQlite3</em> database (in the <code>Props</code> variable), and then
publish it using the <code>http.sys</code> kernel mode HTTP server to the
<code>http://1.2.3.4:8092/remote</code> URI - if the server's IP is
<code>1.2.3.4</code>.</p>
<p>On the client side, you can then write:</p>
<pre>
<strong>uses</strong> SynDB, <em>// RDBMS core</em>
SynDBRemote; <em>// for HTTP server</em>
...
<strong>var</strong> Props: TSQLDBConnectionProperties;
...
Props := TSQLDBWinHTTPConnectionProperties.Create('1.2.3.4:8092','root','user','pass');
</pre>
<p>As you can see, there is no link to <code>SynDBSQLite3.pas</code> nor
<code>SynSQLite3Static.pas</code> on the client side.<br />
Just the HTTP link is needed.<br />
No need to deploy the RDBMS client libraries with your application, nor setup
the local network firewall.</p>
<p>We defined here a single user, with 'user' / 'pass' credentials, but you may
manage more users on the server side, using the <code>Authenticate</code>
property of <code>TSQLDBServerAbstract</code>.<br />
Note that in our remote access, user management does not match the RDBMS user
rights: you should better have your own set of users at application level, for
higher security, and a better integration with your business logic. If creating
a new user on a RDBMS could be painful, it is pretty easy on our
<code>SynDBRemote.pas</code> side.</p>
<p>Then, you execute your favorite SQL using the connection just as usual:</p>
<pre>
<strong>procedure</strong> Test(Props: TSQLDBConnectionProperties);
<strong>var</strong> Stmt: ISQLDBRows;
<strong>begin</strong>
Stmt := Props.Execute('select * from People where YearOfDeath=?',[1519]);
<strong>while</strong> Stmt.Step <strong>do begin</strong>
assert(Stmt.ColumnInt('ID')>0);
assert(Stmt.ColumnInt('YearOfDeath')=1519);
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>You may even use this remote connection e.g. using a stand-alone shared
<em>SQLite3</em> database as high performance but low maintenance client-server
database engine.</p>
<p>The transmission protocol uses an optimized binary format, which is
compressed and digitally signed on both ends, and the remote user
authentication will be performed via a challenge validation scheme.<br />
You could publish your server over HTTPS, if needed.</p>
<p>Even if you may be tempted to use such remote access to implement a
<em>n-Tier architecture</em>, you should rather use <em>mORMot</em>'s
Client-Server ORM instead. Our little <em>mORMot</em> is not an ORM on
which we added a data transmission layer: it is a full RESTful system, with a
true SOA design.<br />
But for integrating some legacy SQL code into a new architecture,
<code>SynDBRemote.pas</code> may have its benefits, used in conjunction with
<em>mORMot</em>'s higher level features.</p>
<p>Feel free to <a href="http://synopse.info/forum/viewtopic.php?id=2178">post
your feedback on our forum</a>, as usual, or <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_131">
browse the updated documentation</a>!</p>Will WebSocket replace HTTP? Does it scale?urn:md5:220c10d8621827708559d8f90e3c2cb32014-08-16T15:04:00+02:002020-07-03T09:29:59+02:00AB4327-GANDImORMot FrameworkAJAXblogDelphiEventCollaborationEventSourcingGoodPracticeHTTPhttp.sysJSONmORMotperformancePublish-SubscribeRestsecuritywebWebSockets<p>You certainly noticed that <em>WebSocket</em> is the current
trendy flavor for any modern web framework.<br />
But does it scale? Would it replace HTTP/REST?<br />
There is a <a href="http://synopse.info/fossil/tktview?name=aa230e5299">feature
request ticket about them for <em>mORMot</em></a>, so here are some thoughts -
matter of debate, of course!<br />
I started all this by <a href="http://stackoverflow.com/a/25340220/458259">answering a StackOverflow
question</a>, in which the actual answers were not accurate enough, to my
opinion.</p>
<p>From my point of view, <em>Websocket</em> - as a protocol -
is some kind of monster.</p>
<p><img src="http://3.bp.blogspot.com/-Oxi9kT_IRM4/UT_E3LjWguI/AAAAAAAABI8/wCgsTmIW-gI/s1600/three-eyed-monster.png" alt="" /></p>
<p>You start a HTTP stateless connection, then switch to <em>WebSocket</em>
mode which releases the TCP/IP dual-direction layer, then you may switch later
on back to HTTP...<br />
It reminds me some kind of monstrosity, just like encapsulating everything over
HTTP, using XML messages... Just to bypass the security barriers... Just
breaking the <a href="http://en.wikipedia.org/wiki/Osi_layers">OSI layered</a>
<ins>model</ins>...<br />
It reminds me the fact that our mobile phone data providers do not use
broadcasting for streaming audio and video, but regular Internet HTTP servers,
so the mobile phone data bandwidth is just wasted when a sport event occurs:
every single smart phone has its own connection to the server, and the same
video is transmitted in parallel, saturating the single communication
channel... Smart phones are not so smart, aren't they?</p>
<p><em>WebSocket</em> sounds like a clever way to circumvent a
limitation...<br />
But why not use a dedicated layer?<br />
I hope <a href="http://en.wikipedia.org/wiki/Http_2">HTTP 2.0</a> would allow
pushing information from the server, as part of the standard... and in one
decade, we probably will see <em>WebSocket</em> as a deprecated
technology.<br />
You have been warned. Do not invest too much in <em>WebSockets</em>..</p>
<p>OK. Back to our existential questions...<br />
First of all, does the <em>WebSocket</em> protocol scale?<br />
Today, any modern single server is able to server millions of clients at
once.<br />
Its HTTP server software has just to be is <em>Event-Driven</em> (IOCP)
oriented (we are not in the old <em>Apache</em>'s <em>one connection = one
thread/process</em> equation any more).<br />
Even the HTTP server built in <em>Windows</em> (http.sys - which is <a href="https://blog.synopse.info?post/post/2011/03/11/HTTP-server-using-fast-http.sys-kernel-mode-server">used in
<em>mORMot</em></a>) is IOCP oriented and very efficient (running in kernel
mode).<br />
From this point of view, there won't be a lot of difference at scaling between
<em>WebSocket </em>and a regular HTTP connection. One TCP/IP connection
uses a little resource (much less than a thread), and modern OS are optimized
for handling a lot of concurrent connections: <em>WebSocket</em> and HTTP
are just OSI 7 application layer protocols, inheriting from this TCP/IP
specifications.</p>
<p>But, from experiment, I've seen two main problems with
<em>WebSocket</em>:</p>
<ol>
<li>It does not support CDN;</li>
<li>It has potential security issues.</li>
</ol> <p>About the potential problems of using <em>WebSocket</em>:</p>
<h3>1. Consider using a CDN</h3>
<p>Today, web scaling involves using <a href="http://en.wikipedia.org/wiki/Content_delivery_network">Content Delivery
Network (CDN)</a> front ends, not only for static content (html,css,js) but
also <a href="https://cdnify.com/blog/cdn-for-apps">your (JSON) application
data</a>.</p>
<p>Of course, you won't put all your data on your CDN cache, but in practice, a
lot of common content won't change often.<br />
I suspect that 80% of your REST resources may be cached...<br />
Even a <em>one minute</em> (or 30 seconds) CDN expiration timeout may be enough
to give your central server a new life, and enhance the application
responsiveness a lot, since CDN can be geographically tuned...</p>
<p>To my knowledge, there is no <em>WebSocket</em> support in CDN yet, and
I suspect it would never be.<br />
The <em>WebSocket</em> protocol is stateful, whereas HTTP is stateless, so
is designed to be cached.<br />
In fact, to make <em>WebSocket</em> CDN-friendly, you may need to switch
to a stateless RESTful approach... which would not be
<em>WebSocket</em> any more.</p>
<h3>2. Security issues</h3>
<p><em>WebSocket</em> has several potential security issues, especially
about <a href="http://en.wikipedia.org/wiki/Denial-of-service_attack">DOS
attacks</a>.<br />
For illustration about new security vulnerabilities , see <a href="http://media.blackhat.com/bh-us-12/Briefings/Shekyan/BH_US_12_Shekyan_Toukharian_Hacking_Websocket_Slides.pdf">
this set of slides</a> and <a href="https://bugs.webkit.org/show_bug.cgi?id=32246">this webkit ticket</a>.</p>
<p>Some quotes from this set of slides:</p>
<ul>
<li><em>WebSockets </em>still fall victim to “old” threats;</li>
<li>If you can intercept or inject you can overtake <em>ws:/wss:</em> (MIM
attack);</li>
<li>Malicious content can exhaust browser by grabbing max. allowed number of
<em>WebSocket</em> connections (DOS attacks on client side by browsers!);</li>
<li>Malicious content can create large number of <em>WebSocket</em> connections
to victim <em>WebSocket</em> server (DOS attacks on server);</li>
<li>Just waiting for mistakes to happen (since developers can put anything on
the wire);</li>
<li><em>wss:</em> means secure transport, not secure app;</li>
<li>Browser support still in flux;</li>
<li><em>WebSockets</em> solve connection problems, not security problems.</li>
</ul>
<p>Furthermore, <em>WebSocket</em> avoid any chance of packet inspection
at OSI 7 application layer level.<br />
<a href="http://en.wikipedia.org/wiki/Application_layer_firewall">Application
firewalls</a>, <a href="http://en.wikipedia.org/wiki/Intrusion_detection_system">IDS</a>, <a href="http://en.wikipedia.org/wiki/Intrusion_prevention_system">IPS</a> are pretty
standard nowadays, in any serious security policy.<br />
In fact, <em>WebSocket</em> makes the transmission obfuscated, so may
become a major breach of security leak.</p>
<h3>When to use WebSockets?</h3>
<p>As a conclusion, I would recommend the following, for any project:</p>
<ul>
<li>Use <em>WebSocket </em>for client notifications only (with a fallback
mechanism to long-polling - there are plenty of libraries around);</li>
<li>Use RESTful / JSON for all other data, using a CDN or proxies for cache, to
make it scale.</li>
</ul>
<p>In practice, full <em>WebSocket's</em> applications do not scale well.<br />
Just use <em>WebSocket </em>for what it was designed to: a protocol to
push notifications from the server to the client.</p>
<p>With <em>mORMot</em>, and for all our end-user projects, this is the
direction we are currently entitled to go.<br />
Still privileging a pure stateless approach, which fits so good with our
RESTful model. </p>
<p>Discussion is <a href="http://synopse.info/forum/viewtopic.php?id=953">welcome in our forum, as
usual</a>!</p>AES encryption over HTTPurn:md5:309765b41f1b1a58b11f71e607c9b6202014-01-05T17:42:00+01:002014-01-05T17:43:29+01:00AB4327-GANDImORMot FrameworkblogDelphiHTTPhttp.sysHttpApiHTTPSmORMotRestsecurity<p>In addition to <a href="https://blog.synopse.info?post/post/2013/09/04/HTTPS-communication-in-mORMot">regular HTTPS flow
encryption</a>, which is not easy to setup due to the needed certificates,
<em>mORMot</em> proposes a proprietary encryption scheme. It is based on
SHA-256 and AES-256/CTR algorithms, so is known to be secure.</p>
<p><img src="http://blogs.scientificamerican.com/extinction-countdown/files/2013/06/marmot-groundhog.jpg" alt="" /></p>
<p>You do not need to setup anything on the server or the client configuration,
just run the <code>TSQLHttpClient</code> and <code>TSQLHttpServer</code>
classes with the corresponding parameters.</p> <p>Note that this encryption uses a global key for the whole process, which
should match on both Server and Client sides. You should better hard-code this
public key in your Client and Server Delphi applications, with some variants
depending on each end-user service. You can use
<code>CompressShaAesSetKey()</code> as defined in <code>SynCrypto.pas</code> to
set globally this Encryption Key, and an optional Initialization Vector. You
can even customize the AES chaining mode, if the default <code>TAESCTR</code>
mode is not what you expect.</p>
<p>When the <code>aHttpServerSecurity</code> parameter is set to
<code>secSynShaAes</code> for the <code>TSQLHttpServer.Create()</code>
constructor, this proprietary encryption will be enabled on the server side.
For instance:</p>
<pre>
MyServer := TSQLHttpServer.Create('888',[DataBase],'+',useHttpApiRegisteringURI,32,secSynShaAes);
</pre>
<p>On the client side, you can just set the
<code>TSQLHttpClientGeneric.Compression</code> property as expected:</p>
<pre>
MyClient.Compression := [hcSynShaAes];
</pre>
<p>Once those parameters have been set, a new proprietary encoding will be
defined in the HTTP headers:</p>
<pre>
ACCEPT-ENCODING: synshaaes
</pre>
<p>Then all HTTP body content will be compressed via our <em>SynLZ</em>
algorithm, and encoded using the very secure AES-CTR/256 encryption.</p>
<p>Since it is a proprietary algorithm, it will work only for Delphi clients.
When accessing for a plain AJAX client, or a Delphi application with
<code>TSQLHttpClientGeneric.Compression = []</code>, there won't be any
encryption at all, due to way HTTP accepts its encoding.<br />
For safety, you should therefore use it in conjunction with our <a href="https://blog.synopse.info?post/post/2013/06/07/Authentication-and-Authorization">per-URI
Authentication</a>.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1553">welcome on our forum</a>, as
usual!</p>HTTPS communication in mORMoturn:md5:4b680cafbfd343c1c6001d36658faaa02013-09-04T12:05:00+02:002013-09-04T13:26:23+02:00AB4327-GANDImORMot FrameworkDelphiHTTPhttp.sysHTTPSmORMotsecuritySource<p>In <em>mORMot</em>, the <code>http.sys</code> <a href="https://blog.synopse.info?post/post/2011/03/11/HTTP-server-using-fast-http.sys-kernel-mode-server">kernel
mode server</a> can be defined to serve HTTPS secure content.</p>
<p>Yes, <em>mORMots</em> do like sophistication:</p>
<p><img src="https://blog.synopse.info?post/public/mORMot/mORMotDrinking.jpg" alt="" title="mORMot drinking, sept. 2013" /></p>
<p>When the <code>aUseSSL</code> boolean parameter is set for
<code>TSQLHttpServer.Create()</code> constructor, the SSL layer will be enabled
within <code>http.sys</code>.<br />
Note that <code>useHttpSocket</code> kind of server does not offer SSL
encryption yet.</p>
<p>We will now define the steps needed to set up a HTTPS server in
<em>mORMot</em>.</p> <p>In order to let the SSL layer work as expected, you need first to create and
import a set of certificates.<br />
Here are the needed steps, as detailed in <a href="http://www.codeproject.com/Articles/24027/SSL-with-Self-hosted-WCF-Service">http://www.codeproject.com/Articles/24027/SSL-with-Self-hosted-WCF-Service</a>
and <a href="http://msdn.microsoft.com/en-us/library/ms733791.aspx">http://msdn.microsoft.com/en-us/library/ms733791.aspx</a> -
you can refer to any WCF related documentation about HTTPS, since it shares the
<code>http.sys</code> kernel-mode server with <em>mORMot</em> and
<em>IIS</em>.</p>
<h3>Certificates</h3>
<p>You need one certificate (cert) to act as your root authority, and one to
act as the actual certificate to be used for the SSL, which needs to be signed
by your root authority. If you don't set up the root authority your single
certificate won't be trusted, and you will start to discover this through a
series of extremely annoying exceptions, long after the fact.</p>
<p>The following command (run in a Visual Studio command prompt) will create
your root certificate:</p>
<pre>
makecert -sv SignRoot.pvk -cy authority -r signroot.cer -a
sha1 -n "CN=Dev Certification Authority" -ss my -sr localmachine
</pre>
<p>Take a look at the above links to see what each of these arguments mean, it
isn't terribly important, but it's nice to know.</p>
<p>The <code>MakeCert</code> tool is available as part of the Windows SDK,
which you can download from <a href="http://go.microsoft.com/fwlink/p/?linkid=84091">http://go.microsoft.com/fwlink/p/?linkid=84091</a>
if you do not want to download the whole <em>Visual Studio</em> package.
Membership in <em>Administrators</em>, or equivalent, on the local computer is
the minimum required to complete this procedure.</p>
<p>Once this command has been run and succeeded, you need to make this
certificate a trusted authority. You do this by using the MMC snap in console.
Go to the run window and type "<code>mmc</code>", hit enter. Then in the window
that opens (called the "<em>Microsoft Management Console</em>", for those who
care) perform the following actions:</p>
<pre>
File -> Add/Remove Snap-in -> Add… -> Double click Certificates -> Select Computer Account and Click Next -> Finish -> Close -> OK
</pre>
<p>Then select the <code>Certificates (Local Computer) -> Personal ->
Certificates</code> node.</p>
<p>You should see a certificate called "<code>Dev Certificate Authority</code>"
(or whatever else you decided to call it as parameter in the above command
line). Move this certificate from the current node to <code>Certificates (Local
Computer) -> Trusted Root Certification Authorities ->
Certificates</code> node, drag and drop works happily.</p>
<p>Now you have NOT the cert you need :)<br />
You have made yourself able to create trusted certs though, which is
nice.<br />
Now you have to create another cert, which you are actually going to use.</p>
<p>Run <code>makecert</code> again, but run it as follows...</p>
<pre>
makecert -iv SignRoot.pvk -ic signroot.cer -cy end -pe -n
CN="localhost" -eku 1.3.6.1.5.5.7.3.1 -ss my -sr
localmachine -sky exchange -sp
"Microsoft RSA SChannel Cryptographic Provider" -sy 12
</pre>
<p>Note that you are using the first certificate as the author for this latest
one. This is important... where I have <code>localhost</code> you need to put
the DNS name of your box. In other words, if you deploy your service such that
its endpoint reads <code>http://bob:10010/Service</code> then the name needs to
be <code>bob</code>. In addition, you are going to need to do this for each
host you need to run as (yes, so one for <code>bob</code> and another one for
<code>localhost</code>).</p>
<p>Get the signature of your cert by double clicking on the cert (Select the
<code>Certificates (Local Computer) ' Personal ' Certificates</code>), opening
the details tab, and scrolling down to the "<em>Thumbprint</em>" option.</p>
<p>Select the thumbprint and copy it. Put it in <code>Notepad</code> or any
other text editor and replace the spaces with nothing. <em>Keep this thumbprint
heaxdecimal value safe</em>, since we will need it soon.</p>
<p>You have your certs set up. Congrats!<br />
But we are not finished yet.</p>
<h3>Configure a Port with an SSL certificate</h3>
<p>Now you get to use another fun tool, <code>httpcfg</code> (for XP/2003), or
its newer version, named aka <code>netsh http</code> (for
Vista/Seven/Eight).</p>
<p>Firstly run the command below to check that you don't have anything running
on a port you want.</p>
<pre>
httpcfg query ssl
</pre>
<p>(under XP)</p>
<pre>
netsh http show sslcert
</pre>
<p>(under Vista/Seven/Eight)</p>
<p>If this is your first time doing this, it should just return a newline. If
there is already SSL set up on the exact IP you want to use (or if later on you
need to delete any mistakes) you can use the following command, where the IP
and the port are displayed as a result from the previous query.</p>
<p>Now we have to bind an SSL certificate to a port number, as such (here
below, <code>0000000000003ed9cd0c315bbb6dc1c08da5e6</code> is the
<em>thumbprint</em> of the certificate, as you copied it into the
<code>notepad</code> in the previous paragraph):</p>
<pre>
httpcfg set ssl -i 0.0.0.0:8012 -h 0000000000003ed9cd0c315bbb6dc1c08da5e6
</pre>
<p>(under XP)</p>
<pre>
netsh http add sslcert ipport=0.0.0.0:8000 certhash=0000000000003ed9cd0c315bbb6dc1c08da5e6 appid={00112233-4455-6677-8899-AABBCCDDEEFF}
</pre>
<p>(under Vista/Seven/Eight)<br />
Here the <code>appid=</code> parameter is a GUID that can be used to identify
the owning application.</p>
<p>To delete an SSL certificate from a port number previously registered, you
can use one of the following commands:</p>
<pre>
httpcfg delete ssl -i 0.0.0.0:8005 -h 0000000000003ed9cd0c315bbb6dc1c08da5e6
httpcfg delete ssl -i 0.0.0.0:8005
</pre>
<p>(under XP)</p>
<pre>
Netsh http delete sslcert ipport=0.0.0.0:8005
</pre>
<p>(under Vista/Seven/Eight)</p>
<p>Note that this is mandatory to first delete an existing certificate for a
given port before replacing it with a new one.</p>
<p>Feedback and information <a href="http://synopse.info/forum/viewtopic.php?id=1402">is welcome on our forum</a>,
as usual.</p>SynPDF now implements 40 bit and 128 bit securityurn:md5:4d503a44fa3af80e99a910a9d681fbf12013-06-19T19:16:00+02:002013-06-19T19:16:00+02:00AB4327-GANDISynopse PDF engineblogDelphimORMotPDFsecurity<p>The trunk version of our <a href="http://synopse.info/fossil/wiki?name=PDF+Engine">Open Source <em>SynPdf</em>
library</a> now <a href="http://synopse.info/fossil/info/6822e8c67f">features
encryption using 40 bit or 128 bit key size</a>.</p>
<p><img src="http://www.verypdf.com/app/pdf-editor-toolkit/encrypt-pdf.jpg" alt="" /></p>
<p>This is a long awaiting feature, and sounds working just fine from my
tests.<br />
Speed has been optimized (as usual with our libraries), as a consequence
encrypting the content will only be slightly slower.</p> <p>In fact, <code>TPdfEncryption.New()</code> will create the expected
<code>TPdfEncryption</code> instance, depending on the supplied encryption
Level:</p>
<pre>
<strong>class function</strong> TPdfEncryption.New(aLevel: TPdfEncryptionLevel;
<strong>const</strong> aUserPassword, aOwnerPassword: <strong>string</strong>;
aPermissions: TPdfEncryptionPermissions): TPdfEncryption;
</pre>
<p>Here are some comments about this new method:</p>
<ul>
<li>to be called as parameter of
<code>TPdfDocument/TPdfDocumentGDI.Create(</code>)</li>
<li>currently, only <em>elRC4_40</em> and <em>elRC4_128</em> levels are
implemented</li>
<li>both passwords are expected to be ASCII-7 characters only</li>
<li><em>aUserPassword</em> will be asked at file opening: to be set to '' for
not blocking display, but optional permission</li>
<li><em>aOwnerPassword</em> shall not be '', and will be used internally to
cypher the pdf file content</li>
<li><em>aPermissions</em> can be either one of the <code>PDF_PERMISSION_ALL /
PDF_PERMISSION_NOMODIF / PDF_PERSMISSION_NOPRINT / PDF_PERMISSION_NOCOPY /
PDF_PERMISSION_NOCOPYNORPRINT</code> set of options</li>
</ul>
<p>In practice, typical use may be:</p>
<pre>
Doc := TPdfDocument.Create(false,0,false,
TPdfEncryption.New(elRC4_40,'','toto',PDF_PERMISSION_NOMODIF));
Doc := TPdfDocument.Create(false,0,false,
TPdfEncryption.New(elRC4_128,'','toto',PDF_PERMISSION_NOCOPYNORPRINT));
</pre>
<p><a href="http://synopse.info/fossil/wiki?name=Get+the+source">Follow this
link</a> to get the latest trunk (unstable) version.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=1319">welcome on our forum, as
usual</a>!</p>Authentication and Authorizationurn:md5:726689ca220250da37e124b30de4e7282013-06-07T13:13:00+02:002013-06-07T13:13:00+02:00AB4327-GANDImORMot FrameworkauthenticationcookieCQRSDocumentationinterfacemORMotRestsecuritysessionSOASourceSQLSQLite3transaction<p>Our <em>mORMot</em> framework tries to implement security via:<br />
- Process safety;<br />
- Authentication;<br />
- Authorization.</p>
<p><em>Process safety</em> is implemented at every <em>n-Tier</em> level:<br />
- Atomicity of the <em>SQLite3</em> database core;<br />
- RESTful architecture to avoid most synchronization issues;<br />
- ORM associated to the Object pascal <em>strong type</em> syntax;<br />
- Extended test coverage of the framework core.</p>
<p><em>Authentication</em> allows user identification:<br />
- Build-in optional authentication mechanism, implementing both <em>per-user
sessions</em> and individual REST <em>Query Authentication</em>;<br />
- <em>Authentication groups</em> are used for proper authorization;<br />
- Several authentication schemes, from very secure SHA-256 based challenging to
weak but simple authentication;<br />
- Class-based architecture, allowing custom extension.</p>
<p><em>Authorization</em> of a given process is based on the group policy,
after proper authentication:<br />
- <em>Per-table access right</em> functionalities built-in at lowest level of
the framework;<br />
- Per-method execution policy for interface-based services;<br />
- General high-level security attributes, for SQL or Service remote
execution.</p>
<p>We will now give general information about both authentication and
authorization in the framework.</p>
<p>In particular, authentication is now implemented via a set of classes.</p> <h3>Authentication</h3>
<p>Extracted from <em>Wikipedia</em>:</p>
<p><em>Authentication (from Greek: "real" or "genuine", from "author") is the
act of confirming the truth of an attribute of a datum or entity. This might
involve confirming the identity of a person or software program, tracing the
origins of an artifact, or ensuring that a product is what its packaging and
labeling claims to be. Authentication often involves verifying the validity of
at least one form of identification.</em></p>
<h3>Principles</h3>
<p>How to handle authentication in a RESTful Client-Server architecture is a
matter of debate.</p>
<p>Commonly, it can be achieved, in the SOA over HTTP world via:<br />
- HTTP <em>basic auth</em> over HTTPS;<br />
- <em>Cookies</em> and session management;<br />
- <em>Query Authentication</em> with additional signature parameters.</p>
<p>We'll have to adapt, or even better mix those techniques, to match our
framework architecture at best.</p>
<p>Each authentication scheme has its own PROs and CONs, depending on the
purpose of your security policy and software architecture:</p>
<table>
<tbody>
<tr>
<td><strong>Criteria</strong></td>
<td><strong>HTTPS <em>basic auth</em></strong></td>
<td><strong>Cookies+Session</strong></td>
<td><strong>Query Auth.</strong></td>
</tr>
<tr>
<td>Browser integration</td>
<td>Native</td>
<td>Native</td>
<td>Via JavaScript</td>
</tr>
<tr>
<td>User Interaction</td>
<td>Rude</td>
<td>Custom</td>
<td>Custom</td>
</tr>
<tr>
<td>Web Service use<br />
(rough estimation)</td>
<td>95%</td>
<td>4%</td>
<td>1%</td>
</tr>
<tr>
<td>Session handling</td>
<td>Yes</td>
<td>Yes</td>
<td>No</td>
</tr>
<tr>
<td>Session managed by</td>
<td>Client</td>
<td>Server</td>
<td>N/A</td>
</tr>
<tr>
<td>Password on Server</td>
<td>Yes</td>
<td>Yes/No</td>
<td>N/A</td>
</tr>
<tr>
<td>Truly Stateless</td>
<td>Yes</td>
<td>No</td>
<td>Yes</td>
</tr>
<tr>
<td>Truly RESTful</td>
<td>No</td>
<td>No</td>
<td>Yes</td>
</tr>
<tr>
<td>HTTP-free</td>
<td>No</td>
<td>No</td>
<td>Yes</td>
</tr>
</tbody>
</table>
<h3>HTTP basic auth over HTTPS</h3>
<p>This first solution, based on the standard HTTPS protocol, is used by most
web services. It's easy to implement, available by default on all browsers, but
has some known draw-backs, like the awful authentication window displayed on
the Browser, which will persist (there is no <em>LogOut</em>-like feature
here), some server-side additional CPU consumption, and the fact that the
user-name and password are transmitted (over HTTPS) into the Server (it should
be more secure to let the password stay only on the client side, during
keyboard entry, and be stored as secure hash on the Server).</p>
<p>The supplied <code>TSQLHttpClientWinHTTP</code> and
<code>TSQLHttpClientWinINet</code> clients classes are able to connect using
HTTPS, and the <code>THttpApiServer</code> server class can send compatible
content.</p>
<h3>Session via Cookies</h3>
<p>To be honest, a session managed on the Server is not truly Stateless. One
possibility could be to maintain all data within the cookie content. And, by
design, the cookie is handled on the Server side (Client in fact don’t even try
to interpret this cookie data: it just hands it back to the server on each
successive request). But this cookie data is application state data, so the
client should manage it, not the server, in a pure Stateless world.</p>
<p>The cookie technique itself is HTTP-linked, so it's not truly RESTful, which
should be protocol-independent. Since our framework does not provide only HTTP
protocol, but offers other ways of transmission, Cookies were left at the
baker's home.</p>
<h3>Query Authentication</h3>
<p><em>Query Authentication</em> consists in signing each RESTful request via
some additional parameters on the URI. See <a href="http://broadcast.oreilly.com/2009/12/principles-for-standardized-rest-authentication.html">
http://broadcast.oreilly.com/2009/12/principles-for-standardized-rest-authentication.html</a>
about this technique. It was defined as such in this article:</p>
<p><em>All REST queries must be authenticated by signing the query parameters
sorted in lower-case, alphabetical order using the private credential as the
signing token. Signing should occur before URI encoding the query
string.</em></p>
<p>For instance, here is a generic URI sample from the link above:</p>
<pre>
GET /object?apiKey=Qwerty2010
</pre>
<p>should be transmitted as such:</p>
<pre>
GET /object?timestamp=1261496500&apiKey=Qwerty2010&signature=abcdef0123456789
</pre>
<p>The string being signed is
"<code>/object?apikey=Qwerty2010×tamp=1261496500</code>" and the signature is
the <em>SHA256</em> hash of that string using the private component of the API
key.</p>
<p>This technique is perhaps the more compatible with a Stateless architecture,
and can also been implemented with a light session management.</p>
<p>Server-side data caching is always available. In our framework, we cache the
responses at the SQL level, not at the URI level (thanks to our optimized
implementation of <code>GetJSONObjectAsSQL</code>, the URI to SQL conversion is
very fast). So adding this extra parameter doesn't break the cache
mechanism.</p>
<h3>Framework authentication</h3>
<p>Even if, theoretically speaking, <em>Query Authentication</em> sounds to be
the better for implementing a truly RESTful architecture, our framework tries
to implement a Client-Server design.</p>
<p>In practice, we may consider two way of using it:<br />
- With no authentication nor user right management (e.g. for local access of
data, or framework use over a secured network);<br />
- With per-user authentication and right management via defined <em>security
groups</em>, and a per-query authentication.</p>
<p>According to RESTful principle, handling per-session data is not to be
implemented in such architecture. A minimal "session-like" feature was
introduced only to handle user authentication with very low overhead on both
Client and Server side. The main technique used for our security is therefore
<em>Query Authentication</em>, i.e. a per-URI signature.</p>
<p>If both <code>AuthGroup</code> and <code>AuthUser</code> are not available
on the Server <code>TSQLModel</code> (i.e. if the
<code>aHandleUserAuthentication</code> parameter was set to <code>false</code>
for the <code>TSQLRestServer. Create constructor</code>), no authentication is
performed. All tables will be accessible by any client, as stated in 19. As
stated above, for security reasons, the ability to execute <code>INSERT /
UPDATE / DELETE</code> SQL statement via a RESTful <em>POST</em> command is
never allowed by default with remote connections: only <code>SELECT</code> can
be executed via this <em>POST</em> verb.</p>
<p>On the Server side, a dedicated service, accessible via the
<code>ModelRoot/Auth</code> URI is to be called to register an User, and create
a session.</p>
<p>If authentication is enabled for the Client-Server process (i.e. if both
<code>AuthGroup</code> and <code>AuthUser</code> are available in the Server
<code>TSQLModel</code>, and the <code>aHandleUserAuthentication</code>
parameter was set to <code>true</code> at the <code>TSQLRestServer</code>
instance construction), the following security features will be added:<br />
- Client <em>should</em> open a session to access to the Server, and provide a
valid <code>UserName / Password</code> pair (see next paragraph);<br />
- Each CRUD statement is checked against the authenticated User security group,
via the <code>AccessRights</code> column and its <code>GET / POST / PUT /
DELETE</code> per-table bit sets;<br />
- Thanks to <em>Per-User</em> authentication, any SQL statement commands may be
available via the RESTful <em>POST</em> verb for an user with its
<code>AccessRights</code> group field containing
<code>AllowRemoteExecute=true</code>;<br />
- Each REST request will expect an additional parameter, named
<code>session_signature</code>, to every URL. Using the URI instead of
<em>cookies</em> allows the signature process to work with all communication
protocols, not only HTTP.</p>
<h3>Per-User authentication</h3>
<p>On the Server side, two tables, defined by the <code>TSQLAuthGroup</code>
and <code>TSQLAuthUser</code> classes will handle respectively per-group access
rights, and user authentication.</p>
<p>The corresponding <code>AccessRights</code> column is a textual CSV
serialization of the <code>TSQLAccessRights</code> record content, as expected
by the <code>TSQLRestServer.URI</code> method. Using a CSV serialization,
instead of a binary serialization, will allow the change of the
<code>MAX_SQLTABLES</code> constant value.</p>
<p>The <code>AuthUser</code> table, as defined by the <code>TSQLAuthUser</code>
class type.</p>
<p>Each user has therefore its own associated <code>AuthGroup</code> table, a
name to be entered at login, a name to be displayed on screen or reports, and a
SHA-256 hash of its registered password. A custom <code>Data</code> BLOB field
is specified for your own application use, but not accessed by the
framework.</p>
<p>By default, the following security groups are created on a void
database:</p>
<table>
<tbody>
<tr>
<td><strong>AuthGroup</strong></td>
<td><strong>POST SQL</strong></td>
<td><strong>Auth Read</strong></td>
<td><strong>Auth Write</strong></td>
<td><strong>Tables R</strong></td>
<td><strong>Tables W</strong></td>
</tr>
<tr>
<td>Admin</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<td>Supervisor</td>
<td>No</td>
<td>Yes</td>
<td>No</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<td>User</td>
<td>No</td>
<td>No</td>
<td>No</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<td>Guest</td>
<td>No</td>
<td>No</td>
<td>No</td>
<td>Yes</td>
<td>No</td>
</tr>
</tbody>
</table>
<p>Then the corresponding '<em>Admin</em>', '<em>Supervisor</em>' and
'<em>User</em>' <code>AuthUser</code> accounts are created, with the default
'<em>synopse</em>' password.</p>
<p><strong>You MUST override those default '<em>synopse</em>' passwords for
each <code>AuthUser</code> record to a custom genuine value.</strong></p>
<p>'<em>Admin</em>' will be the only group able to execute remote not SELECT
SQL statements for POST commands (i.e. to have <code>TSQLAccessRights.
AllowRemoteExecute = true</code>) and modify the <code>Auth*</code> tables
(i.e. <code>AuthUser</code> and <code>AuthGroup</code>) content.</p>
<p>A typical JSON representation of the default security user/group definitions
are the following:</p>
<pre>
[{"AuthUser":[
{"RowID":1,"LogonName":"Admin","DisplayName":"Admin","PasswordHashHexa":"67aeea294e1cb515236fd7829c55ec820ef888e8e221814d24d83b3dc4d825dd","GroupRights":1,"Data":null},
{"RowID":2,"LogonName":"Supervisor","DisplayName":"Supervisor","PasswordHashHexa":"67aeea294e1cb515236fd7829c55ec820ef888e8e221814d24d83b3dc4d825dd","GroupRights":2,"Data":null},
{"RowID":3,"LogonName":"User","DisplayName":"User","PasswordHashHexa":"67aeea294e1cb515236fd7829c55ec820ef888e8e221814d24d83b3dc4d825dd","GroupRights":3,"Data":null}]},
{"AuthGroup":[
{"RowID":1,"Ident":"Admin","SessionTimeout":10,"AccessRights":"11,1-256,0,1-256,0,1-256,0,1-256,0"},
{"RowID":2,"Ident":"Supervisor","SessionTimeout":60,"AccessRights":"10,1-256,0,3-256,0,3-256,0,3-256,0"},
{"RowID":3,"Ident":"User","SessionTimeout":60,"AccessRights":"10,3-256,0,3-256,0,3-256,0,3-256,0"},
{"RowID":4,"Ident":"Guest","SessionTimeout":60,"AccessRights":"0,3-256,0,0,0,0"}]}]
</pre>
<p>Of course, you can change <code>AuthUser</code> and <code>AuthGroup</code>
table content, to match your security requirements, and application
specifications. You can specify a per-table CRUD access, via the
<code>AccessRights</code> column, as we stated above, speaking about the
<code>TSQLAccessRights</code> record layout.</p>
<p>This will implement both <em>Query Authentication</em> together with a
group-defined <em>per-user right</em> management.</p>
<h3>Session handling</h3>
<p>A dedicated RESTful service, available from the <code>ModelRoot/Auth</code>
URI, is to be used for user authentication, handling so called sessions.</p>
<p>In <em>mORMot</em>, a very light in-memory set of sessions is
implemented:<br />
- The unique <code>ModelRoot/Auth</code> URI end-point will create a session
after proper authorization;<br />
- <em>In-memory</em> session allows very fast process and reactivity, on Server
side;<br />
- An <code>integer</code> <em>session identifier</em> is used for all
authorization process, independently from the underlying authentication scheme
(i.e. <em>mORMot</em> is not tied to cookies, and its session process is much
more generic).</p>
<p>Those sessions are in-memory <code>TAuthSession</code> class instances. Note
that this class does not inherit from a <code>TSQLRecord</code> table so won't
be remotely accessible, for performance and security reasons.</p>
<p>The server methods should not have to access those instances directly, but
rely on the <code>SessionID</code> identifier. The only access available is via
the <code>function TSQLRestServer.SessionGetUser(aSessionID: Cardinal):
TSQLAuthUser</code> method.</p>
<p>When the Client is about to close (typically in
<code>TSQLRestClientURI.Destroy</code>), a <code>GET
ModelRoot/auth?UserName=...&Session=...</code> request is sent to the
remote server, in order to explicitly close the corresponding session in the
server memory (avoiding most <em>re-play</em> attacks).</p>
<p>Note that each opened session has an internal <em>TimeOut</em> parameter
(retrieved from the associated <code>TSQLAuthGroup</code> table content): after
some time of inactivity, sessions are closed on the Server Side.</p>
<p>In addition, sessions are used to manage safe cross-client
transactions:<br />
- When a transaction is initiated by a client, it will store the corresponding
client Session ID, and use it to allow client-safe writing;<br />
- Any further write to the DB (Add/Update/Delete) will be accessible only from
this Session ID, until the transaction is released (via commit or
rollback);<br />
- If a transaction began and another client session try to write on the DB, it
will wait until the current transaction is released - a timeout may occur if
the server is not able to acquire the write status within some time;<br />
- This global write locking is implemented in the <code>TSQLRest.AcquireWrite /
ReleaseWrite</code> protected methods, and used on the Server-Side by
<code>TSQLRestServer.URI</code>;<br />
- If the server do not handle Session/Authentication, transactions can be
unsafe, in a multi-client concurrent architecture.</p>
<p>Therefore, for performance reasons in a multi-client environment, it's
mandatory to release a transaction (via commit or rollback) as soon as
possible.</p>
<h3>Authentication schemes</h3>
<h3>Class-driven authentication</h3>
<p>Authentication is implemented in <em>mORMot</em> via the following
classes:</p>
<table>
<tbody>
<tr>
<td><strong>class</strong></td>
<td><strong>Scheme</strong></td>
</tr>
<tr>
<td><code>TSQLRestServerAuthenticationDefault</code></td>
<td><em>mORMot</em> secure authentication scheme, based on a proprietary
dual-pass challenge and SHA-256 hashing</td>
</tr>
<tr>
<td><code>TSQLRestServerAuthenticationSSPI</code></td>
<td>Windows authentication, via the logged user</td>
</tr>
<tr>
<td><code>TSQLRestServerAuthenticationNone</code></td>
<td>Weak but simple authentication, based on user name</td>
</tr>
</tbody>
</table>
<p>All those classes will identify a <code>TSQLAuthUser</code> record from a
user name. The associated <code>TSQLAuthGroup</code> is then used later for
authorization.</p>
<p>You can add you own custom authentication scheme by defining your own class,
inheriting from <code>TSQLRestServerAuthentication</code>.</p>
<p>By default, no authentication is performed.</p>
<p>If you set the <code>aHandleUserAuthentication</code> parameter to
<code>true</code> when calling the constructor
<code>TSQLRestServer.Create()</code>, both default secure <em>mORMot</em>
authentication and Windows authentication are available. In fact, the
<code>constructor</code> executes the following:</p>
<pre>
<strong>constructor</strong> TSQLRestServer.Create(aModel: TSQLModel; aHandleUserAuthentication: boolean);
(...)
<strong>if</strong> aHandleUserAuthentication <strong>then</strong>
<em>// default mORMot authentication schemes</em>
<span style="background-color:yellow;">AuthenticationRegister([</span>
<span style="background-color:yellow;">TSQLRestServerAuthenticationDefault,TSQLRestServerAuthenticationSSPI]);</span>
(...)
</pre>
<p>In order to define one or several authentication scheme, you can call the
<code>AuthenticationRegister()</code> and
<code>AuthenticationUnregister()</code> methods of
<code>TSQLRestServer</code>.</p>
<h3>mORMot secure RESTful authentication</h3>
<p>The <code>TSQLRestServerAuthenticationDefault</code> class implements a
proprietary but secure RESTful 18.</p>
<p>Here are the typical steps to be followed in order to create a new user
session via <em>mORMot</em> authentication scheme:<br />
- Client sends a <code>GET ModelRoot/auth?UserName=...</code> request to the
remote server;<br />
- Server answers with an hexadecimal <em>nonce</em> contents (valid for about 5
minutes), encoded as JSON result object;<br />
- Client sends a <code>GET
ModelRoot/auth?UserName=...&PassWord=...&ClientNonce=...</code> request
to the remote server, in which <code>ClientNonce</code> is a random value used
as Client <em>nonce</em>, and <code>PassWord</code> is computed from the log-on
and password entered by the User, using both Server and Client <em>nonce</em>
as salt;<br />
- Server checks that the transmitted password is valid, i.e. that its matches
the hashed password stored in its database and a time-valid Server
<em>nonce</em> - if the value is not correct, authentication fails;<br />
- On success, Server will create a new in-memory session and return the session
number and a private key to be used during the session (encoded as JSON result
object);<br />
- On any further access to the Server, a <code>&session_signature=</code>
parameter is added to the URL, and will be checked against the valid sessions
in order to validate the request.</p>
<p><em>Query Authentication</em> is handled at the Client side in
<code>TSQLRestClientURI.SessionSign</code> method, by computing the
<code>session_signature</code> parameter for a given URL, according to the
<code>TSQLRestServerAuthentication</code> class used.</p>
<p>In order to enhance security, the <code>session_signature</code> parameter
will contain, encoded as 3 hexadecimal 32 bit cardinals:<br />
- The Session ID (to retrieve the private key used for the signature);<br />
- A Client Time Stamp (in 256 ms resolution) which must be greater or equal
than the previous time stamp received;<br />
- The URI signature, using the session private key, the user hashed password,
and the supplied Client Time Stamp as source for its <em>crc32</em> hashing
algorithm.</p>
<p>Such a classical 3 points signature will avoid most
<em>man-in-the-middle</em> (MITM) or <em>re-play</em> attacks.</p>
<p>Here is typical signature to access the <code>root</code> URL</p>
<pre>
root?session_signature=0000004C000F6BE365D8D454
</pre>
<p>In this case, <code>0000004C</code> is the Session ID, <code>000F6BE3</code>
is the client time stamp (aka nonce), and <code>65D8D454</code> is the
signature, checked by the following Delphi expression:</p>
<pre>
(crc32(crc32(fPrivateSaltHash,PTimeStamp,8),pointer(aURL),aURLlength)=aSignature);
</pre>
<p>For instance, a RESTful GET of the <code>TSQLRecordPeople</code> table with
RowID=6 will have the following URI:</p>
<pre>
root/People/6?session_signature=0000004C000F6DD02E24541C
</pre>
<p>For better Server-side performance, the URI signature will use fast
<em>crc32</em> hashing method, and not the more secure (but much slower)
SHA-256. Since our security model is not officially validated as a standard
method (there is no standard for per URI authentication of RESTful
applications), the better security will be handled by encrypting the whole
transmission channel, using standard HTTPS with certificates signed by a
trusted CA, validated for both client and server side. The security involved by
using <em>crc32</em> will be enough for most common use. Note that the password
hashing and the session opening will use SHA-256, to enhance security with no
performance penalty.</p>
<p>In our implementation, for better Server-side reaction, the
<code>session_signature</code> parameter is appended at the end of the URI, and
the URI parameters are not sorted alphabetically, as suggested by the reference
article quoted above. This should not be a problem, either from a Delphi Client
or from a AJAX / JavaScript client.</p>
<p>On practice, this scheme is secure and very fast, perfect for a Delphi
client.</p>
<h3>Authentication using Windows credentials</h3>
<p>By default, the <em>hash</em> of the user password is stored safely on the
server side. This may be an issue for corporate applications, since a new user
name / password pair is to be defined by each client, which may be
annoying.</p>
<p>Since revision 1.18 of the framework, <em>mORMot</em> is able to use
<em>Windows Authentication</em> to identify any user. That is, the user does
not need to enter any name nor password, but her/his Windows credentials, as
entered at Windows session startup, will be used.</p>
<p>If the <code>SSPIAUTH</code> conditional is defined (which is the default),
any call to <code>TSQLRestClientURI.SetUser()</code> method with a void
<code>aUserName</code> parameter will try to use current logged name and
password to perform a secure Client-Server authentication. It will in fact call
the <code>class function
TSQLRestServerAuthenticationSSPI.ClientSetUser()</code> method.</p>
<p>In this case, the <code>aPassword</code> parameter will just be ignored.
This will be transparent to the framework, and a regular session will be
created on success.</p>
<p>Only prerequisite is that the <code>TSQLAuthUser</code> table shall contain
a corresponding entry, with its <code>LogonName</code> column equals to
'<em>DomainNameUserName</em>' value. This data row won't be created
automatically, since it is up to the application to allow or disallow access
from an authenticated user: you can be member of the domain, but not eligible
to the application.</p>
<h3>Weak authentication</h3>
<p>The <code>TSQLRestServerAuthenticationNone</code> class can be used if you
trust your client (e.g. via a <code>https</code> connection). It will implement
a weak but simple authentication scheme.</p>
<p>Here are the typical steps to be followed in order to create a new user
session via this authentication scheme:<br />
- Client sends a <code>GET ModelRoot/auth?UserName=...</code> request to the
remote server;<br />
- Server checks that the transmitted user name is valid, i.e. that it is
available in the <code>TSQLAuthGroup</code> table - if the value is not
correct, authentication fails<br />
- On success, Server will create a new in-memory session and returns the
associated session number (encoded as hexadecimal in the JSON result
object);<br />
- On any further access to the Server, a <code>&session_signature=</code>
parameter is to be added to the URL with the correct session ID, and will be
checked against the valid sessions in order to validate the request.</p>
<p>For instance, a RESTful GET of the <code>TSQLRecordPeople</code> table with
RowID=6 will have the following URI:</p>
<pre>
root/People/6?session_signature=0000004C
</pre>
<p>Here is some sample code about how to define this authentication scheme:</p>
<pre>
<em>// on the Server side:</em>
Server.AuthenticationRegister(TSQLRestServerAuthenticationNone);
...
<em>// on the Client side:</em>
TSQLRestServerAuthenticationNone.ClientSetUser(Client,'User');
</pre>
<p>The performance benefit is very small in comparison to
<code>TSQLRestServerAuthenticationDefault</code>, so should not be used for
Delphi clients.</p>
<h3>Clients authentication</h3>
<h3>Client interactivity</h3>
<p>Note that with this design, it's up to the Client to react to an
authentication error during any request, and ask again for the User pseudo and
password at any time to create a new session. For multiple reasons (server
restart, session timeout...) the session can be closed by the Server without
previous notice.</p>
<p>In fact, the Client should just use create one instance of the
<code>TSQLRestClientURI</code> classes as presented in 6, then call the
<code>SetUser</code> method as such:</p>
<pre>
Check(Client.SetUser('User','synopse')); <em>// use default user</em>
</pre>
<p>Then an event handled can be associated to the <code>TSQLRestClientURI.
OnAuthentificationFailed</code> property, in order to ask the user to enter its
login name and password:</p>
<pre>
TOnAuthentificationFailed = <strong>function</strong>(Retry: integer;
<strong>var</strong> aUserName, aPassword: <strong>string</strong>): boolean;
</pre>
<p>Of course, if <em>Windows Authentication</em> is defined (see above), this
event handler shall be adapted as expected. For instance, you may add a custom
notification to register the corresponding user to the
<code>TSQLAuthUser</code> table.</p>
<h3>Authentication using AJAX</h3>
<p>Some working JavaScript code has been published in our forum by a framework
user (thanks, "RangerX"), which implements the authentication schema as
detailed above. It uses <code>jQuery</code>, and HTML 5
<code>LocalStorage</code>, not cookies, for storing session information on the
Client side.</p>
<p>See <a href="http://synopse.info/forum/viewtopic.php?pid=2995#p2995">http://synopse.info/forum/viewtopic.php?pid=2995#p2995</a></p>
<p>The current revision of the framework contains the code as expected by this
JavaScript code - especially the results encoded as 2 objects.</p>
<p>In the future, some "official" code will be available for such AJAX clients.
It will probably rely on pure-pascal implementation using such an
<em>Object-Pascal-to-JavaScript</em> compiler - it does definitively make sense
to have Delphi-like code on the client side, not to break the ORM design. For
instance, the Open Source <code>DWS</code> (<em>DelphiWebScript</em>) compiler
matches our needs - see <a href="http://delphitools.info/tag/javascript">http://delphitools.info/tag/javascript</a></p>
<h3>Authorization</h3>
<h3>Per-table access rights</h3>
<p>Even if authentication is disabled, a pointer to a
<code>TSQLAccessRights</code> record, and its <code>GET / POST / PUT /
DELETE</code> fields, is sent as a member of the parameter to the unique access
point of the server class:</p>
<pre>
<strong>procedure</strong> TSQLRestServer.URI(<strong>var</strong> Call: TSQLRestServerURIParams);
</pre>
<p>This will allow checking of access right for all CRUD operations, according
to the table invoked. For instance, if the table <code>TSQLRecordPeople</code>
has 2 as index in <code>TSQLModel.Tables[]</code>, any incoming
<code>POST</code> command for <code>TSQLRecordPeople</code> will be allowed
only if the 2nd bit in <code>RestAccessRights^.POST</code> field is set, as
such:</p>
<pre>
<strong>case</strong> URI.Method <strong>of</strong>
mPOST: <strong>begin</strong> <em>// POST=ADD=INSERT</em>
<strong>if</strong> URI.Table=<strong>nil then begin</strong>
(...)
<strong>end else</strong>
<em>// here, Table<>nil and TableIndex in [0..MAX_SQLTABLES-1]</em>
<span style="background-color:yellow;"><strong>if not</strong> (URI.TableIndex <strong>in</strong> Call.RestAccessRights^.POST) <strong>then</strong> <em>// check User</em></span>
<span style="background-color:yellow;">Call.OutStatus := HTML_FORBIDDEN <strong>else</strong></span>
(...)
</pre>
<p>Making access rights a parameter allows this method to be handled as pure
stateless, thread-safe and session-free, from the bottom-most level of the
framework.</p>
<p>On the other hand, the security policy defined by this global parameter does
not allow tuned per-user authorization. In the current implementation, the
<code>SUPERVISOR_ACCESS_RIGHTS</code> constant is transmitted for all handled
communication protocols (direct access, GDI messages, named pipe or HTTP). Only
direct access via <code>TSQLRestClientDB</code> will use
<code>FULL_ACCESS_RIGHTS</code>, i.e. will have <code>AllowRemoteExecute</code>
parameter set to <code>true</code>.</p>
<p>The light session process, as implemented by 18, is used to override the
access rights with the one defined in the
<code>TSQLAuthGroup.AccessRights</code> field.</p>
<p>Be aware than this per-table access rights depend on the table order as
defined in the associated <code>TSQLModel</code>. So if you add some tables to
your database model, please take care to add the new tables <em>after</em> the
existing. If you insert the new tables within current tables, you will need to
update the access rights values.</p>
<h3>Additional safety</h3>
<p>A <code>AllowRemoteExecute: TSQLAllowRemoteExecute</code> field has been
made available in the <code>TSQLAccessRights</code> record to tune remote
execution, depending on the authenticated user.</p>
<p>It adds some options to tune the security policy.</p>
<h3>SQL remote execution</h3>
<p>In our RESTful implementation, the POST command with no table associated in
the URI allows to execute any SQL statement directly.</p>
<p>This special command should be carefully tested before execution, since SQL
misuses could lead into major security issues. Such execution on any remote
connection, if the SQL statement is not a SELECT, is unsafe. In fact, if it may
affect the data content.</p>
<p>By default, for security reasons, this <code>AllowRemoteExecute</code> field
value in <code>SUPERVISOR_ACCESS_RIGHTS</code> constant does not include
<code>reSQL</code>. It means that no remote call will be allowed but for safe
read-only SELECT statements.</p>
<p>Another possibility of SQL remote execution is to add a
<code>sql=....</code> inline parameter to a GET request (with optional paging).
The <code>reUrlEncodedSQL</code> option is used to enable or disable this
feature.</p>
<p>Last but not least, a <code>WhereClause=...</code> inline parameter can be
added to a DELETE request. The <code>reUrlEncodedDelete</code> option is used
to enable or disable this feature.</p>
<p>You can change the default safe policy by including <code>reSQL</code>,
<code>reUrlEncodedSQL</code> or <code>reUrlEncodedDelete</code> in the
<code>TSQLAuthGroup.AccessRights</code> field if an authentication user
session. But since remote execution of any SQL statements can be unsafe, we
recommend to write a dedicated server-side service (method-based or
interface-based) to execute such statements.</p>
<h3>Service execution</h3>
<p>The <code>reService</code> option can be used to enable or unable the
interface-based services feature of <em>mORMot</em>.</p>
<p>In addition to this global parameter, you can set per-service and per-method
security via dedicated methods.</p>
<p>For method-based services, if authentication is enabled, any method
execution will be processed only for signed URI.</p>
<p>You can use <code>TSQLRestServer.ServiceMethodByPassAuthentication()</code>
to disable the need of a signature for a given service method - e.g. it is the
case for <code>Auth</code> and <code>TimeStamp</code> standard method
services.</p>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?pid=7864#p7864">welcome on our
forum</a>, as usual.</p>