2021-02-22

OpenSSL 1.1.1 Support for mORMot 2

Why OpenSSL?

OpenSSL is the reference library for cryptography and secure TLS/HTTPS communication. It is part of most Linux/BSD systems, and covers a lot of use cases and algorithms. Even if it had some vulnerabilities in the past, it has been audited and validated for business use. Some algorithms have been deeply optimized, with very tuned assembly code.

mORMot 2 can now access all OpenSSL 1.1.1 features, on all supported targets.
OpenSSL 1.1.1 is the latest stable branch, with LTS support, and nice new additions like TLS 1.3 and ed25519.

On Windows, you can use our internal assembly code for encryption, hashing and randomness (e.g. AES or SHA2/3), and we provide SChannel support since mORMot 1, which allows to create a TLS/HTTPS layer without shipping any external library - using OpenSSL .dll files can be a real PITA on Windows, since it is very likely that several versions do exist on your systems, even in your path! We just discovered and fixed some problems with SChannel, which seems now stable and working fine even on latest Windows revisions.

Of course, if you really need to use OpenSSL on Windows, you can, and our little mORMot will try to do its best to find the right .dll for you. Windows' SChannel is behind OpenSSL in terms of ciphers support, and also about performance - especially if your Windows version is old or not updated. But it is part of the system, so you can rely on it if you expect basic TLS support.

On Linux / BSD / POSIX, OpenSSL 1.1.1 is the reference cryptographic library. It is already installed on any recent system.
On MacOS and Android, you can use static linking to your application.

OpenSSL 1.1.1 in mORMot 2

Our OpenSSL integration in mORMot 2 has several key features:

  • Split into two units: mormot.lib.openssl11.pas for the raw API, and mormot.core.crypto.openssl.pas for integration with the other mORMot 2 features;
  • mormot.lib.openssl11.pas leverages OpenSSL 1.1.1, which is the latest stable version, and the current TLS branch - when OpenSSL 3 will be released, the API should remain compatible;
  • mormot.lib.openssl11.pas by default loads dynamically OpenSSL 1.1.1 libraries, so even if your system won't have OpenSSL installed, your application will still start, with explicit error messages when you would try to use OpenSSL features;
  • mormot.lib.openssl11.pas has a "full-API" conditional, which is used as reference when adding new features, and could be defined in your projects if you need specific OpenSSL features;
  • mormot.lib.openssl11.pas publishes a TLS layer, recognized by our mormot.net.sock.pas unit, just like SChannel on Windows, to enabled TLS/HTTPS client connections;
  • mormot.core.crypto.openssl.pas exposes the OpenSSL PRNG, which may be used for regulatory purposes, if our own PRNG doesn't meet your requirements;
  • mormot.core.crypto.openssl.pas exposes the main (and safest) AES ciphers, and Hashing/Signing offered by OpenSSL, in a mormot.core.crypto.pas compatible way;
  • mormot.core.crypto.openssl.pas defines some JWT classes, with full coverage of all the algorithms defined by the official JWT website (it is the only Delphi library defining all of them);
  • mormot.core.crypto.openssl.pas replaces the mormot.core.crypto.pas and mormot.core.ecc256r1.pas functions with the OpenSSL version, if they are faster;
  • We (re)validated our mormot.core.crypto.pas and mormot.core.ecc256r1.pas implementation against OpenSSL as reference, and also enhanced the test coverage (with more reference vectors for instance) for mORMot 2.

In a nutshell, OpenSSL is slower than our mormot.core.crypto.pas assembly on x86_64 for most AES process (ciphers and PRNG) - but AES-GCM. But OpenSSL is much faster than mormot.core.ecc256r1.pas for ECC public key cryptography, so is automatically selected, which is a good idea e.g. on a Linux server environment.

AES-PRNG, AES-CTR and AES-GCM

See this article for actual performance numbers of the AES encryption, and how our internal mormot.core.crypto.pas assembly is faster than OpenSSL in most cases.
Our x86_64 tuned assembly, with full AES-NI and PCLMUL support does wonders.

Public Key Cryptography and JWT

The mormot.core.crypto.openssl.pas unit covers all possible JWT algorithms, with very interesting results, as shown by our regression tests:

  mormot.core.crypto.pas algorithms:
     1000 HS256 in 2.16ms i.e. 461,041/s, aver. 2us
     1000 HS384 in 2.19ms i.e. 456,204/s, aver. 2us
     1000 HS512 in 2.18ms i.e. 457,456/s, aver. 2us
     1000 S3224 in 1.91ms i.e. 521,376/s, aver. 1us
     1000 S3256 in 1.89ms i.e. 526,870/s, aver. 1us
     1000 S3384 in 1.92ms i.e. 518,941/s, aver. 1us
     1000 S3512 in 1.94ms i.e. 513,874/s, aver. 1us
     1000 S3S128 in 1.92ms i.e. 520,833/s, aver. 1us
     1000 S3S256 in 1.94ms i.e. 513,610/s, aver. 1us
     100 ES256 in 14.65ms i.e. 6,823/s, aver. 146us
  mormot.core.crypto.openssl.pas algorithms:
     100 RS256 in 5.35ms i.e. 18,674/s, aver. 53us
     100 RS384 in 5.16ms i.e. 19,368/s, aver. 51us
     100 RS512 in 5.16ms i.e. 19,357/s, aver. 51us
     100 PS256 in 5.56ms i.e. 17,985/s, aver. 55us
     100 PS384 in 5.52ms i.e. 18,102/s, aver. 55us
     100 PS512 in 5.65ms i.e. 17,677/s, aver. 56us
     100 ES256 in 14.38ms i.e. 6,951/s, aver. 143us
     100 ES384 in 123.79ms i.e. 807/s, aver. 1.23ms
     100 ES512 in 94.50ms i.e. 1,058/s, aver. 945us
     100 ES256K in 63.09ms i.e. 1,584/s, aver. 630us
     100 EdDSA in 18.83ms i.e. 5,310/s, aver. 188us

The timing above are to verify a JWT signature.
We can see that the ES256 has very tuned assembly code, and that the other ECC algorithms (ES384/ES512/ES256K) use standard coding, which are much slower. EdDSA is the ed25519 public cryptography algorithm, which is efficient on non-Intel/AMD platforms too, e.g. on ARM.

Raw asymmetric / public key cryptography using OpenSSL is therefore consistent with the expectations:

     3 RSA 2048 Generation in 184.43ms i.e. 16/s, aver. 61.47ms
     30 RSA 2048 Sign in 48.66ms i.e. 616/s, aver. 1.62ms
     30 RSA 2048 Verify in 1.53ms i.e. 19,505/s, aver. 51us
     3 RSA-PSS 2048 Generation in 205.60ms i.e. 14/s, aver. 68.53ms
     30 RSA-PSS 2048 Sign in 47.32ms i.e. 633/s, aver. 1.57ms
     30 RSA-PSS 2048 Verify in 1.57ms i.e. 19,047/s, aver. 52us
     100 prime256v1 Generation in 8.26ms i.e. 12,096/s, aver. 82us
     100 prime256v1 Sign in 7.02ms i.e. 14,226/s, aver. 70us
     100 prime256v1 Verify in 13.45ms i.e. 7,433/s, aver. 134us
     100 ed25519 Generation in 7.33ms i.e. 13,642/s, aver. 73us
     100 ed25519 Sign in 13.67ms i.e. 7,310/s, aver. 136us
     100 ed25519 Verify in 18.44ms i.e. 5,422/s, aver. 184us

RSA generation is really slow in comparison to other options (but verification is fast). In practice, prime256v1 (ES256) is to be privileged - especially since we offer our own pure standalone version if OpenSSL is not available.
Note that any other algorithm exposed by OpenSSL are available - you are not restricted to the list above.

We introduced an easy way to generate public/private key pairs in a single function call, which is mandatory for any automated server work. No need to try the obfuscated openssl ecparam -name prime256v1 -genkey -noout -out key.pem && openssl ec -in key.pem -pubout -out public.pem command line switches: just call OpenSslGenerateKeys() or TJwtAbstractOsl.GenerateKeys and you are ready to go!

TLS Layer

We integrated the OpenSSL TLS layer to our mormot.net.sock.pas unit, with a powerful and easy way of using it:

with THttpClientSocket.Create do
try
  TLS.WithPeerInfo := true;
  TLS.IgnoreCertificateErrors := true;
  TLS.CipherList := 'ECDHE-RSA-AES256-GCM-SHA384';
  OpenBind('synopse.info', '443', {bind=}false, {tls=}true);
  writeln(TLS.PeerInfo);
  writeln(TLS.CipherName);
  writeln(Get('/forum/', 1000), ' len=', ContentLength);
  writeln(Get('/fossil/wiki/Synopse+OpenSource', 1000));
finally
  Free;
end;

The TCrtSocket.TLS new field can be used to set the private and public keys or root certificates, if needed. Its PeerInfo return the whole certificate of the server side, and CipherName returns the current cipher used, e.g. 'ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD'.

The very same code can be used with the SChannel version of mormot.net.sock.windows.inc - but with less features by now.

OpenSource ForEver

The source code of the units is available for review:

It works for FPC and Delphi, in the usual targets supported by mORMot 2.

Interfacing OpenSSL is really tricky sometimes, due to the opaque API names, lack of documentation, and weak/deprecated samples on the net.
We tried to leverage our integration, so that using OpenSSL would be seamless for the mORMot 2 users: you just use the mORMot classes, and don't have to care about which engine (mORMot's or OpenSSL's) is actually running them.

And feedback is welcome on our forum, as usual!

2021-02-13

Fastest AES-PRNG, AES-CTR and AES-GCM Delphi implementation

Last week, I committed new ASM implementations of our AES-PRNG, AES-CTR and AES-GCM for mORMot 2.
They handle eight 128-bit at once in an interleaved fashion, as permitted by the CTR chaining mode. The aes-ni opcodes (aesenc aesenclast) are used for AES process, and the GMAC of the AES-GCM mode is computed using the pclmulqdq opcode.

Resulting performance is amazing: on my simple Core i3, I reach 2.6 GB/s for aes-128-ctr, and 1.5 GB/s for aes-128-gcm for instance - the first being actually faster than OpenSSL!

Continue reading

2021-02-12

New AesNiHash for mORMot 2

I have just committed some new AesNiHash32 AesNiHash64 AesNiHash128 Hashers for mORMot 2. They are using AES-NI and SSE4.1 opcodes on x86_64 and i386. This implementation is faster than the fastest SSE4.1 crc32c and with a much higher usability (less collisions). Logic was extracted from the Go  […]

Continue reading

2020-12-29

mORMot 2 Proposal: Rename RawUTF8 Type As Utf8 ?

One proposal for mORMot 2. What if we renamed the RawUTF8 type into Utf8? With a default compatibility redirection if PUREMORMOT2 is not defined, of course. The "Raw" prefix came from early mORMot code, which used TRichView as reference for the UTF-8 encoding... but it is clearly an  […]

Continue reading

2020-11-16

mORMot 2 Entering Testing Phase

mormot2test.jpg, Nov 2020

After a lot of work, our mORMot 2 fork is entering its testing phase.

The main /src/core /src/lib /src/net /src/db /src/orm /src/soa /src/app folders of our Source Code repository have been implemented.

mormot2test.jpg, Nov 2020

Please check https://github.com/synopse/mORMot2 for the latest version of the source code. The README.md files on each folder would help you discover the new framework design, and the content of each unit.

Continue reading

2020-11-04

EKON 24 Presentation Slides

EKON_24.png, Nov 2020

EKON 24 just finished. "The conference for Delphi & more" was fully online this year, due to the viral context... But this was a great event, and I am very happy to have been part of it. Please find the slides on my two sessions: mORMot 2 Performance: from Delphi to AVX2 Of course,  […]

Continue reading

2020-10-26

mORMot2 Renaming

Last weeks, we introduced REST, ORM and SOA process in the mORMot2 repository.

During this phase, we split the huge mORMot.pas unit into several mormot.rest.*.pas, mormot.orm.*.pas and mormot.soa.*.pas units, to follow SOLID principles.

But we also renamed the base types into something more consistent and easier to work with. Forget about TSQLRecord or TSQLRest, discover TORM and TRest!

Continue reading

2020-09-09

Data Alignment and Delphi 10.4.1

Some regression has been reported with Delphi 10.4.1 and SynPDF. From the Github issue description: Generating a PDF via VLCCanvas and TPdfDocumentGDI causes access violation when compiled with Delphi 10.4.1 with record field alignment compiler option set to "byte" or "off". When  […]

Continue reading

2020-08-11

The RFC, The URI, and The Tilde

For several reasons, only plain ASCII characters are accepted in Web URIs. Other characters should be escaped with % and the hexadecimal value of its code.

The tilde character ~ is not needed to be escaped... at least in theory... because in practice most code expects it...

A journey into confusing RFCs...

Continue reading

2020-07-20

Special Care of Delphi 10.4

sillybug.jpg, Jul 2020

A regression in the Delphi 10.4 compiler was identified. Its optimizer wrongly deletes some code, in one very specific part of the framework.

sillybug.jpg, Jul 2020

As a result a GPF (Access violation) may be triggered with Delphi 10.4 in release mode - the debug mode (when optimization is disabled) has no problem. Thanks to great user feedback, we were able to circumvent it. But we should better stay in alert, like any mORMot, until Delphi 10.4 officially release a patch.

Continue reading

2020-07-02

mORMot on GitHub Sponsors

We just enrolled on GitHub Sponsors!

Aim is to allow proper evolution of our Open Source project, especially the upcoming mORMot2.

Continue reading

2020-06-05

SQlite3 Encryption Not Possible Any More Since 3.32.x

About latest SQlite3 3.32.xxx there is a big problem with codecs.

Critical changes to the public SQLite code were introduced on Feb 7, 2020: “Simplify the code by removing the unsupported and undocumented SQLITE_HAS_CODEC compile-time option”. With the release of SQLite version 3.32.0 on May 22, 2020 these changes finally took officially effect, although they weren't officially announced.

As a sad and unexpected consequence, we are NOT ANY MORE able to compile the new SQlite3 amalgamation with our encryption patch.

Continue reading

2020-05-07

New Multi-thread Friendly Memory Manager for FPC written in x86_64 assembly

As a gift to the FPC community, I just committed a new Memory Manager for FPC.
Check mormot.core.fpcx64mm.pas in our mORMot2 repository.
This is a stand-alone unit for FPC only.

It targets Windows and Linux multi-threaded Service applications - typically mORMot daemons.
It is written in almost pure x86_64 assembly, and some unique tricks in the Delphi/FPC Memory Manager world.

It is based on FastMM4 (not FastMM5), and we didn't follow the path of the FastMM4-AVX version - instead of AVX, we use plain good (non-temporal) SSE2 opcode, and we rely on the mremap API on Linux for very efficient reallocation. Using mremap is perhaps the biggest  benefit of this memory manager - it leverages a killer feature of the Linux kernel for sure. By the way, we directly call the Kernel without the need of the libc.

We tuned our x86_64 assembly a lot, and made it cross-platform (Windows and POSIX). We profiled the multi-threading, especially by adding some additional small blocks for GetMem (which is a less expensive notion of "arenas" as used in FastMM5 and most C allocators), introducing an innovatice and very efficient round-robin of tiny blocks (<128 bytes), and proper spinning for FreeMem and medium blocks.

It runs all our regression tests with huge performance and stability - including multi-threaded tests with almost no slow down: sleep is reported as less than 1 ms during a 1 minute test. It has also been validated on some demanding multi-threaded tasks.

Continue reading

2020-03-30

Debriefing of mORMot2 Survey

Thanks you all for have posted your feedback on our mORMot2 Survey!

Here are some insights.

Continue reading

2020-03-28

Faster Double-To-Text Conversion

On server side, a lot of CPU is done processing conversions to or from text. Mainly JSON these days.

In mORMot, we take care a lot about performance, so we have rewritten most conversion functions to have something faster than the Delphi or FPC RTL can offer.
Only float to text conversion was not available. And RTL str/floattexttext performance, at least under Delphi, is not consistent among platforms.
So we just added a new Double-To-Text set of functions.

Continue reading

2020-03-06

We Need U: Survey about mORMot 2.0

First of all, if it was not clear enough: Delphi will continue to be supported in mORMot 2.0. Some people reported that our previous article may have been misleading. But perhaps not all versions. For sure, Delphi 5 and Kylix will not be supported in mORMot 2. It is also possible that it would not  […]

Continue reading

2020-03-03

Preparing Revision 2.x of the mORMot Framework

The more I think of it, the more I am convinced it is time to change how the framework is versioned.
We have version 1.18 since years... difficult to follow... time to upgrade!


I would like to upgrade mORMot to version 2 - with a major refactoring.

Continue reading

2020-02-17

New move/fillchar optimized sse2/avx asm version

Our Open Source framework includes some optimized asm alternatives to RTL's move() and fillchar(), named MoveFast() and FillCharFast().

We just rewrote from scratch the x86_64 version of those, which was previously taken from third-party snippets.
The brand new code is meant to be more efficient and maintainable. In particular, we switched to SIMD 128-bit SSE2 or 256bit AVX memory access (if available), whereas current version was using 64-bit regular registers. The small blocks (i.e. < 32 bytes) process occurs very often, e.g. when processing strings, so has been tuned a lot. Non temporal instructions (i.e. bypassing the CPU cache) are used for biggest chunks of data. We tested ERMS support, but it was found of no benefit in respect to our optimized SIMD, and was actually slower than our non-temporal variants. So ERMS code is currently disabled in the source, and may be enabled on demand by a conditional.

FPC move() was not bad. Delphi's Win64 was far from optimized - even ERMS was poorly introduced in latest RTL, since it should be triggered only for blocks > 2KB. Sadly, Delphi doesn't support AVX assembly yet, so those opcodes would be available only on FPC.

Resulting numbers are talking by themselves. Working on Win64 and Linux, of course.

Continue reading

2019-12-25

Merry Christmas and Happy New Year!

Let the little mORMot wish you and all yours a merry Christmas and a happy New Year! Happy coding!

2019-10-30

EKON 23 Presentation Slides and Code

I just finished my workshop at EKON 23.
Like every year, it was a great event to attempt to, and I enjoyed presenting 2 sessions and 1 workshop.

Sessions were about "Kingdom Driven Design" (KDD), which is the name I used to define a cut-down version of "Domain Driven Design" (DDD).
Less paranoid, a bit less isolation, but perhaps more common sense for the less sensitive projects.
Some presentations and code are now available!

Continue reading

- page 1 of 19