2021-05-08

Enhanced Faster ZIP Support in mORMot 2

The .zip format is from last century, back to the early DOS days, but can still be found everywhere. It is even hidden when you run a .docx document, a .jar application, or any Android app!
It is therefore (ab)used not only as archive format, but as application file format / container - even if in this respect using SQLite3 may have much more sense.

We recently enhanced our mormot.core.zip.pas unit:

  • to support Zip64,
  • with enhanced .zip read/write,
  • to have a huge performance boost during its process,
  • and to integrate better with signed executables.

Zip64 Support for Huge Files

First of all, our unit now supports the long-awaited Zip64 extension. In a nutshell, it allows to store files bigger than 4GB, or have the total .zip bigger than 4GB - which is the maximum 32-bit stored size.

TZipRead TZipWrite Enhancements

The high-level TZipRead and TZipWrite classes were deeply refactored and enhanced. Not only Zip64 support has been added, but can ignore and skip some files during reading - a very efficient way of deleting files in a .zip. Some additional methods have been introduced, e.g. to quickly validate a .zip file integrity. Cross-platform support has been enhanced.

Previous TZipRead used to map the whole .zip file into memory. It was convenient for small content. But huge files won't fit into a Win32 application memory: you could not use Zip64 on 32-bit executables - not very convenient for sure! And performance of memory-mapped files is typically slower than explicit Seek/Read calls, since the Kernel is involved to handle page faults and read the data from disk. This had to be fixed.
Now a memory buffer size is specified to TZipRead.Create constructors, which will contain the last bytes of the .zip file, in which the directory header would appear - and very efficiently parsed at opening. Then, actual content decompression would use regular Seek/Read calls, only when needed. Of course, if the data is available in the memory buffer - which is the case for the last files, or for smaller .zip - it will take it from there. So the new approach seems a very reasonable implementation - typically faster than other zip library I have seen, and our previous code.

LibDeflate Support

Perhaps the main change of this refactoring, is the libdeflate library integration. It is a library for fast, whole-buffer DEFLATE-based compression and decompression. In practice, when working on memory buffers (not streams), it is able to leverage very efficient ASM code for modern CPUs (like AVX), resulting to be much faster than any other zlib implementation around. If streams are involved - e.g. when decompressing huge files - then we fallback to the regular zlib code.

LibDeflate implementation of crc or adler checsums is astonishing: on my Intel Core i5 crc() went from 800MB/s to 10GB/s. And this crc is used for .zip file checksums, so it really helps.
Also compression and decompression are almost twice faster than regular zlib, thanks to a full rewrite of the deflate engine, targeting modern CPUs, and using tuned asm for bottlenecks.
Last but not least, you can use higher compression levels - regular zlib understand from 0 (stored) to 9 (slowest), but libdeflate accepts 10..12 for even higher compression - at the expense of compression speed which becomes very slow, but decompression will be in par with other levels.

We statically linked libdeflate, so you don't need to have an external library. Sadly, it is currently available for FPC only, since Delphi linking is an incompatible mess.

Note that libdeflate will be used anywhere in mORMot where deflate/zip buffer compression is involved, so for instance regular HTTP/HTTPS on-the-fly gzip compression will be much faster, and even some unexpected part of the framework would benefit from it - e.g. our default RESTful URI authentication used the zlib crc() for its online checksum, so each REST request is slightly faster.

Integrated to Signed Executables

The last enhancement was also the ability to append a .zip content to an existing "signed" executable. Since "mORMot 1", we allowed to find and read any .zip content appended to an executable. But if you digitally signed this executable, you would need to re-sign it after appending. Not very convenient, e.g. when you build a custom Setup.exe.

We added some functions to include the .zip content within the signature itself, allowing to store some additional data or configuration in a convenient format, without requiring to sign the executable again.

Use the Source, Luke!

Check the mormot.core.zip.pas unit in our Open Source repository!

2021-02-22

OpenSSL 1.1.1 Support for mORMot 2

Why OpenSSL? OpenSSL is the reference library for cryptography and secure TLS/HTTPS communication. It is part of most Linux/BSD systems, and covers a lot of use cases and algorithms. Even if it had some vulnerabilities in the past, it has been audited and validated for business use. Some algorithms  […]

Continue reading

2021-02-13

Fastest AES-PRNG, AES-CTR and AES-GCM Delphi implementation

Last week, I committed new ASM implementations of our AES-PRNG, AES-CTR and AES-GCM for mORMot 2.
They handle eight 128-bit at once in an interleaved fashion, as permitted by the CTR chaining mode. The aes-ni opcodes (aesenc aesenclast) are used for AES process, and the GMAC of the AES-GCM mode is computed using the pclmulqdq opcode.

Resulting performance is amazing: on my simple Core i3, I reach 2.6 GB/s for aes-128-ctr, and 1.5 GB/s for aes-128-gcm for instance - the first being actually faster than OpenSSL!

Continue reading

2021-02-12

New AesNiHash for mORMot 2

I have just committed some new AesNiHash32 AesNiHash64 AesNiHash128 Hashers for mORMot 2. They are using AES-NI and SSE4.1 opcodes on x86_64 and i386. This implementation is faster than the fastest SSE4.1 crc32c and with a much higher usability (less collisions). Logic was extracted from the Go  […]

Continue reading

2020-12-29

mORMot 2 Proposal: Rename RawUTF8 Type As Utf8 ?

One proposal for mORMot 2. What if we renamed the RawUTF8 type into Utf8? With a default compatibility redirection if PUREMORMOT2 is not defined, of course. The "Raw" prefix came from early mORMot code, which used TRichView as reference for the UTF-8 encoding... but it is clearly an  […]

Continue reading

2020-11-16

mORMot 2 Entering Testing Phase

mormot2test.jpg, Nov 2020

After a lot of work, our mORMot 2 fork is entering its testing phase.

The main /src/core /src/lib /src/net /src/db /src/orm /src/soa /src/app folders of our Source Code repository have been implemented.

mormot2test.jpg, Nov 2020

Please check https://github.com/synopse/mORMot2 for the latest version of the source code. The README.md files on each folder would help you discover the new framework design, and the content of each unit.

Continue reading

2020-11-04

EKON 24 Presentation Slides

EKON_24.png, Nov 2020

EKON 24 just finished. "The conference for Delphi & more" was fully online this year, due to the viral context... But this was a great event, and I am very happy to have been part of it. Please find the slides on my two sessions: mORMot 2 Performance: from Delphi to AVX2 Of course,  […]

Continue reading

2020-10-26

mORMot2 Renaming

Last weeks, we introduced REST, ORM and SOA process in the mORMot2 repository.

During this phase, we split the huge mORMot.pas unit into several mormot.rest.*.pas, mormot.orm.*.pas and mormot.soa.*.pas units, to follow SOLID principles.

But we also renamed the base types into something more consistent and easier to work with. Forget about TSQLRecord or TSQLRest, discover TORM and TRest!

Continue reading

2020-09-09

Data Alignment and Delphi 10.4.1

Some regression has been reported with Delphi 10.4.1 and SynPDF. From the Github issue description: Generating a PDF via VLCCanvas and TPdfDocumentGDI causes access violation when compiled with Delphi 10.4.1 with record field alignment compiler option set to "byte" or "off". When  […]

Continue reading

2020-08-11

The RFC, The URI, and The Tilde

For several reasons, only plain ASCII characters are accepted in Web URIs. Other characters should be escaped with % and the hexadecimal value of its code.

The tilde character ~ is not needed to be escaped... at least in theory... because in practice most code expects it...

A journey into confusing RFCs...

Continue reading

2020-07-20

Special Care of Delphi 10.4

sillybug.jpg, Jul 2020

A regression in the Delphi 10.4 compiler was identified. Its optimizer wrongly deletes some code, in one very specific part of the framework.

sillybug.jpg, Jul 2020

As a result a GPF (Access violation) may be triggered with Delphi 10.4 in release mode - the debug mode (when optimization is disabled) has no problem. Thanks to great user feedback, we were able to circumvent it. But we should better stay in alert, like any mORMot, until Delphi 10.4 officially release a patch.

Continue reading

2020-07-02

mORMot on GitHub Sponsors

We just enrolled on GitHub Sponsors!

Aim is to allow proper evolution of our Open Source project, especially the upcoming mORMot2.

Continue reading

2020-06-05

SQlite3 Encryption Not Possible Any More Since 3.32.x

About latest SQlite3 3.32.xxx there is a big problem with codecs.

Critical changes to the public SQLite code were introduced on Feb 7, 2020: “Simplify the code by removing the unsupported and undocumented SQLITE_HAS_CODEC compile-time option”. With the release of SQLite version 3.32.0 on May 22, 2020 these changes finally took officially effect, although they weren't officially announced.

As a sad and unexpected consequence, we are NOT ANY MORE able to compile the new SQlite3 amalgamation with our encryption patch.

Continue reading

2020-05-07

New Multi-thread Friendly Memory Manager for FPC written in x86_64 assembly

As a gift to the FPC community, I just committed a new Memory Manager for FPC.
Check mormot.core.fpcx64mm.pas in our mORMot2 repository.
This is a stand-alone unit for FPC only.

It targets Windows and Linux multi-threaded Service applications - typically mORMot daemons.
It is written in almost pure x86_64 assembly, and some unique tricks in the Delphi/FPC Memory Manager world.

It is based on FastMM4 (not FastMM5), and we didn't follow the path of the FastMM4-AVX version - instead of AVX, we use plain good (non-temporal) SSE2 opcode, and we rely on the mremap API on Linux for very efficient reallocation. Using mremap is perhaps the biggest  benefit of this memory manager - it leverages a killer feature of the Linux kernel for sure. By the way, we directly call the Kernel without the need of the libc.

We tuned our x86_64 assembly a lot, and made it cross-platform (Windows and POSIX). We profiled the multi-threading, especially by adding some additional small blocks for GetMem (which is a less expensive notion of "arenas" as used in FastMM5 and most C allocators), introducing an innovatice and very efficient round-robin of tiny blocks (<128 bytes), and proper spinning for FreeMem and medium blocks.

It runs all our regression tests with huge performance and stability - including multi-threaded tests with almost no slow down: sleep is reported as less than 1 ms during a 1 minute test. It has also been validated on some demanding multi-threaded tasks.

Continue reading

2020-03-30

Debriefing of mORMot2 Survey

Thanks you all for have posted your feedback on our mORMot2 Survey!

Here are some insights.

Continue reading

2020-03-28

Faster Double-To-Text Conversion

On server side, a lot of CPU is done processing conversions to or from text. Mainly JSON these days.

In mORMot, we take care a lot about performance, so we have rewritten most conversion functions to have something faster than the Delphi or FPC RTL can offer.
Only float to text conversion was not available. And RTL str/floattexttext performance, at least under Delphi, is not consistent among platforms.
So we just added a new Double-To-Text set of functions.

Continue reading

2020-03-06

We Need U: Survey about mORMot 2.0

First of all, if it was not clear enough: Delphi will continue to be supported in mORMot 2.0. Some people reported that our previous article may have been misleading. But perhaps not all versions. For sure, Delphi 5 and Kylix will not be supported in mORMot 2. It is also possible that it would not  […]

Continue reading

2020-03-03

Preparing Revision 2.x of the mORMot Framework

The more I think of it, the more I am convinced it is time to change how the framework is versioned.
We have version 1.18 since years... difficult to follow... time to upgrade!


I would like to upgrade mORMot to version 2 - with a major refactoring.

Continue reading

2020-02-17

New move/fillchar optimized sse2/avx asm version

Our Open Source framework includes some optimized asm alternatives to RTL's move() and fillchar(), named MoveFast() and FillCharFast().

We just rewrote from scratch the x86_64 version of those, which was previously taken from third-party snippets.
The brand new code is meant to be more efficient and maintainable. In particular, we switched to SIMD 128-bit SSE2 or 256bit AVX memory access (if available), whereas current version was using 64-bit regular registers. The small blocks (i.e. < 32 bytes) process occurs very often, e.g. when processing strings, so has been tuned a lot. Non temporal instructions (i.e. bypassing the CPU cache) are used for biggest chunks of data. We tested ERMS support, but it was found of no benefit in respect to our optimized SIMD, and was actually slower than our non-temporal variants. So ERMS code is currently disabled in the source, and may be enabled on demand by a conditional.

FPC move() was not bad. Delphi's Win64 was far from optimized - even ERMS was poorly introduced in latest RTL, since it should be triggered only for blocks > 2KB. Sadly, Delphi doesn't support AVX assembly yet, so those opcodes would be available only on FPC.

Resulting numbers are talking by themselves. Working on Win64 and Linux, of course.

Continue reading

2019-12-25

Merry Christmas and Happy New Year!

Let the little mORMot wish you and all yours a merry Christmas and a happy New Year! Happy coding!

- page 1 of 19