2026-11-17

ZIP Encryption in mORMot 2

ZIP file compression (including Unicode and zip64 extension) has been supported by mORMot for a long time, with our own TZipRead and TZipWrite classes, optimized for both memory and file processing.
One missing feature was encryption.

This has now changed. Our latest mORMot 2 trunk supports the traditional PKWARE ZipCrypto algorithm, the WinZip AES-128/192/256 format (AE-1 and AE-2), and even the brand-new AES-256-GCM format (AE-3), introduced by WinZip just a few weeks ago.

All this is implemented in a new optional mormot.crypt.zip unit, without adding any external dependency.
The story behind this implementation is also worth sharing. It started with an AI-powered pull request, which we then reviewed, refactored, tested, optimized, and considerably extended.

From an AI pull request to production code

The initial contribution was an AI-assisted pull request adding ZIP encryption support.
AI coding tools are becoming increasingly useful for generating substantial amounts of code, especially when the requested functionality follows an existing specification. But generated code, even when it compiles and passes basic tests, is not necessarily ready for production.

Cryptography makes this distinction particularly important.
An encryption implementation needs to be correct not only for successful round trips, but also for malformed archives, invalid passwords, truncated ciphertext, modified authentication tags, and unusual file sizes. We also need to consider memory consumption, performance, portability, and backward compatibility with existing mORMot applications.

So we took the initial contribution as a starting point, not as a finished implementation.
We reviewed the design against the ZIP specifications and the existing mORMot cryptographic primitives, simplified the class hierarchy, added independent reference vectors, and exercised the code with both Delphi and Free Pascal in mind.

During this work, we also extended the original scope to the new AE-3 format.
Several subtle bugs and missing cases were uncovered by this review, especially in the interaction between AES-GCM authentication and streaming ZIP extraction.

This is probably a good example of how we see AI-generated code fitting into an Open Source project: AI can accelerate the initial development, but the library maintainer remains responsible for the architecture, testing, and actual security properties.

Thanks again to the initial contributor of the initial Pull Request! It is amazing how AI tools do like the mORMot 2 code base, and are aware to propose some nice code, saving countless hours of coding, especially to write extensive regression tests.

Three encryption formats

The new unit implements three families of ZIP encryption.

  • ZipCrypto is the original PKWARE encryption algorithm. It remains useful for compatibility with old tools, including Windows Explorer, but its cryptographic security is very weak. It should not be used to protect sensitive information.
  • AE-1 / AE-2 are the established WinZip AES formats, supporting AES-128, AES-192 and AES-256 in CTR mode, with HMAC-SHA1 authentication. These formats are widely implemented by ZIP utilities, but their password derivation uses only 1,000 PBKDF2-HMAC-SHA1 iterations.
  • AE-3 is the new WinZip format, using AES-256-GCM authenticated encryption and PBKDF2-HMAC-SHA256 with 600,000 iterations by default. It offers a much more modern construction, at the cost of significantly more expensive password derivation.

For most applications, AE-1 or AE-2 remain an interesting compatibility choice. AE-3 is preferable when stronger password protection is required and interoperability with older ZIP software is not a concern. Deprecated ZipCrypto may still be needed if you expect compatibility with Windows or MacOS file explorers, which only support this legacy unsafe encoding - consider it as obfuscation not as encryption.

The new enumeration makes the selection explicit:

TZipEncryption = (
  zeAes256,
  zeZipCrypto,
  zeAes256Gcm);

The existing TZipWrite and TZipRead APIs remain unchanged apart from the new encryption properties.

For instance:

w.Password := 'correct horse battery staple';
w.Encryption := zeAes256Gcm;
w.AddDeflated('hello.txt', pointer(Plain), length(Plain));

Reading an encrypted archive is equally straightforward: set TZipRead.Password before extracting its contents.

Of course, the actual security still depends heavily on the quality of the password.

Keeping the ZIP core lightweight

One important design decision was to keep the cryptographic implementation outside mormot.core.zip.
We did not want every application using ZIP compression to depend on the cryptographic units, nor to increase executable size unnecessarily.
Instead, mormot.core.zip defines a small abstract TZipCrypt interface and an array of implementation classes.

The new mormot.crypt.zip.pas unit implements all the actual algorithms and registers them during initialization:

initialization
ZipCryptClass[zeAes256]    := TZipCryptAes;
ZipCryptClass[zeZipCrypto] := TZipCryptPkware;
ZipCryptClass[zeAes256Gcm] := TZipCryptGcm;

This is a simple form of late binding, following a pattern already used elsewhere in mORMot.

If an application only needs compression, it can keep using mormot.core.zip alone.
If it also needs encryption, it just adds mormot.crypt.zip to its uses clause. The classes are automatically registered, and TZipRead / TZipWrite can instantiate the appropriate implementation.
There is no explicit registration call, no dynamic OpenSSL dependency, and no change to the normal ZIP API.

We like this kind of integration: a small optional unit can add significant functionality without making the core more complicated.

Reusing and optimizing our cryptographic engine

All three ZIP encryption formats reuse the low-level cryptographic code already available in mORMot.
In particular, AES-GCM is implemented by TAesGcmEngine from mormot.crypt.core, which already provides optimized AES-NI and PCLMULQDQ execution paths.

We also took the opportunity to improve the general AES implementation.
TAes.DoBlocks() now includes dedicated four-way interleaved AES-NI assembly routines, for both encryption and decryption, on x86 and x86-64.
Instead of processing each AES block independently, we can process four blocks together, interleaving their AES rounds across several XMM registers. This allows the processor to execute independent AES instructions while previous rounds are still in flight.

The optimized path is selected automatically when AES-NI is available and at least four blocks can be processed. The remaining blocks use the existing single-block path.
These routines are written in assembly for both 32-bit and 64-bit targets, preserve the required calling conventions, and leave the AES context untouched.

The AES-GCM engine has its own eight-way interleaved x86-64 acceleration, combining AES-NI with carry-less multiplication for authentication.
As usual, portable Pascal implementations remain available for platforms without these hardware instructions.

AE-3: the most interesting technical challenge

The newest WinZip AE-3 format deserves a closer look.

To our knowledge, mORMot 2 is the first OpenSource library in any language to support this new format. We used infozip-av source as reference, but it is a tool, not a library.
Compared with AE-2, AE-3 replaces the separate AES-CTR and HMAC-SHA1 combination with authenticated AES-256-GCM encryption. Its password derivation also becomes considerably more expensive: 600,000 PBKDF2-HMAC-SHA256 iterations by default, with a configurable work factor stored in the ZIP entry metadata.
The format derives independent encryption and authentication material using HKDF-SHA256, then stores a 16-byte GCM tag and a supplementary 4-byte FIN authentication value.

The most interesting implementation issue, however, was not AES-GCM itself. We already had that. It was how to handle authenticated ZIP extraction.
A ZIP reader can decompress data directly into a destination stream. But with GCM, the authentication tag is only available at the end of the encrypted entry. We must therefore avoid delivering unauthenticated plaintext to the caller.
For extraction into an arbitrary TStream, TZipRead now makes a preliminary authentication pass over file-backed encrypted entries, before delivering their contents.
We introduced a dedicated TAesGcmEngine.Auth() method for this purpose: it computes the authentication state over the ciphertext without performing the AES-CTR decryption. Then we can reset the GCM state while retaining the derived AES key, and perform the actual extraction. This requires reading the encrypted data twice, but avoids repeating the very expensive PBKDF2 derivation.

For extraction to a filename, we found a simpler solution.
TZipRead writes the extracted content to a temporary file with a .partial extension, located in the destination folder. After successful authentication and content validation, it closes the temporary file and renames it to the requested filename. If authentication fails, the temporary file is removed, and the existing destination remains unchanged.
This allows a single decryption pass even for very large files. It is not intended to provide atomic filesystem replacement, which TZipRead never promised, but prevents a failed authentication from publishing the output under its final name.
Another optimization was made to TestAll(), which no longer needs an unnecessary preliminary pass when validating encrypted entries into a discard stream.

These changes benefit the overall ZIP encryption architecture, not just AE-3.

Testing beyond simple round trips

We considerably extended the regression tests during the implementation.

In addition to generating encrypted archives with TZipWrite and reading them back with TZipRead, we use external reference vectors, including independently generated AE-3 archives.
We test stored, deflated, and empty entries, different password work factors, wrong passwords, corrupted ciphertext, invalid GCM tags, modified FIN fields, malformed metadata, and ZIP entry copying or compaction. We also explicitly test both memory-backed and file-backed readers.
The final regression tests therefore exercise both successful and unsuccessful extraction, including the two-pass authentication and reset sequence.

This work illustrates why cryptographic code needs independent test vectors and careful review, even when built on top of an existing, well-tested cipher engine.

And what about AE-4?

There is still one limitation in the new AE-3 format that we find particularly interesting.

The password derivation is performed independently for every encrypted ZIP entry.
This is a reasonable security choice, but it has a noticeable performance consequence: when an archive contains thousands of small files encrypted with the same password, the repeated 600,000 PBKDF2 iterations can easily dominate the total processing time.
AES-NI optimizations won't help much here, because the bottleneck is no longer AES encryption.

A possible future AE-4 specification could address this by deriving an archive-level master key from the password only once, then deriving independent encryption keys for each ZIP entry using a safe HKDF like HMAC. This would preserve separate keys for individual files, while greatly reducing the latency of archives containing many small entries protected by the same password.
Of course, reusing key hashing would still be an implementation detail: each file could have its own password, but adding a HMAC step could be enough to reuse a single expensive password hash for several files, without reducing the actual security.

It is only a proposal at this stage, not an existing format. We would welcome discussion with other ZIP implementers before going any further. It is worth asking how WinZip actually validated the proper use of their algorithm. Also why they reverted to AE-2 in WinZip 11 to keep a quick crc32 check of the encrypted content, but they force 0 in their AE-3...

Conclusion

ZIP encryption is now fully integrated into mORMot 2, through a small optional unit and the same familiar TZipRead / TZipWrite APIs.

We support the historical ZipCrypto format, the widely used WinZip AE-1 / AE-2 formats, and the very recent AES-256-GCM AE-3 format.
Beyond the new encryption algorithms, this development also brought improved AES-NI assembly routines, safer authenticated extraction, more efficient streaming, and extensive regression testing.
And it started with an AI-assisted pull request.

The experience was positive, but also instructive. AI helped initiate the work, while the detailed human review led to a substantially different and more complete implementation.
This combination of assisted development, careful architecture, independent testing and low-level optimization is probably where such tools bring the most value to a mature library like mORMot.

The full source is available in our mORMot 2 repository, especially mormot.crypt.zip.pas and mormot.core.zip.pas.

As always, feedback and interoperability tests with other ZIP implementations are welcome!

2026-08-28

mORMot 2 HTTP Server: Memory Efficiency and Thread Smoothing

High-performance HTTP servers are often benchmarked primarily by their peak requests per second. For a REST server, however, peak throughput is only part of the story. A server also needs to remain responsive when requests have very different execution times: some may complete in a few microseconds, while others may spend milliseconds waiting for a database, a remote service, or some other slow operation.

The mORMot 2 asynchronous HTTP server was one of the fastest in the TFB benchmark. But it was designed as a scalable REST server. Our project has recently received two related improvements in this area: a closer look at its memory allocation behavior, and a refined thread smoothing mechanism for waking worker threads on POSIX systems.

Continue reading

2026-08-12

A New Native i18n Layer for mORMot 2

Internationalization (aka i18n) is now part of the mORMot 2 core with the new mormot.core.i18n unit, introduced by PR #510 - and deeply rewritten and completed, as always. The goal is not to create yet another translation format, but to provide a small, fast and framework-native i18n layer for both Delphi and Free Pascal.

The basic principle is deliberately simple: the original English text is the translation key. For example, "Hello" can be translated to "Bonjour". If a translation is missing, the original text remains available as the natural fallback. This keeps application code readable and makes the translation tables straightforward to maintain.

Continue reading

2026-08-07

A 2 GB/s XML Parser for Delphi and Free Pascal: SAX, Streaming and DOM Without the Usual Trade-offs

XML may no longer be fashionable, but it is still everywhere.

Scientific datasets, configuration files, industry standards, legacy systems, government data, document formats and countless integration interfaces continue to produce XML. And when XML documents become large, the choice of parser can make a surprisingly large difference.

For mORMot 2, we have recently written a new XML parser, TXmlParser, with a slightly different goal from the traditional XML libraries found in the Delphi and Free Pascal ecosystems. It started from an AI-driven pull request from zen010101, then it was fully rewritten by hand for performance and enhanced to support several API modes.

Instead of choosing between a low-level SAX parser and a convenient DOM parser, the same parser provides several levels of access:

  • a zero-allocation SAX/pull API;
  • a streaming/navigation API for selecting only the parts of the document that matter;
  • a hybrid SAX/DOM API, where selected subtrees can be materialized as TDocVariantData;
  • and a full XML-to-TDocVariant DOM representation, when that is what the application actually needs.

The interesting part is that the performance remains very high at all these levels.

Using the 24 MB nasa.xml reference document from the University of Washington XML repository, the raw parser reaches about 2.1 GB/s, while the full DOM conversion reaches about 424 MB/s. Selective streaming reaches about 1.6 GB/s, and once the DOM exists, traversing it through the new TDocVariantData.Product() enumerator reaches about 7 GB/s.

There is also an interesting memory optimization: enabling dvoInternNames reduces the complete DOM footprint from approximately 60 MB to 40 MB, with essentially no performance penalty.

The complete benchmark is shown below.

Continue reading

2026-07-06

Object Pascal Is Still a Serious Contender for High-Performance REST Servers

In this article, we will deep-dive into the architecture that lets our Open Source mORMot 2 library compete with (and sometimes outperform) nginx, Kestrel, and raw C++ implementations.

The library offers several HTTP servers.
The mormot.net.server unit contains the historical mORMot 1 web servers - mainly THttpServer (thread-pool for HTTP/1.0 short requests + one thread per kept-alive HTTP/1.1 connection), and THttpApiServer (using the efficient http.sys kernel API on Windows).
But since mORMot 2, our mormot.net.async unit is our genuine cross-platform, modern async implementation.

Continue reading

2026-06-02

TPipeStream: A High-Performance In-Memory Pipe for Delphi and FPC

The latest version of mORMot 2 introduces TPipeStream, a new TStream descendant designed to efficiently transmit data between two threads.

At first glance, the class looks deceptively simple: one thread writes data using Write(), another thread reads it using Read().
Behind this familiar interface lies a fast, lock-protected ring buffer with blocking semantics, making it suitable for streaming large amounts of data between independent producer and consumer threads.

Continue reading

2026-05-19

Delphi Linux & macOS Support for mORMot 2

We are pleased to share that mORMot 2 now offers initial compatibility with Delphi's Linux and macOS x86_64 compilers for console/server applications.
This effort originated from GitHub issue #442.

Thanks to community testing and contributions, core units now compile and run on these platforms.

Continue reading

2026-03-03

Rencontre entre Pascaliens à Nantes

Si vous êtes dans l'Ouest de la France, vous êtes les bienvenus pour une escale Delphi/Lazarus le jeudi 9 avril fin d'après midi à Nantes!

J'y serais avec toute l'équipe dev de Tranquil IT pour partager des idées, et certainement quelques petites choses plus solides (ou liquides) ! :)

Continue reading

- page 1 of 53