News from mORMot's Burrow

XML may no longer be fashionable, but it is still everywhere.

Scientific datasets, configuration files, industry standards, legacy systems, government data, document formats and countless integration interfaces continue to produce XML. And when XML documents become large, the choice of parser can make a surprisingly large difference.

For mORMot 2, we have recently written a new XML parser, TXmlParser, with a slightly different goal from the traditional XML libraries found in the Delphi and Free Pascal ecosystems. It started from an AI-driven pull request from zen010101, then it was fully rewritten by hand for performance and enhanced to support several API modes.

Instead of choosing between a low-level SAX parser and a convenient DOM parser, the same parser provides several levels of access:

• a zero-allocation SAX/pull API;
• a streaming/navigation API for selecting only the parts of the document that matter;
• a hybrid SAX/DOM API, where selected subtrees can be materialized as TDocVariantData;
• and a full XML-to-TDocVariant DOM representation, when that is what the application actually needs.

The interesting part is that the performance remains very high at all these levels.

Using the 24 MB nasa.xml reference document from the University of Washington XML repository, the raw parser reaches about 2.1 GB/s, while the full DOM conversion reaches about 424 MB/s. Selective streaming reaches about 1.6 GB/s, and once the DOM exists, traversing it through the new TDocVariantData.Product() enumerator reaches about 7 GB/s.

There is also an interesting memory optimization: enabling dvoInternNames reduces the complete DOM footprint from approximately 60 MB to 40 MB, with essentially no performance penalty.

The complete benchmark is shown below.

OpenAPI, which was formerly called Swagger, is a set of specifications to encode the server API endpoints definitions into text, mostly JSON.
From this reference text, you can generate client code to access the service, in a vast number of languages.

Delphi seems to be far behind other languages, in terms of this code generation. I found nothing even working for FPC.
Since we needed it for our internal tools at Tranquil IT, we just published the new mormot.net.openapi.pas Open Source unit, which is quite a game changer. Thanks Andreas for starting this project, and testing it in its early age!

Starting with its version 5.1, MongoDB disabled the legacy protocol used for communication since its beginning.
As a consequence, our mORMot client was not able to communicate any more with the latest versions of MongoDB instances.

Last week, we made a deep rewrite of mormot.db.nosql.mongodb.pas, which changed the default protocol to use the new layout on the wire. Now messages use regular MongoDB Database Commands, with automated compression if needed.

No change is needed in your end-user MongoDB or ORM/ODM code. The upgrade is as simple as update your mORMot 2 source, then recompile.

The HTTP server is one main part of any SOA/REST service, by design.
It is the main entry point of all incoming requests. So it should better be stable and efficient. And should be able to scale in the future, if needed.

There have always been several HTTP servers in mORMot. You can use the HTTP server class you need.
In mORMot 2, we added two new server classes, one for publishing over HTTP, another able to upgrade to WebSockets. The main difference is that they are fully event-driven, so their thread pool is able to scale with thousands of concurrent connections, with a fixed number of threads. They are a response to the limitations of our previous socket server.

HTTP(S) is the main protocol of the Internet.
We enhanced the mORMot 2 socket client to push its implementation into more use cases. The main new feature is perhaps WGET-like processing, with hashing, resuming, console feedback, and direct file download.

The more I think of it, the more I am convinced it is time to change how the framework is versioned.
We have version 1.18 since years... difficult to follow... time to upgrade!

I would like to upgrade mORMot to version 2 - with a major refactoring.

JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA or ECDSA.

They can be used for:

• Authentication: including a JWT to any HTTP request allows Single Sign On user validation across different domains;
• Secure Information Exchange: a small amount of data can be stored in the JWT payload, and is digitally signed to ensure its provenance and integrity.

See http://jwt.io for an introduction to JSON Web Tokens.

Our mORMot framework now implements JWT:

• HS256 (HMAC-SHA256) and ES256 (256-bit ECDSA) algorithms (with the addition of the "none" weak algo);
• Validates all claims (validation dates, audiences, JWT ID);
• Thread-safe and high performance (2 µs for a HS256 verification under x64), with optional in-memory cache if needed (e.g. for slower ES256);
• Stand-alone and cross-platform code (no external dll, works with Delphi or FPC);
• Enhanced security and strong design - per instance, it is by design immune from https://auth0.com/blog/2015/03/31/critical-vulnerabilities-in-json-web-token-libraries
• Full integration with the framework.