New Client for MongoDB 5.1/6 Support

Starting with its version 5.1, MongoDB disabled the legacy protocol used for communication since its beginning.
As a consequence, our mORMot client was not able to communicate any more with the latest versions of MongoDB instances.

Last week, we made a deep rewrite of mormot.db.nosql.mongodb.pas, which changed the default protocol to use the new layout on the wire. Now messages use regular MongoDB Database Commands, with automated compression if needed.

No change is needed in your end-user MongoDB or ORM/ODM code. The upgrade is as simple as update your mORMot 2 source, then recompile.

The Mongo Wire Protocol

Since its beginning, MongoDB used a simple protocol over TCP, via several binary opcodes and message, for CRUD operations.

A new alternative protocol was introduced in version 3.6, and the former protocol was marked as deprecated.
Two new opcodes were introduced, OP_MSG and OP_COMPRESSED, to replace all other frames. They just encapsulate, with or without compression, some abstract BSON content.
The official documentation details those changes in this web page.

In short (picture extracted from the blog above), the protocol came from this:

to this:

The main benefit is that the commands and answers are just conventional BSON, so the protocol can change at logical/BSON/JSON level by adding or changing some members, with no need of dealing with low-level binary structures.

With the version 5.1 of MongoDB, the previous protocol was not just deprecated, but disabled.
So we had to update the mORMot 2 client code! (yes, the mORMot 1 code has not been updated - it may become a good reason to upgrade)

Deep Rewrite

In fact, the official MongoDB documentation is somewhat vague. And the official drivers are a bit difficult to reverse-engineer, due to the verbose nature of C, Java or C#. The native/node driver was easiest to dissect, and we used it as reference.
Luckily enough, there are some specification document available too, which offers some additional valuable clarifications.

After some testing, we managed to replace all previous OP_QUERY and its brothers to the new OP_MSG frame, which is, as documented in the specification, "One opcode to rule them all". ;)

Once we had the commands working, we needed to rewrite all CRUD operations using commands, and not opcodes.
Queries are now made with find and aggregate commands. Their results are now located in a "cursor": firstBatch": .. BSON array within the response. And a new getMore command is to be used to retrieve the next values within a "cursor": nextBatch": ... resultset.
For writing, insert, update and delete commands are called, with their appropriate BSON content.

During the refactoring, we optimized the BSON process, and also enhanced the whole process, mainly the logs and the execution efficiency. The mORMot client side should not be a bottleneck. And it is not, even with this NoSQL database.

Don't expect any performance enhancement, or new features. It is just some low-level protocol change at TCP level.
But if you used the "non acknowledged write mode" of the former protocol, which was unsafe but very fast, you will have lower performance with the new protocol, because the new protocol always acknowledges the commands it receives. So, in some very specific configurations, the new protocol may reduce the performance.

Backward Compatibility

All those changes were encapsulated in our revised mormot.db.nosql.mongodb.pas unit.

If you have a very old MongoDB instance, and don't want to upgrade, you could just compile your project with the MONGO_OLDPROTOCOL conditional, to use the deprecated opcodes.
If the MongoDB team does not care much with backward compatibility (they could have kept the previous protocol for sure, they still maintain it for the handshake message if needed), we do care about not breaking too much things with mORMot, so we kept the previous code, and tested/validated it too, for legacy systems.

New Sample

We translated and introduced the MongoDB benchmark sample to mORMot 2 code base.

You could find it, and run it, from our source code repository.

This code is a good entry point for what is possible with this unit in our framework, for both direct access or ORM/ODM access.
And you would be able to guess the performance numbers you may achieve with your project.

Running a MongoDB database in a container is as easy as executing the following command:

sudo docker run --name mongodb -d -p 27017:27017 mongo:latest

Then you will have a MongoDB server instance accessible on localhost:27017, so you could run the sample straight away.

Delphi/FPC Open Source Rocks

We hope you will find the change painless and transparent. We did not modify the high-level client methods, nor break the ORM/ODM: you can still write some SELECT complex statements, and our ORM will translate it into MongoDB aggregate commands.

To my knowledge, there is only a single other Delphi/FPC client library which made the upgrade to the new protocol, at today. Once we made our own changes, we notified other library authors, and Stijn made very quickly the needed changes. Congrats! Maybe our code could be used as reference for other library maintainers, because the protocol needs some small tweaks sometimes.
It is important to have some maintenance on the library you use. And our little mORMot is still on the edge: thanks to FPC, it runs very well on Linux and BSD, which makes it perfect for professional services running in the long term! :)

Your feedback is welcome in the forum thread which initiated these modifications, as usual!
Don't hesitate to notify us any missing or broken feature.
Thanks Daniel for your report and support!


Native TLS Support for mORMot 2 REST or WebSockets Servers

Since the beginning, we delegated the TLS encryption support to a reverse proxy server, mainly Nginx. Under Windows, you could setup the http.sys HTTPS layer as usual, as a native - even a bit complicated - solution.
Nginx has several advantages, the first being a proven and efficient technology, with plenty of documentation and configuration tips. It interfaces nicely with Let's Encrypt, and is very good for any regular website, using static content and PHP. This very blog and the Synopse web site is hosted via Ngnix on a small Linux server.

But in mORMot 2, we introduced a new set of asynchronous web server classes. So stability and performance are not a problem any more. Some benchmarks even consider this server to be faster than nginx (the stability issue mentioned in this post has been fixed in-between).
We just introduced TLS support of our socket-based servers, both the blocking and asynchronous classes. It would use OpenSSL if available, or the SChannel API layer of Windows. Serving HTTPS or WSS with a self-signed certificate is just a matter of a single parameter now, and performance seems pretty good, especially with OpenSSL.

Continue reading


New Async HTTP/WebSocket Server on mORMot 2

The HTTP server is one main part of any SOA/REST service, by design.
It is the main entry point of all incoming requests. So it should better be stable and efficient. And should be able to scale in the future, if needed.

There have always been several HTTP servers in mORMot. You can use the HTTP server class you need.
In mORMot 2, we added two new server classes, one for publishing over HTTP, another able to upgrade to WebSockets. The main difference is that they are fully event-driven, so their thread pool is able to scale with thousands of concurrent connections, with a fixed number of threads. They are a response to the limitations of our previous socket server.

Continue reading


mORMot 2 ORM Performance

The official release of mORMot 2 is around the edge. It may be the occasion to show some data persistence performance numbers, in respect to mORMot 1.

For the version 2 of our framework, its ORM feature has been enhanced and tuned in several aspects: REST routing optimization, ORM/JSON serialization, and in-memory and SQL engines tuning. Numbers are talking. You could compare with any other solution, and compile and run the tests by yourself for both framework, and see how it goes on your own computer or server.
In a nutshell, we almost reach 1 million inserts per second on SQLite3, and are above the million inserts in our in-memory engine. Reading speed is 1.2 million and 1.7 million respectively. From the object to the storage, and back. And forcing AES-CTR encryption on disk almost don't change anything. Now we are talking. ;)

Continue reading


Three Locks To Rule Them All

To ensure thread-safety, especially on server side, we usually protect code with critical sections, or locks. In recent Delphi revisions, we have the TMonitor feature, but I would rather trust the OS for locks, which are implemented using Windows Critical Sections, or POSIX futex/mutex.

But all locks are not born equal. Most of the time, the overhead of a Critical Section WinAPI or the pthread library is not needed.
So, in mORMot 2, we introduced several native locks in addition to those OS locks, with multi-read/single-write abilities, or re-entrancy.

Continue reading


mORMot 2 Generics and Collections

Generics are a clever way of writing some code once, then reuse it for several types.
They are like templates, or compiler-time shortcuts for type definitions.

In the last weeks, we added a new mormot.core.collections.pas unit, which features:

  • JSON-aware IList<> List Storage;
  • JSON-aware IKeyValue<> Dictionary Storage.

In respect to Delphi or FPC RTL generics.collections, this unit uses interfaces as variable holders, and leverage them to reduce the generated code as much as possible, as the Spring4D 2.0 framework does, but for both Delphi and FPC. It publishes TDynArray and TSynDictionary high-level features like indexing, sorting, JSON/binary serialization or thread safety as Generics strong typing.

Resulting performance is great, especially for its enumerators, and your resulting executable size won't blow up as with the regular RTL unit.

Continue reading


EKON 25 Slides

EKON 25 at Düsseldorf was a great conference (konference?).

At last, a physical gathering of Delphi developers, mostly from Germany, but also from Europe - and even some from USA! No more virtual meetings, which may trigger the well known 'Abstract Error' on modern pascal coders.
There were some happy FPC users too - as I am now. :)

I have published the slides of my conferences, mostly about mORMot 2.
By the way, I wish we would be able to release officially mORMot 2 in December, before Christmas. I think it starts to be stabilized and already known to be used on production. We expect no more breaking change in the next weeks.

Continue reading


Delphi 10.4 / Delphi 11 Alexandria Breaking Changes

The latest revision of Delphi, named Delphi 11 Alexandria, is out.
A lot of new features, some enhanced platforms. Nice!
But it is also for us the opportunity to come back to some breaking changes, which appeared in Delphi 10.4 earlier this year, and are now "officially" part of Delphi 11.

The main breaking change of Delphi 10.4 and later, as reported by mORMot users, is the new lifetime of local variables.
TL&LR: a local variable which is not explicitly declared, but returned by a function may be released as soon as it is not used any more, whereas in the original implementation, it was allocated as a regular local variable, and we could expect its lifetime to remain active up to the end of the function. With Delphi 10.4, it is not the case any more: the compiler could release/clear the local variable sooner, to reduce the allocation pressure.

Idea behind this change is that it may have better register allocation within the function, so it "may" theoretically result in faster code. Not convinced about it, anyway - we will discuss that.
The main thing is that it could break existing code, because it breaks the Delphi compiler expectation since decades.
Some perfectly fine working code would end to work as expected. We have identified several use cases with mORMot which are affected by this change. Since it seems there will be no coming back from Delphi point of view, it is worth a blog article. ;)

Continue reading

- page 1 of 49