The Mongo Wire Protocol
Since its beginning, MongoDB used a simple protocol over TCP, via several binary opcodes and message, for CRUD operations.
A new alternative protocol was introduced in version 3.6, and the former protocol was marked as deprecated.
Two new opcodes were introduced, OP_MSG and OP_COMPRESSED, to replace all other frames. They just encapsulate, with or without compression, some abstract BSON content.
The official documentation details those changes in this web page.
In short (picture extracted from the blog above), the protocol came from this:
to this:
The main benefit is that the commands and answers are just conventional BSON, so the protocol can change at logical/BSON/JSON level by adding or changing some members, with no need of dealing with low-level binary structures.
With the version 5.1 of MongoDB, the previous protocol was not just deprecated, but disabled.
So we had to update the mORMot 2 client code! (yes, the mORMot 1 code has not been updated - it may become a good reason to upgrade)
Deep Rewrite
In fact, the official MongoDB documentation is somewhat vague. And the official drivers are a bit difficult to reverse-engineer, due to the verbose nature of C, Java or C#. The native/node driver was easiest to dissect, and we used it as reference.
Luckily enough, there are some specification document available too, which offers some additional valuable clarifications.
After some testing, we managed to replace all previous OP_QUERY and its brothers to the new OP_MSG frame, which is, as documented in the specification, "One opcode to rule them all".
Once we had the commands working, we needed to rewrite all CRUD operations using commands, and not opcodes.
Queries are now made with find
and aggregate
commands. Their results are now located in a "cursor": firstBatch": ..
BSON array within the response. And a new getMore
command is to be used to retrieve the next values within a "cursor": nextBatch": ...
resultset.
For writing, insert
, update
and delete
commands are called, with their appropriate BSON content.
During the refactoring, we optimized the BSON process, and also enhanced the whole process, mainly the logs and the execution efficiency. The mORMot client side should not be a bottleneck. And it is not, even with this NoSQL database.
Don't expect any performance enhancement, or new features. It is just some low-level protocol change at TCP level.
But if you used the "non acknowledged write mode" of the former protocol, which was unsafe but very fast, you will have lower performance with the new protocol, because the new protocol always acknowledges the commands it receives. So, in some very specific configurations, the new protocol may reduce the performance.
Backward Compatibility
All those changes were encapsulated in our revised mormot.db.nosql.mongodb.pas unit.
If you have a very old MongoDB instance, and don't want to upgrade, you could just compile your project with the MONGO_OLDPROTOCOL
conditional, to use the deprecated opcodes.
If the MongoDB team does not care much with backward compatibility (they could have kept the previous protocol for sure, they still maintain it for the handshake message if needed), we do care about not breaking too much things with mORMot, so we kept the previous code, and tested/validated it too, for legacy systems.
New Sample
We translated and introduced the MongoDB benchmark sample to mORMot 2 code base.
You could find it, and run it, from our source code repository.
This code is a good entry point for what is possible with this unit in our framework, for both direct access or ORM/ODM access.
And you would be able to guess the performance numbers you may achieve with your project.
Running a MongoDB database in a container is as easy as executing the following command:
sudo docker run --name mongodb -d -p 27017:27017 mongo:latest
Then you will have a MongoDB server instance accessible on localhost:27017
, so you could run the sample straight away.
Delphi/FPC Open Source Rocks
We hope you will find the change painless and transparent. We did not modify the high-level client methods, nor break the ORM/ODM: you can still write some SELECT complex statements, and our ORM will translate it into MongoDB aggregate commands.
To my knowledge, there is only a single other Delphi/FPC client library which made the upgrade to the new protocol, at today. Once we made our own changes, we notified other library authors, and Stijn made very quickly the needed changes. Congrats! Maybe our code could be used as reference for other library maintainers, because the protocol needs some small tweaks sometimes.
It is important to have some maintenance on the library you use. And our little mORMot is still on the edge: thanks to FPC, it runs very well on Linux and BSD, which makes it perfect for professional services running in the long term! :)
Your feedback is welcome in the forum thread which initiated these modifications, as usual!
Don't hesitate to notify us any missing or broken feature.
Thanks Daniel for your report and support!