Synopse Open Source

Starting with its version 5.1, MongoDB disabled the legacy protocol used for communication since its beginning.
As a consequence, our mORMot client was not able to communicate any more with the latest versions of MongoDB instances.

Last week, we made a deep rewrite of mormot.db.nosql.mongodb.pas, which changed the default protocol to use the new layout on the wire. Now messages use regular MongoDB Database Commands, with automated compression if needed.

No change is needed in your end-user MongoDB or ORM/ODM code. The upgrade is as simple as update your mORMot 2 source, then recompile.

EKON 25 at Düsseldorf was a great conference (konference?).

At last, a physical gathering of Delphi developers, mostly from Germany, but also from Europe - and even some from USA! No more virtual meetings, which may trigger the well known 'Abstract Error' on modern pascal coders.
There were some happy FPC users too - as I am now.

I have published the slides of my conferences, mostly about mORMot 2.
By the way, I wish we would be able to release officially mORMot 2 in December, before Christmas. I think it starts to be stabilized and already known to be used on production. We expect no more breaking change in the next weeks.

The .zip format is from last century, back to the early DOS days, but can still be found everywhere. It is even hidden when you run a .docx document, a .jar application, or any Android app!
It is therefore (ab)used not only as archive format, but as application file format / container - even if in this respect using SQLite3 may have much more sense.

We recently enhanced our mormot.core.zip.pas unit:

• to support Zip64,
• with enhanced .zip read/write,
• to have a huge performance boost during its process,
• and to integrate better with signed executables.

As a gift to the FPC community, I just committed a new Memory Manager for FPC.
Check mormot.core.fpcx64mm.pas in our mORMot2 repository.
This is a stand-alone unit for FPC only.

It targets Windows and Linux multi-threaded Service applications - typically mORMot daemons.
It is written in almost pure x86_64 assembly, and some unique tricks in the Delphi/FPC Memory Manager world.

It is based on FastMM4 (not FastMM5), and we didn't follow the path of the FastMM4-AVX version - instead of AVX, we use plain good (non-temporal) SSE2 opcode, and we rely on the mremap API on Linux for very efficient reallocation. Using mremap is perhaps the biggest  benefit of this memory manager - it leverages a killer feature of the Linux kernel for sure. By the way, we directly call the Kernel without the need of the libc.

We tuned our x86_64 assembly a lot, and made it cross-platform (Windows and POSIX). We profiled the multi-threading, especially by adding some additional small blocks for GetMem (which is a less expensive notion of "arenas" as used in FastMM5 and most C allocators), introducing an innovatice and very efficient round-robin of tiny blocks (<128 bytes), and proper spinning for FreeMem and medium blocks.

It runs all our regression tests with huge performance and stability - including multi-threaded tests with almost no slow down: sleep is reported as less than 1 ms during a 1 minute test. It has also been validated on some demanding multi-threaded tasks.

We just committed a deep refactoring of the SynSQlite3Static.pas unit - and all units using static linking for FPC. It also includes a new encryption format for SQlite3, using AES, so much more secure than the previous one. This is a breaking change, so worth a blog article! Now all static .o .a  […]

Marco Cantu, product manager of Delphi/RAD Studio, did publish the official RAD Studio 2016 Product Approach and Roadmap.
The upcoming release has a codename known as "BigBen", and should be called Delphi 10.1 Berlin, as far as I understand.

After this summer, another release, which codename is "Godzilla", will support Linux as a compiler target, in its Delphi 10.2 Tokyo release.
This is a very good news, and some details are given.
I've included those official names to mORMot's internal compiler version detection.
Thanks Marco for the information, and pushing in this direction!

My only concern is that it would be "ARC-enabled"...

It is worth saying again.
I'm not speaking this time about performance issues, but about a common misunderstanding of what the latest version of Delphi offers.

Since Delphi "NextGen" compilers did break the memory model (introducing ARC), and also reducing low-level types (e.g. RawByteString/AnsiString), we can not say that Delphi is a single cross-compiler.
In practice, it has a set of cross-compilers.

I just received a mail from Alfred (aka Alf in the source code), which did a lot of work to let our little mORMot compiles and run with FPC, especially under Linux, and also with an ARM processor. Hello Arnaud, A nice surprise ... Sample 2 native on Android !!!! See picture.  Works 100% !!!    […]

Do you know QDAC3 ?
This is an open source project, from China (with Chinese comments and exception errors messages, but the methods and variables are in English).
It is cross-platform, and told to be very fast about JSON process.

You can download this Open Source project code from http://sourceforge.net/projects/qdac3
And their blog - in Chinese - is at http://blog.qdac.cc/

So I included QDAC3 in our "25 - JSON performance" sample.
Numbers are talking, now.

How would be 2015 like for our little rodents?
Due to popular request of several users of mORMot, we identified and designed some feature requests dedicated to BigData process.

In fact, your data is the new value, especially if you propose SaaS (Software As A Service) hosting to your customers, with a farm of mORMot servers.
Recent Linux support for mORMot servers, together with the high performance and installation ease of our executable, open the gate to cheap cloud-based hosting.
As a consequence, a lot of information would certainly be gathered by your mORMot servers, and a single monolithic database is not an option any more.

For mORMot solutions hosted in cloud, a lot of data may be generated. The default SQLite3 storage engine may be less convenient, once it reaches some GB of file content. Backup becomes to be slow and inefficient, and hosting this oldest data in the main DB, probably stored on an expensive SSD, may be a lost of resource. Vertical scaling is limited by hardware and price factors.

This is were data sharding comes into scene.
Note that sharding is not replication/backup, nor clustering, nor just spreading. We are speaking about application-level data splitting, to ease maintenance and horizontal scalability of mORMot servers.

Data sharding could already be implemented with mORMot servers, thanks to TSQLRestStorage:

• Using TSQLRestStorageExternal: any table may have its own external SQL database engine, may be in its separated DB server;
• Using TSQLRestStorageMongoDB: any table may be stored on a MongoDB cluster, with its own sharding abilities;
• Using TSQLRestStorageRemote: each table may have its own remote ORM/REST server.

But when data stored in a single table tends to grow without limit, this feature is not enough.
Let's see how the close future of mORMot looks like.