To content | To menu | To search

2020, Thursday May 7

New Multi-thread Friendly Memory Manager for FPC written in x86_64 assembly

As a gift to the FPC community, I just committed a new Memory Manager for FPC.
Check mormot.core.fpcx64mm.pas in our mORMot2 repository.
This is a stand-alone unit for FPC only.

It targets Windows and Linux multi-threaded Service applications - typically mORMot daemons.
It is written in almost pure x86_64 assembly, and some unique tricks in the Delphi/FPC Memory Manager world.

It is based on FastMM4 (not FastMM5), and we didn't follow the path of the FastMM4-AVX version - instead of AVX, we use plain good (non-temporal) SSE2 opcode, and we rely on the mremap API on Linux for very efficient reallocation. Using mremap is perhaps the biggest  benefit of this memory manager - it leverages a killer feature of the Linux kernel for sure. By the way, we directly call the Kernel without the need of the libc.
We tuned its x86_64 assembly a lot, and made it cross-platform (Windows and POSIX). We profiled the multi-threading, especially by adding some additional small blocks for GetMem (which is a less expensive notion of "arenas" as used in FastMM5 and most C allocators) and proper spinning for FreeMem and medium blocks.
It runs all our regression tests with huge performance and stability - including multi-threaded tests with almost no slow down (Sleep reported less than 1 ms during a 1 minute test).

Continue reading...

2020, Monday March 30

Debriefing of mORMot2 Survey

Thanks you all for have posted your feedback on our mORMot2 Survey!

Here are some insights.

Continue reading...

2020, Saturday March 28

Faster Double-To-Text Conversion

On server side, a lot of CPU is done processing conversions to or from text. Mainly JSON these days.

In mORMot, we take care a lot about performance, so we have rewritten most conversion functions to have something faster than the Delphi or FPC RTL can offer.
Only float to text conversion was not available. And RTL str/floattexttext performance, at least under Delphi, is not consistent among platforms.
So we just added a new Double-To-Text set of functions.

Continue reading...

2020, Friday March 6

We Need U: Survey about mORMot 2.0

First of all, if it was not clear enough: Delphi will continue to be supported in mORMot 2.0.
Some people reported that our previous article may have been misleading.

But perhaps not all versions. For sure, Delphi 5 and Kylix will not be supported in mORMot 2. It is also possible that it would not be compatible with Delphi 7, which doesn't support inlining and dotted unit names.

So we need your insight!
Please fill this Google Survey, and tell us what you expect!

2020, Tuesday March 3

Preparing Revision 2.x of the mORMot Framework

The more I think of it, the more I am convinced it is time to change how the framework is versioned.
We have version 1.18 since years... difficult to follow... time to upgrade!

I would like to upgrade mORMot to version 2 - with a major refactoring.

Continue reading...

2020, Monday February 17

New move/fillchar optimized sse2/avx asm version

Our Open Source framework includes some optimized asm alternatives to RTL's move() and fillchar(), named MoveFast() and FillCharFast().

We just rewrote from scratch the x86_64 version of those, which was previously taken from third-party snippets.
The brand new code is meant to be more efficient and maintainable. In particular, we switched to SIMD 128-bit SSE2 or 256bit AVX memory access (if available), whereas current version was using 64-bit regular registers. The small blocks (i.e. < 32 bytes) process occurs very often, e.g. when processing strings, so has been tuned a lot. Non temporal instructions (i.e. bypassing the CPU cache) are used for biggest chunks of data. We tested ERMS support, but it was found of no benefit in respect to our optimized SIMD, and was actually slower than our non-temporal variants. So ERMS code is currently disabled in the source, and may be enabled on demand by a conditional.

FPC move() was not bad. Delphi's Win64 was far from optimized - even ERMS was poorly introduced in latest RTL, since it should be triggered only for blocks > 2KB. Sadly, Delphi doesn't support AVX assembly yet, so those opcodes would be available only on FPC.

Resulting numbers are talking by themselves. Working on Win64 and Linux, of course.

Continue reading...

2019, Wednesday December 25

Merry Christmas and Happy New Year!

Let the little mORMot wish you and all yours a merry Christmas and a happy New Year!

Happy coding!

2019, Wednesday October 30

EKON 23 Presentation Slides and Code

I just finished my workshop at EKON 23.
Like every year, it was a great event to attempt to, and I enjoyed presenting 2 sessions and 1 workshop.

Sessions were about "Kingdom Driven Design" (KDD), which is the name I used to define a cut-down version of "Domain Driven Design" (DDD).
Less paranoid, a bit less isolation, but perhaps more common sense for the less sensitive projects.
Some presentations and code are now available!

Continue reading...

2019, Thursday October 17

Nice introductory articles about mORMot

I just found some very nice articles by Stephan Bester about first steps to mORMot's ORM and SOA.

Don't be scared by the mORMot: it is more stressed than you are.
This painful picture just won a wildlife photographer prize... poor little rodent!

Continue reading...

2019, Saturday September 21

SQLite3 static linking for Delphi Win64

A long-awaited feature was the ability to create stand-alone mORMot Win64 applications via Delphi, with no external sqlite3-64.dll required.

It is now available, with proper integration, and encryption is working!

Continue reading...

2019, Wednesday September 18

Introducing Kingdom Driven Design at EKON 23

There are still some days to join EKON 23 conferences with the reduced price!

I will make 2 sessions, and 1 workshop, in English (my German is not good enough), about practical project design and mORMot.
If you find DDD is a bit too much for you, but still want to write clean code, check this!

Continue reading...

2018, Saturday November 24

mORMot running on Delphi 10.3 Rio

Now that Delphi 10.3 Rio is out, we had to ensure that our little mORMot would run with this revision.
Since we don't make any commercial software with Delphi any more (we switched to FPC), I downloaded the Community Edition.
We disabled the Error Insight feature, which seems not very stable especially with our mORMot complex units - as with previous Delphi versions.

In respect to the previous "Starter" editions, this CE version includes everything you expect for good work: Win32 + Win64 + OSX + mobile compilers, and even the source code of the RTL and enabled command-line compilers!
The IDE seems really refreshed with its new layout. You feel at home and with some modern and fresh air!
Great work Embarcadero!

Of course, the "Pro" and "Architect" features are missing (like FireDAC, DataSnap or RADServer).
But mORMot is self-contained even for the database access, so with the CE you can write full-features REST/SOA client-servers and MVC web applications, with a simple ORM and high-performance OleDB/ODBC database access to SQlite3, MSSQL, Oracle, Firebird, MySQL and others (by-passing the TDataSet class for better performance) and even NoSQL like MongoDB.
Ensure you follow the Embarcadero license terms when you use mORMot with the Community Edition.

With Rio, the main breaking change was that the PCRE API switched from an UTF-8 to UTF-16, and did hide the UTF-8 function calls.
But after a small patch, now everything works as expected - at least for our regression tests: 

Feedback is welcome in our forum, as usual!

2018, Monday November 12

Win 100GB of log space from a Real Life mORMot Project

Sometimes, I am asked what could be done with mORMot.
Well, we've been using the library at LiveMon to analyse logs for example.

And we're able to get speed of a few TB/sec. Speed above is no typo: TB/s not GB/s. For a regex search, not a per-word dictionary based lookup.

With a cross-platform collecting agent that's only 2MB.

If you want to try it, I'm able to give some account with 100GB of log space for free - and for life.
Just use this link: 

Hurry up: there are only 10 available ;)

EKON 22 Slides and Code

I've uploaded two sets of slides from my presentations at EKON 22 :

The WorkShop about "Getting REST with mORMot" has a corresponding new Samples folder in our repository.

And don't forget about one example of high-performance object pascal code: what we do at LiveMon - and get 100GB of log space for free!

Feedback is welcome in our forum as usual!

2018, Monday March 12

New AES-based SQLite3 encryption

We just committed a deep refactoring of the SynSQlite3Static.pas unit - and all units using static linking for FPC.
It also includes a new encryption format for SQlite3, using AES, so much more secure than the previous one.
This is a breaking change, so worth a blog article!

Now all static .o .a files are located in a static sub-folder in the source code.
Please delete the previous fpc-* folders, which are deprecated and should not be used.
It has been deployed under GitHub, or you need to download a new version of sqlite3fpc.7z if you used our nightly build from fossil.
This will allow to set "Libraries -fFl" in your FPC project options as safe and sound (...)\static\$(TargetCPU)-$(TargetOS)

The new SQlite3 encryption is based on our SynCrypto unit, so it uses AES-NI acceleration, if available. Performance impact is minimal, much lower than the one included e.g. with wxSqlite3, and with a safer implementation (explicit AES-OFB mode with fast IV derivation and proven PBKDF2 password reduction).
It also allows to use a plain/official/unpatched amalgamation sqlite3.c file, so it is easier to maintain as cross-platform.
You can easily convert existing encrypted databases using IsOldSQLEncryptTable and OldSQLEncryptTablePassWordToPlain functions.

Of course, it will also work with Delphi, so that Win32 statically linked sqlite3.obj will offer this new encryption.

Comments are welcome on our forum, as usual.

2018, Wednesday February 7

Status of mORMot ORM SOA MVC with FPC

In the last weeks/months, we worked a lot with FPC.
Delphi is still our main IDE, due to its better debugging experience under Windows, but we target to have premium support of FPC, on all platforms, especially Linux.

The new Delphi Linux compiler is out of scope, since it is heavily priced, its performance is not so good, and ARC broke memory management so would need a deep review/rewrite of our source code, which we can't afford - since we have FPC which is, from our opinion,  a much better compiler for Linux.
Of course, you can create clients for Delphi Linux and FMX, as usual, using the cross-platform client parts of mORMot. But for server side, this compiler is not supported, and will probably never be.

Continue reading...

2018, Tuesday January 2

Happy New mORMot Year


Happy New mORMot Year 2018!

And thanks Mario for the video!

2017, Saturday November 11

Job Opportunities at LiveMon

You like working with mORMot ?

We're hiring new developers at LiveMon

We're a full-remote team (with a monthly gather-up in Paris), working on a real-time AI-powered monitoring tool.
I joined LiveMon this week, and we will work together on this exciting project, using mORMot and FPC!

Contact us at jobs at livemon dot com !

PS: You can send your resumé to this email address, preferably with link to some code you wrote (we like reading clean code!).
We offer remote work from home, if possible from Western Europe (for easy gathering in Paris) or the USA (for LiveMon, Inc.), using technical English and object pascal as language (French not required).

2017, Tuesday October 24

EKON 21 Slides

After having enjoyed EKON 21 conferences in Köln, some quick post to share material about my presentations.

I also included the "Practical DDD" source code in a new sample folder of the mORMot repository.

This sample was long-awaited from new mORMot users, and anyone willing to see how DDD may be implemented in Delphi.
I choose to store the sample in enumerated sub-folders (01, 02, 03...), to show the typical steps of building such kind of services.

Of course, it follows Test-Driven Design, so we write the test as soon as we add a new feature.
Stubing/mocking is also used for the Domain regression tests.
And we will eventually add new sub-folders, to include new features and services.

About EKON, sad I can't share some fresh beer as pdf or in a git/fossil repository...

2017, Thursday August 10

Faster and cross-platform SynLZ

You probably know about our SynLZ compression unit, in pascal and x86 asm, which is very fast for compression with a good compression ratio, and proudly compete with LZ4 or Snappy.
It is used in our framework everywhere, e.g. for WebSockets communication, for ECC encrypted file content, or to compress executable resources. 

Two news to share:

1. I've added SynLZ support for the NextGen compiler, now available in a new unit of the "CrossPlatform" sub-folder.
Feeback is welcome, since we don't use Delphi for iOS and Android with Delphi, and prefer FPC for Linux!

2. I've also written a new x64 asm optimized version of SynLZ, and profiled the existing x86 asm to be even faster than previously.
For a 100MB text log file, SynLZ is faster than Snappy, and compresses better (93% instead of 84%).
For other kind of files, Snappy is slightly faster at decompression, but SynLZ compresses better, and most of the time faster.
When used on a REST server solution, as with mORMot, compression speed does matter more than decompression.

For Win32:

Win32 Processing DragonFly-devpcm.log = 98.7 MB for 1 times
Snappy compress in 125.07ms, ratio=84%, 789.3 MB/s
Snappy uncompress in 70.35ms, 1.3 GB/s
SynLZ compress in 103.61ms, ratio=93%, 952.8 MB/s
SynLZ uncompress in 68.71ms, 1.4 GB/s

For Win64:

Win64 Processing DragonFly-devpcm.log = 98.7 MB for 1 times
Snappy compress in 107.13ms, ratio=84%, 921.5 MB/s
Snappy uncompress in 61.06ms, 1.5 GB/s
SynLZ compress in 97.25ms, ratio=93%, 1015.1 MB/s
SynLZ uncompress in 61.27ms, 1.5 GB/s

Of course, we didn't change the SynLZ binary format, so it is just perfectly backward compatible with any existing program.
Anyway, from my point of view, the main benefit of SynLZ is that it was designed in plain pascal, so it is clearly cross-platform and well integrated with Delphi/FPC (no external .obj/.o/.dll required).

Feedback is welcome in our forum, as usual!

- page 1 of 18