Synopse

To content | To menu | To search

2016, Monday February 8

Linux support for Delphi to be available end of 2016

Marco Cantu, product manager of Delphi/RAD Studio, did publish the official RAD Studio 2016 Product Approach and Roadmap.
The upcoming release has a codename known as "BigBen", and should be called Delphi 10.1 Berlin, as far as I understand.

After this summer, another release, which codename is "Godzilla", will support Linux as a compiler target, in its Delphi 10.2 Tokyo release.
This is a very good news, and some details are given.
I've included those official names to mORMot's internal compiler version detection.
Thanks Marco for the information, and pushing in this direction!

My only concern is that it would be "ARC-enabled"...

Continue reading...

2016, Saturday January 9

Safe locks for multi-thread applications

Once your application is multi-threaded, concurrent data access should be protected. We already wrote about how debugging multi-thread applications may be hard.
Otherwise, a "race condition" issue may appear: for instance, if two threads modify a variable at the same time (e.g. decrease a counter), values may become incoherent and unsafe to use. Another symptom of broken logic is the "deadlock", by which the whole application appears to be blocked and unresponsive, when two threads have a wrong use of the lock, so are blocking each-others.
On a server system, which is expected to run 24/7 with no maintenance, such issues are to be avoided.

In Delphi, protection of a resource (which may be an object, or any variable) is usually done via Critical Sections.
A critical section is an object used to make sure, that some part of the code is executed only by one thread at a time. A critical section needs to be created/initialized before it can be used and be released when it is not needed anymore. Then, some code is protected using Enter/Leave methods, which would lock its execution: in practice, only a single thread would own the critical section, so only a single thread would be able to execute this code section, and other threads would wait until the lock is released. For best performance, the protected sections should be as small as possible - otherwise the benefit of using threads may be voided, since any other thread would wait for the thread owning the critical section to release the lock.

We will now see that Delphi's TCriticalSection may have potential issues, and what our framework proposes to ease critical section use in your applications.

Continue reading...

2015, Friday December 11

Audit Trail for Services

We have seen previously how the ORM part of the framework is able to provide an Audit Trail for change tracking.
It is a very convenient way of storing the change of state of the data.

On the other side, in any modern SOA solution, data is not at the center any more, but services.
Sometimes, the data is not stored within your server, but in a third-party Service-Oriented Architecture (SOA).
Being able to monitor the service execution of the whole system becomes sooner or later mandatory.

Our framework allows to create an Audit Trail of any incoming or outgoing service operation, in a secure, efficient and automated way.

Continue reading...

2015, Saturday November 21

Try to avoid RTTI (ab)use

There is a very trendy move, since a few years, to value so called "meta-programming".
In short, it is about the ability to treat programs as their data.
It is a very powerful paradigm in functional languages, and it was also introduced to OOP languages, even in SmallTalk a long time before this concept was trendy in Ruby, C# or Java.

In OOP compiled languages, reflection is used to achieve a similar behavior at run-time, mainly via RTTI (Run-Time Type Information).
Delphi supports RTTI since its version 1, as it was heavily used e.g. for all UI streaming.
In our framework, we rely on RTTI for its main features: ORMSOA and MVC - and even in some other parts, like Desktop UI generation.

But RTTI could easily be abused.
Here are some thoughts, started as a comment in a good old Mason's blog article about how RTTI performance may be a bottleneck.
My comment was to get rid of RTTI, and follow a SOLID implementation with explicit OOP code, like use of interface.

Continue reading...

2015, Tuesday November 17

Benefits of interface callbacks instead of class messages

If you compare with existing client/server SOA solutions (in Delphi, Java, C# or even in Go or other frameworks), mORMot's interface-based callback mechanism sounds pretty unique and easy to work with.

Most Events Oriented solutions do use a set of dedicated messages to propagate the events, with a centralized Message Bus (like MSMQ or JMS), or a P2P/decentralized approach (see e.g. ZeroMQ or NanoMsg). In practice, you are expected to define one class per message, the class fields being the message values. You would define e.g. one class to notify a successful process, and another class to notify an error. SOA services would eventually tend to be defined by a huge number of individual classes, with the temptation of re-using existing classes in several contexts.

Our interface-based approach allows to gather all events:

  • In a single interface type per notification, i.e. probably per service operation;
  • With one method per event;
  • Using method parameters defining the event values.

Since asynchronous notifications are needed most of the time, method parameters would be one-way, i.e. defined only as const - in such case, an evolved algorithm would transparently gather those outgoing messages, to enhance scalability when processing such asynchronous events. Blocking request may also be defined as var/out, as we will see below, inWorkflow adaptation.

Behind the scene, the framework would still transmit raw messages over IP sockets (currently over a WebSockets connection), like other systems, but events notification would benefit from using interfaces, on both server and client sides.
We will now see how...

Continue reading...

2015, Friday October 23

Letters of Hope

As we already notified in this blog, Embarcadero has been finally bought by IDERA.

Delphi users received a letter from Randy Jacops, IDERA CEO.
Written in my mother language, in perfect French. Nice!

The letter states that they have 20,000 customers...
It sounds more realistic than the numbers usually given for Delphi "users".
Even if it counts for all their tools.
:)

In our forums, we have 1,384 registered users (real humans: we do not accept bots via a Turing test during registration).
It sounds like if Open Source projects are able to gather a lot of users.
And certainly because we maintain support from Delphi 6 up to Seattle (and even Delphi 5 for some part of our libraries)... we have for sure users using FPC/Lazarus (which we also started to support), and others which did not upgrade to the latest Delphi version!

In Randy's letter, the community has a special place.
I hope future of Delphi would see Open Source projects brought by the community as a chance, not as competition.

I'm currently working on a cloud of mORMot servers, serving content coming from high numbers of connected objects.
Object Pascal powered servers, under Windows or Linux (with FPC), are working 24/7 with very low resource use.
A lot of BigData stream is gathered into MongoDB servers, following the CQRS pattern.
It is so easy to deploy those servers (including their high performance embedded SQlite3 database), that almost everyone in my company did install their own "cloud", mainly for testing purpose of the objects we are selling...
Real-time remote monitoring of the servers is very easy and integrated. You could even see the log changing in real-time, or run your SQL requests on the databases, with ease.
When I compare to previous projects I had to write or maintain using Java or .Net, I can tell you that it is "something else".
The IT administrators were speechless when they discovered how it worked: no need of containers, no need of virtual machines (but for infrastructure needs)...
The whole stack is SOA oriented, in an Event-Driven design (thanks to WebSockets callbacks). It follows DDD principles, thanks to the perfect readability of the object pascal language.
Delphi, and Open Source, could be great to create Internet Of Things servers...

2015, Monday October 5

Delphi 10 Seattle Win64 compiler Heisenbug: unusable target

Andy reported that he was not able to validate its IDE fix pack for Delphi 10 Seattle, due to its Win64 compiler not being deterministic anymore.
The generated code did vary, from one build to other.

Sadly, on our side, we identified that the code generated by the Win64 compiler of Delphi 10 Seattle is broken.
We have observed some weird code generation with the Win64 platform as a target. Some unexpected exception do occur (like a EPrivilege or EAccessViolation exception).
But it is a random issue, very difficult to reproduce.  After a recompile, no problem any more. Or a problem at another place... A typical Heisenbug...
And, to be clear, no such problem when using an older version of Delphi...

We hope that the corresponding QC entry would be quickly fixed.
So we will stay away from Delphi 10's Win64 compiler, and use Delphi XE8 instead, in the meanwhile.

Update: Issue fixed!
Allen Bauer recognized that "It was an uninitialized memory allocation" in the QC, and that he is pushing to include the fix into the upcoming Seattle 10 Update 1.
Nice seeing such a quick reaction. Delphi is not dead, even if Embarcadero was just acquired! :)

2015, Monday September 28

mORMot show case: Illustrated Spare Parts Catalog

Illustrated Spare Parts Catalog is, as its name suggests, a software for creating and publishing spare parts catalogs.
It uses mORMot for client-server communication and ORM, and SynPdf for the reporting.

Sounds like a powerful solution.
It is also a testimony that you could use big databases (20 GB of blobs) with a SQlite3 engine, and access them via REST using mORMot, without the hassle of setting up a regular RDBMS.

If you (or Google Translate or via this direct link on translate.ru) know a little of Russian, it is worth reading this previous blog article, about how the software author interacted with our Open Source project.
In fact, Chaa did provide a lot of feedback, patches and new features (like direct authentication via Active Directory).
Open Source could be great!

Thanks Chaa for the feedback, and interest!

2015, Friday September 25

ORM TNullable* fields for NULL storage

In Delphi code, NULLable types do not exist as such. There is no native int? type, as in C#.
But at SQL and JSON levels, the NULL value does exist and should be converted as expected by the ORM.

In SQLite3 itself, NULL is handled as stated in http://www.sqlite.org/lang_expr.html (see e.g. IS and IS NOT operators).
It is worth noting that NULL handling is not consistent among all existing database engines, e.g. when you are comparing NULL with non NULL values... so we recommend using it with care in any database statements, or only with proper (unit) testing, when you switch from one database engine to another.

By default, in the mORMot ORM/SQL code, NULL will appear only in case of a BLOB storage with a size of 0 bytes.
Otherwise, you should not see it as a value, in most kinds of ORM properties.

Null-oriented value types have been implemented in our framework, since the object pascal language does not allow defining a nullable type (yet)

We choose to store those values as variant, with a set of TNullable dedicated types, as defined in mORMot.pas:

type
  TNullableInteger = type variant;
  TNullableBoolean = type variant;
  TNullableFloat = type variant;
  TNullableCurrency = type variant;
  TNullableDateTime = type variant;
  TNullableTimeLog = type variant;
  TNullableUTF8Text = type variant;

Continue reading...

2015, Monday September 21

Embarcadero bought by Idera

Just a link found on Internet.

Jefferies is also leading a US$425m covenant-lite credit to back Idera's acquisition of Embarcadero Technologies. Idera is backed by TA Associates. The deal, which launches on Thursday, includes a US$25m revolving credit, a US$300m first-lien term loan and a US$100m second-lien term loan.

Perhaps some company name change!
I wonder what it may lead for Delphi's future. May be some time to "optimize" the acquisition, and "manage" a new line of products.
But if Embarcadero is worth buying, it means that Delphi's is worth betting on!

They may find out that they should focus on the compiler stuff, and that Open Source is a chance, not a competitor.

2015, Thursday September 17

AVAST did detect ALL Delphi programs as dangerous

Today, an avalanche of "false postitive detection" of AVAST heuristic engine did occur.
Any executable built with Delphi XE8 or Delphi 10 Seattle was identified as a Win32:Banker-MGC [Trj] threat!

Heuristic analysis is a method employed by many computer antivirus programs designed to detect previously unknown computer viruses, as well as new variants of viruses already in the "wild".

Continue reading...

2015, Wednesday September 16

Feedback from the Wild

We just noticed a nice feedback from a mORMot user.

Vojko Cendak commented the well-known DataSnap analysis based on Speed & Stability tests blog article written by Roberto some months years (!) ago.
It is not meant to be the final word, perhaps there was some tuning possible for RTC (which is told to be very reliable), but it is worth a look:

We used 3 products: RO, RTC and Mormot.. I won’t speak about RO ( slow and heavy ). We tried RTC but was too very slow and CPU consuming in getting lots of 1000 .. 5000 dynamically fetching OPC tags (let’s say list of small objects) – at least once per second (one client). I mean Mormot is FAST and we’re glad to be so. We use Mormot in actual productions 24/7 on several sites: servers don’t even blink on client requests and run smoothly and reliably.

Thanks for the kind words!
We have a lot of feedback, around the world, from users of our little Open Source project, very happy with its abilities.
We try to make it always better! Open Source, and Delphi as a platform, do rock!

2015, Monday September 14

Performance issue in NextGen ARC model - much better now

Back in 2013, I found out an implementation weakness in the implementation of ARC weak references in the RTL.
A giant lock was freezing all threads and cores, so would decrease a lot the performance abilities of any ARC application, especially in multi thread.

I just investigated that things are now better.

Continue reading...

2015, Monday August 31

Delphi 10 = DX Seattle is out, mORMot supports it

We expected Delphi XE9, and now we have Rad Studio 10 Seattle, with Delphi renamed as Delphi 10 Seattle, or simply DX.

No big news for the Delphi compiler itself (we are still waiting for Linux server support), but a lot of FireMonkey updates, Windows 10 compatibility enhancements, enhancements to JSON (better performance using a SAX approach), and NoSQL/MongoDB support in FireDAC.
The documentation is rather sparse for the new features, but it goes into the right direction (we support MongoDB since a long time, in our ORM/ODM).
See what's new in details.

Of course, our Open Source mORMot framework supports this version.
Feedback is welcome, as usual!
Enjoy the new DX IDE!

2015, Monday August 24

Introducing Delphinus PackageManager

You may have missed the news.
So I relay here the information from Delphinus PackageManager blog article.

Delphinus is a new Package Manager, which runs on Delphi XE and newer, directly from within your IDE.
It is GitHub-based, and fairly easy to add a reference to.

Of course, I tried to add mORMot to the list. We were the first external project linked in!
In contrast to GetIt (which rejected mORMot to be part of it - as we expected), it is very open-minded.

We wish great success to this project, which sounds Open and Efficient!

2015, Sunday August 23

"SQL and NoSQL", not "SQL vs NoSQL"

You know certainly that our mORMot Open Source framework is an ORM, i.e. mapping objects to a relational / SQL database (Object Relational Mapping).
You may have followed also that it is able to connect to a NoSQL database, like MongoDB, and that the objects are then mapped via an ODM (Object Document Mapping) - the original SQL SELECT are even translated on the fly to MongoDB queries.

But thanks to mORMot, it is not "SQL vs NoSQL" - but "SQL and NoSQL".
You are not required to make an exclusive choice.
You can share best of both worlds, depending on your application needs.

In fact, the framework is able to add NoSQL features to a regular relational / SQL database, by storing JSON documents in TEXT columns.

In your end-user code, you just define a variant field in the ORM, and store a TDocVariant document within.
We also added some dedicated functions at SQL level, so that SQLite3 could be used as embedded fast engine, and provide advanced WHERE clauses on this JSON content.

Continue reading...

2015, Saturday August 15

Breaking Change in mORMot WebSockets binary protocol

Among all its means of transmission, our mORMot framework features WebSockets, allowing bidirectional communications, and interface-based callbacks for real time notification of SOA events.
After several months of use in production, we identified some needed changes for this just emerged feature.

We committed today a breaking change of the data layout used for our proprietary WebSockets binary protocol.
From our tests, it would increase the performance and decrease the resource consumption, especially in case of high number of messages.

Continue reading...

2015, Tuesday July 14

New blog about mORMot

An enthusiastic mORMot user, named willo in the forum, just started a blog about his experiments with our framework.

The information there is clear, simple, and right to the point.
If you are a little lost in our huge documentation, it is a good place to start!

Continue reading...

2015, Tuesday June 30

Faster String process using SSE 4.2 Text Processing Instructions STTNI

A lot of our code, and probably yours, is highly relying on text process.
In our mORMot framework, most of its features use JSON text, encoded as UTF-8.
Profiling shows that a lot of time is spent computing the end of a text buffer, or comparing text content.

You may know that In its SSE4.2 feature set, Intel added STTNI (String and Text New Instructions) opcodes.
They are several new instructions that perform character searches and comparison on two operands of 16 bytes at a time.

I've just committed optimized version of StrComp() and StrLen(), also used for our TDynArrayHashed wrapper.
The patch works from Delphi 5 up to XE8, and with FPC - unknown SSE4.2 opcodes have been entered as hexadecimal bytes, for compatibility with the last century compilers!
The resulting speed up may be worth it!

Next logical step would be to use those instruction in the JSON process itself.
It may speed up the parsing speed of our core functions (which is already very optimized, but written in a classical one-char-at-a-time reading).
Main benefit would be to read the incoming UTF-8 text buffer by blocks of 16 bytes, and performing several characters comparison in a few CPU cycles, with no branching.
Also JSON writing would benefit for it, since escaping could be speed up thanks to STTNI instructions.

Any feedback is welcome, as usual!

2015, Sunday June 21

Why FPC may be a better compiler than Delphi

Almost every time I'm debugging some core part of our framework, I like to see the generated asm, and trying to optimize the pascal code for better speed - when it is worth it, of course!
I just made a nice observation, when comparing the assembler generated by Delphi to FPC's output.

Imagine you compile the following lines (extracted from SynCommons.pas), to convert some number into ASCII characters:

...
c100 := val div 100;
dec(val,c100*100);
PWord(P)^ := TwoDigitLookupW[val];
...

This divides a number by 100, then computes the modulo in val, to store two digits at a time. We did not use val := val mod 100 here, since mod would do another division, so we rely on a simple multiplication to compute the modulo.
You may know that for today's CPUs, integer multiplication is very optimized, taking a cycle (or less, thanks to its pipelines), whereas a division is much more expensive - if you have some spare time, take a look at this document, and you will find out that a div opcode could use 10 times more cycles then a mul - even with the latest CPU architectures.
Let's see how our two beloved compilers do their homework (with optimization enabled, of course)...

Delphi generates the following code for c100 := val div 100:

005082AB 8BC1             mov eax,ecx
005082AD BE64000000       mov esi,$00000064
005082B2 33D2             xor edx,edx
005082B4 F7F6             div esi

Whereas FPC generates the following:

0043AC48 8b55f8                   mov    -0x8(%ebp),%edx
0043AC4B b81f85eb51               mov    $0x51eb851f,%eax
0043AC50 f7e2                     mul    %edx
0043AC52 c1ea05                   shr    $0x5,%edx
0043AC55 8955f0                   mov    %edx,-0x10(%ebp)

Even if you are assembler agnostic, and once you did get rid of the asm textual representation (Delphi uses Intel's, whereas FPC/GDB follows AT&T), you can see that Delphi generates a classic (and slow) div esi opcode, whereas FPC uses a single multiplication, followed by a bit shift.

This optimization is known as "Reciprocal Multiplication", and I would let you read this article for mathematical reference - or this one.
It multiplies (mul) the number by the power of two reciprocal of 100 (which is the hexadecimal 51eb851f value), followed by a right shift (shr) of 5 bits.
Thanks to 32 bit rounding of the integer operations, this would in fact divide the number per 100.
Even it consists in two assembler opcodes, a mul + shr is in fact faster than a single div.

It is a shame that the Delphi compiler did not include this very common optimization, which is clearly a win for some very common tasks. 

Of course, the LLVM back-end used on the NextGen compiler can do it, be we may expect this classic optimization be part of the decades-old Delphi compiler.
And I'm still not convinced about the performance of the NextGen generated code, since the associated RTL is known to be slow, so won't benefit of LVVM optimization - which takes  a LOT of time to compile, by the way (much more than FPC).

Congrats, FPC folks!

- page 1 of 17