Synopse Open Source

In any modern application, especially on Client/Server nTier architecture as our little mORMot offers, we often have to persist some private keys in a safe way.
Problem with such keys is that they consist in small amount of bytes (typically 16 or 32 bytes), easy to be left somewhere in disk or memory.
Given the abilities of recent forensic data recovery methods, data can't be destroyed on magnetic or flash storage media reliably.

We have just added to our SynCrypto OpenSource library the Anti-forensic Information Splitter algorithm, as proposed in TKS1, and implemented in the LUKS standard.
LUKS is the de-facto standard of platform-independent standard on-disk format for use in various tools.

Once your application is multi-threaded, concurrent data access should be protected. We already wrote about how debugging multi-thread applications may be hard.
Otherwise, a "race condition" issue may appear: for instance, if two threads modify a variable at the same time (e.g. decrease a counter), values may become incoherent and unsafe to use. Another symptom of broken logic is the "deadlock", by which the whole application appears to be blocked and unresponsive, when two threads have a wrong use of the lock, so are blocking each-others.
On a server system, which is expected to run 24/7 with no maintenance, such issues are to be avoided.

In Delphi, protection of a resource (which may be an object, or any variable) is usually done via Critical Sections.
A critical section is an object used to make sure, that some part of the code is executed only by one thread at a time. A critical section needs to be created/initialized before it can be used and be released when it is not needed anymore. Then, some code is protected using Enter/Leave methods, which would lock its execution: in practice, only a single thread would own the critical section, so only a single thread would be able to execute this code section, and other threads would wait until the lock is released. For best performance, the protected sections should be as small as possible - otherwise the benefit of using threads may be voided, since any other thread would wait for the thread owning the critical section to release the lock.

We will now see that Delphi's TCriticalSection may have potential issues, and what our framework proposes to ease critical section use in your applications.

If you compare with existing client/server SOA solutions (in Delphi, Java, C# or even in Go or other frameworks), mORMot's interface-based callback mechanism sounds pretty unique and easy to work with.

Most Events Oriented solutions do use a set of dedicated messages to propagate the events, with a centralized Message Bus (like MSMQ or JMS), or a P2P/decentralized approach (see e.g. ZeroMQ or NanoMsg). In practice, you are expected to define one class per message, the class fields being the message values. You would define e.g. one class to notify a successful process, and another class to notify an error. SOA services would eventually tend to be defined by a huge number of individual classes, with the temptation of re-using existing classes in several contexts.

Our interface-based approach allows to gather all events:

• In a single interface type per notification, i.e. probably per service operation;
• With one method per event;
• Using method parameters defining the event values.

Since asynchronous notifications are needed most of the time, method parameters would be one-way, i.e. defined only as const - in such case, an evolved algorithm would transparently gather those outgoing messages, to enhance scalability when processing such asynchronous events. Blocking request may also be defined as var/out, as we will see below, inWorkflow adaptation.

Behind the scene, the framework would still transmit raw messages over IP sockets (currently over a WebSockets connection), like other systems, but events notification would benefit from using interfaces, on both server and client sides.
We will now see how...

In Delphi code, NULLable types do not exist as such. There is no native int? type, as in C#.
But at SQL and JSON levels, the NULL value does exist and should be converted as expected by the ORM.

In SQLite3 itself, NULL is handled as stated in http://www.sqlite.org/lang_expr.html (see e.g. IS and IS NOT operators).
It is worth noting that NULL handling is not consistent among all existing database engines, e.g. when you are comparing NULL with non NULL values... so we recommend using it with care in any database statements, or only with proper (unit) testing, when you switch from one database engine to another.

By default, in the mORMot ORM/SQL code, NULL will appear only in case of a BLOB storage with a size of 0 bytes.
Otherwise, you should not see it as a value, in most kinds of ORM properties.

Null-oriented value types have been implemented in our framework, since the object pascal language does not allow defining a nullable type (yet). 

We choose to store those values as variant, with a set of TNullable dedicated types, as defined in mORMot.pas:

type
  TNullableInteger = type variant;
  TNullableBoolean = type variant;
  TNullableFloat = type variant;
  TNullableCurrency = type variant;
  TNullableDateTime = type variant;
  TNullableTimeLog = type variant;
  TNullableUTF8Text = type variant;

Among all its means of transmission, our mORMot framework features WebSockets, allowing bidirectional communications, and interface-based callbacks for real time notification of SOA events.
After several months of use in production, we identified some needed changes for this just emerged feature.

We committed today a breaking change of the data layout used for our proprietary WebSockets binary protocol.
From our tests, it would increase the performance and decrease the resource consumption, especially in case of high number of messages.

A lot of our code, and probably yours, is highly relying on text process. In our mORMot framework, most of its features use JSON text, encoded as UTF-8. Profiling shows that a lot of time is spent computing the end of a text buffer, or comparing text content. You may know that In its SSE4.2 feature  […]

One year ago, we published a set of slides about the main concepts implemented by our framework. Mainly about ORM (and ODM), NoSQL, JSON, SOA, MVC (and MVVM), SOLID, DDD, CQRS and some patterns like Stubs, Mocks, Factory, Repository, Unit-Of-Work. Worth a look, if you want to find out the benefits  […]

We identified and fixed today several issues which may affect applications creating a lot of threads (i.e. not using a thread pool). The symptom was an unexpected access violation, when you reach a multiple of 256 threads count. You should better upgrade to at least revision 1.18.1351 if your  […]

When publishing SOA services, most of them are defined as stateless, in a typical query/answer pattern - see Service-Oriented Architecture (SOA).
This fits exactly with the RESTful approach of Client-Server services via interfaces, as proposed by the framework.

But it may happen that a client application (or service) needs to know the state of a given service. In a pure stateless implementation, it will have to query the server for any state change, i.e. for any pending notification - this is called polling.

Polling may take place for instance:

• When a time consuming work is to be processed on the server side. In this case, the client could not wait for it to be finished, without raising a timeout on the HTTP connection: as a workaround, the client may start the work, then ask for its progress status regularly using a timer and a dedicated method call;
• When an unpredictable event is to be notified from the server side. In this case, the client should ask regularly (using a timer, e.g. every second), for any pending event, then react on purpose.

It may therefore sounds preferred, and in some case necessary, to have the ability to let the server notify one or several clients without any prior query, nor having the requirement of a client-side timer:

• Polling may be pretty resource consuming on both client and server sides, and add some unwanted latency;
• If immediate notification is needed, some kind of "long polling" algorithm may take place, i.e. the server will wait for a long time before returning the notification state if no event did happen: in this case, a dedicated connection is required, in addition to the REST one;
• In an event-driven systems, a lot of messages are sent to the clients: a proper publish/subscribe mechanism is preferred, otherwise the complexity of polling methods may increase and become inefficient and unmaintainable;
• Explicit push notifications may be necessary, e.g. when a lot of potential events, associated with a complex set of parameters, are likely to be sent by the client.

Our mORMot framework is therefore able to easily implement asynchronous callbacks over WebSockets, defining the callbacks as interface parameters in service method definitions - see Available types for methods parameters.

As stated during TSQLRecord fields definition, the ORM is able to maintain a revision number for any TSQLRecord table, so that it the table may be easily synchronized remotely by another TSQLRestServer instance.
If you define a TRecordVersion published property, the ORM core will fill this field just before any write with a monotonically increasing revision number, and will take care of any deletion, so that those modifications may be replayed later on any other database.

This synchronization will work as a strict master/slave replication scheme, as a one-way on demand refresh of a replicated table.
Each write operation on the master database on a given table may be easily reflected on one or several slave databases, with almost no speed nor storage size penalty.