Synopse

To content | To menu | To search

mORMot Framework

Entries feed

2014, Friday April 18

Introducing mORMot's architecture and design principles

We have just released a set of slides introducing 

  • ORM, SOA, REST, JSON, MVC, MVVM, SOLID, Mocks/Stubs, Domain-Driven Design concepts with Delphi, 
  • and showing some sample code using our Open Source mORMot framework.

You can follow the public link on Google Drive!

This is a great opportunity to discovers some patterns you may not be familiar with, and find out how mORMot try to implement them.
This set of slides may be less intimidating than our huge documentation - do not be terrified by our 1400 pages Software Architecture Design pdf!

Feedback is welcome on our forum, as usual.

2014, Friday March 7

Support of MySQL, DB2 and PostgreSQL

We just tested, benchmarked and validated Oracle MySQL, IBM DB2 and PostgreSQL support for our SynDB database classes and the mORMot's ORM core.
This article will also show all updated results, including our newly introduced multi-value INSERT statement generations, which speed up a lot BATCH insertion.

Stay tuned!

Purpose here is not to say that one library or database is better or faster than another, but publish a snapshot of mORMot persistence layer abilities, depending on each access library.

In this timing, we do not benchmark only the "pure" SQL/DB layer access (SynDB units), but the whole Client-Server ORM of our framework.

Process below includes all aspects of our ORM:

  • Access via high level CRUD methods (Add/Update/Delete/Retrieve, either per-object or in BATCH mode);
  • Read and write access of TSQLRecord instances, via optimized RTTI;
  • JSON marshaling of all values (ready to be transmitted over a network);
  • REST routing, with security, logging and statistic;
  • Virtual cross-database layer using its SQLite3 kernel;
  • SQL on-the-fly generation and translation (in virtual mode);
  • Access to the database engines via several libraries or providers.

In those tests, we just bypassed the communication layer, since TSQLRestClient and TSQLRestServer are run in-process, in the same thread - as a TSQLRestServerDB instance. So you have here some raw performance testimony of our framework's ORM and RESTful core, and may expect good scaling abilities when running on high-end hardware, over a network.

On a recent notebook computer (Core i7 and SSD drive), depending on the back-end database interfaced, mORMot excels in speed, as will show the following benchmark:

  • You can persist up to 570,000 objects per second, or retrieve 870,000 objects per second (for our pure Delphi in-memory engine);
  • When data is retrieved from server or client 38, you can read more than 900,000 objects per second, whatever the database back-end is;
  • With a high-performance database like Oracle, and our direct access classes, you can write 70,000 (via array binding) and read 160,000 objects per second, over a 100 MB network;
  • When using alternate database access libraries (e.g. Zeos, or DB.pas based classes), speed is lower (even if comparable for DB2, MS SQL, PostgreSQL, MySQL) but still enough for most work, due to some optimizations in the mORMot code (e.g. caching of prepared statements, SQL multi-values insertion, direct export to/from JSON, SQlite3 virtual mode design, avoid most temporary memory allocation...).

Difficult to find a faster ORM, I suspect.

Continue reading...

2014, Monday March 3

ORM enhanced for BATCH insert

We just committed some nice features to the ORM kernel, and SynDB* classes of our mORMot framework.

During BATCH insertion, the ORM is able to generate some optimized SQL statements, depending on the target database, to send several rows of data at once.
It induces a noticeable speed increase when saving several objects into an external database.

This feature is available for SQlite3 (3.7.11 and later), MySQL, PostgreSQL, MS SQL Server (2008 and up), Oracle, Firebird and NexusDB.
Since it is working at SQL level, it is available for all supported access libraries, e.g. ODBC, OleDB, Zeos/ZDBC, UniDAC, FireDAC.
It means that even properties not implementing array binding (like OleDB, Zeos or UniDAC) are able to have a huge boost at data insertion, ready to compete with the (until now) more optimized libraries.

Continue reading...

2014, Friday February 28

Are NoSQL databases ACID?

One of the main features you may miss when discovering NoSQL ("Not-Only SQL"?) databases, coming from a RDBMS background, is ACID.

ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties that guarantee that database transactions are processed reliably. In the context of databases, a single logical operation on the data is called a transaction. For example, a transfer of funds from one bank account to another, even involving multiple changes such as debiting one account and crediting another, is a single transaction. (Wikipedia)

But are there any ACID NoSQL database?

Please ensure you read the Martin Fowler introduction about NoSQL databases.
And the corresponding video.

First of all, we can distinguish two types of NoSQL databases:

  1. Aggregate-oriented databases;
  2. Graph-oriented databases (e.g. Neo4J).

By design, most Graph-oriented databases are ACID!
This is a first good point.

Then, what about the other type?
In Aggregate-oriented databases, we can identify three sub-types:

  • Document-based NoSQL databases (e.g. MongoDB, CouchDB);
  • Key/Value NoSQL databases (e.g. Redis);
  • Column family NoSQL databases (e.g. Cassandra).
Whatever document/key/column oriented they are, they all use some kind of document storage.
It may be schema-less, blob-stored, column-driven, but it is always some set of values bound together to be persisted.
This set of values define a particular state of one entity, in a given model.
Which we may call Aggregate.

Continue reading...

2014, Tuesday February 25

TDocVariant custom variant type

With revision 1.18 of the framework, we just introduced two new custom types of variants:

  • TDocVariant kind of variant;
  • TBSONVariant kind of variant.

The second custom type (which handles MongoDB-specific extensions - like ObjectID or other specific types like dates or binary) will be presented later, when dealing with MongoDB support in mORMot, together with the BSON kind of content. BSON / MongoDB support is implemented in the SynMongoDB.pas unit.

We will now focus on TDocVariant itself, which is a generic container of JSON-like objects or arrays.
This custom variant type is implemented in SynCommons.pas unit, so is ready to be used everywhere in your code, even without any link to the mORMot ORM kernel, or MongoDB.

TDocVariant documents

TDocVariant implements a custom variant type which can be used to store any JSON/BSON document-based content, i.e. either:

  • Name/value pairs, for object-oriented documents;
  • An array of values (including nested documents), for array-oriented documents;
  • Any combination of the two, by nesting TDocVariant instances.

Here are the main features of this custom variant type:

  • DOM approach of any object or array documents;
  • Perfect storage for dynamic value-objects content, with a schema-less approach (as you may be used to in scripting languages like Python or JavaScript);
  • Allow nested documents, with no depth limitation but the available memory;
  • Assignment can be either per-value (default, safest but slower when containing a lot of nested data), or per-reference (immediate reference-counted assignment);
  • Very fast JSON serialization / un-serialization with support of MongoDB-like extended syntax;
  • Access to properties in code, via late-binding (including almost no speed penalty due to our VCL hack as already detailed);
  • Direct access to the internal variant names and values arrays from code, by trans-typing into a TDocVariantData record;
  • Instance life-time is managed by the compiler (like any other variant type), without the need to use interfaces or explicit try..finally blocks;
  • Optimized to use as little memory and CPU resource as possible (in contrast to most other libraries, it does not allocate one class instance per node, but rely on pre-allocated arrays);
  • Opened to extension of any content storage - for instance, it will perfectly integrate with BSON serialization and custom MongoDB types (ObjectID, RegEx...), to be used in conjunction with MongoDB servers;
  • Perfectly integrated with our Dynamic array wrapper and its JSON serialization as with the record serialization;
  • Designed to work with our mORMot ORM: any TSQLRecord instance containing such variant custom types as published properties will be recognized by the ORM core, and work as expected with any database back-end (storing the content as JSON in a TEXT column);
  • Designed to work with our mORMot SOA: any interface-based service is able to consume or publish such kind of content, as variant kind of parameters;
  • Fully integrated with the Delphi IDE: any variant instance will be displayed as JSON in the IDE debugger, making it very convenient to work with.

To create instances of such variant, you can use some easy-to-remember functions:

  • _Obj() _ObjFast() global functions to create a variant object document;
  • _Arr() _ArrFast() global functions to create a variant array document;
  • _Json() _JsonFast() _JsonFmt() _JsonFastFmt() global functions to create any variant object or array document from JSON, supplied either with standard or MongoDB-extended syntax.

Continue reading...

2014, Friday January 10

RESTful mORMot

Our Synopse mORMot Framework was designed in accordance with Fielding's REST architectural style without using HTTP and without interacting with the World Wide Web.
Such Systems which follow REST principles are often referred to as "RESTful".

Optionally, the Framework is able to serve standard HTTP/1.1 pages over the Internet (by using the mORMotHttpClient / mORMotHttpServer units and the TSQLHttpServer and TSQLHttpClient classes), in an embedded low resource and fast HTTP server.

Continue reading...

REpresentational State Transfer (REST)

Representational state transfer (REST) is a style of software architecture for distributed hypermedia systems such as the World Wide Web.
As such, it is not just a method for building "web services". The terms "representational state transfer" and "REST" were introduced in 2000 in the doctoral dissertation of Roy Fielding, one of the principal authors of the Hypertext Transfer Protocol (HTTP) specification, on which the whole Internet rely.

 

There are 5 basic fundamentals of web which are leveraged to create REST services:

  1. Everything is a Resource;
  2. Every Resource is Identified by a Unique Identifier;
  3. Use Simple and Uniform Interfaces;
  4. Communication is Done by Representation;
  5. Every Request is Stateless.

Continue reading...

2014, Tuesday January 7

Some enhancements to REST routing of interface-based services

We have just committed some enhancements to interface-based service process.

TSQLRestRoutingREST will now recognize several URI schemes, like new  root/Calculator/Add?n1=1&n2=2 alternative could be pretty convenient to be consumed from some REST clients.

Please find here a documentation update.

Continue reading...

2014, Sunday January 5

AES encryption over HTTP

In addition to regular HTTPS flow encryption, which is not easy to setup due to the needed certificates, mORMot proposes a proprietary encryption scheme. It is based on SHA-256 and AES-256/CTR algorithms, so is known to be secure.

You do not need to setup anything on the server or the client configuration, just run the TSQLHttpClient and TSQLHttpServer classes with the corresponding parameters.

Continue reading...

2014, Saturday January 4

Domain-Driven Design: part 4

One year ago, we already made a quick presentation of Domain-Driven Design, in the context of our mORMot framework.
After one year of real-world application of those patterns, it is now time to give more light to DDD.

Let's continue with part 4, which will define Domain-Driven Design as could be implemented with our Synopse mORMot framework

Continue reading...

Domain-Driven Design: part 3

One year ago, we already made a quick presentation of Domain-Driven Design, in the context of our mORMot framework.
After one year of real-world application of those patterns, it is now time to give more light to DDD.

Let's continue with part 3, which will define Domain-Driven Design patterns and principles - this will be the main article of the whole serie!

Continue reading...

Domain-Driven Design: part 2

One year ago, we already made a quick presentation of Domain-Driven Design, in the context of our mORMot framework.
After one year of real-world application of those patterns, it is now time to give more light to DDD.

Let's continue with part 2, which will define Domain-Driven Design high-level model principles.

Continue reading...

Domain-Driven Design: part 1

One year ago, we already made a quick presentation of Domain-Driven Design, in the context of our mORMot framework.
After one year of real-world application of those patterns, and a training made by a great French software designer named Jérémie Grodziski, it is now time to give more light to DDD.

Let's start with part 1, which will be a general introduction to Domain-Driven Design, trying to state how it may be interesting (or not) for your projects.

Continue reading...

2013, Tuesday December 10

JSON record serialization

In Delphi, the record has some nice advantages:

  • record are value objects, i.e. accessed by value, not by reference - this can be very convenient, e.g. when defining a Domain-Driven Design
  • record can contain any other record or dynamic array, so are very convenient to work with (no need to define sub-classes or lists); 
  • record variables can be allocated on stack, so won't solicit the global heap; 
  • record instances automatically freed by the compiler when they come out of scope, so you won't need to write any try..finally Free; end block.

Serialization of record values are therefore a must-have for a framework like mORMot.

In recent commits, this JSON serialization of record has been enhanced.
In particular, we introduced JSON serialization via a new text-based record definition.

Continue reading...

2013, Monday November 4

Updated mORMot database benchmark - including MS SQL and PostgreSQL

On an recent notebook computer (Core i7 and SSD drive), depending on the back-end database interfaced, mORMot excels in speed:

  • You can persist up to 570,000 objects per second, or retrieve more than 900,000 objects per second (for our pure Delphi in-memory engine);
  • When data is retrieved from server or client cache, you can read more than 900,000 objects per second, whatever the database back-end is;
  • With a high-performance database like Oracle and our direct access classes, you can write 65,000 (via array binding) and read 160,000 objects per second, over a 100 MB network;
  • When using alternate database access libraries (e.g. Zeos, or DB.pas based classes), speed is lower, but still enough for most work.

Difficult to find a faster ORM, I suspect.

The following tables try to sum up all available possibilities, and give some benchmark (average objects/second for writing or read).

In these tables:

  • 'SQLite3 (file full/off/exc)' indicates use of the internal SQLite3 engine, with or without Synchronous := smOff and/or DB.LockingMode := lmExclusive;
  • 'SQLite3 (mem)' stands for the internal SQLite3 engine running in memory;
  • 'SQLite3 (ext ...)' is about access to a SQLite3 engine as external database - either as file or memory;
  • 'TObjectList' indicates a TSQLRestServerStaticInMemory instance, either static (with no SQL support) or virtual (i.e. SQL featured via SQLite3 virtual table mechanism) which may persist the data on disk as JSON or compressed binary;
  • 'Oracle' shows the results of our direct OCI access layer (SynDBOracle.pas);
  • 'NexusDB' is the free embedded edition, available from official site;
  • 'Zeos *' indicates that the database was accessed directly via the ZDBC layer;
  • 'FireDAC *' stands for FireDAC library;
  • 'UniDAC *' stands for UniDAC library;
  • 'BDE *' when using a BDE connection;
  • 'ODBC *' for a direct access to ODBC;
  • 'Jet' stands for a MSAccess database engine, accessed via OleDB;
  • 'MSSQL local' for a local connection to a MS SQL Express 2008 R2 running instance (this was the version installed with Visual Studio 2010), accessed via OleDB.

This list of database providers is to be extended in the future. Any feedback is welcome!

Numbers are expressed in rows/second (or objects/second). This benchmark was compiled with Delphi 7, so newer compilers may give even better results, with in-lining and advanced optimizations.

Note that these tests are not about the relative speed of each database engine, but reflect the current status of the integration of several DB libraries within the mORMot database access.

Purpose here is not to say that one library or database is better or faster than another, but publish a snapshot of current mORMot persistence layer abilities.

In this timing, we do not benchmark only the "pure" SQL/DB layer access (SynDB units), but the whole Client-Server ORM of our framework: process below includes read and write RTTI access of a TSQLRecord, JSON marshaling, CRUD/REST routing, virtual cross-database layer, SQL on-the-fly translation. We just bypass the communication layer, since TSQLRestClient and TSQLRestServer are run in-process, in the same thread - as a TSQLRestServerDB instance. So you have here some raw performance testimony of our framework's ORM and RESTful core.

You can compile the "15 - External DB performance" supplied sample code, and run the very same benchmark on your own configuration.

Continue reading...

2013, Tuesday September 10

Thread-safety of mORMot

We tried to make mORMot at the same time fast and safe, and able to scale with the best possible performance on the hardware it runs on.
Multi-threading is the key to better usage of modern multi-core CPUs, and also client responsiveness.

As a result, on the Server side, our framework was designed to be thread-safe.

On typical production use, the mORMot HTTP server will run on its own optimized thread pool, then call the TSQLRestServer.URI method. This method is therefore expected to be thread-safe, e.g. from the TSQLHttpServer. Request method. Thanks to the RESTful approach of our framework, this method is the only one which is expected to be thread-safe, since it is the single entry point of the whole server. This KISS design ensure better test coverage.

Let us see now how this works, and publish some benchmarks to test how efficient it has been implemented.

Continue reading...

2013, Wednesday September 4

HTTPS communication in mORMot

In mORMot, the http.sys kernel mode server can be defined to serve HTTPS secure content.

Yes, mORMots do like sophistication:

When the aUseSSL boolean parameter is set for TSQLHttpServer.Create() constructor, the SSL layer will be enabled within http.sys.
Note that useHttpSocket kind of server does not offer SSL encryption yet.

We will now define the steps needed to set up a HTTPS server in mORMot.

Continue reading...

2013, Monday September 2

Summer videos of mORMot

During this summer, warleyalex did meet some mORMots in the mountains of REST, Java, AJAX and JSON.

(picture may differ from actual user:) )

He did some videos of his experiment with our little rodent.
At this time, there are 11 videos available!

Latest one is a Java client application, communicating with a mORMot server and a SQLite3 database.

Please go to this forum post to find all URIs, and put your feedback.

Thanks a lot for sharing!

2013, Thursday July 4

Let mORmot's applications be even more responsive

In mORmot applications, all the client communication is executed by default in the current thread, i.e. the main thread for a typical GUI application.
This may become an issue in some reported environments.

Since all communication is performed in blocking mode, if the remote request takes long to process (due to a bad/slow network, or a long server-side action), the application may become unresponsive, from the end-user experience.
Even Windows may be complaining about a "non responsive application", and may propose to kill the process, which is far away from an expected behavior.

In order to properly interacts with the user, a OnIdle property has been defined in TSQLRestClientURI, and will change the way communication is handled.
If a callback event is defined, all client communication will be processed in a background thread, and the current thread (probably the main UI thread) will wait for the request to be performed in the background, running the OnIdle callback in loop in the while.

Continue reading...

2013, Sunday June 16

FireDAC / AnyDAC support for mORMot

Our SynDB classes feature now FireDAC / AnyDAC access, with full speed!

Up to now, only UniDAC, BDE or ZEOS components were available as source, but we just added FireDAC / AnyDAC.

FireDAC is an unique set of Universal Data Access Components for developing cross platform database applications on Delphi. This was in fact a third-party component set, bought by Embarcadero to DA-SOFT Technologies (formerly known as AnyDAC), and included with several editions of Delphi XE3 and up. This is the new official platform for high-speed database development in Delphi, in favor to the now deprecated DBExpress.

Our integration within SynDB.pas units and the mORMot persistence layer has been tuned. For instance, you can have direct access to high-speed FireDAC Array DML feature, via the ORM batch process, via so-called array binding.

Continue reading...

- page 1 of 5