Synopse

To content | To menu | To search

mORMot Framework

Entries feed

2014, Friday May 30

Software Design, Brook, mORMot, RAD, SOLID and OOP

We got a very instructive discussion in our forums, with Silvio, the maintainer of the Brook Framework.
Brook is a nice framework for writing web applications using Free Pascal.

It comes to my mind what mORMot can offer.
We did not want to compare the features or say that one framework is better than the other, but it appeared to me that a lot of object pascal programmers are tied to 20th century programming model.

In fact, to embrace the potentials of mORMot, you need to switch your mind, and enhanced your RAD and OOP background, into 21th century SOLID model.

Continue reading...

2014, Sunday May 25

New crc32c() function using optimized asm and SSE 4.2 instruction

Cyclic Redundancy Check (CRC) codes are widely used for integrity checking of data in fields such as storage and networking.
There is an ever-increasing need for very high-speed CRC computations on processors for end-to-end integrity checks.

We just introduced to mORMot's core unit (SynCommons.pas) a fast and efficient crc32c() function.

It will use either:

  • Optimized x86 asm code, with unrolled loops;
  • SSE 4.2 hardware crc32 instruction, if available.

Resulting speed is very good.
This is for sure the fastest CRC function available in Delphi.
Note that there is a version dedicated to each Win32 and Win64 platform - both performs at the same speed!

In fact, most popular file formats and protocols (Ethernet, MPEG-2, ZIP, RAR, 7-Zip, GZip, and PNG) use the polynomial $04C11DB7, while Intel's hardware implementation is based on another polynomial, $1EDC6F41 (used in iSCSI and Btrfs).
So you would not use this new crc32c() function to replace the zlib's crc32() function, but as a convenient very fast hashing function at application level.
For instance, our TDynArray wrapper will use it for fast items hashing.

Continue reading...

2014, Sunday May 18

Automatic JSON serialization of record or dynamic arrays via Enhanced RTTI

Since Delphi 2010, the compiler generates additional RTTI at compilation, so that all record fields are described, and available at runtime.
By the way, this enhanced RTTI is one of the reasons why executables did grow so much in newer versions of the compiler.

Our SynCommons.pas unit is now able to use this enhanced information, and let any record be serialized via RecordLoad() and RecordSave() functions, and all internal JSON marshalling process.

In short, you have nothing to do.
Just use your record as parameters, and, with Delphi 2010 and up, they will be serialized as valid JSON objects.

Of course, text-based definition or callback-based registration are still at hand, and will be used with older versions of Delphi.
But you could be used to by-pass or extend the enhanced-RTTI serialization, even on newer versions of the compiler.

Continue reading...

New sample for JSON performance: mORMot vs SuperObject/XSuperObject/dwsJSON/DBXJSON

We have just added a new "25 - JSON performance" sample to benchmark JSON process, using well most known Delphi libraries...

A new fight
featuring
mORMot vs SuperObject/XSuperObject/dwsJSON/DBXJSON

On mORMot side, it covers TDocVariant, late binding, TSQLTable, ORM, record access, BSON...

We tried to face several scenarios:

  • parse/access/write iteration over a small JSON document,
  • read of deeply nested 680 KB JSON (here mORMot is slower than SO/dwsJSON),
  • read of one 180 MB JSON file (with on-the-fly adaptation to fit a record layout),
  • named access to all rows and columns of a 1 MB JSON table, extracted from a SQL request (with comparison with our ORM performance).

On average and in details, mORMot is the fastest in almost all scenarios (with an amazing performance for table/ORM processing), dwsJSON performs very well (better than SuperObject), and DBXJSON is the slowest (by far, but XE6 version is faster than XE4).

Continue reading...

2014, Wednesday May 7

Benchmarking Mustache libraries: native SynMustache vs mustache.js/SpiderMonkey

I just wrote a small sample program, for benchmarking Mustache libraries: native SynMustache vs mustache.js running on SpiderMonkey 24...

And the winner is ...SynMustache, which is 10 times faster, uses almost no memory during process, and handles inlined {{>partials}} natively (whereas we have to handle them manually with mustache.js)!

Who says that Garbage Collection and immutable strings in modern JITted runtimes are faster than "native" Delphi applications?
Are you still preferring the "NextGen" roadmap?

Continue reading...

Direct MongoDB database access

MongoDB (from "humongous") is a cross-platform document-oriented database system, and certainly the best known NoSQL database.
According to http://db-engines.com in April 2014, MongoDB is in 5th place of the most popular types of database management systems, and first place for NoSQL database management systems.
Our mORMot framework gives premium access to this database, featuring full NoSQL and Object-Document Mapping (ODM) abilities to the framework.

Integration is made at two levels:

  • Direct low-level access to the MongoDB server, in the SynMongoDB.pas unit;
  • Close integration with our ORM (which becomes defacto an ODM), in the mORMotMongoDB.pas unit.

MongoDB eschews the traditional table-based relational database structure in favor of JSON-like documents with dynamic schemas (MongoDB calls the format BSON), which matches perfectly mORMot's RESTful approach.

In this first article, we will detail direct low-level access to the MongoDB server, via the SynMongoDB.pas unit.

Continue reading...

2014, Friday May 2

mORMot on GitHub

There was a long-standing request from customers, about putting all our source code repository to GitHub.

We like a lot our self-hosted Fossil repository, and will continue to use it as our main system, including issue tracking and wiki, for our official web site.

But we created a repository on GitHub, on https://github.com/synopse/mORMot

Continue reading...

2014, Monday April 7

JavaScript support in mORMot via SpiderMonkey

As we already stated, we finished the first step of integration of the SpiderMonkey engine to our mORMot framework.
Version 1.8.5 of the library is already integrated, and latest official revision will be soon merged, thanks to mpv's great contribution.
It can be seen as stable, since it is already used on production site to serve more than 1,000,000 requests per day.

You can now easily uses JavaScript on both client and server side.
On server side, mORMot's implementation offers an unique concept, i.e. true multi-threading, which is IMHO a huge enhancement when compared to the regular node.js mono-threaded implementation, and its callback hell.
In fact, node.js official marketing states its non-blocking scheme is a plus. It allows to define a HTTP server in a few lines, but huge server applications need JavaScript experts not to sink into a state a disgrace.

Continue reading...

2014, Saturday March 29

Enhanced and fixed late-binding of variants for Delphi XE2 and up

For several units of our framework, we allow late-binding of data values, using a variant and direct named access to properties:
- In SynCommons, we defined our TDocVariant custom variant type, able to store any JSON/BSON document-based content;
- In SynBigTable, we use the TSynTableVariantType custom variant type, as defined in SynCommons;
- In SynDB, we defined a TSQLDBRowVariantType, ready to access any column of a RDBMS data result set row;
- In mORMot, we allow access to TSQLTableRowVariantType column values.

It's a very convenient way of accessing result rows values. Code is still very readable, and safe at the same time.

For instance, we can write:

var V: variant;
 ...
  TDocVariant.New(V); // or slightly slower V := TDocVariant.New;
  V.name := 'John';
  V.year := 1972;
  // now V contains {"name":"john","year":1982}

This is just another implementation of KISS design in our framework.

Since Delphi XE2, some modifications were introduced to the official DispInvoke() RTL implementation:

  1. A new varUStrArg kind of parameter has been defined, which will allow to transmit UnicodeString property values;
  2. All text property values would be transmitted as BSTR / WideString / varOleStr variants to the invoked variant type;
  3. All textual property names were normalized to be in UPPERCASE.

Those modifications are worth considering...
And we may have discovered two regressions: one about speed, and the other about an unexpected logic bug...

Continue reading...

2014, Thursday March 13

ORM mapping class fields to external table columns

When working with an ORM, you have mainly two possibilites:

  1. Start from scratch, i.e. write your classes and let the ORM creates all the database structure - it is also named "code-first";
  2. From an existing database, you define in your model how your classes map the existing database structure - this is "database-first".

We have just finalized ORM external table field mapping in mORMot, using e.g.
aModel.Props[aExternalClass].ExternalDB.MapField(..)
See this last commit.

So you can write e.g.

fProperties := TSQLDBSQLite3ConnectionProperties.Create(
  SQLITE_MEMORY_DATABASE_NAME,'','','');
VirtualTableExternalRegister(fExternalModel,
  TSQLRecordPeopleExt,fProperties,'PeopleExternal');
fExternalModel.Props[TSQLRecordPeopleExt].ExternalDB.
  MapField('ID','Key').
  MapField('YearOfDeath','YOD');

Then you use your TSQLRecordPeopleExt table as usual from Delphi code, with ID and YearOfDeath fields:

  • The "internal" TSQLRecord class will be stored within the PeopleExternal external table;
  • The "internal" TSQLRecord.ID field will be an external "Key: INTEGER" column;
  • The "internal" TSQLRecord.YearOfDeath field will be an external "YOD: BIGINT" column;
  • Other internal published properties will be mapped by default with the same name to external column.

Continue reading...

2011, Sunday September 25

Synopse SQLite3 framework is now mORMot

In case you were redirected from the previous "Synopse SQLite3 framework" category link, here is the new thread to be used instead:
http://blog.synopse.info/category/Open-Source-Projects/mORMot-Framework

Since revision 1.15 of the framework, it is able to connect to any database engine (therefore is not limited to SQLite3), and is now called mORMot.

2011, Thursday September 1

DataSnap-like Client-Server JSON RESTful Services in Delphi 6-XE5

Article update:
The server side call back signature changed since this article was first published in 2010. 
Please refer to the documentation or this forum article and associated commit.
The article was totally rewritten to reflect the enhancements.
And do not forget to see mORMot's interface-based services!

Note that the main difference with previous implementation is the signature of the service implementation event, which should be now exactly:
procedure MyService(Ctxt: TSQLRestServerURIContext);
(note that there is one unique class parameter, with no var specifier)
Please update your code if you are using method-based services!


You certainly knows about the new DataSnap Client-Server features, based on JSON, introduced in Delphi 2010.
http://docwiki.embarcadero.com/RADStudi … plications

We added such communication in our mORmot Framework, in a KISS (i.e. simple) way: no expert, no new unit or new class. Just add a published method Server-side, then use easy functions about JSON or URL-parameters to get the request encoded and decoded as expected, on Client-side.

Continue reading...

2011, Monday July 4

WinINet vs WinHTTP

If you want to implement an HTTP client access in your application, you may consider several choices:

  • Use the provided Indy components;
  • Use third-party components like Synapse, ICS or your own WinSock-based wrapper;
  • Use WinINet;
  • Use WinHTTP.

For our ORM, we tried to avoid external dependencies, and did not have the need of all Indy's features and overhead.
We fist wrote our own WinSock wrapper, then tried out WinInet.

When used on our testing benchmark, we found out that WinINet was dead slow.
Then we tried WinHTTP, the new API provided by Microsoft, and we found out this was blazing fast. As fast as direct WinSock access, without the need of writing all the wrapper code.

Continue reading...

2011, Sunday July 3

"Sharding" or "Share nothing" architecture

Here is what wikipedia states at http://en.wikipedia.org/wiki/Shared_nothing_architecture:

A shared nothing architecture (SN) is a distributed computing architecture in which each node is independent and self-sufficient, and there is no single point of contention across the system. People typically contrast SN with systems that keep a large amount of centrally-stored state information, whether in a database, an application server, or any other similar single point of contention.

This is just one approach of "sharding". Sharding is indeed related to a shared nothing architecture - once sharded, each shard can live in a totally separate logical schema instance.
"I sharded, therefore it scales"...

You can do this in Delphi... and opens a new world of scaling opportunities... Just as Google, Facebook, or eBay do... 

Continue reading...

2011, Wednesday June 29

Synopse SQLite3 framework 1.14

Our ORM framework has been released as version 1.14.

It's mainly a bug-fix release:

  • Integrated SQLite3 engine updated to latest version 3.7.7.1;
  • Fix several issues about JSON generation layout;
  • Enhanced automated User Interface generation for object on-screen edition;
  • SynPdf unit now handles Bézier curves from TCanvas, and some CMYK functions; also enhanced PDF/A-1compatibility;
  • Some speed enhancements, and new functions for the SynOleDB unit.

Continue reading...

2011, Sunday June 5

Synopse SQLite3 Framework 1.13

This is a major step for the framework.

Among a lot of new features and bug fixes:

Open Source project, for Delphi 6 up to XE, licensed under a MPL/LGPL/GPL tri-license.

Continue reading...

2011, Thursday June 2

Custom SQL functions

The SQLite3 engine defines some standard SQL functions, like abs() min() max() or upper().
A complete list is available at http://www.sqlite.org/lang_corefunc.html

One of the greatest SQLite3 feature is the ability to define custom SQL functions in high-level language. In fact, its C API allows to implement new functions which may be called within a SQL query. In other database engine, such functions are usually named UDF (for User Defined Functions).

Our framework allows you to add easily such custom functions, directly from Delphi classes.

Continue reading...

Fast JSON parsing

When it deals with parsing some (textual) content, two directions are usually envisaged. In the XML world, you have usually to make a choice between:
- A DOM parser, which creates an in-memory tree structure of objects mapping the XML nodes;
- A SAX parser, which reads the XML content, then call pre-defined events for each XML content element.

In fact, DOM parsers use internally a SAX parser to read the XML content. Therefore, with the overhead of object creation and their property initialization, DOM parsers are typically three to five times slower than SAX. But, DOM parsers are much more powerful for handling the data: as soon as it's mapped in native objects, code can access with no time to any given node, whereas a SAX-based access will have to read again the whole XML content.

Most JSON parser available in Delphi use a DOM-like approach. For instance, the DBXJSON unit included since Delphi 2010 or the SuperObject or DWS libraries create a class instance mapping each JSON node.

In a JSON-based Client-Server ORM like ours, profiling shows that a lot of time is spent in JSON parsing, on both Client and Server side. Therefore, we tried to optimize this part of the library.

Continue reading...

BATCH sequences for adding/updating/deleting records

When use the so-called BATCH sequences?

In a standard Client-Server architecture, especially with the common understanding (and most implementations) of a RESTful service, any Add / Update / Delete method call requires a back and forth flow to then from the remote server.

In case of a remote connection via the Internet (or a slow network), you could have some 100 ms of latency: it's just the "ping" timing, i.e. the time spent for your IP packet to go to the server, then back to you.

If you are making a number of such calls (e.g. add 1000 records), you'll have 100*1000 ms = 100 s = 1:40 min just because of this network latency!

The BATCH sequence allows you to regroup those statements into just ONE remote call. Internally, it builds a JSON stream, then post this stream at once to the server. Then the server answers at once, after having performed all the modifications.

Continue reading...

TSQLRecordRTree to implement R-Tree virtual tables

An R-Tree is a special index that is designed for doing range queries.

R-Trees are most commonly used in geospatial systems where each entry is a rectangle with minimum and maximum X and Y coordinates. Given a query rectangle, an R-Tree is able to quickly find all entries that are contained within the query rectangle or which overlap the query rectangle.

 This idea is easily extended to three dimensions for use in CAD systems. R-Trees also find use in time-domain range look-ups. For example, suppose a database records the starting and ending times for a large number of events. A R-Tree is able to quickly find all events, for example, that were active at any time during a given time interval, or all events that started during a particular time interval, or all events that both started and ended within a given time interval. And so forth. See http://www.sqlite.org/rtree.html

Since the 2010-06-25 source code repository update, the RTREE extension is compiled by default within all supplied .obj files of the framework.

A dedicated ORM class, named TSQLRecordRTree, is available to create such tables. It inherits from TSQLRecordVirtual, like the other virtual tables types (e.g. TSQLRecordFTS3 or our custom virtual tables).

Continue reading...

- page 1 of 3