Synopse

To content | To menu | To search

Tag - blog

Entries feed

2017, Thursday August 10

Faster and cross-platform SynLZ

You probably know about our SynLZ compression unit, in pascal and x86 asm, which is very fast for compression with a good compression ratio, and proudly compete with LZ4 or Snappy.
It is used in our framework everywhere, e.g. for WebSockets communication, for ECC encrypted file content, or to compress executable resources. 

Two news to share:

1. I've added SynLZ support for the NextGen compiler, now available in a new unit of the "CrossPlatform" sub-folder.
Feeback is welcome, since we don't use Delphi for iOS and Android with Delphi, and prefer FPC for Linux!

2. I've also written a new x64 asm optimized version of SynLZ, and profiled the existing x86 asm to be even faster than previously.
For a 100MB text log file, SynLZ is faster than Snappy, and compresses better (93% instead of 84%).
For other kind of files, Snappy is slightly faster at decompression, but SynLZ compresses better, and most of the time faster.
When used on a REST server solution, as with mORMot, compression speed does matter more than decompression.

For Win32:

Win32 Processing DragonFly-devpcm.log = 98.7 MB for 1 times
Snappy compress in 125.07ms, ratio=84%, 789.3 MB/s
Snappy uncompress in 70.35ms, 1.3 GB/s
SynLZ compress in 103.61ms, ratio=93%, 952.8 MB/s
SynLZ uncompress in 68.71ms, 1.4 GB/s

For Win64:

Win64 Processing DragonFly-devpcm.log = 98.7 MB for 1 times
Snappy compress in 107.13ms, ratio=84%, 921.5 MB/s
Snappy uncompress in 61.06ms, 1.5 GB/s
SynLZ compress in 97.25ms, ratio=93%, 1015.1 MB/s
SynLZ uncompress in 61.27ms, 1.5 GB/s

Of course, we didn't change the SynLZ binary format, so it is just perfectly backward compatible with any existing program.
Anyway, from my point of view, the main benefit of SynLZ is that it was designed in plain pascal, so it is clearly cross-platform and well integrated with Delphi/FPC (no external .obj/.o/.dll required).

Feedback is welcome in our forum, as usual!

2017, Wednesday March 22

Delphi 10.2 Tokyo Compatibility: DCC64 broken

We are proud to announce compatibility of our mORMot Open Source framework with the latest Delphi 10.2 Tokyo compiler...
At least for Win32.

For Win64, the compiler was stuck at the end of the compilation, burning 100% of one CPU core...

A bit disappointing, isn't it?

Continue reading...

2017, Saturday March 18

Application Locking using Asymmetric Encryption

A common feature request for professional software is to prevent abuse of published applications.
For licensing or security reasons, you may be requested to "lock" the execution of programs, maybe tools or services.

Our Open-Souce mORMot framework can leverage Asymmetric Cryptography to ensure that only allowed users could run some executables, optionally with dedicated settings, on a given computer.
It offers the first brick on which you may build your own system upon.

From the User point of view, he/she will transmit a user@host.public file, then receives a corresponding user@host.unlock file, which will unlock the application.
Pretty easy to understand - even if some complex asymmetric encryption is involved behind the scene.

Continue reading...

2016, Monday December 19

JSON Web Tokens (JWT)

JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA or ECDSA.

They can be used for:

  • Authentication: including a JWT to any HTTP request allows Single Sign On user validation across different domains;
  • Secure Information Exchange: a small amount of data can be stored in the JWT payload, and is digitally signed to ensure its provenance and integrity.

See http://jwt.io for an introduction to JSON Web Tokens.

Our mORMot framework now implements JWT:

  • HS256 (HMAC-SHA256) and ES256 (256-bit ECDSA) algorithms (with the addition of the "none" weak algo);
  • Validates all claims (validation dates, audiences, JWT ID);
  • Thread-safe and high performance (2 µs for a HS256 verification under x64), with optional in-memory cache if needed (e.g. for slower ES256);
  • Stand-alone and cross-platform code (no external dll, works with Delphi or FPC);
  • Enhanced security and strong design - per instance, it is by design immune from https://auth0.com/blog/2015/03/31/critical-vulnerabilities-in-json-web-token-libraries
  • Full integration with the framework.

Continue reading...

2016, Thursday November 10

EKON20 mORMot Conferences

EKON20 is now over, and there was a lot of people, great speakers, beautiful T-Shirt, and fresh beer!

I've published the slides of my mORMot conferences on SlideShare...

The "classic" Synopse/mORMot slides have also been uploaded to their latest revision, so don't hesitate to check what's new!

The PDF are also available for direct download from our server.

Enjoy!

2016, Tuesday October 18

Cheat Mode for Private Keys

In order to follow best practice, our .private key files are always protected by a password.  A random value with enough length and entropy is always proposed by the ECC tool when a key pair is generated, and could be used directly.
It is always preferred to trust a computer to create true randomness (and SynCrypto.pas's secure TAESPRNG was designed to be the best possible seed, using hardware entropy if available), than using our human brain, which could be defeated by dictionary-based password attacks.
Brute force cracking would be almost impossible, since PBKDF2_HMAC_SHA256 Password-Based Key Derivation Function with 60,000 rounds is used, so rainbow tables (i.e. pre-computed passwords list) will be inoperative, and each password trial would take more time than with a regular Key Derivation Function.

The issue with strong passwords is that they are difficult to remember. If you use not pure random passwords, but some easier to remember values with good entropy, you may try some tools like https://xkpasswd.net/s which returns values like $$19*wrong*DRIVE*read*61$$.
But even then, you will be able to remember only a dozen of such passwords. In a typical public key infrastructure, you may create hundredths of keys, so remembering all passwords is no option for an average human being as (you and) me.

At the end, you end up with using a tool to store all your passwords (last trend is to use an online service with browser integration), or - admit it - store them in an Excel document protected by a password. Most IT people - and even security specialists - end with using such a mean of storage, just because they need it.
The weaknesses of such solutions can be listed:

  • How could we trust closed source software and third-party online services?
  • Even open source like http://keepass.info/help/base/security.html may appear weak (no PBKDF, no AFSplit, managed C#, SHA as PRNG);
  • The storage is as safe as the "master password" is safe;
  • If the "master password" is compromised, all your passwords are published;
  • You need to know the master password to add a new item to the store.

The ECC tool is able to work in "cheat mode", storing all .private key files generated passwords in an associated .cheat local file, encrypted using a cheat.public key.

As a result:

  • Each key pair will have its own associated .cheat file, so you only unleash one key at a time;
  • The .cheat file content is meaningless without the cheat.private key and its master password, so you can manage and store them together with your .private files;
  • Only the cheat.public key is needed when creating a key pair, so you won't leak your master password, and even could generate keys in an automated way, on a distant server;
  • The cheat.private key will be safely stored in a separated place, only needed when you need to recover a password;
  • It uses strong File Encryption, with proven PBKDF, AFSplit, AES-PRNG, and ECDH/ECIES algorithms.

Continue reading...

2016, Saturday September 24

Public-key Asymmetric Cryptography via SynECC

After weeks of implementation and testing, we introduce today a new feature of our mORMot Open-Source Framework.

Asymmetric encryption, also known as public-key cryptography, uses pairs of keys:

  • Public keys that may be disseminated widely;
  • Paired with private keys which are known only to the owner.

The framework SynEcc unit features a full asymmetric encryption system, based on Elliptic curve cryptography (ECC), which may be used at application level (i.e. to protect your application data, by signing or encrypting it), or at transmission level (to enhance communication safety).
A full set of high-level features, including certificates and command line tool, offers a stand-alone but complete public-key infrastructure (PKI).

Continue reading...

2016, Tuesday September 6

Ride the mORMot at EKON 20 in Dusseldorf!

There are still a few days for "very early birds" offer for EKON 20 conference, and meet us for 3 sessions (including a half-day training/introduction to mORMot)!
Join us the 7-9th of November in Düsseldorf!

Our sessions are not restricted to mORMot, but will use mORMot to illustrate some concepts and design ideas:

You can contact me if you want to join, so that I may give you an additional discount password!

Hope we can meet for a chat and (a few) beers!

2016, Saturday May 14

Anti-forensic, safe storage of private keys

In any modern application, especially on Client/Server nTier architecture as our little mORMot offers, we often have to persist some private keys in a safe way.
Problem with such keys is that they consist in small amount of bytes (typically 16 or 32 bytes), easy to be left somewhere in disk or memory.
Given the abilities of recent forensic data recovery methods, data can't be destroyed on magnetic or flash storage media reliably.

We have just added to our SynCrypto OpenSource library the Anti-forensic Information Splitter algorithm, as proposed in TKS1, and implemented in the LUKS standard.
LUKS is the de-facto standard of platform-independent standard on-disk format for use in various tools.

Continue reading...

2016, Friday April 22

Support of Delphi 10.1 Berlin

You should have noticed that Delphi 10.1 Berlin has been released.

Our Open Source projects, including mORMot and SynPDF and their associated documentation have been updated to support this new revision.
Any additional feedback is welcome, as usual!

2016, Saturday April 9

AES-256 based Cryptographically Secure Pseudo-Random Number Generator (CSPRNG)

Everyone knows about the pascal random() function.
It returns some numbers, using a linear congruential generator, with a multiplier of 134775813, in its Delphi implementation.
It is fast, but not really secure. Output is very predictable, especially if you forgot to execute the RandSeed() procedure.

In real world scenarios, safety always requires random numbers, e.g. for key/nonce/IV/salt/challenge generation.
The less predictable, the better.
We just included a Cryptographically Secure Pseudo-Random Number Generator (CSPRNG) into our SynCrypto.pas unit.
The TAESPRNG class would use real system entropy to generate a sequence of pseudorandom bytes, using AES-256, so returning highly unpredictable content.

Continue reading...

2016, Saturday January 9

Safe locks for multi-thread applications

Once your application is multi-threaded, concurrent data access should be protected. We already wrote about how debugging multi-thread applications may be hard.
Otherwise, a "race condition" issue may appear: for instance, if two threads modify a variable at the same time (e.g. decrease a counter), values may become incoherent and unsafe to use. Another symptom of broken logic is the "deadlock", by which the whole application appears to be blocked and unresponsive, when two threads have a wrong use of the lock, so are blocking each-others.
On a server system, which is expected to run 24/7 with no maintenance, such issues are to be avoided.

In Delphi, protection of a resource (which may be an object, or any variable) is usually done via Critical Sections.
A critical section is an object used to make sure, that some part of the code is executed only by one thread at a time. A critical section needs to be created/initialized before it can be used and be released when it is not needed anymore. Then, some code is protected using Enter/Leave methods, which would lock its execution: in practice, only a single thread would own the critical section, so only a single thread would be able to execute this code section, and other threads would wait until the lock is released. For best performance, the protected sections should be as small as possible - otherwise the benefit of using threads may be voided, since any other thread would wait for the thread owning the critical section to release the lock.

We will now see that Delphi's TCriticalSection may have potential issues, and what our framework proposes to ease critical section use in your applications.

Continue reading...

2015, Friday December 11

Audit Trail for Services

We have seen previously how the ORM part of the framework is able to provide an Audit Trail for change tracking.
It is a very convenient way of storing the change of state of the data.

On the other side, in any modern SOA solution, data is not at the center any more, but services.
Sometimes, the data is not stored within your server, but in a third-party Service-Oriented Architecture (SOA).
Being able to monitor the service execution of the whole system becomes sooner or later mandatory.

Our framework allows to create an Audit Trail of any incoming or outgoing service operation, in a secure, efficient and automated way.

Continue reading...

2015, Saturday November 21

Try to avoid RTTI (ab)use

There is a very trendy move, since a few years, to value so called "meta-programming".
In short, it is about the ability to treat programs as their data.
It is a very powerful paradigm in functional languages, and it was also introduced to OOP languages, even in SmallTalk a long time before this concept was trendy in Ruby, C# or Java.

In OOP compiled languages, reflection is used to achieve a similar behavior at run-time, mainly via RTTI (Run-Time Type Information).
Delphi supports RTTI since its version 1, as it was heavily used e.g. for all UI streaming.
In our framework, we rely on RTTI for its main features: ORMSOA and MVC - and even in some other parts, like Desktop UI generation.

But RTTI could easily be abused.
Here are some thoughts, started as a comment in a good old Mason's blog article about how RTTI performance may be a bottleneck.
My comment was to get rid of RTTI, and follow a SOLID implementation with explicit OOP code, like use of interface.

Continue reading...

2015, Tuesday November 17

Benefits of interface callbacks instead of class messages

If you compare with existing client/server SOA solutions (in Delphi, Java, C# or even in Go or other frameworks), mORMot's interface-based callback mechanism sounds pretty unique and easy to work with.

Most Events Oriented solutions do use a set of dedicated messages to propagate the events, with a centralized Message Bus (like MSMQ or JMS), or a P2P/decentralized approach (see e.g. ZeroMQ or NanoMsg). In practice, you are expected to define one class per message, the class fields being the message values. You would define e.g. one class to notify a successful process, and another class to notify an error. SOA services would eventually tend to be defined by a huge number of individual classes, with the temptation of re-using existing classes in several contexts.

Our interface-based approach allows to gather all events:

  • In a single interface type per notification, i.e. probably per service operation;
  • With one method per event;
  • Using method parameters defining the event values.

Since asynchronous notifications are needed most of the time, method parameters would be one-way, i.e. defined only as const - in such case, an evolved algorithm would transparently gather those outgoing messages, to enhance scalability when processing such asynchronous events. Blocking request may also be defined as var/out, as we will see below, inWorkflow adaptation.

Behind the scene, the framework would still transmit raw messages over IP sockets (currently over a WebSockets connection), like other systems, but events notification would benefit from using interfaces, on both server and client sides.
We will now see how...

Continue reading...

2015, Friday October 23

Letters of Hope

As we already notified in this blog, Embarcadero has been finally bought by IDERA.

Delphi users received a letter from Randy Jacops, IDERA CEO.
Written in my mother language, in perfect French. Nice!

The letter states that they have 20,000 customers...
It sounds more realistic than the numbers usually given for Delphi "users".
Even if it counts for all their tools.
:)

In our forums, we have 1,384 registered users (real humans: we do not accept bots via a Turing test during registration).
It sounds like if Open Source projects are able to gather a lot of users.
And certainly because we maintain support from Delphi 6 up to Seattle (and even Delphi 5 for some part of our libraries)... we have for sure users using FPC/Lazarus (which we also started to support), and others which did not upgrade to the latest Delphi version!

In Randy's letter, the community has a special place.
I hope future of Delphi would see Open Source projects brought by the community as a chance, not as competition.

I'm currently working on a cloud of mORMot servers, serving content coming from high numbers of connected objects.
Object Pascal powered servers, under Windows or Linux (with FPC), are working 24/7 with very low resource use.
A lot of BigData stream is gathered into MongoDB servers, following the CQRS pattern.
It is so easy to deploy those servers (including their high performance embedded SQlite3 database), that almost everyone in my company did install their own "cloud", mainly for testing purpose of the objects we are selling...
Real-time remote monitoring of the servers is very easy and integrated. You could even see the log changing in real-time, or run your SQL requests on the databases, with ease.
When I compare to previous projects I had to write or maintain using Java or .Net, I can tell you that it is "something else".
The IT administrators were speechless when they discovered how it worked: no need of containers, no need of virtual machines (but for infrastructure needs)...
The whole stack is SOA oriented, in an Event-Driven design (thanks to WebSockets callbacks). It follows DDD principles, thanks to the perfect readability of the object pascal language.
Delphi, and Open Source, could be great to create Internet Of Things servers...

2015, Monday October 5

Delphi 10 Seattle Win64 compiler Heisenbug: unusable target

Andy reported that he was not able to validate its IDE fix pack for Delphi 10 Seattle, due to its Win64 compiler not being deterministic anymore.
The generated code did vary, from one build to other.

Sadly, on our side, we identified that the code generated by the Win64 compiler of Delphi 10 Seattle is broken.
We have observed some weird code generation with the Win64 platform as a target. Some unexpected exception do occur (like a EPrivilege or EAccessViolation exception).
But it is a random issue, very difficult to reproduce.  After a recompile, no problem any more. Or a problem at another place... A typical Heisenbug...
And, to be clear, no such problem when using an older version of Delphi...

We hope that the corresponding QC entry would be quickly fixed.
So we will stay away from Delphi 10's Win64 compiler, and use Delphi XE8 instead, in the meanwhile.

Update: Issue fixed!
Allen Bauer recognized that "It was an uninitialized memory allocation" in the QC, and that he is pushing to include the fix into the upcoming Seattle 10 Update 1.
Nice seeing such a quick reaction. Delphi is not dead, even if Embarcadero was just acquired! :)

2015, Monday September 28

mORMot show case: Illustrated Spare Parts Catalog

Illustrated Spare Parts Catalog is, as its name suggests, a software for creating and publishing spare parts catalogs.
It uses mORMot for client-server communication and ORM, and SynPdf for the reporting.

Sounds like a powerful solution.
It is also a testimony that you could use big databases (20 GB of blobs) with a SQlite3 engine, and access them via REST using mORMot, without the hassle of setting up a regular RDBMS.

If you (or Google Translate or via this direct link on translate.ru) know a little of Russian, it is worth reading this previous blog article, about how the software author interacted with our Open Source project.
In fact, Chaa did provide a lot of feedback, patches and new features (like direct authentication via Active Directory).
Open Source could be great!

Thanks Chaa for the feedback, and interest!

2015, Friday September 25

ORM TNullable* fields for NULL storage

In Delphi code, NULLable types do not exist as such. There is no native int? type, as in C#.
But at SQL and JSON levels, the NULL value does exist and should be converted as expected by the ORM.

In SQLite3 itself, NULL is handled as stated in http://www.sqlite.org/lang_expr.html (see e.g. IS and IS NOT operators).
It is worth noting that NULL handling is not consistent among all existing database engines, e.g. when you are comparing NULL with non NULL values... so we recommend using it with care in any database statements, or only with proper (unit) testing, when you switch from one database engine to another.

By default, in the mORMot ORM/SQL code, NULL will appear only in case of a BLOB storage with a size of 0 bytes.
Otherwise, you should not see it as a value, in most kinds of ORM properties.

Null-oriented value types have been implemented in our framework, since the object pascal language does not allow defining a nullable type (yet)

We choose to store those values as variant, with a set of TNullable dedicated types, as defined in mORMot.pas:

type
  TNullableInteger = type variant;
  TNullableBoolean = type variant;
  TNullableFloat = type variant;
  TNullableCurrency = type variant;
  TNullableDateTime = type variant;
  TNullableTimeLog = type variant;
  TNullableUTF8Text = type variant;

Continue reading...

2015, Monday September 21

Embarcadero bought by Idera

Just a link found on Internet.

Jefferies is also leading a US$425m covenant-lite credit to back Idera's acquisition of Embarcadero Technologies. Idera is backed by TA Associates. The deal, which launches on Thursday, includes a US$25m revolving credit, a US$300m first-lien term loan and a US$100m second-lien term loan.

Perhaps some company name change!
I wonder what it may lead for Delphi's future. May be some time to "optimize" the acquisition, and "manage" a new line of products.
But if Embarcadero is worth buying, it means that Delphi's is worth betting on!

They may find out that they should focus on the compiler stuff, and that Open Source is a chance, not a competitor.

- page 1 of 13