Synopse

Performance issue in NextGen ARC model

A.Bouchez — Tue, 21 May 2013 08:55:00 +0200

Apart from being very slow during compilation, the Delphi NextGen compiler introduced a new memory model, named ARC.

We already spoke about ARC years ago, so please refer to our corresponding blog article for further information, especially about how Apple did introduce ARC to iOS instead of the Garbage Collector model.

About how ARC is to be used in the NextGen compiler, take a look at Marco's blog article, and its linked resources.

But the ARC model, as implemented by Embarcadero, has at least one huge performance issue, in the way weak references, and zeroing weak pointers have been implemented.
I do not speak about the general slow down introduced during every class/record initialization/finalization, which is noticeable, but not a big concern.

If you look at XE4 internals, you will discover a disappointing global lock introduced in the RTL.

The main issue is that XE4 RTL implements weak references with a global lock, which will slow down the whole process a lot, especially in multi-thread.

procedure TInstHashMap.RegisterWeakRef(Address: Pointer; Instance: Pointer);
var
  H: Integer;
  Item: PInstItem;
begin
  Lock;
  try
    H := Hash(Instance);
    Item := FindInstItem(Instance, H);
    if Item = nil then
      Item := AddInstItem(Instance, H);
    Item.RegisterWeakRef(Address);
  finally
    Unlock;
  end;
end;

The Lock/Unlock methods are implemented via TMonitor.
This synchronization class has the benefit to be cross-platform, but the drawback of being slower than other approaches.

Such a global lock will just kill the performance in multi-thread process.

Our weak pointer implementation for interfaces in mORMot uses a diverse approach, with a list per class type and small critical sections, so will be much more multi-thread friendly than XE4's implementation.

We can use ARC when targeting mobile platforms.
But in its current implementation, ARC would be a disaster about performance for a server application.

I'm waiting for XE5!

Please do not kill Delphi desktop/server application performance!

In October last year we were already speaking about this global lock implementation issue, when we discovered a pre-version of it in the XE3 RTL source code.
And the version shipped with XE4 did not improve anything.
How could EMB say that performance is a concern for them?
The more I see it, the more I think that enforcing strings to be immutable for performance reasons is just a joke, when you look at the current RTL state.

Are we the milch cow of Embarcadero?

A.Bouchez — Mon, 20 May 2013 11:23:00 +0200

So Marco did answer to our blog article.
Thanks a lot for taking your time, Marco!
I knew you are an open minded fellow.

When I read this Marco's proposal to make an Open Source third-party library to support AnsiStrings, I was stunned.

This demonstrates a naive (and I know he is not such a guy) misunderstanding of what Open Source is.
Open Source is not the milch cow of big Companies.
Open Source is not the solution when you do not want to fix their own problems.

Marco is fair enough to propose his own contribution to the project.
But why on earth won't they make it part of the official RTL?
But why on earth wouldn't they make it compiler-supported, as it always was?
It is 100% part of their duty.

I understand the fact that Extended was not supported any more at compiler level in Win64.
It did make sense at compiler level (SSE2 only supports single or double).
And some third party users proposed a third party library to handle it via records.

But for such low-level feature as AnsiString, since there was no compiler-level nor CPU-level explanation, I do not see any reason to get rid of it.
LLVM is perfectly able to handle both AnsiString and UnicodeString, with the COW paradigm.

Open Source

IMHO the main point about Open Source is not it is free (as a beer), but that it is free (as a bird).

I mean, the community is always worth consulting and learning from.
I mean, you can contribute to the project, from the internals, if you are good enough.
I mean, if you do not like the roadmap or orientation, you are free to fork the project.

Milch cow

Marco's answer sounded to me as:

"OK. I admit I'm not so happy with this breaking change.
But we do not want to maintain the existing code base.
We do want to make our life easier.
So we get rid of AnsiString/RawByteString/UTF8String.
If some of you are hurt and will loose a lot of money because of OUR decision, let's do together a workaround, with performance penalties, and we will sleep well tonight".

Andreas Hausladen did great work on fixing IDEs.
His work is not Open Source (some part is), but his process sounds just familiar to the one suggested by Marco.
Someone is doing EMB's work to fix their mistakes.
BTW Andy, why is there no fix pack for XE4 yet?

Performance penalties

Implementing a string type with a record type and a nested TBytes will be definitively slower than the previous implementation, for several obvious reasons:

You will loose the COW pattern (TBytes only have reference counting);
You will loose a lot of compiler magic;
All TBytes are first filled with 0 whereas AnsiStrings are not (and do not need to - only the latest byte needs to be 0);
Record initialization, finalization and assignments were never optimized for speed in the Delphi compiler and RTL.

IDE integration

Since we are requested to do EMB's work, do not forget to write some debugger extension for the Delphi IDE.

In fact, we need a mean of seeing a string content, with characters, and not a set of bytes, when we are debugging our new AnsiString type.

NextGen is SlowGen

By the way, I was stunned when I saw how slow was the so-called NextGen Compiler.

I do not know if it is the new front-end or the LLVM back-end which is so slow, but the NextGen compiler is more than 10 or 20 times slower than the Win32 compiler!
The Win64 edition was already slower, but to a certain extend.
I did not make accurate benchmark, but it sounds just another regression.

Compilation speed is another big advantage of Delphi which is just voided by this NextGen.

If I wanted something slow, I would use Visual Studio and C#.
In comparison, my Delphi compiler was so fast.
OK, I admit I may abuse of the "build all" feature. Some kind of superstition, I guess...
Working on both C# and Delphi everyday, I'm still so pleased when I switch to Delphi, and let full projects compiled almost with no delay.

Conclusion

Marco's answer did not solve any issue of my previous post.

I never wrote that it would not be possible to replace AnsiChar by Byte.
I wrote it would be difficult to migrate, slower, and less convenient.
What we expect is to have such support at compiler level.

And now I'm convinced that I will have to pay twice: first to buy new Delphi version (and even more for the new Mobile target), then to spend time developing / debugging / maintaining what they do not want do do.

Fair enough?

"The shorter code, the better?"

A.Bouchez — Sun, 19 May 2013 13:24:00 +0200

One quick Sunday post, from a comment I wrote in a blog article.

I'm always wondering why a lot of programmers tend to implicitly assume that "the shorter source code, the better".

It is true when it means that with proper code refactoring, making small objects with dedicated methods, the code of your methods will be smaller.
It is true when you do not like to write as a "Copy & Paste" coder, without searching to put common code in shared places.

But is it true at the language level?
I mean, is it just because your ARC or GC model allow you not to manage the object memory, that it is always better?

Just some ideas...

1. Delphi interface types already allow to write such short code

First of all, it’s worth saying that you can use interfaces to write perfectly safe and short code, without ARC, just with the current version of Delphi.
See for instance how is implemented this GDI+ library.

Implicit try..finally free blocks are already written by the compiler.

2. Delphi Owner property already allows you to manage object life time in the VCL

The whole Delphi components life time is based on ownership.
A form owns its sub components.
A module owns its sub components.
And so on...

It is when the owner is freed, that all its components are also released.

Nice and easy.
Safe and efficient.

3. Code length does not make it less readable

We all know that we read much more code than we write.

So first priority is to let the code be as readable as possible.
IMHO try…finally free patterns do not pollute the readability of the code.
On the contrary, from my own taste, it shows exact scope of a class instance life time.

4. Managing life time objects can help you write better code

When you manage the object life time, you are not tempted to re-create the same again and again. You factorize, optimize your code. And it results… in faster code.

I have seen plenty of Java and C# programmers how do not give a clue about memory allocation and internal process: they write code which works, without asking themselves what will be executed in fact.
Then, especially on a server side, performance scaling is a nightmare.
It works on the developer PC, but won't pass the first performance tests pass.

Having to manage memory life time by hand does not bother me.
On the contrary, it made me a better (less bad) lazy programmer.
And it helped me write faster code, which I know what it is doing when executed.

Of course, it is not automatic.
You can just write try…finally blocks and weep, without searching for code refactoring.
I have seen Delphi code written like that, especially from programmers with a GC model background.

So do not be afraid to learn how to manage your memory!

5. Managing life time objects is worth learning

I was a bit afraid about managing memory, when I came from old BASIC and ASM programming on 8 bit systems (good old days!).
A time where there was no heap, but only static allocation, with less than 64 KB of RAM.
It was working well. And such programs can run for years without any memory leak!

But managing life time is a good way of known how your objects are implemented.
When using an object method, you are not just getting the right result, you are calling perhaps a lot of process.
Worth looking at the internals.

In practice, when writing efficient code, in a GC world, you will have to learn a lot of unofficial information from the runtime designers, to know how GC is implemented.
As such, performance may vary on one revision of the runtime engine, in comparison to another.
If you manage your object life time by hand, you know what you are doing.

The ARC model is in the middle.
But introduces some issues, like need for weak references, and zeroing weak pointers.
AFAIK the RTL implements weak references with a global lock using the very slow TMonitor, which will slow down the whole process a lot, especially in multi-thread (whereas the weak pointer implementation in mORMot core is much more scalable, by the way).
BTW, in October last year I was already speaking about this global lock implementation issue, when I discovered a pre-version of it in the XE3 RTL source code. And the version shipped with XE4 did not improve anything.
And could they say that performance is a concern for them? Forcing the immutable strings use for performance is a just joke, when you look at the current RTL.

6. Source code size has nothing to do with execution speed

In practice, the more the compiler magic or the runtime will execute under the hood, the slower it will be.
So shorter code is most of the time slower code.

Of course, I know that using some high-level structures (like a hashed dictionary or an optimized sort) can be much faster than using a manual search (with a for ... loop), or writing a naive bubble sort.
It does not mean that the more verbose code means the faster.
But my point is that if you rely on some hidden low-level mechanisms, like memory handling, auto-generated structures (like closures), or some RTTI-based features, you will probably write less code, but it will be slower, or less stable.

If you do not handle memory, you are not able to tune the execution process, when needed.
It is not for nothing that the most speed-effective Java projects just use POJO and statically allocated instances, to by-pass the GC.

Worth some minutes thinking about...

Delphi XE4 NextGen compiler: using byte instead of ansichar?

A.Bouchez — Sat, 11 May 2013 10:59:00 +0200

When I first read the technical white paper covering all of the language changes in XE4 for mobile development (tied to the new ARM LLVM-based Delphi compiler), I have to confess I was pretty much confused.

Two great mORMot users just asked for XE4/iOS support of mORMot.

Win32/Win64 support for XE4 will be done as soon as we got a copy of it.
I suspect the code already works, since it was working as expected with XE3, and we rely on our own set of low-level functions for most internal work.

But iOS-targetting is more complex, due to the NextGen compiler, mainly.

FireMonkey

It is first time we are explicitly asked for non-Windows support of mORMot.
This is the reason why we did not put the cross-platform item of the roadmap at first place.

We did not do any support for this yet, because:

No one did ask for cross-platform use of mORMot;
FireMonkey was broken several times, has some part of it very poorly written, and do not support L2R languages - is it mature enough?
iOS support was broken once - and I prefer FPC to this NextGen compiler (see below);
We do not use FireMonkey in any of our applications;
SmartMobileStudio is an innovative, fast growing, cheap, and stable alternative (with lack of documentation and 3rd party components, I admit);
We also considered WxForms (which seems not supported any more, but did work well);
Linux support is a goal for mORMot, on the server side.

Immutable non-AnsiString

Immutable strings are something I do not understand well, in the context of Delphi.

I still do not understand any benefit, in comparison to the copy-on-write paradigm (COW) implemented in Delphi since the beginning for reference-counted value types.
With COW, you have the advantages of immutable strings and private copies and in-place modification, if needed, e.g. for fast parsing.
COW can allow your text buffer access to be safe and fast at the same time.

Using "array of byte" as a workaround from AnsiString/RawByteString is possible, but will be slower and less convenient.
It will implements COW (if const is used as expected in method parameters), but it will fill the content with zeros, so slow down the process.
And it won't be displayed as text in the debugger, nor allow direct conversion to string.

Honestly, changing from everything from AnsiChar to Byte is just an awful workaround and breaking change.
Just like a regression from the modern/turbo Pascal paradigm to a low-level C data type.

The switch introduced by NextGen/ARM/LLVM is IMHO much bigger than the one introduced with Delphi 2009.
For instance, for third party libraries (like our Open Source mORMot), you can maintain an existing code base for all versions of Delphi (e.g. Delphi 6 up to XE4), but you will have to maintain two versions of the code (or nest it with IFDEF) if you want to support NextGen syntax.

I understand that conversion to NextGen compiler can be easy.

See for instance how TMS reported it to be not difficult for Aurelius.
But... do not forget that it may be on the depend of the performance.
Using pointers is not evil, if done with knowledge of it.
See this user feedback about FireBird ODBC access using Aurelius or our Open Source mORMot (which allows remote access, by the way, in addition to plain ORM).

IMHO this is one of the great features of compiled object pascal, in comparison to managed code, or the "NextGen" model.
My point is that pointers are not evil, especially for performance.
Of course, I'm speaking about typed pointers, not blank untyped pointers.

Huge code modification... for nothing?

We could switch the mORMot code to be Next-Gen compatible, but since we use UTF-8 at the lowest level, it will need a lot of IFDEF.
Using "array of byte" instead of "AnsiString(CP_UTF8)" and "byte" instead of "AnsiChar" is just an awful regression and compatibility break.

We would have to use a lot of function wrappers, or perhaps re-create at hand a UTF-8 compatibility layer.
The whole mORMot core is depending on UTF-8, and IMHO this was *not* a wrong choice, on the contrary.

But why go in this direction?

I'm confused with the Embarcadero NextGen compiler.
Performance is not a goal. The RTL is just worse at every Delphi version.
... and compilation time is just dead slow, in comparison to the "PreviousGen" compiler. More than 20 times slower.
Is it worth it?

Deprecation of AnsiString was never prepared by Embarcadero.
We knew it about shortstring - OK.
We were told that the with keyword is the root of all evil, and should be avoided - OK.
But deprecation of AnsiString in the NextGen compiler sounds like a showstopper to me.

And don't tell me it is required by the LLVM compiler to have immutable strings and UTF-16 encoding.
This is a pure Embarcadero choice.

And don't tell me it is for performance optimization.
The Delphi RTL can be dead slow and not scalable.
Current string process was not the main speed bottleneck.
And you have easy alternatives to circumvent those bottlenecks and unleash your CPU power.

And do not tell me that TStringBuilder is the answer.
In-place parsing of buffers is the fastest mean e.g. for JSON or XML performance.
TStringBuilder just replaces classic string concatenation of mutable strings associated with a modern memory manager.
It is a workaround to circumvent a performance problem. There is no benefit of using it.

I was impressed by the Win64 support of latest versions of Delphi.
Very small breaking changes, when adapting mORMot to this platform.
IDE stable enough.
Resulting executable fast enough (at least when it relies on our SynCommons unit, and with some tuned asm code).
Even nice features where re-introduced into the compiler, after a complain in the newsgroups, like ability to compile x64 assembler functions/methods.
But here, I do not understand the direction.

Future Object Pascal

I do not want Delphi to be another managed-but-compiled language.
I like the object pascal language because:

It has the benefits of high-level languages like C# or Java, with readability, strong typing, classes, generics, interfaces, dedicated (ansi/wide)char type, string and dynamic array reference counted types;
It has some nice features I miss in C# for instance, like sets, enumerates or array of enumerations;
The unit layout, with clear distinction between interface and implementation sections, is very powerful, in respect to C# or Java all-in-one syntax;
It has the low-level power of C, if needed, especially for the library cores;
It has a truly working class system (you can use TMyClassClass = class of TMyClass with success);
It has a strong typing paradigm, which mitigates e.g. the use of pointer;
Memory can be managed just as we need, with no Garbage Collector glitches;
We have the whole RTL source code at hand, still readable (the C# RTL is much bigger and complex);
It compiles very quickly, even for huge projects;
It generates stand-alone applications, with no dll hell;
It has a strong backward compatible history, with huge code available on the Internet.

If I want a C# or Java syntax, I would switch to those.
Trying to make Delphi more C#-like is a mistake.
Just think how the attributes syntax in modern Delphi is not pascal-oriented: it should be defined after the type definition, as in the free pascal syntax, not before it, as in C#/Java.

Please fix NextGen roadmap!

Using byte instead of AnsiChar is IMHO not a feature.
This is not "next generation".
This is a regression.
This is a breaking show-stopper.

I suspect (hope?) Embarcadero will be clever enough to re-introduce AnsiString process to their compiler.
Forget about zero-based strings.
Allow mutable strings and pointer access to their buffer.

Otherwise, I honestly do not have much hope of continuing in this direction.
Not worth it.

The "next gen" object pascal is not C#.
It may be the Oxygene compiler, even if I do not find so appealing a cross-compiler language with no cross-platform library.
It may either be FreePascal and its FCL/RTL, when targeting native compilation.
Either DelphiWebScript / SmartMobileStudio, when targeting interpreted / JIT / JavaScript platforms.

Your feedback is needed!

I understand that I may be one of the last one using Delphi as such.
That is, using Delphi not as a RAD tool, but as a strong platform to build huge applications.
With some unique benefits, in regard to alternatives (mainly Java or C#).
Huge list of users of mORMot, and continuous feedback on the forum, tends to prove that I'm not the only one!

Knowing that x64 assembler was re-introduced to Win64 compiler.
That Client-Server licensing was re-allowed after some newsgroup protest.

I do not want to troll, just to let Delphi have a long life!
I like Delphi, I do not want to stay with Delphi 7 for ever, I want the platform to be maintained and evolving!
But this NextGen roadmap may kill my hope, and switch to FPC and SMS.

Feedback is welcome on our forum!

Or even better please react on the Embarcadero forum directly!

Update: a lot of "reaction" did occur in the Embarcadero forum.
But no Embarcadero official did react by now.
Even Team-B members agreed about the need of a petition to integrate feedback of a lot of existing Delphi users.
Stay tuned!

mORMots know how to swim like fishes

A.Bouchez — Wed, 24 Apr 2013 15:53:00 +0200

Another great video by warleyalex.

This time, a full FishFacts demo in AJAX, using mORMot and its SQLite3 ORM as server.

See it on YouTube!

Feedback is welcome on our forum.

Update:

I've just uploaded the corresponding source code to our repository.
See sample 19 - AJAX ExtJS FishFacts.
You need to download the corresponding DB file to run the sample.
Enjoy!

TDataSet... now I'm confused

A.Bouchez — Mon, 22 Apr 2013 07:34:00 +0200

You perhaps know that I'm not a big fan of the TDataSet / RAD DB approach for end-user applications.
They are easy to define, almost no code to write, and you are able to publish a working solution very fast.

But it is a nightmare to debug and maintain. I prefer the new DataBinding feature, or... of course... ORM!
In mORMot, we have some auto-generated screens, and in our roadmap, we forcast to use some auto-binding features, using a KISS by-convention MVC pattern.

For some users, we made a ORM / TDataSet conversion unit.
And we discovered that TDataSet has a weird, and very misleading definition of its AsString property, for Unicode versions of Delphi.

In the Delphi 2009+ implementation, you have to use AsString property for AnsiString and AsWideString for string=UnicodeString.

In fact, the As*String properties are defined as such:

property AsString: string read GetAsString write SetAsString;
property AsWideString: UnicodeString read GetAsWideString write SetAsWideString;
property AsAnsiString: AnsiString read GetAsAnsiString write SetAsAnsiString;

How on earth may we be able to find out that AsString: string returns in fact an AnsiString?
It just does not make sense at all, when compared to the rest of the VCL/RTL.

The implementation, which uses TStringField class for AnsiString and TWideStringField for string=UnicodeString just appear to be broken.

Furthermore, the documentation is also broken:

Data.DB.TField.AsString
Represents the field's value as a string (Delphi) or an AnsiString (C++).

This does not represent a string in Delphi, but an AnsiString!
The fact that the property uses a plain string=UnicodeString type is perfectly misleading.

From the database point of view, it is up to the DB driver to handle Unicode or work with a specific charset.
But from the VCL point of view, in Delphi 2009+ you should only know about one string type, and be confident that using AsString: String will be Unicode-ready.

If you use our mORMotVCL.pas unit, it will behave as expected.
Thanks you sjerinic for your input and patience about this issue!
Feedback is welcome!

Two videos about EXTjs client of mORMot server

A.Bouchez — Tue, 02 Apr 2013 20:35:00 +0200

Two nice videos, posted by a framework user.

The first one presents a remote RESTful access of a SQLite3 database, hosted by a mORMot server:

After one post in the forum, warleyalex was able to easily add remote filtering of the request:

In addition to the previous video about security (by which the mORMot authentication model seems much more secure than DataSnap's), this is a very nice demo!
Thanks a lot, warleyalex for the feedback and information!

Adding some generic-based methods to mORMot

A.Bouchez — Thu, 28 Mar 2013 16:51:00 +0100

We have just added a TSQLTable.ToObjectList<T: class>() method.

I want to add some generics-based methods to mORMot, for newer versions of Delphi.

What do you think of it?
Have you some method prototypes to propose?

My first concern is that FreePascal does not offer the same syntax.
I suppose some $ifdef... will be necessary here..

Feel free to post your proposal to the forum.

Introducing TSQLTable.Step() method

A.Bouchez — Wed, 27 Mar 2013 17:26:00 +0100

We have just added TSQLTable.Step(), FieldBuffer() and Field() methods, handling a cursor at TSQLTable level, with optional late-binding column access.

It allows to retrieve results from a TSQLTable / TSQLTableJSON result sets within a "cursor-like" orientation.
That is, no need to specify the row number, but write a simple while aList.Step do ... loop.

Of course, you should better use TSQLRecord.FillPrepare most of the time, and access the data from a TSQLRecord instance.
But it can be very useful, e.g. when working on a custom JOINed SQL statement.

TSQLTableJSON will expect some JSON content as input, will parse it in rows and columns, associate it with one or more optional TSQLRecord class types, then will let you access the data via its Get* methods.

You can use this TSQLTableJSON class as in the following example:

procedure WriteBabiesStartingWith(const Letters: RawUTF8; Sex: TSex);
var aList: TSQLTableJSON;
    Row: integer;
begin
  aList := Client.MultiFieldValues(TSQLBaby,'ID,BirthDate',
    'Name LIKE ? AND Sex = ?',[Letters+'%',ord(Sex)]);
  if aList=nil then
    raise Exception.Create('Impossible to retrieve data from Server');
  try
    for Row := 1 to aList.RowCount do
      writeln('ID=',aList.GetAsInteger(Row,0),' BirthDate=',aList.Get(Row,1));
  finally
    aList.Free;
  end;
end;

For a record with a huge number of fields, specifying the needed fields could save some bandwidth. In the above sample code, the ID column has a field index of 0 (so is retrieved via aList.GetAsInteger(Row,0)) and the BirthDate column has a field index of 1 (so is retrieved as a PUTF8Char via aList.Get(Row,1)). All data rows are processed via a loop using the RowCount property count - first data row is indexed as 1, since the row 0 will contain the column names.

The TSQLTable class has some methods dedicated to direct cursor handling, as such:

procedure WriteBabiesStartingWith(const Letters: RawUTF8; Sex: TSex);
var aList: TSQLTableJSON;
begin
  aList := Client.MultiFieldValues(TSQLBaby,'ID,BirthDate',
    'Name LIKE ? AND Sex = ?',[Letters+'%',ord(Sex)]);
  try
    while aList.Step do
      writeln('ID=',aList.Field(0),' BirthDate=',aList.Field(1));
  finally
    aList.Free;
  end;
end;

By using the TSQLTable.Step method, you do not need to check that aList<>nil, since it will return false if aList is not assigned. And you do not need to access the RowCount property, nor specify the current row number.

We may have used not the field index, but the field name, within the loop:

      writeln('ID=',aList.Field('ID'),' BirthDate=',aList.Field('BirthDate'));

You can also access the field values using late-binding and a local variant, which gives some perfectly readable code:

procedure WriteBabiesStartingWith(const Letters: RawUTF8; Sex: TSex);
var baby: variant;
begin
  with Client.MultiFieldValues(TSQLBaby,'ID,BirthDate'
    'Name LIKE ? AND Sex = ?',[Letters+'%',ord(Sex)]) do
  try
    while Step(false,@baby) do
      writeln('ID=',baby.ID,' BirthDate=',baby.BirthDate);
  finally
    Free;
  end;
end;

In the above code, late-binding will search for the "ID" and "BirthDate" fields at runtime. But the ability to write baby.ID and baby.BirthDate is very readable. Using a with ... do statement makes the code shorter, but should be avoided if it leads into confusion, e.g. in case of more complex process within the loop.

See also the following methods of TSQLRest: OneFieldValue, OneFieldValues, MultiFieldValue, MultiFieldValues which are able to retrieve either a TSQLTableJSON, or a dynamic array of integer or RawUTF8. And also List and ListFmt methods of TSQLRestClient, if you want to make a JOIN against multiple tables at once.

Feedback is welcome in our forum, as usual.

Delphi is just a perfect fit for the average programmer

A.Bouchez — Tue, 26 Mar 2013 20:20:00 +0100

On the Embarcadero forums, some user did have a perfectly sane reaction, about a non obvious integer type cast like Int64Var := Int32Var*Int32Var, which may overflow.

We've got to stop becoming, as one poster put it, "human pre-compilers" for Delphi.
The compiler ought to have the common sense to not need the programmer to cast the two integer values.

I respectfully think just the opposite.

Such a type cast is part of the language grammar.
If you know the grammar, you will know how it will be compiled.

To be honest, you have the same in all languages, with more or less range checking, optimization, and implicit conversion.
This is why I like Delphi: it can be mastered by any programmers, whereas truly mastering Java or .Net needs a genius.

Delphi is just... human...

This is exactly what I like so much with Delphi, just as with C or C++.
I like to know which asm code will be generated.
I like to know how memory is allocated/released.
I do not like to play with a black box, but I enjoy to rely on identified source code (e.g. the Delphi RTL).
I like to know how the hardware will react to my code.
I like to run a profiler and see what is going on here.
I like to know how my software will run on the customer network and database.

When you know how the language and the compiler works, when you know about the various data structures used and generated by the compiler and the RTL, you are more efficient with it.
Compilers are just tools.
Computer languages are just abstraction of the hardware.

Even with more "high level" languages like JavaScript, you have to use some dedicated structures, and be aware of the potential of the JIT compiler, if you want the process to be as efficient as possible.

One of the problems of today's programmers is that most of them do not know any more what is "under the hood".
Coding is seen as a list of recipes.
It is pretty easy to (ab)use of Intellisense/Resharper, Linq, and whatever the language/IDE is offering to you, and you forget about what will be executed.
Everything is fine on your own computer, but on the customer's side, it does not work as expected.
It just remembers me coding in ZX81 basic... already typing on a black box... good old days doing plenty of things with 768 bytes of free RAM...

You can be a lazy programmer (which is something good, IMHO), and at the same time know what happens.
In fact, a true lazy programmer needs to know how it works.

Delphi is simple enough to be understood by an average human being like me (after years of daily efforts), whereas I consider mastering .Net or Java frameworks is out of scope of my poor brain.
Object pascal is a abstract enough to allow high-level style of coding (e.g. as I've done with SynProject), and also low-level enough to release the whole hardware potential.

SynProject tool 1.18

A.Bouchez — Mon, 25 Mar 2013 22:58:00 +0100

We have uploaded an updatetd compiled version of our Open Source SynProject tool in SynProject.zip.

Synopse SynProject is an open source application for code source versioning and automated documentation of software projects.
Licensed under a GPL license.

Main feature is a new (better-looking?) template for the generated files.
See our mORMot framework documentation for a good sample of rendering content.

The internal wiki pages related to this tool has also been refreshed.

Feedback is welcome on our forum!

Download latest version of sqlite3.dll for Windows 64 bit

A.Bouchez — Sat, 23 Mar 2013 10:16:00 +0100

Up to now, there is no official Win64 version of the SQlite3 library released in http://sqlite.org..
It is in fact very difficult to find a ready-to-use and up-to-date SQLite3-64.dll from Internet, for Win64.

You can find our own 3.7.16 version of the SQlite3 external library, to be used in 64 bit mode, to be downloaded from SQLite3-64.7z.

It includes FTS3/FTS4 virtual tables, and was compiled in release mode.

Compiled with the latest version of Rad Studio XE3, which uses the LLVM 64 bit compiler as back-end.
Thanks to Hans for compiling and sharing the binary!

This is the version we use when our mORMot framework targets Win64, using Delphi XE2/XE3.

x64 optimized asm of FillChar() and Move() for Win64

A.Bouchez — Wed, 13 Mar 2013 13:18:00 +0100

We have included x64 optimized asm of FillChar() and Move() for Win64 - for corresponding compiler targets, i.e. Delphi XE2 and XE3.
It will handle properly cache prefetch and appropriate SSE2 move instructions.

The System.pas unit of Delphi RTL will be patched at startup, unless the NOX64PATCHRTL conditional is defined.
Therefore, whole application may benefit for this optimized version.

Performance improvement is noticeable, when compared with the original pascal-based version included in System.pas.

By the way, the Delphi x64 built-in assembler does not recognize the movnti opcode... so we had to inline it as plain db hexadecimal values.
A bit disappointing. Until now, we did not suffer from anything in regard to the x64 compatibility at Delphi level.

No stand-alone unit available yet, since it is included in our SynCommons.pas shared unit, starting with the 1.18 revision of mORMot.

Feedback is welcome, as usual!

64 bit compatibility of mORMot units

A.Bouchez — Thu, 07 Mar 2013 17:19:00 +0100

I'm happy to announce that mORMot units are now compiling and working great in 64 bit mode, under Windows.
Need a Delphi XE2/XE3 compiler, of course!

ORM and services are now available in Win64, on both client and server sides.
Low-level x64 assembler stubs have been created, tested and optimized.
UI part is also available... that is grid display, reporting (with pdf export and display anti-aliasing), ribbon auto-generation, SynTaskDialog, i18n... the main SynFile demo just works great!

Overall impression is very positive, and speed is comparable to 32 bit version (only 10-15% slower).

Speed decrease seems to be mostly due to doubled pointer size, and some less optimized part of the official Delphi RTL.
But since mORMot core uses its own set of functions (e.g. for JSON serialization, RTTI support or interface calls or stubbing), we were able to release the whole 64 bit power of your hardware.

Delphi 64 bit compiler sounds stable and efficient. Even when working at low level, with assembler stubs.
Generated code sounds more optimized than the one emitted by FreePascalCompiler - and RTL is very close to 32 bit mode.
Overall, VCL conversion worked as easily than a simple re-build.
Embarcadero's people did a great job for VCL Win64 support, here!

SQlite3 works great in 64 bit mode.

You can find our own 3.7.16 version of the SQlite3 external library, to be used in 64 bit mode, from SQLite3-64.7z, since there is no official Win64 library released yet in http://sqlite.org
No problem so far, and pretty good performance.
Jut a weird bug about SQLITE_TRANSIENT constant, which should be pointer(integer(-1)) instead of pointer(-1) when working with virtual tables columns - but nothing to care of in your user code, since the framework will handle it for you.

I suspect some part of official System.RTTI.pas unit as provided in XE2/XE3 is broken in Win64.
For instance, I think it does not handle a method returning a string.
Our mORMot.pas implementation has been tested with the same regression code as in 32 bit mode.

Your own tests and feedback are welcome!
Feedback and detailed tests results are available in our forum!

Using external MinGW/VisualC++ sqlite3.dll - including benchmark

A.Bouchez — Mon, 25 Feb 2013 23:18:00 +0100

With upcoming revision 1.18 of the framework, our SynSQlite3.pas unit is able to access the SQLite3 engine in two ways:

Either statically linked within the project executable;
Or from an external sqlite3.dll library file.

The SQLite3 APIs and constants are defined in SynSQlite3.pas, and accessible via a TSQLite3Library class definition. It defines a global sqlite3 variable as such:

var
  sqlite3: TSQLite3Library;

To use the SQLite3 engine, an instance of TSQLite3Library class shall be assigned to this global variable. Then all mORMot's calls will be made through it, calling e.g. sqlite3.open() instead of sqlite3_open().

There are two implementation classes:

Class	Unit	Purpose
`TSQLite3LibraryStatic`	`SynSQLite3Static.pas`	Statically linked engine (`.obj` within the `.exe`)
`TSQLite3LibraryDynamic`	`SynSQLite3.pas`	Instantiate an external `sqlite3.dll` instance

Referring to SynSQLite3Static.pas in the uses clause of your project is enough to link the .obj engine into your executable.

Warning - breaking change: before version 1.18 of the framework, link of static .obj was forced - so you must add a reference to SynSQLite3Static in your project uses clause to work as expected.

In order to use an external sqlite3.dll library, you have to set the global sqlite3 variable as such:

 FreeAndNil(sqlite3); // release any previous instance (e.g. static)
 sqlite3 := TSQLite3LibraryDynamic.Create;

Of course, FreeAndNil(sqlite3) is not mandatory, and should be necessary only to avoid any memory leak if another SQLite3 engine instance was allocated (may be the case if SynSQLite3Static is referred somewhere in your project's units).

Here are some benchmarks, compiled with Delphi XE3, run in a 32 bit project, using either the static bcc-compiled engine, or an external sqlite3.dll, compiled via MinGW or Microsoft Visual C++.

Static bcc-compiled .obj

First of all, our version included with SynSQLite3Static.pas unit, is to be benchmarked.

Writing speed

	Direct	Batch	Trans	Batch Trans
SQLite3 (file full)	477	389	97633	122865
SQLite3 (file off)	868	869	96827	125862
SQLite3 (mem)	84642	108624	104947	135105
TObjectList (static)	338478	575373	337336	572147
TObjectList (virtual)	338180	554446	331873	575837
SQLite3 (ext full)	486	496	101419	7011
SQLite3 (ext off)	799	303	105402	135109
SQLite3 (ext mem)	93893	129550	109027	152811

Reading speed

	By one	All Virtual	All Direct
SQLite3 (file full)	26924	494559	500200
SQLite3 (file off)	27750	496919	502714
SQLite3 (mem)	124402	444404	495392
TObjectList (static)	332778	907605	910249
TObjectList (virtual)	331038	404891	905961
SQLite3 (ext full)	102707	261547	521322
SQLite3 (ext off)	131130	255806	513505
SQLite3 (ext mem)	135784	248780	502664

Good old Borland C++ builder produces some efficient code here.
Those numbers are very good, when compared to the other two options.
Probably, using FastMM4 as memory manager and tuned compilation options do make sense.

Official MinGW-compiled sqlite3.dll

Here we used the official sqlite3.dll library, as published in the http://sqlite.org web site, and compiled with the MinGW/GCC compiler.

Writing speed

	Direct	Batch	Trans	Batch Trans
SQLite3 (file full)	418	503	86322	119420
SQLite3 (file off)	918	873	93196	127317
SQLite3 (mem)	83108	106951	99892	138003
TObjectList (static)	320204	573723	324696	547465
TObjectList (virtual)	323247	563697	324443	564716
SQLite3 (ext full)	501	410	100152	133679
SQLite3 (ext off)	913	438	102806	135545
SQLite3 (ext mem)	96028	122798	108363	150920

Reading speed

	By one	All Virtual	All Direct
SQLite3 (file full)	26883	473529	438904
SQLite3 (file off)	27729	472188	451304
SQLite3 (mem)	116550	459432	457959
TObjectList (static)	318248	891265	905469
TObjectList (virtual)	327739	359040	892697
SQLite3 (ext full)	127346	180812	370288
SQLite3 (ext off)	127749	227759	438096
SQLite3 (ext mem)	129792	224386	436338

Visual C++ compiled sqlite3.dll

The Open Source wxsqlite project provides a sqlite3.dll library, compiled with Visual C++, and including RC4 and AES 128/256 encryption (better than the basic encryption implemented in SynSQLite3Static.pas) - not available in the official library.

See http://sourceforge.net/projects/wxcode/files/Components/wxSQLite3 to download the corresponding source code, and compiled .dll.

Writing speed

	Direct	Batch	Trans	Batch Trans
SQLite3 (file full)	470	498	93801	112170
SQLite3 (file off)	886	819	90298	132883
SQLite3 (mem)	86897	110287	105207	140896
TObjectList (static)	332005	596445	321357	570776
TObjectList (virtual)	327225	585000	329272	579240
SQLite3 (ext full)	459	503	91086	140599
SQLite3 (ext off)	501	519	110338	150394
SQLite3 (ext mem)	98112	133276	117346	158634

Reading speed

	By one	All Virtual	All Direct
SQLite3 (file full)	28527	516689	521159
SQLite3 (file off)	28927	513769	519156
SQLite3 (mem)	127740	529100	523176
TObjectList (static)	335053	869262	879352
TObjectList (virtual)	334739	410374	885269
SQLite3 (ext full)	132594	258371	506277
SQLite3 (ext off)	138159	260892	507717
SQLite3 (ext mem)	139567	254919	516208

Under Windows, the Microsoft Visual C++ compiler gives very good results.
It is a bit faster than the other two, despite a somewhat less efficient virtual table process.

As a conclusion, our statically linked implementation sounds like the best overall approach: best speed for virtual tables (which is the core of our ORM), and no dll hell.
No library to deploy and copy, everything is embedded in the project executable, ready to run as expected.

Using external SQLite3 is also the open door to easy cross-platform of mORMot.
First step will be to finish the 64 bit compatibility of the framework, using the official x64 SQLite3.dll, which is much easier to work with than linking to .obj in Delphi.

Feedback is welcome in our forum.

Interface-based service sample: remote SQL access

A.Bouchez — Sun, 17 Feb 2013 13:14:00 +0100

You will find in the SQLite3\Sample\16 - Execute SQL via services folder of mORMot source code a Client-Server sample able to access any external database via JSON and HTTP.
It is a good demonstration of how to use an interface-based service between a client and a server.
It will also show how our SynDB classes have a quite abstract design, and are easy to work with, whatever database provider you need to use.

The corresponding service contract has been defined:

  TRemoteSQLEngine = (rseOleDB, rseODBC, rseOracle, rseSQlite3, rseJet, rseMSSQL);

  IRemoteSQL = interface(IInvokable)
    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
    procedure Connect(aEngine: TRemoteSQLEngine; const aServerName, aDatabaseName,
      aUserID, aPassWord: RawUTF8);
    function GetTableNames: TRawUTF8DynArray;
    function Execute(const aSQL: RawUTF8; aExpectResults, aExpanded: Boolean): RawJSON;
  end;

Purpose of this service is:
- To Connect() to external databases, given the parameters of a standard TSQLDBConnectionProperties. Create() constructor;
- Retrieve all table names of this external database as a list;
- Execute any SQL statement, returning the content as JSON array, ready to be consumed by AJAX applications (if aExpanded is true), or a Delphi client (e.g. via a TSQLTableJSON and the mORMotUI unit).

Of course, this service will be define as sicClientDriven mode, that is, the framework will be able to manage a client-driven TSQLDBProperties instance life time.

Benefit of this service is that no database connection is required on the client side: a regular HTTP connection is enough.
No need to install nor configure any database provider, and full SQL access to the remote databases.

Due to our optimized JSON serialization, it will probably be faster to work with such plain HTTP / JSON services, instead of a database connection through a VPN. In fact, database connections are made to work on a local network, and do not like high-latency connections, which are typical on the Internet.

Note that the Execute() method returns a RawJSON kind of variable, which is in fact a sub-type of RawUTF8. Its purpose is to transmit the UTF-8 encoded content directly, with no translation to a JSON string, as would be the case with a RawUTF8 variable. In fact, escaping some JSON array within a JSON string is quite verbose. Using RawJSON in this case ensure the best client-side and server-side speed, and also reduce the transmission bandwidth.

The server part is quite easy to follow:

type
  TServiceRemoteSQL = class(TInterfacedObject, IRemoteSQL)
  protected
    fProps: TSQLDBConnectionProperties;
  public
    destructor Destroy; override;
  public // implements IRemoteSQL methods
    procedure Connect(aEngine: TRemoteSQLEngine; const aServerName, aDatabaseName,
      aUserID, aPassWord: RawUTF8);
    function GetTableNames: TRawUTF8DynArray;
    function Execute(const aSQL: RawUTF8; aExpectResults, aExpanded: Boolean): RawJSON;
  end;

 { TServiceRemoteSQL }
procedure TServiceRemoteSQL.Connect(aEngine: TRemoteSQLEngine; const aServerName, aDatabaseName, aUserID, aPassWord: RawUTF8);
const // rseOleDB, rseODBC, rseOracle, rseSQlite3, rseJet, rseMSSQL
  TYPES: array[TRemoteSQLEngine] of TSQLDBConnectionPropertiesClass = (
     TOleDBConnectionProperties, TODBCConnectionProperties,
     TSQLDBOracleConnectionProperties, TSQLDBSQLite3ConnectionProperties,
     TOleDBJetConnectionProperties, TOleDBMSSQL2008ConnectionProperties);
begin
  if fProps<>nil then
    raise Exception.Create('Connect called more than once');
  fProps := TYPES[aEngine].Create(aServerName,aDatabaseName,aUserID,aPassWord);
end;

function TServiceRemoteSQL.Execute(const aSQL: RawUTF8; aExpectResults, aExpanded: Boolean): RawJSON;
var res: ISQLDBRows;
begin
  if fProps=nil then
    raise Exception.Create('Connect call required before Execute');
  res := fProps.ExecuteInlined(aSQL,aExpectResults);
  if res=nil then
    result := '' else
    result := res.FetchAllAsJSON(aExpanded);
end;

function TServiceRemoteSQL.GetTableNames: TRawUTF8DynArray;
begin
  if fProps=nil then
    raise Exception.Create('Connect call required before GetTableNames');
  fProps.GetTableNames(result);
end;

destructor TServiceRemoteSQL.Destroy;
begin
  FreeAndNil(fProps);
  inherited;
end;

Any exception during SynDB process, or raised manually in case of wrong use case will be transmitted to the client, just as expected. The fProps instance life-time is handled by the client, so all we need is to release its pointer in the service implementation destructor.

From the client point of view, it will be consumed as such:

procedure TMainForm.FormShow(Sender: TObject);
  (...)
  fModel := TSQLModel.Create([],ROOT_NAME);
  fClient := TSQLHttpClient.Create('localhost','888',fModel);
  if not fClient.ServerTimeStampSynchronize then begin
    ShowLastClientError(fClient,'Please run Project16ServerHttp.exe');
    Close;
    exit;
  end;
  if (not fClient.SetUser('User','synopse'))  or
     (not fClient.ServiceRegisterClientDriven(TypeInfo(IRemoteSQL),fService)) then begin
    ShowLastClientError(fClient,'Remote service not available on server');
    Close;
    exit;
  end;
end;

Note the use of ShowLastClientError() function of mORMotUILogin unit, which is able to use our SynTaskDialog unit to report standard and detailed information about the latest error.

Note that in this sample, no table has been defined within the ORM model. It is not necessary, since all external process will take place at the SQL level. As we need authentication (see the call to fClient.SetUser method), the ORM core will by itself add the TSQLAuthUser and TSQLAuthGroup tables to the model - no need to add them explicitly.

From now on, we have a fService: IRemoteSQL instance available to connect and process any remote SQL request.

procedure TMainForm.btnOpenClick(Sender: TObject);
var TableNames: TRawUTF8DynArray;
  (...)
  with fSettings do
    fService.Connect(Engine,ServerName,DatabaseName,UserID,PassWord);
  TableNames := fService.GetTableNames;
  cbbTableNames.Items.Text := UTF8ToString(RawUTF8ArrayToCSV(TableNames,#13#10));
  (...)

Now we are connected to the database via the remote service, and we retrieved the table names in a TComboBox.

Then a particular SQL statement can be executed as such:

procedure TMainForm.btnExecuteClick(Sender: TObject);
var SQL: RawUTF8;
begin
  SQL := trim(StringToUTF8(mmoQuery.Text));
  Screen.Cursor := crHourGlass;
  try
    try
      if isSelect(pointer(SQL)) then begin
        fTableJSON := fService.Execute(SQL,True,False);
        TSQLTableToGrid.Create(drwgrdData,
          TSQLTableJSON.Create([],SQL,pointer(fTableJSON),Length(fTableJSON)),fClient);
      end else
        fService.Execute(SQL,False,False);
    except
      on E: Exception do
        ShowException(E);
    end;
  finally
    Screen.Cursor := crDefault;
  end;
end;

Here, TSQLTableToGrid.Create(), from the mORMotUI unit, will "inject" the returned data to a standard TDrawGrid, using a TSQLTableJSON instance to unserialize the returned JSON content.

Note that in case of any exception (connection failure, or server side error, e.g. wrong SQL statement), the ShowExecption() method is used to notify the user with appropriate information.

Feedback is welcome on our forum.

Introducing ZEOS, UniDAC, NexusDB, BDE, any TDataset to SynDB and mORMot's ORM

A.Bouchez — Tue, 12 Feb 2013 17:42:00 +0100

Up to now, our SynDB database classes were handling ODBC, OleDB providers and direct Oracle or SQLite3 connection.

We have added a DB.pas based layer, ready to be used with UniDAC, NexusDB, or the BDE.
Any other TDataset based component is ready to be interfaced, including UIB, AnyDAC or DBExpress.

The ZEOS library (in its latest 7.0.3 stable version, which works from Delphi 7 up to XE3) has also been interfaced, but without the TDataset/DB.pas layer: our SynDBZEOS.pas unit calls the ZDBC layer, which is not tied to DB.pas nor its RAD components, and is therefore faster. By the way, it will work also with the Starter edition of Delphi (which does not include the DB components) - just like the other "regular" SynDB classes.

This is a work in progress, any testing and feedback is welcome!
We had to circumvent some particularities of the libraries, but I guess we have something interesting.

A dedicated "SynDBDataset" sub-folder has been created in the repository, to contain all SynDBDataset.pas-based database providers.
SynDBNexusDB.pas unit has been moved within this sub-folder, as SynDBUniDAC.pas + SynDBBDE.pas units have been added.
SynDBZeos.pas has a direct access to the ZDBC layer, so is not part of the "SynDBDataset" sub-folder.

Here is some benchmark, mainly about Oracle and SQlite3 database access.
Of course, our direct SynDBOracle / SynDBSQLite3 layers are the fastest around, and we can see that ZDBC layer is sometimes more efficient than the TDataset components.

This preliminary results came from the same hardware than previously used on this blog.
In short: notebook with a SSD and Core i7 mobile.

Note that these tests are not about the relative speed of each database engine, but reflect the current status of the integration of several DB libraries within the mORMot database layer.
Purpose here is not to say that one library is better or faster than another, but publish a snapshot of mORMot persistence layer abilities.
In this timing, we do not benchmark only the "pure" SQL-DB layer access (SynDB* units), but the whole Client-Server ORM of our framework: process below includes read and write RTTI access of a TSQLRecord, JSON marshalling, CRUD/REST routing, virtual cross-database layer, SQL on-the-fly translation. We just bypass the communication layer, since TSQLRestClient and TSQLRestServer are run in-process, in the same thread - as a TSQLRestServerDB instance. So you have here some raw performance testimony of our framework's ORM and RESTful core.

Jet/MSAccess database engine is accessed via OleDB.
NexusDB is the free embedded edition, which is a perfect match for a Client-Server ORM framework like mORMot.
Oracle server is a remote 11g instance, accessed with a "slow" 100 MB network.
SQLite3 is our static embedded engine, either as internal or external database, or the latest 3.7.15.2 official sqlite3.dll library, for ZEOS and UniDAC.
TObjectList is our in-memory optimized engine, with hashed indexes, and able to persist its data as JSON file or an optimized and compressed binary content.

Insertion speed

We use the sample provided within the source code tree, i.e. "15 - External DB performance".

	Direct	Batch	Trans	Batch Trans
SQLite3 (file full)	536	525	93687	113527
SQLite3 (file off)	941	610	94222	98590
SQLite3 (mem)	84722	104727	100425	126974
TObjectList (static)	298560	423657	300951	385089
TObjectList (virtual)	298721	445473	299293	444207
SQLite3 (ext full)	262	259	94916	120293
SQLite3 (ext off)	337	330	103941	112336
SQLite3 (ext mem)	95418	119442	110238	143135
UniDAC SQlite3	239	253	7916	38696
ZEOS SQlite3	372	374	362	467
Oracle	538	62295	1228	62156
ODBC Oracle	639	605	1743	1606
ZEOS Oracle	441	424	1109	1140
UniDAC Oracle	528	557	1061	1237
BDE Oracle	489	511	1024	1003
Jet	4243	4442	4924	4982
NexusDB	4697	6711	5723	8786
ZEOS Firebird	1791	1870	6522	7213
UniDAC Firebird	3670	7135	4751	10125

NexusDB has a pretty good insertion speed, but no "BATCH" mode, nor prepared statement re-use. But not bad for such a small unit.
ZEOS also lacks of some optimizations like prepared statement cache, but does work well for an Open Source free solution.

Read speed

Read speed is comparable to insertion speed.
Depending on the design of each library, results are similar.

BDE performance is not so good, indeed. When working with Oracle, it is pretty slow, especially when retrieving data.
But it (still) works!

Both Jet/MSAccess and NexusDB are pretty good, as embedded engines, for reading.
Of course, our direct SQlite3 or in-memory TObjectList engines are incredibly fast. Just as usual.

	By one	All Virtual	All Direct
SQLite3 (file full)	26755	435995	440683
SQLite3 (file off)	26527	435995	438519
SQLite3 (mem)	126253	438711	423155
TObjectList (static)	289955	689369	681477
TObjectList (virtual)	299383	234126	707714
SQLite3 (ext full)	122524	202208	356760
SQLite3 (ext off)	137358	228102	432376
SQLite3 (ext mem)	139209	228738	432226
UniDAC SQlite3	2001	74155	96140
ZEOS SQlite3	3486	81959	115856
Oracle	1753	77429	92423
ODBC Oracle	1909	36789	42530
ZEOS Oracle	601	47028	55233
UniDAC Oracle	653	25960	28083
BDE Oracle	860	3870	4036
Jet	2605	150051	231792
NexusDB	1360	128816	204976
ZEOS Firebird	2508	56771	71822
UniDAC Firebird	1850	66882	88021

We are working on direct integration of Firebird (either directly via an unfinished SynDBFirebird.pas unit, or via ZEOS / Unidac / any TDataset based component), but we are still fighting with some configuration issues.
Any help is welcome. Don't be shy!

The SAD pdf (in its 1.18 revision) has been updated to include information about external database connection via those libraries.

Feedback is welcome on our forum.

Log to the console

A.Bouchez — Sun, 03 Feb 2013 14:37:00 +0100

Our framework features an integrated logging class, ready to be enabled for support and statistics.

For debugging purposes, it could be very handy to output the logging content to a console window.
It enables interactive debugging of a Client-Server process, for instance: you can interact with the Client, then look in real time at the server console window, and inspect which requests are processed, without the need to open the log file.

Depending on the events, colors will be used to write the corresponding information. Errors will be displayed as light red, for instance.

Logging could be very handy for interactive debug of a client application.

Since our TSynLog / TSQLLog class feature an optional ouput to a console, you are able to see in real-time the incoming requests - see for instance how 14 - Interface based servicesProject14ServerHttp.pas sample is initialized:

begin
  // define the log level
  with TSQLLog.Family do begin
    Level := LOG_VERBOSE;
    EchoToConsole := LOG_VERBOSE; // log all events to the console
  end;
  // create a Data Model
  aModel := TSQLModel.Create([],ROOT_NAME);
  (...)

You can select which events are to be echoed on the console (perhaps you expect only errors to appear, for instance).

Note that this echoing process slow down the logging process a lot, since it is currently implemented in a blocking mode, and flushes each row to the log file. This feature is therefore disabled by default, and not to be enabled on a production server, but only to make interactive debugging easier.

Feedback is welcome on our forum.

External database speed improvements

A.Bouchez — Mon, 28 Jan 2013 08:16:00 +0100

Some major speed improvements have been made to our SynDB* units, and how they are used within the mORMot persistence layer.
It results in an amazing speed increase, in some cases.

Here are some of the optimizations how took place in the source code trunk:

SQL statement client-side cache in ODBC and OleDB;
SQL statement server-side cache for Oracle;
When inserting individual data rows in an external table, the last inserted IDs are maintained in memory instead of executing "select max(id)" - we added a new property EngineAddUseSelectMaxID to unset this optimization - we noted that this modification circumvented a known limitation of Firebird very efficiently.

Overall, I observed from x2 to x10 performance boost with simple Add() operations, using ODBC, OleDB and direct Oracle access, when compare to previous benchmarks (which were already impressive).
BATCH mode performance is less impacted, since it by-passed some of those limitations, but even in this operation mode, there is some benefits (especially with ODBC and OleDB).

Here are some results, directly generated by the supplied "15 - External DB performance" sample.

Insertion speed

Here we test insertion of some records, for most of our supplied engines.
We did the test with UNIK conditional undefined, i.e. with no index of the Name field.

A Core i7 notebook has been used, as hardware platform.
Oracle 11g database is remotely accessed over a corporate network, so latency and bandwidth is not optimal.
The hardrive is a SSD this time - so we will see how it affects the results.

	SQLite3 (file full)	SQLite3 (file off)	SQLite3 (mem)	TObjectList (static)	TObjectList (virtual)	SQLite3 (ext file full)	SQLite3 (ext file off)	SQLite3 (ext mem)	Oracle	ODBC Oracle	Jet
Direct	501	911	81870	281848	288234	548	952	72697	518	512	4159
Batch	523	891	102614	409836	417257	557	868	91617	77155	509	4441
Trans	90388	95884	96612	279579	286188	99681	70950	105674	1024	1432	4920
Batch Trans	110869	117376	125190	412813	398851	127424	126627	121368	62601	1019	4926

Performance gain is impressive, especially for "ODBC Oracle" and also "OleDB Jet".
Since Jet/MSAccess is a local engine, it is faster than Oracle for one record retrieval - it does not suffer from the network latency. But it is faster than SQlite3 at insertion, due to a multi-thread design - which is perhaps less ACID nor proven.
Note that this hardware configuration run on a SSD, so even "SQLite3 (file full)" configuration is very much boosted - about 3 times faster.
Our direct Oracle access classes achieve more than 77,000 inserts per second in BATCH mode (using the Array Binding feature).
Direct TObjectList in-memory engine reaches amazing speed, when used in BATCH mode - more than 400,000 inserts per second!

Read speed

	SQLite3 (file full)	SQLite3 (file off)	SQLite3 (mem)	TObjectList (static)	TObjectList (virtual)	SQLite3 (ext file full)	SQLite3 (ext file off)	SQLite3 (ext mem)	Oracle	ODBC Oracle	Jet
By one	26777	26933	122016	298400	301041	135413	133571	131877	1289	1156	2413
All Virtual	429331	427423	447227	715717	241289	232385	167420	202839	63473	35029	127772
All Direct	443773	433463	427094	711035	700574	432189	334179	340136	90184	39485	186164

Reading speed was also increased. ODBC results have the biggest improvement.
Server-side statement cache for Oracle makes individual reading of records 2 times faster. Wow.
The SQLite3 engine is still the more reactive SQL database here, when it comes to reading.
Of course, direct TObjectList engine is pretty fast - more than 700,000 records per second.

Feedback is welcome on our forum, as usual.

Video about mORMot authentication

A.Bouchez — Sun, 27 Jan 2013 10:13:00 +0100

A new mORMot user notified on our forum that he just made a short video, about authentication and security with our framework, from the perspective of an AJAX Client.
Many thanks for sharing your experiences!

This video illustrate how RESTful authentication is implemented by mORMot.
It compares also with the unsecured scheme used with DataSnap - pretty informative.

Click here to watch the 5 minutes video.

Each time you are logged, a light session is created on the server, and is the root of all mORMot advanced security attributes.
Also a big difference with the heavy implementation of DataSnap session handling.

Note that the 1.18 upcoming revision feature Windows Authentication, i.e. automatic log with your Windows credentials.
With it, it is not even necessary to enter/remember/manage your login/password pair: mORMot is able to use your Windows domain security to let you connected.

Feedback is welcome on our forum.