Synopse

Modification of TSQLRestServerCallBack method prototype (bis)

A.Bouchez — Mon, 06 Feb 2012 22:05:00 +0100

In order to implement some RESTful Services, a callback has to be defined on the server side.

The prototype of these methods has been modified one more time, to supply an unique parameter:
This is a CODE BREAK change and you shall refresh ALL your server-side code to match the new signature.

This unique parameter will let the signature remain untouched in your code implementation, even if the framework evolves (like adding a new parameter).

Here is the updated declaration of the class:

TSQLRestServerTest = class(TSQLRestServerDB)
  published
    function Sum(var aParams: TSQLRestServerCallBackParams): Integer;
  end;

This method name will be used for the URL encoding, and will be called here with ModelRoot/Sum URL. The ModelRoot is the one defined in the Root parameter of the model used by the application.

This method, like all Server-side methods, MUST have all parameters of the TSQLRestServerCallBack prototype:

type
  TSQLRestServerCallBack = function(var aParams: TSQLRestServerCallBackParams): Integer of object;

Then we implement this method:

function TSQLRestServerTest.Sum(var aParams: TSQLRestServerCallBackParams): Integer;
var a,b: Extended;
begin
  if not UrlDecodeNeedParameters(aParams.Parameters,'A,B') then
  begin
    result := 404; // invalid Request
    exit;
  end;
  while aParameters<>nil do
  begin
    UrlDecodeExtended(aParams.Parameters,'A=',a);
    UrlDecodeExtended(aParams.Parameters,'B=',b,@aParams.Parameters);
  end;
  aParams.Resp := JSONEncodeResult([a+b]);
  // same as : aResp := JSONEncode(['result',a+b],TempMemoryStream);
  result := 200; // success
end;

On the Server side, you can use the UrlDecodeNeedParameters function to check that an expected parameters were supplied by the caller, then call UrlDecodeInteger / UrlDecodeInt64 / UrlDecodeExtended / UrlDecodeValue functions (all defined in SynCommons.pas) to retrieve each individual parameter as standard JSON content. The powerful UrlDecodeObject function (defined in SQLite3Commons.pas) can be used to unserialize most class instance from its textual JSON representation. Note that due to this implementation pattern, the mORMot service implementation is very fast, and not sensitive to the "Hash collision attack" security issue, as reported with Apache - see this blog entry for details.

The aParams.Session parameter may contain at calling time the current session identifier. If authentication is not used, this parameter is meaningless. Server-side implementation can use the TSQLRestServer.SessionGetUser method to retrieve the corresponding user details (the returned TSQLAuthUser instance is a local thread-safe copy which shall be freed when done).

The aParams.Head parameter may be overridden on the server side to set a custom header which will be provided to the client - it may be useful for instance to specify another mime-type than the default constant JSON_CONTENT_TYPE, i.e. 'application/json; charset=UTF-8'.

You can use the forum to ask additional questions.

Microsoft states: OleDB out - enjoy ODBC!

A.Bouchez — Sun, 29 Jan 2012 14:30:00 +0100

For our native connection to any DB, we developed a set of classes and several units.

We implemented at first OleDB, then native Oracle direct access and SQlite3 static engine.

Now, Microsoft is officially deprecating OleDB, and urge all developers to switch to the open and cross-platform ODBC API for native connection.

It is worth reading the official announcement from Microsoft:

Microsoft is Aligning with ODBC for Native Relational Data Access

...

We encourage you to adopt ODBC in the development of your new and future versions of your application. You don’t need to change your existing applications using OLE DB, as they will continue to be supported on Denali throughout its lifecycle. While this gives you a large window of opportunity for changing your applications before the deprecation goes into effect, you may want to consider migrating those applications to ODBC as a part of your future roadmap.

They created OLE DB to make a "pure Windows" concurrent of ODBC.
We were told that OLE DB was much better than ODBC, by Micro$oft evangelists.

Now they are going back to ODBC because Cross-Platform is now mandatory. This is all marketing and money here. This won't be the first abandoned technology from Microsoft (or any other company).

Open source is another world, even if money is still within, you can always fork a dead project to continue supporting it.

In fact, OleDB was mostly about using MS SQL under Windows.
Oracle provider is buggy - no cry for me...

I'm working on adding ODBC support for our SynDB classes.
This will be necessary for (Linux and) Mac OS support of DBs other than Oracle or SQLite3.
Follow the evolution of the new SynDBODBC.pas unit.

Feel free to discuss about it in our forum.

SynDBExplorer fast direct export

A.Bouchez — Tue, 17 Jan 2012 23:02:00 +0100

The Open Source SynDBExplorer tool has been enhanced these days.

Main new features are:

Execution of the current selected text (if any) instead of the whole memo content;
"Exec & Export" new button, for direct export to file.

I really like the selection execution feature - this speed up SQL process a lot, and allow to switch from one statement to another.
And the new exporting features are opening new possibilities.

In particular, the "Exec & Export" new button is able to export any SQL statement result into:

Regular CSV file (ready to be imported in Excel);
Regular TXT file (columns are separated with a tab character);
Standard JSON content (ready for any AJAX application);
Our more compact JSON format (column names are not duplicated on each row);
Synopse BigTable records (with fixed sized records for integers columns);
Synopse BigTable records (with variable-length records for integers columns);
A SQLite3 database file (creating a table with all necessary columns).

This direct export feature won't use a temporary memory buffer, just with the regular "Export" button, which exports the grid on screen.
Therefore, it will be faster, and will be able to handle any amount of data. You can export gigabytes of data directly from the tool, then consume it offline as regular SQLite3 tables or very efficient Synopse BigTable records. With a map & reduce architecture, it can be very handy to process huge amount of data. Or it can be used to convert an existing database layout into a SQLite3 database.

Thanks to these enhancements, this little tool begins to be a very nice SQL and database management tool.

All this is made possible due to the unique architecture of our SynDB classes. Their design appears to be powerful and open, at the same time: it allows easy mix of very diverse data structures like BigTable or SQLite3, far away from the RAD layout.

A visual Query Designer is also in alpha stage (select several tables, then right click on your mouse), and will become a powerful entry point for the database reverse engineering of our mORMot framework (i.e. the upcoming TSQLRecordMapped* classes). Up to now, you can JOIN visualy your tables with this right-click approach. But it will shortly become a powerful way of managing any RDBMS as objects.

Feedback and comments are welcome in our forum.

Hash collision attack

A.Bouchez — Fri, 30 Dec 2011 10:49:00 +0100

A variety of programming languages suffer from a denial-of-service (DoS) condition against storage functions of key/value pairs in hash data structures, the condition can be leveraged by exploiting predictable collisions in the underlying hashing algorithms.

The issue finds particular exposure in web server applications and/or frameworks. In particular, the lack of sufficient limits for the number of parameters in POST requests in conjunction with the predictable collision properties in the hashing functions of the underlying languages can render web applications vulnerable to the DoS condition. The attacker, using specially crafted HTTP requests, can lead to a 100% of CPU usage which can last up to several hours depending on the targeted application and server performance, the amplification effect is considerable and requires little bandwidth and time on the attacker side.

Source: #2011-003 multiple implementations denial-of-service via hash algorithm collision

How will our mORMot framework handle such DOS attacks?

The current implementation of parameters parsing is not using a hash table, nor using any storage list.

The parameters are parsed and checked only once. No global list is created, just to avoid such DOS problems.

Strong-typing just rocks

A.Bouchez — Sun, 11 Dec 2011 09:59:00 +0100

To my understanding, the so-called "strong-typing" feature is one big benefit of the Delphi object pascal language.

As stated by wikipedia:

Most generally, "strong typing" implies that the programming language places severe restrictions on the intermixing that is permitted to occur, preventing the compiling or running of source code which uses data in what is considered to be an invalid way. For instance, an addition operation may not be used with an integer and string values; a procedure which operates upon linked lists may not be used upon numbers. However, the nature and strength of these restrictions is highly variable.

Some Delphi users may find this is a limitation of the language, in comparison with other "fashionable" script idioms (like Python, Javascript of Ruby). For me, runtime strong typing (alla Python or Ruby) is not true strong typing. Simon Stuart just proposed a smartstring kind of string, which is in fact a weakstring type. As far as I understood his point, he wanted to get rid of all the warnings emitted by Unicode-version of the Delphi compiler, about explicit string conversion.

In fact, I use to go in the opposite direction. For wide projects, strong-typing is one of the big benefit of using Delphi (like other main "serious" languages like Java, C, C++, Ada or C#).

Our SQLite3/mORMot Framework has 100% UNICODE compatibility, that is compilation under Delphi 2009/2010/XE/XE2. The code has been deeply rewritten and tested, in order to provide compatibility with the String=UnicodeString paradigm of these compilers. But the code will also handle safely Unicode for older version, i.e. from Delphi 6 up to Delphi 2007.

Since our framework is natively UTF-8 (this is the better character encoding for fast text - JSON - streaming/parsing and it is natively supported by the SQLite3 engine), we had to establish a secure way our framework used strings, in order to handle all versions of Delphi (even pre-Unicode versions, especially the Delphi 7 version we like so much), and provide compatibility with the Free Pascal Compiler.

Some string types have been defined, and used in the code for best cross-compiler efficiency (avoiding most conversion between formats):
- RawUTF8 is used for every internal data usage, since both SQLite3 and JSON do expect UTF-8 encoding;
- WinAnsiString where WinAnsi-encoded AnsiString (code page 1252) are needed;
- Generic string for i18n (e.g. in unit SQLite3i18n), i.e. text ready to be used within the VCL, as either AnsiString (for Delphi 2 to 2007) or UnicodeString (for Delphi 2009/2010/XE/XE2);
- RawUnicode in some technical places (e.g. direct Win32 *W() API call in Delphi 7) - note: this type is NOT compatible with Delphi 2009/2010/XE/XE2 UnicodeString;
- RawByteString for byte storage (e.g. for FileFromString() function);
- SynUnicode is the fastest available Unicode native string type, depending on the compiler used (i.e. WideString before Delphi 2009, and UnicodeString since);
- Some special conversion functions to be used for Delphi 2009/2010/XE/XE2 UnicodeString (defined inside $ifdef UNICODE...$endif blocks);
- Never use AnsiString directly, but one of the types above.

Note that RawUTF8 is the preferred string type to be used in our framework when defining textual properties in a TSQLRecord and for all internal data processing. It's only when you're reaching the User Interface layer that you may convert explicitly the RawUTF8 content into the generic VCL string type, using either the Language. UTF8ToString method (from SQLite3i18n.pas unit) or the following function from SynCommons.pas:

/// convert any UTF-8 encoded String into a generic VCL Text
// - it's prefered to use TLanguageFile.UTF8ToString() in SQLite3i18n,
// which will handle full i18n of your application
// - it will work as is with Delphi 2009/2010/XE/XE2 (direct unicode conversion)
// - under older version of Delphi (no unicode), it will use the
// current RTL codepage, as with WideString conversion (but without slow
// WideString usage)
function UTF8ToString(const Text: RawUTF8): string;

Of course, the StringToUTF8 method or function are available to send back some text to the ORM layer.
A lot of dedicated conversion functions (including to/from numerical values) are included in SynCommons.pas. Those were optimized for speed and multi-thread capabilities, and to avoid implicit conversions involving a temporary string variable.

In fact, this enforce the n-tier layered architecture of the project:

RawUTF8 is to be used at Storage and Business layers;
string type for the User Interface (at the VCL level).

Warnings during the compilation process are not allowed, especially under Unicode version of Delphi (e.g. Delphi 2010): all string conversion from the types above are made explicitly in the framework's code, to avoid any unattended data loss.

With such an implementation pattern, we keep all our code benefit from the strong typing feature of Delphi, whereas we can handle safely Unicode content even with older Ansi versions of Delphi.

Comments are welcome on our forum.

Avoiding Garbage Collector: Delphi and Apple side by side

A.Bouchez — Thu, 08 Dec 2011 10:09:00 +0100

Among all trolling subject in forums, you'll find out the great Garbage Collection theme.

Fashion languages rely on it. At the core of the .Net and Java framework, and all scripting languages (like JavaScript, Perl, Python or Ruby), you'll find a Garbage Collector. New developers, just released from schools, do learn about handling memory only in theory, and just can't understand how is memory allocated - we all have seen such rookies involved in Delphi code maintenance, leaking memory as much as they type. In fact, most of them did not understood how a computer works. I warned you this will be a trolling subject.

And, in Delphi, there is no such collector. We handle memory in several ways:

Creating static variables - e.g. on the stack, inside a class or globally;
Creating objects with class instances allocated on heap - in at least three ways: with a try..finally Free block, with a TComponent ownership model in the VCL, or by using an interface (which creates an hidden try..finally Free block);
Creating reference-counted variables, i.e. string, array of, interface or variant kind of variables.

It is a bit complex, but it is also deadly powerful. You have several memory allocation models at hand, which can be very handy if you want to tune your performance and let program scale. Just like manual recycling at home will save the planet. Some programmers will tell you that it's a waste of cell brain, typing and time. Linux kernel gurus would not say so, I'm afraid.

Then came the big Apple company, which presented its new ARC model (introduced in Mac OS X 10.7 Lion) as a huge benefit for Objective-C in comparison with the Garbage Collection model. And let's face it: this ARC just sounds like the Delphi memory model.

What is this ARC model?

No, ARC is not only an old packer format, nor the company responsible of solid waste removal and recycling services for Chicago and suburbs - this acronym stands for Automatic Reference Counting.

Formerly, class instances used to be manually handled in Objective C code. You had to call explicitly the Release method to decrement its reference count. When an object's reference count is zero, it is deallocated.

This sounds just like our well known reference-counting mechanism included in Delphi, for string, array of, interface or variant. What is new with ARC is that, as with these Delphi types, the compiler will generate the release code. And all those kind of variable instanced are filled with zero (this is not the same as the famous Zeroing Weak pointers feature) - which is implemented by _InitializeRecord() and _FinalizeRecord() in low-level System.pas unit. Of course, the ARC implementation is certainly more sophisticated than the basic implementation in the Delphi compiler: it is told (at least from the marketing paper point of view) to use some deep knowledge of the software architecture to provide an accurate access to all instances. Whereas the Delphi compiler just relies on a out-of-scope pattern.

There is a very interesting analysis of the ARC design available on Ars Technica, which is worth reading. Especially on modest hardware (I do not know if it is honest when this concept is applied to modern smartphones, which are much more powerful than the PCs on which the first Delphi version was running on), Apple's experts say it does make a difference, in term of memory use and program latency - the UI won't glitch during background garbage recycling.

So what about the issues?

This just sounds too much perfect. And there is no perfection on earth, nor in computing.

One common problem with reference counting is the potential circular reference issue.

This occurs when one object has a strong pointer to another, but the target object has a strong pointer back to the original. Even when all other references to these objects are removed, they still will hold on to one another and will not be released. This can also happen indirectly, by a chain of objects that might have the last one in the chain referring back to an earlier object.

This is where the ARC's Zeroing Weak pointers comes to mind. When you read the official reference material at llvm.org, there are also some other potential issues, but the __weak ownership qualifier is the main point of this document.

The implementation pattern is a bit more complex that the one used for Delphi's interface (taken from Ars Technica):

The "zeroing" part means that weak references will be set to nil when the object they reference is deallocated. (Under ARC, all object pointers are initially set to zero.) Under normal circumstances, an object shouldn't be deallocated if there are still outstanding references to it. But since weak references don't contribute to an object's reference count, an object can be deallocated when there are outstanding weak references to it. When this happens, the automatic zeroing of the outstanding weak references prevents them from becoming dangling pointers. (In Objective-C, sending a message to nil is a no-op.)

As far as I understood the problem, simple types like our reference-counted Delphi variables solve the circular reference issue by providing a copy-on-write behavior. But the interface kind of variable, in its current state, may suffer from this issue.

Since I want to use interface everywhere in our next version of mORMot (to implement a true Domain-Driven architecture - including SOLID principles), some strict rules may be defined in its usage, not to suffer from similar issues. To my understanding, as soon as you use interfaces in the internal scope of methods (i.e. stack-allocated), you may never suffer for circular reference.

Additional investigation is certainly needed.
Your feedback is warmly welcome on our forum!

Automatic JOIN query

A.Bouchez — Tue, 06 Dec 2011 22:08:00 +0100

In mORMot, all the methods available to handle many-to-many relationship (ManySelect, DestGetJoined...) are used to retrieve the relations between tables from the pivot table point of view. This saves bandwidth, and can be used in most simple cases, but it is not the only way to perform requests on many-to-many relationships. And you may have several TSQLRecordMany instances in the same main record - in this case, those methods won't help you.

It is very common, in the SQL world, to create a JOINed request at the main "Source" table level, and combine records from two or more tables in a database. It creates a set that can be saved as a table or used as is. A JOIN is a means for combining fields from two or more tables by using values common to each. Writing such JOINed statements is not so easy by hand, especially because you'll have to work with several tables, and have to specify the exact fields to be retrieved; if you have several pivot tables, it may start to be a nightmare.

Let's see how our ORM will handle it.

A dedicated FillPrepareMany method has been added to the TSQLRecord class, in conjunction with a new constructor named CreateAndFillPrepareMany. This particular method will:
- Instantiate all Dest properties of each TSQLRecordMany instances - so that the JOINed request will be able to populate directly those values;
- Create the appropriate SELECT statement, with an optional WHERE clause.

Here is the corresponding test included in our regression suite:

Check(MS.FillPrepareMany(aClient,
  'DestList.Dest.SignatureTime<>% and id>=? and DestList.AssociationTime<>0 '+
  'and SignatureTime=DestList.Dest.SignatureTime '+
  'and DestList.Dest.Signature<>"DestList.AssociationTime"',[0],[sID[1]]));

Of course, the only useful parameter here is id>=? which is used to retrieve the just added relationships in the pivot table. All other conditions will always be true, but it will help testing the generated SQL.

Our mORMot will generate the following SQL statement:

select A.ID AID,A.SignatureTime A00,A.Signature A01,
  B.ID BID,B.AssociationTime B02,
  C.ID CID,C.SignatureTime C00,C.Signature C01
from ASource A,ADests B,ADest C
where B.Source=A.ID and B.Dest=C.ID
  and (C.SignatureTime<>0 and A.id>=:(1): and B.AssociationTime<>0
  and A.SignatureTime=C.SignatureTime and C.Signature<>"DestList.AssociationTime")

You can notice the following:
- All declared TSQLRecordMany instances (only one, renamed B in our case) are included in the statement, with all corresponding Dest instances (renamed as C);
- Fields are aliased with short unique identifiers (AID, A01, BID, B02...), for all simple properties of every classes;
- The JOIN clause is created (B.Source=A.ID and B.Dest=C.ID);
- Our manual WHERE clause has been translated into proper SQL, including the table internal aliases (A,B,C) - in fact, DestList.Dest has been replaced by C, the main ID property has been declared properly as A.ID, and the "DestList.AssociationTime" text remained untouched, because it was bounded with quotes.

That is, our ORM did make all the dirty work for you! You can use Delphi-level conditions in your query, and the engine will transparently convert them into a valid SQL statement. Benefit of this will become clear in case of multiple pivot tables, which are likely to occur in real-world applications.

After the statement has been prepared, you can use the standard FillOne method to loop through all returned rows of data, and access to the JOINed columns within the Delphi objects instances:

  Check(MS.FillTable.RowCount=length(sID));
  for i := 1 to high(sID) do begin
   MS.FillOne;
    Check(MS.fID=sID[i]);
    Check(MS.SignatureTime=MD.fSignatureTime);
    Check(MS.DestList.AssociationTime=i);
    Check(MS.DestList.Dest.fID=dID[i]);
    Check(MS.DestList.Dest.SignatureTime=MD.fSignatureTime);
    Check(MS.DestList.Dest.Signature=FormatUTF8('% %',[aClient.ClassName,i]));
  end;
  MS.FillClose;

Note that in our case, an explicit call to FillClose has been added in order to release all Dest instances created in FillPrepareMany. This call is not mandatory if you call MS.Free directly, but it is required if the same MS instance is about to use some regular many-to-many methods, like MS.DestList.ManySelect() - it will prevent any GPF exception to occur with code expecting the Dest property not to be an instance, but a pointer(DestID) value.

Feedback and comments are welcome in our forum - this thread also contains the customer request for this nice feature.

Total Commander 64 bit is using... Lazarus and FPC

A.Bouchez — Sun, 04 Dec 2011 09:46:00 +0100

I'm a long-time registered user of Total Commander.

This tool is my daily file manager. I never use Windows Explorer, since Total Commander is just faster, more easy to use (especially with the keyboard), has a lot of plug-ins. I even created my own plug-ins to access some custom file formats, and navigate into them just like with any folder. And it includes a lot of well written commands for FTP access or file comparison, which made other tools (like WinMerge) unnecessary.

There is a new beta version of Total Commander available, which targets Windows 64 bit. I just thought: 'Whoo, this is a real-world Delphi XE2 64 application'. I downloaded and tried it. Worked as expected, and integrates seamlessly with Windows Seven (for the shell extensions). Then I took a look at the executable... and discovered it was not compiled with Delphi XE2... but with FPC !

The 32 bit version appears to be compiled with Delphi 2 - yes, this is an OLD version (the first 32 bit version) - but this is what the TC author himself states.

As a result, the 32 bit version executable is small and efficient. And, honestly, you can write a nice Unicode software without using generics and all the new Delphi features...

When looking at the 64 bit supplied version, it was obvious to me that even if the tool was behaving the same as the 32 bit version (fast and native user interface), the gears were not the same. It seems to have been compiled with the great Open Source Free Pascal Compiler - using not the VCL, but the cross-platform Lazarus project. I do not know if the LCL components where used, or some Total Commander-custom components (which I think is more than probable, knowing Ghisler's skill and hand-tuning abilities).

It is even clear if you look at the application history:

14.07.10 Added: Start work on conversion to Lazarus/Free Pascal in preparation for 64-bit version

As I wrote some time ago in this blog, FPC team published a 64 bit compiler using the Delphi/object pascal language in 2006. Whereas the "official" Win64 support just appeared this year, with Delphi XE2.

Christian Ghisler is working with FPC for a lot of time for the PocketPC (... and now Android!) freeware versions of his tool.

I'm very happy to see such a world-wide spread application using FPC and Lazarus tandem. There are open source alternatives to FireMonkey and Delphi XE2. Not a feature-like, but with another architecture, another rendering, another coding spirit, another cross-platform abilities, another mind openness... Even Delphi is using FPC as compiler when targeting iOS.

Feedback and comments are welcome on our forum!

AJAX authentication

A.Bouchez — Wed, 30 Nov 2011 06:36:00 +0100

A nice framework user, named esmondb, did write and publish some JavaScript code to handle our RESTful authentication mechanism.

It seems to work well, and implements all secure hashing and challenging.
Our authentication mechanism is much more advanced than the one used by DataSnap - which is a basic HTTP authentication with the password transmitted in clear (this is the reason why it shall better be used over HTTPS, whereas mORMot can be used over plain HTTP).
Resulting JavaScript code seems not difficult to follow, even for a no JS expert like me.

The full JavaScript code is available in this forum post.

I've modified the Delphi framework code (in the current trunk version) to match the authentication JavaScript code.
In fact, the server response is now a true JSON result object. I've modified the TSQLRestURI.SetUser code to let pure Delphi client code work as expected.

In the same discussion, another user involved in developing AJAX applications with mORMot, named RangerX, made a nice proposal: add a generic JSON error message mechanisms in the framework.
I've added this to the project to-do list.

SOLID design principles

A.Bouchez — Sun, 27 Nov 2011 23:39:00 +0100

Delphi is sometimes assimilated to a RAD product - and this is a marketing label - but IMHO Delphi is much more than RAD.
With Delphi, you can make very serious and clean programming.

Including SOLID style of coding.

The acronym SOLID is derived from the following OOP principles (quoted from the corresponding Wikipedia article):

Single responsibility principle: the notion that an object should have only a single responsibility;
Open/closed principle: the notion that “software entities ... should be open for extension, but closed for modification”;
Liskov substitution principle: the notion that “objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program” - also named as "design by contract";
Interface segregation principle: the notion that “many client specific interfaces are better than one general purpose interface.”;
Dependency inversion principle: the notion that one should “Depend upon Abstractions. Do not depend upon concretions.”. Dependency injection is one method of following this principle.

If you have some programming skills, those principles are general statements you may already found out by yourself. If you start doing serious object-oriented coding, those principles are best-practice guidelines you would gain following.

They certainly help to fight the three main code weaknesses:

Rigidity – Hard to change something because every change affects too many other parts of the system;
Fragility – When you make a change, unexpected parts of the system break;
Immobility – Hard to reuse in another application because it cannot be disentangled from the current application.

Single responsibility principle

When you define a class, it shall be designed to implement only one feature. The so-called feature can be seen as an "axis of change" or a "a reason for change".

Therefore:

One class shall have only one reason that justifies changing its implementation;
Classes shall have few dependencies on other classes;
Classes shall be abstract from the particular layer they are running.

For instance, a TRectangle object should not have both ComputeArea and Draw methods defined at once - they would define two responsibilities or axis of change: the first responsibility is to provide a mathematical model of a rectangle, and the second is to render it on GUI.

When you define an ORM object, do not put GUI methods within. In fact, the fact that our TSQLRecord class definitions are common to both Client and Server sides makes this principle mandatory. You won't have any GUI related method on the Server side, and the Client side could use the objects instances with several GUI implementations (Delphi Client, AJAX Client...).

Therefore, if you want to change the GUI, you won't have to recompile the TSQLRecord class and the associated database model.

Another example is how our database classes are defined in SynDB.pas:

The connection properties feature is handled by TSQLDBConnectionProperties classes;
The actual living connection feature is handled by TSQLDBConnection classes;
And database requests feature is handled by TSQLDBStatement instances using dedicated NewConnection / ThreadSafeConnection / NewStatement methods.

Therefore, you may change how a database connection is defined (e.g. add a property to a TSQLDBConnectionProperties child), and you won't have to change the statement implementation itself.

Following this Single responsibility principle may sound simple and easy, but in fact, it is one of the hardest principles to get right. Naturally, we tend to join responsibilities in our class definitions. Our ORM architecture will enforce you, by its Client-Server nature, to follow this principle, but it is always up to the end coder to design properly his/her interfaces.

Open/closed principle

When you define a class or a unit, at the same time:

They shall be open for extension;
But closed for modification.

When designing our ORM, we tried to follow this principle. In fact, you should not have to modify its implementation. You should define your own units and classes, without the need to hack the framework source code.

Even if Open Source paradigm allows you to modify the supplied code, this shall not be done unless you are either fixing a bug or adding a new common feature. This is in fact the purpose of our http://synopse.info web site, and most of the framework enhancements have come from user requests.

The framework Open Source license may encourage user contributions in order to fulfill the Open/closed design principle:

Your application code extends the Synopse SQLite3/mORMot Framework by defining your own classes or event handlers - this is how it is open for extension;
The main framework units shall remain inviolate, and common to all users - this illustrates the closed for modification design.

Furthermore, this principle will ensure your code to be ready to follow the main framework updates (which are quite regular). When a new version is available, you would be able to retrieve it for free from our web site, replace your files locally, then build a new enhanced version of your application. Even the source code repository is available - at http://synopse.info/fossil - and allows you to follow the current step of evolvment of the framework.

In short, abstraction is the key. All your code shall not depend on a particular implementation.

In order to implement this principle, several conventions could be envisaged:

You shall better define some abstract classes, then use specific overridden classes for each and every implementation: this is for instance how Client-Server classes were implemented;
All object members shall be declared private or protected - this is a good idea to use an Service-Oriented-Architecture for defining server-side process, and/or make the TSQLRecord published properties read-only and using some client-side constructor with parameters;
No singleton nor global variable - ever;
RTTI is dangerous - that is, let our framework use RTTI functions for its own cooking, but do not use it in your code.

Some other guidelines may be added, but you got the main idea. Conformance to this open/closed principle is what yields the greatest benefit of OOP, i.e.:

Code re-usability;
Code maintainability;
Code extendibility.

Following this principle will make your code far away from a regular RAD style. But benefits will be huge.

Liskov substitution principle

Even if her name is barely unmemorable, Barbara Liskov is a great computer scientist, we should better learn from.

Her "substitution principle" states that, if TChild is a subtype of TParent, then objects of type TParent may be replaced with objects of type TChild (i.e., objects of type TChild may be substitutes for objects of type TParent) without altering any of the desirable properties of that program (correctness, task performed, etc.).

For our framework, it would signify that TSQLRestServer or TSQLRestClient instances can be substituted to a TSQLRest object. Most ORM methods expect a TSQLRest parameter to be supplied.

Your code shall refer to abstractions, not to implementations. By using only methods and properties available at classes parent level, your code won't need to change because of a specific implementation.

The main advantages of this coding pattern are the following:

Thanks to this principle, you will be for instance able to stub or mock an interface or a class - this principle is therefore mandatory for implementing unitary testing to your project;
Furthermore, testing would be available not only at isolation level (testing each child class), but also at abstracted level, i.e. from the client point of view - you can have implementation which behave correctly when tested individually, but which failed when tested at higher level if the Liskov principle was broken;
If this principle is violated, the open/close principle will be - the parent class would need to be modified whenever a new derivative of the base class is defined;
Code re-usability is enhanced by method re-usability: a method defined at a parent level does not require to be implemented for each child.

Some patterns which shall not appear in your code:

Statements like if aObject is TAClass then begin .... end else if aObject is TAnotherClass then ... in a parent method;
Use an enumerated item and a case ... of or nested if ... then to change a method behavior (this will also probably break the single responsibility principle: each enumeration shall be defined as a class);
Define a method which will stay abstract for some children;
Need to explicitly add all child classes units to the parent class unit uses clause.

In order to fulfill this principle, you should:

Use the "behavior" design pattern, when defining your objects hierarchy - for instance, if a square may be a rectangle, a TSquare object is definitively not a TRectangle object, since the behavior of a TSquare object is not consistent with the behavior of a TRectangle object (square width always equals its height, whereas it is not the case for most rectangles);
Write your tests using abstract local variables (and this will allow test code reuse for all children classes);
Follow the concept of Design by Contract, i.e. the Meyer's rule defined as "when redefining a routine [in a derivative], you may only replace its precondition by a weaker one, and its postcondition by a stronger one" - use of preconditions and postconditions also enforce testing model;
Separate your classes hierarchy: typically, you may consider using separated object types for implementing persistence and object creation (this is the common separation between Factory and Repository).

The SOA and ORM concepts as used by our framework are compatible with the Liskov substitution principle. Future versions shall enforce this, by providing some direct Design by Contract methods (involving a more wide usage of interfaces).

Interface segregation principle

This principle states that once an interface has become too 'fat' it shall be split into smaller and more specific interfaces so that any clients of the interface will only know about the methods that pertain to them. In a nutshell, no client should be forced to depend on methods it does not use.

As a result, it will help a system stay decoupled and thus easier to re-factor, change, and redeploy.

In its current state, our framework does not make direct use of interfaces. But its Client-Server SOA architecture helps decoupling all services to individual small methods. And its stateless design will also reduce the use of 'fat' session-related processes.

Dependency Inversion Principle

Another form of decoupling is to invert the dependency between high and low level of a software design:

High-level modules should not depend on low-level modules. Both should depend on abstractions;
Abstractions should not depend upon details. Details should depend upon abstractions.

In conventional application architecture, lower-level components are designed to be consumed by higher-level components which enable increasingly complex systems to be built. This design limits the reuse opportunities of the higher-level components, and certainly breaks the Liskov's substitution principle.

The goal of the dependency inversion principle is to decouple high-level components from low-level components such that reuse with different low-level component implementations becomes possible. A simple implementation pattern could be to use only interfaces owned by, and existing only with the high-level component package.

In other languages (like Java or .Net), various patterns such as Plug-in, Service Locator, or Dependency Injection are then employed to facilitate the run-time provisioning of the chosen low-level component implementation to the high-level component.

Our Client-Server architecture facilitated this decoupling pattern, even if no interface is used by default in the framework yet.
But such a WCF-like or SOAP/Datasnap interface-based service definition is on the top of my to-do list.

Comments and feedback are welcome on our forum.

Modification of TSQLRestServerCallBack method prototype

A.Bouchez — Sun, 27 Nov 2011 07:21:00 +0100

In order to implement some RESTful Services, a callback has to be defined on the server side.

The prototype of these method has been modified, to supply an additional aSession: cardinal parameter: this is a CODE BREAK change and you shall refresh ALL your server-side code to match the new signature.

This new aSession parameter will identify the authentication session of the remote client, or 1 (if authentication mode is not set), or 0 (if the session not started yet).
Service implementation code may then use the new SessionGetUser() protected method to retrieve the session details, e.g. the user logon name and display name, or the associated BLOB data.

This is a code break of existing implementation, but it will provide mandatory additional information for a true Client-Server design.
Next step is to implement a true Interface-based service feature, similar to WCF or SOAP/DataSnap (in my to-do list).

See this link for the corresponding commit.

For instance, here is the difference applied on the corresponding sample folder:

TServiceServer = class(TSQLRestServer) published - function Sum(aRecord: TSQLRecord; aParameters: PUTF8Char; - const aSentData: RawUTF8; var aResp, aHead: RawUTF8): Integer; + function Sum(aSession: Cardinal; aRecord: TSQLRecord; aParameters: PUTF8Char; + const aSentData: RawUTF8; out aResp, aHead: RawUTF8): Integer; end;

Note that the callback prototype changed again, for simplicity and to avoid most further change.

Does speed matters?

A.Bouchez — Wed, 23 Nov 2011 08:37:00 +0100

Luigi Sandon wrote on Embarcadero's forum:

And then you ask yourself: "why use a native compiler if its code may be even slower than jitted one?". Hope the new developers will also develop better and faster code - and not viceversa.

Embarcadero is just following the Wirth's law slower than others:

"Software is getting slower more rapidly than hardware becomes faster"

Speed is only a matter of compiler for mathematical computing intensive tasks.
Most of the time, in real apps (like business apps), the main speed issue is more the framework size (and the number of dll invoked), memory consumption, and general design (e.g. how caching and SQL are written).

Delphi, Java or .Net can do slow apps.
Delphi, Java or .Net can do fast apps.

You can do small and fast stand-alone apps with Delphi, running from Windows 2000 to Windows 8.
It is not possible with Java nor .Net.

This is the main difference IMHO with native code and JIT - about memory use, ease of distribution and no need of an external runtime framework.

For instance, most of our tools (like SynProject documentation or SynDBExplorer tool) are stand-alone, but powerful.

SynDBExplorer is able to connect to databases with no installation: the same tool written in .Net would make a smaller executable in size, but will be slower (even unworkable with huge amount of data - millions of rows), need additional requirements for database access (e.g. native Oracle), and will consume much more memory.

Of course, there are plenty of great libraries in .Net, a lot of books to be read... Nice features I miss with Delphi.

Currency is your friend

A.Bouchez — Tue, 08 Nov 2011 21:51:00 +0100

The currency type is the standard Delphi type to be used when storing and handling monetary values. It will avoid any rounding problems, with 4 decimals precision. It is able to safely store numbers in the range -922337203685477.5808 .. 922337203685477.5807. Should be enough for your pocket change.

As stated by the official Delphi documentation:

Currency is a fixed-point data type that minimizes rounding errors in monetary calculations. On the Win32 platform, it is stored as a scaled 64-bit integer with the four least significant digits implicitly representing decimal places. When mixed with other real types in assignments and expressions, Currency values are automatically divided or multiplied by 10000.

In fact, this type matches the corresponding OLE and .Net implementation of currency, and the one used by most database providers (when it comes to money, a dedicated type is worth the cost in a "rich man's world"). It is still implemented the same in the Win64 platform (since XE 2). The Int64 binary representation of the currency type (i.e. value*10000 as accessible via PInt64(aCurrencyValue)^) is a safe and fast implementation pattern.

In our framework, we tried to avoid any unnecessary conversion to float values when dealing with currency values. Some dedicated functions have been implemented for fast and secure access to currency published properties via RTTI, especially when converting values to or from JSON text. Using the Int64 binary representation can be not only faster, but also safer: you will avoid any rounding problem which may be introduced by the conversion to a float type. Rounding issues are a nightmare to track - it sounds safe to have a framework handling natively a currency type from the ground up.

Faster and safer way of comparing two currency values is certainly to map the variables to their internal Int64 binary representation, as such:

function CompCurrency(var A,B: currency): Int64;
var A64: Int64 absolute A;
    B64: Int64 absolute B;
begin
  result := A64-B64;
end;

This will avoid any rounding error during comparison (working with *10000 integer values), and will be faster than the default implementation, which uses the FPU (or SSE2 under x64 architecture) instructions.

In SynCommons.pas, there are some functions using the Int64 binary representation (accessible either as PInt64(aCurrencyVar)^ or the absolute syntax).
They will by-pass the FPU/SSE2 use, and are therefore both fast and safe at the same time:

- function Curr64ToString(Value: Int64): string;
- function StrToCurr64(P: PUTF8Char): Int64;
- function Curr64ToStr(Value: Int64): RawUTF8;
- function Curr64ToPChar(Value: Int64; Dest: PUTF8Char): PtrInt;
- function StrCurr64(P: PAnsiChar; const Value: Int64): PAnsiChar;

Using those functions can be much faster for textual conversion than using the standard FloatToText() implementation. They are validated with provided regression tests.

Of course, in normal code, it is certainly not worth using the Int64 binary representation of currency, but rely on the default compiler/RTL implementation. In all cases, having optimized functions was a need for both speed and accuracy of our ORM data processing, and also for external database handling.

Comments and feedback are welcome on our forum.

Yes we can... fight bugs

A.Bouchez — Thu, 27 Oct 2011 05:17:00 +0200

From a StackOverflow question about a freezing Delphi application, I posted some experiment-based debugging tricks.

May help any developer in his/her fight against random bugs...

Some (general-purpose) suggestions may be the following:

If your application is not multi-threaded, do the process in background threads, then leave the main thread ready to process GDI messages;
If your application is multi-threaded, take care that all VCL accesses from background threads are made via a Synchronize call;
If your application is multi-threaded or use timers, take care that no method is re-entrant (in some circonstances, you may come into a race condition;
Hunt any memory leak;
Use a detailed logging of the program execution, logging all exceptions risen, to guess the context of the program hang (it may be used on the customer side also to hunt race conditions);
Download the great free tool named ProcessExplorer (now hosted by Microsoft), and check out the state of your frozen program: you will see detailed information about threads, CPU use, memory, network, libraries, handles - this is a must have for any serious debugging - track especially the GDI handles leaks (number of those should remain stable);
If you did not check it already, take a look at the global Windows system event log: there may be some information here;
Perhaps a third party component or library is responsible of the process hang: try to isolate the part of your code which may be responsible of this hang.

I've Delphi application running for months without any problem. Issue is definitively in application code, not in the Delphi architecture (its RTL and VCL are very stable).

Feedback and comments are welcome on our forum!

Synopse SQLite3/mORMot framework 1.15

A.Bouchez — Sun, 25 Sep 2011 21:46:00 +0200

Our Client-Server ORM framework is now available in revision 1.15.

This is a major upgrade of the framework:

It is now called mORMot - so please update your T-Shirts or coffee cups
It is able to use any Database engine back-end - in fact, it is SQLite3 powered, not SQLite3 limited;
In particular, direct OleDB and native Oracle have been implemented;
It makes use of the genuine SQlite3 Virtual Table mechanism everywhere to allow mixed access to any database engine;
New TModTime / TCreateTime kind of fields;
Enhanced stability, speed and multi-thread implementation;
Methods and functions have been enhanced, according to user feedback (thanks you all for your interest and forum posts!);
Extended documentation (more than 700 pdf pages), with new diagrams and a lot of new content;
New associated tools, like LogViewer or SynDBExplorer;
The SQLite3 core can now be used without our ORM - it has been updated to the latest 3.7.8 version;
Open Source (under GPL/LGPL/MPL license), running from Delphi 6 up to XE2.

As you can see, if we step into the implementation details, there is a huge list of enhancements and new features.

Let's face each unit modification list:

SynCommons

unit now tested with Delphi XE2 (32 Bit)
TSynLog now writes the elapsed time (in us) for Enter/Leave events, and will flush the log content to disk on any exception (for safety)
new sllTrace and sllWarning levels for TSynLog class
new TSynLog.DefaultExtension property (set to '.log' by default)
new TSynLogFile.LogProc[] property for customer-side method profiling, with LogProcSort method available for sorting the resulting array, and LogProcMerged property to merge the location name timing
new TSynMapFile.FindLocation method for high-level .map symbol access
TSynMapFile now handles huge .map file (bigger default in-memory buffer)
fix potential GPF issue in code using ConvertHexToBin[]
new TSynLog.EventCount method
new TMemoryMapText.LineContains method for fast case-insensitive search
TSynTests now writes the elapsed time in each test in the final report
faster late binding process for our variants custom types (i.e. TSynTableVariantType and TSQLDBRowVariantType): you can call SynRegisterCustomVariantType() function to register any other custom variant type, and enhance GetProperty/SetProperty process speed
includes our optimized RecordCopy procedure in replacement to the slower default System.@CopyRecord internal RTL function
our optimized Move() and FillChar() will replac the default System RTL function, for Delphi versions prior to 2007 (which didn't contain those)
new AnsiCharToUTF8(), StringToWinAnsi(), WideStringToWinAnsi(), WideStringToUTF8(), CSVOfValue(), IdemPCharArray(), FindUnicode(), UpperCaseUnicode(), LowerCaseUnicode() and Split() functions
faster GetInt64() function
Iso8601ToSecondsPUTF8Char() now returns 0 in case of unexpected format
fixed issue in StrCurr64() low-level conversion routine
fixed issue in Utf8DecodeToRawUnicodeUI() function
new TSynTableFieldProperties.OrderedIndexRefresh method, to allow access on OrderedIndex[] even if the index needs to be refreshed
new TDynArrayHashed.AddAndMakeUniqueName() method and Hash[] property
new TRawByteStringStream class (a TStream using a RawByteString as internal storage), especially useful since Delphi 2009
new TSynNameValue object, to efficiently handle Name/Value RawUTF8 pairs
TTextWriter.CreateOwnedStream now create an internal TRawByteStringStream instance for faster process and direct retrieval in the Text method
JSONEncode*() global functions will use an internal TRawByteStringStream instead of a supplied TMemoryStream
new FormatUTF8() overloaded function, handling both '%' and '?' parameters (inserting '?' as inlined :(...): parameters, with proper string quote) - with associated regression tests

SQLite3Commons

unit now tested with Delphi XE2 (32 Bit)
new sftModTime / TModTime published field type in TSQLRecord, which will be set to the current server time stamp before update/adding
new sftCreateTime / TCreateTime published field type in TSQLRecord, which will be set to the current server time stamp at record creation
new TSQLRest.ServerTimeStamp property, which will return the current server time as TTimeLog/Int64 value (will use the new /TimeStamp RESTful service to retrieve the exact server time)
TSQLRestServerStaticInMemory uses a per-Table Critical Section to have its EngineList, EngineRetrieve, EngineAdd, EngineUpdate, EngineDelete, EngineRetrieveBlob, EngineUpdateBlob methods begin thread-safe
enhanced TSQLRestServer.URI thread-safety (e.g. Sessions access)
TSQLTable.InitFieldTypes will now also use column type retrieved during JSON parsing
new TSQLTable.GetCSVValues method
GetJSONValues() is now using an internal TRawByteStringStream when the expected result is a RawUTF8 (avoid copying content twice, and is perfectly thread-safe)
the shared fTempMemoryStream is not available any more (not thread-safe)
new TSQLRest.AcquireWrite/ReleaseWrite protected methods, used by TSQLRestServer.URI to safely write to the DB (e.g. for POST/PUT/DELETE...) with TSQLRest.AcquireWriteTimeOut, both thread-safe and transaction-safe
TSQLRest.TransactionBegin / Commit / RollBack methods now expect a SessionID parameter in order to allow safe concurent access: writing to the database is queued within a single client session
CreateSQLMultiIndex and CreateSQLIndex methods now working on external DB virtual tables (using SynDB.TSQLDBConnectionProperties.SQLAddIndex)
new TSQLRecordProperties.ExternalTableName and ExternalDatabase fields used by SQLite3DB to handle external SynDB-based database access
code refactoring to make TSQLRestServerStatic more generic (for SQLite3DB)
TSQLRestServer.UpdateField now accepts to search by ID or by value (used e.g. by rewritten TSQLRestServer.AfterDeleteForceCoherency method)
introducing TSQLRecordExternal kind of record, able to use any SynDB external database engine (e.g. OleDB/MSSQL/Oracle/SQLite3)
new ExtractInlineParameters procedure to handle :(1234): SQL statements
new MakePrivateCopy property in TSQLTableJSON.Create, which will avoid creating a private copy of the JSON (used e.g. in SynDBExplorer to handle very large result sets, with half the memory)
new TSQLRecordProperties.SQLUpdateSet, SQLInsertSet and AppendFieldName properties/method (used for external DB handling)
new TSQLRecord.Create, TSQLRecord.FillPrepare, TSQLRecord.CreateAndFillPrepare, TSQLRest.OneFieldValue, TSQLRest.MultiFieldValues, TSQLRestClient.EngineExecuteFmt and TSQLRestClient.ListFmt overloaded methods, accepting both '%' and '?' characters in the SQL WHERE format text, inlining '?' parameters with :(...): and auto-quoting strings
new UnicodeComparison parameter in TSQLTable.SearchValue to handle property non WinAnsi (code page 1252) characters
fixed issue in TPropInfo.GetBinary method with dynamic arrays (used e.g. by TSQLRestServerStaticInMemory.SaveToBinary)
fixed issue with TAuthSession.IDCardinal=0 after 76 connections
fixed issue in SetInt64Prop() with a setter method
fixed potential issue in TSQLTable.SearchValue in case of invalid Client supplied parameter (now checks TSQLRest class type)

SQLite3

updated SQLite3 engine to version 3.7.8
unit now tested with Delphi XE2 (32 Bit)
transactions now following a safe concurent access (both thread-safe and client/connection-safe) - but authentication should be enabled
the SQLite3 wrapper is now located in a separate SynSQLite3 unit: this will allow to use it as a separate database engine, e.g. using SynDB classes without the overhead/features of our mORMot framework
statement cache is now shared with SynDBSQLite3, via the new TSQLStatementCached object as defined in SynSQLite3.pas
now TSQLRestServerDB will unregister any TSQLVirtualTableModuleServerDB to avoid random GPF in TSQLVirtualTable.Destroy
TSQLRestClientDB and TSQLRestServerDB constructors now accept an optional Password parameter, associated to the supplied file name, in order to use database encryption

SynSQLite3

first public release, corresponding to mORMot Framework 1.15
new unit extracting the SQLite3 wrapper from the previous SQLite3 unit: this unit can therefore be used with our SynDB classes (via SynDBSQLite3), without SQLite3Commons overhead (and features)
added TSQLRequest.BindNull method and associated sqlite3_bind_null function
fixed issue with TSQLDataBase with UseCache=false
new TSQLStatementCached object, for caching of prepared SQLite3 statements
TSQLDatabase constructors now accepts an optional Password parameter, associated to the supplied file name, in order to use database encryption (not finished nor fully tested feature)

SynPDF

unit now tested with Delphi XE2 (32 Bit)

SynGdiPlus

unit now tested with Delphi XE2 (32 Bit)
handle TIFF saving with diverse compression methods

SynBigTable

unit now tested with Delphi XE2 (32 Bit)

SynZip

unit now tested with Delphi XE2 (32 Bit)

SynCrypto

unit now tested with Delphi XE2 (32 Bit)

SynCrtSock

unit now tested with Delphi XE2 (32 Bit)
fixed issue in HTTP_RESPONSE.SetHeaders

SQLite3DB

first public release, corresponding to mORMot Framework 1.15
new unit handling Virtual tables for DB direct access classes for the mORMot framework

SynDB

SynDB unit extracted from previous SynOleDB.pas
TQueryValue.As* methods now handle NULL column as 0 or ''
added new TSQLDBRowVariantType custom variant type, allowing late binding access to row columns (not for Delphi 5) - see RowData method
fixed transaction handling in a safe abstract manner
TSQLDBStatement class now expects a prepared statement behavior, therefore TSQLDBStatementPrepared class has been merged into its parent class, and inherited classes have been renamed TSQLDBStatementWithParams[AndColumns]
new TSQLDBStatement.FetchAllAsJSON method for JSON retrieval as RawUTF8
exposed FetchAllAsJSON method for ISQLDBRows interface
made the code compatible with Delphi 5
new TSQLDBConnectionProperties.SQLIso8601ToDate virtual method
code refactoring for better metadata (database and table schemas) handling, including GetTableNames, GetFields, GetFieldDefinitions and GetForeignKey methods - will work with OleDB metadata and direct Oracle sys.all_* tables
new TSQLDBConnectionProperties.SQLCreate/SQLAddColumn/SQLAddIndex virtual methods (SQLCreate and SQLAddColumn will use the new protected SQLFieldCreate virtual method to retrieve the SQL field definition from protected fSQLCreateField[Max] properties) - as a result, SQL statement generation as requested for mORMot is now much more generic than previously
new overloaded TSQLDBStatement.Execute() method, able to mix % and ? parameters in the SQL statement
new TSQLDBStatement.BindNull() method
new TSQLDBConnectionProperties.NewThreadSafeStatementPrepared and TSQLDBConnection.NewStatementPrepared methods, able to be overriden to implement a SQL statement caching (used e.g. for SynDBSQLite3)
new TSQLDBConnection.ServerTimeStamp property, which will return the external database Server current date and time as TTimeLog/Int64 value (current implementation handle Oracle, MSSQL and MySQL database engines - with SQLite3, this will be the local PC time, just as for other DB engines)
new overloaded TSQLDBStatement.Bind() method, which can bind an array of const (i.e. an open list of Delphi arguments) to a statement
new overloaded TSQLDBStatement.Bind() and ColumnToVarData() methods, able to bind or retrieve values from a TVarData/TVarDataDynArray (used e.g. for direct access to/from SQLite3 virtual table in the SQLite3DB unit)
new ColumnTimeStamp method for TSQLDBStatement/ISQLDBRows, returning a TTimeLog/Int64 value for a date/time column

SynOleDB

SynDB unit extracted from previous SynOleDB.pas
several fixes and speed enhancements
made the code compatible with Delphi 5
TOleDBStatement class now follows the prepared statement pattern introduced with its parent TSQLDBStatement
now able to retrieve table names and column information from OleDB metadata including GetTableNames, GetFields, GetFieldDefinitions and GetForeignKey methods - able to use faster direct SQL retrieval (e.g. for Oracle / MS SQL)

SynDBOracle

first public release, corresponding to mORMot Framework 1.15
new unit implementing fast Oracle DB direct access classes (via OCI)

SynDBSQLite3

first public release, corresponding to mORMot Framework 1.15
new unit implementing SQLite3 direct access classes to be used with our SynDB architecture
use the SQLite3 engine, as wrapped via the new separated SynSQLite3 unit

To get it, go to this download page, or use the source...

Feedback and questions are welcome in our forum, just as usual.

Some thoughts about OSX integration in XE2

A.Bouchez — Sun, 25 Sep 2011 19:54:00 +0200

You know all that one of the most exciting features of Delphi XE2 is the MaxOSX Cross-Platform feature.
You've got the UI part, that is FireMonkey, but underneath, you did have some RTL modifications in order to let our Windows-centric solutions be OSX ready.

The first main step was to make our code speak with the "Objective-C" way of coding.

Objective-C is the primary language used for Apple's Cocoa API, and it was originally the main language on NeXT's NeXTSTEP OS. It's some object-oriented C variant, but something other than C++ or Java. In fact, Objective-C sounds more like a SmallTalk variance of C than another C++/Java/C# flavor. For instance, the Objective-C model of object-oriented programming is based on message passing to object instances: this is just another way of doing it. It has some advantages, and disadvantages (I don't want to troll here) - but it is definitively nice. And the memory model is just something else, more close to our reference-counting way (as in Delphi interface implementation) than a garbage collector.

About Objective-C, it sounds like if Embarcadero did make its own Delphi-based wrapper around the OOP layout of this technology. In short, messages were encapsulated inside methods, and exposed as interfaces. Embarcadero RTL team used the new RTTI and generics language features to expose the Objective-C API into standard Delphi interfaces.

FPC did extend the compiler so that Objective-C style of coding appeared in the code: this is the so-called "Objective Pascal" mode, which handle both its message and memory model as native features, at the compiler level.

I don't have any preference. Both have pros and cons. The FPC solution sounds lighter and automated (the OSX APIs are created using an automated process). The EMB solution sounds more "pascalish", since the language was not modified, but some wrappers were designed over the OSX APIs.

What is nice in both case is that we are not stick into the flat Carbon API, which is 32 bit only. Calling directly the Cocoa APIs will release the full power of Mac OSX, and allow to write 64 bit applications (already for FPC, in the future for Delphi).

The new "unit scope names" introduced in Delphi XE2 for handling several platforms (Windows/Posix or VCL/FireMonkey) do make sense to me. By the way, it seems definitively better than the previous LibC huge unit, as used for Kylix. And perhaps more rigid than the FPC library.
In practice, I had to modify our framework source code for all VCL-refering unit names. But nothing difficult - just one step preparing for FireMonkey.

In all cases, this is a quite interesting step into cross-platform.

Nice talking about it on our forum, if you wish.

Synopse SQLite3 framework is now mORMot

A.Bouchez — Sun, 25 Sep 2011 13:59:00 +0200

In case you were redirected from the previous "Synopse SQLite3 framework" category link, here is the new thread to be used instead:
http://blog.synopse.info/category/Open-Source-Projects/mORMot-Framework

Since revision 1.15 of the framework, it is able to connect to any database engine (therefore is not limited to SQLite3), and is now called mORMot.

Synopse PDF Engine 1.15

A.Bouchez — Sun, 25 Sep 2011 10:49:00 +0200

For our PDF generation Open-Source library, this is a small fix update.

It can now be compiled under Delphi XE2.
But it's still working from all previous IDE versions, starting with Delphi 5, and still 100% free - released under GPL/LGPL/MPL license, choice is yours.

Source code is to be downloaded from this page.

Feedback and questions are welcome on our forum, just as usual.
See http://synopse.info/forum/viewtopic.php?id=447

L10n and i18n in our framework

A.Bouchez — Wed, 14 Sep 2011 20:31:00 +0200

In computing, internationalization and localization (also spelled internationalisation and localisation) are means of adapting computer software to different languages, regional differences and technical requirements of a target market:

Internationalization (i18n) is the process of designing a software application so that it can be adapted to various languages;
Localization (L10n) is the process of adapting internationalized software for a specific region or language by adding locale-specific components and translating text, e.g. for dates display.

Our framework handle both features, via the SQLite3i18n.pas unit. For instance, resourcestring defined in the source code are retrieved from the executable and can be translated on the fly. The unit extends this to visual forms, and even captions generated from RTTI.

In short, making your software open to any language is handled by the framework, from the bottom-up.

Application i18n and L10n

The unit expects all textual content (both resourcestring and RTTI derived captions) to be correct English text. A list of all used textual elements will be retrieved then hashed into an unique numerical value. When a specific locale is set for the application, the unit will search for a .msg text file in the executable folder matching the expected locale definition. For instance, it will search for FR.msg for translation into French.

In order to translate all the user interface, a corresponding .msg file is to be supplied in the executable folder. Neither the source code, nor the executable is to be rebuild to add a new language. And since this file is indeed a plain textual file, even a non developer (e.g. an end-user) is able to add a new language, starting from another .msg.

Creating the reference file

In order to begin a translation task, the SQlite3i18n.pas unit is able to extract all textual resource from the executable, and create a reference text file, containing all English sentences and words to be translated, associated with their numerical hash value.

It will in fact:

Extract all resourcestring text;
Extract all captions generated from RTTI (e.g. from enumerations or class properties names);
Extract all embedded dfm resources, and create per-form sections, allowing a custom translation of displayed captions or hints.

This creation step needs a compilation of the executable with the EXTRACTALLRESOURCES conditional defined, globally to the whole application (a full rebuild is necessary after having added or suppressed this conditional from the Project / Options / Folders-Conditionals IDE field).

Then the ExtractAllResources global procedure is to be called somewhere in the code.

For instance, here is how this is implemented in !TMainForm.FormShow!LibMainDemoFileMain.pas, for the framework main demo:

procedure TMainForm.FormShow(Sender: TObject);
begin
{$ifdef EXTRACTALLRESOURCES}
  ExtractAllResources(
    // first, all enumerations to be translated
    [TypeInfo(TFileEvent),TypeInfo(TFileAction),TypeInfo(TPreviewAction)],
    // then some class instances (including the TSQLModel will handle all TSQLRecord)
    [Client.Model],
    // some custom classes or captions
    [],[]);
  Close;
{$else}
  //i18nLanguageToRegistry(lngFrench);
{$endif}
  Ribbon.ToolBar.ActivePageIndex := 1;
end;

The TFileEvent and TFileAction enumerations RTTI information is supplied, together with the current TSQLModel instance. All TSQLRecord classes (and therefore properties) will be scanned, and all needed English caption text will be extracted.

The Close method is then called, since we don't want to use the application itself, but only extract all resources from the executable.

Running once the executable will create a SynFile.messages text file in the SynFile.exe folder, containing all English text:

[TEditForm]
Name.EditLabel.Caption=_2817614158   Name
KeyWords.EditLabel.Caption=_3731019706   KeyWords

[TLoginForm]
Label1.Caption=_1741937413   &User name:
Label2.Caption=_4235002365   &Password:

[TMainForm]
Caption=_16479868    Synopse SQLite3 Framework demo - SynFile

[Messages]
2784453965=Memo
2751226180=Data
744738530=Safe memo
895337940=Safe data
2817614158=Name
1741937413=&User name:
4235002365=&Password:
16479868= Synopse SQLite3 Framework demo - SynFile
940170664=Content
3153227598=None
3708724895=Page %d / %d
2767358349=Size: %s
4281038646=Content Type: %s
2584741026=This memo is password protected.|Please click on the "Edit" button to show its content.
3011148197=Please click on the "Extract" button to get its content.
388288630=Signed,By %s on %s
(...)

The main section of this text file is named [Messages]. In fact, it contains all English extracted texts, as NumericalKey=EnglishText pairs. Note this will reflect the exact content of resourcestring or RTTI captions, including formating characters (like %d), and replacing line feeds (#13) by the special character (a line feed is not expected on a one-line-per-pair file layout). Some other text lines are separated by a comma. This is usual for instance for hint values, as expected by the code.

As requested, each application form has its own section (e.g. [TEditForm], [TMainForm]), proposing some default translation, specified by a numerical key (for instance Label1.Caption will use the text identified by 1741937413 in the [Messages] section). The underline character before the numerical key is used to refers to this value. Note that if no _NumericalKey is specified, a plain text can be specified, in order to reflect a specific use of the generic text on the screen.

Adding a new language

In order to translate the whole application into French, the following SynFile.FR file could be made available in the SynFile.exe folder:

[Messages]
2784453965=Texte
2751226180=Données
744738530=Texte sécurisé
895337940=Données sécurisées
2817614158=Nom
1741937413=&Nom utilisateur:
4235002365=&Mot de passe:
16479868= Synopse mORMot Framework demo - SynFile
940170664=Contenu
3153227598=Vide
3708724895=Page %d / %d
2767358349=Taille: %s                                                        4281038646=Type de contenu: %s                                               2584741026=Le contenu de ce memo est protégé par un mot de passe.|Choisissez "Editer" pour le visualiser.
3011148197=Choisissez "Extraire" pour enregistrer le contenu.
388288630=Signé,Par %s le %s
(....)

Since no form-level custom captions have been defined in this SynFile.FR file, the default numerical values will be used. In our case, Name.EditLabel.Caption will be displayed using the text specified by 2817614158, i.e. 'Nom'.

Note that the special characters %s %d , markup was preserved: only the plain English text has been translated to the corresponding French.

Language selection

User Interface language can be specified at execution.

By default, it will use the registry to set the language. It will need an application restart, but it will also allow easier translation of all forms, using a low-level hook of the TForm.Create constructor.

For instance, if you set in FileMain.pas, for the framework main demo:

procedure TMainForm.FormShow(Sender: TObject);
begin
  (...)
  i18nLanguageToRegistry(lngFrench);
  Ribbon.ToolBar.ActivePageIndex := 1;
end;

Above code will set the main application language as French. At next startup, the content of a supplied SynFileFR.msg file will be used to translate all screen layout, including all RTTI-generated captions.

Of course, for a final application, you'll need to change the language by a common setting. See i18nAddLanguageItems, i18nAddLanguageMenu and i18nAddLanguageCombo functions and procedures to create your own language selection dialog, using a menu or a combo box, for instance.

Localization

Take a look at the TLanguageFile class. After the main language has been set, you can use the global Language instance in order to localize your application layout.

The SQlite3i18n unit will register itself to some methods of SQlite3Commons.pas, in order to translate the RTTI-level text into the current selected language. See for instance i18nDateText.

As usual, feedbacks and comments are welcome on our forum!.

Using Extended in Delphi XE2 64 bit

A.Bouchez — Mon, 12 Sep 2011 23:13:00 +0200

Unfortunately, Delphi's 64-bit compiler (dcc64) and RTL do not support 80-bit extended floating point values on Win64, but silently alias Extended = Double on Win64.

There are situations, however, where this is clearly undesirable, e.g. if the additional precision gained from Extended is required.

The Open-source uTExtendedX87 unit provides a replacement FPU-backed 80-bit Extended floating point type (TExtendedX87) for Win64.

Quoted from the unit presentation:

The aim of this unit is to provide basic 80-bit extended capabilities for Delphi x64 by providing:

A dedicated FPU-backed TExtendedX87 type that behaves as an 80-bit extended value both on 32-bit and 64-bit platforms

Re-implementing basic routines from the System unit for this type on 32-bit and 64-bit

Users should be able to either re-declare their variables explictly as TExtendedX87 or redeclare the type extended as
Extended = TExtendedX87;

Freely downloaded from Embarcadero's Code Central.

This unit is very well written. It uses x87 assembly code and operator overloading to provide 80 bit fast computation under Delphi XE2 64 bit compiler.
Most of existing code base on the Extended type can be reused directly, or shared among both 32 bit and 64 bit versions.

Just another gem of Philipp M. Schlüter.
Open Source and Delphi rock!