Synopse

To content | To menu | To search

Tag - dynamic array

Entries feed

2014, Saturday September 13

Some thoughts about "modern" pascal, generics, code and data structures

In a comment of a Google+ announcement about new C# features, Stephan did react about my naive enthusiasm about SmartPascal.

Apart from the fact that he did miss the numerous ways of creating Windows executable in this dialect (I quoted at least 5 diverse ways), he was chocked by the fact that the SmartPascal syntax, in its actual idiom, does not support generics.

But are generics mandatory?
I'm not speaking about any drug identified by its chemical name rather than its brand name (Wikipedia).
I would neither comment on the current Delphi implementation of generics (which may appear not so polished, nor very widely used, even in the Delphi RTL/FMX, but for TList<T> TDictionary<>).
Just share some thoughts about what is, to my opinion - which may be wrong and biased! - the most important part of today's programming.

Continue reading...

2014, Sunday May 18

Automatic JSON serialization of record or dynamic arrays via Enhanced RTTI

Since Delphi 2010, the compiler generates additional RTTI at compilation, so that all record fields are described, and available at runtime.
By the way, this enhanced RTTI is one of the reasons why executables did grow so much in newer versions of the compiler.

Our SynCommons.pas unit is now able to use this enhanced information, and let any record be serialized via RecordLoad() and RecordSave() functions, and all internal JSON marshalling process.

In short, you have nothing to do.
Just use your record as parameters, and, with Delphi 2010 and up, they will be serialized as valid JSON objects.

Of course, text-based definition or callback-based registration are still at hand, and will be used with older versions of Delphi.
But you could be used to by-pass or extend the enhanced-RTTI serialization, even on newer versions of the compiler.

Continue reading...

2014, Friday April 18

Introducing mORMot's architecture and design principles

We have just released a set of slides introducing 

  • ORM, SOA, REST, JSON, MVC, MVVM, SOLID, Mocks/Stubs, Domain-Driven Design concepts with Delphi, 
  • and showing some sample code using our Open Source mORMot framework.

You can follow the public link on Google Drive!

This is a great opportunity to discovers some patterns you may not be familiar with, and find out how mORMot try to implement them.
This set of slides may be less intimidating than our huge documentation - do not be terrified by our Online Documentation!
The first set of pages (presenting architecture and design principles) is worth reading.

Feedback is welcome on our forum, as usual.

2014, Tuesday February 25

TDocVariant custom variant type

With revision 1.18 of the framework, we just introduced two new custom types of variants:

  • TDocVariant kind of variant;
  • TBSONVariant kind of variant.

The second custom type (which handles MongoDB-specific extensions - like ObjectID or other specific types like dates or binary) will be presented later, when dealing with MongoDB support in mORMot, together with the BSON kind of content. BSON / MongoDB support is implemented in the SynMongoDB.pas unit.

We will now focus on TDocVariant itself, which is a generic container of JSON-like objects or arrays.
This custom variant type is implemented in SynCommons.pas unit, so is ready to be used everywhere in your code, even without any link to the mORMot ORM kernel, or MongoDB.

TDocVariant documents

TDocVariant implements a custom variant type which can be used to store any JSON/BSON document-based content, i.e. either:

  • Name/value pairs, for object-oriented documents;
  • An array of values (including nested documents), for array-oriented documents;
  • Any combination of the two, by nesting TDocVariant instances.

Here are the main features of this custom variant type:

  • DOM approach of any object or array documents;
  • Perfect storage for dynamic value-objects content, with a schema-less approach (as you may be used to in scripting languages like Python or JavaScript);
  • Allow nested documents, with no depth limitation but the available memory;
  • Assignment can be either per-value (default, safest but slower when containing a lot of nested data), or per-reference (immediate reference-counted assignment);
  • Very fast JSON serialization / un-serialization with support of MongoDB-like extended syntax;
  • Access to properties in code, via late-binding (including almost no speed penalty due to our VCL hack as already detailed);
  • Direct access to the internal variant names and values arrays from code, by trans-typing into a TDocVariantData record;
  • Instance life-time is managed by the compiler (like any other variant type), without the need to use interfaces or explicit try..finally blocks;
  • Optimized to use as little memory and CPU resource as possible (in contrast to most other libraries, it does not allocate one class instance per node, but rely on pre-allocated arrays);
  • Opened to extension of any content storage - for instance, it will perfectly integrate with BSON serialization and custom MongoDB types (ObjectID, RegEx...), to be used in conjunction with MongoDB servers;
  • Perfectly integrated with our Dynamic array wrapper and its JSON serialization as with the record serialization;
  • Designed to work with our mORMot ORM: any TSQLRecord instance containing such variant custom types as published properties will be recognized by the ORM core, and work as expected with any database back-end (storing the content as JSON in a TEXT column);
  • Designed to work with our mORMot SOA: any interface-based service is able to consume or publish such kind of content, as variant kind of parameters;
  • Fully integrated with the Delphi IDE: any variant instance will be displayed as JSON in the IDE debugger, making it very convenient to work with.

To create instances of such variant, you can use some easy-to-remember functions:

  • _Obj() _ObjFast() global functions to create a variant object document;
  • _Arr() _ArrFast() global functions to create a variant array document;
  • _Json() _JsonFast() _JsonFmt() _JsonFastFmt() global functions to create any variant object or array document from JSON, supplied either with standard or MongoDB-extended syntax.

Continue reading...

2013, Tuesday December 10

JSON record serialization

In Delphi, the record has some nice advantages:

  • record are value objects, i.e. accessed by value, not by reference - this can be very convenient, e.g. when defining a Domain-Driven Design
  • record can contain any other record or dynamic array, so are very convenient to work with (no need to define sub-classes or lists); 
  • record variables can be allocated on stack, so won't solicit the global heap; 
  • record instances automatically freed by the compiler when they come out of scope, so you won't need to write any try..finally Free; end block.

Serialization of record values are therefore a must-have for a framework like mORMot.

In recent commits, this JSON serialization of record has been enhanced.
In particular, we introduced JSON serialization via a new text-based record definition.

Continue reading...

2013, Saturday May 11

Delphi XE4 NextGen compiler: using byte instead of ansichar?

When I first read the technical white paper covering all of the language changes in XE4 for mobile development (tied to the new ARM LLVM-based Delphi compiler), I have to confess I was pretty much confused.

Two great mORMot users just asked for XE4/iOS support of mORMot.

Win32/Win64 support for XE4 will be done as soon as we got a copy of it.
I suspect the code already works, since it was working as expected with XE3, and we rely on our own set of low-level functions for most internal work.

But iOS-targetting is more complex, due to the NextGen compiler, mainly.

Continue reading...

2013, Sunday February 17

Interface-based service sample: remote SQL access

You will find in the SQLite3\Sample\16 - Execute SQL via services folder of mORMot source code a Client-Server sample able to access any external database via JSON and HTTP.
It is a good demonstration of how to use an interface-based service between a client and a server.
It will also show how our SynDB classes have a quite abstract design, and are easy to work with, whatever database provider you need to use.

The corresponding service contract has been defined:

  TRemoteSQLEngine = (rseOleDB, rseODBC, rseOracle, rseSQlite3, rseJet, rseMSSQL);

IRemoteSQL = interface(IInvokable) ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}'] procedure Connect(aEngine: TRemoteSQLEngine; const aServerName, aDatabaseName, aUserID, aPassWord: RawUTF8); function GetTableNames: TRawUTF8DynArray; function Execute(const aSQL: RawUTF8; aExpectResults, aExpanded: Boolean): RawJSON; end;

Purpose of this service is:
- To Connect() to external databases, given the parameters of a standard TSQLDBConnectionProperties. Create() constructor;
- Retrieve all table names of this external database as a list;
- Execute any SQL statement, returning the content as JSON array, ready to be consumed by AJAX applications (if aExpanded is true), or a Delphi client (e.g. via a TSQLTableJSON and the mORMotUI unit).

Of course, this service will be define as sicClientDriven mode, that is, the framework will be able to manage a client-driven TSQLDBProperties instance life time.

Benefit of this service is that no database connection is required on the client side: a regular HTTP connection is enough.
No need to install nor configure any database provider, and full SQL access to the remote databases.

Due to our optimized JSON serialization, it will probably be faster to work with such plain HTTP / JSON services, instead of a database connection through a VPN. In fact, database connections are made to work on a local network, and do not like high-latency connections, which are typical on the Internet.

Continue reading...

2012, Monday December 31

Enhance existing projects with mORMot

Even if mORMot will be more easily used in a project designed from scratch, it fits very well the purpose of evolving any existing Delphi project, or even creating the server side part of an AJAX application. 

One benefit of such a framework is to facilitate the transition from a Client-Server architecture to a N-Tier layered pattern.

Continue reading...

2012, Tuesday November 13

Go language and Delphi

Do you know the Go language?

It is a strong-typed, compiled, cross-platform, and concurrent.
It features some nice high-level structures, like maps and strings, and still have very low-level access to the generated code: pointers are there, in a safe strong-typed implementation just like in pascal, and there is even a "goto", which sounds like an heresy to dogmatic coders, but does make sense to me, at least when you want to optimize code speed, in some rare cases.

It is created/pushed by Google, used internally by the company in their computer farms, and was designed by one of the original C creators.

Continue reading...

2012, Saturday October 6

Delphi XE3 is preparing (weak) reference counting for class instances

In Delphi, you have several ways of handling data life time, therefore several ways of handling memory:

  • For simple value objects (e.g.  byte integer double shortstring and fixed size arrays or record containing only such types), the value is copied in fixed-size buffers;
  • For more complex value objets (e.g. string and dynamic arrays or record containing such types), there is a reference counter handled by each instance, with copy-on-write feature and compiler-generated reference counting at code scope level (with hidden try..finally blocks);
  • For most class instances (e.g. deriving from TObject), you have to Create then Free each instance, and manage its life time by hand - with explicit try..finally blocks;
  • For class deriving from TInterfacedObject, you have a RefCount property, with _AddRef _Release methods (this is the reference-counted COM model), and you can use Delphi interface to work with such instances - see this blog article.
With Delphi XE3, we were told that some automatic memory handling at class level are about to be introduced at the compiler and RTL level.
Even if this feature is not finished, and disabled, there are a lot of changes in the Delphi XE3 Run Time Library which sounds like a preparation of such a new feature.

Continue reading...

2012, Thursday July 12

One ORM to rule them all

If you discovered the mORMot framework, you may have found out that its implementation may sound restricted, in comparison to other ORMs, due to its design. It would be easy to answer that "it is not a bug, it is a feature", but I suspect it is worth a dedicated article.

Some common (and founded) criticisms are the following (quoting from our forum - see e.g. this question):
- "One of the things I don't like so much about your approach to the ORM is the mis-use of existing Delphi constructs like "index n" attribute for the maximum length of a string-property. Other ORMs solve this i.e. with official Class-attributes";
- "You have to inherit from TSQLRecord, and can't persist any plain class";
- "There is no way to easily map an existing complex database".

I understand very well those concerns.
Our mORMot framework is not meant to fit any purpose, but it is worth understanding why it has been implemented as such, and why it may be quite unique within the family of ORMs - which almost all are following the Hibernate way of doing.

Continue reading...

2012, Monday June 18

Circular reference and zeroing weak pointers

The memory allocation model of the Delphi interface type uses some kind of Automatic Reference Counting (ARC). In order to avoid memory and resource leaks and potential random errors in the applications (aka the terrible EAccessViolation exception on customer side) when using interface, a SOA framework like mORMot has to offer so-called Weak pointers and Zeroing Weak pointers features.

Note that garbage collector based languages (like Java or C#) do not suffer from this problem, since the circular references are handled by their memory model: objects lifetime are maintained globally by the memory manager. Of course, it will increase memory use, slowdown the process due to additional actions during allocation and assignments (all objects and their references have to be maintained in internal lists), and may slow down the application when garbage collector enters in action. In order to avoid such issues when performance matters, experts tend to pre-allocate and re-use objects: this is one common limitation of this memory model, and why Delphi is still a good candidate (like unmanaged C or C++ - and also Objective C) when it deals with performance and stability.

Continue reading...

2012, Friday May 25

Domain-Driven design

With a previous article, we introduced the concept of "Domain-Driven design" into our framework presentation.

It's now time to detail a bit more this very nice software architecture design, and how mORMot is able to achieve such an implementation pattern.

Continue reading...

2012, Thursday May 3

Custom JSON serialization of records

By default, during interface-based service call, any record parameter or function result will be serialized with our proprietary binary (and optimized layout) - i.e. RecordLoad and RecordSave functions - then encoded in Base-64, to be stored as plain text within the JSON stream.

But custom record JSON serialization can be defined, as with any class - see this article - or dynamic array - see this article.

Continue reading...

2012, Thursday April 12

Custom JSON serialization of any dynamic array content

One nice feature of the TDynArray wrapper and its associated methods, as defined and used in our framework, is the ability to serialize any dynamic array as JSON content.

By default, only "standard" dynamic arrays (like TIntegerDynArray) are serialized as true JSON array: other not known kind of arrays are serialized as binary content, within a Base64 encoding.

This is a very efficient solution for a pure Delphi application, since it will be fast and always works, but won't be easy to deal with from an AJAX client.

Applications can now supply a custom JSON serialization for any other dynamic array, via the TTextWriter.RegisterCustomJSONSerializer() class method.
Two callbacks are to be supplied for a dynamic array type information, in order to handle proper serialization and un-serialization of the JSON array.

Continue reading...

2011, Sunday December 11

Strong-typing just rocks

To my understanding, the so-called "strong-typing" feature is one big benefit of the Delphi object pascal language.

As stated by wikipedia:

Most generally, "strong typing" implies that the programming language places severe restrictions on the intermixing that is permitted to occur, preventing the compiling or running of source code which uses data in what is considered to be an invalid way. For instance, an addition operation may not be used with an integer and string values; a procedure which operates upon linked lists may not be used upon numbers. However, the nature and strength of these restrictions is highly variable.

Some Delphi users may find this is a limitation of the language, in comparison with other "fashionable" script idioms (like Python, Javascript of Ruby). For me, runtime strong typing (alla Python or Ruby) is not true strong typing. Simon Stuart just proposed a smartstring kind of string, which is in fact a weakstring type. As far as I understood his point, he wanted to get rid of all the warnings emitted by Unicode-version of the Delphi compiler, about explicit string conversion.

In fact, I use to go in the opposite direction. For wide projects, strong-typing is one of the big benefit of using Delphi (like other main "serious" languages like Java, C, C++, Ada or C#).

Continue reading...

2011, Thursday December 8

Avoiding Garbage Collector: Delphi and Apple side by side

Among all trolling subject in forums, you'll find out the great Garbage Collection theme.

Fashion languages rely on it. At the core of the .Net and Java framework, and all scripting languages (like JavaScript, Perl, Python or Ruby), you'll find a Garbage Collector. New developers, just released from schools, do learn about handling memory only in theory, and just can't understand how is memory allocated - we all have seen such rookies involved in Delphi code maintenance, leaking memory as much as they type. In fact, most of them did not understood how a computer works. I warned you this will be a trolling subject.

And, in Delphi, there is no such collector. We handle memory in several ways:

  • Creating static variables - e.g. on the stack, inside a class or globally;
  • Creating objects with class instances allocated on heap - in at least three ways: with a try..finally Free block, with a TComponent ownership model in the VCL, or by using an interface (which creates an hidden try..finally Free block);
  • Creating reference-counted variables, i.e. string, array of, interface or variant kind of variables.

It is a bit complex, but it is also deadly powerful. You have several memory allocation models at hand, which can be very handy if you want to tune your performance and let program scale. Just like manual recycling at home will save the planet. Some programmers will tell you that it's a waste of cell brain, typing and time. Linux kernel gurus would not say so, I'm afraid.

Then came the big Apple company, which presented its new ARC model (introduced in Mac OS X 10.7 Lion) as a huge benefit for Objective-C in comparison with the Garbage Collection model. And let's face it: this ARC just sounds like the Delphi memory model.

Continue reading...

2011, Tuesday November 8

Currency is your friend

The currency type is the standard Delphi type to be used when storing and handling monetary values. It will avoid any rounding problems, with 4 decimals precision. It is able to safely store numbers in the range -922337203685477.5808 .. 922337203685477.5807. Should be enough for your pocket change.

As stated by the official Delphi documentation:

Currency is a fixed-point data type that minimizes rounding errors in monetary calculations. On the Win32 platform, it is stored as a scaled 64-bit integer with the four least significant digits implicitly representing decimal places. When mixed with other real types in assignments and expressions, Currency values are automatically divided or multiplied by 10000.

In fact, this type matches the corresponding OLE and .Net implementation of currency, and the one used by most database providers (when it comes to money, a dedicated type is worth the cost in a "rich man's world"). It is still implemented the same in the Win64 platform (since XE 2). The Int64 binary representation of the currency type (i.e. value*10000 as accessible via PInt64(aCurrencyValue)^) is a safe and fast implementation pattern.

In our framework, we tried to avoid any unnecessary conversion to float values when dealing with currency values. Some dedicated functions have been implemented for fast and secure access to currency published properties via RTTI, especially when converting values to or from JSON text. Using the Int64 binary representation can be not only faster, but also safer: you will avoid any rounding problem which may be introduced by the conversion to a float type. Rounding issues are a nightmare to track - it sounds safe to have a framework handling natively a currency type from the ground up.

Continue reading...

2011, Monday July 25

Close future of the framework: database agnosticism

Our ORM RESTful Framework is about to access any available database engine.

It will probably change its name (since it won't use only SQlite3 as database), to become mORMot - could be an acronym for "Manage Object Relational Mapping Over Tables", or whatever you may think of...

We'll still rely on SQLite3 on the server, but a dedicated mechanism will allow to access via OleDB any remote database, and mix those tables content with the native ORM tables of the framework. A flexible Virtual Tables and column mapping will allow any possible architecture: either a new project in pure ORM, either a project relying on an existing database with its own table layout.

Continue reading...

2011, Sunday July 3

"Sharding" or "Share nothing" architecture

Here is what wikipedia states at http://en.wikipedia.org/wiki/Shared_nothing_architecture:

A shared nothing architecture (SN) is a distributed computing architecture in which each node is independent and self-sufficient, and there is no single point of contention across the system. People typically contrast SN with systems that keep a large amount of centrally-stored state information, whether in a database, an application server, or any other similar single point of contention.

This is just one approach of "sharding". Sharding is indeed related to a shared nothing architecture - once sharded, each shard can live in a totally separate logical schema instance.
"I sharded, therefore it scales"...

You can do this in Delphi... and opens a new world of scaling opportunities... Just as Google, Facebook, or eBay do... 

Continue reading...

- page 1 of 2