Synopse

To content | To menu | To search

Tag - SQL

Entries feed

2012, Sunday January 29

Microsoft states: OleDB out - enjoy ODBC!

For our native connection to any DB, we developed a set of classes and several units.

We implemented at first OleDB, then native Oracle direct access and SQlite3 static engine.

Now, Microsoft is officially deprecating OleDB, and urge all developers to switch to the open and cross-platform ODBC API for native connection.

Continue reading...

2012, Tuesday January 17

SynDBExplorer fast direct export

The Open Source SynDBExplorer tool has been enhanced these days.

Main new features are:

  • Execution of the current selected text (if any) instead of the whole memo content;
  • "Exec & Export" new button, for direct export to file.
I really like the selection execution feature - this speed up SQL process a lot, and allow to switch from one statement to another.
And the new exporting features are opening new possibilities.

Continue reading...

2011, Tuesday December 6

Automatic JOIN query

In mORMot, all the methods available to handle many-to-many relationship (ManySelect, DestGetJoined...) are used to retrieve the relations between tables from the pivot table point of view. This saves bandwidth, and can be used in most simple cases, but it is not the only way to perform requests on many-to-many relationships. And you may have several TSQLRecordMany instances in the same main record - in this case, those methods won't help you.

It is very common, in the SQL world, to create a JOINed request at the main "Source" table level, and combine records from two or more tables in a database. It creates a set that can be saved as a table or used as is. A JOIN is a means for combining fields from two or more tables by using values common to each. Writing such JOINed statements is not so easy by hand, especially because you'll have to work with several tables, and have to specify the exact fields to be retrieved; if you have several pivot tables, it may start to be a nightmare.

Let's see how our ORM will handle it.

Continue reading...

2011, Wednesday November 23

Does speed matters?

Luigi Sandon wrote on Embarcadero's forum

And then you ask yourself: "why use a native compiler if its code may be even slower than jitted one?". Hope the new developers will also develop better and faster code - and not viceversa.

Embarcadero is just following the Wirth's law slower than others:

"Software is getting slower more rapidly than hardware becomes faster"

Speed is only a matter of compiler for mathematical computing intensive tasks.
Most of the time, in real apps (like business apps), the main speed issue is more the framework size (and the number of dll invoked), memory consumption, and general design (e.g. how caching and SQL are written).

Delphi, Java or .Net can do slow apps.
Delphi, Java or .Net can do fast apps.

You can do small and fast stand-alone apps with Delphi, running from Windows 2000 to Windows 8.
It is not possible with Java nor .Net.

This is the main difference IMHO with native code and JIT - about memory use, ease of distribution and no need of an external runtime framework.

Continue reading...

2011, Tuesday November 8

Currency is your friend

The currency type is the standard Delphi type to be used when storing and handling monetary values. It will avoid any rounding problems, with 4 decimals precision. It is able to safely store numbers in the range -922337203685477.5808 .. 922337203685477.5807. Should be enough for your pocket change.

As stated by the official Delphi documentation:

Currency is a fixed-point data type that minimizes rounding errors in monetary calculations. On the Win32 platform, it is stored as a scaled 64-bit integer with the four least significant digits implicitly representing decimal places. When mixed with other real types in assignments and expressions, Currency values are automatically divided or multiplied by 10000.

In fact, this type matches the corresponding OLE and .Net implementation of currency, and the one used by most database providers (when it comes to money, a dedicated type is worth the cost in a "rich man's world"). It is still implemented the same in the Win64 platform (since XE 2). The Int64 binary representation of the currency type (i.e. value*10000 as accessible via PInt64(aCurrencyValue)^) is a safe and fast implementation pattern.

In our framework, we tried to avoid any unnecessary conversion to float values when dealing with currency values. Some dedicated functions have been implemented for fast and secure access to currency published properties via RTTI, especially when converting values to or from JSON text. Using the Int64 binary representation can be not only faster, but also safer: you will avoid any rounding problem which may be introduced by the conversion to a float type. Rounding issues are a nightmare to track - it sounds safe to have a framework handling natively a currency type from the ground up.

Continue reading...

2011, Sunday September 25

Synopse SQLite3 framework is now mORMot

In case you were redirected from the previous "Synopse SQLite3 framework" category link, here is the new thread to be used instead:
http://blog.synopse.info/category/Open-Source-Projects/mORMot-Framework

Since revision 1.15 of the framework, it is able to connect to any database engine (therefore is not limited to SQLite3), and is now called mORMot.

2011, Saturday August 20

Enhanced Log viewer

We already shipped a sophisticated set of logging classes some month ago.

Since then, its features have been enhanced, and the system has been deeply interfaced with our main ORM framework. Now almost all low-level or high-level operations can be logged on request.

But since the log files tend to be huge (for instance, if you set the logging for our unitary tests, the 6,000,000 unitary tests creates a 280 MB log file), a log viewer was definitively in need.

Continue reading...

2011, Wednesday August 10

Framework documentation updated for revision 1.15

The framework documentation was just updated.

The general organization of the SAD document (which is the one to be read in all cases) has been refreshed, and is now separated in smaller chapters.

The new official name has been changed into "Synopse SQLite3/mORMot framework"...

Continue reading...

2011, Sunday August 7

SQLite3-powered, not SQLite3-limited

Our downloadable documentation has been enhanced, and contains now a description about the main feature of the near-to-come 1.15 version, i.e. "database agnosticism".

The core database of our mORMot / SQLite3 framework uses the SQLite3 library, which is a Free, Secure, Zero-Configuration, Server-less, Single Stable Cross-Platform Database File database engine.

As stated below, you can use any other database access layer, if you wish. A fast in-memory engine is included, and can be used instead or together with the SQLite3 engine. Since revision 1.15 of the framework you may be able to access any remote database, and use one or more OleDB (or Oracle) connections to store your precious ORM objects. The SQlite3 will be used as the main SQL engine, able to JOIN all those tables, thanks to its Virtual Table unique feature.

Continue reading...

2011, Monday July 25

Close future of the framework: database agnosticism

Our ORM RESTful Framework is about to access any available database engine.

It will probably change its name (since it won't use only SQlite3 as database), to become mORMot - could be an acronym for "Manage Object Relational Mapping Over Tables", or whatever you may think of...

We'll still rely on SQLite3 on the server, but a dedicated mechanism will allow to access via OleDB any remote database, and mix those tables content with the native ORM tables of the framework. A flexible Virtual Tables and column mapping will allow any possible architecture: either a new project in pure ORM, either a project relying on an existing database with its own table layout.

Continue reading...

2011, Friday July 22

SynDBSQLite3: SQLite3 direct access

For our ORM framework, we implemented an efficient SQLite3 wrapper, joining statically (i.e. without any external dll) the SQLite3 engine to the executable. SQLite3 is in fact used as the DB kernel of the framework. For instance, thanks to its unique virtual table mechanism, even tables in other databases (like Oracle or MSSQL) are available as if they were SQLite3 tables.

We just made this wrapper independent from our ORM, in a new dedicated unit, named SynSQLite3.pas.

It was an easy task to let this unit be called from our SynDB database abstract classes.

Continue reading...

2011, Saturday July 9

SynDBOracle: Open Source native Oracle access

For our upcoming mORMot framework, and in completion to our SynOleDB unit, we just added a new Open Source unit, named SynDBOracle. It allows direct access to any remote Oracle server, using the Oracle Call Interface.

Oracle Call Interface (OCI) is the most comprehensive, high performance, native unmanaged interface to the Oracle Database that exposes the full power of the Oracle Database. We wrote a direct call of the oci.dll library, using our DB abstraction classes introduced for SynOleDB.

We tried to implement all best-practice patterns detailed in the official Building High Performance Drivers for Oracle document

Resulting speed is quite impressive: for all requests, SynDBOracle is 3 to 5 times faster than a SynOleDB connection using the native OleDB Provider supplied by Oracle. We noted also that our implementation is 10 times faster than the one provided with ZEOS/ZDBC, which is far from optimized.

You can use the latest version of the Oracle Instant Client provided by Oracle - see this link - which allows you to run your applications without installing the standard (huge) Oracle client or having an ORACLE_HOME. Just deliver the dll files in the same directory than your application, and it will work.

Continue reading...

2011, Saturday July 2

Is Object-Relational Mapping the Paradise of Computer Science?

There is a well known syndrome around, against ORM.

Do you remember The Vietnam of Computer Science article?

It is worth reading... and commenting.
Sounds a bit outdated by now. Tempus fugit!

Continue reading...

Faster variant late binding

For both our SynOleDB and SynBigTable units, we allow late-binding of data row values, using a variant and direct named access of properties. Thanks to this unique feature (as far as I know in the Delphi database world),

This allows clear and valid code as such:

var Customer: Variant;
begin
  with Props.Execute(
    'select * from Sales.Customer where AccountNumber like ?',
    ['AW000001%'],@Customer) do
    while Step do
      assert(Copy(Customer.AccountNumber,1,8)='AW000001');
end;

In practice, this code is slower than using a standard property based access, like this:

    while Step do
      assert(Copy(Column['AccountNumber'],1,8)='AW000001');

But the first version, using late-binding of column name, just sounds more natural.

Of course, since it's late-binding, we are not able to let the compiler check at compile time for the column name. If the column name in the source code is wrong, an error will be triggered at runtime only. But it would not be an issue, since it would be the same for the SQL code inserted: it's only executed at runtime (this is one of the benefits of using an ORM, by the way: the ORM will generate correct SQL code for you...).

The default VCL implementation of this late-binding was a bit slow for our purpose.
Since it has to deal with Ole Automation, and because it's fun, we hacked the VCL to provide a lighter and faster version for our custom variant types.

Continue reading...

2011, Friday July 1

SynOleDB: OpenSource Unit for direct access to any database via OleDB

That's it, our SynOleDB unit seems alive and running well.

OLE DB (Object Linking and Embedding, Database, sometimes written as OLEDB or OLE-DB) is an API designed by Microsoft for accessing data from a variety of sources in a uniform manner. It was designed as a higher-level replacement for, and successor to, ODBC, extending its feature set to support a wider variety of non-relational databases, such as object databases and spreadsheets that do not necessarily implement SQL.

SynOleDB unit implementation has been made with several points in mind:

  • Tested with SQL Server 2008 R2 and Oracle 11g providers from Microsoft and Oracle; 
  • Ability to be truly Unicode, even with pre-Unicode version of Delphi (like Delphi 7 or 2007); 
  • Could access any local or remote Database, from any version of Delphi, since it doesn't use the DB.pas unit or any related part of the VCL (even the Delphi 7 personal or the Turbo Explorer editions), just for free; 
  • Handle NULL or BLOB content for parameters and results; 
  • Avoid most memory copy or unnecessary allocation: we tried to access the data directly from the retrieved data buffer, just as given from OleDB; 
  • Was therefore designed to achieve the best performance possible: most time is spent in OleDB: the code layer added to the OleDB customer is very thin; 
  • True OOP architecture, to be used with any OleDB provider (allowing custom parameters or such), and even without OleDB (in the future, direct access to any DB client could be used); 
  • Could be safely used in a multi-threaded application/server (with one TOleDBConnection per thread); 
  • Allow parameter bindings of requests, with fast access to any parameter or column name (thanks to TDynArrayHashed);
  • Late binding of column values in Delphi code;
  • Direct JSON content creation, with no temporary data copy nor allocation; 
  • Designed to be used with our mORMot ORM, but could be used stand-alone (a full Delphi 7 client executable is just about 200 KB), or even in any existing Delphi application, thanks to a TQuery-like wrapper.

Continue reading...

2011, Wednesday June 29

Synopse SQLite3 framework 1.14

Our ORM framework has been released as version 1.14.

It's mainly a bug-fix release:

  • Integrated SQLite3 engine updated to latest version 3.7.7.1;
  • Fix several issues about JSON generation layout;
  • Enhanced automated User Interface generation for object on-screen edition;
  • SynPdf unit now handles Bézier curves from TCanvas, and some CMYK functions; also enhanced PDF/A-1compatibility;
  • Some speed enhancements, and new functions for the SynOleDB unit.

Continue reading...

2011, Thursday June 2

Custom SQL functions

The SQLite3 engine defines some standard SQL functions, like abs() min() max() or upper().
A complete list is available at http://www.sqlite.org/lang_corefunc.html

One of the greatest SQLite3 feature is the ability to define custom SQL functions in high-level language. In fact, its C API allows to implement new functions which may be called within a SQL query. In other database engine, such functions are usually named UDF (for User Defined Functions).

Our framework allows you to add easily such custom functions, directly from Delphi classes.

Continue reading...

2011, Thursday April 14

Enhanced logging in SynCommons

Logging is everything... unless you never wrote a bug in your program! :)

Let us introduce a new logging class:

  • logging with a set of levels;
  • fast, low execution overhead;
  • can load .map file symbols to be used in logging;
  • compression of .map into binary .mab (900 KB -> 70 KB);
  • inclusion of the .map/.mab into the .exe;
  • reading of an external .map to add unit names and line numbers to a log file without .map available information at execution;
  • exception logging (Delphi or low-level exceptions) with unit names and line numbers;
  • optional stack trace with units and line numbers;
  • methods or procedure recursive tracing, with Enter and auto-Leave;
  • high resolution time stamps, for customer-side profiling of the application execution;
  • set / enumerates / TList / TPersistent / TObjectList / TContainer / dynamic array JSON serialization;
  • per-thread or global logging;
  • multiple log files on the same process;
  • integrated log archival (in zip or any other format);
  • Open Source, works from Delphi 6 up to XE.

Continue reading...