Synopse Open Source - Tag - SynDBExplorermORMot MVC / SOA / ORM and friends2024-02-02T17:08:25+00:00urn:md5:cc547126eb580a9adbec2349d7c65274DotclearHTTP remote access for SynDB SQL executionurn:md5:8ac51ed0a6e45f22746c8ba9e6d3e63c2014-11-18T01:08:00+01:002014-11-18T01:11:52+01:00AB4327-GANDIOpen Source librariesarray bindingblogcompressionDatabaseDelphiFirebirdFireDACHTTPhttp.sysHTTPSmORMotMSSQLMySQLOpenSourceOracleORMperformancesecuritySQLSynDBSynDBExplorerTDataSetwebWinHTTPWinINetZEOS<p>For <em>mORMot</em>, we developed a fully feature direct access layer to any
RDBMS, implemented <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_126">
in the SynDB.pas unit</a>.</p>
<p>You can use those <code>SynDB</code> classes to execute any SQL statement,
without any link to the framework ORM.<br />
At reading, the resulting performance is much higher than using the standard
<code>TDataSet</code> component, which is in fact a true performance
bottleneck.<br />
It has genuine features, like <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_136">
column access via late-binding</a>, <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_137">
an innovative <code>ISQLDBRows</code> interface, and ability to directly access
the low-level binary buffers of the database clients.</a></p>
<p><img src="http://vpn-services.bestreviews.net/files/vpn-remote-access.jpg" alt="" /></p>
<p>We just added a nice feature to those classes: the ability to access
remotely, via plain HTTP, to any <code>SynDB</code> supported database!</p> <p>To define a HTTP server, you may write:</p>
<pre>
<strong>uses</strong> SynDB, <em>// RDBMS core</em>
SynDBSQLite3, SynSQLite3Static, <em>// static SQLite3 engine</em>
SynDBRemote; <em>// for HTTP server</em>
...
<strong>var</strong> Props: TSQLDBConnectionProperties;
HttpServer: TSQLDBServerAbstract;
...
Props := TSQLDBSQLite3ConnectionProperties.Create('data.db3','','','');
<span style="background-color:yellow;">HttpServer := TSQLDBServerHttpApi.Create(Props,'remote','8092','user','pass');</span>
</pre>
<p>The above code will initialize a connection to a local <code>data.db3</code>
<em>SQlite3</em> database (in the <code>Props</code> variable), and then
publish it using the <code>http.sys</code> kernel mode HTTP server to the
<code>http://1.2.3.4:8092/remote</code> URI - if the server's IP is
<code>1.2.3.4</code>.</p>
<p>On the client side, you can then write:</p>
<pre>
<strong>uses</strong> SynDB, <em>// RDBMS core</em>
SynDBRemote; <em>// for HTTP server</em>
...
<strong>var</strong> Props: TSQLDBConnectionProperties;
...
Props := TSQLDBWinHTTPConnectionProperties.Create('1.2.3.4:8092','root','user','pass');
</pre>
<p>As you can see, there is no link to <code>SynDBSQLite3.pas</code> nor
<code>SynSQLite3Static.pas</code> on the client side.<br />
Just the HTTP link is needed.<br />
No need to deploy the RDBMS client libraries with your application, nor setup
the local network firewall.</p>
<p>We defined here a single user, with 'user' / 'pass' credentials, but you may
manage more users on the server side, using the <code>Authenticate</code>
property of <code>TSQLDBServerAbstract</code>.<br />
Note that in our remote access, user management does not match the RDBMS user
rights: you should better have your own set of users at application level, for
higher security, and a better integration with your business logic. If creating
a new user on a RDBMS could be painful, it is pretty easy on our
<code>SynDBRemote.pas</code> side.</p>
<p>Then, you execute your favorite SQL using the connection just as usual:</p>
<pre>
<strong>procedure</strong> Test(Props: TSQLDBConnectionProperties);
<strong>var</strong> Stmt: ISQLDBRows;
<strong>begin</strong>
Stmt := Props.Execute('select * from People where YearOfDeath=?',[1519]);
<strong>while</strong> Stmt.Step <strong>do begin</strong>
assert(Stmt.ColumnInt('ID')>0);
assert(Stmt.ColumnInt('YearOfDeath')=1519);
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>You may even use this remote connection e.g. using a stand-alone shared
<em>SQLite3</em> database as high performance but low maintenance client-server
database engine.</p>
<p>The transmission protocol uses an optimized binary format, which is
compressed and digitally signed on both ends, and the remote user
authentication will be performed via a challenge validation scheme.<br />
You could publish your server over HTTPS, if needed.</p>
<p>Even if you may be tempted to use such remote access to implement a
<em>n-Tier architecture</em>, you should rather use <em>mORMot</em>'s
Client-Server ORM instead. Our little <em>mORMot</em> is not an ORM on
which we added a data transmission layer: it is a full RESTful system, with a
true SOA design.<br />
But for integrating some legacy SQL code into a new architecture,
<code>SynDBRemote.pas</code> may have its benefits, used in conjunction with
<em>mORMot</em>'s higher level features.</p>
<p>Feel free to <a href="http://synopse.info/forum/viewtopic.php?id=2178">post
your feedback on our forum</a>, as usual, or <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_131">
browse the updated documentation</a>!</p>SynDBOracle: Open Source native Oracle accessurn:md5:8ae19f8c26e82f0a275f5530cbd56cc42012-10-28T22:13:00+01:002012-10-28T22:16:29+01:00AB4327-GANDImORMot Frameworkarray bindingBatchblogDatabaseDelphiODBCOleDBOracleORMSourceSQLSQLite3SynDBSynDBExplorerUnicode<p><em>(this is an update of the article published in 2011/07)</em></p>
<p>For our <em>mORMot</em> framework, and in completion to our
<em>SynOleDB</em> unit, we added a new Open Source unit, named
<em>SynDBOracle</em>. It allows direct access to any remote Oracle server,
using the <em>Oracle Call Interface</em>.</p>
<p><em>Oracle Call Interface</em> (OCI) is the most comprehensive, high
performance, native unmanaged interface to the Oracle Database that exposes the
full power of the Oracle Database. We wrote a direct call of the
<code>oci.dll</code> library, using our DB abstraction classes introduced for
<em>SynOleDB</em>.</p>
<p>We tried to implement all best-practice patterns detailed in the official
<a href="http://www.oracle.com/technetwork/topics/php/whatsnew/building-best-drivers-131920.pdf">
<em>Building High Performance Drivers for Oracle</em> document</a></p>
<p>Resulting speed is quite impressive: for all requests, <em>SynDBOracle</em>
is 3 to 5 times faster than a <em>SynOleDB</em> connection using the native
<em>OleDB Provider</em> supplied by Oracle. We noted also that our
implementation is 10 times faster than the one provided with ZEOS/ZDBC, which
is far from optimized.</p>
<p>You can use the latest version of the <em>Oracle Instant Client</em>
provided by Oracle - see <a href="http://www.oracle.com/technetwork/database/features/instant-client">this
link</a> - which allows you to run your applications without installing
the standard (huge) Oracle client or having an <code>ORACLE_HOME</code>. Just
deliver the <code>dll</code> files in the same directory than your application,
and it will work.</p> <h3>Main features</h3>
<p>Here are the main features of this unit:</p>
<ul>
<li><em>Direct access</em> to the <em>Oracle Call Interface</em> (OCI) client,
with no BDE, Midas, DBExpress, nor OleDB or ODBC provider necessary; </li>
<li>Dedicated to work with <em>any version</em> of the Oracle OCI interface,
starting from revision 8; </li>
<li><em>Optimized for the latest features</em> of Oracle 11g (e.g. using native
<code>Int64</code> for retrieving NUMBER fields with no decimal); </li>
<li>Able to work with the <em>Oracle Instant Client</em> for <em>No Setup</em>
applications; </li>
<li><em>Natively Unicode</em> (uses internal UTF-8 encoding), for all version
of Delphi, with special handling of each database char-set; </li>
<li>Tried to achieve <em>best performance available</em> from every version of
the Oracle client, and the lowest memory use; </li>
<li>Designed to work under <em>any version of Windows</em>, either in 32 or 64
bit architecture; </li>
<li><em>Late-binding</em> access to column names, using a new dedicated
<code>Variant</code> type (similar to Ole Automation runtime
properties); </li>
<li>Connections are <em>multi-thread ready</em> with low memory and CPU
resource overhead; </li>
<li>Can use connection strings like
<code>'//host[:port]/[service_name]'</code>, avoiding use of the
<code>TNSNAME.ORA</code> file; </li>
<li>Use <em>Rows Array</em> and <em>BLOB fetching</em>, for best performance
(ZEOS/ZDBC did not handle this, for instance);</li>
<li>Very fast insertion, deletion or updates, using <a href="https://blog.synopse.info?post/post/2012/07/19/Oracle-Array-Binding-and-BATCH-performance">batch-mode
sending</a> (aka Array Binding) - more than 50,000 inserts per second;</li>
<li><code>TQuery</code> <em>emulation class</em>, for direct re-use with
existing code, in replacement to the BDE; </li>
<li>Handle <em>Prepared Statements</em> - but by default, we rely on OCI-side
statement cache, if available; </li>
<li>Native <em>export to JSON</em> methods, which will be the main entry point
for our <em>mORMot</em> framework; </li>
<li>By changing one class, you can switch to another <em>SynDB*.pas</em> unit,
like <a href="https://blog.synopse.info?post/post/2011/06/27/SynOleDB%3A-OpenSource-Unit-for-direct-access-to-any-database-via-OleDB">
OleDB</a>, <a href="https://blog.synopse.info?post/post/2012/02/29/Microsoft-states%3A-OleDB-out-enjoy-ODBC%21">ODBC</a>, or
even a <a href="https://blog.synopse.info?post/post/2011/07/22/SynDBSQLite3%3A-SQLite3-direct-access">direct
<em>SQLite3</em> embedded database</a>;</li>
<li>Compatible with <em>Delphi 5 up to XE3</em>; </li>
<li>Since it doesn't use the <em>DB</em> unit, nor <em>DBExpress</em> or such
other technologies, works <em>with any edition of Delphi</em> (even Delphi
XE/XE2/XE3 Stater or Delphi 7 Personal); </li>
<li>Open Source, released under a <em>MPL/GPL/LGPL</em> license.</li>
</ul>
<h3>End-user distribution</h3>
<p>In comparison to the the default Delphi database implementations, here is
how our <em>SynDBOracle</em> classes can be distributed, on the customer
side:</p>
<p><img src="https://blog.synopse.info?post/public/SynDB/.SynDBOracleConnectivity_m.jpg" alt="" title="SynDBOracle connectivity, oct. 2012" /></p>
<p>It appears that one of the most difficult support issue with Oracle, which
is the client itself, is solved by our classes.<br />
Using the official <em>Oracle Instant Client</em> is a much better option that
using the <em>.dcu</em> provided by some vendors, which are in fact a
deprecated <em>OCI</em> client for Oracle 8.</p>
<h3>Units presentation</h3>
<p>Here are the units involved in this library:<br />
- <code>SynCommons</code> is used for all low-level stuff (like UTF-8 or
dynamic arrays), and is common to all our units (including <em>mORMot</em> or
<em>SynPDF</em>);<br />
- <code>SynDB</code> is the main unit, providing abstract classes to implement
connection properties, connections, and statements;<br />
- <code>SynDBOracle</code> will implement all Oracle specific classes, hidding
most of the complexity of the OCI interface;<br />
- <code>SynOleDB</code> is not required, but can be used to connect to other
databases (like <em>Microsoft SQL Server</em>), from the same class
architecture.</p>
<h3>Classes involved</h3>
<p>First of all, the solution expect to use a
<code>TSQLDBConnectionProperties</code> sub-class, which will contain all
connection settings (server, user and password, general parameters like
<code>CodePage</code> or custom internal buffer size).</p>
<p>Here are the classes handling the connection properties:</p>
<p><img src="http://synopse.info/files/pictures/DBConnectionProperties.png" alt="" /></p>
<p>Then you can open a per-thread connection using those
<code>TSQLDBConnection</code> classes:</p>
<p><img src="http://synopse.info/files/pictures/DBConnection.png" /></p>
<p>And finally, it does rely on <code>TSQLDBStatement</code> classes to
implement the actual SQL request.</p>
<h3>Sample code</h3>
<p>Once you have a <code>TOleDBConnectionProperties</code> instance, you can
execute a statement on it directly, as such:</p>
<pre>
<strong>procedure</strong> Test(Props: TOleDBConnectionProperties; <strong>const</strong> aName: RawUTF8);
<strong>var</strong> I: ISQLDBRows;
Customer: <strong>Variant</strong>;
<strong>begin</strong>
I := Props.Execute('select * from Domain.Customers where Name=?',[aName],@Customer);
<strong>while</strong> I.Step <strong>do</strong>
writeln(Customer.Name,' ',Customer.FirstName,' ',Customer.Address);
<strong>end</strong>;
<br /><strong>var</strong> Props: TOleDBConnectionProperties;
<strong>begin</strong>
Props := TSQLDBOracleConnectionProperties.Create(
'TnsName','UserName','Password',CODEPAGE_US);
<strong>try</strong>
Test(Props,'Smith');
<strong>finally</strong>
Props.Free;
<strong>end</strong>;
<strong>end</strong>;
</pre>
<p>I think this above code is not difficult to follow...<br />
The late-binding of column names via our custom Variant type allows to write
code using expressions like <code>Customer.Name</code> - Delphi can be quite
powerful, can't it?</p>
<p>Of course, our <em>mORMot</em> framework uses this unit for direct access to
<em>Oracle</em>, for its <a href="https://blog.synopse.info?post/post/2011/08/07/SQLite3-powered%2C-not-SQLite3-limited">ORM feature for
external databases</a>.<br />
But you do not need to use the <em>mORMot</em>'s ORM: our SynDB*.pas units are
designed to be used stand-alone.</p>
<p>This unit is now in stable state, and is available as part as <em><a href="http://mormot.net">mORMot</a></em>.</p>
<p>Feedback and comments are <a href="http://synopse.info/forum/viewtopic.php?id=365">welcome on our forum</a>, just
as usual.</p>SynDBExplorer enhancementsurn:md5:c4fd28522d51a5c8b2aa7600eeeed5e22012-06-23T12:21:00+02:002012-06-23T11:26:10+02:00AB4327-GANDIOpen Source librariesblogDatabaseDelphiOleDBOracleperformanceSourceSQLSQLite3SynDBSynDBExplorer <p>Our <a href="https://blog.synopse.info?post/tag/SynDBExplorer">SynDBExplorer</a> free tool has been
enhanced.</p>
<p>A SQL request history has been added to the software.<br />
It is now able to <a href="http://synopse.info/forum/viewtopic.php?id=740">handle directly Jet / MSAccess
.mdb</a> files.<br />
It has also several fixes included (including <em>Oracle</em> direct link), and
the internal <em>SQLite3</em> engine has been updated to its latest
revision.</p>
<div>The <a href="http://synopse.info/files/SynDBExplorer.zip">executable
download link content</a> has been updated with its latest version.<br />
<p>And of course, all source code is available in <a href="http://synopse.info/fossil/dir?name=SQLite3/Samples/12+-+SynDB+Explorer">our
source code repository</a>.</p>
</div>
<p>Feedback is <a href="http://synopse.info/forum/viewtopic.php?id=756">welcome
on our forum</a>!</p>SynDBExplorer fast direct exporturn:md5:241a21708ff87d206c47f6301173ad5a2012-01-17T23:02:00+01:002012-06-23T11:15:33+02:00AB4327-GANDIOpen Source librariesblogDatabaseDelphiOleDBOracleperformanceSourceSQLSQLite3SynDBSynDBExplorer<p style="margin-top: 0;">The Open Source <a href="http://blog.synopse.info/post/2011/07/22/SynDBSQLite3%3A-SQLite3-direct-access">SynDBExplorer</a> tool
has been enhanced these days.</p>
<p>Main new features are:</p>
<ul>
<li>Execution of the current selected text (if any) instead of the whole memo
content;</li>
<li>"<em>Exec & Export</em>" new button, for direct export to file.</li>
</ul>
<div>I really like the <em>selection execution</em> feature - this speed up SQL
process a lot, and allow to switch from one statement to another.<br />
And the new exporting features are opening new possibilities.</div> <p>In particular, the "<em>Exec & Export</em>" new button is able to
export any SQL statement result into:</p>
<div>
<ul>
<li>Regular CSV file (ready to be imported in Excel);</li>
<li>Regular TXT file (columns are separated with a tab character);</li>
<li>Standard JSON content (ready for any AJAX application);</li>
<li>Our more compact JSON format (column names are not duplicated on each
row);</li>
<li><em>Synopse BigTable</em> records (with fixed sized records for integers
columns);</li>
<li><em>Synopse BigTable</em> records (with variable-length records for
integers columns);</li>
<li>A <em>SQLite3</em> database file (creating a table with all necessary
columns).</li>
</ul>
<p>This direct export feature won't use a temporary memory buffer, just with
the regular "<em>Export</em>" button, which exports the grid on screen.<br />
Therefore, it will be faster, and will be able to handle any amount of data.
You can export gigabytes of data directly from the tool, then consume it
offline as regular <em>SQLite3</em> tables or very efficient <a href="https://blog.synopse.info?post/post/2011/01/22/Synopse-Big-Table-1.12a"><em>Synopse
BigTable</em> records</a>. With a map & reduce architecture, it can be
very handy to process huge amount of data. Or it can be used to convert an
existing database layout into a <em>SQLite3</em> database<em>.</em></p>
</div>
<p>Thanks to these enhancements, this little tool begins to be a very nice SQL
and database management tool.</p>
<p>All this is made possible due to the unique architecture of our
<em>SynDB</em> classes. Their design appears to be powerful and open, at the
same time: it allows <a href="http://synopse.info/fossil/fdiff?v1=63e5ce5a727fb868&v2=c7d227d26119b35a">easy
mix of very diverse data structures like <em>BigTable</em> or
<em>SQLite3</em></a>, far away from the RAD layout.</p>
<p>A visual <em>Query Designer</em> is also in alpha stage (select several
tables, then right click on your mouse), and will become a powerful entry point
for the <a href="https://blog.synopse.info?post/post/2011/06/09/Close-future-of-the-framework%3A-database-agnosticism">database
reverse engineering</a> of our <em>mORMot</em> framework (i.e. the upcoming
<code>TSQLRecordMapped*</code> classes). Up to now, you can JOIN visualy your
tables with this right-click approach. But it will shortly become a powerful
way of managing any RDBMS as objects.</p>
<p>Feedback and comments are <a href="http://synopse.info/forum/viewtopic.php?id=585">welcome in our forum</a>.</p>SynDBSQLite3: SQLite3 direct accessurn:md5:90778ed81c0f8acacf468e5be3463aa62011-07-22T17:14:00+02:002012-08-17T17:40:48+02:00AB4327-GANDIOpen Source librariesblogDatabaseDelphiOleDBOracleSQLSQLite3SynDBSynDBExplorerUnicode<p>For our ORM framework, we implemented an efficient <em>SQLite3</em> wrapper,
joining statically (i.e. without any external dll) the SQLite3 engine to the
executable. SQLite3 is in fact used as the DB kernel of the framework. For
instance, thanks to its unique <a href="https://blog.synopse.info?post/post/2011/05/14/Virtual-Tables-in-the-SQLite3-framework">virtual table
mechanism</a>, even tables in other databases (like Oracle or MSSQL) are
available <a href="https://blog.synopse.info?post/post/2011/06/09/Close-future-of-the-framework%3A-database-agnosticism">as if
they were SQLite3 tables</a>.</p>
<p>We just made this wrapper independent from our ORM, in a new dedicated unit,
<a href="http://synopse.info/fossil/finfo?name=SynSQLite3.pas">named
<code>SynSQLite3.pas</code></a>.</p>
<p>It was an easy task to let this unit be called from our <em>SynDB</em>
database abstract classes.</p> <p>That is, you'll have a generic class hierarchy, able to access:</p>
<ul>
<li>Any <a href="https://blog.synopse.info?post/post/2011/06/27/SynOleDB%3A-OpenSource-Unit-for-direct-access-to-any-database-via-OleDB">
<em>OleDB</em> providers</a> (including latest version of MSSQL, or good old
Access/Jet); </li>
<li>Any <a href="https://blog.synopse.info?post/post/2012/02/29/Microsoft-states%3A-OleDB-out-enjoy-ODBC%21">ODBC
provider</a> (including e.g. MySQL / FireBird);</li>
<li>Oracle via <a href="https://blog.synopse.info?post/post/2011/07/09/SynDBOracle%3A-Open-Source-native-Oracle-access">direct
access to the OCI layer</a>; </li>
<li>and now <em>SQLite3</em>, statically embedded to the executable.</li>
</ul>
<p>As a result, the <em>SynDBExplorer</em> sample tool is now able to select a
SQLite3 database as source, in addition to the <em>OleDB, ODBC</em> or
<em>Oracle/OCI</em> database types. </p>
<p><a href="https://blog.synopse.info?post/public/SynDBExplorerSQLite3.png"><img src="https://blog.synopse.info?post/public/.SynDBExplorerSQLite3_m.jpg" alt="" title="SynDBExplorerSQLite3.png, juil. 2011" /></a><br />
<br />
Select "<em>Sqlite3</em>" as the database type when creating a new connection,
and specify the full SQLite3 database as "Server name". Other properties
(Database name, User ID, Password) are just ignored. </p>
<p><a href="https://blog.synopse.info?post/public/SynDBExplorer.png"><img src="https://blog.synopse.info?post/public/.SynDBExplorer_m.jpg" alt="" title="SynDBExplorer.png, juil. 2011" /></a></p>
<p>The <em>SynDBExplorer</em> tool is a multi-tabbed software able to list
all tables, and also column names and types - including indexes. It can also
export all result content into csv, json, text or pdf files. And you have some
popup-menus to help you enter field names or values into your SQL
statement.<br />
You can download this free tool from this <a href="http://synopse.info/files/SynDBExplorer.zip">direct link</a>.</p>
<p>As you can see in the <a href="http://synopse.info/fossil/finfo?name=SynDBSQLite3.pas">SynSQLite3.pas source
code</a>, adding such another Database to the SynDB classes model is just a
very thin layer, calling the new shared <em>SynSQLite3</em> unit.</p>
<p>Now I will work on integrating all those <em>SynDB</em> classes into
our main framework, which is now named <em>mORMot</em> (at least in the source
code).</p>
<p>Feedback and comments are <a href="http://synopse.info/forum/viewtopic.php?pid=2370#p2370">welcome on our
forum</a>.</p>