Synopse

A.Bouchez — Sun, 01 May 2011 12:00:00 +0000

About the Synopse Informatique company and our Open Source approach.

If your project needs Delphi software development, it's the right place to search.

About Synopse

"Synopse Informatique" (aka Synopse) is a French software development firm founded by Arnaud Bouchez, specialized in providing custom database and search engines, written in the object pascal language.

"Synopse Informatique" in french translates more or less in "Synopsis Software"... here "Synopsis" doesn't mean a brief summary, but comes from late latin, itself from Ancient Greek σύνοψις (sunopsis), i.e. σύν (sun, “with”) + ὄψις (opsis, “view”). It's commonly used to describe a book displaying some similar documents in parallel, e.g. the three synoptic gospels. Using such a concept for a software company indicates a global approach, a kwow-how and proven experience used to create innovative solutions. By having a global vision, using agile methods and mastering software advanced technologies, Synopse is able to create the software you need, not what computer or marketing specialists think of, and, sometimes, more than you ever expected.

Synopse projetcs

Synopse is proud to release some Delphi/Object pascal projects and tools as Open Source:

Enhanced Delphi Run Time library
The SQLite3 framework (full SQLite3-based JSON client/server framework and UI tools)
The SynProject automated documentation and code versioning tool
The Synopse database engine

The Synopse database engine

Synopse is releasing a new open source database engine, founded on some successfull projects like Ictus Win or ROC. It's a whole new database engine, not SQL-based, but purely object/XML related.

The Synopse company is able to develop business rules and tuned solutions using the Synopse database engine. This software development can be made private by an appropriate contract, but will rely on the proven Synopse database engine.

Open Source model

Synopse does believe in the Open Source approach:

Open source is a development method for software that harnesses the power of distributed peer review and transparency of process. The promise of open source is better quality, higher reliability, more flexibility, lower cost, and an end to predatory vendor lock-in.

Therefore, the company will provide some usefull components they use, from User Interface to communication layers, under a MPL/GPL/LGPL tri-license to the Delphi community.

The fully featured Synopse database engine itself is published under a dual licence model, similar to the one used by Sun™ for MySQL™: a commercial licence for OEMs, ISVs, VARs and other distributors of commercial applications; and a GPL licence for open source projects (FOSS).

Synopse Blog

We decided to use this simple blog site as a major place to inform the community about our projects and open source initiative.

Stay tuned on this web site!

Freelance Programming

Our company is able to help you in whatever Delphi project. We may help with Delphi expertise, corrective or preventive maintenance, customization of one of our libraries according to your purposes.

You can contact us at freelance at synopse dot info

Synopse SQLite3 Framework 1.9.1

A.Bouchez — Tue, 24 Aug 2010 11:16:00 +0200

The Synopse SQLite3 Database Framework was just released under version 1.9.1:
- internal SQLite3 database engine is updated to version 3.7.2;
- new TSQLRecordFTS3 record, for using FTS3 virtual tables, i.e. implementing full-text search;
- new SQLite3UIEdit unit, to edit table content with a dialog created from RTTI;
- new dedicated BLOB methods and JSON array serialization;
- a lot of fixes and speed enhancements (including our HTTP/1.1 RESTful server now using Thread Pool).

The new 3.7.2 version of the SQLite3 engine, which is mandatory according to SQLite3's authors, is included.

Changes in SynCommons.pas unit:
- now compiles under CrossKylix, and tested under Linux
- new JSONEncodeArray procedures, to create JSON array content from supplied Delphi arrays (handle RawUTF8 text, double or integer arrays)
- new AddCSV methods in TTextWriter handling Delphi arrays to be added as Comma-Separated-Values (handle RawUTF8 text, double or integer arrays)
- new definition of PtrInt/PtrUInt, to match NativeInt/NativeUInt types, available since Delphi 2007 - some code rewrite in order to avoid any implicit conversion from/to integer/cardinal

Changes in SynCrtSock.pas unit:
- avoid some GPF during client deconnection when the server shut down
- rewrite HTTP Server handle request loop keep alive timing
- HTTP Server now use a Thread Pool to speed up multi-connections: this speed up a lot HTTP/1.0 requests, by creating a Thread only if necessary

Changes in SynCrypto.pas unit:
- now use direct Windows threads, since we don't need any exception handling nor memory usage inside the AES encryption Thread handler
-> avoid classes.TThread and system.BeginThread() use
-> application is still "officialy" mono-threaded (i.e. IsMultiThread=false), for faster System.pas and FastMM4 (prevent CPU locking - see http://synopse.info/forum/viewtopic.php?id=57 about Delphi & multi-core)
- some other minor fixes and enhancements

Changes in SynZip.pas unit:
- crc32 is now coded in inlined fast asm (crc32.obj is no longer necessary)
- crc32 hashing is performed using 8 tables, for better CPU pipelining and faster execution
- crc32 tables are created on the fly during unit initialization, therefore save 8 KB of code size from standard crc32.obj, with no speed penalty

Changes in SynSelfTests.pas unit:
- test multi-threaded AES encryption/decryption of 4 MB blocks
- added crc32 tests

Changes in SynGdiPlus.pas unit:
- small enhancements for framework Main Demo release (RectNotBiggerThan and MaxPixelsForBiggestSide parameter in SaveAs function e.g.)

Changes in SQLite3.pas unit:
- database engine updated to 3.7.1
- fix issue in TSQLRestServerDB.CreateMissingTables when an exception occured
- handle now RowID as a valid alias to the ID field (needed for TSQLRecordFTS3)
- handle TSQLRecordFTS3 record, for using FTS3 virtual tables, i.e. implementing full-text search (including dedicated regression tests)

Changes in SQLite3Commons.pas unit:
- update engine to version 3.7.1
- fixed error calling a TSQLRestServerCallBack with both record & parameters
- TSQLRecordSigned must now be signed with a specified content (content was formerly a fixed field of type RawUTF8, but it didn't apply in all cases, e.g. if content field is defined as TSQLRawBlob)
- new TSQLRestClientURI.ForceBlobTransfert property which enable to get and set BLOB fields values with usual Add/Update/Retrieve methods
- new TSQLRestClientURI.RetrieveBlobFields/UpdateBlobFields methods for retrieving/updating all BLOB fields of a record at once
- better handling of sftID in the User Interface and database use (e.g. creates a dedicated index for the TSQLRecord published fields)
- some code rewrite in order to avoid any implicit conversion from/to integer/cardinal after new definition of PtrInt/PtrUInt (matching NativeInt/NativeUInt types, available since Delphi 2007)
- updated TSQLRibbonTabParameters object, with some new fields dedicated to the automatic edition of records, via the new SQLite3UIEdit unit
- new sftSet SQL field kind, handling a TSQLRecord published property with a set of enumeration as Delphi type (stored as bit-mapped INTEGER)
- handle now RowID as a valid alias to the ID field (needed for TSQLRecordFTS3)
- defines a new TSQLRecordFTS3 type, for defining a FTS3 virtual table, i.e. implementing full-text search

Changes in SQLite3UILogin.pas unit:
- improved Delphi 2009/2010 UnicodeString compatibility
- new PassWord method to enter a password e.g. for Cypher (i.e. with no UserName)

Changes in SQLite3ToolBar.pas unit:
- improved Delphi 2009/2010 UnicodeString compatibility
- add TMS components stylers coherency update
- added TSQLRibbon to factorize most used Ribbon-related User Interface data, functions and events in one class

Changes in SQLite3UIQuery.pas unit:
- custom queries can now handle standard operators, following a specified set
- TSQLQueryEvent() usage has therefore been modified for custom queries

Changes in SQLite3UI.pas unit:
- improved Delphi 2009/2010 UnicodeString compatibility
- fix some issues, and complete implementation of marking from time elapsed

New SQLite3UIEdit.pas unit:
- Initial Release, handling most common kind of SQL field (sftRecord, sftTimeLog, sftCurrency, sftDateTime, sftFloat and sftBlob) are not handled yet, because is not needed; perhaps sftTimeLog, sftCurrency, sftDateTime and sftFloat should be handled in the future (using TDateTimePicker or a to be written TSynExtendedLabeledEdit components)
- all user interface (fields, layout, etc...) is created from RTTI data and some custom parameters

Changes in SQLite3Pages.pas unit:
- new AppendRichEdit method to draw RichEdit content
- new WordWrapLeftCols property used to optionaly word wrap caLeft columns into multiple lines, i.e. if the text is wider than the column width, its content is wrapped to the next line (set to false by default) - this also will handle #13/#10 in column text as a "go to next line" command

Changes in SQLite3i18n.pas unit:
- new TLanguageFile.BooleanToString method, returning 'No' or 'Yes'
- new TLanguageFile.PropToString method to convert a TSQLRecord published property value into ready to be displayed text

Changes in SQLite3UIOptions.pas unit:
- some code refactoring to share code with the new SQLite3UIEdit unit
- minor fixes and enhancements

Last minute update:
Main version updated to 1.9.1:

We had to update the SQLite3 internal engine to version 3.7.2: an obscure but very old bug makes SQLite authors recommend to use 3.7.2 for all new development.
Upgrading from all prior SQLite versions is also recommended. See http://sqlite.org/releaselog/3_7_2.html

So we do. And increase our version number to 1.9.1.

Don't forget to consult the draft version of the documentation of this framework!

The full source code from the framework is available to download from SynopseSQLite3.zip licensed under a MPL/GPL/LGPL tri-license.
Compatible with Delphi 7 to Delphi 2010.

Comments and feedback are welcome on our forum:
http://synopse.info/forum/viewtopic.php?id=96

How to implement multi-tier architecture in our SQLite3 Framework

A.Bouchez — Thu, 19 Aug 2010 10:48:00 +0200

In software engineering, multi-tier architecture (often referred to as n-tier architecture) is a client–server architecture in which the presentation, the application processing, and the data management are logically separate processes. For example, an application that uses middleware to service data requests between a user and a database employs multi-tier architecture. The most widespread use of multi-tier architecture is the three-tier architecture.

Both ORM and RESTful aspects of our framework makes it easy to develop using such a three-tier architecture.

In our SQLite3 Open Source framework, for Delphi 7-2010, you can follow this development pattern:
- Data Tier is either SQLite3 and/or an internal very fast in-memory database, most SQL queries are created on the fly;
- Logic Tier is performed by pure ORM aspect: you write Delphi classes which are mapped by the Data Tier into the database, and you can write your business logic in both Client or Server side, just by adding some events or methods to the classes;
- Presentation Tier is either a Delphi Client, either an AJAX application, because the framework can communicate using RESTful JSON over HTTP/1.1 (the Delphi Client User Interface is generated from Code, by using RTTI and structures, not as a RAD - and the Ajax applications need to be written by using your own tools and JavaScript framework, there is no "official" Ajax framework used yet).

If you need a RAD approach, it's perhaps not the right tool for you.
But if you like describing your application with classes, it's worth taking a look at it, and contributes to this open source project!

We can give you more details about each Tier/Layer in our forum.

FPC and Delphi: toward a "fratricidal war"?

A.Bouchez — Sat, 14 Aug 2010 10:28:00 +0200

Speaking about [upcoming] 64 bit Delphi compiler, I did forecast they will surely release their compiler 5 years later than FPC's... hoping they will follow the FPC way, just as Intel did about 64 bits instructions, against AMD.

What I was referring to is that AMD was the first to publish a 64 bit x86 CPU, with a new instruction set. Intel released somewhat later their own 64 bit x86 CPU, following the same instruction set.
There are some variants between AMD and Intel CPU, some dedicated instructions or implementations.
But most instruction set is perfectly compatible.

And I suspect the more it gets, the less compatible FPC and Delphi will be.

The main difference between Delphi and FPC is that the first is a business company project (Embarcadero), the second an Open Source solution. They are not two billions dollars companies fighting each other, with some state-level institutions watching about any unfair competition or monopoly... FPC compiler is technically a great and robust project, but EMB follows business rules. I don't think anyone will play "Anti-monopoly" in favor of FPC.

FPC and 64 bit: the future is today

FPC published a 64 bit compiler using the Delphi/object pascal language in 2006.

It could have been fair and less confusing that Delphi engineers just follow the same patterns, for example use PtrInt and PtrUInt.

I guess that in 2006, the 64 bit Delphi compiler was already a R&D project at Borland, (and Codegear since February).
That's why NativeUInt and NativeInt were made available in Delphi 2007.
But FPC published its 64 bit compiler in 2006... and EMB is not able to promise such a working x86-64 compiler before end of 2011.

The preview 64-bit compiler is still indicated with a "first half of 2011" like it was and future versions should have both 64-bit and Mac support, although from the slides it is not clear which features actually go into Pulsar ("Introduction of 64-bit and cross-platform to the RAD Studio product line"), Wheelhouse ("Extending support for 64-bit and additional platforms"), and Commodore ("Full support for 64-bit compilers for Windows, Mac OS, and Linux"). Part of the differentiators comes from the fact Delphi language will be there before C++, according to come of the details.
See http://edn.embarcadero.com/article/39934

Some kind of messy

About the IDE, RadStudio is much more advanced than Lazarus. But there are some newcomers around, like mseide, which is light and powerful.

Newest Delphi compiler have some language enhancements (like generics and attributes, without naming Unicode which is mostly VCL-related), but FPC evolves (it includes generics and in the future clever property attributes).
But the more it goes, the less compatible.
If you want your code to compile with both Delphi and FPC, you need a lot of {$ifdef} and/or custom types... and a good knowledge of every compiler.

That's quite a mess, and a pity!

Sometimes, I miss a "standardized language", like C#, even if the libraries are not standardized (if you want to work cross-platform, e.g. using Mono, you can't rely simply on latest Microsoft libraries).

Will there be a "fratricidal war"?

If such a war happens (but isn't it already there?) in the object pascal world, we can worry.
My first reaction is to think about "poor little FPC". And imagine me at the compiler graveyard, praying on its tomb.
But after some deeper thinking, it's not so simple.
I'm not able to see which one will be the winner.

First of all, FPC is Open Source: it can't dye, if people have interest in it.
And because of object pascal language strength, even a small team of coders can handle it.

At a glance, Delphi seems more powerful than FPC, because of EMB. But it could be exactly the contrary...
Remember the Delphi for DotNet compiler: it was replaced by Oxygene, a not EMB product, but a more reactive product.

Since FPC is written in pure object pascal code (and the Delphi compiler isn't, by the way), it could be easy for EMB to embed the FPC compiler into their RadStudio XE... And by using wxforms, extending it with DB components, RadStudio could easily cross-compile to both Linux and MacOsX.

So will the Delphi for Win32 compiler dye, just as the Delphi for DotNet did?
Isn't FPC at the right place, at the right moment?

Feedback and comments are welcome on our forum!

Writing Delphi code for 64 bits compiler

A.Bouchez — Tue, 10 Aug 2010 09:19:00 +0200

There will be an upcoming 64 bits Delphi compiler. Embarcadero promised it.

Florian (the architect of FPC) showed a first "Hello world" program for Win64 in March 2006.
This was remarkable since GCC and the binutils don't even support this target at this time.
In fact, FPC used its Internal linker on Win32 and Win64 platforms, just like Delphi does.

Here are some points on how you could make your code ready to compile under FPC 64 bits, therefore (I hope) under future Delphi 64 bits compiler.

Under FPC, all common types (like integer, cardinal, etc...) have the same size. They are all 32 bits.

So most of you code don't need to be rewritten. But if you have some asm part in your code, you'll have to write it in pascal.

Use the PUREPASCAL conditional, and always provide a pure pascal version of any asm function. It's therefore a good practice to always code in pascal, then add an optimized asm version only if it's necessary.
Here is one example extracted from our framework:

function IdemPropName(const P1,P2: shortstring): boolean; overload;

{$ifdef PUREPASCAL}

var i: integer;

begin

  result := false;

  if P1[0]<>P2[0] then

    exit;

  for i := 1 to ord(P1[0]) do

    if (ord(P1[i]) xor ord(P2[i])) and $DF<>0 then

      exit;

  result := true;

end;

{$else}

asm // eax=P1 edx=P2

    mov cl,[eax]

    cmp cl,[edx]

    jne @z      // length differs

    or cl,cl

@1: lea eax,[eax+1]

    lea edx,[edx+1]

    jz @ok

    mov ch,[eax]

    xor ch,[edx]

    and ch,$DF   // case insensitive compare

    jne @z

    dec cl

    jmp @1

@ok:mov al,1

    ret

@z: xor eax,eax

end;

{$endif}

If you use pointer arithmetic, you should use PtrInt instead of integer to typecast pointers.
See http://wiki.freepascal.org/Multiplatform_Programming_Guide#32.2F64_bit It's perfectly cross-compiler. We defined these types for working with Delphi and FPC on the same code:

{$ifdef FPC}

  {$ifdef CPU64}

    {$define PUREPASCAL}

  {$endif}

{$else}

type

  /// a CPU-dependent unsigned integer type cast of a pointer / register

  // - used for 64 bits compatibility, native under Free Pascal Compiler

  PtrUInt = cardinal;

  /// a CPU-dependent unsigned integer type cast of a pointer of pointer

  // - used for 64 bits compatibility, native under Free Pascal Compiler

  PPtrUInt = ^PtrUInt;



  /// a CPU-dependent signed integer type cast of a pointer / register

  // - used for 64 bits compatibility, native under Free Pascal Compiler

  PtrInt = integer;

  /// a CPU-dependent signed integer type cast of a pointer of pointer

  // - used for 64 bits compatibility, native under Free Pascal Compiler

  PPtrInt = ^PtrInt;



  /// unsigned Int64 doesn't exist under Delphi, but is defined in FPC

  QWord = Int64;

{$endif}

So it's always a good practice to use these PtrInt instead of integer in any pointer arithmetic, just as you should use AnsiString/AnsiChar/PAnsiChar for any ansi content, even on Delphi 7.
And if you use loop variables like for i := 0 to Lines.Count-1, you should better use var i: PtrInt instead of i: integer

Note that there is a CPU64 define set if you're compiling for a 64 bits CPU.

For example, since PtrInt = Int64 in fact, you can test {$ifdef CPU64} to avoid writing a Int64 specific function, if a PtrInt one has already been defined.

I just hope that EMB will follow the same rules for their future 64 bits compiler.

They will be 5 years late... I hope they will follow the FPC way, just as Intel did about 64 bits instructions, against AMD.

As always, feedback and comments are welcome on our forum.

Every Delphi DB program contains hidden code for EMB license checking

A.Bouchez — Sat, 07 Aug 2010 17:16:00 +0200

After looking further at the SysUtils unit, I found out some small piece of asm code I didn't notice at first.

There is some hidden and not documented code in every program using the DB features, in order to verify that you have the right to use some drivers or DB features.

The license checking performed by Borland (it was a trick existing in Delphi 4, or perhaps before) is somewhat childish.

So here are the three function procedures in SysUtils.pas, not documented:

function GDAL: LongWord;

procedure RCS;

procedure RPR;

Here is a first explanation: http://assarbad.net/stuff/!export/dvclal/jeremyres.html

The purpose of these functions is to check the presence of the DVCLAL resource in the exe (or one of the current module).

This resource is just an array of 4 longword, i.e. 4 cardinal values.

The function GDAL, for example, returns either 0, -16, -5136, or -1 depending of the version of your current Delphi license (i.e. Professional, Entreprise, Architecture, GPL, whatever). The returned value of GDAL is checked in the following units: SqlExpr.pas BDEReg.pas DBReg.pas DBTables.pas BDEReg.pas DBReg.pas MidReg.pas

And the protection takes place easily, for example here is an extract of the BDEReg.pas code:

procedure Register;
begin
  { Database Components are excluded from the STD SKU }
  if GDAL <> LongWord(-16) then
  begin
    // Restrict these components to only be used with VCL components.
    GroupDescendentsWith(TBDEDataSet, Controls.TControl);

....

    { Components that are excluded from the STD & PRO SKUs }
    if GDAL = 0 then
      RegisterComponents(srBDE, [TNestedTable]);

The RPR procedure is a bit more agressive: it raises an exception in case you don't have the right to use some components registered in DBXClient.pas DbxFirebird.pas DbxInterbase.pas DbxMySql.pas DbxSybaseASA.pas DSServer.pas DB.pas dbcgrids.pas HTTPApp.pas HTTPProd.pas Sockets.pas SOAPConn.pas units.

And within the RCS procedure, the exception is raised for other components registered in DBXClient.pas DbxDb2.pas DbxInformix.pas DbxMSSQL.pas DbxOracle.pas DbxSybaseASE.pas DBClient.pas MConnect.pas TConnect.pas HTTPApp.pas HTTPProd.pas MXDB.PAS MXGRAPH.PAS MXGRID.PAS MXPIVSRC.PAS units.

I let you guess which component is what, just from the official EMB feature matrix.

Why all this?

Since the raw compiler is the same, that is the language feature and RTL is the same for all Delphi versions (from Professional up to Architect), you could be tempted buying the Professional version, then copy the dcu/bpl from an Architect, and use it... but in fact, you won't have the "magic" resource in your exe, so the trick won't play.

Writing a patcher for your exe is very easy. You don't even have to patch the memory, just some standard Win32 API like BeginUpdateResource / UpdateResource / EndUpdateResource like stated in msdn will do the work.
I won't commit Delphi code here, because I want to stay friend with EMB. But guess what, it's not a great protection system!

I don't blame Borland/EMB. But lol ! Such a weak "protection", giving all the source code at once. Thanks EMB for making me smile after my "object type" funeral.

Comments and more source code in our forum, just as usual.

Save object, stop class hegemony!

A.Bouchez — Fri, 06 Aug 2010 11:31:00 +0200

In a recent thread in the Embarcadero Discussion Forums, Vedran Vuk posted some questions about object keyword.
His purpose was to use object instead of class to improve speed and memory consumption:

I do use classes. I just want to use objects on smaller classes that don't really need initialization or RTTI. Plus, I can directly operate on it like a record with no need for constructors and it can be sealed and has inheritance. Every byte matters in this case.

Rudy posted on the EMB forum that

The "object" type is deprecated. As was said, it mainly exists for compatibility with old Turbo Pascal. That is why it is not documented very well. It's use is not promoted.

I have the same requirement sometimes, for example for our framework or for low-level units.

I do like such plain-old object type, and don't want to see this object feature marked as deprecated in future EMB versions.

Delphi has always had three kind of coders:

high-level coders, using components and RAD, standard library and its classes;
low-level coders, using pure code SW development, looking at the asm code generated and tracking down memory and CPU usage;
and to be honest, all low-level coders use the high-level tools, because for a simple form, RAD is great, and standard library is to be used anyway.

In fact, the object type is one feature which low-level coders like a lot.
For example, the whole KOL library relies and it, and performs a great job with it. The XMLRad framework uses it a lot for all basic types, in order to increase speed, and improve multi-threaded server response time.

Here are some cases I use object for:

a memory mapped file, which I want to parse very quickly: a pointer to such an object is just great, and you still have methods at hand; I use this for TFileHeader or TFileInfo which map the .zip header, in SynZip.pas;
a Win32 structure, as defined by a API call, in which I put handy methods;
a temporary structure defined on the stack, just used during a procedure: I use this for TZStream in SynZip.pas, or for our RTTI related classes, which map the Delphi generated RTTI in an Object-Oriented way not as the TypeInfo which is function/procedure oriented. By mapping the RTTI memory content directly, our code is faster than using the new RTTI classes created on the heap. We don't instanciate any memory, which, for an ORM framework like ours, is good for its speed. We need a lot of RTTI info, but we need it quick, we need it directly.

In fact, I discovered that new Delphi compiler doesn't like packed object.
See http://qc.embarcadero.com/wc/qcmain.aspx?d=79792
I've an Internal Error DT5830 under Delphi 2009 Update 3, but previous compiler versions (including Delphi 2009 Update 2) did accept a packed object declaration.
There is a simple workaround: define it as packed record... this works, but is not compatible with previous version of the compiler.
So here is how such a packed object must be coded, if you want your code to work on most Delphi compiler versions:

type

{$A-} { force packed object (not allowed under Delphi 2009) }

  /// the internal memory structure as expected by the ZLib library

  TZStream = {$ifdef UNICODE}record{$else}object{$endif}

...

It's a pity that Embarcadero seems not to a have a full regression test of all compiler/language features, and run it before every release, like Free Pascal compiler does.

It's a pity that EMB changed the record type, by allowing adding methods on it. IMHO, it's exactly a no-feature. Noone asked for it, or for any good reason. If you want a "pure object pascal language", with no history, just created from scratch, with nice high-level features, just use PRISM.

So here is my today's claim:

Don't forget low-level Delphi coders! Save our object type!

Feedback and discussion is welcome on our forum.

Synopse SQLite3 Framework Main Demo - draft

A.Bouchez — Wed, 04 Aug 2010 00:48:00 +0200

The SQLite3 MainDemo source code has just been uploaded to the repository.

This application is just a database tool which stores text content and files into the database, in both clear and "safe" manner. Safe records are stored using AES-256/SHA-256 encryption. There is an Audit Trail table for tracking the changes.

Here are some screen shots of this sample application:
http://synopse.info/files/pictures/synfilebeta.png
http://synopse.info/files/pictures/synfilebeta1.png
http://synopse.info/files/pictures/synfilebeta2.png

It shows you some interesting features like integrated User Interface generation and i18n, reporting and export to PDF, client/server services.

And you'll discover how our ORM and client/server database system is implemented: all is done in code (no RAD/IDE approach here), but with very little code lines count...

An executable (still in draft status) can be downloaded from http://synopse.info/files/samples/synfile.zip.

More details and feedback in our dedicated forum page.

Compiler enhancement proposal: threadlocalvar

A.Bouchez — Fri, 30 Jul 2010 11:25:00 +0200

As I wrote in a previous post, Delphi string, dynamic array and memory manager don't like multi-core CPU.

My proposal is to add a threadlocalvar keyword, to be used instead of var in your code, to mark some variables to be used in only the current thread. Then the compiler and RTL won't have to use the LOCK instruction, and the application will be MUCH faster in multi-thread environment.

I made this proposal in both the Embarcadero forum and in the FreePascal forum.

And what about using a threadlocalvar new reserved word, in a threadvar way, which could define the variable (string, dynamic array) to be accessed by the current thread only, and won't have any LOCK call?

My guess is that any server thread uses mostly such local variables (e.g. for string concatenation of XML or HTML stream), and don't need the thread safe approach most of the time.

It could lead into some confusion for less advanced Delphi users, but the threadvar itself does exist for years, but is used only by those how need this feature. If you need it, you use it. Otherwise you ignore it, and your application will be slower in some cases, but wil work. For some CPU intensive applications, like multi-thread servers, it should be a great improvement to have such a threadlocalvar at hand.

Note 1: the threadvar implements one variable instance per thread, and the threadlocalvar should implements one variable instance for the current thread only.

Note 2: any threadlocalvar could be assigned to a normal var, when a multi-thread safe variable is needed (e.g. for communication between threads): in this case, it will be a direct write, not a copy on write. Copy on write would be enabled between threadlocalvar, of course, for better performance.

Note 3: it's up to the programmer to take care of the multi-thread approach, but writing threadlocalvar (so many characters to type!) will prevent for doing it without knowing it.

This could be something like that:

function TServer.ComputePage: string; 
threadlocalvar
  tmp: string; // differs from var tmp: string 
  i: integer;  // i is defined as threadlocalvar, but is the same as normal var 
begin
  for i := 0 to high(ListStr) do // ListStr[] is a dynamic array
   tmp := tmp+ListStr[i]+#13#10; // very fast computation, without LOCK
  result := tmp; // copy from local heap to global heap 
end;

To implement this at the compiler and RTL level, we could use reference count <=-2 for these variables (-1=const, -2=ref 0, -3=ref 1..).

The RTL (i.e. system.pas unit) should be modified as such, for example for handling a threadlocalvar string reference counting add:

function _LStrAddRef(var str): Pointer;
var P: PStrRec;
begin
  if Integer(str)<>0 then begin
    P := Pointer(Integer(str) - sizeof(StrRec));
    if P.refcnt >= 0 then
      InterlockedIncrement(P.refcnt) else // slow multi-thread safe reference count
    if P.refcnt < -1 then // -1 for const
      dec(P.refcnt);      // -2,-3... for threadlocalvar
  end;
  Result := Pointer(str);
end;

And a local threadheap should be implemented for such threadlocalvar, with threadlocalgetmem() and such functions.

Another possibility should be do add a "threadlocal" attribute for types, to be used for local variables and properties:

TServer = class
public
  ListStr: threadlocal array of string;
...

In this case, the method above should begin with:

function TServer.ComputePage: string;
var
  tmp: threadlocal string; // differs from var tmp: string
....

But this syntax sound a bit not "pascalish", whereas the threadlocalvar does (it sounds like the treadvar feature) ... another idea?

Comments and ideas are welcome on our forum.

See http://synopse.info/forum/viewtopic.php?pid=305

Hyphenation in Delphi

A.Bouchez — Tue, 27 Jul 2010 10:32:00 +0200

We made some years ago a Delphi unit for hyphenation of text.

Such an unit, together with our SynPdf library for example, could be useful in your application. So we released the source!

What it hyphenation?

As stated by wikipedia:

The hyphen ( ‐ ) is a punctuation mark used to join words and to separate syllables of a single word. It is often confused with dashes ( –, —, ― ), which are longer and have different uses, and with the minus sign ( − ) which is also longer. The use of hyphens is called hyphenation.

Our engine does rely on Libhnj, more exactly on the version enhanced for use in Open Office.

See http://lingucomponent.openoffice.org/hyphenator.html

You can download the source code from http://synopse.info/files/hyphen.zip

There are english, french, italian and spanish hyphen tables included (as pure delphi .res) in this release. But you can use official Open Office tables if you want to include any other language.

No external dll is necessary, you can hyphen any word directly in your Delphi project.

It was developed and tested under Delphi 7 and Kylix 3, under both Windows and Linux. If you need it, I could make a version compatible up to Delphi 2010.

Source code released as dual licensed under LGPL and MPL, like the original Libhnj library, so you can use it in both GPL and commercial products.

This is a first release, as draft. If you find it useful, we may enhance and document it.

Comments and questions are welcome on our forum!

Damnatio Memoriae

A.Bouchez — Sat, 24 Jul 2010 18:50:00 +0200

You sure heard about the "CrossKylixGate"... A Great tool I used for years (CrossKylix) has just been updated. Its author (named Simon) posted an announcement in the Embarcadero forums. Then the post has been deleted by "Team B" members, because of some old story.

But not only Simon has been censured. Since I like and use this free tool, I posted a very "soft" post just with the announcement of the CrossKylix update, with some explanations and precisions. No debate. Just a link to the download page. Then my whole thread was deleted.

It just sounds like a "damnatio memoriae" to me.

From wikipedia: "Damnatio memoriae is the Latin phrase literally meaning "damnation of memory" in the sense of removal from remembrance. It was a form of dishonor that could be passed by the Roman Senate upon traitors or others who brought discredit to the Roman State."

Are we back to the tyrany of the Caesars in old Rome?
Will Simon feed the lions?
I hope not, at least because of CrossKylix, I'll thumbs up and pray for mercy!

The worse in this case is that it's not the Senate (i.e. Embarcadero) who made the decision, but some "Team B" zealous and proud anonymous.
Thumbs down for him!

"Iste asinus hominis currit; ergo illius hominis asinus currit"... I let you guess who is the donkey, and who is the man!

Comments are welcome on our forum.

Unit Testing light in Delphi

A.Bouchez — Fri, 23 Jul 2010 15:35:00 +0200

Automated Unit Testing is a great improvement in coding safe applications.

If you don't know about it, visit http://xprogramming.com/index.php then come back here, and you'll discover how we implement unit testing in a KISS way, in pure Delphi code.

What about automated testing?

You know that testing is (almost) everything if you want to avoid regression problems in your application.

How can you be confident that any change made to your software code won't create any error in other part of the software?

So automated unit testing is the good candidate for implementing this.

And even better, testing-driven coding is great:
0. write a void implementation of a feature, that is code the interface with no implementation;
1. write a test code;
2. launch the test - it must fail;
3. implement the feature;
4. launch the test - it must pass;
5. add some features, and repeat all previous tests every time you add a new feature.

It could sounds like a waste of time, but such coding improve your code quality a lot, and, at least, it help you write and optimize every implementation feature.

But don't forget that unit testing is not enough: you have to do tests with your real application, and perform tasks like any user, in order to validate it works as expected. That's why we added the writing and cross-referencing of test protocols in our SynProject documentation tool.

So how is testing implemented in our framework?

We may have used DUnit - http://sourceforge.net/projects/dunit

But I didn't like the fact that it relies on IDE and create separated units for testing. I find it useful to make tests in pure code, in the same unit which implement them. Smartlink of the Delphi compiler won't put the testing code in your final application, so I don't see a lot of drawbacks. And I don't like visual interfaces with red or green lights... I prefer text files and command line. And DUnit code is bigger than mine, and I don't need so many options. That's a matter of taste - you can not agree, that's fine.

So what about using RTTI for adding tests to your program?

The SynCommons unit implements two classes:

type
  /// a class used to run a suit of test cases
  TSynTests = class(TSynTest)

which is used to register/list the tests; and

type
  /// a class implementing a test case
  // - should handle a test unit, i.e. one or more tests
  // - individual tests are written in the published methods of this class
  TSynTestCase = class(TSynTest)

which is the parent of any test case: in published methods of these classes, you write your own tests.

Sample code

Here are the functions we want to test:

function Add(A,B: double): Double; overload;
begin
  result := A+B;
end;
 
function Add(A,B: integer): integer; overload;
begin
  result := A+B;
end;
 
function Multiply(A,B: double): Double; overload;
begin
  result := A*B;
end;
 
function Multiply(A,B: integer): integer; overload;
begin
  result := A*B;
end;

So we create three classes one for the whole test suit, one for testing addition, one for testing multiplication:

type
  TTestNumbersAdding = class(TSynTestCase)
  published
    procedure TestIntegerAdd;
    procedure TestDoubleAdd;
  end;
 
  TTestNumbersMultiplying = class(TSynTestCase)
  published
    procedure TestIntegerMultiply;
    procedure TestDoubleMultiply;
  end;
 
  TTestSuit = class(TSynTests)
  published
    procedure MyTestSuit;
  end;

The trick is to create published methods.

Here is how one of these test methods are implemented (I let you guess the others):

procedure TTestNumbersAdding.TestDoubleAdd;
var A,B: double;
    i: integer;
begin
  for i := 1 to 1000 do
  begin
    A := Random;
    B := Random;
    Check(SameValue(A+B,Adding(A,B)));
  end;
end;

The SameValue() is necessary because of floating-point precision problem, we can't trust plain = operator.

And here is the test case implementation:

procedure TTestSuit.MyTestSuit;
begin
  AddCase([TTestNumbersAdding,TTestNumbersMultiplying]);
end;

And the main program:

with TTestSuit.Create do
  try
    ToConsole := @Output; // so we will see something on screen
    Run;
    readln;
  finally
    Free;
  end;

Just run this program, and you'll get:

Suit
  ------
 
 
1. My test suit
 
 1.1. Numbers adding:
  - Test integer add: 1000 assertions passed
  - Test double add: 1000 assertions passed
  Total failed: 0 / 2000  - Numbers adding PASSED
 
 1.2. Numbers multiplying:
  - Test integer multiply: 1000 assertions passed
  - Test double multiply: 1000 assertions passed
  Total failed: 0 / 2000  - Numbers multiplying PASSED
 
 
Generated with: Delphi 7 compiler
 
Time elapsed for all tests: 1.96ms
Tests performed at 23/07/2010 15:24:30
 
Total assertions failed for all test suits:  0 / 4000
 
! All tests passed successfully.

You can see that all text on screen was created by "uncamelcasing" the method names, and that the test suit just follows the classes defined.

I've uploaded this test in the SQLite3\Sample\07 - SynTest folder of our Source Code Repository.

You can post comments and get feedback in our forum.

Synopse SQLite3 Framework 1.8

A.Bouchez — Thu, 22 Jul 2010 22:44:00 +0200

The Synopse SQLite3 Database Framework was just released under Version 1.8.

SQLite3 just officially reached version 3.7.0, so it's time to commit our changes from our source code repository to the main "stable" zip.

SynCommons.pas
- includes Unitary Testing class and functions
- bug fixed in WinAnsiBufferToUtf8() and all WinAnsi to UTF-8 encoding
functions (issue identified thanks to new _UTF8 testing function)
- bug fixed in val() under Delphi 2009/2010 for some values (issue identified
thanks to new NumericalConversion testing function)
- bug fixed in AnsiICompW() - used in SynPdf unit
- ambiguous SameText() function rewritten as SameTextU() with UTF-8 decoding
- TTextWriter class moved from SQLite3Commons to SynCommons
- new JSONEncode and JSONDecode functions to directly encode or decode any
content to/from a valid UTF-8 JSON object content
- enhanced URLEncode() and URLDecode() functions
- new ExtendedToStr/ExtendedToString functions
- new tests added (mostly relative to the new functions or classes)

SynCrypto.pas
- mostly code review for Delphi 2009/2010 integration

SynGdiPlus.pas
- small modifications to better handling Jpeg saving

SynPdf.pas
- font substitution if the font is not existing in the system (worse case
will use Arial for all fonts)
- now handle ETO_GLYPH_INDEX in metafile rendering
- new feature: allow forced JPEG compression for graphics
- new feature: UNDERLINE + STRIKEOUT support (also in RICH TEXT and rotated text !)
- new USE_SYNGDIPLUS conditional if you want to use the default jpeg unit
instead of our SynGdiPlus (but you loose TIF, PNG, and GIF support)
- enhanced: PenWidth changed to Single -> better precision (e.g. for underlined text)
- fix issue: Rotated text was misplaced for some angles
- some small fixes about FillRect() + scaling, and move/line stroke
- REDDWARF / ONDREJ made a very good work - I had very few thinks to rewrite
- added optional XOff,YOff parameters to RenderMetaFile()
- now handle EMR_STRETCHDIBITS (used in Html2Pdf)
- fix strike out line position (was too low)
- fixed TextWidth() and TextMeasure()
- fixed font enumeration problem (trigerred with asiatic windows)
- system font enumeration is now stored using UTF-8, and any non ASCII font
name will used in the PDF content the official Postscript name extracted
from its TrueType font content
- optional charset parameter is now available in TPdfCanvas.SetFont: this
was needed in case of TMetaFile rendering to fix some encoding problems
- bitmap embedding fix - see http://synopse.info/forum/viewtopic.php?pid=237
- now initializes the Gdi+ library if necessary
- fix small issue with font orientation in metafile enumaration

SynSelfTests.pas
- first public release, corresponding to SQLite3 Framework 1.8
- includes Unitary Testing class and functions

SQLite3Commons.pas
- includes Unitary Testing class and functions
- update engine to version 3.7
- new direct methods to handle BLOB fields from clients or servers
- new URI to GET or PUT BLOB fields: ModelRoot/TableName/ID/BlobFieldName
- fixed bug in TSQLTable.GetJSONValues: FirstRow parameter not used
- TTextWriter class moved from SQLite3Commons to SynCommons
- new ModelRoot/[TableName/ID/]MethodName RESTful GET/PUT request
(see TSQLRestServerCallBack definition and comments): implements some custom
Client/Server request, similar to the DataSnap technology, but in a KISS way;
it's fully integrated in the Client/Server architecture of our framework,
and extend its ORM feature to Object-less communication - see new
ModelRoot/Stat method implement in TSQLRestServer
- floating point numbers are now encoded using new ExtendedStr[ing] methods:
such values are now encoded in a more human readable way in the JSON content
- new tests added (mostly relative to the new functions or classes)

SQLite3.pas
- includes Unitary Testing class and functions
- update engine to version 3.7
- SetWALMode() method for enabling Write-Ahead Logging for the database
- the RTREE extension is now compiled by default into the engine
- new tests added (mostly relative to the new functions or classes)

SQLite3HttpServer.pas
- includes Unitary Testing class and functions
- allows content-type changing for GET blob fields

SQLite3Pages.pas
- some fixes for compilation under Delphi 2009/2010

SQLite3Toolbar.pas
- possibility to add a background picture to the ribbon pages
(creating a TSQLAdvPage class from TAdvPage with a BackgroundPicture property)

SQLite3UI.pas
- added TSynIntegerLabeledEdit component (e.g. used in SQLite3UIOptions)

Don't forget to consult the draft version of the documentation of this framework!

The full source code from the framework is available to download from SynopseSQLite3.zip licensed under a MPL/GPL/LGPL tri-license.
Compatible with Delphi 7 to Delphi 2010.

Comments and feedback are welcome on our forum:
http://synopse.info/forum/viewtopic.php?pid=260

DataSnap-like Client-Server JSON RESTful Services in Delphi 7-2010

A.Bouchez — Sun, 18 Jul 2010 17:44:00 +0200

You certainly knows about the new DataSnap Client-Server features, based on JSON, introduced in Delphi 2010.
http://docwiki.embarcadero.com/RADStudi … plications

We added such communication in our SQLite3 Framework, in a KISS (i.e. simple) way: no expert, no new unit or new class. Just add a published method Server-side, then use easy functions about JSON or URL-parameters to get the request encoded and decoded as expected, on Client-side.

We'll implement the same example as in the official Embarcadero docwiki page above. Add two numbers. Very useful service, isn't it?

We'll first code the Server-side:

First the declaration of the class:

TSQLRestServerTest = class(TSQLRestServerDB)
published
function Sum(aRecord: TSQLRecord; aParameters: PUTF8Char;
const aSentData: RawUTF8; var aResp, aHead: RawUTF8): Integer;
end;

This method name will be used for the URL encoding, and will be called here with ModelRoot/Sum URL.

This method, like all Server-side methods, MUST have all these parameters, and follow the TSQLRestServerCallBack prototype:

type
TSQLRestServerCallBack = function(aRecord: TSQLRecord;
aParameters: PUTF8Char; const aSentData: RawUTF8;
var aResp, aHead: RawUTF8): Integer of object;

Then we implement this method:

function TSQLRestServerTest.Sum(aRecord: TSQLRecord; aParameters: PUTF8Char;
const aSentData: RawUTF8; var aResp, aHead: RawUTF8): Integer;
var a,b: Extended;
begin
if not UrlDecodeNeedParameters(aParameters,'A,B') then
begin
result := 404; // invalid Request
exit;
end;
while aParameters<>nil do
begin
UrlDecodeExtended(aParameters,'A=',a);
UrlDecodeExtended(aParameters,'B=',b,@aParameters);
end;
aResp := JSONEncodeResult([a+b]);
// same as : aResp := JSONEncode(['result',a+b],TempMemoryStream);
result := 200; // success
end;

Not difficult to follow, isn't?

OK, now the client-side:

function Sum(aClient: TSQLRestClientURI; a, b: double): double;
var err: integer;
begin
val(aClient.CallBackGetResult('sum',['a',a,'b',b]),Result,err);
end;

And... that's all!

You have to create the server instance, and the corresponding TSQLRestClientURI, with the same database model, just as usual...

Note that this Client-Server protocol uses JSON here, but you can serve any kind of data, binary, HTML, whatever...
The usual protocols of our framework can be used: HTTP/1.1, Named Pipe, Windows GDI messages, direct in-memory/in-process access.

Just to be noticed that the data transmitted is a valid JSON content, like this one:

{ "result":3.141592653}

So you can consume these services, implemented Server-Side in fast Delphi code, with any AJAX application Client-Side.

Of course, these services can be related to any table/class of our ORM framework, so you would be able to create easily any RESTful compatible requests on URL like ModelRoot/TableName/ID/MethodName. For example, here we return a BLOB field content as hexa:

function TSQLRestServerTest.DataAsHex(aRecord: TSQLRecordPeople; aParameters: PUTF8Char;
const aSentData: RawUTF8; var aResp, aHead: RawUTF8): Integer;
var aData: TSQLRawBlob;
begin
result := 404; // invalid Request
if (self=nil) or (aRecord=nil) or not aRecord.InheritsFrom(TSQLRecord) or
(aRecord.ID<0) then
exit; // we need a valid record and its ID
if not EngineRetrieveBlob(TSQLRecordPeople,aRecord.ID,
aRecord.FieldProp('Data'),aData) then
exit; // impossible to retrieve the Data BLOB field
aResp := JSONEncodeResult([SynCommons.BinToHex(aData)]);
// idem: aResp := JSONEncode(['result',BinToHex(aRecord.fData)],TempMemoryStream);
result := 200; // success
end;

Full source code is available in our Source Code Repository.
It should work from Delphi 7 to Delphi 2010.

Your comments are welcome on our forum.

Delphi doesn't like multi-core CPUs (or the contrary)

A.Bouchez — Thu, 15 Jul 2010 17:46:00 +0200

If you're like me, you are proud of the new CPU your computer runs on - in my case a i7-720Q with 8 embedded cores...

But Delphi is not very multi-thread or multi-core friendly... guess why....

The root cause the is the LOCK prefix

All X86 CPUs are equipped with the ability to lock a specific memory address, preventing other system buses to read or modify it while the following instruction runs.

The LOCK prefix to an assembly instruction causes the CPUs to assert the LOCK# signal, and practically ensures exclusive use of the memory address in multiprocessors / multi-thread environments.

The LOCK prefix works only with the following instructions:

BT, BTS, BTR, BTC   (mem, reg/imm)
XCHG, XADD  (reg, mem / mem, reg)
ADD, OR, ADC, SBB   (mem, reg/imm)
AND, SUB, XOR   (mem, reg/imm)
NOT, NEG, INC, DEC  (mem)

Note: XCHG and XADD (and all the ‘X’ family of instructions) are planned to be thread-safe, and always asserts LOCK# regardless of the presence of the LOCK prefix.

These low-level LOCK mechanisms ensure that some memory is modified by only one thread at a time.

So what is wrong with these LOCKed instructions?

On a multi-core CPU, all cores just freeze in order to make this LOCKed asm function threadsafe. If you have a lot of threads with more than one CPU, the context of every CPU core has to be frozen, cleared, all cores wait for the LOCKed asm instruction to complete, then the context is to be retrieved, and execution continue.

So guess what... when the CPU has to execute such instructions, all cores just freeze and your brand new 8 cores CPU just run as a 1 core CPU...

This is the same LOCKed asm function which is used internally by Windows with its Critical Sections. That's why Windows itself is told not to be very multi-core friendly, because it does use a lot of critical sections in its internal... Linux is much more advanced, and scales pretty well on massive multi-core architectures.

What about Delphi?

In Delphi, I discovered at least two performance problem in its RTL:

Default memory manager, i.e. FastMM4, uses a LOCKed asm instruction for every memory allocation or dis-allocation.
string types and dynamic arrays just use the same LOCKed asm instruction everywhere, i.e. for every access which may lead into a write to the string.

See what I wrote in the Embarcadero forum... this post was not very popular, but indeed I think I've raised a big issue on the Delphi compiler internals and performance here - and I don't think Embarcadero has plans to resolve this...

IMHO if you use strings in your application and need speed, using another memory manager than FastMM4 is not enough. You'll have to avoid most string use, and implement a safe TStringBuilder-like class.

ShortStrings could be handy here, even if they are limited to 255 character long.

Using regular PAnsiChar, and fixed buffers in the stack is also a solution, but it must be safe...

Our enhanced RTL

In our enhanced RTL for Delphi 7, we avoid use of this LOCKed asm instruction if your application has only one thread: so if you use our enhanced RTL, and make thread by yourself (not using the TThread object), you'll have the best multi-thread performance possible.

For example, here is how we coded the clearing of a string:

asm     { ->    EAX pointer to str      }
        MOV     EDX,[EAX]                       { fetch str                     }
        TEST    EDX,EDX                         { if nil, nothing to do         }
        JE      @@done 
        lea ecx,[edx-skew].StrRec.refCnt
        mov dword ptr [EAX],0  // clear str
        cmp [ecx],0
        jl @@done     // refCnt=-1: literal str
{$ifdef AVOIDLOCK}
        cmp IsMultiThread,false
        jnz @@lock
        dec [ecx]     // not threadsafe dec refCount, but faster
{$else} lock  dec [ecx]     // threadsafe dec refCount
{$endif}jne @@done
@@free: push eax
        mov eax,ecx
        call MemoryManager.FreeMem // inlined _FreeMem() code
        or eax,eax
        jnz _JmpInvalidPtr
        pop eax
{$ifdef AVOIDLOCK}
        ret
@@lock: lock  dec [ecx]     // threadsafe dec refCount
        je @@free
{$else}{$ifndef NOAMD}ret{$endif} // AMD trick: avoid branch misprediction
{$endif}
@@done:{$ifndef NOAMD}db $F3{$endif} // rep ret AMD trick: avoid branch misprediction
end;

So if the AVOIDLOCK conditional is defined, and there is only one thread in your application (the IsMultiThread is false), the lock dec [ecx] instruction won't be called, but a much faster (and core-friendly) dec [ecx] instruction is used.

Note that there is a similar check already in FastMM4: if IsMultiThread is false, no LOCKed instruction will be used.

The only drawback is that if you want to use threads in your application, you'll have:

TThread is not to be used: the creation of one TThread just set IsMultiThread to true, so enable LOCKed instructions;
BeginThread() function must be avoided also (it set also the flag);
So you'll have to call directly CreateThread() Win32 APIs for your threads;
And none of your units should use either TThread either BeginThread!

That's why it could be useful that Embarcadero take this problem in account, and try to resolve it at the compiler level....

You may post comments or react on our forum

http://synopse.info/forum/viewtopic.php?pid=205

htm2pdf free tool

A.Bouchez — Wed, 14 Jul 2010 19:07:00 +0200

Using our SynPDF library and a public domain THtmlView component, and discussion with some of our users, we were able to make a html to pdf conversion tool.

After a discussion with Pawel Stroinski on our forum, the THtmlView component was used to make a tool converting any html document into pdf, i.e. "printing" a html document as a pdf.

I downloaded via SVN the latest THtmlView source code available from the successor of the well-known HTMLViewer components for Delphi (previously sold by PBear.com), i.e. r36.

Then I compiled it under Delphi 7. Some errors in the code. I therefore fixed them. Then try to compile under Delphi 2010 - a lot of warnings, and some main feature not working as expected (like css support). Fixed most bugs, and made a lot of code refactoring - a lot of warning remains, and the component is not truly Unicode: THtmlView use internaly UTF-8 encoding. But it works better than the original version.

Then I used Pawel Stroinski's code using our SynPdf unit to produce a pdf file from any html content.

Some bitmaps where not printed: fixed some part of the SynPdf library (now in version 1.8.3 in our source code repository).

And since the code from THtmlView did rely on jpeg.pas and was handling PNG just with Delphi 2010, I added a compilation define (set by default) in order to use our SynGdiPlus library to read GIF, PNG and JPEG files. So the resulting executable is smaller in size (less than 890 KB), and work as expected. If you don't like all this, and need GIF animated in your PDF (??? it's a joken pdf doesn't allow this), you can turn the compilation define OFF, and use the previous code.

I've uploaded the source code in the HtmlView sub directory of our Source Code Repository.

You can download an executable file from
http://synopse.info/files/pdf/htm2pdf.zip

Just extract the htm2pdf.exe file in a directory, then launch it:

Click on the "Open" button to open a html file, then browse it on the screen, update the parameters below (like margins or paper size), and click on the "Save as" or "Save" buttons.
A pdf file will be created, containing the whole html content, together with the files.

As a demo, I included in the HtmlView/demo sub directory of our Source Code Repository some html, css and picture files.

You can download the resulting pdf from http://synopse.info/files/pdf/sample.pdf

This htm2pdf tool can be used from the command line, without the GUI window above:

1. you'll observe that the last used parameters are saved in a htm2pdf.ini file in the same directory than the executable.

If you launch

htm2pdf /silent

all parameters from htm2pdf.ini will be used, no GUI will be displayed, and the conversion will be made.

2. you can force to specify a html source file name, and an optional destination pdf file:

htm2pdf sample.htm

will create a sample.pdf file in the same directory as sample.htm

htm2pdf sample.htm dest.pdf

will create a dest.pdf file

This tool is released under the Public Domain, free but with no warranty at all.

Discussion and comments are welcome on our forum:
http://synopse.info/forum/viewtopic.php?pid=187

Draft Documentation of the Synopse SQLite3 Framework

A.Bouchez — Mon, 12 Jul 2010 14:55:00 +0200

As a tutorial about SynProject, the documentation for the SQLite3 framework itself has been generated for the first time.

You asked for documentation about our Synopse SQLite3 framework, we have generated it automatically, thanks to SynProject!

The resulting documentation of our SQLite3 framework (in draft status) can be downloaded from sampledoc.zip.

If you want to know how this documentation was created in a few click using SynProject, follow this link.

This documentation will progress, and the associated .pro file will be available in our source code repository.

Comments are welcome in our forum!

How to unzip or zip files content

A.Bouchez — Sun, 11 Jul 2010 20:34:00 +0200

Didn't you ever wanted to unzip some archive content, or embed a zip file into your exe?

Didn't you ever wanted to create a zip archive file, from some data in memory, in pure Delphi code, without using any external dll?

One of the open source unit we use allow you to do these tasks in a easy way.

First of all, all the zip compression is embedded in an unique Delphi unit, named SynZip which is meant to replace the default zlib unit of Delphi, and enhance it. It works from Delphi 7 to Delphi 2010, from the same source code, which is released under a GPL/LGPL/MPL tri-license.

Do you want to compress some data in memory? Use CompressString() and UnCompressString() methods. Both work with RawByteString encoded data:

function test: boolean;

var  tmp: RawByteString;

begin

  tmp := 'some data';

  result := (UnCompressString(CompressString(tmp,False,comp))=tmp);

end;

Do you want to create a zip archive file? Use our TZipWrite class.

procedure TestCompression;

var FN: TFileName;

begin

  FN := ChangeFileExt(paramstr(0),'.zip');

  with TZipWrite.Create(FN) do

  try

    AddDeflated('one.exe',pointer(Data),length(Data));

    assert(Count=1);

    AddDeflated('ident.txt',M.Memory,M.Position);

    assert(Count=2);

  finally

    Free;

  end;

end;

Do you want to read a zip archive file? Use our TZipRead class.

procedure TestUnCompression;

var FN: TFileName;

i: integer;

begin

  FN := ChangeFileExt(paramstr(0),'.zip');

  with TZipRead.Create(FN) do

  try

    i := NameToIndex('ONE.exe');

    assert(i>=0);

    assert(UnZip(i)=Data);

    i := NameToIndex('ident.TXT');

    assert(i>=0);

    assert(Entry[i].info^.zcrc32=crc32(0,M.Memory,M.Position));

  finally

    Free;

  end;

  DeleteFile(FN);

end;

And if you want the file to be embedded to your executable, there is a dedicated Create constructor in the TZipRead just for handling that:

create your exe file with a TZipRead instance created with it;
append your zip content to your exe (by using TZipWrite or by using copy /b on command line);
that's all!

Couldn't it be easier?

Code above was extracted and adapted from our SynSelfTests test unit.

Comments and questions are welcome on our forum.

SynProject screen shots available

A.Bouchez — Sat, 10 Jul 2010 17:30:00 +0200

On the wiki of the Source Code repository, I've added some screen shots of our SynProject tool.

See this page for all this new content.

I hope these screen shots and comments will give you some interest about using and collaborating on this great Open Source (GPL) project!

Named Pipe, Vista, Seven and Service

A.Bouchez — Sun, 04 Jul 2010 16:38:00 +0200

If you want some local communicate between a service software and a front-end GUI application, named pipes are a viable mechanism for this communication. It worked fine until Windows XP, then came Vista, Seven, and the UAC...

By default, when you create a named pipe under Windows, it is accessible only by other applications running under the same user account. Under XP, it worked if the two users were part of the same group, or at least did share some security policy attributes. But, for security reasons, if the background service runs as local system while the application with the user interface runs as the logged on user, you'll receive a ERROR_ACCESS_DENIED flag under Vista and Seven.

If you are willing to open up access to your pipe to all clients, you'd have to explicitly create a TSecurityAttributes record instance, initialized with full user rights:

procedure InitializeSecurity(var SA: TSecurityAttributes; var SD);
begin
fillchar(SD,SECURITY_DESCRIPTOR_MIN_LENGTH,0);
// Initialize the new security descriptor
if InitializeSecurityDescriptor(@SD, SECURITY_DESCRIPTOR_REVISION) then begin
// Add a NULL descriptor ACL to the security descriptor
if SetSecurityDescriptorDacl(@SD, true, nil, false) then begin
// Set up the security attributes structure
SA.nLength := sizeof(TSecurityAttributes);
SA.lpSecurityDescriptor := @SD;
SA.bInheritHandle := true;
exit; // mark OK
end;
end;
fillchar(SA,sizeof(SA),0); // mark error: no security
end;

This works fine, but it doesn't allow you remote access to a service from another computer.

I've tried to implement what Microsoft told officially on http://support.microsoft.com/kb/813414, but I didn't succeed as expected. The trick is to allow access as anonymous user to the pipe...

Here is the code I wrote from original c sample

function GetUserSid(var SID: PSID; var Token: THandle): boolean;
var TokenUserSize: DWORD;
TokenUserP: PSIDAndAttributes;
begin
result := false;
if not OpenThreadToken(GetCurrentThread, TOKEN_QUERY, True, Token) then
if (GetLastError <> ERROR_NO_TOKEN) or
not OpenProcessToken(GetCurrentProcess, TOKEN_QUERY, Token) then
Exit;
TokenUserP := nil;
TokenUserSize := 0;
try
if not GetTokenInformation(Token, TokenUser, nil, 0, TokenUserSize) and
(GetLastError <> ERROR_INSUFFICIENT_BUFFER) then
Exit;
TokenUserP := AllocMem(TokenUserSize);
if not GetTokenInformation(Token, TokenUser, TokenUserP,
TokenUserSize, TokenUserSize) then
Exit;
SID := TokenUserP^.Sid;
result := true;
finally
FreeMem(TokenUserP);
end;
end;
{$ALIGN ON}
type
ACE_HEADER = record
AceType: BYTE;
AceFlags: BYTE;
AceSize: WORD;
end;
ACCESS_ALLOWED_ACE = record
Header: ACE_HEADER;
Mask: ACCESS_MASK;
SidStart: DWORD;
end;
{$A8}
procedure InitializeSecurity(var SA: TSecurityAttributes; var SD);
const
SECURITY_NT_AUTHORITY: TSIDIdentifierAuthority = (Value: (0, 0, 0, 0, 0, 5));
SECURITY_ANONYMOUS_LOGON_RID = ($00000007);
ACL_REVISION = 2;
var pSidAnonymous, pSidOwner: PSID;
dwAclSize: integer;
ACLP: PACL;
Token: THandle;
begin
fillchar(SD,SECURITY_DESCRIPTOR_MIN_LENGTH,0);
// Initialize the new security descriptor
if InitializeSecurityDescriptor(@SD, SECURITY_DESCRIPTOR_REVISION) and
GetUserSid(pSidOwner,Token) then begin
AllocateAndInitializeSid(SECURITY_NT_AUTHORITY,1,
SECURITY_ANONYMOUS_LOGON_RID,0,0,0,0,0,0,0,pSidAnonymous);
try
dwAclSize := sizeof(TACL) +
2 * ( sizeof(ACCESS_ALLOWED_ACE) - sizeof(DWORD) ) +
GetLengthSid(pSidAnonymous) + GetLengthSid(pSidOwner) ;
ACLP := AllocMem(dwAclSize);
try
InitializeAcl(ACLP^,dwAclSize,ACL_REVISION);
if not AddAccessAllowedAce(ACLP^,ACL_REVISION,
GENERIC_ALL,pSidOwner) then
exit;
if not AddAccessAllowedAce(ACLP^,ACL_REVISION,
GENERIC_READ or GENERIC_WRITE,pSidAnonymous) then
exit;
if SetSecurityDescriptorDacl(@SD,true,ACLP,false) then begin
// Set up the security attributes structure
SA.nLength := sizeof(TSecurityAttributes);
SA.lpSecurityDescriptor := @SD;
SA.bInheritHandle := true;
exit; // mark OK
end;
finally
FreeMem(ACLP);
end;
finally
FreeSid(pSidAnonymous);
CloseHandle(Token);
end;
end;
fillchar(SA,sizeof(SA),0); // mark error: no security
end;

But it didn't work as expected. Of course, I added the name of the pipe to the SYSTEM\CurrentControlSet\Services\lanmanserver\parameters\NullSessionPipes registry entry, but it didn't work...

Whole source code is available from our source code repository, file SQLite3\SQlite3Commons.pas

You can continue to discuss on our forum, and share any solution...

Of course, we have implemented HTTP/1.1 connection, which works very well remotely over a network, so if you need networking, consider using HTTP... but it could be an interesting challenge to make named pipes work over the network!