Default Binary/Base64 serialization

By default, any record value will be serialized with a proprietary binary (and optimized) layout - i.e. via RecordLoad and RecordSave functions - then encoded as Base64, to be stored as plain text within the JSON stream.

A special UTF-8 prefix (which does not match any existing Unicode glyph) is added at the beginning of the resulting JSON string to identify this content as a BLOB, as such:

 { "MyRecord": "ï¿°w6nDoMOnYQ==" }

You will find in SynCommons unit both BinToBase64 and Base64ToBin functions, very optimized for speed.
Base64 encoding was chosen since it is standard, much more efficient than hexadecimal, and still JSON compatible without the need to escape its content.

When working with most part of the framework, you do not have anything to do: any record will by default follow this Base64 serialization, so you will be able e.g. to publish or consume interface-based services with records.

Custom serialization

Base64 encoding is pretty convenient for a computer (it is a compact and efficient format), but it is very limited about its interoperability. Our format is proprietary, and will use the internal Delphi serialization scheme: it means that it won't be readable nor writable outside the scope of your own mORMot applications. In a RESTful/SOA world, this sounds not like a feature, but a limitation.

Custom record JSON serialization can therefore be defined, as with any class.
It will allow writing and parsing record variables as regular JSON objects, ready to be consumed by any client or server. Internally, some callbacks will be used to perform the serialization.

In fact, there are two entry points to specify a custom JSON serialization for record:

  • When setting a custom dynamic array JSON serializer - the associated record will also use the same Reader and Writer callbacks; 
  • By setting explicitly serialization callbacks for the TypeInfo() of the record, with the very same TTextWriter. RegisterCustomJSONSerializer method used for dynamic arrays.

Then the Reader and Writer callbacks can be defined by two means:

  • By hand, i.e. coding the methods with manual conversion to JSON text or parsing; 
  • Via some text-based type definition, which will follow the record layout, but will do all the marshaling (including memory allocation) on its own.

Defining callbacks

For instance, if you want to serialize the following record:

  TSQLRestCacheEntryValue = record
    ID: integer;
    TimeStamp: cardinal;
    JSON: RawUTF8;
  end;

With the following code:

  TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TSQLRestCacheEntryValue),
    TTestServiceOrientedArchitecture.CustomReader,
    TTestServiceOrientedArchitecture.CustomWriter);

The expected format will be as such:

 {"ID":1786554763,"TimeStamp":323618765,"JSON":"D:\TestSQL3.exe"}

Therefore, the writer callback could be:

class procedure TTestServiceOrientedArchitecture.CustomWriter(
  const aWriter: TTextWriter; const aValue);
var V: TSQLRestCacheEntryValue absolute aValue;
begin
  aWriter.AddJSONEscape(['ID',V.ID,'TimeStamp',Int64(V.TimeStamp),'JSON',V.JSON]);
end;

In the above code, the cardinal field named TimeStamp is type-casted to a Int64: in fact, as stated by the documentation of the AddJSONEscape method, an array of const will handle by default any cardinal as an integer value (this is a limitation of the Delphi compiler). By forcing the type to be an Int64, the expected cardinal value will be transmitted, and not a wrongly negative versions for numbers > $7fffffff.

On the other side, the corresponding reader callback would be like:

class function TTestServiceOrientedArchitecture.CustomReader(P: PUTF8Char;
  var aValue; out aValid: Boolean): PUTF8Char;
var V: TSQLRestCacheEntryValue absolute aValue;
    Values: TPUtf8CharDynArray;
begin
  result := JSONDecode(P,['ID','TimeStamp','JSON'],Values);
  if result=nil then
    aValid := false else begin
    V.ID := GetInteger(Values[0]);
    V.TimeStamp := GetCardinal(Values[1]);
    V.JSON := Values[2];
    aValid := true;
  end;
end;

Text-based definition

Writing those callbacks by hand could be error-prone, especially for the Reader event.

You can use the TTextWriter.RegisterCustomJSONSerializerFromText method to define the record layout in a convenient text-based format.

The very same TSQLRestCacheEntryValue can be defined as with a typical pascal record:

 const
  __TSQLRestCacheEntryValue = 'ID: integer; TimeStamp: cardinal; JSON: RawUTF8';

Or with a shorter syntax:

 const
  __TSQLRestCacheEntryValue = 'ID integer TimeStamp cardinal JSON RawUTF8';

Both declarations will do the same definition. Note that the supplied text should match exactly the original record type definition: do not swap or forget any property!

By convention, we use two underscore characters (__) before the record type name, to easily identify the layout definition. It may indeed be convenient to write it as a constant, close to the record type definition itself, and not in-lined at RegisterCustomJSONSerializerFromText() call level.

Then you register your type as such:

  TTextWriter.RegisterCustomJSONSerializerFromText(
    TypeInfo(TSQLRestCacheEntryValue),__TSQLRestCacheEntryValue);

Now you are able to serialize any record value directly:

  Cache.ID := 10;
  Cache.TimeStamp := 200;
  Cache.JSON := 'test';
  U := RecordSaveJSON(Cache,TypeInfo(TSQLRestCacheEntryValue));
  Check(U='{"ID":10,"TimeStamp":200,"JSON":"test"}');

You can also unserialize some existing JSON content:

  U := '{"ID":210,"TimeStamp":2200,"JSON":"test2"}';
  RecordLoadJSON(Cache,@U[1],TypeInfo(TSQLRestCacheEntryValue));
  Check(Cache.ID=210);
  Check(Cache.TimeStamp=2200);
  Check(Cache.JSON='test2');

Note that this text-based definition is very powerful, and is able to handle any level of nested record or dynamic arrays.

By default, it will write the JSON content in a compact form, and will expect only existing fields to be available in the incoming JSON. You can specify some options at registration, to ignore all non defined fields. It can be very useful when you want to consume some remote service, and are interested only in a few fields.

For instance, we may define a client access to a RESTful service like api.github.com.

type
  TTestCustomJSONGitHub = packed record
    name: RawUTF8;
    id: cardinal;
    description: RawUTF8;
    fork: boolean;
    owner: record
      login: RawUTF8;
      id: cardinal;
    end;
  end;
  TTestCustomJSONGitHubs = array of TTestCustomJSONGitHub;

const
  __TTestCustomJSONGitHub = 'name RawUTF8 id cardinal description RawUTF8 '+
    'fork boolean owner{login RawUTF8 id cardinal}';

Note the format to define a nested record, as a shorter alternative to a nested record .. end syntax.

It is also mandatory that you declare the record as packed.
Otherwise, you may have unexpected access violation issues, since alignement may vary, depending on local setting, and compiler revision.

Now we can register the record layout, and provide some additional options:

  TTextWriter.RegisterCustomJSONSerializerFromText(TypeInfo(TTestCustomJSONGitHub),
    __TTestCustomJSONGitHub).Options := [soReadIgnoreUnknownFields,soWriteHumanReadable];

Here, we defined:

  • soReadIgnoreUnknownFields to ignore any non defined field in the incoming JSON; 
  • soWriteHumanReadable to let the output JSON be more readable.

Then the JSON can be parsed then emitted as such:

var git: TTestCustomJSONGitHubs;
 ...
  U := zendframeworkJson;
  Check(DynArrayLoadJSON(git,@U[1],TypeInfo(TTestCustomJSONGitHubs))<>nil);
  U := DynArraySaveJSON(git,TypeInfo(TTestCustomJSONGitHubs));

You can see that the record serialization is auto-magically available at dynamic array level, which is pretty convenient in our case, since the api.github.com RESTful service returns a JSON array.

It will convert 160 KB of very verbose JSON information:

[{"id":8079771,"name":"Component_ZendAuthentication","full_name":"zendframework/Component_ZendAuthentication","owner":{"login":"zendframework","id":296074,"avatar_url":"https://1.gravatar.com/avatar/460576a0866d93fdacb597da4b90f233?d=https%3A%2F%2Fidenticons.github.com%2F292b7433472e2946c926bdca195cec8c.png&r=x","gravatar_id":"460576a0866d93fdacb597da4b90f233","url":"https://api.github.com/users/zendframework","html_url":"https://github.com/zendframework","followers_url":"https://api.github.com/users/zendframework/followers","following_url":"https://api.github.com/users/zendframework/following{/other_user}","gists_url":"https://api.github.com/users/zendframework/gists{/gist_id}","starred_url":"https://api.github.com/users/zendframework/starred{/owner}{/repo}",...

Into the much smaller (6 KB) and readable JSON content, containing only the information we need:

[
{
"name": "Component_ZendAuthentication",
"id": 8079771,
"description": "Authentication component from Zend Framework 2",
"fork": true,
"owner":
{
"login": "zendframework",
"id": 296074
}
},
{
"name": "Component_ZendBarcode",
"id": 8079808,
"description": "Barcode component from Zend Framework 2",
"fork": true,
"owner":
{
"login": "zendframework",
"id": 296074
}
},
...

During the parsing process, all unneeded JSON members will just be ignored.
The parser will jump the data, without doing any temporary memory allocation.
This is a huge difference with other existing Delphi JSON parsers, which first create a tree of all JSON values into memory, then allow to browse all the branches on request.

Note also that the fields have been ordered following the TTestCustomJSONGitHub record definition, which may not match the original JSON layout (here name/id fields order is inverted, and owner is set at the end of each item, for instance).

With mORMot, you can then access directly the content from your Delphi code as such:

  if git[0].id=8079771 then begin
    Check(git[0].name='Component_ZendAuthentication');
    Check(git[0].description='Authentication component from Zend Framework 2');
    Check(git[0].fork=true);
    Check(git[0].owner.login='zendframework');
    Check(git[0].owner.id=296074);
  end;

Note that we do not need to use intermediate objects (e.g. via some obfuscated expressions like gitarray.Value[0].Value['owner'].Value['login']).
Your code will be much more readable, will complain at compilation if you misspell any field name, and will be easy to debug within the IDE (since the record layout can be easily inspected).

The serialization is able to handle any kind of nested record or dynamic arrays, including dynamic arrays of simple types (e.g. array of integer or array of RawUTF8), or dynamic arrays of record:

type
  TTestCustomJSONRecord = packed record
    A,B,C: integer;
    D: RawUTF8;
    E: record E1,E2: double; end;
    F: TDateTime;
  end;
  TTestCustomJSONArray = packed record
    A,B,C: integer;
    D: RawUTF8;
    E: array of record E1: double; E2: string; end;
    F: TDateTime;
  end;
  TTestCustomJSONArraySimple = packed record
    A,B: Int64;
    C: array of SynUnicode;
    D: RawUTF8;
  end;

The corresponding text definitions may be:

const
  __TTestCustomJSONRecord = 'A,B,C integer D RawUTF8 E{E1,E2 double} F TDateTime';
  __TTestCustomJSONArray  = 'A,B,C integer D RawUTF8 E[E1 double E2 string] F TDateTime';
  __TTestCustomJSONArraySimple = 'A,B Int64 C array of synunicode D RawUTF8';

The following types are handled by this feature:

Delphi type Remarks
boolean Serialized as JSON boolean
byte word integer cardinal Int64 single double Serialized as JSON number
string RawUTF8 SynUnicode WideString Serialized as JSON string
DateTime TTimeLog Serialized as JSON text, encoded as ISO-8601
TGUID Serialized as JSON text
nested record Serialized as JSON object
Identified as record ... end; or ... with its nested definition
nested registered record Serialized as JSON corresponding the the defined callbacks
dynamic array of record Serialized as JSON array
Identified as array of ... or [ ... ]
dynamic array of simple types Serialized as JSON array
Identified e.g. as array of integer

For other types (like enumerations or sets), you can simply use the unsigned integer types corresponding to the binary value, e.g. byte word cardinal Int64 (depending on the sizeof() of the initial value).

You can refer to the supplied regression tests (in TTestLowLevelTypes.EncodeDecodeJSON) for some more examples of custom JSON serialization.

Feedback is welcome on our forum, as usual!