Default Binary/Base64 serialization
By default, any record
value will be serialized with a
proprietary binary (and optimized) layout - i.e. via RecordLoad
and RecordSave
functions - then encoded as Base64, to be
stored as plain text within the JSON stream.
A special UTF-8 prefix (which does not match any existing Unicode glyph) is added at the beginning of the resulting JSON string to identify this content as a BLOB, as such:
{ "MyRecord": "ï¿°w6nDoMOnYQ==" }
You will find in SynCommons
unit both BinToBase64
and Base64ToBin
functions, very optimized for speed.
Base64 encoding was chosen since it is standard, much more efficient
than hexadecimal, and still JSON compatible without the need to escape its
content.
When working with most part of the framework, you do not have anything to do: any record will by default follow this Base64 serialization, so you will be able e.g. to publish or consume interface-based services with records.
Custom serialization
Base64 encoding is pretty convenient for a computer (it is a compact and efficient format), but it is very limited about its interoperability. Our format is proprietary, and will use the internal Delphi serialization scheme: it means that it won't be readable nor writable outside the scope of your own mORMot applications. In a RESTful/SOA world, this sounds not like a feature, but a limitation.
Custom record
JSON serialization can therefore be defined,
as
with any class
.
It will allow writing and parsing record
variables as regular JSON
objects, ready to be consumed by any client or server. Internally, some
callbacks will be used to perform the serialization.
In fact, there are two entry points to specify a custom JSON serialization
for record
:
- When setting a custom dynamic array JSON serializer - the
associated
record
will also use the sameReader
andWriter
callbacks; - By setting explicitly serialization callbacks for the
TypeInfo()
of the record, with the very sameTTextWriter. RegisterCustomJSONSerializer
method used for dynamic arrays.
Then the Reader
and Writer
callbacks can be
defined by two means:
- By hand, i.e. coding the methods with manual conversion to JSON text or parsing;
- Via some text-based type definition, which will follow the
record
layout, but will do all the marshaling (including memory allocation) on its own.
Defining callbacks
For instance, if you want to serialize the following
record
:
TSQLRestCacheEntryValue = record ID: integer; TimeStamp: cardinal; JSON: RawUTF8; end;
With the following code:
TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TSQLRestCacheEntryValue), TTestServiceOrientedArchitecture.CustomReader, TTestServiceOrientedArchitecture.CustomWriter);
The expected format will be as such:
{"ID":1786554763,"TimeStamp":323618765,"JSON":"D:\TestSQL3.exe"}
Therefore, the writer callback could be:
class procedure TTestServiceOrientedArchitecture.CustomWriter( const aWriter: TTextWriter; const aValue); var V: TSQLRestCacheEntryValue absolute aValue; begin aWriter.AddJSONEscape(['ID',V.ID,'TimeStamp',Int64(V.TimeStamp),'JSON',V.JSON]); end;
In the above code, the cardinal
field named
TimeStamp
is type-casted to a Int64
: in fact, as
stated by the documentation of the AddJSONEscape
method, an
array of const
will handle by default any cardinal
as
an integer
value (this is a limitation of the Delphi
compiler). By forcing the type to be an Int64
, the expected
cardinal
value will be transmitted, and not a wrongly negative
versions for numbers > $7fffffff
.
On the other side, the corresponding reader callback would be like:
class function TTestServiceOrientedArchitecture.CustomReader(P: PUTF8Char; var aValue; out aValid: Boolean): PUTF8Char; var V: TSQLRestCacheEntryValue absolute aValue; Values: TPUtf8CharDynArray; begin result := JSONDecode(P,['ID','TimeStamp','JSON'],Values); if result=nil then aValid := false else begin V.ID := GetInteger(Values[0]); V.TimeStamp := GetCardinal(Values[1]); V.JSON := Values[2]; aValid := true; end; end;
Text-based definition
Writing those callbacks by hand could be error-prone, especially for the
Reader
event.
You can use the
TTextWriter.RegisterCustomJSONSerializerFromText
method to define
the record layout in a convenient text-based format.
The very same TSQLRestCacheEntryValue
can be defined as with a
typical pascal record
:
const __TSQLRestCacheEntryValue = 'ID: integer; TimeStamp: cardinal; JSON: RawUTF8';
Or with a shorter syntax:
const __TSQLRestCacheEntryValue = 'ID integer TimeStamp cardinal JSON RawUTF8';
Both declarations will do the same definition. Note that the supplied text
should match exactly the original record
type definition:
do not swap or forget any property!
By convention, we use two underscore characters (__
) before the
record
type name, to easily identify the layout definition. It may
indeed be convenient to write it as a constant, close to the
record
type definition itself, and not in-lined at
RegisterCustomJSONSerializerFromText()
call level.
Then you register your type as such:
TTextWriter.RegisterCustomJSONSerializerFromText( TypeInfo(TSQLRestCacheEntryValue),__TSQLRestCacheEntryValue);
Now you are able to serialize any record
value directly:
Cache.ID := 10;
Cache.TimeStamp := 200;
Cache.JSON := 'test';
U := RecordSaveJSON(Cache,TypeInfo(TSQLRestCacheEntryValue));
Check(U='{"ID":10,"TimeStamp":200,"JSON":"test"}');
You can also unserialize some existing JSON content:
U := '{"ID":210,"TimeStamp":2200,"JSON":"test2"}';
RecordLoadJSON(Cache,@U[1],TypeInfo(TSQLRestCacheEntryValue));
Check(Cache.ID=210);
Check(Cache.TimeStamp=2200);
Check(Cache.JSON='test2');
Note that this text-based definition is very powerful, and is able to handle
any level of nested record
or dynamic arrays.
By default, it will write the JSON content in a compact form, and will expect only existing fields to be available in the incoming JSON. You can specify some options at registration, to ignore all non defined fields. It can be very useful when you want to consume some remote service, and are interested only in a few fields.
For instance, we may define a client access to a RESTful service like
api.github.com
.
type TTestCustomJSONGitHub = packed record name: RawUTF8; id: cardinal; description: RawUTF8; fork: boolean; owner: record login: RawUTF8; id: cardinal; end; end; TTestCustomJSONGitHubs = array of TTestCustomJSONGitHub;
const __TTestCustomJSONGitHub = 'name RawUTF8 id cardinal description RawUTF8 '+ 'fork boolean owner{login RawUTF8 id cardinal}';
Note the format to define a nested record, as a shorter alternative to a
nested record .. end
syntax.
It is also mandatory that you declare the record
as
packed
.
Otherwise, you may have unexpected access violation issues, since alignement
may vary, depending on local setting, and compiler revision.
Now we can register the record
layout, and provide some
additional options:
TTextWriter.RegisterCustomJSONSerializerFromText(TypeInfo(TTestCustomJSONGitHub), __TTestCustomJSONGitHub).Options := [soReadIgnoreUnknownFields,soWriteHumanReadable];
Here, we defined:
soReadIgnoreUnknownFields
to ignore any non defined field in the incoming JSON;soWriteHumanReadable
to let the output JSON be more readable.
Then the JSON can be parsed then emitted as such:
var git: TTestCustomJSONGitHubs; ... U := zendframeworkJson; Check(DynArrayLoadJSON(git,@U[1],TypeInfo(TTestCustomJSONGitHubs))<>nil); U := DynArraySaveJSON(git,TypeInfo(TTestCustomJSONGitHubs));
You can see that the record
serialization is auto-magically
available at dynamic array level, which is pretty convenient in our case, since
the api.github.com
RESTful service returns a JSON array.
It will convert 160 KB of very verbose JSON information:
[{"id":8079771,"name":"Component_ZendAuthentication","full_name":"zendframework/Component_ZendAuthentication","owner":{"login":"zendframework","id":296074,"avatar_url":"https://1.gravatar.com/avatar/460576a0866d93fdacb597da4b90f233?d=https%3A%2F%2Fidenticons.github.com%2F292b7433472e2946c926bdca195cec8c.png&r=x","gravatar_id":"460576a0866d93fdacb597da4b90f233","url":"https://api.github.com/users/zendframework","html_url":"https://github.com/zendframework","followers_url":"https://api.github.com/users/zendframework/followers","following_url":"https://api.github.com/users/zendframework/following{/other_user}","gists_url":"https://api.github.com/users/zendframework/gists{/gist_id}","starred_url":"https://api.github.com/users/zendframework/starred{/owner}{/repo}",...
Into the much smaller (6 KB) and readable JSON content, containing only the information we need:
[ { "name": "Component_ZendAuthentication", "id": 8079771, "description": "Authentication component from Zend Framework 2", "fork": true, "owner": { "login": "zendframework", "id": 296074 } }, { "name": "Component_ZendBarcode", "id": 8079808, "description": "Barcode component from Zend Framework 2", "fork": true, "owner": { "login": "zendframework", "id": 296074 } }, ...
During the parsing process, all unneeded JSON members will just be
ignored.
The parser will jump the data, without doing any temporary memory
allocation.
This is a huge difference with other existing Delphi JSON parsers, which first
create a tree of all JSON values into memory, then allow to browse all the
branches on request.
Note also that the fields have been ordered following the
TTestCustomJSONGitHub
record definition, which may not match the
original JSON layout (here name/id
fields order is inverted, and
owner
is set at the end of each item, for instance).
With mORMot, you can then access directly the content from your Delphi code as such:
if git[0].id=8079771 then begin Check(git[0].name='Component_ZendAuthentication'); Check(git[0].description='Authentication component from Zend Framework 2'); Check(git[0].fork=true); Check(git[0].owner.login='zendframework'); Check(git[0].owner.id=296074); end;
Note that we do not need to use intermediate objects (e.g. via some
obfuscated expressions like
gitarray.Value[0].Value['owner'].Value['login']
).
Your code will be much more readable, will complain at compilation if you
misspell any field name, and will be easy to debug within the IDE (since the
record
layout can be easily inspected).
The serialization is able to handle any kind of nested record
or dynamic arrays, including dynamic arrays of simple types
(e.g. array of integer
or array of RawUTF8
), or
dynamic arrays of record
:
type TTestCustomJSONRecord = packed record A,B,C: integer; D: RawUTF8; E: record E1,E2: double; end; F: TDateTime; end; TTestCustomJSONArray = packed record A,B,C: integer; D: RawUTF8; E: array of record E1: double; E2: string; end; F: TDateTime; end; TTestCustomJSONArraySimple = packed record A,B: Int64; C: array of SynUnicode; D: RawUTF8; end;
The corresponding text definitions may be:
const __TTestCustomJSONRecord = 'A,B,C integer D RawUTF8 E{E1,E2 double} F TDateTime'; __TTestCustomJSONArray = 'A,B,C integer D RawUTF8 E[E1 double E2 string] F TDateTime'; __TTestCustomJSONArraySimple = 'A,B Int64 C array of synunicode D RawUTF8';
The following types are handled by this feature:
Delphi type | Remarks |
boolean |
Serialized as JSON boolean |
byte word integer cardinal Int64 single double |
Serialized as JSON number |
string RawUTF8 SynUnicode WideString |
Serialized as JSON string |
DateTime TTimeLog |
Serialized as JSON text, encoded as ISO-8601 |
TGUID |
Serialized as JSON text |
nested record |
Serialized as JSON object Identified as record ... end; or ... with its nested
definition |
nested registered record |
Serialized as JSON corresponding the the defined callbacks |
dynamic array of record |
Serialized as JSON array Identified as array of ... or [ ... ] |
dynamic array of simple types | Serialized as JSON array Identified e.g. as array of integer |
For other types (like enumerations or sets), you can simply use the unsigned
integer types corresponding to the binary value, e.g. byte word cardinal
Int64
(depending on the sizeof()
of the initial value).
You can refer to the supplied regression tests (in
TTestLowLevelTypes.EncodeDecodeJSON
) for some more examples of
custom JSON serialization.
Feedback is welcome on our forum, as usual!