Synopse Open Source

Introducing TXmlParser

The mormot.core.fmt.pas TXmlParser record operates directly on an in-memory UTF-8 buffer. Its basic token API returns element starts and ends, attributes, text, CDATA, comments and processing instructions.

It deliberately remains a relatively small XML parser rather than attempting to implement every XML feature ever standardized. In particular, it does not support DTD processing or namespace URI binding, and it implements only the XPath-like navigation that is useful for its intended role. The parser nevertheless verifies element nesting and reports malformed XML with detailed error information.

The parser is zero-allocation by design: the TXmlParser type is a record, statically allocated on stack, and token names and values normally point directly into the original UTF-8 input buffer. In pure SAX/pull mode, it consumes only 2KB of stack space.

The benchmark begins by loading the complete XML file into a RawUtf8 string:

var x: TXmlParser; // stack-allocated, no x.Free needed

u := StringFromFile('nasa.xml');
x.Init(u, []);
while x.ParseNext <> xtEof do ; // warmup
writeln('warmup to last position = ', KB(x.Position));

The warmup reaches the end of the input:

warmup to last position = 23.8 MB

So the reference document processed by the benchmark is approximately 23.8 MB.

1. Raw SAX/pull mode

The first test deliberately uses the lowest-level API.

// raw SAX/pull mode API
for i := 1 to 4 do
begin
  t.Start;
  x.Rewind;
  while x.ParseNext <> xtEof do ;
  writeln('TXmlParser.ParseNext: ',t.Stop,' ',
    KB(t.PerSec(length(u))),'/s');
end;

This is essentially the simplest possible parser benchmark: rewind to the beginning and ask ParseNext to walk through the entire XML document until xtEof.

There is no DOM creation and no application-level processing.

The measured results were:

TXmlParser.ParseNext: 11.38ms 2 GB/s
TXmlParser.ParseNext: 11.33ms 2 GB/s
TXmlParser.ParseNext: 11.09ms 2.1 GB/s
TXmlParser.ParseNext: 11.02ms 2.1 GB/s

So the parser processes this 23.8 MB XML document in roughly 11 milliseconds, corresponding to approximately 2.0--2.1 GB/s of input throughput.

For applications doing their own SAX-style processing, this is the fast path.

2. Streaming API with ForEach

The next test uses the higher-level streaming API.

The objective is to find every:

datasets/dataset/tableHead/fields/field

and inspect its units value.

The benchmark is:

// Streaming API
n := 0;
t.Start;
if x.Rewind.Find('datasets') then
  while x.ForEach('dataset', 0) do
    while x.ForEach('tableHead', 1) do
      while x.ForEach('fields', 2) do
        while x.ForEach('field', 3) do
          if x.Find('units') and
             x.ConsumeText(s) and
             (s = 'arcsec') then
            inc(n);
writeln('ForEach: n=',n,' in ',t.Stop,' ',
  KB(t.PerSec(length(u))),'/s');

The result is:

ForEach: n=2770 in 14.23ms 1.6 GB/s

This is approximately 1.6 GB/s while actually navigating the XML hierarchy, finding the relevant elements, consuming their text and comparing it with arcsec.

This is particularly interesting because it is much closer to a real application workload than the raw ParseNext benchmark.

The code expresses the structure of the XML directly:

while x.ForEach('dataset', 0) do
  while x.ForEach('tableHead', 1) do
    while x.ForEach('fields', 2) do
      while x.ForEach('field', 3) do
        ...

The parser remains streaming. No DOM representing the complete 24 MB document is created.

Note the weird 0,1,2,3 numbers as second parameter to every ForEach() call: they are how the method identifies each loop and maintains its state, without the need of a more verbose Open/Close methods.

3. Full DOM with TDocVariant

Sometimes streaming is not what an application needs. Perhaps the XML document is reasonably small, or many different parts of it need to be accessed repeatedly. In that case, constructing a DOM-like representation can be much more convenient. The parser therefore supports consuming an element subtree directly into TDocVariantData.

The corresponding benchmark is:

// single DOM using the TDocVariant.Product() enumerator
t.Start;
if x.Rewind.Find('datasets') then // or XmlToVariant()
  x.Consume(ds);
writeln('Consume All: in ',t.Stop,' ',
  KB(t.PerSec(length(u))),'/s');

The result from the updated run is:

Consume All: in 56.28ms 424.4 MB/s

So building the complete TDocVariant representation takes about 56 ms, corresponding to approximately 424 MB/s.

This is naturally slower than raw parsing. The raw ParseNext() test merely walks through the input. Consume() has to construct an in-memory document representation: objects, arrays, fields and values.

The mapping follows a simple and natural convention:

• XML attributes become object members prefixed with @
• Text content becomes #text only when mixed with attributes or child elements
• Repeated sibling elements automatically become JSON arrays
• By default, all values are stored as "json text" unless xpoVariantGuessType is defined, and numbers/booleans are recognized

For example:

<book id="123">
  <title>mORMot</title>
</book>

becomes:

{
  "@id": "123",
  "title": "mORMot"
}

Mixed content:

<title lang="en">XML</title>

becomes:

{
  "@lang": "en",
  "#text": "XML"
}

xpoVariantGuessType mode:

<invoice id="123">
  <total>12.75</total>
</invoice>

becomes:

{
  "@id": "123",
  "total": 12.75
}

So the 424 MB/s figure measures something substantially more useful than simple XML scanning: it measures how quickly the complete dynamic representation can be constructed.

4. Memory consumption and string interning

Performance is only one part of DOM design.

A DOM has to store the document representation in memory, and XML documents often repeat the same element and attribute names thousands or millions of times. For this reason, TDocVariantData provides the dvoInternNames option. With the default options, consuming the NASA document into TDocVariantData uses approximately 60 MB of memory.

Using:

x.Consume(ds, JSON_XML + [dvoInternNames]);

reduces the memory consumption to approximately 40 MB.

That is roughly a one-third reduction in memory usage, despite the fact that the original XML document itself is about 23.8 MB. More importantly, this reduction does not come with a meaningful performance penalty.

The benchmark without interning is:

// single DOM using the TDocVariant.Product() enumerator
t.Start;
if x.Rewind.Find('datasets') then // or XmlToVariant()
  x.Consume(ds);
writeln('Consume All: in ',t.Stop,' ',
  KB(t.PerSec(length(u))),'/s');

for i := 1 to 2 do
begin
  n := 0;
  t.Start;
  for a in ds.Product('dataset.tableHead.fields.field') do
    if a^.U['units'] = 'arcsec' then
      inc(n);
  writeln('Product: n=',n,' in ',t.Stop,' ',
  KB(t.PerSec(length(u))),'/s');
end;

t.Start;
ds.Clear;
writeln('Clear Consume All: in ',t.Stop,' ',
KB(t.PerSec(length(u))),'/s');

The results are:

Consume All: in 56.28ms 424.4 MB/s
Product: n=2770 in 3.48ms 6.6 GB/s
Product: n=2770 in 3.24ms 7.2 GB/s
Clear Consume All: in 20.73ms 1.1 GB/s

Now compare this with string interning:

// single DOM using the TDocVariant.Product() enumerator and string interning
t.Start;
if x.Rewind.Find('datasets') then // or XmlToVariant()
  x.Consume(ds, JSON_XML + [dvoInternNames]);
writeln('Consume All Interning: in ',t.Stop,' ',
  KB(t.PerSec(length(u))),'/s');

for i := 1 to 2 do
begin
  n := 0;
  t.Start;
  for a in ds.Product('dataset.tableHead.fields.field') do
    if a^.U['units'] = 'arcsec' then
      inc(n);
  writeln('Product: n=',n,' in ',t.Stop,' ',
  KB(t.PerSec(length(u))),'/s');
end;

t.Start;
ds.Clear;
writeln('Clear Consume All Interning: in ',t.Stop,' ',
KB(t.PerSec(length(u))),'/s');

The results are:

Consume All Interning: in 58.04ms 411.5 MB/s
Product: n=2770 in 3.25ms 7.1 GB/s
Product: n=2770 in 3.12ms 7.4 GB/s
Clear Consume All Interning: in 16.85ms 1.3 GB/s

The comparison is striking:

In other words, the benchmark shows no significant performance penalty from interning. The DOM construction measurement changes from 56.28 ms to 58.04 ms. That is only about 1.8 ms difference on a 24 MB document. The Product() measurements are effectively indistinguishable, and the second run is actually slightly faster with interning.

The most interesting difference is memory usage: approximately 60 MB versus 40 MB. There is also less data to clear, which is reflected in the measured destruction time dropping from about 20.7 ms to 16.9 ms.

This makes dvoInternNames a particularly powerful option for large or repetitive XML documents. String interning is especially useful when the document contains many repeated element or attribute names. Instead of keeping separate copies of identical names throughout the dynamic object tree, the representation can share the interned name. The result is a useful combination: less memory, less cleanup work, and essentially the same execution speed. For applications processing large numbers of XML documents, this can be more important than a few percent of raw parsing throughput.

5. TDocVariantData.Product() traversal

After constructing the DOM, the benchmark traverses it twice.

Without interning, the measurements were:

Product: n=2770 in 3.48ms 6.6 GB/s
Product: n=2770 in 3.24ms 7.2 GB/s

With interning:

Product: n=2770 in 3.25ms 7.1 GB/s
Product: n=2770 in 3.12ms 7.4 GB/s

Obviously this does not mean that the program is reparsing XML at 7 GB/s. The XML has already been converted into the TDocVariant representation.

What this measures is the cost of traversing that representation through the Product() enumerator. For such task, the for .. in ... enumerator syntax offered by modern pascal is really powerful. There is no memory allocation involved: each item is retrieved on demand, and returned as convenient PDocVariantData reference.

The expression:

ds.Product('dataset.tableHead.fields.field')

provides a convenient projection over the matching elements.

The application can therefore write:

for a in ds.Product('dataset.tableHead.fields.field') do
  if a^.U['units'] = 'arcsec' then
    inc(n);

The traversal itself is therefore extremely cheap compared with constructing the DOM.

6. Hybrid streaming + partial DOM

The final benchmark combines the two approaches. There is no reason to build a complete 24 MB DOM if only selected parts of the document need convenient DOM-style processing.

The benchmark therefore streams through the outer structure and consumes only the relevant fields subtree:

// hybrid Streaming + partial DOM mode
n := 0;
t.Start;
if x.Rewind.Find('datasets') then
  while x.ForEach('dataset', 0) do
    while x.ForEach('tableHead', 1) do
      while x.Next('fields') do
      begin
        x.Consume(field);
        for a in field.Product('field') do
          if a^.U['units'] = 'arcsec' then
            inc(n);
      end;
writeln('ForEach + Consume Field: n=',n,' in ',t.Stop,' ',
  KB(t.PerSec(length(u))),'/s');

The result:

ForEach + Consume Field: n=2770 in 30.87ms 773.8 MB/s

So the complete operation runs at approximately 774 MB/s.

This is a useful middle ground:

• the complete XML document is never represented as a DOM;
• the parser streams through the outer structure;
• only the relevant fields sections are materialized;
• those temporary DOM fragments can then be processed with Product().

Finally, the temporary DOM is cleared:

t.Start;
field.Clear;
writeln('Clear last Consume: in ',t.Stop,' ',
  KB(t.PerSec(length(u))),'/s');

Result:

Clear last Consume: in 3us 7.5 TB/s

The apparent 7.5 TB/s figure is obviously not a meaningful XML throughput number. The operation is only three microseconds, so expressing that tiny operation against the original 23.8 MB input produces an artificially enormous normalized bandwidth.

7. XPath //field lookup

The parser also supports a simple XPath-like lookup syntax, which makes it possible to express the same search more directly.

For this NASA document, looking up all <field> elements with:

// XPath //field lookup
n := 0;
t.Start;
x.Rewind;
while x.Find('//field') do
begin
  x.Consume(Field);
  if Field.U['units'] = 'arcsec' then
    inc(n);
end;
writeln('XPath //field lookup: n=',n,' in ',t.Stop,' ',
  KB(t.PerSec(length(u))),'/s');

gives:

XPath //field lookup: n=2770 in 27.46ms 869.7 MB/s

So the XPath-like '//field' lookup processes the complete 23.8 MB document at approximately 870 MB/s while finding the same 2,770 matching elements. Of course, if some <field>...</field> nodes would have appeared outside the regular dataset/tableHead/fields/field location, it would have returned some false positives.

Each x.Find('//field') lookup is very fast, but x.Consume(Field) computes the DOM of each <field> content, so it is slower than the explicit nested ForEach() streaming traversal, which reaches 1.6 GB/s. But it provides a concise way to express a document-wide lookup, with some core nodes DOM representation. The result is also still comfortably below 30 ms for the complete document.

This illustrates another useful point of the API: the parser does not force a choice between maximum streaming performance and convenient path-based navigation. Explicit ForEach() is available when every bit of performance matters, while the XPath-like //field syntax can make less performance-critical queries considerably simpler.

The XPath support of Find() is limited to the 90% useful patterns: /root/catalog calls Rewind to search from the document <root>, catalog/book search nested <catalog><book> from the current position, //book path will find <book> anywhere from the current position. There is no XPath //book/title, predicates, wildcards, attributes or namespaces support by design.

Complete benchmark results

The measurements can now be summarized as follows:

* These throughput figures are useful mainly for normalizing the measured time against the 23.8 MB input. They should not be interpreted as parser bandwidth.

The most meaningful numbers are therefore:

Why the hybrid model matters

XML libraries have traditionally forced developers toward one of two programming models.

The first is SAX-style processing. SAX is efficient because it processes the XML sequentially rather than constructing an entire tree. The downside is that application code must understand the event stream and maintain enough state to reconstruct the logical structure it is interested in.

The second is DOM. DOM is much easier to navigate because the entire XML document becomes a tree. But constructing and retaining that tree consumes memory and introduces allocation and object-management overhead.

TXmlParser tries to make the compromise part of the API itself.

Raw UTF-8 buffer
      |
      +-- ParseNext()
      |      raw SAX/pull
      |
      +-- Find()/Next()/ForEach()
      |      selective streaming
      |
      +-- Consume()
      |      partial DOM
      |
      +-- XmlToVariant()
             complete DOM
                    |
                    +-- Product()
                         efficient traversal

These are not four unrelated XML implementations. They are different views over the same parser state and data representation.

How this compares with existing XML solutions

There is already no shortage of XML support in Pascal. Free Pascal's fcl-xml, for example, provides XML DOM and reader functionality, and even its xpath unit. There are also established SAX interfaces for Pascal, as well as other XML libraries such as OXml that provide combinations of DOM and SAX functionality. On Delphi, developers can also use platform XML implementations such as MSXML, or bindings to native XML engines such as libxml2. Therefore, the benchmark should not be presented as a claim that TXmlParser is universally faster than every existing XML library. That would require a proper cross-library benchmark using the same XML corpus, CPU, operating system, compiler and optimization settings, with equivalent parser features and equivalent workloads. Endless apples-to-oranges comparison for sure.

An interesting comparison may be the numbers published in NesLib XML DOM blog article. Reducing each node to 8 bytes is impressive, and mORMot DOM via TDocVariant uses more memory for sure. But the blog article, and associated benchmark code was in fact plain wrong about its query tests: they parsed for matching units=arcsec but only for the first appearance in each loop, so they identified only 3 and not 2770 occurrences. It is a good indication that, when mining XML data, the API itself is more important than raw performance or memory consumption. Our nested ForEach() loops or our Product() iterator are simple to understand and very effective to use. They found what they are supposed to with no surprise.

The benchmark presented here demonstrates something narrower and more relevant to the mORMot toolbox:

A native Delphi/Free Pascal XML parser can provide very high raw parsing performance while simultaneously offering convenient selective streaming and dynamic DOM APIs without requiring separate parsing engines.

There are extremely mature XML implementations in C and C++, Java, .NET and many other ecosystems. Some are highly optimized, some prioritize complete XML standards compliance, and some deliberately restrict their feature set to maximize performance. Historical XML parser benchmarks have repeatedly shown the fundamental difference between SAX-style processing and DOM construction: sequential event processing can be substantially cheaper than building a complete object tree.

The real advantage is not the 2 GB/s number. The raw number is impressive, but it is not actually the most important part of the benchmark. The interesting result is the range of abstraction levels.

If maximum speed is needed:

while x.ParseNext <> xtEof do
  ...

gives direct access to the parser stream.

If the application needs to locate selected elements:

x.Find(...)
x.ForEach(...)

provides navigation without building the whole document.

If only a particular subtree needs convenient dynamic access:

x.Consume(field);

creates a temporary TDocVariantData.

And if the entire document should simply become a dynamic object:

XmlToVariant(...)

provides the convenient high-level representation.

Then Product() makes querying the resulting document particularly concise:

for a in ds.Product('dataset.tableHead.fields.field') do
  if a^.U['units'] = 'arcsec' then
    inc(n);

And there are also some other TDocVariantData methods accepting returning a PVariant or directly a RawUtf8, for even cleaner code:

var v: PVariant;
    for v in ds.ProductValue('dataset.tableHead.fields.field.units') do
      if VariantEquals(v^, 'arcsec') then // faster than naive v^ = 'arcsec'
        inc(n);
  // ProductValue v^=arcsec: n=2770 in 5.27ms 4.4 GB/s
  // ProductValue VariantEquals: n=2770 in 2.92ms 7.9 GB/s

var s: RawUtf8;
    for s in ds.ProductU('dataset.tableHead.fields.field.units') do
      if s ='arcsec' then
        inc(n);
  // ProductU: n=2770 in 3.49ms 6.6 GB/s

Finally, dvoInternNames allows the same dynamic representation to use substantially less memory when XML names are repeated.

Just compare with the standard Delphi-XML way of computing the very same request:

var
  Dataset, TableHead, Fields, Field, Units: IXMLNode;
  I, J, K, L, N: Integer;

N := 0;
Datasets := XMLDoc.DocumentElement.ChildNodes.FindNode('datasets');
if Datasets <> nil then
  for I := 0 to Datasets.ChildNodes.Count - 1 do
  begin
    Dataset := Datasets.ChildNodes[I];
    if Dataset.NodeName <> 'dataset' then
      Continue;
    for J := 0 to Dataset.ChildNodes.Count - 1 do
    begin
      TableHead := Dataset.ChildNodes[J];
      if TableHead.NodeName <> 'tableHead' then
        Continue;
      for K := 0 to TableHead.ChildNodes.Count - 1 do
      begin
        Fields := TableHead.ChildNodes[K];
        if Fields.NodeName <> 'fields' then
          Continue;
        for L := 0 to Fields.ChildNodes.Count - 1 do
        begin
          Field := Fields.ChildNodes[L];
          if Field.NodeName <> 'field' then
            Continue;
          Units := Field.ChildNodes.FindNode('units');
          if (Units <> nil) and (Units.Text = 'arcsec') then
            Inc(N);
        end;
      end;
    end;
  end;

Which one do you find more natural and less error prone?
Which one do you guess is the fastest and uses less memory?

The design of our TXmlParser was to optimize the common cases:

• UTF-8 input;
• sequential parsing;
• simple element navigation;
• selective subtree consumption;
• dynamic variant DOM representation;
• none or low allocation;
• efficient string interning;
• straightforward error handling.

This is a very mORMot-like approach: provide a small, composable primitive that is fast enough for demanding workloads and convenient enough that application developers do not need to abandon the abstraction to get performance.