Synopse Open Source - Tag - MetaProgrammingmORMot MVC / SOA / ORM and friends2024-02-02T17:08:25+00:00urn:md5:cc547126eb580a9adbec2349d7c65274DotclearModern Pascal is Still in the Raceurn:md5:af5e15cc3f4bde55561318a1bb5dd3bb2022-11-26T10:05:00+00:002022-11-29T13:21:57+00:00Arnaud BouchezPascal ProgrammingblogcollectionsCrossPlatformDatabaseDelphiFPCGarbageCollectorgenericsGoGoodPracticeMetaProgrammingmORMotmORMot2performanceRTTIRust<p>A recent poll <a href="https://forum.lazarus.freepascal.org/index.php/topic,61276.0.html">on the Lazarus/FPC forum</a> highlighted a fact: pascal coders are older than most coders. Usually, at our age, we should be managers, not developers. But we like coding in pascal. It is still fun after decades!<br />
But does it mean that you should not use pascal for any new project? Are the language/compilers/libraries outdated?<br />
In the company I currently work for, we have young coders, just out-of-school or still-in-school, which joined the team and write great code!</p>
<p><img src="https://blog.synopse.info?post/public/blog/performance.jpg" alt="" /></p>
<p>And a recent thread <a href="https://forum.lazarus.freepascal.org/index.php/topic,61035.0.html">in this very same forum</a> was about comparing languages to implement a REST server, in C#, Go, Scala, TypeScript, Elixir and Rust.<br />
Several pascal versions are about to be contributed, one in which <em>mORMot</em> shines.</p> <h4>The Challenge and the Algorithms</h4>
<p>The original challenge is available at <a href="https://github.com/losvedir/transit-lang-cmp">transit-lang-cmp</a> with the original source code, of all those fancy languages and libraries.</p>
<p>In practice, the goal of this test program is to load two big CSVs into memory (80MB + 2MB), then serve over HTTP some JSON generated by route identifiers, joining both CSVs.<br />
The resulting JSON could be of 30KB up to 2MB. And all data is generated on the fly from the CSV in memory.</p>
<p>To be fair, a regular/business coder would have used a database for this. Not silly memory structures. And asked for money to setup a huge set of cloud machines with load balancing. <img src="https://blog.synopse.info?pf=smile.svg" alt=":-)" class="smiley" /></p>
<h4>Reference Implementations in Today Languages</h4>
<p>The "modern" / "school" approach, as implemented in the reference project in Go/Rust/C#/... is using two lists for the CSVs data, then two maps/dictionaries between route ID and lists indexes.</p>
<ul>
<li>The <a href="https://github.com/losvedir/transit-lang-cmp/blob/main/trogsit/app.go">Golang version</a> has a good expressiveness, and is nice to read, even if you don't know the language.</li>
<li>The <a href="https://github.com/losvedir/transit-lang-cmp/tree/main/Trannet">C# version</a> is also readable, but making a webserver is still confusing because it is not built from code, but from config files.</li>
<li><a href="https://github.com/losvedir/transit-lang-cmp/tree/main/trexit">Elixir</a> is a bit over-complicated to my taste.</li>
<li><a href="https://github.com/losvedir/transit-lang-cmp/tree/main/trala">Scala</a> and <a href="https://github.com/losvedir/transit-lang-cmp/tree/main/trypsit">TypeScript/Deno</a> versions, are fine to read, but really slow. You may better use a database instead.</li>
<li>Just for fun, check <a href="https://github.com/losvedir/transit-lang-cmp/blob/main/trustit/src/main.rs">the Rust version</a> - do you think Rust is good for big maintainable projects with junior developers?</li>
</ul>
<p>There was a first attempt to write a FPC version of it, by Leledumbo.<br />
His <a href="https://github.com/leledumbo/transit-lang-cmp/blob/main/trascal/app.pas">Source Code repository</a> is a nice pascal conversion of above code. But performance was disappointing. Especially because the standard JSON library can not work directly with high level structures like collections or arrays.</p>
<p>So is Pascal out of the race?<br />
Let's call the <em>mORMot</em> to the rescue!</p>
<h4>Following the mORMot Way</h4>
<p>For the <em>mORMot</em> version in FPC, I used another approach, with two diverse algorithms:</p>
<ul>
<li>I ensured the lists were sorted in memory, then made a O(log(n)) binary lookup in it;</li>
<li>All stored strings were "interned", i.e. the same text was sharing a single string instance, and FPC reference counting did its magic.</li>
</ul>
<p>There is no low-level tricks like generating the JSON by hand or using complex data structures - data structures are still are high-level, with readable field names and such. The logic and the intent are clearly readable.<br />
We just leveraged the pascal language, and <em>mORMot</em> features. For instance, string interning is part of the framework, if needed.</p>
<p>Please <a href="https://github.com/synopse/mORMot2/tree/master/ex/lang-cmp/LangCmp.dpr">check the source code in our repository</a>.</p>
<p>As a result:</p>
<ul>
<li>Code is still readable, short and efficient (most of the process is done by <em>mORMot</em>, i.e. CSV, searching, JSON);</li>
<li>It uses much less memory - 10 times less memory than Go when holding the data, 5 times less memory than Go when serving the data;</li>
<li>Performance is as fast as Go, and its very tuned/optimized compiler and RTL.</li>
</ul>
<p><img src="https://blog.synopse.info?post/public/blog/mORMot2-small.png" alt="" /></p>
<h4>Algorithms Matters</h4>
<p>Main idea was to let the algorithms match the input data and the expected resultset.<br />
As programmers do when programming games. Not as coders do when pissing out business software. <img src="https://blog.synopse.info?pf=wink.svg" alt=";-)" class="smiley" /></p>
<ul>
<li>The source code is still pretty readable, thanks to using <em>mORMot</em> efficient <code>TDynArray</code> to map the dynamic array storage, and its CSV and JSON abilities.</li>
<li>I guess source is still understandable for out-of-school programmers - much more readable than Rust for instance.</li>
</ul>
<p>To by fair, I used typed pointers in <code>TScheduler.BuildTripResponse</code> but it is not so hard getting their purpose, and FPC compiles this function into very efficient assembly. I could have used regular dynamic array access with indexes, it would have been slightly slower, but not really easier to follow, nor safer (if we compile with no range checking).</p>
<p>Worth noting that we did not make any specific tuning, like pre-allocating the results with constants, as other frameworks did. We just specified the data, then let <em>mORMot</em> play with it - that's all.<br />
The <em>mORMot</em> RTTI level matches what we expect for modern frameworks: not only some classes to store JSON, but convenient serialization/unserialization using structures like class or record.<br />
Using modern Pascal dynamic arrays and records to define the data structures let the compiler leverage the memory for us, with no need to write any <code>try..finally..Free</code> blocks, and use interfaces. "Manual memory management" with Pascal is not mandatory and can easily be bypassed. Only for the WebServer, we have a <code>Free</code>, which is expected to close it.</p>
<h4>Give Me Some Numbers</h4>
<p>Here are a performance comparison with Go (FPC on the left, Go on the right):</p>
<pre>
parsed 1790905 stop times in 968.43ms | parsed 1790905 stop times in 3.245251432s
parsed 71091 trips in 39.54ms | parsed 71091 trips in 85.747852ms
running (0m33.4s), 00/50 VUs, 348 complete and 0 interrupted | running (0m32.3s), 00/50 VUs, 320 complete and 0 interrupted
default ✓ [======================================] 50 VUs 30 default ✓ [======================================] 50 VUs 30
data_received..................: 31 GB 933 MB/s | data_received..................: 31 GB 971 MB/s
data_sent......................: 3.2 MB 97 kB/s | data_sent......................: 3.0 MB 92 kB/s
http_req_blocked...............: avg=9µs min=1.09µs | http_req_blocked...............: avg=6.77µs min=1.09µs
http_req_connecting............: avg=2.95µs min=0s | http_req_connecting............: avg=1.73µs min=0s
http_req_duration..............: avg=47.59ms min=97.28µs | http_req_duration..............: avg=49.02ms min=123.81µ
{ expected_response:true }...: avg=47.59ms min=97.28µs | { expected_response:true }...: avg=49.02ms min=123.81µ
http_req_failed................: 0.00% ✓ 0 ✗ | http_req_failed................: 0.00% ✓ 0 ✗ 3
http_req_receiving.............: avg=9.66ms min=15.35µs | http_req_receiving.............: avg=5.92ms min=14.76µs
http_req_sending...............: avg=87.24µs min=5.2µs | http_req_sending...............: avg=70.71µs min=5.2µs
http_req_tls_handshaking.......: avg=0s min=0s | http_req_tls_handshaking.......: avg=0s min=0s
http_req_waiting...............: avg=37.83ms min=54.74µs | http_req_waiting...............: avg=43.02ms min=91.84µs
http_reqs......................: 34452 1032.205528/s | http_reqs......................: 31680 981.949476/s
iteration_duration.............: avg=4.72s min=3.54s | iteration_duration.............: avg=4.86s min=2.19s
iterations.....................: 348 10.426318/s | iterations.....................: 320 9.918682/s
vus............................: 30 min=30 ma | vus............................: 15 min=15 max
vus_max........................: 50 min=50 ma | vus_max........................: 50 min=50 max
</pre>
<p>So CSV loading was much faster, then the HTTP server performance was almost the same.</p>
<h4>No Alzheimer</h4>
<p>Here are some numbers about memory consumption:</p>
<blockquote><p>Upon finished loading the CSV, mORMot only eats 80MB, heck so little. Sounds a bit magical. But during load test, it fluctuates between 250-350MB, upon which it returns to 80MB at the end.
The Go version eats 925MB upon finished loading the CSV. During load test, it tops at 1.5GB, returning to 925MB afterwards.</p></blockquote>
<p>Nice to read. :)</p>
<h4>Pascal has a Modern and Capable Ecosystem</h4>
<p>This article was not only about Pascal, but about algorithms and libraries.<br />
The challenge was initially about comparing them. Not only as unrealistic micro-benchmarks, or "computer language benchmark games", but as data processing abilities on a real usecase.</p>
<p><strong>And... Pascal is still in the race for sure!</strong><br />
Not only for "old" people like me - I just got 50 years old. ;-)</p>
<p>The more we spread such kind of information, the less people would make jokes about pascal programmers.<br />
Delphi and FPC are as old as Java, so it is time to get the big picture, not following marketing trends.</p>Try to avoid RTTI (ab)useurn:md5:d296cf380ab8ae68e214ecaab44a8ef22015-11-21T14:30:00+01:002015-11-22T13:06:03+01:00AB4327-GANDIPascal ProgrammingblogDelphiGoodPracticeMetaDataMetaProgrammingmORMotMVCMVVMORMRTTISOAUserInterface<p>There is a very trendy move, since a few years, to value so called "<a href="https://en.wikipedia.org/wiki/Metaprogramming">meta-programming</a>".<br />
In short, it is about the ability to treat programs as their data.<br />
It is a very powerful paradigm in functional languages, and it was also
introduced to OOP languages, <a href="http://csl.ensm-douai.fr/MetaclassTalk">even in SmallTalk</a> a long time
before this concept was trendy in Ruby, C# or Java.</p>
<p><a href="https://pragprog.com/book/cmelixir/metaprogramming-elixir"><img src="https://imagery.pragprog.com/products/430/cmelixir_xlargecover.jpg?1415371472" alt="" /></a></p>
<p>In OOP compiled languages, reflection is used to achieve a similar behavior
at run-time, mainly via RTTI (<a href="https://en.wikipedia.org/wiki/Run-time_type_information">Run-Time Type
Information</a>).<br />
Delphi <a href="http://www.blong.com/Conferences/BorConUK98/DelphiRTTI/CB140.htm">supports
RTTI since its version 1</a>, as it was heavily used e.g. for all UI
streaming.<br />
In our framework, we rely on RTTI for its main features: <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_3">
ORM</a>, <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_63">SOA</a>
and <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_465">
MVC</a> - and even in some other parts, like <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_5">
Desktop UI generation</a>.</p>
<p>But RTTI could easily be abused.<br />
Here are some thoughts, started as a comment in a <a href="http://tech.turbu-rpg.com/473/the-next-rtti-bottleneck/comment-page-1#comment-197674">
good old Mason's blog article</a> about how RTTI performance may be a
bottleneck.<br />
My comment was to get rid of RTTI, and follow a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_47">SOLID</a>
implementation with explicit OOP code, like use of <code><a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_46">
interface</a></code>.</p> <p>To be fair, our UI generation from RTTI was more a proof of concept.<br />
It works on some existing projects, with a <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#SYNFILETMS_PNG">
nice rendering and user experience</a>.<br />
But the auto-generated UI result has a fixed layout. To customize it, we had to
include additional parameters, and introduce complexity in the generator.<br />
Good old RAD, with a SOA backend, or a MVC/MVVM design approach, as <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_465">
we propose for web servers</a>, would have been more productive.</p>
<p>We admit that some patterns like <a href="https://en.wikipedia.org/wiki/Aspect-oriented_programming">Aspect Oriented
Programing</a> could easily add some behavior to an existing code
base.<br />
The most obvious example is to write some attributes in code and enable method
execution logging.<br />
Sounds like a magical way of writing code.</p>
<p>Just like with our UI generation trick, adding logging could be just as easy
as writing an attribute to the class. Magic! Feature implemented in 5 minutes.
Now you can take a coffee for the whole day - task was budgeted for a
day.<br />
But logging content is poor (only method name/parameters). So you start adding
some attributes at the method level, to customize the log message. And you
start polluting your class. And, sadly, at attribute level, you do not have
access to all needed information (e.g. values or functions in other classes).
So you add an explicit log method call within the method code. And find it
convenient, finally.</p>
<p>In <em>mORMot</em>, you can have auto-generated logging (e.g. all SOA
interface calls, including input parameters and output values), in the log
system, or even in a log database (very convenient, in practice).<br />
This uses RTTI, and is indeed magic. Just a parameter, in the settings file,
and you obtain some additional tracking information. And you did not change the
business logic!<br />
But this automated logging is not enough. You should now have to write a lot of
manual logging, even at business layer, to track real execution context.</p>
<p>But all this run-time injected behavior make it harder to maintain and
debug.<br />
By looking at the code, you could not guess what is really executed.<br />
There is a lot of hidden mechanisms, which are not explicit any more.<br />
When you start to put some business logic using such injection/reflection
techniques, it eventually add complexity.<br />
We have seen unexpected rebounds in <a href="http://martinfowler.com/eaaDev/EventSourcing.html">Event-Sourcing</a>
architectures, using such hidden code injection to propagate events.<br />
I've seen recently servers becoming unstable because of real-time logging
issues: sending the log information remotely did create some kind of recursion,
due to the remote sending logging itself its own process!</p>
<p>Maintainable code should always try to make the implicit explicit.<br />
We would not make explicit all the low-level plumbing - otherwise we <a href="http://www.menuetos.net/">would code everything in assembler</a>.<br />
But we should better make all <em>logical process</em> explicit.<br />
This is where <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_54">
DDD excels</a>.</p>
<p>Using RTTI is a tricky business... which should be exceptional.<br />
Relying on RTTI at runtime, for usual code, smells like a break of <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_159">
Liskov Substitution Principle</a> (and the <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITL_158">
Open/Closed Principle</a>).<br />
Most of the time, code would be much cleaner when using proper OOP and
<code>interface</code> types definition, instead of relying on RTTI and
<code>TValue</code>.<br />
In <em>mORMot</em>, we achieve amazing results without the <a href="http://www.tindex.net/Language/EnhancedRTTI.html">enhanced RTTI</a>, as
introducing in Delphi 10. If it is available, we would use it. But it is not,
we would be able to <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_219">
find workarounds</a>, even for <a href="http://synopse.info/files/html/Synopse%20mORMot%20Framework%20SAD%201.18.html#TITLE_624">
compilation with FPC</a>.</p>
<p>And it is not my own opinion.<br />
In fact, Microsoft, in its newest orientation to "Native" compilation in .Net,
does advice to avoid RTTI (ab)use.<br />
Even at technical level, all this run-time "injection" and "inspection" stuff
is <a href="https://msdn.microsoft.com/en-us/library/dn600640">not so trendy
any more</a>...</p>
<p>RTTI is great, but it may be abused.<br />
There is good laziness, and unsafe laziness.<br />
If you add cross-cutting features (like logging, or authorization) via RTTI -
fine.<br />
But if you put some business logic via code injection - beware.<br />
This blog post just aims to state that, as always, RTTI use is a matter of
balance: design principles should rule, not code magic!</p>
<p>As <a href="https://plus.google.com/107629505609094966321">Markus</a>
commented, as one of the <a href="http://www.amazon.com/dp/0132350882">clean code principles</a>: "Someone
should be able to read the source code like a book and be able to figure out
what it is doing without comments."</p>
<p>Feedback may be <a href="https://plus.google.com/+ABouchez/posts/RVMdYpn4j2b">shared on
Google+</a>.<br />
<em>Edit</em>: I've added one more example, about logging via AOP.</p>