Memory management: very high allocation traffic, very small live footprint
The mORMot 2 TFB server was tested on an Intel Core i5-13500 running Debian 13, with 10 asynchronous server threads and epoll, using wrk with up to 8192 concurrent connections.
The TFB server sample was launched as such:
$ ./raw -s 1 -t 10 THttpAsyncServer running on localhost:8080 num servers=1, threads per server=10, total threads=10, total logical CPU=20, accessible CPU=20, pinned=0, db=SQLite
For 512 connections, the raw /plaintext URI returned more than 1,200,000 requests per second:
$ wrk -c 512 -t 10 -d 5 http://localhost:8080/plaintext
Running 5s test @ http://localhost:8080/plaintext
10 threads and 512 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 426.33us 1.11ms 35.80ms 99.59%
Req/Sec 126.87k 6.65k 191.43k 86.25%
6423899 requests in 5.10s, 771.91MB read
Requests/sec: 1259565.32
Transfer/sec: 151.35MB
For this simple /json endpoint, mormot.core.fpcx64mm, libc malloc and mimalloc all reached roughly 740,000 requests/sec at 8192 connections. The older FPC 3.2.2 RTL memory manager was around 648,000 requests/sec, approximately 12% slower in this particular test.
$ wrk -c 8192 -t 10 -d 30 http://localhost:8080/json
Running 30s test @ http://localhost:8080/json
10 threads and 8192 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 10.66ms 1.22ms 81.92ms 94.53%
Req/Sec 76.17k 4.92k 142.86k 94.38%
22657711 requests in 30.06s, 3.08GB read
Requests/sec: 753740.66
Transfer/sec: 104.95MB
Thanks to very efficient buffer reuse, each /plaintext request consumes only two getmem (the URI and the headers), whereas /json needs three (one more for the JSON itself) - using real serialization via RTTI.
The more interesting difference was memory usage at process level. mormot.core.fpcx64mm maintained an unusually small resident footprint: during one 8192-connection test the server was around 24 MB RSS, compared with approximately 45–51 MB for libc and 71–80 MB for mimalloc. These are measurements from this particular environment rather than universal characteristics of the allocators, but they illustrate the benefit of an allocator designed around the allocation patterns of a high-performance server.
The allocation counters also show why looking only at the amount of memory allocated by an application can be misleading. Across repeated /json endpoint tests, mormot.core.fpcx64mm recorded approximately 188 million small allocations, representing about 14 GB of cumulative allocation traffic, while the current small-block footprint remained only about 159 KB. The allocator is therefore reusing these short-lived blocks extremely aggressively. For a REST server processing millions of requests, this distinction between cumulative allocation traffic and live memory is particularly important.
FPCMM_SERVER and FPCMM_BOOST produced essentially identical results for this workload. The additional BOOST options did not produce a measurable improvement, which is itself useful information: once the allocator is already well adapted to the application's allocation pattern, additional optimizations do not necessarily translate into higher HTTP throughput.
Thread smoothing: don't wake everybody just because work arrived
The other optimization addresses a different problem.
An asynchronous server typically has one thread responsible for polling sockets and a pool of worker threads responsible for processing pending connections. When new work arrives, the polling thread needs to wake workers. A naive implementation can easily become too aggressive: wake too many threads, let them compete for a small amount of work, put them back to sleep, and repeat.
mORMot's thread smoothing deliberately avoids this behavior. This algorithm was particularly tuned for REST workloads. A pool can remain small when requests are uniformly fast, but when some requests become slower, additional workers are progressively activated. The result is a smoother distribution of work without trying to synchronize the entire pool on every event.
The new POSIX wakeup path: CAS instead of waiting for a lock
The recent change in our repository makes this mechanism even less intrusive.
Previously, the wakeup path could contend on the lock protecting the worker state even though a wakeup itself does not require strict synchronization. The new implementation uses atomic counters as pending wakeup hints:
fWakeupOne, fWakeupEvents: cardinal;
The main poll/epoll thread does:
new := fOwner.fSockets.fRead.PollForPendingEvents(ms); if new <> 0 then fOwner.ThreadPollingWakeupEvents(new) // distribute those events else if fOwner.fSockets.fRead.fPending.Count <> 0 then fOwner.ThreadPollingWakeupOne; // scale up by waking a new thread
For example, the wakeup methods can now essentially do:
procedure TAsyncConnections.ThreadPollingWakeupOne;
begin
// wake up one thread after accept() on idle server or on slow REST process
if acoThreadSmooting in fOptions then
fThreadPollingLastWakeUpTix := mormot.core.os.GetTickCount64; // 16ms / 4ms
if LockedExc32(fWakeupOne, 1, 0) then // if not already notified
if fWakeupSafe.TryLock then
ThreadPollingWakeupLocked;
end;
procedure TAsyncConnections.ThreadPollingWakeupEvents(Events: integer);
begin
// after poll/epoll pending events
if acoThreadSmooting in fOptions then
begin
fThreadPollingLastWakeUpTix := mormot.core.os.GetTickCount64; // 16ms / 4ms
LockedAdd32(fWakeupEvents, Events); // up to ThreadPollingWakeupLoad events
end
else
LockedAdd32(fWakeupOne, Events); // default/legacy is one thread per event
if fWakeupSafe.TryLock then
ThreadPollingWakeupLocked;
end;
The important detail is the combination of CAS LockedExc32 LockedAdd32 and TLightLock.TryLock, with update of the OS ticks to identify slow process.
The CAS atomically consumes one pending wakeup request. If another thread has already consumed it, there is nothing to do. If the wakeup-management lock is currently busy, the caller simply leaves the advisory notification for another opportunity.
There is deliberately no blocking lock acquisition here. This is important because these functions are normally called by the main epoll thread, but they can also be called by worker threads themselves. A worker processing a slow request may decide that one of its siblings should be activated. It would be counterproductive for that worker to block merely because another thread is currently performing the very small wakeup-management operation.
The resulting principle is simple: If we can cheaply wake a worker, do it. If somebody else is already handling the wakeup, let them do it. If all workers are already active, do nothing.
There is no correctness dependency on the wakeup being performed immediately. At worst, a missed advisory wakeup costs a few CPU cycles or a small amount of latency; it cannot deadlock the server.
On POSIX this is now a particularly lightweight path: atomic counters collect wakeup hints, TryLock avoids waiting, and the actual operating-system event notification is performed only after the worker selection has been made. The selected workers are signalled outside the management lock.
Windows follows a different architecture because the asynchronous server uses IOCP, whose completion queue already provides its own kernel-managed work distribution.
Reusing already-running workers
Since mormot.core.os.GetTickCount64 has a 4ms typical resolution on POSIX, the web server is able to monitor the REST load, and isolate longstanding/slow requests.
In acoThreadSmooting mode, inside the protected wakeup section, a first pass looks for workers that are already running and are expected to become available soon:
if (wuPossible in t.fWakeUp) or // not running (t.fThreadPollingLastWakeUpEvents <= 0) or // no spare process (t.fThreadPollingLastWakeUpTix <> tix) then // slow process continue;
A worker with spare expected capacity effectively consumes some of the pending event pressure without requiring another thread to be awakened. This is the core of the smoothing algorithm.
For example, imagine:
- 100 pending events
- 10 workers already running
If several of those workers are processing quickly, there may be no reason to make another ten threads runnable. The algorithm therefore attempts to reuse the concurrency that already exists. Only when the estimated existing capacity is insufficient does it start selecting sleeping workers.
Slow requests provide feedback
The most interesting part of the design is that worker threads themselves can request additional concurrency.
A worker processing a pending read checks the smoothing timestamp:
if (acoThreadSmooting in fOwner.Options) and (fThreadPollingLastWakeUpTix <> fOwner.fThreadPollingLastWakeUpTix) and not (wuFromSlowProcess in fWakeUp) then begin include(fWakeUp, wuFromSlowProcess); fOwner.ThreadPollingWakeupOne; end;
The idea is simple. A worker discovers that its processing loop has taken long enough to cross into a new smoothing interval.
That is evidence that the currently running workers may be insufficient.
Is this unusual?
Most high-performance web servers have sophisticated worker-pool and event-distribution mechanisms, but this particular combination is less common: event-driven socket polling, a fixed asynchronous worker pool, deliberately conservative worker activation, and advisory wakeups based on the observed processing speed of individual workers.
Servers such as Nginx traditionally emphasize event-driven processing with relatively few workers, while systems based on thread pools generally rely on a work queue and its associated worker wakeup mechanism. Frameworks such as Tokio and other asynchronous runtimes also have sophisticated scheduling and worker-stealing strategies.
mORMot's approach is somewhat different because the objective is not simply to maximize the number of runnable workers. The server tries to keep the pool smooth: fast requests should not cause a storm of worker wakeups, while slow requests should gradually cause additional workers to become active.
That distinction matters for a REST server whose request cost is highly variable. A benchmark consisting entirely of identical, tiny requests can favor aggressive scheduling. A real API may simultaneously contain trivial cache lookups, JSON processing, authentication, filesystem operations, and database queries. The optimal worker population is therefore not constant.
Conclusion: stability matters as much as peak RPS
The goal of the acoThreadSmooting mode is not to win a synthetic benchmark by waking the maximum possible number of threads. It is to keep a REST server responsive under changing workloads.
Fast requests naturally keep the existing workers busy. When processing becomes slower, additional workers are progressively woken. When workers are already active, no unnecessary wakeup is generated. And with the new CAS-based wakeup counters and non-blocking TryLock, even the notification path itself does not become a source of contention.
Combined with mormot.core.fpcx64mm's very low live allocation footprint, this gives mORMot 2 an interesting performance profile: high throughput without requiring an aggressively growing number of runnable threads or a large amount of retained memory.
For real-world REST services, that balance can be more valuable than a small improvement in peak requests/sec. The important property is that the server remains predictable when the workload stops being predictable. A mORMot 2 REST server is expected to run continuously, and adapt itself to the load, in a sustainable and stable way.
