Synopse Open Source

Especially if you are dealing with a lot of data, you often need a way to identify if a value is available in a value set.
A typical use case is if you have data sharded among several nodes, and you want to avoid asking each node for each incoming request.

A naive approach could be to store all data in a memory list.
But here we are really talking about a lot of data, and it would simply not fit into a memory list.

We may say that it is the purpose of a database to maintain such a list.
So you start a good CREATE TABLE on your RDBMS with a single indexed primary key column, fill it with your data, and run a proper SELECT.
But it takes a lot of storage, insertion is slow, and this database becomes a bottleneck.

Then you consider using some NoSQL database like Redis.
It is faster than a RDBMS, but it tends to use a lot of memory, and it is still resource consuming to update the values.

No comes Bloom Filter magic.
It allows to store the presence of high-number of values with a small memory space, with a predefined ratio of potential false positives.

We just introduced a TSynBloomFilter class in our Open Source mORMot framework trunk, which features an optimized and self-tuning Bloom Filter storage, with potential low-bandwidth synchronization over the wire.

In any modern application, especially on Client/Server nTier architecture as our little mORMot offers, we often have to persist some private keys in a safe way.
Problem with such keys is that they consist in small amount of bytes (typically 16 or 32 bytes), easy to be left somewhere in disk or memory.
Given the abilities of recent forensic data recovery methods, data can't be destroyed on magnetic or flash storage media reliably.

We have just added to our SynCrypto OpenSource library the Anti-forensic Information Splitter algorithm, as proposed in TKS1, and implemented in the LUKS standard.
LUKS is the de-facto standard of platform-independent standard on-disk format for use in various tools.

Everyone knows about the pascal random() function.
It returns some numbers, using a linear congruential generator, with a multiplier of 134775813, in its Delphi implementation.
It is fast, but not really secure. Output is very predictable, especially if you forgot to execute the RandSeed() procedure.

In real world scenarios, safety always requires random numbers, e.g. for key/nonce/IV/salt/challenge generation.
The less predictable, the better.
We just included a Cryptographically Secure Pseudo-Random Number Generator (CSPRNG) into our SynCrypto.pas unit.
The TAESPRNG class would use real system entropy to generate a sequence of pseudorandom bytes, using AES-256, so returning highly unpredictable content.