Synopse Open Source

One language, or all languages

The implementation is built around two complementary classes.

TLanguageFile represents one language. It contains a thread-safe dictionary mapping the original English text to its translation, together with optional date and date/time formatting settings. It can load translations from GNU gettext .po and .mo files, as well as INI, YAML and JSON variants.

TLanguageFiles is the application-level container: it manages all the TLanguageFile instances for all languages. It can load a complete directory of language files, expose the languages currently available, and provide a default language. Most importantly for server applications, the current language can be selected per thread.

This makes the model particularly natural for an HTTP server: load all translations once at startup, then call SetThreadLanguage() when a request starts, based for example on a cookie, URI parameter or user preference. Translation calls automatically use the language selected for the current request.

Of course, if you don't like the per-thread context trick, you could just store the proper TLanguageFile in your server-side-connection state instead. And since TLanguageFiles.Language lookup is very fast, so you can just store the per-connection TLanguage enumeration if you prefer.

A native .m18n deployment format

TLanguageFiles also goes beyond simply loading individual translation files.

Because it inherits from TObjectStore, the complete collection of language tables can be persisted as a compressed binary representation. mORMot defines .m18n as its canonical extension for this format, and the same binary data can be embedded directly into an executable as a resource.

This creates a useful separation between development and deployment. Translators can work with familiar .po or .json files, while developers can package all the application's translations into one compact .m18n file or executable resource. A multilingual application can therefore be deployed with a single self-contained translation database instead of a collection of separate files.

resourcestring support on both Delphi and FPC

Another important aspect is the support for Delphi/FPC resourcestring values.

TLanguageFiles.TranslateResourceStrings() can translate the whole executable resourcestring table using the same language dictionaries. The original resourcestring values are expected to be English text and therefore naturally become the translation keys.

This is implemented according to the actual RTL mechanisms of each compiler rather than pretending that Delphi and FPC handle resourcestrings identically. On Delphi, mORMot redirects LoadResString() through its own translation/cache mechanism. On FPC, resourcestring values are kept in per-unit writable tables, which mORMot updates through objpas.SetResourceStrings() once when the method is called. The language can consequently be changed at runtime, with the original English values restored when no translation language is selected.

This is a particularly nice feature for existing Delphi and FPC code: applications can continue using the familiar resourcestring mechanism, while the actual translations are managed by the same TLanguageFiles infrastructure used by the rest of the application.

Designed to fit mORMot

The translation engine can also be connected to several existing mORMot hooks, including Mustache translation tags, framework caption translation and language-specific date/time formatting. GNU gettext .po and .mo remain supported, so existing translation workflows can be reused rather than replaced.

The difference from many traditional Delphi/FPC localization libraries is therefore mainly integration and scope. It is not intended to be a large desktop form-localization framework. Instead, it provides a small common translation engine in the mORMot core, usable by both Delphi and FPC and especially convenient for multilingual server applications.

The resulting workflow is simple:

1. load all languages once
2. select the language per request/thread
3. translate through the shared tables
4. optionally deploy everything as one compressed .m18n resource

This combination of gettext compatibility, native resourcestring support on both Delphi and FPC, per-thread language selection, framework integration and a compact multi-language .m18n deployment format makes mormot.core.i18n a useful addition to the mORMot 2 core.

See mormot.core.i18n.pas for the implementation unit.

Show me some code

Here is an extract from the regression tests:

resourcestring
  MyResource = 'Hello';

  // each TLanguageFiles instance has an internal name
  langs := TLanguageFiles.Create('ProjectV1');
  try
    // manuall add some entries, here as JSON
    CheckEqual(langs.AddFromJson(lngFrench, '{"Hello":"Bonjour"}'), 1);
    CheckEqual(langs.AddFromJson(lngChinese, '{"Hello":"NiHao"}'), 1);

    // per-thread selection
    TLanguageFiles.SetThreadLanguage(lngFrench);
    Check(TLanguageFiles.ThreadLanguage = lngFrench);
    Check(langs.Current = langs.Language[lngFrench]);
    u := 'Hello';
    Check(langs.Translate(u));
    CheckEqual(u, 'Bonjour');

    // fallback to DefaultLanguage when no thread language is set
    TLanguageFiles.SetThreadLanguage(lngUndefined);
    langs.DefaultLanguage := lngChinese;
    Check(langs.Current = langs.Language[lngChinese]);
    u := 'Hello';
    Check(langs.Translate(u));
    CheckEqual(u, 'NiHao');

    // Mustache {{"text}} channel end-to-end
    TLanguageFiles.SetThreadLanguage(lngFrench);
    m := TSynMustache.Parse('{{"Hello}} {{name}}!');
    CheckEqual(m.Render(_ObjFast(['name', 'world']), nil, nil,
      langs.TranslateString), 'Bonjour world!');

    // trigger globally resourcestring translations
    Check(MyResource = 'Hello');
    langs.TranslateResourceStrings(lngFrench);
    Check(MyResource = 'Bonjour');
  finally
    langs.Free;
  end;

More is available in the TTestCoreProcess._i18n method.

Two Breaking Changes

During the development of this feature, two breaking changes were introduced.
We would like to document them:

1. The TObjectStore.fReader field is now a PFastReader and not a TFastReader. You may need to fix compilation by writing fReader^ in some places.
2. The { { " English text } } Mustache tag now properly escape into clean HTML - as it should have been. But if you already escaped manually the text in your resources, you may need to use plain UTF-8 text instead.