Scripting abilities of mORMot

As a Delphi framework, mORMot premium language support is for the object pascal language. But it could be convenient to have some part of your software not fixed within the executable. In fact, once the application is compiled, execution flow is written in stone: you can't change it, unless you modify the Delphi source and compile it again. Since mORMot is Open Source, you can ship the whole source code to your customers or services with no restriction, and diffuse your own code as pre-compiled .dcu files, but your end-user will need to have a Delphi IDE installed (and paid), and know the Delphi language.

This is when scripting does come on the scene.
For instance, scripting may allow to customize an application behavior for an end-user (i.e. for reporting), or let a domain expert define evolving appropriate business rules - following Domain Driven Design.

If your business model is to publish a core domain expertise (e.g. accounting, peripheral driving, database model, domain objects, communication, AJAX clients...) among several clients, you will sooner or later need to adapt your application to one or several of your customers. There is no "one exe to rule them all". Maintaining several executables could become a "branch-hell". Scripting is welcome here: speed and memory critical functionality (in which mORMot excels) will be hard-coded within the main executable, then everything else could be defined in script.

There are plenty of script languages available.
We considered DelphiWebScript which is well maintained and expressive (it is the code of our beloved SmartMobileStudio), but is not very commonly used. We still want to include it in the close future.
Then LUA defines a light and versatile general-purpose language, dedicated to be embedded in any application. Sounds like a viable solution: if you can help with it, your contribution is welcome!
We did also take into consideration Python and Ruby but both are now far from light, and are not meant to be embedded, since they are general-purpose languages, with a huge set of full-featured packages.

Then, there is JavaScript:

  • This is the World Wide Web assembler. Every programmer in one way or another knows JavaScript.
  • JavaScript can be a very powerful language - see Crockford's book "JavaScript - The Good Parts";
  • There are a huge number of libraries written in JavaScript: template engines (jade, mustache...), SOAP and LDAP clients, and many others (including all node.js libraries of course);
  • It was the base for some strongly-typed syntax extensions, like CoffeScript, TypeScript, Dart;
  • In case of AJAX / Rich Internet Application we can directly share part of logic between client and server (validation, template rendering...) without any middle-ware;
  • One long time mORMot's user (Pavel, aka mpv) already integrated SpiderMonkey to mORMot's core. His solution is used on production to serve billion of requests per day, with success. We officially integrated his units.
    Thanks a lot, Pavel!

As a consequence, mORMot introduced direct JavaScript support via SpiderMonkey.
It allows to:

  • Execute Delphi code from JavaScript - including our ORM or SOA methods, or even reporting;
  • Consume JavaScript code from Delphi (e.g. to define and customize any service or rule, or use some existing .js library);
  • Expose JavaScript objects and functions via a TSMVariant custom variant type: it allows to access any JavaScript object properties or call any of its functions via late-binding, from your Delphi code, just as if it was written in native Object-Pascal;
  • Follow a classic synchronous blocking pattern, rooted on mORMot's multi-thread efficient model, easy to write and maintain;
  • Handle JavaScript or Delphi objects as UTF-8 JSON, ready to be published or consumed via mORMot's RESTful Client-Server remote access.

SpiderMonkey integration

A powerful JavaScript engine

SpiderMonkey, the Mozilla JavaScript engine, can be embedded in your mORMot application. It could be used on client side, within a Delphi application (e.g. for reporting), but the main interest of it may be on the server side.

The word JavaScript may bring to mind features such as event handlers (like onclick), DOM objects, window.open, and XMLHttpRequest.
But all of these features are actually not provided by the SpiderMonkey engine itself.

SpiderMonkey provides a few core JavaScript data types—numbers, strings, Arrays, Objects, and so on—and a few methods, such as Array.push. It also makes it easy for each application to expose some of its own objects and functions to JavaScript code. Browsers expose DOM objects. Your application will expose objects that are relevant for the kind of scripts you want to write. It is up to the application developer to decide what objects and methods are exposed to scripts.

Direct access to the SpiderMonkey API

The SynSMAPI.pas unit is a tuned conversion of the SpiderMonkey API, providing full ECMAScript 5 support and JIT.
You could take a look at the full description of this low-level API.

But the SynSM.pas unit will encapsulate most of it into higher level Delphi classes and structures (including a custom variant type), so you probably won't need to use SynSMAPI.pas directly in your code:

Type Description
TSMEngineManager main access point to the SpiderMonkey per-thread scripting engines
TSMEngine implements a Thread-Safe JavaScript engine instance
TSMObject wrap a JavaScript object and its execution context
TSMValue wrap a JavaScript value, and interfaces it with Delphi types
TSMVariant /
TSMVariantData		 define a custom variant type, for direct access to any JavaScript object, with late-binding

We will see know how to work with all those classes.

Execution scheme

The SpiderMonkey JavaScript engine compiles and executes scripts containing JavaScript statements and functions. The engine handles memory allocation for the objects needed to execute scripts, and it cleans up—garbage collects—objects it no longer needs.

In order to run any JavaScript code in SpiderMonkey, an application must have three key elements:

  1. A JSRuntime,
  2. A JSContext,
  3. And a global JSObject.

A JSRuntime, or runtime, is the space in which the JavaScript variables, objects, scripts, and contexts used by your application are allocated. Every JSContext and every object in an application lives within a JSRuntime. They cannot travel to other runtimes or be shared across runtimes.

A JSContext, or context, is like a little machine that can do many things involving JavaScript code and objects. It can compile and execute scripts, get and set object properties, call JavaScript functions, convert JavaScript data from one type to another, create objects, and so on.

Lastly, the global JSObject is a JavaScript object which contains all the classes, functions, and variables that are available for JavaScript code to use. Whenever a web browser code does something like window.open("http://www.mozilla.org/"), it is accessing a global property, in this case window. SpiderMonkey applications have full control over what global properties scripts can see.

Every SpiderMonkey instance starts out every execution context by creating its JSRunTime, JSContext instances, and a global JSObject. It populates this global object with the standard JavaScript classes, like Array and Object. Then application initialization code will add whatever custom classes, functions, and variables (like window) the application wants to provide; it may be, for a mORMot server application, ORM access or SOA services consumption and/or implementation.

Each time the application runs a JavaScript script (using, for example, JS_EvaluateScript), it provides the global object for that script to use. As the script runs, it can create global functions and variables of its own. All of these functions, classes, and variables are stored as properties of the global object.

Creating your execution context

The main point about those three key elements is that, in the current implementation pattern of SpiderMonkey, runtime, context or global objects are not thread-safe.

Therefore, in the mORMot's use of this library, each thread will have its own instance of each.

In the SynSM.pas unit, a TSMEngine class has been defined to give access to all those linked elements:

  TSMEngine = class
  ...
    /// access to the associated global object as a TSMVariant custom variant
    // - allows direct property and method executions in Delphi code, via
    // late-binding
    property Global: variant read FGlobal;
    /// access to the associated global object as a TSMObject wrapper
    // - you can use it to register a method
    property GlobalObject: TSMObject read FGlobalObject;
    /// access to the associated global object as low-level PJSObject
    property GlobalObj: PJSObject read FGlobalObject.fobj;
    /// access to the associated execution context
    property cx: PJSContext read fCx;
    /// access to the associated execution runtime
    property rt: PJSRuntime read frt;
  ...

Our implementation will define one Runtime, one Context, and one global object per thread, i.e. one TSMEngine class instance per thread.

A JSRuntime, or runtime, is created for each TSMEngine instance. In practice, you won't need access to this value, but rely either on a JSContext or directly a TSMEngine.

A JSContext, or context, will be the main entry point of all SpiderMonkey API, which expect this context to be supplied as parameter. In mORMot, you can retrieve the running TSMEngine from its context by using the function TSMObject.Engine: TSMEngine - in fact, the engine instance is stored in the private data slot of each JSContext.

Lastly, the TSMEngine's global object contains all the classes, functions, and variables that are available for JavaScript code to use. For a mORMot server application, ORM access or SOA services consumption and/or implementation, as stated above.

You can note that there are several ways to access this global object instance, from high-level to low-level JavaScript object types. The TSMEngine.Global property above is in fact a variant. Our SynSM.pas unit defines in fact a custom variant type, identified as the TSMVariant class, able to access any JavaScript object via late-binding, for both variables and functions:

  engine.Global.MyVariable := 1.0594631;
  engine.Global.MyFunction(1,'text');

Most web applications only need one runtime, since they are running in a single thread - and (ab)use of callbacks for non-blocking execution. But in mORMot, you will have one TMSEngine instance per thread, using the TSMEngineManager.ThreadSafeEngine method. Then all execution may be blocking, without any noticeable performance issue, since the whole mORMot threading design was defined to maximize execution resources.

Blocking threading model

This threading model is the big difference with other server-side scripting implementation schemes, e.g. the well-known node.js solution.

Multi-threading is not evil, when properly used. And thanks to the mORMot's design, you won't be afraid of writing blocking JavaScript code, without any callbacks. In practice, those callbacks are what makes most JavaScript code difficult to maintain.

On the client side, i.e. in a web browser, the JavaScript engine only uses one thread per web page, then uses callbacks to defer execution of long-running methods (like a remote HTTP request).
If fact, this is one well identified performance issue of modern AJAX applications. For instance, it is not possible to perform some intensive calculation in JavaScript, without breaking the web application responsiveness: you have to split your computation task in small tasks, then let the JavaScript code pause, until a next piece of computation could be triggered... On server side, node.js allows to define Fibers and Futures - but this is not available on web clients. Some browsers did only start to uncouple the JavaScript execution thread from the HTML rendering thread - and even this is hard to fix... we reached here the limit of a technology rooted in the 80's...

On the server side, node.js did follow this pattern, which did make sense (it allows to share code with the client side, with some name-space tricks), but it is also IMHO a big waste of resources. Why should we stick to an implementation pattern inherited from the 80's computing model, when all CPUs were mono core, and threads were not available?

The main problem when working with one single thread, is that your code shall be asynchronous. Soon or later, you will face a syndrome known as "Callback Hell". In short, you are nesting anonymous functions, and define callbacks. The main issue, in addition to lower readability and being potentially sunk into function() nesting, is that you just lost the JavaScript exception model. In fact, each callback function has to explicitly check for the error (returned as a parameter in the callback function), and handle it.

Of course, you can use so-called Promises and some nice libraries - mainly async.js.
But even those libraries add complexity, and make code more difficult to write. For instance, consider the following non-blocking/asynchronous code:

getTweetsFor("domenic") // promise-returning function
  .then(function (tweets) {
    var shortUrls = parseTweetsForUrls(tweets);
    var mostRecentShortUrl = shortUrls[0];
    return expandUrlUsingTwitterApi(mostRecentShortUrl); // promise-returning function
  })
  .then(httpGet) // promise-returning function
  .then(
    function (responseBody) {
      console.log("Most recent link text:", responseBody);
    },
    function (error) {
      console.error("Error with the twitterverse:", error);
    }
  );

Taken from this web site.

This kind of code will be perfectly readable for a JavaScript daily user, or someone fluent with functional languages.

But the following blocking/synchronous code may sound much more familiar, safer and less verbose, to most Delphi / Java / C# programmer:

try {
  var tweets = getTweetsFor("domenic"); // blocking
  var shortUrls = parseTweetsForUrls(tweets);
  var mostRecentShortUrl = shortUrls[0];
  var responseBody = httpGet(expandUrlUsingTwitterApi(mostRecentShortUrl)); // blocking x 2
  console.log("Most recent link text:", responseBody);
} catch (error) {
  console.error("Error with the twitterverse: ", error);
}

Thanks to the blocking pattern, it becomes obvious that code readability and maintainability is as high as possible, and error detection is handled nicely via JavaScript exceptions, and a global try .. catch.

Last but not least, debugging blocking code is easy and straightforward, since the execution will be linear, following the code flow.

Upcoming ECMAScript 6 should go even further thanks to the yield keyword and some task generators - see taskjs - so that asynchronous code may become closer to the synchronous pattern. But even with yield, your code won't be as clean as with plain blocking style.

In mORMot, we did choose to follow an alternate path, i.e. write blocking synchronous code. Sample above shows how easier it is to work with. If you use it to define some huge business logic, or let a domain expert write the code, blocking syntax is much more straightforward.

Of course, mORMot allows you to use callbacks and functional programming pattern in your JavaScript code, if needed. But by default, you are allowed to write KISS blocking code.

Interaction with existing code

Within mORMot units, you can mix Delphi and JavaScript code by two ways:

  • Either define your own functions in Delphi code, and execute them from JavaScript; 
  • Or define your own functions in JavaScript code (including any third-party library), and execute them from Delphi.

Like for other part of our framework, performance and integration has been tuned, to follow our KISS way.

You can take a look at "22 - JavaScript HTTPApi web server\JSHttpApiServer.dpr" sample for reference code.

Proper engine initialization

As was previously stated, the main point to interface the JavaScript engine is to register all methods when the TSMEngine instance is initialized.

For this, you set the corresponding OnNewEngine callback event to the main TSMEngineManager instance.
See for instance, in the sample code:

constructor TTestServer.Create(const Path: TFileName);
begin
  ...
  fSMManager := TSMEngineManager.Create;
  fSMManager.OnNewEngine := DoOnNewEngine;
  ...

In DoOnNewEngine, you will initialize every newly created TSMEngine instance, to register all needed Delphi methods and prepare access to JavaScript via the runtime's global JSObject.

Then each time you want to access the JavaScript engine, you will write for instance:

function TTestServer.Process(Ctxt: THttpServerRequest): cardinal;
var engine: TSMEngine;
...
   engine := fSMManager.ThreadSafeEngine;
...  // now you can use engine, e.g. engine.Global.someMethod()

Each thread of the HTTP server thread-pool will be initialized on the fly if needed, or the previously initialized instance will be quickly returned otherwise.

Once you have the TSMEngine instance corresponding to the current thread, you can launch actions on its global object, or tune its execution.
For instance, it could be a good idea to check for the JavaScript VM's garbage collection:

function TTestServer.Process(Ctxt: THttpServerRequest): cardinal;
...
   engine := fSMManager.ThreadSafeEngine;
   engine.MaybeGarbageCollect; // perform garbage collection if needed
...

We will now find out how to interact between JavaScript and Delphi code.

Calling Delphi code from JavaScript

In order to call some Delphi method from JavaScript, you will have to register the method.
As just stated, it is done by setting a callback within TSMEngineManager.OnNewEngine initialization code. For instance:

procedure TTestServer.DoOnNewEngine(const Engine: TSMEngine);
...
  // add native function to the engine
  Engine.RegisterMethod(Engine.GlobalObj,'loadFile',LoadFile,1);
end;

Here, the local LoadFile() method is implemented as such in native code:

function TTestServer.LoadFile(const This: variant; const Args: array of variant): variant;
begin
  if length(Args)<>1 then
    raise Exception.Create('Invalid number of args for loadFile(): required 1 (file path)');
  result := AnyTextFileToSynUnicode(Args[0]);
end;

As you can see, this is perfectly easy to follow.
Its purpose is to load a file content from JavaScript, by defining a new global function named loadFile().
Remember that the SpiderMonkey engine, by itself, does not know anything about file system, database or even DOM. Only basic objects were registered, like arrays. We have to explicitly register the functions needed by the JavaScript code.

In the above code snippet, we used the TSMEngineMethodEventVariant callback signature, marshaling variant values as parameters. This is the easiest method, with only a slight performance impact.

Such methods have the following features:

  • Arguments will be transmitted from JavaScript values as simple Delphi types (for numbers or text), or as our custom TSMVariant type for JavaScript objects, which allows late-binding;
  • The This: variant first parameter map the "callee" JavaScript object as a TSMVariant custom instance, so that you would be able to access the other object's methods or properties directly via late-binding;
  • You can benefit of the JavaScript feature of variable number of arguments when calling a function, since the input arguments is a dynamic array of variant;
  • All those registered methods are registered in a list maintained in the TSMEngine instance, so it could be pretty convenient to work with, in some cases;
  • You can still access to the low-level JSObject values of any the argument, if needed, since they can be trans-typed to a TSMVariantData instance (see below) - so you do not loose any information;
  • The Delphi native method will be protected by the mORMot wrapper, so that any exception raised within the process will be catch and transmitted as a JavaScript exception to the runtime;
  • There is also an hidden set of the FPU exception mask during execution of native code (more on it later on) - you should not bother on it here.

Now consider how you should have written the same loadFile() function via low-level API calls.

First, we register the callback:

procedure TTestServer.DoOnNewEngine(const Engine: TSMEngine);
...
  // add native function to the engine
 Engine.GlobalObject.DefineNativeMethod('loadFile', nsm_loadFile, 1);
end;

Then its implementation:

function nsm_loadFile(cx: PJSContext; argc: uintN; vp: Pjsval): JSBool; cdecl;
var in_argv: PjsvalVector;
    filePath: TFileName;
begin
  TSynFPUException.ForDelphiCode;
  try
    if argc<>1 then
      raise Exception.Create('Invalid number of args for loadFile(): required 1 (file path)');
    in_argv := JS_ARGV(cx,vp);
    filePath := JSVAL_TO_STRING(in_argv[0]).ToString(cx);
    JS_SET_RVAL(cx, vp, cx^.NewJSString(AnyTextFileToSynUnicode(filePath)).ToJSVal);
    Result := JS_TRUE;
  except
    on E: Exception do begin // all exceptions MUST be catched on Delphi side
      JS_SET_RVAL(cx, vp, JSVAL_VOID);
      JSError(cx, E);
      Result := JS_FALSE;
    end;
  end;
end;

As you can see, this nsm_loadFile() function is much more difficult to follow:

  • Your code shall begin with a cryptic TSynFPUException.ForDelphiCode instruction, to protect the FPU exception flag during execution of native code (Delphi RTL expects its own set of FPU exception mask during execution, which does not match the FPU exception mask expected by SpiderMonkey);
  • You have to explicitly catch any Delphi exception which may raise, with a try...finally block, and marshal them back as JavaScript errors;
  • You need to do a lot of manual low-level conversions - via JS_ARGV() then e.g. JSVAL_TO_STRING() macros - to retrieve the actual values of the arguments;
  • And the returning function is to be marshaled by hand - see the JS_SET_RVAL() line.

Since the variant-based callback has only a slight performance impact (nothing measurable, when compared to the SpiderMonkey engine performance itself), and still have access to all the transmitted information, we strongly encourage you to use this safer and cleaner pattern, and do not define any native function via low-level API.

Note that there is an alternate JSON-based callback, which is not to be used in your end-user code, but will be used when marshaling to JSON is needed, e.g. when working with mORMot's ORM or SOA features.

TSMVariant custom type

As stated above, the SynSM.pas unit defines a TSMVariant custom variant type. It will be used by the unit to marshal any JSObject instance as variant.

Via the magic of late-binding, it will allow access of any JavaScript object property, or execute any of its functions. Only with a slightly performance penalty, but with much better code readability than with low-level access of the SpiderMonkey API.

The TSMVariantData memory structure can be used to map such a TSMVariant variant instance. In fact, the custom variant type will store not only the JSObject value, but also its execution context - i.e. JSContext - so is pretty convenient to work with.

For instance, you may be able to write code as such:

function TMyClass.MyFunction(const This: variant; const Args: array of variant): variant;
var global: variant;
begin
  TSMVariantData(This).GetGlobal(global);
  global.anotherFunction(Args[0],Args[1],'test');
  // same as:
  global := TSMVariantData(This).SMObject.Engine.Global;
  global.anotherFunction(Args[0],Args[1],'test');
  // but you may also write directly:
  with TSMVariantData(This).SMObject.Engine do
    Global.anotherFunction(Args[0],Args[1],'test');
  result := AnyTextFileToSynUnicode(Args[0]);
end;

Here, the This custom variant instance is trans-typed via TSMVariantData(This) to access its internal properties.

Calling JavaScript code from Delphi

In order to execute some JavaScript code from Delphi, you should first define the JavaScript functions to be executed.
This shall take place within TSMEngineManager.OnNewEngine initialization code:

procedure TTestServer.DoOnNewEngine(const Engine: TSMEngine);
var showDownRunner: SynUnicode;
begin
  // add external JavaScript library to engine (port of the Markdown library)
  Engine.Evaluate(fShowDownLib, 'showdown.js');
  // add the bootstrap function calling loadfile() then showdown's makeHtml()
  showDownRunner := AnyTextFileToSynUnicode(ExeVersion.ProgramFilePath+'showDownRunner.js');
  Engine.Evaluate(showDownRunner, 'showDownRunner.js');
  ...

This code first evaluates (i.e. "executes") a general-purpose JavaScript library contained in the showdown.js file, available in the sample executable folder. This is an open source library able to convert any Markdown markup into HTML. Plain standard JavaScript code.

Then we evaluate (i.e. "execute") a small piece of JavaScript code, to link the makeHtml() function of the just defined library with our loadFile() native function:

function showDownRunner(pathToFile){
  var src = loadFile(pathToFile);            // call Delphi native code
  var converter = new Showdown.converter();  // get the Showdown converted
  return converter.makeHtml(src);            // convert .md content into HTML via showdown.js
}

Now we have a new global function showDownRunner(pathToFile) at hand, ready to be executed by our Delphi code:

function TTestServer.Process(Ctxt: THttpServerRequest): cardinal;
var content: variant;
    FileName, FileExt: TFileName;
    engine: TSMEngine;
  ...
  if FileExt='.md' then begin
  ...
    engine := fSMManager.ThreadSafeEngine;
  ...
    content := engine.Global.showDownRunner(FileName);
  ...

As you can see, we access the function via late-binding. Above code is perfectly readable, and we call here a JavaScript function and a whole library as natural as if it was native code.

Without late-binding, we may have written, accessing not the Global TSMVariant instance, but the lower level GlobalObject: TSMObject property:

  ...
    content := engine.GlobalObject.Run('showDownRunner',[SynUnicode(FileName)]);
  ...

It is up to you to choose which kind of code you prefer, but late-binding is worth considering.

Next step on our side is to directly allow access to mORMot's ORM and SOA features, including interface-based services.
Feedback is welcome on our forum, as usual.