Embedding Lua in Torque2D

For the past couple of weeks, I’ve been experimenting with integrating lua into Torque2D. If you’ve used Torque2D before, you may wonder “Why stick lua in if Torque2D already has TorqueScript?”. There are several reasons, one main one being many times I’ve felt TorqueScript’s lack of advanced language primitives (e.g. lists, tables, closures) to be an annoyance.

Adding simple keywords to TorqueScript is relatively simple since it uses a simple Yacc grammar coupled with an AST. However the implementation of the AST and the design of the stack makes it quite difficult to implement anything more elaborate. Even worse, TorqueScript has no unit tests, so you can easily break something if you are not careful.

So rather than having the burden of maintaining TorqueScript itself, I decided to try integrating another scripting language.

Its important to note that TorqueScript itself is tightly coupled with the object system in Torque, so its worth understanding how it functions before replacing it.

Brief overview of TorqueScript

TorqueScript is a fairly simple scripting language backed by a byte-code interpreter. Internally all variables are strings, though they are simultaneously casted to float and integer types to speed up any math operations. All variable access goes through either a local or global hash table (apart from the name of the variable itself, nothing is cache’d).

Local variables are prefixed using the % sigil, and global variables are prefixed using $. Some variables are registered by the engine itself, in which case they directly reflect whatever variable they point to.

TorqueScript’s object system is rather ingenious. Objects are referenced by either a numeric id or a name with optional quotation marks. You can also use more elaborate search strings, e.g.

1234
"MyObject"
"RootGroup/MyObject"

May all refer to the same object.

As far as the scripting engine is concerned, objects have two things: a list of properties, and a namespace. Through the namespace functions are accessed. The class hierarchy is mirrored through the use of linked namespaces.

When accessing properties, it searches for the property in the class property list first, then looks in the objects property list. When accessing functions, it checks the classes current namespace for the function. If the function doesn’t exist in that namespace, it checks the parent namespace and so on  – though internally this check is cache’d using a hash table in the first namespace which contains the complete set of functions.

Example Namespace Linkage (1)

When you create an object you can also give it a name, in which case it will link in an extra namespace at the top of the chain allowing you to do cool things such as this:

new SimObject(FudgeCollector);

function FudgeCollector::collectFudge()
{
   echo("Collected fudge");
}

FudgeCollector.collectFudge(); // prints "Collected fudge

Namespaces in TorqueScript don’t necessarily have to be linked to a class. For instance you can create a new namespace at any time by defining a function.

function MyNewNamespace::doThis()
{
   return 123;
}

MyNewNamespace::doThis(); // No problem

Internally all objects are derived from a ConsoleObject class which handles fields, though most of the action happens in the SimObject class which handles assigning names and identifiers to objects, and well as setting up namespaces. There is also a SimSet class which adds nesting support.

Safe to say TorqueScript is a fairly loose and flexible language.

On to lua…

“So why lua?” you may ask. Several reasons. Firstly out of all the languages I tried, lua seemed to be the most flexible. This meant that it was reasonably easy to implement something which resembled how TorqueScript functions. There is no real class system, so you are free to invent your own using linked metatables. Variable access can easily be overridden so you can expose globals and object properties pretty easily.

It’s also ridiculously easy to bind lua code in torque using the current TorqueScript binding system. Instead of having to adding a lua thunk function for every function and method, it’s possible to use a single thunk function and use “upvalues” to point to the correct C function. Although ultimately it would be a better idea to use a specialised thunk for each exposed function, the fact that it isn’t strictly required reduces the complexity of integration  substantially while ensuring existing TorqueScript code can still function without the need of a rewrite.

Comparing lua to other embeddable scripting languages with class systems (e.g. ruby, python), they tend to be a bit stricter when it comes to object classes, making things such as the namespace name trick harder (if not, impossible) to implement correctly. i.e. you can’t just create an object on the fly and bind new methods to it, you need to make a proper class or make the functions references to delegates.

Lua does have a few of its own oddities – for example, arrays start at 1, and there is no switch statement. But these are a reasonable trade-off for flexibility.

On to the binding…

To be honest, it took me a while to get the right implementation which reflects the way things already work in TorqueScript. Since I wanted to keep the existing binding system, I had to write my own bindings from scratch using some of the existing C++ binding libraries as references to how things should work on a basic level.

For my implementation I ended up using userdata objects bound to class metatables. The userdata objects point directly to the instance of an object. Property access goes through one of two thunk methods (__index and __newindex), while functions are binded through one of 11 thunk methods depending on the return type and whether or not the function is meant to be used with an object. Functions are assigned to the relevant table in lua (e.g. SimObject methods go in the SimObject table, global functions go in Torque). These tables are linked together to reflect the linkage of each namespace in Torque.

Lua Namespace (1)

When looking up properties, the __index method first checks the relevant top namespace table in lua. If it doesn’t exist there, it checks the parent table, and so on. If nothing is found in the end, it assumes the key is meant to refer to a property and acts the same way as in TorqueScript when you get a property.

When creating objects, the equivalent lua reference is stored in a global table called InstanceTable. e.g. “InstanceTable[1234]”.

So for example the following lua script…

myObject = SimObject()
myObject.customValue = 'candy'

function SimObject:printInfo()
   Torque.echo(self:getId())
   Torque.echo(self.customValue)
end

myObject:printInfo()

Will print the id of the object “myObject” and the word “candy” in the console.

In the engine code, everything is abstracted into a LuaScriptEngine class which handles binding classes, registering objects and functions, and also executing code. There’s also an equivalent implementation of the “Con::executef” function for calling back into script. Unlike the TorqueScript executef, this implementation uses a list of ScriptStackValueRef objects which transparently handles pushing pushing values onto the lua stack with the desired type.

Unfortunately one thing I haven’t implemented as of yet is nested objects, where you can create an object within an object and have the script engine automatically add it to that objects group for you. However given that Torque2D uses TAML to serialise nested objects this feature can be safely ignored.

Conclusion

Screen Shot 2014-02-03 at 20.17.33

Looks like lua

So was it worth embedding lua into Torque2D? I’d say yes. At the very least I now know how deep the TorqueScript rabbit hole goes, making it easier to incorporate any of the other alternative embeddable scripting languages should they become a better fit for my needs.