After realizing I’ve not posted a thing on here for over 2 years, I figured now is about time for an update.
For the past 2 years i’ve mainly been working with Mode7 Games on their upcomming title Frozen Synapse 2 as well as their previous, Frozen Cortex. As previous subject material might suggest, both of these products have involved working with technology built up from Torque 3D.
The indie game development scene has certainly been interesting, though quite demanding in terms of technical challenges - even moreso compared to your typical web development backend job. I hope at some point to write up about some of these challenges on this blog.
Old content has now been pruned, due to it being mostly out of date. Old site content pre-2012 can now be found at the vault.
For the past couple of weeks, I’ve been experimenting with integrating lua into Torque2D. If you’ve used Torque2D before, you may wonder “Why stick lua in if Torque2D already has TorqueScript?”. There are several reasons, one main one being many times I’ve felt TorqueScript’s lack of advanced language primitives (e.g. lists, tables, closures) to be an annoyance.
Adding simple keywords to TorqueScript is relatively simple since it uses a simple Yacc grammar coupled with an AST. However the implementation of the AST and the design of the stack makes it quite difficult to implement anything more elaborate. Even worse, TorqueScript has no unit tests, so you can easily break something if you are not careful.
So rather than having the burden of maintaining TorqueScript itself, I decided to try integrating another scripting language.
Its important to note that TorqueScript itself is tightly coupled with the object system in Torque, so its worth understanding how it functions before replacing it.
Brief overview of TorqueScript
TorqueScript is a fairly simple scripting language backed by a byte-code interpreter. Internally all variables are strings, though they are simultaneously casted to float and integer types to speed up any math operations. All variable access goes through either a local or global hash table (apart from the name of the variable itself, nothing is cache’d).
Local variables are prefixed using the % sigil, and global variables are prefixed using $. Some variables are registered by the engine itself, in which case they directly reflect whatever variable they point to.
TorqueScript’s object system is rather ingenious. Objects are referenced by either a numeric id or a name with optional quotation marks. You can also use more elaborate search strings, e.g.
May all refer to the same object.
As far as the scripting engine is concerned, objects have two things: a list of properties, and a namespace. Through the namespace functions are accessed. The class hierarchy is mirrored through the use of linked namespaces.
When accessing properties, it searches for the property in the class property list first, then looks in the objects property list. When accessing functions, it checks the classes current namespace for the function. If the function doesn’t exist in that namespace, it checks the parent namespace and so on – though internally this check is cache’d using a hash table in the first namespace which contains the complete set of functions.
When you create an object you can also give it a name, in which case it will link in an extra namespace at the top of the chain allowing you to do cool things such as this:
FudgeCollector.collectFudge(); // prints "Collected fudge
Namespaces in TorqueScript don’t necessarily have to be linked to a class. For instance you can create a new namespace at any time by defining a function.
MyNewNamespace::doThis(); // No problem
Internally all objects are derived from a ConsoleObject class which handles fields, though most of the action happens in the SimObject class which handles assigning names and identifiers to objects, and well as setting up namespaces. There is also a SimSet class which adds nesting support.
Safe to say TorqueScript is a fairly loose and flexible language.
On to lua…
“So why lua?” you may ask. Several reasons. Firstly out of all the languages I tried, lua seemed to be the most flexible. This meant that it was reasonably easy to implement something which resembled how TorqueScript functions. There is no real class system, so you are free to invent your own using linked metatables. Variable access can easily be overridden so you can expose globals and object properties pretty easily.
It’s also ridiculously easy to bind lua code in torque using the current TorqueScript binding system. Instead of having to adding a lua thunk function for every function and method, it’s possible to use a single thunk function and use “upvalues” to point to the correct C function. Although ultimately it would be a better idea to use a specialised thunk for each exposed function, the fact that it isn’t strictly required reduces the complexity of integration substantially while ensuring existing TorqueScript code can still function without the need of a rewrite.
Comparing lua to other embeddable scripting languages with class systems (e.g. ruby, python), they tend to be a bit stricter when it comes to object classes, making things such as the namespace name trick harder (if not, impossible) to implement correctly. i.e. you can’t just create an object on the fly and bind new methods to it, you need to make a proper class or make the functions references to delegates.
Lua does have a few of its own oddities – for example, arrays start at 1, and there is no switch statement. But these are a reasonable trade-off for flexibility.
On to the binding…
To be honest, it took me a while to get the right implementation which reflects the way things already work in TorqueScript. Since I wanted to keep the existing binding system, I had to write my own bindings from scratch using some of the existing C++ binding libraries as references to how things should work on a basic level.
For my implementation I ended up using userdata objects bound to class metatables. The userdata objects point directly to the instance of an object. Property access goes through one of two thunk methods (__index and __newindex), while functions are binded through one of 11 thunk methods depending on the return type and whether or not the function is meant to be used with an object. Functions are assigned to the relevant table in lua (e.g. SimObject methods go in the SimObject table, global functions go in Torque). These tables are linked together to reflect the linkage of each namespace in Torque.
When looking up properties, the __index method first checks the relevant top namespace table in lua. If it doesn’t exist there, it checks the parent table, and so on. If nothing is found in the end, it assumes the key is meant to refer to a property and acts the same way as in TorqueScript when you get a property.
When creating objects, the equivalent lua reference is stored in a global table called InstanceTable. e.g. “InstanceTable”.
So for example the following lua script…
myObject = SimObject()
myObject.customValue = 'candy'
Will print the id of the object “myObject” and the word “candy” in the console.
In the engine code, everything is abstracted into a LuaScriptEngine class which handles binding classes, registering objects and functions, and also executing code. There’s also an equivalent implementation of the “Con::executef” function for calling back into script. Unlike the TorqueScript executef, this implementation uses a list of ScriptStackValueRef objects which transparently handles pushing pushing values onto the lua stack with the desired type.
Unfortunately one thing I haven’t implemented as of yet is nested objects, where you can create an object within an object and have the script engine automatically add it to that objects group for you. However given that Torque2D uses TAML to serialise nested objects this feature can be safely ignored.
Looks like lua
So was it worth embedding lua into Torque2D? I’d say yes. At the very least I now know how deep the TorqueScript rabbit hole goes, making it easier to incorporate any of the other alternative embeddable scripting languages should they become a better fit for my needs.
Last month I unveiled a port of Torque2D which runs in web browsers via emscripten. It was developed largely over a period of a couple or weeks with the help of a few emscripten developers along the way.
After the unveil it generated some interest, but not really enough for me to make a product or anything commercial out of it especially given the current trend for game development engine tech: release everything for free.
Considering this, I’ve decided to release the code for the emscripten port under the same lovely MIT license as Torque2D itself. I hope someone can find it useful, perhaps to help promote their Torque2D game. Or who knows, maybe they can release a web-based browser game for it! The way web browsers are heading, I guess anything can happen!
Picture this scenario. You are developing a game engine, but get to an all-important question: “which scripting language do I use?”. The problem is, there are tons of scripting languages out there. Which one do you choose?
Unfortunately, picking a scripting language is not always easy. A lot of the time its a trade-off between performance, support, and something you like. There’s not much sense in picking the fastest scripting language in the world if you hate it after all.
For me, I don’t particularly care about the speed of a language (this has been explored before), so long as it’s reasonably fast. After all, if I’m going to implement something CPU intensive, I’m probably going to just end up writing it in C. I’m also open to anything, so long as it meets the following requirements:
- Reasonably easy to embed
- Uses an encapsulated state instead of globals
- Has a capable binding API
- Has some sort of OOP system with objects bound with functions
- Is used by a community of people
Since I had time to spare, I decided to evaluate a few popular scripting language implementations which have been described as being embeddable and fit the above criteria. Basically I wanted to:
- Bind an object to the API
- Pass an existing instance of an object to the API
- Create an object in the scripting language
- Call methods and get/set properties on the object
- Handle errors sanely (i.e. no stack corruption)
In addition I didn’t want to rely on any third-party wrappers – instead I went straight to the C API, since otherwise I would have been evaluating the scripting engine + the “helpful” binding library, rather than the scripting engine itself.
I ended up making a test application which binded a test object to each target language and ran a sample script in order to get a feel for how everything worked. The code is on github, so feel free to check it out if you are wondering what its like to bind a C++ object to lua, mruby, squirrel and angelscript.
I wrote a few notes in the code, but I felt the need to express my thoughts on each language based on my experience in working with both the API and language myself. So behold! My evaluation of each language is as follows…
Lua is a powerful, fast, lightweight, embeddable scripting language – or so it is claimed. It seems to be the number 1 choice for game developers, as this wikipedia article shows.
Scratching the surface, it appears to be a simple functional programming language. However it features some very powerful constructs centred around the concept of a “metatable” which amongst other things can be used to create class-based object systems. The API is well documented, and its fairly easy to find information on how to do just about anything in Lua since its commonly embedded in apps and games.
An often-heard complaint about lua is how it indexes arrays from 1, though in practice I don’t really find this to be much of an annoyance especially considering luas support for iterators. Another issue is how it isn’t very C-like in general, but IMO this isn’t too big of an issue since the syntax is fairly trivial to learn and get used to anyway.
The binding API in lua is a bit confusing as it uses the stack to set fields on objects, which almost feels like reading some assembler code. Once you realize how the stack works though, it’s not too bad.
Binding objects themselves is an open-ended problem, with many solutions. At a bare minimum you need to create a metatable object for the class, and for each instance of the class you need a userdata object bound to the metatable. You can just use a userdata for object instances and implement “__index” and to return methods,
but things start to get tricky when you need to call functions as if you just return a C function from “__index” lua won’t provide any reference to the userdata, so you need to pass in whats known as an “upvalue” when returning the function.
UPDATE: turns out it does provide a reference to the userdata, you just need to use the “:” instead of “.” when calling the function. Basic Lua 101.
One gotcha with the C API to look out for is when calling methods within methods. If the method you are calling returns an error, lua will pop the C stack all the way down to your first lua_call. This will ignore destructors for any stack variables you still have in scope in-between. To mitigate this, make sure to use lua_pcall instead.
For derived classes you can simply make a new metatable and set its “__metatable” to the parent class metatable, ad infinitum.
Apart from that binding objects to lua is reasonably simple, and if you are having problems there are a ton of binding libraries out there to help. A fairly strong contender.
Squirrel is a high level imperative, object-oriented programming language, designed to be a light-weight scripting language that fits in the size, memory bandwidth, and real-time requirements of applications like video games – or so the squirrel website claims.
In reality, I like to think of it as the result of what happens when someone takes a lot of the bad things about lua and fixes them to create a rather neat C-like language.
My only big problem with squirrel from a language perspective is its odd use of the “<-” operator to set object slots, though this can be “fixed” by simply changing the bytecode used by the “=” operator.
Similar to lua you use a stack to bind functions and classes which can be confusing. Thankfully though as with lua there are comprehensive API docs, but with one fatal flaw: I couldn’t find any good example for how to create a class bound to a C++ object, so once again I had to guess. I could excuse this a bit in lua since it’s more ambiguous, but in squirrel classes are a first level construct, making it strange how it isn’t really better explained in the documentation.
In general while I found the API documentation to be comprehensive, I didn’t feel it explained how to do things very well, and combined with the relative lack of resources online it made things a little harder to figure out.
If you are fed up with luas syntax and lack of certain basic constructs, be sure to give squirrel a try.
mruby is a light-weight, embeddable implementation of the Ruby programming language which has been in development for the past couple of years, so its fairly new.
Ruby itself is a very capable dynamic, object-oriented language which to me at least feels like a cross between perl and python. Ruby also has a proper inbuilt class system which IMO sets it on a higher level than lua. Not only that, it even supports composition through the use of modules.
Unfortunately the binding api is a little under-documented. In fact at time of writing I couldn’t find any documentation other than blog posts. In order to write a binding I had to essentially guess what you had to do using a combination of code on github and deduction.
However upon reflection binding an actual object is rather simple, with some rather descriptive function names to define both the class and its methods. For object instances, you use userdata instances bound to a data type.
Similar to lua there is also a gotcha when calling methods within methods. If the method you are calling raises an exception, mruby will pop the C stack all the way down to your first mrb_funcall. This will ignore destructors for any stack variables you still have in scope in-between. To mitigate this you need to save the “jmp” variable before the call and restore it afterwards. If you don’t want the exception to propagate down the stack you also need to clear the “exc” variable. If mruby had mrb_funcall_safe, this hack could have been avoided.
Compared to lua & squirrel, ruby is in general a more extensive language. But mruby itself still feels like it’s not quite there yet, so if you are considering implementing ruby in your app you might want to consider embedding the reference implementation instead.
Angelscript is a real curiosity. In essence it answers the question: “What would happen if you made a scripting language out of C++?”.
What results is a strongly typed, inflexible language. When compiling, everything needs to be defined or declared up-front. There are no fancy metaprogramming constructs, nor any late binding of functions. From what I gathered, if you make your whole game in a set of AngelScript files, you need to compile everything together in a single module. It makes me wonder, why not just use a reloadable DLL written in C++?
Unlike C++ Angelscript prefers that you define all class and namespace methods within the class and namespace blocks, making it feel more like C#. This is a fairly minor issue, but if you start thinking “maybe I’d like it better if I just wrote everything in C++”, it does not bode well for the scripting language.
However out of all the scripting languages, I’d have to say AngelScript was by far the easiest to embed. You simply define your function prototype and BAM! It figures out how to call it without having to write your own binding function (using its own FFI system). You can also directly expose object fields too which is a nice touch.
If you want something more direct, Angelscript is a safe bet. If you want something more freeform, look towards the other scripting languages.
So which scripting language do you use? Ultimately there is no perfect solution, you just have to go with the one you feel is the best. Failing that, you can always make your own scripting language, which is not at all an unusual thing to do when making a game engine!
» View more in the Archives
When developing 3d games, the allure of using a pre-built game engine is all too great. The trouble is nowadays you really need a good content pipeline in order to take 3d objects from your modelling tool and render them in your game, which in the past has usually required someone on the development team to write an elaborate tool-specific exporter script or plugin.
Fortunately nowadays with solutions such as collada you can now pretty much skip the “write exporter” stage and move the logic to generate runtime model data into the engine itself. However you still need something to handle animation and skinning. There are a few solutions out there, but they either contain too much baggage or the licensing isn’t so great.
So I decided to try making something myself. My requirements were as follows:
- Written in standard C++
- Proven codebase
Most importantly I wanted to base my solution off of something I knew already worked, and I knew just the thing: the “dts” code from Torque3D, which has been in use in some form or another for over a decade.
DTS itself is quite a featured format, being suitable for rendering level geometry with basic culling and fully rigged and skinned character models. In Torque3D there is even a whole content pipeline so you can go straight from collada dae to “compiled” dts shape.
You might be thinking now “Wait a minute, this DTS stuff must be tightly integrated into Torque3D. Won’t I have to include 99% of Torque3D to use this?”
This is where the beauty of libDTShape comes in. libDTShape is an extremely stripped down version of the DTS code from Torque3D. It includes only the essential code which is needed to both load and create a DTS shape, while leaving you to implement the rendering. It lives in its own namespace, so it can work independently of your own code. It is also configurable, allowing you to strip down the code even further (e.g. to remove collada support).
For those who want to just quickly load a shape and see what you can do, don’t worry! libDTShape also includes a basic example OpenGL renderer.
Admittedly as it currently stands libDTShape is a bit of an experiment, but still I hope this can be useful to others.
libDTShape can be found here: https://github.com/jamesu/libdtshape