I’ve decided to resurrect this blog, in the hope that perhaps I can infuse some much needed useful long-form content about what I’ve been up to for the past decade. Hopefully it will be insightful to those who manage to read it!

Honestly, I’m looking forward to the writing some new engaging blog posts!

It started development on the 1st gen iPod touch, all the way up to the iPad 3 and iPhone 5s - truly it was an interesting ride in terms of keeping it reasonably up to date.

At its most popular, it was getting around 3.75k downloads a month. These days, it’s around 100. Not too bad considering I last updated it in 2014. Still, considering the increasing irrelevance of the app given the prevalence of superior media players and even GIF keyboards, it’s presence on the store is questionable.

So I have decided to Open Source it under GPLV3. I hope it might be of use to someone making a gif or apng playback app.

Grab it here

NOTE: while the app is currently still available in the App Store, I am planning on removing it within the next couple of weeks so in the future compiling it yourself or downloading it from one of the various jailbreak pirate platforms will be the only way of installing it on new devices.

For the past 2 years i’ve mainly been working with Mode7 Games on their upcomming title Frozen Synapse 2 as well as their previous, Frozen Cortex. As previous subject material might suggest, both of these products have involved working with technology built up from Torque 3D.

The indie game development scene has certainly been interesting, though quite demanding in terms of technical challenges - even moreso compared to your typical web development backend job. I hope at some point to write up about some of these challenges on this blog.

Old content has now been pruned, due to it being mostly out of date. Old site content pre-2012 can now be found at the vault.

Adding simple keywords to TorqueScript is relatively simple since it uses a simple Yacc grammar coupled with an AST. However the implementation of the AST and the design of the stack makes it quite difficult to implement anything more elaborate. Even worse, TorqueScript has no unit tests, so you can easily break something if you are not careful.

So rather than having the burden of maintaining TorqueScript itself, I decided to try integrating another scripting language.

Its important to note that TorqueScript itself is tightly coupled with the object system in Torque, so its worth understanding how it functions before replacing it.

Brief overview of TorqueScript

TorqueScript is a fairly simple scripting language backed by a byte-code interpreter. Internally all variables are strings, though they are simultaneously casted to float and integer types to speed up any math operations. All variable access goes through either a local or global hash table (apart from the name of the variable itself, nothing is cache’d).

Local variables are prefixed using the % sigil, and global variables are prefixed using $. Some variables are registered by the engine itself, in which case they directly reflect whatever variable they point to.

TorqueScript’s object system is rather ingenious. Objects are referenced by either a numeric id or a name with optional quotation marks. You can also use more elaborate search strings, e.g.

1234
"MyObject"
"RootGroup/MyObject"

May all refer to the same object.

As far as the scripting engine is concerned, objects have two things: a list of properties, and a namespace. Through the namespace functions are accessed. The class hierarchy is mirrored through the use of linked namespaces.

When accessing properties, it searches for the property in the class property list first, then looks in the objects property list. When accessing functions, it checks the classes current namespace for the function. If the function doesn’t exist in that namespace, it checks the parent namespace and so on  – though internally this check is cache’d using a hash table in the first namespace which contains the complete set of functions.

When you create an object you can also give it a name, in which case it will link in an extra namespace at the top of the chain allowing you to do cool things such as this:

new SimObject(FudgeCollector);

function FudgeCollector::collectFudge()
{
   echo("Collected fudge");
}

FudgeCollector.collectFudge(); // prints "Collected fudge

Namespaces in TorqueScript don’t necessarily have to be linked to a class. For instance you can create a new namespace at any time by defining a function.

function MyNewNamespace::doThis()
{
   return 123;
}

MyNewNamespace::doThis(); // No problem

Internally all objects are derived from a ConsoleObject class which handles fields, though most of the action happens in the SimObject class which handles assigning names and identifiers to objects, and well as setting up namespaces. There is also a SimSet class which adds nesting support.

Safe to say TorqueScript is a fairly loose and flexible language.

On to lua…

“So why lua?” you may ask. Several reasons. Firstly out of all the languages I tried, lua seemed to be the most flexible. This meant that it was reasonably easy to implement something which resembled how TorqueScript functions. There is no real class system, so you are free to invent your own using linked metatables. Variable access can easily be overridden so you can expose globals and object properties pretty easily.

It’s also ridiculously easy to bind lua code in torque using the current TorqueScript binding system. Instead of having to adding a lua thunk function for every function and method, it’s possible to use a single thunk function and use “upvalues” to point to the correct C function. Although ultimately it would be a better idea to use a specialised thunk for each exposed function, the fact that it isn’t strictly required reduces the complexity of integration  substantially while ensuring existing TorqueScript code can still function without the need of a rewrite.

Comparing lua to other embeddable scripting languages with class systems (e.g. ruby, python), they tend to be a bit stricter when it comes to object classes, making things such as the namespace name trick harder (if not, impossible) to implement correctly. i.e. you can’t just create an object on the fly and bind new methods to it, you need to make a proper class or make the functions references to delegates.

Lua does have a few of its own oddities – for example, arrays start at 1, and there is no switch statement. But these are a reasonable trade-off for flexibility.

On to the binding…

To be honest, it took me a while to get the right implementation which reflects the way things already work in TorqueScript. Since I wanted to keep the existing binding system, I had to write my own bindings from scratch using some of the existing C++ binding libraries as references to how things should work on a basic level.

For my implementation I ended up using userdata objects bound to class metatables. The userdata objects point directly to the instance of an object. Property access goes through one of two thunk methods (__index and __newindex), while functions are binded through one of 11 thunk methods depending on the return type and whether or not the function is meant to be used with an object. Functions are assigned to the relevant table in lua (e.g. SimObject methods go in the SimObject table, global functions go in Torque). These tables are linked together to reflect the linkage of each namespace in Torque.

When looking up properties, the __index method first checks the relevant top namespace table in lua. If it doesn’t exist there, it checks the parent table, and so on. If nothing is found in the end, it assumes the key is meant to refer to a property and acts the same way as in TorqueScript when you get a property.

When creating objects, the equivalent lua reference is stored in a global table called InstanceTable. e.g. “InstanceTable[1234]”.

So for example the following lua script…

myObject = SimObject()
myObject.customValue = 'candy'

function SimObject:printInfo()
   Torque.echo(self:getId())
   Torque.echo(self.customValue)
end

myObject:printInfo()

Will print the id of the object “myObject” and the word “candy” in the console.

In the engine code, everything is abstracted into a LuaScriptEngine class which handles binding classes, registering objects and functions, and also executing code. There’s also an equivalent implementation of the “Con::executef” function for calling back into script. Unlike the TorqueScript executef, this implementation uses a list of ScriptStackValueRef objects which transparently handles pushing pushing values onto the lua stack with the desired type.

Unfortunately one thing I haven’t implemented as of yet is nested objects, where you can create an object within an object and have the script engine automatically add it to that objects group for you. However given that Torque2D uses TAML to serialise nested objects this feature can be safely ignored.

Conclusion

Looks like lua

So was it worth embedding lua into Torque2D? I’d say yes. At the very least I now know how deep the TorqueScript rabbit hole goes, making it easier to incorporate any of the other alternative embeddable scripting languages should they become a better fit for my needs.

After the unveil it generated some interest, but not really enough for me to make a product or anything commercial out of it especially given the current trend for game development engine tech: release everything for free.

Considering this, I’ve decided to release the code for the emscripten port under the same lovely MIT license as Torque2D itself. I hope someone can find it useful, perhaps to help promote their Torque2D game. Or who knows, maybe they can release a web-based browser game for it! The way web browsers are heading, I guess anything can happen!

Unfortunately, picking a scripting language is not always easy. A lot of the time its a trade-off between performance, support, and something you like. There’s not much sense in picking the fastest scripting language in the world if you hate it after all.

For me, I don’t particularly care about the speed of a language (this has been explored before), so long as it’s reasonably fast. After all, if I’m going to implement something CPU intensive, I’m probably going to just end up writing it in C. I’m also open to anything, so long as it meets the following requirements:

• Reasonably easy to embed
• Uses an encapsulated state instead of globals
• Has a capable binding API
• Has some sort of OOP system with objects bound with functions
• Is used by a community of people

Since I had time to spare, I decided to evaluate a few popular scripting language implementations which have been described as being embeddable and fit the above criteria. Basically I wanted to:

1. Bind an object to the API
2. Pass an existing instance of an object to the API
3. Create an object in the scripting language
4. Call methods and get/set properties on the object
5. Handle errors sanely (i.e. no stack corruption)

In addition I didn’t want to rely on any third-party wrappers – instead I went straight to the C API, since otherwise I would have been evaluating the scripting engine + the “helpful” binding library, rather than the scripting engine itself.

I ended up making a test application which binded a test object to each target language and ran a sample script in order to get a feel for how everything worked. The code is on github, so feel free to check it out if you are wondering what its like to bind a C++ object to lua, mruby, squirrel and angelscript.

I wrote a few notes in the code, but I felt the need to express my thoughts on each language based on my experience in working with both the API and language myself. So behold! My evaluation of each language is as follows…

Lua(http://www.lua.org)

Lua is a powerful, fast, lightweight, embeddable scripting language – or so it is claimed. It seems to be the number 1 choice for game developers, as this wikipedia article shows.

Scratching the surface, it appears to be a simple functional programming language. However it features some very powerful constructs centred around the concept of a “metatable” which amongst other things can be used to create class-based object systems. The API is well documented, and its fairly easy to find information on how to do just about anything in Lua since its commonly embedded in apps and games.

An often-heard complaint about lua is how it indexes arrays from 1, though in practice I don’t really find this to be much of an annoyance especially considering luas support for iterators. Another issue is how it isn’t very C-like in general, but IMO this isn’t too big of an issue since the syntax is fairly trivial to learn and get used to anyway.

The binding API in lua is a bit confusing as it uses the stack to set fields on objects, which almost feels like reading some assembler code. Once you realize how the stack works though, it’s not too bad.

Binding objects themselves is an open-ended problem, with many solutions. At a bare minimum you need to create a metatable object for the class, and for each instance of the class you need a userdata object bound to the metatable. You can just use a userdata for object instances and implement “__index” and to return methods, but things start to get tricky when you need to call functions as if you just return a C function from “__index” lua won’t provide any reference to the userdata, so you need to pass in whats known as an “upvalue” when returning the function.

UPDATE: turns out it does provide a reference to the userdata, you just need to use the “:” instead of “.” when calling the function. Basic Lua 101.

One gotcha with the C API to look out for is when calling methods within methods. If the method you are calling returns an error, lua will pop the C stack all the way down to your first lua_call. This will ignore destructors for any stack variables you still have in scope in-between. To mitigate this, make sure to use lua_pcall instead.

For derived classes you can simply make a new metatable and set its “__metatable” to the parent class metatable, ad infinitum.

Apart from that binding objects to lua is reasonably simple, and if you are having problems there are a ton of binding libraries out there to help. A fairly strong contender.

Squirrel(http://squirrel-lang.org/)

Squirrel is a high level imperative, object-oriented programming language, designed to be a light-weight scripting language that fits in the size, memory bandwidth, and real-time requirements of applications like video games – or so the squirrel website claims.

In reality, I like to think of it as the result of what happens when someone takes a lot of the bad things about lua and fixes them to create a rather neat C-like language.

My only big problem with squirrel from a language perspective is its odd use of the “<-” operator to set object slots, though this can be “fixed” by simply changing the bytecode used by the “=” operator.

Similar to lua you use a stack to bind functions and classes which can be confusing. Thankfully though as with lua there are comprehensive API docs, but with one fatal flaw: I couldn’t find any good example for how to create a class bound to a C++ object, so once again I had to guess. I could excuse this a bit in lua since it’s more ambiguous, but in squirrel classes are a first level construct, making it strange how it isn’t really better explained in the documentation.

In general while I found the API documentation to be comprehensive, I didn’t feel it explained how to do things very well, and combined with the relative lack of resources online it made things a little harder to figure out.

If you are fed up with luas syntax and lack of certain basic constructs, be sure to give squirrel a try.

mruby(https://github.com/mruby/mruby)

mruby is a light-weight, embeddable implementation of the Ruby programming language which has been in development for the past couple of years, so its fairly new.

Ruby itself is a very capable dynamic, object-oriented language which to me at least feels like a cross between perl and python. Ruby also has a proper inbuilt class system which IMO sets it on a higher level than lua. Not only that, it even supports composition through the use of modules.

Unfortunately the binding api is a little under-documented. In fact at time of writing I couldn’t find any documentation other than blog posts. In order to write a binding I had to essentially guess what you had to do using a combination of code on github and deduction.

However upon reflection binding an actual object is rather simple, with some rather descriptive function names to define both the class and its methods. For object instances, you use userdata instances bound to a data type.

Similar to lua there is also a gotcha when calling methods within methods. If the method you are calling raises an exception, mruby will pop the C stack all the way down to your first mrb_funcall. This will ignore destructors for any stack variables you still have in scope in-between. To mitigate this you need to save the “jmp” variable before the call and restore it afterwards. If you don’t want the exception to propagate down the stack you also need to clear the “exc” variable. If mruby had mrb_funcall_safe, this hack could have been avoided.

Compared to lua & squirrel, ruby is in general a more extensive language. But mruby itself still feels like it’s not quite there yet, so if you are considering implementing ruby in your app you might want to consider embedding the reference implementation instead.

Angelscript(http://www.angelcode.com/angelscript/)

Angelscript is a real curiosity. In essence it answers the question: “What would happen if you made a scripting language out of C++?”.

What results is a strongly typed, inflexible language. When compiling, everything needs to be defined or declared up-front. There are no fancy metaprogramming constructs, nor any late binding of functions. From what I gathered, if you make your whole game in a set of AngelScript files, you need to compile everything together in a single module. It makes me wonder, why not just use a reloadable DLL written in C++?

Unlike C++ Angelscript prefers that you define all class and namespace methods within the class and namespace blocks, making it feel more like C#. This is a fairly minor issue, but if you start thinking “maybe I’d like it better if I just wrote everything in C++”, it does not bode well for the scripting language.

However out of all the scripting languages, I’d have to say AngelScript was by far the easiest to embed. You simply define your function prototype and BAM! It figures out how to call it without having to write your own binding function (using its own FFI system). You can also directly expose object fields too which is a nice touch.

If you want something more direct, Angelscript is a safe bet. If you want something more freeform, look towards the other scripting languages.

Conclusion

So which scripting language do you use? Ultimately there is no perfect solution, you just have to go with the one you feel is the best. Failing that, you can always make your own scripting language, which is not at all an unusual thing to do when making a game engine!

Fortunately nowadays with solutions such as collada you can now pretty much skip the “write exporter” stage and move the logic to generate runtime model data into the engine itself. However you still need something to handle animation and skinning. There are a few solutions out there, but they either contain too much baggage or the licensing isn’t so great.

So I decided to try making something myself. My requirements were as follows:

• Written in standard C++
• Engine-Agnostic
• Renderer-Agnostic
• Cross-Platform
• Proven codebase

Most importantly I wanted to base my solution off of something I knew already worked, and I knew just the thing: the “dts” code from Torque3D, which has been in use in some form or another for over a decade.

DTS itself is quite a featured format, being suitable for rendering level geometry with basic culling and fully rigged and skinned character models. In Torque3D there is even a whole content pipeline so you can go straight from collada dae to “compiled” dts shape.

You might be thinking now “Wait a minute, this DTS stuff must be tightly integrated into Torque3D. Won’t I have to include 99% of Torque3D to use this?”

This is where the beauty of libDTShape comes in. libDTShape is an extremely stripped down version of the DTS code from Torque3D. It includes only the essential code which is needed to both load and create a DTS shape, while leaving you to implement the rendering. It lives in its own namespace, so it can work independently of your own code. It is also configurable, allowing you to strip down the code even further (e.g. to remove collada support).

For those who want to just quickly load a shape and see what you can do, don’t worry! libDTShape also includes a basic example OpenGL renderer.

Admittedly as it currently stands libDTShape is a bit of an experiment, but still I hope this can be useful to others.

libDTShape can be found here: https://github.com/jamesu/libdtshape

The tools for doing this in the past sadly either required the end-user to install something (if you made your own plugin), or conforming to a strange runtime (flash, unity, java). Torque itself at one point went with the “make your own plugin” approach, which to say the least was a disaster since every developer had to go through the hurdle of making a plugin for each major system and ultimately the end-user had to install something. Unless your game is the best thing since Minecraft, this approach will never work.

Thankfully now there are several more promising solutions on the horizon, one of which is Emscripten. Emscripten is a compiler that takes standard C/C++ code and compiles it to Javascript suitable for running in any modern web browser. In essence as long as you have a modern 3d graphics card and HTML5-capable browser, you should be able to run just about any game code on all major platforms!

Getting to the point, over the past few weeks I’ve been working on a port of the Torque2D game engine to Emscripten. After a bit of help from the Emscripten developers, nearly everything works (excluding networking).

On a time scale, the basic port took a weekend while the bug fixing was spread over the next few weeks. Porting across code to Emscripten was relatively easy since it exposes a pretty standard unix API with SDL, OpenGL, OpenAL, and a few other useful APIs. However the difficulty was multiplied somewhat when I discovered some rather nasty memory corruption bugs.

Firstly there was a problem with the dynamic_cast handler, in which a cast would fail for no apparent reason. It turns out a key map in the input code was a bit too small, causing an overflow which corrupted some memory related to handling the cast. There was also a problem in the scripting engine in which script strings would randomly be corrupted which I ended up hacking around.

Sadly Emscriptens debugging facilities were not up to the task in this case. While there were several facilities to debug memory corruption, none of them picked up on these issues. Unfortunately this meant  I had to go through every major bit of code to eliminate what was causing the problem, which took a considerable amount of time.

Another thing to note is Emscriptens OpenAL implementation was missing a few crucial functions, though these were fairly trivial to implement.

Overall I was quite impressed with Emscripten, though it would be nice if there were more debugging facilities or even some form of GDB frontend so I could more easily step through code.

For your enjoyment, A build of the port can be found here: http://stuff.cuppadev.co.uk/torque2d-web/

Apart from the problem of not being able to find time to update, I also had a problem of not being to update at all. A while ago I migrated the entire site to Jekyll, a static site generator. This worked fine for a while, but I quickly grew tired of the long delay when regenerating the site when typing up new posts.

I also felt a bit of disconnect needing to type up each post in a text editor and having to guess how the post formatting would turn out. In short it felt like a big chore to post anything. And for me the ability to post things quickly and get instant feedback is a much needed attribute of a blogging system. Thus I have now migrated back to WordPress.

I have a few projects I would like to release before the end of the year, so expect me to write about them here. I’m also going to write about what actually happened this year, including what went wrong during the rather disastrous deployment of the Frozen Synapse server rewrite.

Stay Tuned.

This time I decided to tackle TorqueScript’s type conversion system. This basically handles converting strings such as “0.1” into native values such as 0.1f, and vice versa. Since everything must be converted to and from strings all the time when manipulating object fields and object variables, it’s a great bottleneck for performance if scripting is used too heavily.

A Little Background on the Type System…

In TorqueScript, values can be one of four types:

• Strings
• Unsigned Integers
• Floats
• Custom Types

The first three are self-explanatory. The fourth is more involved as it is used to expose engine variables and object fields as strings. Getters are implemented using the ConsoleGetType macro which implements a function returning a string based on the value, setters are implemented with the ConsoleSetType macro which implements a function which parses a string and sets the value.

All access to registered engine variables and objects goes through this type system, so any improvements to it have a good potential to improve overall performance. For instance point values in TorqueScript are always manipulated as strings, so any complex vector math incurs the overhead of string conversion. For this reason I also decided to implement value-based arrays.

What went wrong

One big problem with altering something as monolithic as TorqueScript is little changes can cause big problems since the whole implementation is essentially an inter-connected mess. Changed the syntax? Be prepared to debug the AST. Changed the AST? Be prepared to debug the bytecode. Changed the bytecode? Be prepared to debug the interpreter.

For instance, when changing parts of the AST I frequently found that it wouldn’t generate the correct opcodes. The issue was that the AST nodes had been designed assuming the other statement nodes behaved in a specific way, so when I added new variable types to support arrays it completely failed.

In another case simple changes to the interpreter and the AST caused the float stack to become unbalanced, which required painfully tedious step-by-step debugging in order to resolve the issue.

Problems from the ground up

TorqueScript suffers from a key design issue: stack values are split into three pools: strings, integers and floats. The only generic value type is a string e.g. when calling functions, the compiler has to assume the return value is a string.

There is some code present in the interpreter which skips the conversion if the next opcodes indicate the value will be converted to an integer or float, but this is more of an elaborate hack around a poorly designed system.

Since there is no generic stack value, operator overloading is impossible (thus the need for specific string concatenation operators). Opcodes relating to field access need to be duplicated in order to handle different types. Object references need to be resolved every time a field is accessed or when an object function is invoked.

TorqueScript also lacks tests to determine if it functions correctly or not, which ultimately means everything needs to be manually tested in order to verify it still works correctly… unless you have the time and resources to create a complete testing suite.

To Conclude

While this project has been interesting, ultimately I’ve decided to shelve it for the time being.

It turns out fixing a scripting language is harder than it first looks. While I managed to implement arrays and improve the way certain types are set, fixing all the bugs caused by the changes turned out to be an even bigger hurdle. In the end, I was stuck trying to resolve some tricky killer bugs in the editor scripts.

In light of my recent experience, I have the uttermost respect for anyone who maintains a scripting language.

For anyone interested, this code is available in my Torque3D fork.