Scripting, down the rabbit hole

27 Jul 2015

It’s been a long time since the last update, but it hasn’t been idle time. I’ve made a lot of progress on the Lua scripting layer and achieved a lot more than I had planned for between the last update and this one. The initial idea was to first just extend and clean up the prototype scripting layer, to the point it would be stable and flexible enough to start moving entity behavior into Lua scripts. Along the way it became increasingly obvious there were just too many strings attached to the game, planet and entity classes and their interactions, to be able to move them piece-wise into the scripting layer. While trying to prototype a single scriptable K14Entity I went down a giant rabbit hole that stretched almost all of the engine code related to game logic, including sharing of game state between Objective-C and Lua, calling between the two worlds in both directions, posting gameplay events from Lua, input handling, everything. I found myself in a dark and grim place, where nothing worked and everything appeared to require additional effort to fix properly ;-). But eventually, I emerged with a more-or-less complete separation of the pure core engine logic (implemented as native code), and the game logic running on top of it (implemented as Lua code).

In this post I will try to highlight some of the more interesting bits of the end result, without going into too much detail how much code was changed behind the scenes to implement them. No new features have been added besides the ability (and requirement) of all game logic to be written in Lua, and most of the elbow grease that went into implementing scriptable game logic is simply not that interesting to read (or write) about, so I’ll refrain from that ;-)

Scripting layer - the binding interface class

Most of the lower-level parts of the Objective-C parts of the scripting layer are now contained in a new class K14LuaVM, which implements general-purpose utility functions used by the higher-level, application-specific binding classes for K14Game, K14Planet and K14Entity, which will be discussed in a later section. The K14LuaVM class has the following responsibilities:

Creating and initializing the Lua VM

Every part of the game state that is implemented in Lua needs to live inside the same Lua VM. This includes the wrapper base class definitions, utility functions, global state, etc. The K14LuaVM class creates the Lua VM using luaL_newstate, and automatically finds and loads the Lua scripts containing the wrapper base class definitions from the application bundle. Inside the Lua VM, the K14LuaVM instance also creates an instance of the Lua VM class to wrap itself, and binds some general-purpose enumeration values and functions to it for use by scripts. From this point onward, any Lua script loaded into the VM can use the types, functions and classes added when the K14LuaVM instance was constructed, without having to require any external Lua scripts. For example, entity classes like Ship or Turret can directly derive from the Entity wrapper base class.

Loading additional Lua scripts

To allow the higher-level binding classes to load additional Lua scripts containing e.g. the wrapper classes for a specific entity (such as Turret.lua or Ship.lua), the K14LuaVM provides a loadScriptFile method that reads a script file and sources into the VM.

Querying the Lua VM

Supported queries include global variables, class tables (which are also just global variables in Lua), class variables, subclasses derived from a native class. The variables to query can be specified using a dotted path (e.g. Planet0.description), and the query result is pushed to the Lua stack.

Lua stack manipulation and data type conversion

All communication between native code and scripts works by means of the Lua stack, including but not limited to function parameters and return values, access to global variables, etc. Because Lua only supports a very minimal set of data types, values pushed and popped to-, and from the Lua stack need to be converted between their Objective-C and Lua representations, and vice-versa. The K14LuaVM class provides some utility functions to push, pop, encode and decode between Objective-C and Lua data types, including some that have no direct Lua equivalent, such as arrays, dictionaries, points, sizes, etc. I’ve defined my own set of data types that can be encoded using K14LuaVM, which currently contains BOOL, INT, REAL, STRING, REF (a reference to a wrapped instance, such as an entity), VEC2 and SIZE, plus arrays of any of these types.

Creating wrapper instances

Instances of native classes that need to be accessed and/or extended by Lua scripts need to have an associated wrapper class instance inside the Lua VM, which is used both as a proxy for the native instance, and as as a context for callbacks and roundtrip calls between native code and Lua scripts. The wrapper instance stores a pointer back to the native instance as Lua light user data, and is passed as the first argument (the self pointer) of any method call from Lua to Objective-C and vice-versa.

The K14LuaVM class provides facilities to create Lua wrapper classes and associate them with native instances, by means of the K14LuaVM createWrapperForInstance:wrapperClass: method. This method will try to retrieve the specified Lua wrapper class definition from the VM, and call its new method (constructor) with the Objective-C instance as its single argument. This will return a new instance of the wrapper class, which is stored in the Lua registry from where it can be retrieved using an integer index. When the Objective-C instance goes out of scope, it releases its wrapper instance by means of the K14LuaVM deleteWrapper method.

Strictly speaking, the moniker ‘wrapper class’ is not doing justice to the scope of the wrapper classes that are currently implemented for games, planets and entities. Besides establishing a two-way binding between Objective-C and Lua, these classes also implement the actual game, planet and entity logic, store their own set of Lua properties, etc. The wrapper classes for games, planets and entities will be discussed in a later section, but for now just keep in mind that when I refer to a ‘wrapper class’ it means ‘the Lua class that implements everything about a game, planet, or entity, except its core engine properties and functions implemented as native code’.

Binding enumeration types shared between native code and Lua

Enumeration types that need to be shared between native code and scripts have to be defined both in Objective-C and Lua, using the same enumeration values. For instance when Lua code wants to the Box2D body type of an entity to dynamic using its native property bodyType, it needs to be able to retrieve the Objective-C K14EntityBodyTypeDynamic enumeration value.

The K14LuaVM class provides a method bindEnumWithName:values:inClass: method, which takes a dictionary mapping string-type names to enumeration values, and adds them to the specified Lua class as a table property. This allows exchanging enumeration values between the two worlds without having to resort to duplication and/or hard-coding of enumeration values:

// In Objective-C
K14LuaVM *vm = ...

NSDictionary *enum_values = @{ 
  "STATIC" : @(K14EntityBodyTypeStatic),
  "DYNAMIC" : @(K14EntityBodyTypeDynamic),
  "KINEMATIC" : @(K14EntityBodyTypeKinematic),
@}

[vm bindEnumWithName:@"BodyType" values:enum_values inClass:@"Entity"];
-- In Lua
planet:getOrb():setBodyType(Entity.BodyType["DYNAMIC"])

Binding native functions for use by scripts

As mentioned the Lua wrapper classes act as a proxy to their underlying Objective-C instance, and any native method that needs to be accessible from Lua needs to be bound to the wrapper class. This is done using the K14LuaVM bindMethodWithName:inClass:function: method, which takes the name of the Lua method to bind, the class to bind it to, and a C wrapper function that adheres to the lua_CFunction signature. The wrapper function has the following responsibilities:

  1. Popping and decoding the function call arguments passed by the Lua script from the Lua stack. The first argument will always be a reference to the Lua wrapper instance (the self pointer)
  2. Getting the native instance stored with the wrapper instance
  3. Calling the Objective-C method wrapped by the function, on the native instance retrieved from the Lua wrapper instance
  4. Pushing the return value back onto Lua script and returning the number of return values pushed.

As an example consider the K14Entity glueToSurfaceEdgeWithIndex:fraction: method, which is bound to the Lua Entity class using the following binding code and C wrapper function:

// In Objective-C
K14LuaVM *vm = ...;

[vm bindMethodWithName:@"glueToSurfaceEdge" inClass:@"Entity" function:wrap_glue_to_surface_edge];

// ...

static int wrap_glue_to_surface_edge(lua_State *L)
{
  // Get K14ScriptableEntity instance from wrapper
  lua_getfield(L, -3, "entity");
  K14ScriptableEntity *entity = (__bridge K14ScriptableEntity *) lua_topointer(L, -1);
  lua_pop(L, 1);

  // Get parameters
  int edge_index = (int) lua_tointeger(L, -2);
  float fraction = lua_tonumber(L, -1);

  // Glue entity to edge
  [entity glueToSurfaceEdgeWithIndex:edge_index atFraction:fraction];

  return 0;
}

Binding native properties for use by scripts

Besides binding native functions to the Lua wrapper classes, we also want to be able to read and write native properties of the underlying Objective-C instance. For instance, an entity wrapper class should be able to get and set the position and angle properties of the underlying K14Entity.

To avoid having to bind properties by binding accessor functions for each and every property that needs to be accessible from scripts, I used a combination of [Key-Value Coding][obj-kvc] and a Lua property registration function. From the Lua-perspective, native properties are bound to wrapper classes as follows:

-- In VM.lua, define a function that returns a getter and setter function
-- for the native property to bind
function VM.bindNativeProperty(wrappedObject, name, dataType)·

  local get = function(instance)·
    return instance:getNativeProperty(instance[wrappedObject], name, dataType)
  end

  local set = function(instance, value)
    return instance:setNativeProperty(instance[wrappedObject], name, dataType, value)
  end

  return get, set
end


-- In Entity.lua, add native properties to entity wrapper base class
Entity.getPlanet, _ = VM.bindNativeProperty("entity", "planet", VM.PropertyType["REF"])
Entity.getGame, _ = VM.bindNativeProperty("entity", "planet.game", VM.PropertyType["REF"])

Entity.getId, _ = VM.bindNativeProperty("entity", "id", VM.PropertyType["INT"])
Entity.getAngle, Entity.setAngle = VM.bindNativeProperty("entity", "angle", VM.PropertyType["REAL"])
Entity.getPosition, Entity.setPosition = VM.bindNativeProperty("entity", "position", VM.PropertyType["POINT"]);

Note that the getter and setter functions returned by VM.bindNativeProperty are closures: function objects that capture the values of local variables in the same scope, at the time the function object is created. In this case these are the field name of the pointer to the native object inside the wrapper class, the name of the property to get, and the property data type. The values for these three variables are bound to the returned functions (get and set), and are automatically added to an invocation of the getNativeProperty and getNativeProperty functions when the get and set functions are called.

The getNativeProperty and setNativeProperty functions are defined in Objective-C, and get/set the requested property using key-value coding. The property data type is passed as a PropertyType enumeration, which is bound to the Lua VM class, and is used to decode/encode the property value from/to the Lua stack. To get/set the actual property value, the NSKeyValueCoding protocol functions getValueForKeyPath and setValue:forKeyPath:` methods are used.

The nice thing about this approach is not just that we can now bind any property that has a supported data type by just adding a single line to the Lua wrapper class, but that we can also use key paths to traverse relations between Objective-C instances accessible through their properties. For example, the getGame accessor is bound to the Entity class using the key-value path planet.game. Getting the property will first find the K14Planet instance associated with the K14Entity wrapped by the Entity instance by retrieving its planet property, then retrieve the game property of the returned K14Planet instance to get the game. Because the data type for this property is defined as REF (a reference variable), the getNativeProperty method will find the wrapper class for the K14Game instance, and return it to the Lua script.

Scripting layer: the game, planet and entity binding classes

The application-specific binding classes K14ScriptableGame, K14ScriptablePlanet and K14ScriptableEntity sit between the Lua VM and the engine classes K14Game, K14Planet and K14Entity, implementing the necessary evil required to provide a ‘view’ for the script wrappers onto the native engine classes and vice-versa. In terms of code and architecture these are quite boring classes, so I’ll skip over them quickly. The main responsibilities of the entity-specific binding classes are the following:

  1. Binding native functions to the wrapper base class
  2. Loading wrapper subclass scripts implementing the actual game, planet or entity wrappers
  3. Binding Lua wrapper class functions to native API functions
  4. Relaying native engine callbacks to Lua wrapper instances

The first three tasks on the above list are implemented by the registerClasses method of each to the K14Scriptable... classes. These are static methods that are called by the K14LuaVM class on initialization, to populate the Lua VM with the Lua class definitions, functions, constants, etc. required by the scripting facilities. For the K14ScriptablePlanet class, the registerClass method looks like this:

+(void) registerClasses: (K14LuaVM *) vm
{
  // Load and execute base class script
  [vm loadScriptFile:[[NSBundle mainBundle] pathForResource:@"Planet" ofType:@"lua"]];
  
  // Bind native property get/set functions
  [vm bindMethodWithName:@"getNativeProperty" inClass:@"Planet" function:vm.getNativeProperty];
  [vm bindMethodWithName:@"setNativeProperty" inClass:@"Planet" function:vm.setNativeProperty];
  
  // Bind K14Planet native functions
  [vm bindMethodWithName:@"createEntity" inClass:@"Planet" function:wrap_create_entity];
  [vm bindMethodWithName:@"peekFromPosition" inClass:@"Planet" function:wrap_peek_from_position];
  
  // Load planet scripts from the 'Planets' subdirectory in the application bundle
  NSString *planet_script_path = [NSBundle.mainBundle.resourcePath stringByAppendingPathComponent:@"Planets"];
  
  NSDirectoryEnumerator *planet_scripts = [[NSFileManager defaultManager] enumeratorAtPath:planet_script_path];
  
  for (NSString *planet_script in planet_scripts)
  {
    if ([planet_script.pathExtension isEqualToString:@"lua"])
    {
      [vm loadScriptFile:[planet_script_path stringByAppendingPathComponent:planet_script]];
    }
  }
}

The code for this function should be self-explanatory. The only interesting thing about it worth mentioning is how the wrapper subclass scripts are automatically loaded by finding all .lua files in an hard-coded path of the application bundle, and sourcing them into the Lua VM. Adding additional game, planet or entities is a matter of just dropping an extra Lua file into the application bundle, which will be loaded automatically. To query classes loaded into the Lua VM, the wrapper class for K14LuaVM (the VM class in Lua) provides a method getSubclassesOfClass, which when passed the name of a base class will find all registered classes deriving from it. This is used for example when populating the level select screen, which queries all Planet subclasses and reads their name and description class variables to add them to the table of planets.

The last responsibility of the application-specific binding class, as mentioned in the summary above, is related to binding Lua to Objective-C, ie: the binding direction opposite to what we’ve mostly been talking about so far. In some cases, engine code should be able to access state living inside the Lua VM, for example to ask a Lua wrapper class to make a snapshot of itself, or to communicate engine events. At this point, the scripting layer only contains examples of the latter (relaying engine events), as the snapshot functionality has not been ported over to support scriptable entities yet.

The most obvious examples of engine callbacks that need to be relayed from native code to Lua scripts are the game loop update event (the step method of K14Game, K14Planet, and K14Entity, and the entity collision callbacks didCollideWithSurface and didCollideWithEntity originating from the physics engine. Since the K14ScriptableGame, K14ScriptablePlanet and K14ScriptableEntity classes directly derive from K14Game, K14Planet and K14Entity, they can simply override their base-class methods to relay them to the Lua wrapper classes associated with the binding class instance. For example this what the K14ScriptablePlanet step method looks like, which will call the Lua method Planet:step(dt):

-(void) step: (NSTimeInterval) dt
{
  lua_State *L = self.vm.state;

  // Get 'step' method from wrapper and push it to the Lua stack  
  [self.vm pushWrapperWithIndex:self.wrapper];
  lua_getfield(L, -1, "step");
  lua_replace(L, -2);

  // Push wrapper, as 'self' argment for the call
  [self.vm pushWrapperWithIndex:self.wrapper];

  // Push time-delta parameter
  lua_pushnumber(L, (lua_Number) dt);
  
  // Perform Lua call with 2 parameters and 0 return values
  lua_call(L, 2, 0);
}

As you can see the binding function is still implemented directly using Lua C API calls, which looks a little obscure and non-obvious. Eventually I’d like to encapsulate all the binding stuff using K14LuaVM interfaces.

Scripting layer: the game, planet and entity Lua wrapper classes

The last part of the scripting layer are the Lua wrapper classes themselves. These are where all the game logic will be implemented: game rules (ie: the set of gameplay events, when they are posted, and how they are handled), win/lose conditions, planet setup and behavior (dynamic surfaces, switches, …), entity behavior and appearance, etc. The engine provides the basic building blocks (game loop control, physics, rendering, sound output, etc), the Lua wrapper classes actually make the game.

Right now the three main wrapper base classes are Game, Planet and Entity. These each provide the facilities required by subclasses to implement the desired game logic, by means of a combination of methods directly implemented in Lua and methods bound to the native classes K14Game, K14Planet and K14Entity, by means of the binding classes K14ScriptableGame, K14ScriptablePlanet and `K14ScriptableEntity.

Here’s a summary of the responsibilities and functionality of the three types of wrapper classes. Note that these are still a work in progress, and what’s listed below only reflects the current state of the wrapper classes. When more gameplay elements are added, the functionality provided by the wrapper classes will expand.

  • The Game wrapper classes

    1. Binding accessors for native K14Game properties: elapsed game time.
    2. Binding accessors for native K14Player properties: number of lives, player score, fuel left.
    3. Binding the native function getInputs, which returns the current input state as a Lua table
    4. Providing a step callback method that is called by the game engine on every iteration of the game loop.
    5. Providing a postEvent method that can be used by Game or Entity subclasses to post gameplay events.
    6. Providing a processEvent method that handles a single event. This method will be called from the default implementation of the step method of the wrapper base class Game.
  • The Planet wrapper classes

    1. Exposing a set of static class variables to define the planet surface, gravity vector, a description of the planet, and a set that indicates the entity classes that are used by the planet. These properties will be read by the K14ScriptablePlanet class to perform initialization of its superclass K14Planet.

    2. Binding accessors for K14Planet properties: the parent game, the planet surface and gravity vector, and a reference to the ship and orb entities.

    3. Binding native functions: the createEntity function (to create entities), and the peekFromPosition function (to query the planet for visible entities)

    4. Providing an ‘init’ method, which is called by the game engine after the wrapper class is instantiated, and should populate the planet with entities. Planet initialization cannot be done directly from the Planet constructor, because the K14ScriptablePlanet instance referenced by the wrapper will only be fully constructed after the wrapper class has been instantiated.

  • The Entity wrapper base class

    1. Exposing a set of static class variables defining the entity size, its type (player, enemy, static, or projectile), and its sprite frames (a map from frame names to image filenames).

    2. Binding accessors for native K14Entity properties: the parent planet, the entity ID, its size, position and angle.

    3. Binding native functions: applyForce, createBoxFixture, glueToSurfaceEdge, joinUsingDistanceJoint, kill and respawn. This may seem like a random collection of methods, but they are currently the only native K14Entity methods needed to replicate all current entity logic using Lua scripts.

    4. Providing a step callback method that is called by the game engine on every iteration of the game loop. The step method is where entity behavior is defined.

    5. Providing callbacks didCollideWithSurface and didCollideWithEntity, which are called by the engine when it collides with the surface or with another entity (e.g. a projectile).

    6. Providing a callback getSprites, which returns a set of sprite definitions representing the current state of the entity. A sprite definition is a combination of a sprite frame,
      a size, a position and an angle. This method is called by the game engine whenever the entity is rendered.

The way these functions are used right now basically amounts to ‘replicating the behavior implemented by the old, native implementation of the game, planet and entity classes’. There’s a Game subclass StandardGame that defines the same set of gameplay events as what was previously implemented by K14Game, and they are all handled exactly as they were in native code. Then there’s Planet subclasses that setup the two test planets we were using all along, and Entity subclasses for each of the native entity types we had before, again implementing the same behavior and engine callbacks as their native counterparts. Together, they implement the game we had before, but now running inside the Lua VM, on top of an engine implemented in native code :-)

Anatomy of an entity script

To give an impression what a typical Lua script implementing an entity looks like, I’ll paste the complete source code of the Turret entity here, which was previously implemented by the native class K14Turret. The Lua code together with the comments should be self-explanatory.

-- Define 'Turret' as a Lua class deriving from the 'Entity' base class
Turret = {}
Turret.__index = Turret

setmetatable(Turret, { __index = Entity })

-- Initialize static properties & constants
Turret.size = { width=2.5, height=0.8 }
Turret.entityType = Entity.Type["ENEMY"]
Turret.spriteFrameImages = { default="turret_sprite_default" }

Turret.killScore = 750
Turret.firingRate = 0.25
Turret.bulletVelocity = 8

-- Constructor. This is called by `K14ScriptableEntity` whenever an entity with the 'Turret' 
-- class is constructed. The single argument passed is a pointer to the `K14ScriptableEntity`
-- instance.
function Turret:new(turret)

  -- Create wrapper instance by calling the base class constructor. This will initialize
  -- instance variables such as the 'game' and 'planet' references.
  local instance = Entity:new(turret)

  setmetatable(instance, Turret)

  -- Create fixture
  instance:createBoxFixture("turret", Turret.size, 1)

  -- Set property defaults. Note that the distinction between class constants/static 
  -- properties and instance variables with default values may not be directly obvious.
  -- The line between is drawn based on whether a property will be modified during 
  -- gameplay: if so, the property value needs to be registered with the instance, and
  -- stored/restored to/from when taking a snapshot of the entity. If the property never
  -- changes, its value can be defined as a static class variablem, and its value can be
  -- restored by simply sourcing the script file.
  instance.lastBulletFired = 0

  return instance
end

-- Return current entity sprites
function Turret:getSprites()

  local default_sprite = 
  {
    spriteFrame="default",
    position=self:getPosition(),
    size=self:getSize(),
    angle=self:getAngle()
  }

  return { default_sprite }
end

-- Entity step method
function Turret:step(dt)

  self.lastBulletFired = self.lastBulletFired + dt;

  if self.lastBulletFired > (1 / self.firingRate) then

    local angle = self:getAngle() + math.pi / 2

    -- Communicating game state changes can only be done using gameplay events. This may seem a little
    -- cumbersome, but it has some important advantages. First of all the surface area of the scripting
    -- layer can be kept small, as we don't need to bind functions for every little thing that may occur
    -- that changes game state. Second, using gameplay events as the means of communication, allows the
    -- additional processing based on the event type, such as playing sound, or overriding the default
    -- action performed as a result of the event. Last but not least, it makes the scripting code much
    -- more resilient to changes to the interface or architecture of the native engine code. As long as 
    -- the postEvent method doesn't change, the Lua script should stay compatible.
    self.game:postEvent(Game.Event["PROJECTILE_LAUNCHED"], self, { angle=angle, velocity=self.bulletVelocity, projectileClass="Bullet" })

    self.lastBulletFired = 0;
  end
end

-- Entity-entity collision callback. This is called automatically from, whenever Box2D 
-- triggers the ` K14ScriptableEntity didCollideWithEntity:` method
function Turret:didCollideWithEntity(entity)

  if entity.entityType == Entity.Type["PROJECTILE"] then
    self.game:postEvent(Game.Event["ENITITY_HIT"], self, {})
    self.game:postEvent(Game.Event["POINTS_SCORED"], self, { points=self.killScore })
  end
end

Pasting the code for the Ship entity (previously K14Ship) would have been an even better way to illustrate the capabilities of the scripting layer, but it’s a little long (over 200 lines of Lua code), and the length of this post is already getting out of hand. But don’t worry, we’ll see a lot more Lua game logic in the future ;-)

Current state

The current state of the scripting layer is that all native entity classes have been deprecated, and all game, planet and entity logic has been moved into Lua classes. Obviously the scripting layer is far from complete, but its a solid base that’s already flexible and complete enough to support all of the game and entity logic that was previously implemented as native code.

I could of course post a video of the game running on top of the scripting engine, but at this point it would look identical to the last video, when everything was still implemented as native code. I’ll hold off making a video until I can show some of the more interesting possibilities the scripting layer should enable ;-)

Loose ends

There’s a few loose ends in the scripting layer that don’t feel ‘right’ or ‘done’, but aren’t bothering me enough yet to consider fixing them. I’ll keep these in the back of my mind while working on the scripting layer:

  • The K14LuaVM class is a very leaky abstraction, as it doesn’t draw a clear boundary between parts of the code that interface directly with Lua, and parts that use our own interfaces. Lua C API calls can still be found all around the scripting layer, and K14LuaVM functions make assumptions about the contents of the Lua stack, and/or leave the stack in a different state after returning. It would be nice to have a better abstraction that concentrates all Lua C API calls in one place.

  • Taking snapshots of the game, planet and entity wrapper classes, which is done for rendering or to freeze/unfreeze game state currently only stores native properties. This means we cannot restore the game state from a snapshot, and we cannot visualize things like the joint between the ship and the orb, because the renderer has no way to determine whether the orb is lifted: this is now a property of the Lua wrapper class Game, and wrapper properties are not accessible from native code.

  • Action replays are currently broken as a result of the broken snapshot functionality.

  • The native entity classes are not used anymore, but they are still part of the XCode project, to be able to build the unit tests. The unit tests will have to be rewritten to use the scriptable game, planet and entity classes.

Lessons learned

  • Manually writing binding code for Lua has proven to be a relatively minor effort, which (at least for now) pays off in flexibility (we can do the same thing differently in places where it makes more sense to do so) and low-friction prototyping (we can quickly try out things at every level of abstraction in the code). Wrapper generators are supposed to make your life easier by doing all the work for you, but they often make things much more difficult if you need something that is very specific or unorthodox. With manual binding you have full control.

  • Don’t try to incrementally move things into your scripting layer, keeping your code in a twilight state where e.g. half of your entities are implemented in native code, and the other half in scripts. I found out the hard way that adding a scripting layer to a working project quickly turns into a rabbit hole impacting almost every part the code related to game logic. Trying to keep the native K14Entity entity classes alive and working, while at the same time implementing other entities as K14ScriptableEntity classes backed by Lua wrappers, brought me nothing but trouble. Either take the plunge and move everything you want to script at once, or (even better), start out with scripting support as a day-1 feature.

  • Things will get worse before they get better. In the process of adding scripting functionality you may get the feeling you are turning a perfectly fine and working piece of code into a big, unstable mess that forces you to fix all kinds of problems that didn’t exist before you started. For example, I didn’t realize beforehand how many dependencies (function calls, data sharing) between the core engine and the game logic were hiding throughout the code. I first had to make a big mess, putting up a barrier between the engine and the game logic to reduce the surface area of the scripting layer, but eventually this has lead to a much cleaner separation between these two components. I think this is one of these examples that show that constraints (having to write binding code for all function calls and data-transfers between engine and game logic) often force you to write better code (limiting the interactions results in a better separation of concerns).

  • It’s strangely satisfying to see your game run like before, knowing that half of it is now running inside its own VM, its own memory space, implemented using its own instruction set, executed by a virtual ‘CPU’ (the interpreter). We haven’t gained anything in terms of gameplay features or anything, but it’s still fascinating a computer that fits inside your pocket pulls this off without breaking a sweat.

Next steps

As mentioned in the ‘loose ends’ section above, the scripting layer is far from done, improving it will be an ongoing effort for some time to come. One major feature that has been lost in translation is taking snapshots of the complete game state, which is a requirement for action replays and to be able to freeze/unfreeze the game. Freezing/unfreezing game state is an essential feature, not just to avoid losing the game state when the game is interrupted by e.g. a phone call, but also to enable things like live/looped code editing features possible now that the game logic is running inside a Lua VM, so I think this is what I’ll be working on next.

Development scoreboard

I lost count of the development time a little, but I estimate that all of the above took me about ~40 hours of development time, for a total of ~220 hours. A significant increase, which was mainly caused by my lack of experience manually writing scripting bindings. I had to experiment and rework a lot, which added a lot to the development time.

A nasty surprise was waiting for me when I wanted to get the SLOC count including the Lua scripts. I had to switch to CLOC to get the SLOC count because sloccount does not support Lua. It also, apparently, does not support Objective-C++. Which accounts for more than two thirds of the game code… All this time I had assumed that what sloccount listed as ‘objective-c’ code, also included Objective-C++. It doesn’t, which means the SLOC count has been dramatically under-reported all along :-/

CLOC does support both Lua and Objective-C++ and reports a total SLOC count of 6546 lines of native code, and 476 lines of Lua. I should have realized earlier that it was impossible all this code added up to less then 2500 lines… Anyway, I’m not planning to retroactively adjust the SLOC counts for all previous posts, so this post will be the new benchmark for all SLOC counts from this point onward.