SC Engine: Part 2 - System Overview: Messages and Applications

- Introduction
- Part One: System Overview
- Part Two: System Overview: Messages and Applications
- Part Three: Screen Scraping
- Part Four: YouTube Parsing
- Part Five: Linking the Video to the Game
- Part Six: Messaging Middleware
- Part Seven: The Console
- Part Eight: The Site

This post will go over some of the general implementation for SC Engine.

The system is written in python, and as stated in Part 1, uses a Message Bus to pass messages with information between "applications" inside the engine.

Messages

A message is a simple entity meant to store data that is being sent from one part of the system to another. Here is what a message might look like...

class PlayerFetchRequested(BaseMessage):
     def __init__(self, player_id):
         # int
         self.player_id = player_id

For those interetested, you can see all the messages in the source:

As you can see, there's not much meat here. It's just a class that inherits from BaseMessage, with a constructor that has the data. It could've just been implemented as a tuple with a string for the message name and a dictionary for the data, but it was nice to be able to have an error thrown if I don't include all the arguments. BaseMessage also provides a few methods onto the messages that are helpful, and is used by the messaging middleware I wrote (which will be discussed in another part).

See all the messages for SC Engine

Applications

Messages are sent between applications. An "application" is a python object that defines a number of methods which respond to incoming messages and optionally sending out resulting messages. The point is that each application does one job, and sends out messages of the results of what it's done. Where the messages go or who use them is completely not of the concern of the application. Nor does it care where the messages it receives have come from. In this sense, the entire system is implicitly "pub/sub", or "publish and subscribe".

Typically, an application consists of two parts: a builder, and the application itself. Let's look at an application first:

Player Fetch App

This application takes into its constructor a callable (in the end, this callable turns out to be a function that launches the actual fetching and scraping of a web page containing the player data). It is using constructor dependency injection so that I can replace lookup with a mock and easily test the app.

There is one method here that handles incoming messages, and that is player_fetch_requested. This message is called when the application receives a PlayerFetchMessage. An application can support handling of more than just one type of message, just add more methods for each handling operation. The fact that the name of the function is similar to the message name is just a convention, and doesn't have any significance other than to to be self-documenting.

The method takes in one argument; the message to handle. A method that handles messages like this can then return from the function a message or list of messages of it's own, which is how it sends out messages. The point is that this application is run on top of some messaging middleware that passes the message to the correct function, gets the results of the function (which are always messages) and then sends out the resulting messages.

In this case, the lookup is done, and depending on the results sends out a resulting message. In this case, either a messages with the info on the player, or a message saying that the player doesn't exist.

Now, the application itself is useless, since there's no way of telling the middleware that will be using it what messages to map to what methods. I experimented with naming conventions on the methods and decorators, but eventually settled on a simpler solution: a builder function that does this mapping, as well as giving the application object an dependencies...

def make_app(app_name, root_config, config):
    lookup = get_player_info
    app = FetchPlayerApp(lookup)

    return {
        msgs.PlayerFetchRequested: app.player_fetch_requested,
    }   

Originally, this method was parameterless, but over time has grown to three parameters. app_name is the name as determined by the message bus for the application. The application shouldn't hard-code this or figure it out by itself because it might have parts prepended or appended to it by other parts of the system. The main reason an application would need this is to use it as the name of the logger or data filenames.

The root_config and config parameters are dictionaries containing items that might be run-time options (typically stored in ini files and parsed by a lower part of the engine). root_config typically contains configuration information for all application (such as the directory to store any data files), while config stores information specific to that application (an example might be the exact url to use when fetching, although I've decided to hardcode this right now).

The application builder uses the arguments to create the application object. It then returns a dictionary that maps the message type to the method on the app that should handle that message. Typically, this method is simple enough that it doesn't need testing.

Notice that the application is a POPO (Plain Ole' Python Object). It does not have any dependencies on messaging systems on it. It just takes in the messages, and returns resulting messages. The actual job of sending and receiving those messages on any sort of message bus is up to the object which calls the applications methods.

This allowed me to easily set up prototypes in the early stages by using a hacked together message bus that would just repeatedly send a message to an apps handler, get the resulting messages, then send those out as well. In fact, I pretty much used this system through most of development, not setting up RabbitMQ or anything like that until much later.

Also, because it's a POPO, this application is extremely simple to test. Here is what the test looks like:

class TestPlayerFetchApp(AppTestBase):
    def setUp(self):
        self.lookup = Mock()
        self.sut = PlayerFetchApp(self.lookup)

    def test_fetches_player(self):
        self.lookup.return_value = {'name' : 'test', 'race' : 'zerg'}

        input = msgs.PlayerFetchRequested(1)
        expected = msgs.PlayerFetchAnnouncement(1, 'test', 'zerg')

        result = self.sut.player_fetch_requested(input)

        self.assertContainsMsg(result, expected)

    def test_fetches_non_existant_player(self):
        self.lookup.return_value = None

        input = msgs.PlayerFetchRequested(1)
        expected = msgs.PlayerFetchNonExistantPlayer(1)

        result = self.sut.player_fetch_requested(input)

        self.assertContainsMsg(result, expected)

assertContainsMsg is a helper function that covers all the cases of an application returning a message (returning the message itself, or returning a list of messages with that message in the list). Also, the messages are easy to spot because the BaseMessage object implements value equality. This means...

>>> a = PlayerFetchAnnouncement(1, 'test', 'zerg')
>>> b = PlayerFetchAnnouncement(1, 'test', 'zerg')
>>> c = PlayerFetchAnnouncement(2, 'test2', 'zerg')
>>> a == b
True
>>> a == c
False

Even though object a and b are different objects, BaseMessage overrides the equality operators to ensure that messages with the same data are equal. The main point is to make testing much easier: just make the message you're expecting, rather than having to check all the individual attributes yourself.

That's a pretty simple overview of what an individual application might look like. Typically, your application is either a simple application in itself, just storing and sending data, or is a front-end to more advanced functionality such as this fetch application. It's easy to test, and while is built with messaging in mind has no dependencies on any messaging framework.