SC Engine: Part 7 - The Console

- Introduction
- Part One: System Overview
- Part Two: System Overview: Messages and Applications
- Part Three: Screen Scraping
- Part Four: YouTube Parsing
- Part Five: Linking the Video to the Game
- Part Six: Messaging Middleware
- Part Seven: The Console
- Part Eight: The Site

For the most part, SC Engine is designed to be automatic: Read the schedules, get the video data, put them together, upload to a site. However, there are plenty of things that need to be manually done:

The engine might not be advanced enough to parse enough data from the title and description of a youtube video to determine the game. If I detect one of these situations, I want to be able to see the video and what the engine DID manage to get from it so I can either fix the engine and retry the video, manually link the video to a game, or just ignore the video. Also, while the messaging middleware should do well to ensure that the engine stays alive during application's raising errors, I do want some way to see what these errors were so I can figure out what needs fixing.

For this, I've written a web site designed to work as a frontend to the workings of the engine, called "SC Console". The console is a simple pylons web site, that looks like this:



The two main things are error reports and youtube partial parses:

Error Reports

The error report lists all of the errors that have been encountered that still need to be dealt with. It gives some quick data for all the errors (in the screenshot, have just one) such as the time the error occurred, the name of the application that the error occurred in, the name of the message that was being handled when the error occurred, and the error message itself (constructed by converting the error raises into a string). There's also the ability to retry (that is, the message will be sent out again) or deleted.

It's important to realize that retrying will send the message out again, so if a message is handled by two applications and one raises an error that we later retry, BOTH applications will be asked to handle the message. Luckily, it typically isn't too difficult to write applications in to gracefully handle the same message more than once, but it is something to be aware of. An application that would keep track of how many games a player has ever played, for example, would need to keep track of the actual games, not just increment a counter every time a message arrives stating that a player is in a game.

Anyway, clicking on "View" gives you more info on the error:



In addition to the data seen in the list screen, we also get to see the actual data that was contained in the message (which is typically a dictionary of values), the traceback of the error, as well as any log messages that were written during the application's execution of handling the message.

Youtube Partial Parses

This screen shows me all the cases where a youtube video came to my attention, had enough data to possibly be a proleague game, but I couldn't determine what game it was for. It shows the date & time of the attempted link, the video id (with a link that I can use to easily open the video), and the title of the video for quick reference Additionally, the dates, players, teams and game numbers parsed from the videos are shown, as well as the number of possible results found. Of course, the same retry and delete buttons.

Clicking on view gives me more info:



The date played links to the kespa website, specifically to the page for that schedule, which is a handy feature. The player ids also have a link to the kespa page.

Down below, information on the games that are seen as possible candidates are shown. These help me figure out if the engine is anywhere in the ballpark in terms of guessing the game, and also give me all the info I need that, if I wanted to, add to the script that tells the engine to manually mark videos for games. However, for most situations (including this one), the solution is to add aliases for players. In this case, players "Baxter" and "[Oops]Cloud" were not recognized. Some sluethwork reveals that "Baxter" is another name for a player typically known as "Killer" (id: 1305), and "[Oops]Cloud" is typically just known as "Cloud" (id: 348). For something like this I would add these two new aliases to the engine, and then hit "retry". This time around, the link would be made, because the one other game would no longer be reported as a candidate. This is a better result than manually linking, because now future games that use these aliases will also be immediately recognized, causing less work for me.

To quickly go over the rest of the menu:

Proleague Partial Fetch PP:

This is a feature I still haven't implemented, but I put it in the menu anyway. It will be similar to YouTube partial parse, except instead of showing ambiguous YouTube videos, it'll show ambiguous game schedules.

Utils:

This contains a bunch of forms that can be filled out to send specific messages, such as a schedule fetch for a certain day. Was very useful when the only messaging system was just a locally storing the messages in a list in a single application, where the only way I could add new messages was to shut down the application and change the code. However, it still works, so I keep it around.

Status:

Will eventually show a "heartbeat" status of all running apps, which is information on how long an application's message queue is by sending out a heartbeat and seeing how long before the heartbeat is responded too. Just a quick way to check the current load in addition to viewing the actual messaging systems queues.