A little known feature of EVE Online, disabled in the client but very much active in the server, is a web server. This was added early in the development, before I started on the project back in 2003. It is the main back-end access point to the game server, used for all kinds of management, status information and debugging.
Back then, Python was much less mature as a web serving framework. Also we initially wanted just very rudimentary functionality. So we wrote our own web server. It, and the site it presents, were collectively called ESP, for Eve Server Pages and over the years it has grown in features and content. The heaviest use that it sees is as the dashboard for Game Managers, where everything a GM needs to do is done through HTML pages. It is also one of the main tools for content authoring, where game designers access a special authoring server. ESP presents a content authoring interface that then gets stored in the backend database.
Recently we have increasingly started to look for alternatives to our homegrown HTTP solution, though. The main reasons are:
- We want to use a standard Python web framework with all the bells and whistles and support that such frameworks offer
- We want modern Web 2.0 features without having to write them ourselves
- We want something that our web developers can be familiar with already
- We want to share expertise between the ESP pages and other web projects run by CCP. Confluence of synergies and all that.
EVE Online is based on Stackless Python. Embedded into the the game engine is a locally patched version of Stackless Python 2.7. We have been using Stackless since the very beginning, the existence of Stackless being the reason we chose Python as a scripting solution for our game.
Systems like the web server have always depended heavily on the use of Stackless. Using it we have been able to provide a blocking programming interface to IO which uses asynchronous IO behind the scenes. This allows for a simple and intuitive programming interface with good performance and excellent scalability. For some years now we use the in-house developed StacklessIO solution which provides an alternative implementation of the socket module. By also providing emulation of the threading module by using tasklets instead of threads, many off the shelf components simply work out of the box. As an example, the standard library’s xmlrpc module, itself based on the socketserver module, just works without modification.
Of course, the use of Stackless python is not limited to IO. A lot of the complicated game logic takes advantage of its programming model, having numerous systems that run as their own tasklets to do different things. As with IO, this allows for a more intuitive programming experience. We can write code in an imperative manner where more traditional solutions would have to rely on event driven approaches, state machines and other patterns that are better left for computers than humans to understand. This makes CCP very much a Stackless Python shop and we are likely to stay that way for quite a bit.
It was therefore with a great deal of interest that we noticed the announcement of Nagare on the Stackless mailing list a few years ago.
Nagare promises a different approach to web development. Instead of web applications that are in effect giant event handlers (responding to the HTTP connectionless requests) it allows the user to write the web applications imperatively much as one would write desktop applications. Their web site has a very interesting demo portal with a number of applications demonstrating their paradigm, complete with running apps and source code.
This resonates well with me. I am of the opinion that the programmer is the slowest part of software development. Anything that a development environment can do to make a programmer able to express himself in familiar, straight-forward manner is a net win. This is why we we use tasklet blocking tasklet IO instead of writing a game based on something as befuddling as Twisted. And for this reason I thought it worthwhile to see if a radical, forward thinking approach to web development might be right for CCP.
Nagare achieves its magic by using a little known Stackless feature called tasklet pickling. A tasklet that isn’t running can have its execution state pickled and stored as binary data. The pickle contains the execution frames, local variables and other such things.
Stackless Python contains some extensions to allow pickling of various objects whose pickling isn’t supported in regular Python. These include frames, modules and generators among other things.
When a Nagare web application reaches the point where interaction with the user is required, its state is pickled and stored with the Nagare server, and a HTTP response sent back to the client. When a subsequent request arrives, the tasklet is unpickled and its execution continues. From the programmer’s point of view, a function call simply took a long time. Behind the scenes, Stackless and Nagare were working its magic, making the inherently stateless HTTP protocol seem like smooth program flow to the application.
In other ways, Nagare is a very traditional Python web framework. It is based around WSGI and so uses whatever threading and IO model provided to it by a WSGI server.
Unlike some smaller frameworks, Nagare is designed and distributed as a complete web server. One typically installs it as the single application of a virualenv root, and then configures and runs it using a script called nagare_admin. This is very convenient for someone simply running a web server, but it becomes less obvious how to use it as a component in a larger application.
What we were interested in doing was to see if Nagare would work within the application that is EVE. To do this we would have to:
- Extract Nagare and its dependencies as a set of packages that can be used with the importing framework that EVE uses. As I have blogged about before, Python by default assumes a central package directory and doesn’t lend itself well to isolation by embedded applications. We have done that with EVE, however, and would want nagare to be an “install free” part of our application.
- Set up the necessary WSGI infrastructure within EVE for Nagare to run on
- Configure and run Nagare programmatically rather than by using the top level application scripts provided by the Nagare distribution.
I specifically didn’t intend to evaluate Nagare from the web developer’s point of view. This is because I am not one of those and find the whole domain rather alien. This would be a technical evaluation from an embedding point of view.
My first attempt at setting up Nagare was to fetch the source from its repository.
svn co svn://www.nagare.org/trunk/nagare
I then intended to fetch any dependencies in a similar manual manner. However, I soon found that to be a long and painstaking process. In the end I gave up and used the recommended approach: Set up a virtualenv and use:
This turned out to install a lot of stuff. This is the basic install and a total of 13 packages were installed in addition to Nagare itself, a total of almost 12Mb. The full install of Nagare increases this to a total of 22 packages and 22Mb.
The original plan was to take this and put it in a place where EVE imports its own files from. But because of the amount of files in question, we ended up keeping them in place, and hacking EVE to import from <NAGARE_HOME>Libsite-packages.
Setting up WSGI
Nagare requires Paste and can make use of the Paste WSGI server. Fortunately, EVE already has a built-in WSGI service, based on Paste. It uses the standard socket module, monkeypatched to use StacklessIO tasklet-blocking sockets. So this part is easy.
Fitting it together
This is where it finally got tricky. Normally, Nagare is a stand-alone application, managed with config files. A master script, nagare_admin, then reads those config files and assembles the various Nagare components into a working application. Unfortunately, documentation about how to do this programmatically was lacking.
However, the good people of Nagare were able to help me out with the steps I needed to take, thus freeing me from having to reverse-engineer a lot of configuration code. What I needed to do was to create a Publisher, a Session manager and the Nagare application I need to run. For testing purposes I just wanted to run the admin app that comes with Nagare.
After configuring EVE’s importer to find Nagare and its dependencies in its default install location, the code I ended up with was this:
#Instantiate a Publisher (WSGI application) from nagare.publishers.common import Publisher p = Publisher() #Register the publisher as a WSGI app with our WSGI server sm.StartService('WSGIService').StartServer(p.urls, 8080) #our WSGI server #instantiate a simple session manager from nagare.sessions.memory_sessions import SessionsWithMemoryStates sm = SessionsWithMemoryStates() #import the app and configure it from nagare.admin import serve, util, admin_app app = admin_app.app app.set_sessions_manager(sm) #register the app with the publisher p.register_application('admin', 'admin', app, app) #register static resources def lookup(r, path=r"D:nagareLibsite-packagesnagare-0.3.0-py2.5.eggstatic"): return serve.get_file_from_root(path, r) p.register_static('nagare', lookup)
This gave the expected result. Browsing to port 8080 gave this image:
So, success! EVE was serving a Nagare app from its backend.
These tests showed that Nagare does indeed work as a backend webserver for EVE. In particular, the architecture of StaclessIO sockets allows most socket-based applications to just work out of the box.
Also, because Nagare is a Python package, it is inherently programmable. So, it is possible to configure it to be a part of a larger application, rather than the stand-alone application that it is primarily designed to be. Using Nagare as a library and not an application, however, wasn’t well documented and I had to have some help from its friendly developers and read the source code to get it to work.
On the other hand, Nagare is a large application. Not only is Nagare itself a substantial package, it also has a lot of external dependencies. For an embedded application of Python, such as a computer game, this is a serious drawback. We like to be very selective about what modules we make available within EVE. The reasons range from the purely practical (space restraints, versioning hell, build management complexity) to externally driven issues like security and licensing.
It is for this reason that we ultimately decided that Nagare wasn’t right for us as part of the EVE backend web interface. The backend web interface started out as a minimal HTTP server and we want to keep it as slim as possible. We are currently in the process of picking and choosing some standard WSGI components and writing special case code for our own use. This does, however, mean that we miss out on the cool web programming paradigm that is Nagare within EVE.