Month: September 2010

selectmodule on PS3

Kristján Valur Python, Stackless , ,

As part of a game that we are developing on the Sony PS3, we are porting Stackless Python 2.7 to run on it.  What would be more natural?

Python is to do what it does best, act as a supervising puppetmaster, running high-level logic that you don’t want to code using something primitive such as C++.

While there are many issues that we have come across and addressed (most are minor), I’m just going to mention a particular issue with sockets.

An important role of Python will be to manage network connection using sockets.  We are using the Stacklesssocket module with Stackless Python do do this.  For this to work, we need not only the socketmodule but also the selectmodule.

The PS3 networking api is mostly BSD compliant with a few quirks.  Porting these modules to use it is mostly a straightforward affair of providing #defines for some API calls that have different names, dealing with APIs that are missing (return NotImplementedError and such) and mapping error values to something sensible.  For the most part, it behaves exactly as you would expect.

We ran into one error though, prompting me to write special handling code for select() and poll().  the Sony implementation of these functions not only returns with an error if they themselves fail (which would be serious) but they also indicate a error return if a socket error is pending on one of the sockets.  For example, if a socket receives a ECONNRESET, while you are waiting for data to arrive, select() will return with an ECONNRESET error indicator.  Not what you would expect.

The workaround is to simply filter the error values from select() and poll() and ignore the unexpected socket errors.  Rather, such a return must be considered a successful select()/poll() and for the latter function, ‘n’, the number of valid file descriptors, having been set as -1, must be recreated by walking the list of file descriptors.

Embedding Python and sys.path

Kristján Valur Python ,

When embedding a Python interpreter in another application, such as a game, a tricky parts is to get it to start up and find the right modules.  You typically want Python to:

  • Find your version of the standard library
  • Find your application modules
  • Not be confused by any other Python system that may be installed on the machine.

The application has absolute knowledge of the location of the python modules.  It has been installed in a known location and it knows where its resources are.

Unfortunately, Python often appears to be designed for the specific needs of python.exe, the “canonical embedding application”.  When one calls Py_Initialize(), a number of magic steps are invoked whereby the Python runtime tries to guess the initial setting for sys.path.  This may involve finding the program location, looking at the PYTHONPATH environment variable, and so on.  When Py_Initialize() returns, it may be too late to modify sys.path because in the mean time, module imports could have occured, such as that of copy_reg.

I have found that circumventing this involves patching pythoncore itself.  What we do in EVE before calling Py_Initialize() is to call a custom API, Py_SetPath() with an application-crafted semicolon-separated path string.  This will turn off any path-guessing in pythoncore and gives us absolute control.

Before we started doing this, for example, we could not encourage developers to have Python installed on their workstations, for example.  We would invariably run into conflicts where the embedded python picked up modules from the installed python and mysterious things would happen.  Now there is no need, and the embedded Python and the installed Python can coexsist without conflict.

I’ve recently contributed this patch to python.org.