What is Stackless?

I sometimes get this question. And instead of starting a rant about microthreads, co-routines, tasklets and channels, I present the essential piece of code from the implementation:

The Code:

/*
    the frame dispatcher will execute frames and manage
    the frame stack until the "previous" frame reappears.
    The "Mario" code if you know that game :-)
 */

PyObject *
slp_frame_dispatch(PyFrameObject *f, PyFrameObject *stopframe, int exc, PyObject *retval)
{
    PyThreadState *ts = PyThreadState_GET();

    ++ts->st.nesting_level;

/*
    frame protocol:
    If a frame returns the Py_UnwindToken object, this
    indicates that a different frame will be run.
    Semantics of an appearing Py_UnwindToken:
    The true return value is in its tempval field.
    We always use the topmost tstate frame and bail
    out when we see the frame that issued the
    originating dispatcher call (which may be a NULL frame).
 */

    while (1) {
        retval = f->f_execute(f, exc, retval);
        if (STACKLESS_UNWINDING(retval))
            STACKLESS_UNPACK(retval);
        /* A soft switch is only complete here */
        Py_CLEAR(ts->st.del_post_switch);
        f = ts->frame;
        if (f == stopframe)
            break;
        exc = 0;
    }
    --ts->st.nesting_level;
    /* see whether we need to trigger a pending interrupt */
    /* note that an interrupt handler guarantees current to exist */
    if (ts->st.interrupt != NULL &&
        ts->st.current->flags.pending_irq)
        slp_check_pending_irq();
    return retval;
}

(This particular piece of code is taken from an experimental branch called stackless-tealet, selected for clarity)

What is it?

It is the frame execution code. A top level loop that executes Python function frames. A “frame” is the code sitting inside a Python function.

Why is it important?

It is important in the way it contrasts to C Python.

Regular C Python uses the C execution stack, mirroring the execution stack of the Python program that it is interpreting. When a Python function foo(), calls a python function bar(), this happens by a recursive invocation of the C function PyEval_EvalFrame(). This means that in order to reach a certain state of execution of a C Python program, the interpreter needs to be in a certain state of recursion.

In Stackless Python, the C stack is decoupled from the Python stack as much as possible. The next frame to be executed is placed in ts->frame and the frame chain is executed in a loop.

This allows two important things:

  1. The state of execution of a Stackless python program can be saved and restored easily. All that is required is the ability to pickle execution frames and other runtime structures (Stackless adds that pickling functionality). The recursion state of a Python program can be restored without having the interpreter enter the same level of C recursion.
  2. Frames can be executed in any order. This allows many tasklets to be created and code that switches between them. Microthreads, if you will. Co-routines, if you prefer that term. But without forcing the use of the generator mechanism that C python has (in fact, generators can be more easily and elegantly implemented using this system).

That’s it!

Stackless Python is stackless, because the C stack has been decoupled from the python stack. Context switches become possible, as well as the dynamic management of execution state.

Okay, there’s more:

  • Stack slicing: A clever way of switching context even when the C stack gets in the way
  • A framework of tasklets and channels to exploint execution context switching
  • A scheduler to keep everything running

Sadly, development and support for Stackless Python has slowed down in the last few years. It however astonishes me that the core idea of stacklessness hasn’t been embraced by C Python even yet.