In our port of Python 2.7 to the PS3 console, we have deliberately removed the python compiler. This was mainly done to save on the code size, since on a console every byte is sacred. An additional benefit is slight hardening against certain kinds of attacks, since evil constructs such as eval() and exec() now raise the NotImplementedError when used.
Program code is pre-compiled and put in .zip archives so there is no need for regular compilation on the console. The most serious problem we encountered though, was with the new namedtuple construct.
The namedtuple is implemented in the collections module by constructing a class declaration with string interpolation and then calling exec() on it. With exec() removed, a lot of the standard library turned out to fail on import.
Our initial fix was simply to replace the namedtuples with regular tuples:
[python]
def namedtuple(typename, field_names, verbose=False, rename=False):
return tuple
[/python]
This worked surprisingly well. The parts of the library we were using were still using namedtuples just like regular tuples and all was well.
Recently, however, we found that the urlparse module was making non-trivial use of it so something needed to be done. My initial reflex was to dive in and reimplement it using a metaclass or some such. But then I thought of asking the internet.
It turns out that this exists as an issue in the Python bug tracker. Someone else had come across this oddity in the standard library and submitted an alternative implementation. This works perfectly for our purposes.
I know that there is nothing inherently evil about using exec in Python, but this particular case still doesn’t quite ring true to me: If the best way to implement a class is by resorting to the meta-language, doesn’t that indicate some shortcoming in the language itself?
Kristján's Cosmic Percolator 
eval()/exec are also critical in order to produce signature-preserving decorators. They are at the core of the ubiquitously-used decorator module.
I see. I imagine that a custom wrapper is then created using exec every time.
This btw has a downside common to namedtuple which I didn’t bring up: Code duplication.
Every namedtuple class will define n almost identical versions of common functions such as __repr__ which is wasteful. Similarly, the wrappers in the decorator module are presumably created uniquely for each function.
sure, that it has to use exec at all is horrific. So there’s your “missing python feature” – a “decorator” built-in that just does “that thing” – produces a function that appears identical to documentation tools like Sphinx.
The __signature__ object proposed by brett would have solved that.
How?
Interesting. You’d have a different __signature__ for each closure, then?
The use of
exec in namedtuple has bothered me since it was introduced to the standard library. I even rewrote it at one point to avoidexec()calls.Well yes, there’s a missing feature – but it isn’t just being able to create functions with specific signatures. The function then has to use the arguments from the signatures (which are then local variables), and how *can* you do that without code generation?
Mock does something similar to the mock module (generating functions with the same signature as a function / method they’re copying) and it’s something I’ve thought about. It may be a failure of imagination, but I can’t think of how this “feature” *could* look (ast generation and compilation is just a different version of the same thing so wouldn’t really be any better).
Some of the core features of namedtuple, by the way, can be got with very little code at all. Here’s my take in 12 lines of Python (and no exec):
http://paste.pocoo.org/show/396729/
It’s missing many of the features of course. Given that exec *is* a core language feature (although as Kristjan points out there are reasons to remove it), I don’t see that using it is *automatically* evil.
I’m not just arguing that it’s not evil. I’m also arguing that a Python implementation that doesn’t have it is nearly useless, as a tremendous number of widely used libraries rely upon it.
I think “nearly useless” is overstating it a bit. I think for example that it makes perfect sense for “deployed” python applications (think py2exe). I was surprised at how much of the standard library worked without it. Namedtuple is the only thing we’ve come across yet that has caused us problems.
[…] second post is on the NamedTuple kerfluffle that was stirred up Kristjan Valur’s post on the use of exec() and namedtuple (short version: namedtuple creates a string defining the class and then calls exec to […]
[…] namedtuple and exec() by Kristján Valur […]
Generating code from templates is a somewhat basic and powerful technique. Tools like ANTLR and templating languages leverage this technique to good effect.
Early versions of namedtuple used type instead of exec but that approach didn’t scale well — it was hard to maintain, somewhat unreadable, had odd bugs, and wasn’t self-documenting.
After some clean-up, the exec version ended-up being somewhat beautiful: hg.python.org/cpython/file/3.2/Lib/collections.py#l235
I acknowledge that exec makes some developers feel ikcy but that is a psychological issue, not a technical one.