Comments for Kristján's Cosmic Percolator

Comment on Surprising Python by Kristján Valur

Kristján Valur — Mon, 27 May 2013 10:41:50 +0000

“But again how do you install it into $PATH “
The answer is that you don’t. You add the directory to $PATH.
Symbolic links aren’t “shortcuts”. Shortcuts need help from the shell to manually translate them to the proper place. The “dereferencing” of the link should not be the job of the executable. It is a terrible, terrible, hack if the program starts performing the job of the shell for the convenience of a subset of the users.

You mention a number of programs that have this property. I’m curious as to what they are. I gave a pretty good counter-example with the world’s most ubiquitous programming language.

It think that if Python, or any language, decides to perform this sort of shortcut patching for the shell, it should at least be enabled via an optional argument.

As for hard links, nobody in their right mind uses hard links after symbolic links were invented

Comment on Surprising Python by Beni Paskin-Cherniavsky

Beni Paskin-Cherniavsky — Sat, 25 May 2013 02:11:35 +0000

Let me argue that it’s not an obscure feature but a widely applicable unixy behavior.
Consider *any* program whose installation consists of more than one file. One (at least) is an executable, which you would like to install somewhere on $PATH (/usr/bin, a personal ~/bin etc.). The other supporting files (whether data or dynamically loaded code) do not belong there. How will the executable find them?

One approach is to hardcoding: something like “./configure –prefix=/usr; make” creates an executable that always looks under /usr/lib/foo and/or /usr/share/foo. A common variant is not compiling the path into the main binary but generating a small wrapper script that knows the path.

But that’s not good enough.
(1) If you’re developing the program, you want to test it without installing in a centralized location.
(2) Same you’re a user who just checked it out from version control. You might prefer to just run it from there — less of a mess then installing, and you can just “git pull” the newest version without rebuilding (esp. if it’s a purely interpreted code).
(3) Remote execution (like Hadoop) is easier if your program can run unmodified from any location.

The obvious — unavoidable — solution is the executable inspecting argv[0], e.g.
do_something $(dirname $0)/supporting-file

But again how do you install it into $PATH without dragging the supporting files? Copying or hardlinking doesn’t leave a trace of where to look for the supporting files, and has to be repeated if you modify/upgrade the program. But symlinking is perfect! For it to work, the program must do:
do_something $(dirname (readlink -f $0))/supporting-file
Indeed I’ve seen this pattern in many programs, and it is precisely what Python automates when filling in sys.path[0]!

[P.S. There is also the self-extracting archive approach, which gives you a standalone fully relocatable binary. This is handy for remote deployment, but it's pretty ugly... Python has builtin zipimport allowing you to avoid extraction — but only for pure python, extension modules must be extracted since you can't dlopen() from memory.]

=>
IMHO the Right solution would be for Hadoop to use a hardlink tree rathan symlink tree. Hardlinks are truly transparent so python — or any other app — would not break on them.
[I believe most uses of linktrees to save space, including mercurial clones, employ hardlinks. Mostly because they're symmetric - you can delete the original tree and the clone would not be broken.]

Comment on Surprising Python by Kristján Valur

Kristján Valur — Fri, 24 May 2013 09:27:06 +0000

It’s a an misconceived contract though, IMHO, and I would like to see us take steps to correct our mistakes. There are ways to do it, optional flags, for example.
But I’m not going to fight that battle, it is a relatively minor issue as it is. The point of my post was to point out a very surprising, and very obscure, feature. No one seemed to even know about it, except the BDFL

Comment on Surprising Python by Israel Fruchter

Israel Fruchter — Fri, 24 May 2013 09:13:04 +0000

I agree it’s weird, but Guido has a point there.
Not breaking the contract with users (same approach as linux kernel)

Last time Python did it with Strings in Python3. the debacle about it isn’t finished to this day.

Comment on Atomic by Kristján Valur

Kristján Valur — Sat, 23 Mar 2013 16:14:09 +0000

Yes, I should have mentioned that I had a conversation with Armin about this when he originally mentioned this on the PyPy blog. His patch was more ambitious, and involved not releasing the GIL even when explicitly doing so around blocking IO calls or other long duration calls. The point of my proposal is to limit ourselves to merely stopping the volountary release of the GIL, but leaving explicit GIL release alone. This does not introduce any new deadlock cases.

Comment on Atomic by Nick Coghlan (@ncoghlan_dev)

Nick Coghlan (@ncoghlan_dev) — Sat, 23 Mar 2013 15:44:12 +0000

Armin Rigo suggested much the same thing as a hook for C extensions to play with when he first started working on his STM experiment, so it sounds good to me. (Armin ultimately rejected his own patch, http://bugs.python.org/issue12850, as not sufficiently interesting, but I think a cheap “threading.atomic()” context manager is interesting enough to be worthwhile, particularly since it allows certain data structure manipulation code to be made safe across multiple interpreter implementations)

Comment on Blog moved by Kristján Valur

Kristján Valur — Wed, 13 Mar 2013 18:22:36 +0000

And: It turned out that wordpress didn’t actually copy the images in the old blog and stored it locally. So, I must now resurrect the blog again, and find a way to have those files moved.

Comment on Blog moved by Kristján Valur

Kristján Valur — Sat, 09 Mar 2013 10:16:59 +0000

And it is done!
After resurrecting my blog on a virtual machine at the office, I realized that it needed to be accessible in the Interzone so that wordpress.com could migrate its images and other content.
So, I zipped it up, transfered it to my phone, and fired it up at home again, on my home workstation which is accessible via NAT traversal at krisvale.servebeer.com.

So, now it is back to doing interesting stuff!

Comment on Optimizing Python Condition Variables with Telemetry by Kristján Valur

Kristján Valur — Tue, 29 May 2012 09:25:46 +0000

Yes, in many other cases we have non-python worker threads and then can keep the queueing non-Python. In this case, I specifically wanted to use Python worker threads to spare me some pain.
One particularly bothersome thing with using non-python workers, is that object lifetime management becomes more difficult, since the worker cannot use reference counting. For example, the worker needs its own reference to arguments, or needs to copy them, because the client may get killed before the worker exits.

Comment on Optimizing Python Condition Variables with Telemetry by Rene Dudfield

Rene Dudfield — Mon, 28 May 2012 14:34:44 +0000

Hi,

nice article, thanks. I’ve found using a non-python queue is quite useful for speeding up things. The worker threads can insert into the queue without obtaining the python GIL. Works nicely if your worker code is mostly C stuff. Seems like you’ve sped up the python side queue stuff quite a lot where that might not be necessary anymore.