Countergram

Group a list into chunks in Python

Thu, 06 May 2010 00:00:00 EST

More than once I've had to take a list of items and group them. For instance, maybe I have a flat list of key-value pairs in alternation, like this:

[key1, value1, key2, value2, key3, value3]

It could have been that they were easier to enter that way (fewer parentheses or brackets), or maybe that's just how they come from an outside source.

Nevertheless, to actually use them, it would be better to have them in tuples:

[(key1, value1), (key2, value2), (key3, value3)]

That way, I could make them into a dictionary, urlencode them, etc.

I looked through Python's entire itertools module for a pre-existing solution, but didn't find one. itertools does have the reverse operation, chain, but not this one.

So, to save you the trouble, here's the code:

group_iter.py (license: public domain)

def group_iter(iterator, n=2, strict=False):
    """ Transforms a sequence of values into a sequence of n-tuples.
    e.g. [1, 2, 3, 4, ...] => [(1, 2), (3, 4), ...] (when n == 2)
    If strict, then it will raise ValueError if there is a group of fewer
    than n items at the end of the sequence. """
    accumulator = []
    for item in iterator:
        accumulator.append(item)
        if len(accumulator) == n: # tested as fast as separate counter
            yield tuple(accumulator)
            accumulator = [] # tested faster than accumulator[:] = []
            # and tested as fast as re-using one list object
    if strict and len(accumulator) != 0:
        raise ValueError("Leftover values")

YouTube video embedding with XSLT

Mon, 03 May 2010 00:00:00 EST

A while ago, I published an article on how to use Python's docutils to expand some markup in reStructuredText into the HTML for a YouTube embedded video. Just for fun, here's an XSLT stylesheet that does something similar. It hasn't had a chance to be used in production, but if you have an XSLT-based web application, it might come in useful.

youtube.inc.xslt

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns="http://www.w3.org/1999/xhtml">

    <!-- Replaces elements of thei form <youtube id="asdfasdf" /> with the
    correct XHTML object element to embed a YouTube video -->

    <!-- If desired, one can also place arbitrary elements inside the youtube
    element that will be converted into parameters of the object element.
    E.g. the element <fullscreen>true</fullscreen> will be converted to
    <param name="fullscreen" value="true" /> -->

    <xsl:strip-space elements="youtube"/>

    <xsl:template match="youtube">
        <xsl:element name="object">
            <xsl:attribute name="type">application/x-shockwave-flash</xsl:attribute>
            <xsl:attribute name="width">425</xsl:attribute>
            <xsl:attribute name="height">344</xsl:attribute>
            <xsl:attribute name="class">youtube-embed</xsl:attribute>
            <xsl:attribute name="data">
                http://www.youtube.com/v/<xsl:value-of select="@id"/>
            </xsl:attribute>
            <xsl:element name="param">
                <xsl:attribute name="name">movie</xsl:attribute>
                <xsl:attribute name="value">
                    http://www.youtube.com/v/<xsl:value-of select="@id"/>
                </xsl:attribute>
            </xsl:element>
            <xsl:element name="param">
                <xsl:attribute name="name">wmode</xsl:attribute>
                <xsl:attribute name="value">transparent</xsl:attribute>
            </xsl:element>
            <xsl:apply-templates mode="youtube-params" />
        </xsl:element>
    </xsl:template>

    <xsl:template match="youtube/*" mode="youtube-params">
        <xsl:element name="param">
            <xsl:attribute name="name"><xsl:value-of select="local-name()" /></xsl:attribute>
            <xsl:attribute name="value"><xsl:value-of select="text()" /></xsl:attribute>
        </xsl:element>
    </xsl:template>

</xsl:stylesheet>

As the comment notes, you need to run this on an XML document that contains elements like <youtube id="asdfasdf" />. It's not currently configurable as to the width, height, and other parameters (except by changing the XSLT) but could be made so.

Pylons deployment with Lighttpd and Flup

Thu, 22 Apr 2010 00:00:00 EST

What's fast, easy to set up, and good for running web applications?

Wait, did that sound too much like a joke? Anyway, I'm talking about the combination of Lighttpd (a web server and alternative to Apache) and Pylons (a Python web framework).

Lighttpd (as well as the similar nginx) has been getting an increasing amount of attention in the past few years, especially for use with web applications in combination with dynamic languages such as Python and Ruby. It has a relatively small memory footprint and, due to its asynchronous network layer, can easily handle both dynamic web applications and lots of static files at the same time.

The "catch" is that it never runs apps in embedded interpreters, as Apache modules such as mod_php, mod_perl and mod_wsgi do. Instead, it farms requests out to apps running in separate processes, using protocols such as FastCGI or SCGI, or acting as an HTTP proxy.

This article will show you a basic setup for Pylons and Lighttpd, using the FastCGI protocol. I'm going to assume you already have a basic Pylons app set up and know how to run it; if not, hop over to the Pylons Getting Started documentation. Lighttpd installation instructions can be found on the Lighttpd documentation page.

Let's go.

Pylons configuration

I've started off with the FastCGI protocol here for reasons discussed below. SCGI, for those who prefer, is a drop-in replacement (you can actually just change all instances of fcgi and fastcgi to scgi in the Pylons and Lighttpd configuration examples.) HTTP proxying is only slightly different.

You will need the flup package installed. Use easy_install flup or pip install flup, or visit the Flup home page.

Only a small change needs to be made to the INI file for your Pylons application (e.g.: development.ini) to use Flup instead of the default Paste HTTP server.

development.ini (partial)

[server:main]
use = Flup#fcgi
host = localhost
port = 5000

You can then run the app with paster serve --reload development.ini, as usual.

Lighttpd configuration

Below is a minimal lighttpd.conf containing configuration that you can adapt to your needs. It's missing a lot of what you would actually want, such as logging and MIME types, so cut and paste liberally.

lighttpd.conf

server.modules = (
    "mod_fastcgi"
)

server.port = 80
server.document-root = "/dev/null"

$HTTP["host"] =~ "^example.com|localhost$" {
    server.document-root = "/path/to/files/"
    $HTTP["url"] !~ "^/static/" {
        fastcgi.server = ("/" => ((
            "host" => "127.0.0.1",
            "port" => 5000,
            "check-local" => "disable",
            "disable-time" => 1,
            "fix-root-scriptname" => "enable"
        )))
    }
}

This configuration uses Lighttpd's regular-expression selectors to create a host that can be accessed either as example.com or as localhost on port 80. This is suitable if you are sharing one configuration file between your local development machine and the example.com server, but unnecessary if you are using separate configuration files, or remapping example.com to your local machine for development purposes.

The positioning of server.document-root and the second regular-expression selector means that URLs that start with /static/ will be served by Lighttpd from /path/to/files/static/, while everything else will be served by the application.

Disabling check-local means that for those URLs handled by the application, Lighttpd will not attempt to look at the filesystem at all. disable-time, on the other hand, is the amount of time Lighttpd will wait between tries if it fails to connect to the application server. Setting this to a small value makes restarts or redeployments of your app go faster, if you have not implemented a seamless restarting process for your app.

The fix-root-scriptname argument is necessary for any WSGI app, Pylons or otherwise, that is mounted on /. Otherwise SCRIPT_NAME and PATH_INFO will not be set properly.

Why FastCGI?

Some perceive FastCGI as outdated, but it works well, and given that Flup and Lighttpd handle the protocol for you there's no real need to worry about it. Still, there are options.

If you would like to use SCGI instead of FastCGI, simply load mod_scgi in server.modules and change fastcgi.server to scgi.server in the config above. Lighttpd 1.4.15 and previous have a segfaulting bug in mod_scgi, and it should not be used. Since SCGI and FastCGI are otherwise drop-in replacements and their performance seems equivalent, I simply used FastCGI.

You could also use Paste's HTTP server (the same one used for development) behind mod_proxy. This looks like the above, but load mod_proxy in server.modules, and put following in place of the fastcgi.server block:

proxy.server = ("" => ((
    "host" => "127.0.0.1",
    "port" => 5000
)))

Unfortunately, in my tests Paste's HTTP server (based on the standard library's BaseHTTPServer) did not hold up under load, whereas Flup did spectacularly. You could also try another Python HTTP server that speaks WSGI.

Load balancing

All three options (FastCGI, SCGI, and proxying) in Lighttpd come with load-balancing capability. See the "extra" set of parentheses around the host and port definition in the configuration examples above? Simply add multiple such blocks, like this:

proxy.server = ("" => (
    ("host" => "10.0.0.10",
    "port" => 5000),
    ("host" => "10.0.0.11",
    "port" => 5000)
))

If using mod_proxy, you can then alter the load-balancing algorithm using the proxy.balance parameter (see the mod_proxy docs).

What about mod_wsgi?

Apache (not Lighttpd) has mod_wsgi, which runs one or more embedded Python processes in each Apache process, passing requests directly to WSGI applications. This is a solid way to deploy Pylons apps.

See the Integration With Pylons document on the mod_wsgi site for information on how to use paste.deploy.loadapp to make Pylons compatible with mod_wsgi.

An advantage of mod_wsgi is that you don't have to keep track of as many processes. However, if you use Supervisor to manage your Pylons apps and Lighttpd, it's no trouble at all.

A disadvantage of mod_wsgi is that you don't easily have the ability to split your HTTP server and app servers across multiple machines, which is easy with Lighttpd and a separate Python process. You can do it, of course, by running Apache on each machine and using mod_proxy or a load balancer.

Also, if you are building your own Apache and Python from source, you need to be very careful about library versions. If Apache and Python are built against different versions of the Expat library, for instance, mod_wsgi will segfault. This is not a concern when Python is running as a separate process, rather than embedded.

(See Graham Dumpleton's comment, below, for some updates -- mod_wsgi can run daemon processes, although they're managed by mod_wsgi, and apparently the Expat library hasn't been a problem for a while.)

This isn't too farfetched -- if you want to use a more recent version of Python than is available in your package manager (some stable Linux distributions seem to habitually lag behind Python point releases), or if you want to use the exact same version on different platforms, you'll probably build your own.

One final note on mod_wsgi: writing to STDOUT (e.g. using the print statement) will cause serious errors, so make sure to excise all print statements and use logging instead.

Addendum

Lighttpd is my personal favorite way to deploy Python web applications. (Some people prefer nginx, also a solid option.)

Not only does separating the HTTP process from the Python process allow for easy and flexible deployment, but Lighttpd has an amazingly useful configuration system. The basic syntax is easy to work with, and for advanced uses you can generate configuration files dynamically (see: include_shell) or run Lua scripts in the server process to do advanced URL rewriting/redirecting, maintenance notices, application firewalls, etc. (see: mod_magnet).

Show your IP address using a Lua/Lighttpd script

Mon, 19 Apr 2010 00:00:00 EST

Sometimes it's useful to be able to determine your external WAN IP address from a script. There are web sites that will tell you your IP, but they're formatted for human readership and might go away or change their HTML format at any time. You could access the HTML administration interface of your router, but that's subject to changes in router brand or login information.

Another way is to set up a script on your own web server that reports the IP of the client accessing it.

There are tons of ways to do this... a CGI script, a web application in your language of choice, a PHP file even if it's not your main development environment. But here's one you might not have thought of. If you're using Lighttpd, you can use its Lua scripting support to do this extremely simply, without much setup and without involving any of your (possibly more complex) web development environments.

I don't even have CGI or PHP enabled on my web server, and my primary web-application development style based on long-running processes and frameworks is a bit "heavy" for a single-purpose script like this. And I didn't want to add it as a view to an existing app, as there would be no meaningful relationship between the two.

So, Lua it was. Here's how the script looks:

remote_ip.lua

lighty.content = { lighty.env["request.remote-ip"] }
return 200

And here's the Lighttpd configuration required (adapt to your domain/URL structure):

lighttpd.conf (excerpt)

server.modules = (
    "mod_lua"
)

# ...

$HTTP["host"] =~ "^ip.example.com$" {
    magnet.attract-physical-path-to = ( "/path/to/remote_ip.lua" )
}

Simple, eh?

Pylons multiple-domain (virtual hosting) configuration

Sat, 17 Oct 2009 00:00:00 EST

The conventional approach to Python web-application frameworks such as Pylons is to run multiple Python processes when multiple apps need to be deployed on the same physical server. This would also often apply to serving up more than one domain.

This works fine, especially if you use a process manager such as Supervisor to manage the web-app processes, but in some cases, you may want to pack more than one domain into the same process space, effectively moving your virtual hosts into the Python app configuration. This article will show you how.

Reasons to run Pylons apps in the same process

Save lots of memory: In memory-constrained environments such as virtual private servers (VPS), using one process for multiple apps may be a requirement.
Share: In some cases, you may want multiple apps to be able to share database connections or other resources.

Reasons not to

Global state may conflict.
Specifically, SQLAlchemy models (which have some global state) may conflict.
A syntax or import error in one app, or another error that triggers on initialization rather than on a page view, will take down the other apps.
The apps will use the same server thread pool, so fewer threads will be available per site on average.

Using Paste#urlmap for multiple domains

You've probably seen Paste#urlmap used to route different paths on the same domain to different applications. It's the main example of composite applications in the Paste Deploy documentation, where it looks something like this:

[composite:main]
use = egg:Paste#urlmap
/ = mainapp
/files = staticapp

But an apparently as-yet undocumented syntax also allows you to run apps on different domains:

[composite:main]
use = egg:Paste#urlmap
domain example.com / = mainapp
domain files.example.com / = staticapp

The syntax is domain <hostname> <path> = <appname>; the inclusion of <path> means that you can also set up apps on several paths under the same domain.

Setting `cache_dir`

Whenever you have multiple Pylons applications running from config files that are in the same directory, whether the same file or multiple files, you must be careful to set the cache_dir for each application separately.

The default created by paster looks like this:

cache_dir = %(here)s/data

%(here)s will be filled in with the directory where the configuration file is located. If two applications are configured in the same file, or in files in the same directory, and cache_dir is left as-is, their cache directories will overlap. They will overwrite or, worse, share each other's cached data. For example, Mako stores compiled templates in the cache directory, so one application will end up using the other's templates.

To avoid this, always set cache_dir to different values for applications whose configuration is in the same directory, using either relative or absolute paths:

#relative
cache_dir = %(here)s/appname-data

#absolute
cache_dir = /tmp/pylons-cache/appname

Serving an app on more than one path

When you set up the same application on multiple paths or domains using urlmap, it will be initialized multiple times. For example:

[composite:main]
use = egg:Paste#urlmap
domain foo.example.com / = app1
domain example.com /foo = app1

This will cause app1 to be loaded and initialized twice, which will cause problems if, for example, you create any models inside the init_model() function. (You'll get an exception because they're being created twice in the same Python process).

It's probably best, in this case, to place the app behind a web server such as Apache or Lighttpd and configure the web server to direct multiple hosts or URLs to the same application.

Global data

As I mentioned earlier, running multiple Pylons apps in the same process will cause potential conflicts in module-global data.

The init_model() conflict mentioned in the previous section is one; SQLAlchemy models are stored globally, and trying to declare the same models more than once can cause errors. In order to facilitate this, you will need a separate piece of code for model creation that checks whether the shared models are already in existence before creating them.

Some builtin Python modules also use global state, including locale and timezone.

Alternative to this approach

If your use-case involves a number of very similar sites or apps, with small configuration differences, you could use the approach described above. Or, you could use just one app and handle multiple domains inside the app itself.

For example, if you have a number of small content sites with identical structure, you could use just one app, with different template directories for each site, and a database column that indicates which site each content item belongs to.

Pylons apps on Google App Engine

Sun, 11 Oct 2009 00:00:00 EST

This will show you how to get a Pylons web app running on Google App Engine (GAE) in about 15 minutes, assuming you already have your GAE account and a familiarity with Pylons and virtualenv.

(Otherwise, start out with the Google App Engine docs and the Pylons docs).

The approach described here is focused on using as "normal" a process as possible for creating a Pylons app, with minimal modifications for GAE. It uses a stock virtualenv, and a stock paster create template, instead of an entirely different setup just for GAE. Several manual edits are required (but fewer now!)

This means that you could use an existing app or an existing virtualenv, with a few modifications. This is up to you, as I don't know what your exact setup looks like, but most of the steps below should apply.

Getting started

To begin, run one of the following scripts in an appropriate directory, replacing $MYPROJ and $MYAPP with whatever values you like (they can be the same, though if they are the directory structure can end up slightly confusing. paster create will make two directories named $MYAPP as it is.)

I've provided instructions for both pip and easy_install, so you can continue to use your preferred package manager.

You could also browse app.yaml and app.py to get an idea of what's going on.

Getting started: `pip` version

console

virtualenv --python=python2.5 --no-site-packages --unzip-setuptools $MYPROJ
cd $MYPROJ
pip install -U -E . pylons
touch ./lib/python2.5/site-packages/paste/__init__.py
rm -f ./lib/python2.5/site-packages/simplejson*/_speedups*
./bin/paster create -t pylons $MYAPP template_engine=mako sqlalchemy=False
wget http://files.countergram.com/pylons-gae/app.yaml
wget http://files.countergram.com/pylons-gae/app.py

Getting started: `easy_install` version

console

virtualenv --python=python2.5 --no-site-packages --unzip-setuptools $MYPROJ
cd $MYPROJ
./bin/easy_install -UZa pylons
rm -f ./lib/python2.5/site-packages/simplejson*/_speedups*
./bin/paster create -t pylons $MYAPP template_engine=mako sqlalchemy=False
wget http://files.countergram.com/pylons-gae/app.yaml
wget http://files.countergram.com/pylons-gae/app.py

If you use easy_install, you must then look in the lib/python2.*/site-packages directory and see if multiple versions of any packages (especially WebOb) were installed due to differing dependencies. Delete all but the most recent; otherwise they will conflict. The -U argument to easy_install should prevent this but does not necessarily do so.

Python versions other than 2.5

GAE runs Python 2.5. As long as your code runs on 2.5, you can use whatever (2.x) version of Python you want locally. You'll need to change all instances of python2.5 in the script above, and also edit the value of libdir in app.py.

Other required edits

You'll need to manually edit a few files to get things working.

Edit app.yaml and replace $REGNAME with the name of your application as registered with Google. For example, if your hosted URL is myapp.appspot.com, the registered name is myapp.
Edit app.py and replace $APPNAME with the name of your application that you passed to paster create (i.e. the directory name).
In your Pylons app's environment.py, comment out or delete the line beginning module_directory= in the instantiation of TemplateLookup. GAE does not have a writable filesystem, so Mako's file-based template caching will not work on it. If you are using a different template system, you'll have to figure out what is required to stop it from trying to write any files.

Running locally

You can run your app locally in a GAE environment using the Google App Engine SDK/Launcher. It's fairly simple, and documentation is available and beyond the scope of this article, but here are a few tips:

In the GUI version of the SDK/Launcher, be sure to use File->Add Existing Application rather than File->New Application. The path to use is the directory that contains app.yaml and app.py (i.e. $MYPROJ).
Using the command-line version, dev_appserver.py, don't run it from within the virtualenv; it won't be able to find the Google libraries.
The SDK/Launcher has a primitive (compared to paster serve) reloading capability. In order to force an "automatic" reload while your app is running, you'll need to edit (or touch) app.py.

Deploying

There are two ways to deploy your application: the Google App Engine SDK/Launcher application, or the appcfg.py command-line script.

The SDK/Launcher application gives you a nice push-button GUI, while appcfg.py is more flexible with various options such as verbose mode (-v), which prints a list of which files are being uploaded, and which are being skipped.

File count

When I deployed a barebones Pylons app in this way, with the included skip_list, the file count came out to 821 using pip and 814 using easy_install, both, well under the 3,000-file limit. Your "base" file count will probably vary slightly as new library versions come out.

If you are using easy_install (but apparently not if you are using pip), you can reduce the file count by excluding EGG-INFO directories from being uploaded. This will probably not break anything (but see below), unless part of your code depends on entry points of modules other than your app. You'll need to add a line to the middle of the skip_list regex in your app.yaml, as follows:

(.*/site-packages/.*\.egg/EGG-INFO/.*)|

For me, this reduced the "base" file count to 691, a savings of about 120 files. Unfortunately, excluding .egg-info directories when using pip breaks the setup.

If you do this, you must also add template_engine=None as a keyword argument to the config.init_app function call in your Pylons app's environment.py. Otherwise, some legacy code in Pylons (supporting the outdated Buffet template system) will break because it can't find some egg info.

Notes on workarounds, for the curious

GAE does not appear to use .pth files, which are files that are placed on the Python path and contain additions to the path. This is why sys.path must be set up in app.py. Also, easy_install uses its easy_install.pth file to make sure only the latest installed version of each package is on the path. This is why you have to manually delete duplicate packages using this GAE method.
The provided app.py has a rather odd line: os.mkdir = lambda *x:None. The latest version of setuptools (as of 0.6c11) attempts to import mkdir from os, and mkdir does not exist on GAE. This monkey-patch allows setuptools to be imported.
The provided app.yaml has a skip_list that prevents files that are part of the virtualenv but useless on GAE, such as binary shared objects, from being uploaded.
The "speedups" part of simplejson (_speedups.py and _speedups.so) uses a C extension, which is not supported by GAE. But the package is set up so that it will still work if these files are missing.
Paste uses a wonky system to allow separate packages (Paste, Paste Deploy, Paste Script) to inhabit the same paste namespace. When installed using pip, this system is dependent on several .pth files, which don't work on GAE. But sticking an __init__.py in makes it work normally.

Conclusion

At this point, you should be up and running and see the Pylons welcome page when you visit your local server run by the SDK. There are still more steps to a real GAE app, of course, such as familiarizing yourself with Google's datastore models.

I can't provide "support" on any of this, per se, but if the steps here don't work for you, please e-mail me and I'll try to correct or amend the article.

Managing multiple Pylons apps with Supervisor

Fri, 28 Aug 2009 00:00:00 EST

Wouldn't it be nice if your long-running processes, such as web applications and even HTTP servers, could be managed from one place?

There are a number of tools, including Supervisor, daemontools, and runit, that will launch processes, allow them to be configured in various ways, and restart them if they stop unexpectedly. This means you only have to make sure one process is started upon system startup, and it will take care of the rest. This article concentrates on how to manage Pylons web applications using Supervisor, which is fairly easy to configure.

This is something of a whistle-stop tour, aimed at getting you started with a working configuration. Supervisor is quite flexible, and it's a good idea to read the manual.

Assuming you have easy_install (and if you're using Pylons you almost certainly do) but not supervisor yet, here's how to get started. Note that Supervisor does not work on Windows, so this is a Linux/BSD/OS X/etc. tutorial.

Console

easy_install supervisor
mkdir /apps
cd /apps
paster create -t pylons app1
paster create -t pylons app2

I don't necessarily expect you to create /apps in the root of your filesystem. Create it where you like, but substitute your directory when I talk about /apps, which I'm using for simplicity in this article, later on.

After executing these commands, you'll have two Pylons applications. They won't do much, but they'll serve as a demonstration. Use real apps instead if you like.

Configuring Supervisor

The Supervisor configuration file should be located at /etc/supervisord.conf. If you would like to place it elsewhere for deployment purposes, I strongly recommend creating a symlink at /etc/supervisord.conf as that seems to be where Supervisor will look for configuration when restarted, even if you gave it a different initial configuration file.

I'll show you the necessary configuration for Supervisor in several parts. Here's the first configuration block:

/etc/supervisord.conf (partial)

[supervisord]
logfile = /tmp/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = /tmp/supervisord.pid
nodaemon = false
minfds = 1024
minprocs = 200
umask = 022
identifier = supervisor
directory = /tmp
nocleanup = true
childlogdir = /tmp
strip_ansi = false

Supervisor uses an INI-style configuration file, which is divided into sections. Each section has a name enclosed in square brackets. This section sets up some basic parameters that you can adjust if you want.

Now, let's talk about permissions. If run as root with this configuration, Supervisor will retain root permissions. You can then configure a user for each program that Supervisor runs, and it will drop root permissions for those specific programs. This is useful if some of the programs you manage need to run as root. For example, if you manage lighttpd as well as your Pylons apps, you may need lighttpd to run as root while your apps run as a non-root user.

If you want Supervisord to drop root permissions across the board, add a user = <username> line to the [supervisord] section of the configuration file.

If you allow Supervisord to remain root, make sure the configuration file is writable only by root!

Next, let's set up supervisorctl, which is a command-line utility that lets you see the status and output of processes and start, stop, and restart them.

/etc/supervisord.conf (cont.)

[unix_http_server]
file = /tmp/supervisord.sock

[supervisorctl]
serverurl = unix:///tmp/supervisord.sock

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

With this configuration, supervisorctl will only be available from the command line on the local machine, because it's using a file-based socket. Supervisor also supports HTTP and has a web interface, but I'm leaving it disabled for security reasons.

After saving supervisord.conf, you should be able to run supervisord, followed by supervisorctl, and end up in an interactive shell. There are no processes running, so it's not very interesting, but you can type help for a list of options, and exit to leave.

Finally, let's add some processes. Open that config file back up.

/etc/supervisord.conf (cont.)

[program:app1]
user = <yourusername>
command = paster serve --reload /apps/app1/development.ini
environment = PYTHONPATH=/apps/app1/,PYTHON_EGG_DIR=/tmp/python-eggs/

[program:app2]
user = <yourusername>
command = paster serve --reload /apps/app2/development.ini
environment = PYTHONPATH=/apps/app2/,PYTHON_EGG_DIR=/tmp/python-eggs/

Although in this example the names of the program blocks are the same as the names of the applications, they don't have to be. For example, if your application was named reallylongappname, your Supervisor configuration could read [program:rla], or whatever you like. The name in the program heading is what will appear in supervisorctl.

The environment can be set individually for each process. Here, the Python path is set because we're in development and have not installed the applications as packages.

After saving supervisord.conf, either type supervisorctl update, or type supervisorctl to enter the interactive shell and type update. Either style works, so it's a matter of whether you want to leave a shell open for convenience.

Now, type supervisorctl status, or in the interactive shell type status. You should see two items, app1 and app2, both marked RUNNING. If they instead show an error or exit condition, something went wrong.

Say app1 failed to start. The best thing to do is supervisorctl tail app1 stderr, which will show you the last log messages that app1 printed. You can also add a -f switch after tail, just as with the real tail utility, to see live updates of an application's output.

Where to go from here

It's up to you which programs you would like to manage using Supervisor, and which are better off being managed by your operating system's init system of choice, such as init.d, rc.d, or launchd.

It can be useful, however, to run your HTTP server, such as lighttpd, under Supervisor. This makes it easy to restart the server without su or sudo, since the Supervisor process is running as root (if configured as above — and remember to make the configuration file writable only by root to prevent abuse).

/etc/supervisord.conf (cont.)

[program:http]
command = /usr/sbin/lighttpd -D -f /etc/lighttpd.conf

(Remember that -D argument, which prevents lighttpd from daemonizing itself.)

If necessary, you can even run multiple HTTP servers on different ports, which gives you more flexibility than a system-wide startup script.

Supervisor is well documented, so the logical next step is reading the manual for more ideas of what it can do for you.

Pylons from scratch 1: a barebones app

Sun, 23 Aug 2009 00:00:00 EST

Typically when you start a new application using the Python web framework Pylons, you might use paster create -t pylons <appname> to do a little code generation for you and create a conventional directory structure.

But in this series, I'll show you how to build a working Pylons app by hand, from scratch.

Why do this? Two reasons:

Going behind the scenes, and building from scratch what code-generation tools would do for you, helps you learn how Pylons works.
You will also find out how to wrap an existing, non-Pylons WSGI application so that it can be deployed using paster and the same INI configuration files as Pylons applications, potentially easing legacy integration with Pylons apps.

If you do not already have Pylons installed, please easy_install Pylons==0.9.7. Or, use the virtualenv instructions in the Pylons installation documentation.

Starting out

First of all, we'll be using setuptools and creating a package structure. Pylons, and particularly the Paste Deploy component, is really set up to work with apps that are packaged as eggs, which requires setuptools and a setup.py. So start by creating a few directories and (empty) files, in a structure looking like this:

Directory structure

barebones
|-- barebones
|   |-- __init__.py
|   `-- app.py
|-- development.ini
`-- setup.py

Next, let's make an application. This is going to look almost exactly like a "hello, world" WSGI application, but with an extra function. For reference, a basic WSGI application would look like this:

def app(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    return ['Hello, world']

If you're unfamiliar with the functions or concepts used here, read the WSGI wiki to understand the spec, environ, start_response, and returning iterators.

Now, the approach we need to use in order to work with Paste's configuration and deployment system is a "factory," a function that creates a WSGI application. So our app.py will actually start out looking like this:

app.py

def make_app(global_conf, **app_conf):
    def app(environ, start_response):
        start_response('200 OK', [('Content-Type', 'text/plain')])
        return ['Hello, ', app_conf.get('foo', 'world')]
    return app

This is merely a function that takes configuration parameters and returns the WSGI app, which is the same as the one above it except for the use of a setting from app_conf.

In a standard Pylons app structure created by paster create, this make_app function would be in config/middleware.py instead. That's an appropriate name, since a lot of middleware is applied in it, but it's also not immediately obvious that it contains the main application factory.

Speaking of configuration, this is the barebones configuration file we'll be using:

development.ini

[DEFAULT]
debug = true

[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 8000

[app:main]
use = egg:barebones

It's rather spartan, isn't it? This configuration file is missing a lot of the sections a real Pylons app has, mostly to do with logging, which we can add later. It also does not configure components that we're not actually using yet, like SQLAlchemy and Beaker. This minimal configuration is plenty to satisfy the requirements of paster serve and get our application running and serving on localhost:8000. Not yet, however. We still need to use setuptools to create an egg for our application with the proper entry point.

By the way, the global_conf argument to make_app above will be drawn from the DEFAULT section, while the keyword arguments present in app_conf will be drawn from the app:main section.

Explanation of setup and entry points

The setup.py file is what makes your package distributable as an egg, which is what allows paster serve to serve your app.

Below is the barebones setup.py file. Stock Pylons apps also include a script that automatically installs setuptools if it's not found, which I have left out, along with some i18n-related configuration.

The important part of this file to note for future use and changes is the entry_points argument, which (explain)

zip_safe=False is also important, as it causes your package not to be made into a compressed archive when installed. Instead, it will be installed as directories and files, which means your templates and static files will be accessible.

setup.py

from setuptools import setup, find_packages

setup(
    name='barebones',
    version='0.1',
    description='',
    author='',
    author_email='',
    url='',
    install_requires=[
        "Pylons>=0.9.7",
    ],
    setup_requires=["PasteScript>=1.6.3"],
    packages=find_packages(),
    include_package_data=True,
    zip_safe=False,
    paster_plugins=['PasteScript', 'Pylons'],
    entry_points="""
    [paste.app_factory]
    main = barebones.app:make_app
    """,
)

Starting the server

With all three of these files saved, run the following command in the project directory:

Console

python setup.py egg_info

This will create a directory named barebones.egg-info, which is required for paster serve to work. Remember, development.ini specifies the main application as using egg:barebones. It won't work if your app is not packaged as an egg. The paster create utility normally does this for you when you create a new Pylons app, though you sometimes need to re-run this command if you significantly change the structure of your app, including adding entry points or renaming the package.

Now, at last, you can start your application:

Console

paster serve --reload development.ini

If all goes well, you now have a HTTP server running on localhost:8000 that you can visit in your web browser to see the message Hello, world.

Configuration and next steps

Now, to make sure the configuration file works properly, add the following line to the end, in the [app:main] section:

development.ini (cont.)

foo = Pylons

If you have left paster running while you did this, you should see in its output a message to the effect that it reloaded the app when you saved the new development.ini file. This reloading feature is extremely useful for rapid development. Changes to Python source files in your app and the main INI configuration file will result in a reload, while changes to secondary INI files, templates, and static files will not. Syntax and import errors will cause the server to terminate.

Visit localhost:8000 again, and you should see the message Hello, Pylons.

Are we there yet?

Is this a Pylons app? Not yet. Pylons is a collection of libraries and tools, one of which is Paste, which we are using. But this app includes very little of what makes Pylons what it is, including the special way it handles controllers and pseudo-global, context-specific variables. Still, it's a start, and this provides you with a way to create any WSGI application you want in a way that can be configured using INI files and deployed with Paste.

What's more important is that this is a way for you to easily package existing WSGI applications so that they can be served up alongside your Pylons applications, using most of the same configuration and deployment. You'll just need to create a setup.py that resembles the one above, create an app factory that returns your WSGI application, and set up your INI file and egg_info.

Using this method, I successfully converted three existing web sites, which used a mostly homebrew WSGI-based system, to using Paste Deploy and running alongside my Pylons apps. I then started to convert them into actual Pylons apps at my leisure, knowing that at least I had eliminated extra deployment steps and could manage everything in a consistent way.

So that's the basic setup in a nutshell. In the rest of this series, we'll add the Pylons environment, controllers, Routes, Mako templates, logging, and SQLAlchemy.

Create Pylons filters: middleware the configurable way

Thu, 20 Aug 2009 00:00:00 EST

If you're familiar with the use of WSGI middleware in Python web applications, you know that it's a potentially powerful way to modify both requests and responses without changing any code in the WSGI application wrapped by the middleware. But without something like the "filters" of Paste Deploy, used by Pylons and discussed here, the middleware must itself be instantiated in code.

This article will show you how to package a WSGI middleware component so it can be added to your application using only configuration and no code, assuming you are using Pylons (or at least Paste Deploy).

Here's a simple piece of WSGI middleware that adds a value to the WSGI environment. It's about as simple as middleware gets.

modulename/__init__.py

class AddValue(object):
    def __init__(self, app, key, value):
        self.app = app
        self.key = key
        self.value = value

    def __call__(self, environ, start_response):
        environ = dict(environ)
        environ[self.key] = self.value
        return self.app(environ, start_response)

To use this middleware class as it is, we would have to instantiate it in code. For example, if app is an existing WSGI application, we could write app = AddValue(app, 'foo', 4). Then, for the wrapped application, environ['foo'] would equal 4.

Creating the filter

Converting this middleware into a filter will allow it to be applied using configuration only, with no code changes. Filters are added to Pylons applications in the INI configuration file, and are themselves configurable. The downside is that middleware applied in this way can only be applied to the "outside" of an application, not in between other middleware in its internal stack.

Creating a filter is not hard at all, but can look confusing at first because there are three layers:

The middleware component itself (The AddValue class, defined above, which will remain unchanged)
The filter function, which takes a WSGI application as an argument and returns a WSGI application that has had the middleware applied
The filter factory function, which takes configuration as its arguments and returns the filter function.

What you need to deploy a filter in practice is a filter factory function, in a package deployed as an egg, with extra lines in the package's ``setup.py`` to make it usable from a configuration file.

Here's a filter factory that applies our AddValue middleware class:

modulename/__init__.py (cont.)

def addvalue_filter_factory(global_conf, key, value):
    def filter(app):
        return AddValue(app, key, value)
    return filter

The key and value arguments here come from the INI configuration file. Since they are specified as arguments individually, they are required. To make them optional, we would rephrase the code like this:

modulename/__init__.py (optional arguments)

def addvalue_filter_factory(global_conf, **app_conf):
    def filter(app):
        return AddValue(app,
            app_conf.get('key', 'default'),
            app_conf.get('value', 'defaultvalue'))
    return filter

Deployment

The addvalue_filter_factory function is not meant to be used directly. Instead, it should be part of a package deployed as an egg using setuptools, and the following should be placed in the setup.py:

setup.py (partial)

setup(
    # ...
    entry_points="""
    [paste.filter_factory]
    addvalue = modulename:addvalue_filter_factory
    """
    # ...
    )

This sets up an entry point named addvalue that points to the function. Note that the punctuation between the module name and the function name is a colon, not a dot.

Assuming this package is built as an egg and available on the Python path, the filter can now be used by adding something like this to the INI file for the Pylons application:

development.ini (partial)

[app:main]
# ...
filter-with = myvalue

[filter:myvalue]
use = egg:modulename#addvalue
key = foo
value = 4

Conclusion

It seems like more work up front to create a filter rather than just a plain middleware component. For many types of middleware, however, the ability to easily drop in a component using configuration and no code is valuable. It can ease deployment, and makes the conditional application of middleware for different configurations easier.

An example of a useful filter might be an authorization filter. Although it's generally more powerful to build authorization into an app, authorization could be added on top of an app that doesn't have any using a filter that assigns certain required roles to any part of the app under certain paths. The configuration might look something like this:

[filter:auth]
use = egg:myauth#rolebased
/admin = admin
/edit = writer,editor

Implementation is left as an exercise for the reader!

Rename your Pylons app without going mad

Tue, 18 Aug 2009 00:00:00 EST

Eventually, it happens: You realize that the name you gave your Pylons app conflicts with another package, or just needs to be changed. Unfortunately, you've now got a bunch of references to the old name to change. Not only that, but they come in different contexts: import statements, expressions, configuration, strings, etc.

Here's how to rename that Pylons app.

First, and perhaps, most important, delete the dist, build, and any .egg-info directories in your project directory.

If you leave these directories laying around, they can hold old information that references the previous name of your project, which will cause problems whether you are running your app from the project directory or installing it. Running python setup.py clean will not necessarily fix this problem, so delete the directories.

Next, you need to start replacing references to your project in code and configuration files. But you can't simply do a global search-and-replace, because your project name might be used in multiple contexts; for example, maybe you named your project foo because it would be running on foo.com, and now you want it to be named foo_site to avoid a naming conflict. But a global search-and-replace could end up replacing foo.com with foo_site.com in your templates.

Instead, try doing global search-and-replace on from foo and import foo.

Of course, you'll also need to change any references to foo in your own code. But here's a list of places in a stock Pylons app, created by paster create -t pylons, where your app name appears, other than import statements:

Config file (e.g. development.ini, foo/config/deployment.ini_tmpl): use under [app:main], keys under [loggers], and the [logger_foo] section.
foo/config/environment.py: config.init_app call, config['pylons.h'] assignment.
MANIFEST.in
setup.py: name, package_data, and entry_points arguments.
Documentation.

Finally, change the project directory name and module directory name, which are usually identical. In other words, if this is your project directory layout, you'll want to change both foo directory names:

foo
|-- foo
|   |-- __init__.py
|   |-- config
|   |-- controllers
|   |-- ...
|   `-- websetup.py
|-- development.ini
`-- setup.py

Once you're pretty sure you've changed all references to foo, run python setup.py egg_info to regenerate the .egg-info directory you deleted earlier.

Also check your system's site-packages directory or other directories on your Python path, to make sure there is not still an installed version of the package with the previous name.

Now you're ready to test. If all goes well, your app should run under its new name. If you see any ImportError exceptions or other unexpected errors, you missed one.

Finally, run python setup.py install (unless you have a reason not to want your app in site-packages) to make sure the packaging works.

Done.

GMP bignums vs. Python bignums: performance and code examples

Sat, 18 Apr 2009 00:00:00 EST

Interpreted languages such as Python are traditionally used in IO-bound applications where their lack of high CPU-bound performance does not matter. On the other hand, extensions written in C (or C++) can be used to implement computationally heavy operations that can be called from Python as if they were native functions.

Of course, it's not quite that simple. As soon as data types and structures more complicated than simple integers, floats, and strings become involved, you have to consider both the implementation of the types in C and data interchange between the C layer and the high-level language, whether through serialization or through a data-access API. Furthermore, you might be surprised at what actually does, and does not, result in a gain in performance.

In this article, I've implemented n-choose-k, or choose(n, k) for languages that don't support arbitrary infix operators, in Python and C. The choose function is useful in combinatorics and evaluates to the number of unordered subsets of size k that can be chosen from a set of size n.

Besides being practical, choose requires dividing factorials, which means very large numbers and lots of arithmetic operations. N-choose-k can be implemented in terms of factorials like this:

choose(n, k) == factorial(n) / (factorial(k) * factorial(n - k))

Because k ≤ n in all normal cases, a small optimization is possible by simplifying factorial(n) / factorial(k) into a stopped factorial function. In other words, evaluate [n × (n - 1) × ... × k]. This saves 2k multiplications versus computing the full factorials of both n and k.

Pure Python Version

Versions of the factorial and choose functions implemented in pure Python will serve as references; theoretically, the C version should improve on the performance of the Python version, or there's no point. Because Python has a 1,000-call recursion limit and does not optimize tail recursion, an iterative algorithm is best. The choice of 50,000 and 50 as parameters is arbitrary and will take a long enough time to execute that differences in performance between versions will be obvious.

testpure.py

def factorial(n, stop=0):
    o = 1
    while n > stop:
        o *= n
        n -= 1
    return o

def choose(n, k):
    return factorial(n, stop=k) / factorial(n - k)

if __name__ == '__main__':
    print choose(50000, 50)

The First C Version

A naive version in C would duplicate the Python code above, using C unsigned long long arguments instead of Python objects. This would lead to an upper bound of 1.8e19, a value easily exceeded by the factorial function at n=21.

In order to get around this limitation, we need a bignum or bigint library, which can represent arbitrary-precision integers. As it happens, Python intrinsically supports bignums in pure Python code, and also exposes a C API to its bignum objects. Perhaps we could use Python's bignum library while writing the algorithm in C for speed.

Additionally, a Pyrex wrapper, shown below the C code, will make it easy to generate a module usable from Python.

The code below implements C functions for stopped factorial and choose, which use Python objects throughout. We'll call this choose1.c, which distinguishes it from the choose.c that Pyrex will generate (and foreshadows that there will be a choose2.c.)

choose1.c

#include <Python.h>

PyObject* factorial1(PyObject* n, PyObject* stop) {
    PyObject* dec = PyLong_FromLong(1);
    PyObject* counter = PyLong_FromLong(1);
    PyObject* n1 = PyNumber_Long(n);
    PyObject* placeholder;
    while(1 == PyObject_Compare(n1, stop)) {
        placeholder = PyNumber_Multiply(counter, n1);
        Py_DECREF(counter);
        counter = placeholder;

        placeholder = PyNumber_Subtract(n1, dec);
        Py_DECREF(n1);
        n1 = placeholder;
    }
    Py_DECREF(dec);
    Py_DECREF(n);
    return counter;
}

PyObject* choose1(PyObject* n, PyObject* k) {
    PyObject* zero = PyLong_FromLong(0);
    PyObject* nk = PyNumber_Subtract(n, k);
    PyObject* fact1 = factorial(n, k);
    PyObject* fact2 = factorial(nk, zero);
    PyObject* result = PyNumber_FloorDivide(fact1, fact2);
    Py_DECREF(nk);
    Py_DECREF(zero);
    Py_DECREF(fact1);
    Py_DECREF(fact2);
    return result;
}

You might wonder why this code uses a placeholder value and does not use the available in-place arithmetic functions. This is because Python integers are immutable objects. Using PyNumber_InPlaceMultiply and PyNumber_InPlaceSubtract actually produces the same results as the above.

Using immutable types has performance consequences. In calculating 50,000 choose 50, for instance, this code will instantiate and destroy about 199,850 objects inside the main loop of the factorial function (given that instances of the integers from -1 to 100 are kept in a static table and not instantiated dynamically). More on performance later.

The following Pyrex source code and distutils configuration wrap the C code as a Python function called choose().

choose.pyx

cdef extern from "choose1.c":
    object _choose1 "choose1" (object n, object k)

def choose(n, k):
    return _choose1(n, k)

setup.py

from distutils.core import setup
from distutils.extension import Extension
from Pyrex.Distutils import build_ext
setup(
  name = "choose",
  ext_modules=[
    Extension("choose", ["choose.pyx"])
    ],
  cmdclass = {'build_ext': build_ext}
)

Finally, the following code uses the library and will be the point of comparison to the pure Python version.

testc.py

if __name__ == '__main__':
    from choose import choose
    print choose(50000, 50)

Performance Comparison

There are numerous ways of comparing execution times. In this example, I'm using the time utility rather than something in Python such as the timeit module.

time python testpure.py

284958500315333290867708487072990268397101930544468658
476216100935982755508148971449700622210078705183923286
686402942943816349032142836981589618876226813174803825
580124000

real    0m14.921s
user    0m9.211s
sys     0m4.151s

time python testc.py

284958500315333290867708487072990268397101930544468658
476216100935982755508148971449700622210078705183923286
686402942943816349032142836981589618876226813174803825
580124000

real    0m14.460s
user    0m9.146s
sys     0m4.000s

What a surprise! Accounting for normal variations in performance based on background CPU load, the two versions take effectively the same amount of time. It looks as though the overhead in the pure Python version is not, as you might assume, in the fact that Python is interpreted rather than compiled. By using Python objects and memory management, the C version runs just as slowly.

Replacing Python bignums with GMP

The GNU Multiple Precision Arithmetic Library (GMP) is a bignum library that supports integers, fractions, and floating-point numbers, and is usable from C or C++.

GMP allows true in-place arithmetic operations, which might make it faster than the Python version, which allocates and deallocates objects on every iteration of the main loop. It's also very easy to mix GMP's bignums and unsigned longs, as well as producing string representations of GMP bignums in different bases.

Be sure to specify your host platform when building GMP. The first time I built it, I relied on configure to auto-detect everything, and it built a 64-bit library for my definitively 32-bit computer. Read the section in the manual on GMP Build Options to figure out the right host argument to configure; for instance, mine on a 32-bit OS X system was --host=i386-apple-darwin.

The following code listings implement the factorial and choose functions using GMP and update the Pyrex wrapper. While I won't try to duplicate the GMP Manual here, it's worth noting that its functions tend to operate on arguments rather than returning values, and variables need to be initialized and cleaned up.

The prefixes used are mi for GMP multi-precision integer (distinct from the mpz prefix used by the library), pi for Python integer, and ps for Python string.

choose2.c

#include <Python.h>
#include <gmp.h>

void mi_factorial(mpz_t mi_result, unsigned long n, unsigned long stop) {
    unsigned long int n1 = n;
    mpz_set_ui(mi_result, 1);
    while(n1 > stop) {
        mpz_mul_ui(mi_result, mi_result, n1--);
    }
}

PyObject* choose2(unsigned long n, unsigned long k) {
    mpz_t mi_fact1;
    mpz_t mi_fact2;
    mpz_t mi_result;
    char* buffer;
    PyObject* ps_result;
    PyObject* pi_result;

    mpz_init(mi_fact1);
    mpz_init(mi_fact2);
    mpz_init(mi_result);

    mi_factorial(mi_fact1, n, k);
    mi_factorial(mi_fact2, n - k, 0);
    mpz_cdiv_q(mi_result, mi_fact1, mi_fact2);

    buffer = malloc(mpz_sizeinbase(mi_result, 10) + 2);
    mpz_get_str(buffer, 10, mi_result);
    ps_result = PyString_FromString(buffer);
    pi_result = PyNumber_Long(ps_result);

    mpz_clear(mi_fact1);
    mpz_clear(mi_fact2);
    mpz_clear(mi_result);
    free(buffer);
    Py_DECREF(ps_result);

    return pi_result;
}

choose.pyx (revised)

cdef extern from "choose1.c":
    object _choose1 "choose1" (object n, object k)

cdef extern from "choose2.c":
    object _choose2 "choose2" (unsigned long n, unsigned long k)

def choose(n, k):
    return _choose1(n, k)

def gmpchoose(n, k):
    return _choose2(n, k)

setup.py (revised)

from distutils.core import setup
from distutils.extension import Extension
from Pyrex.Distutils import build_ext
setup(
  name = "choose",
  ext_modules=[
    Extension("choose", ["choose.pyx"], libraries=['gmp'])
    ],
  cmdclass = {'build_ext': build_ext}
)

testgmp.py

if __name__ == '__main__':
    from choose import gmpchoose
    print gmpchoose(50000, 50)

And, finally, the result we've been waiting for:

time python testgmp.py

284958500315333290867708487072990268397101930544468658
476216100935982755508148971449700622210078705183923286
686402942943816349032142836981589618876226813174803825
580124000

real    0m2.042s
user    0m1.993s
sys     0m0.033s

Using the GMP library has cut execution time on this hardware down to around 2 seconds, from around 13-15 seconds.

Conclusion

When implementing C extensions to speed up Python programs, there are more considerations than just "switch to compiled language, see speed increase." Python provides many data types and structures that must be duplicated in the C program, and using the Python API heavily may lead to a slight or nonexistent gain in performance. Meanwhile, using other libraries may require interconversion (e.g. here, the GMP multi-precision integer is converted to a Python integer by way of a decimal string representation).

For purely numeric functions that deal only in values that fit within standard C types, it's relatively easy to create large speedups. Once other types, such as strings and bignums, become involved, the work starts to build up. However, the reward can still be substantial: here, the final C extension using GMP resulted in an 86% reduction in execution time. The results above do not mean that GMP is more performant than Python bignums in general, but the difference in performance in this case is obvious.

YouTube video embedding in reStructuredText

Thu, 19 Mar 2009 00:00:00 EST

The reStructuredText markup language is used in a variety of ways, including as a documentation format for Python applications and a general markup language for web content, in the same vein as Textile or Markdown. It's also extensible in Python: by adding new functions as "directives" to the doctools module, you can implement new commands. In this article, I'll show you how to implement a youtube directive so that a line like the following in your reStructured Text file will become an embedded YouTube video when it's rendered to HTML:

.. youtube:: <video-id>

The outline of what we're going to do is simple: create a youtube function that returns a document node list, then register this function with docutils' directives list. (By the way, I'm using the XHTML-valid approach that uses only an object element, not embed.) The specifics are a little more involved. The following code listing shows how I did it:

youtube.py

from docutils import nodes
from docutils.parsers.rst import directives

CODE = """\
<object type="application/x-shockwave-flash"
        width="%(width)s"
        height="%(height)s"
        class="youtube-embed"
        data="http://www.youtube.com/v/%(yid)s">
    <param name="movie" value="http://www.youtube.com/v/%(yid)s"></param>
    <param name="wmode" value="transparent"></param>%(extra)s
</object>
"""

PARAM = """\n    <param name="%s" value="%s"></param>"""

def youtube(name, args, options, content, lineno,
            contentOffset, blockText, state, stateMachine):
    """ Restructured text extension for inserting youtube embedded videos """
    if len(content) == 0:
        return
    string_vars = {
        'yid': content[0],
        'width': 425,
        'height': 344,
        'extra': ''
        }
    extra_args = content[1:] # Because content[0] is ID
    extra_args = [ea.strip().split("=") for ea in extra_args] # key=value
    extra_args = [ea for ea in extra_args if len(ea) == 2] # drop bad lines
    extra_args = dict(extra_args)
    if 'width' in extra_args:
        string_vars['width'] = extra_args.pop('width')
    if 'height' in extra_args:
        string_vars['height'] = extra_args.pop('height')
    if extra_args:
        params = [PARAM % (key, extra_args[key]) for key in extra_args]
        string_vars['extra'] = "".join(params)
    return [nodes.raw('', CODE % (string_vars), format='html')]
youtube.content = True
directives.register_directive('youtube', youtube)

Much of this code is unnecessary if you only want to implement the single-line version of the command I used above. But this code also allows you to customize the width and height and add other param elements:

.. youtube:: asdfjkl
    width=600
    height=400
    someOtherParam=value

Quick code walk-through

The argument signature of the youtube function is the same as you would use for any directive of this type (i.e. a block directive, as opposed to something inserted into running text.)
The only argument actually used here is content, which would contain everything from asdfjkl through 400 in the above example.
content is a list of lines.
The code involving extra_args simply splits each line after the first on the equals sign and makes a dictionary out of the (key, value) pairs, treating width and height specially.
The return value is a list of nodes. The nodes.raw function used to create the (single) node here takes three parameters: the raw source for the element, which is not really used in this case and can be left blank; the string to output; and the format, which in this case is HTML.

How to render

If you're using reStructured Text seriously on your web site, your content management system probably already has a renderer for it. But just in case, here's how to render it to HTML manually:

source = """\
This is some text.

.. youtube:: 2A2XBoxtcUA

This is some more text.
"""

from docutils.core import publish_parts

doc_parts = publish_parts(source,
    settings_overrides={'output_encoding': 'utf8',
                        'initial_header_level': 2},
    writer_name="html")

print doc_parts['html_body']

The initial_header_level override is useful if you want your section headings to start as h2 elements, rather than h1 elements. On this site, I store the title attribute of each article separately from the body, and render it as a h1 element separately. The reStructuredText documents begin with body text. If you place the title of your documents in reStructuredText, you will still need to use the initial_header_level setting, because the default doctools behavior is to use h1 elements for both the title and the top-level section headings.

And that's it. Now embedding videos is a one-line statement, and you can make other helpful extensions using the same technique.

Addendum: The Python source code in this article is under the terms of the MIT License.

WSGI decorators considered inconvenient

Tue, 03 Mar 2009 00:00:00 EST

In the grand tradition of "considered harmful" essays, here's one on why it's not a great idea to use Python function decorators to wrap WSGI applications (e.g. for authentication/authorization). But unlike using goto to implement control structures, the mixture of decorators and WSGI is just inconvenient. In this essay, I'll explore the problem, show how it applies more generally to function decorators that work with arguments, and suggest a hackish solution if you don't want to avoid WSGI decorators altogether.

WSGI applications are functions (well, callables) that expect two arguments (environ, start_response): a dictionary of CGI-style and extension variables and a callback function. Pop quiz: in what position in a function's argument list is the environ dict?

The correct answer is "maybe 0 and maybe 1," and this is the start of the problems with WSGI decorators. Specifically, when you're calling a WSGI function from normal code, environ is in position 0; and when you're implementing a WSGI callable as an ordinary function, it's in position 0; but when you're implementing a WSGI callable as a method of a class, it's in position 1 because there's a self argument before it.

When you wrap a bound method in another function, as is commonly done with WSGI middleware, there's no problem. Bound methods carry their self arguments with them, in their im_self attribute. The argument signature looks the same from the outside.

In a decorator, though, you do have to worry about the self argument. When you apply a decorator to a function inside a class, the decorator receives a reference to a function that has not become a method yet. In fact, only the final return value of the last decorator on the function will be "blessed" into an unbound method of the class, after the class block closes. From the decorator you can't tell whether the function is destined to be a method or not.

What this means is that, if your decorator expects a particular argument signature such as (environ, start_response), whenever the decorator is applied to a method, the arguments will be shifted over one place. And you don't know that this has happened unless you (1) check the length of the argument list, assuming there are no optional arguments, or (2) check the types of the arguments.

Our method-safe WSGI decorator might look like this:

import functools
def wsgi_decorator(function):
    @functools.wraps(function)
    def wrapper(*args):
        if len(args) == 3:
            environ, start_response = args[1:]
        else:
            environ, start_response = args
        environ['mykey'] = "my value" # Do stuff
        return function(*args)
    return wrapper

By checking the length of *args, we sidestep the problem of the potential self argument. You can see that the above decorator works through the following example:

import webtest
import webob

@wsgi_decorator
def app(environ, start_response):
    return webob.Response(environ['mykey'])(environ, start_response)

class C(object):
    @wsgi_decorator
    def app(self, environ, start_response):
        return webob.Response(environ['mykey'])(environ, start_response)

print webtest.TestApp(app).get('/').body
print webtest.TestApp(C().app).get('/').body

This approach can be simplified with a simple utility function:

def wsgi_args(args):
    if len(args) == 3:
        return args[1:3]
    else:
        return args[0:2]

The original function, however, must still be called with *args, not environ, start_response, because it will expect a self argument if it's a method.

If you use decorators that ship with third-party WSGI modules, they might work with only functions, only modules, or both. Sometimes the only way to be sure is to check the source code.

And unfortunately, other cases are not as simple as this. The example above only works because WSGI applications are expected to have exactly two regular arguments. If the expected argument signature involves optional arguments, it gets trickier. It may be a better idea to avoid such decorators.

Add bound methods to instantiated objects in Python

Sun, 01 Mar 2009 00:00:00 EST

In Python, as in other dynamic languages, you can add new methods to a class after it's been defined. To do this in Python, you define a function that takes self as its first parameter, then assign that function to an attribute of the class:

class C(object):
    pass # No methods... yet

def foo(self):
    print self

C.foo = foo

When executing the final line, Python automagically wraps the function foo in an unbound method called foo, and assigns the unbound method to C.foo. (You need to be aware of this if you ever try to assign a function to an attribute of a class and want to use it as just a function: in that case, you would need to put it inside a data structure instead or use C.foo.im_func to get at the original function.)

This principle is not at work when you are dealing with an instance of a class. If you assign a function to an attribute of an instance, it will not be converted into anything. The creation of a bound method must be done explicitly, like this:

import types
instance = C()
instance.bar = types.MethodType(foo, instance, instance.__class__)

In this example, invoking MethodType creates a new bound method object. Its parameters are a function, the instance to which the method should be bound, and the class (the last parameter could simply be C, but using instance.__class__ allows you to use this approach when you don't have a direct reference to the class.)

Basic Formencode usage pattern

Fri, 20 Feb 2009 00:00:00 EST

Here's a useful Python snippet that shows the basics of using Formencode to validate form submissions inside a typical web-application controller. There's a reusable pattern here that isn't totally clear from the official documentation. This is non-framework code. First, here's the setup (the definition part):

from formencode import Schema, validators

# Declarative style for schemas
class MyForm(Schema):
    name = validators.String(not_empty=True)
    email = validators.Email(resolve_domain=False)
    month = validators.Int(not_empty=True)
    allow_extra_fields = True

# Adding to schemas after the fact
MyForm.fields['address'] = validators.String(not_empty=True)

Now, here's what goes in your request handler, adapted as necessary to the particularities of your web application framework or architecture (i.e. the arguments passed to the request handler and the way of getting at POST arguments will be different depending on how your application is set up):

from formencode import Invalid, htmlfill

def your_request_handler(request):
    errors = {}
    data = {}
    # Put data access common to GET/POST requests here
    if request.method == 'POST':
        data = dict(request.POST)
        try:
            data = MyForm().to_python(data)
            # Put use of the data here
            # Put a redirect to a GET-based confirmation page here
        except Invalid, e:
            errors = e.unpack_errors()
    else:
        pass # Put code for GET requests here
    html = your_template()
    print htmlfill.render(html, data, errors)

The control flow works like this. If there's no post data, then data and errors will both be empty, and your template will be rendered with an empty form. If there's post data but it's in error, the errors will be displayed and the posted data will be pre-filled into the form for the user. And if the data validates, you should return a redirect to a GET-based page after data processing is complete.

Install Python Imaging Library (PIL) on Intel Mac

Sun, 15 Feb 2009 00:00:00 EST

Recently I found the Python Imaging Library (PIL) would not install properly on my Intel Mac. The error message when trying to install PIL was:

ld: in /Developer/SDKs/MacOSX10.4u.sdk/usr/local/lib/libxml2.2.dylib,
file is not of required architecture for architecture ppc

It turned out that gcc was being called as gcc -arch ppc -arch i386, which is used to build a universal binary on OS X. Since the installed libxml2 was Intel-only, and setuptools did not detect this, the PIL installation failed.

Force gcc to build only an Intel binary when you install PIL, rather than a universal binary, by setting an environment variable:

sudo env ARCHFLAGS="-arch i386" python setup.py install

Since universal binaries are not needed on Intel Macs unless you are preparing them for distribution, this lets you install PIL normally.

Countergram

Group a list into chunks in Python

YouTube video embedding with XSLT

Pylons deployment with Lighttpd and Flup

Pylons configuration

Lighttpd configuration

Why FastCGI?

Load balancing

What about mod_wsgi?

Addendum

Show your IP address using a Lua/Lighttpd script

Pylons multiple-domain (virtual hosting) configuration

Reasons to run Pylons apps in the same process

Reasons not to

Using Paste#urlmap for multiple domains

Setting cache_dir

Serving an app on more than one path

Global data

Alternative to this approach

Pylons apps on Google App Engine

Getting started

Getting started: pip version

Getting started: easy_install version

Python versions other than 2.5

Other required edits

Running locally

Deploying

File count

Notes on workarounds, for the curious

Conclusion

Managing multiple Pylons apps with Supervisor

Configuring Supervisor

Where to go from here

Pylons from scratch 1: a barebones app

Starting out

Explanation of setup and entry points

Starting the server

Configuration and next steps

Are we there yet?

Create Pylons filters: middleware the configurable way

Creating the filter

Deployment

Conclusion

Rename your Pylons app without going mad

GMP bignums vs. Python bignums: performance and code examples

Pure Python Version

The First C Version

Performance Comparison

Replacing Python bignums with GMP

Conclusion

YouTube video embedding in reStructuredText

Quick code walk-through

How to render

WSGI decorators considered inconvenient

Add bound methods to instantiated objects in Python

Basic Formencode usage pattern

Install Python Imaging Library (PIL) on Intel Mac

Setting `cache_dir`

Getting started: `pip` version

Getting started: `easy_install` version