Deploying TurboGears with Lighttpd and SCGI · Monday, October 31, 2005

Last week, I decided it was finally time for me to learn a bit more about TurboGears, a great new Python web application "mega-framework" from Kevin Dangoor. I had watched the excellent screencast, peered through the documentation, and was impressed with what I saw. At work, I had been tasked with creating a web-based tool, and I had the basic requirements down in my head. With no prior experience with TurboGears, I was able to create a fully functional, featureful, and pretty web application to meet those needs in less than a week. Now, I had an application running under the built-in web server, which is a great tool for development, but I couldn't find any solid information out there on deploying my application as a WSGI application behind the great lighttpd web server. After a little work, I am happy to say that I have this working, and I would like to share the process with anyone else who was interested in getting this working.

Get thee lighttpd!

The first step of this exercise is to download and compile lighttpd, the world's most difficult to pronounce web server. Make sure that you have the pcre regular expression library installed first as, with most things in life, this is going to require a few regular expressions. Done with that? Good.

Flup it up

The next step is to download and install the flup WSGI toolkit from Allan Saddi. This is a great little toolkit that I have referenced before in my previous post on lighttpd and python. Flup provides a host of WSGI servers for FastCGI, SCGI, and more in both threaded and pre-forking versions. The setup process for flup is a standard "python setup.py install" one, although I really wish that Allan would make a Python egg of flup available so that the process was even simpler. Come to think of it, I think that Kevin should just ask Allan if he can include flup as a part of TurboGears so that people won't have to come here and read my blathering just to find a good deployment option for their TurboGears applications... but I digress.

SCGI

The SCGI protocol is, and I quote, "a replacement for the Common Gateway Interface (CGI) protocol. It is a standard for applications to interface with HTTP servers. It is similar to FastCGI but is designed to be easier to implement." Neat. Lets get it and install it. Thanks to easy_install, this is a very straightforward task. Simply execute:

easy_install scgi

You will need to do this as root, or sudo the command. This will automagically download, install, and configure the scgi module. Every Python package should do this!

A TurboGears WSGI Server

Now that you have all the various parts and pieces that you will be needing, you need to create a script that will run your TurboGears application as a WSGI server over the SCGI protocol. Thanks to flup, this is pretty easy. In your TurboGears project, you already have a script to start your application with the built-in web server. Copy it to a new file called "myproject-start-scgi.py" and replace its contents with the following:

#!/usr/bin/env python
import pkg_resources
import cherrypy
import sys

pkg_resources.require("TurboGears")

from cherrypy._cpwsgi       import wsgiApp
from flup.server.scgi_fork  import WSGIServer
from os.path                import *

if len(sys.argv) > 1:
    cherrypy.config.update(file=sys.argv[1])
elif exists(join(dirname(__file__), "setup.py")):
    cherrypy.config.update(file="dev.cfg")
else:
    cherrypy.config.update(file="prod.cfg")

from myproject.controllers import Root

cherrypy.root = Root()

cherrypy.server.start(initOnly=True, serverClass=None)

WSGIServer(application=wsgiApp).run()

Make sure that you change the line that says from myproject.controllers import Root to reflect your application's name. Executing this script will start your application as a WSGI application that listens on port 4000 for SCGI requests.

Lighttpd SCGI Configuration

Now, you just need to configure lighttpd to listen to your SCGI server. You will want to take into account the fact that you won't want your static files (css, images, javascript, etc.) to go through TurboGears, but just to be served directly through lighttpd. This is pretty straightforward, and this is what your lighttpd should look like, minus the MIME type stuff, which I will let you find elsewhere:

server.modules = ( "mod_access",
                   "mod_scgi",
                   "mod_accesslog",
                   "mod_rewrite",
                   "mod_staticfile" )

server.document-root = "/full/path/to/your/application"
server.errorlog      = "/tmp/lighttpd.error.log"
server.event-handler = "freebsd-kqueue"  # on Mac OS X or FreeBSD

accesslog.filename   = "/tmp/access.log"

$HTTP["url"] !~ "^/static/" {
  scgi.server = (
                  "/" =>
                    ( "127.0.0.1" =>
                      (
                        "host" => "127.0.0.1",
                        "port" => 4000,
                        "check-local" => "disable"
                      )
                    )
                )
}

With this lighttpd configuration, and the above script, you should be all set. Run your startup script, then run your lighttpd with this configuration file, and you should be good to go! To really feel the speed, make sure that you pass prod.cfg to the startup script so that your TurboGears application is running under "production mode" which turns off a host of debugging features.

Some Caveats

I elected to use the pre-forking WSGI server from flup, and if you are using in-memory sessions, this seems to go insane. I would recommend using the database session backend of TurboGears, or even the on-disk session backend. You could also try using the threaded WSGI server in flup, and I am sure this would also do the trick. I have had good luck with the above, and I hope that someone finds this useful!
UPDATE: This information is now all available in the TurboGears documentation. I am turning comments off for now because of comment spam on this post, but if you have questions, feel free to send me an email.