Metaclasses Demystified · Saturday, January 10, 2009

This article originally appeared in the July 2008 issue of Python Magazine and was adapted from Jonathan LaCour’s CLEVERtrain professional Python training services.

The word “metaclass” often strikes fear into the hearts of Python programmers. This article explores metaclasses in an approachable way, in order to take the mystery out of metaclasses, and enable you to leverage metaprogramming techniques to solve real problems.

There are few things in the Python world that are so misunderstood and feared as metaclasses. In fact, when I recently was training a group of seasoned programmers about metaclasses, I asked the room what sprung to mind when they heard the word “metaclass.” The students shouted out their feelings in chorus – “confusing,” “magical,” and worst of all, “difficult.”

Well, as I told my students that day, when it comes to metaclasses, I’d recommend that we called upon the wise words of Douglas Adams in his famed book “The Hitchhiker’s Guide to the Galaxy.”

“Don’t panic!”

Once you take a closer look at how object-oriented programming works in Python, you’ll quickly discover that not only are metaclasses easy, they’re also extremely useful. Here, we’ll do our best to take all of the mystery out of metaclasses, and show you that there’s nothing to be afraid of!

Revisiting Classes

Most Python programmers are familiar with creating classes, and might even create or modify their own classes several times a week in the course of their work. Many of us were even taught about classes and object-oriented programming in the course of our education. But, when was the last time any of us really thought about classes in depth since we were first introduced to the concept? I’d wager that most of us take classes for granted, and don’t really think about what they are providing for us, and more importantly, how they provide it to us.

Understanding metaclasses can be greatly simplified by taking another look at Python classes, from a slightly different perspective. In order to do this, lets pose a simple question. What exactly does a class do?

My first instinct when attempting to answer this question is to rely on my education, which unearths big, fancy, computer-sciency words like “encapsulation” and “abstraction.” Its tempting to allow this to cloud our thinking, so lets do our best to think at a much simpler level when considering this question. So, I ask again – what does a class do?

Fundamentally, a class is used to construct objects called instances. This is essentially all a class does – creates instances. The process of creating instances using a class is called instantiation. For example, given a class Person, we can instantiate it to create an “instance” of that Person class.

    class Person(object):
        pass

    jonathan = Person()

This instance is related back to its class in that the class is what constructs the instance. What do we mean by “construct”? Well, when a class is instantiated, it constructs an instance by providing it with its “namespace.” We all know from Tim Peters’ “The Zen of Python” that namespaces are a “honking great idea,” and classes are a great example of why! The attributes in the namespace of a class are used to define the namespace of its instances, thus providing those instances with behavior and state. Lets enhance our Person class, to see how this works.

    >>> class Person(object):
    ...    greeting = 'Hello'
    ...    
    ...    def greet(self, who):
    ...        print self.greeting, who

    >>> jonathan = Person()
    >>> jonathan.greet('Readers')
    'Hello Readers'

The Person class now has two attributes in its namespace – greeting and greet. When we instantiate our Person class, our instance is given both state, in the form of the greeting attribute, and behavior, in the form of the greet method. In this way, a class is of critical and primary importance in defining how its instances will behave.

So, what does a class do? Lets summarize. In a nutshell, a class is an object with constructs instances through a process called instantiation, therefore defining the namespace, behavior, and state of that instance.

Defining Metaclasses

Now that we’ve established clearly what a class is, we can confidently pose the question – what exactly is a metaclass? When sifting through documentation, the most common answer you’ll find to that question is short, but often difficult to unpack:

A metaclass is the class of a class.

While this answer is certainly correct and concise, and tells us what a metaclass is, it still doesn’t tell us what a metaclass does. Lets look at an alternate answer to the question:

A metaclass is a class whose instances are classes.

I will concede that this answer is just as much of a mouthful as its counterpart, but it has the benefit of giving us a clue as to what a metaclass does! In short, a metaclass constructs classes. We also know that a metaclass is just another class, and we’ve just spent some time outlining what a class does. As a result, we should be able to build upon what we’ve learned about classes earlier to determine a bit more about metaclasses.

Recall that a class helps define the behavior and state of its instances. Metaclasses provide the same basic capability to classes, giving you the ability to change the way a class will behave at runtime. This technique is commonly referred to as “metaprogramming.”

Having a definition of metaclasses and metaprogramming is useful, but lets take a deeper look by investigating an existing metaclass.

The type Metaclass

In Python, classes which inherit from object are called “new-style classes.” All such new-style classes have a default metaclass called type. We can prove that type is the metaclass of object through simple introspection in the Python interpreter. Recall, a metaclass is the class of a class, so what is the class of object:

    >>> print object.__class__
    <type 'type'>

Just as we expected! The metaclass of object is type, and thus all classes which inherit from object will be provided with this metaclass by default. Classes that do not inherit from object are called “old-style classes” and will disappear in Python 3.0. While old-style classes also support metaclasses and metaprogramming, we’ll focus on new-style classes in this article for the sake of simplicity.

Typically, classes are defined using the class statement in Python, as we saw in our earlier Person example. However, we have just learned that metaclasses create classes when they are instantiated. This means that we should be able to define a class by instantiating the type metaclass manually.

Lets define our original Person class again, but this time, lets do it without using the class statement by instantiating the type metaclass:

    >>> def greet(self, who):
    ...    print self.greeting, who

    >>> Person = type(
    ...    'Person', 
    ...    (object,),
    ...    {'greet': greet, 'greeting': 'Hello'}
    ... )
    >>>
    >>> jonathan = Person()
    >>> jonathan.greet('Readers')
    'Hello Readers'

This method of creating classes is equivalent to using the class statement, and reveals quite a bit about how metaclasses work. The constructor for a metaclass expects very specific arguments:

1. The first argument is the name of the class.
2. The second argument is a tuple of the base classes for the class.
3. The last argument is a dictionary representing the namespace of the class. This dictionary contains all of the attributes that would typically appear within the body of a class statement.

Now, we’ve seen a metaclass in action, and we know how to instantiate them to create classes. Armed with this knowledge of the default type metaclass, we can now tackle the much more interesting problem of creating our own metaclasses.

Defining Metaclasses

Defining metaclasses in Python is as simple as creating a class that inherits from the built-in type metaclass. The constructor for our metaclass will take the same arguments as the constructor for the type metaclass:

1. The class name.
2. A tuple of the class bases.
3. A dictionary representing the namespace of the class.

Constructors typically perform some action on their instances, so lets make our constructor set a flag on the instance that we can inspect to make sure that our metaclass is being used.

    class MyFirstMeta(type):
        def __init__(cls, name, bases, ns):
            cls.uses_my_metaclass = True

One important thing to note here is that the first argument to the constructor of a class is typically called self, as it refers to the instance being constructed. It is conventional to name the first argument of a metaclass constructor cls as the metaclass instance is actually a class.

Now that we’ve defined our metaclass, we know that we can construct a new class called MyClass simply by instantiating the metaclass:

    >>> MyClass = MyFirstMeta('MyClass', (object,), {})
    >>> print MyClass.uses_my_metaclass
    True

This very simple metaclass has given us our first glimpse into the power of metaclasses. Within our metaclass constructor, we gave our class some state in the form of a boolean attribute uses_my_metaclass. Metaclasses have the power to add, remove, or modify any attribute of the class being constructed. Metaclasses will frequently add or replace methods on their instances, based upon the data in the namespace of the class. Many Python object-relational mappers use metaclasses to transform the attributes of a class into database table definitions, for example.

While you can certainly construct classes by manually instantiating custom metaclasses, it is much more convenient to use the class statment to create your classes. Python allows you to define the metaclass for a class by using the special __metaclass__ attribute in your class statement:

    class MyClass(object):
        __metaclass__ = MyFirstMeta

This is the preferred method of attaching metaclasses to your classes. An important thing to note about this syntax is that while you are not manually instantiating the metaclass when defining your classes this way, the Python interpeter will instantiate the metaclass. The metaclass instantiation will occur immediately after the class statement has been fully executed. As a result, bugs in metaclasses often are triggered during imports. In a way, the class statement is simply syntactic sugar for instantiating metaclasses!

Mystery Metaclass Methods

Before we move onto some practical examples of metaclasses, lets investigate metaclass definition a bit further. Our Person class defines a method called greet within its namespace. Instances of Person will thus have an “instance method” called greet. We know what happens when we define methods on a class, but what happens if we define a method on a metaclass?

Methods defined on a class become instance methods. Since instances of metaclasses are classes, methods defined on metaclasses become _class methods_. Lets take a look at this in practice:

    >>> class MysteryMeta(type):
    >>> ... def mystery_method(cls):
    >>> ...     return 'I am a class method!'
    >>> 
    >>> class Mystery(object):
            __metaclass__ = MysteryMeta
    >>> 
    >>> print Mystery.mystery_method()
    I am a class method!

This revelation often surprises people, but is a logical outcome of the fact that metaclasses are simply classes which produce classes. Many metaclasses utilize this capability of defining class methods, but it is often preferable to define such class methods on a base class, which is easier to document and understand.

We’ve now established what metaclasses are, how they work, and how we can define our own by inheriting from type. But, what about practical use cases for metaclasses? Lets take a look at several examples of how we might come to use metaprogramming in practice.

Example: The Enforcer

When I am teaching Python to programmers with strong backgrounds in “bondage and discipline” languages like Java, I frequently hear the same complaints about Python. One such complaint is that class definitions are “loose” and you cannot enforce the type of variables by declaring them up-front. In order to illustrate how to use metaclasses, lets define a little library that will address this particular complaint.

Our goal is to create a base class called Enforcer that will enforce the types of attributes on its subclasses. Lets say we wanted to enforce that the name attribte of our Person class was a string, and that the age attribute was an integer. Attempting to set attribtues with the wrong type should trigger a TypeError to be raised, just like the Java compiler would catch such errors.

    class Person(Enforcer):
        name = Field(str)
        age = Field(int)

The first thing we need to do is define the Field class, which we will use to hold onto the type of the variable we’re attempting to restrict. Lets also give it the ability to validate whether or not a value is of the right type for that particular attribute:

    class Field(object):
        def __init__(self, ftype):
            self.ftype = ftype

        def is_valid(self, value):
            return isinstance(value, self.ftype)

Now that we have our Field class, we need to create a metaclass that will look at the namespace of our class, searching for Field definitions, and then storing them in a dictionary for later use. Recall that the last argument to the constructor of a metaclass is a dictionary containing the namespace of the class. We can loop through this namespace to find Field definitions:

    class EnforcerMeta(type):
        def __init__(cls, name, bases, ns):
            # store the field definitions on the class as a dictionary 
            # mapping the field name to the Field instance.
            cls._fields = {}

            # loop through the namespace looking for Field instances
            for key, value in ns.items():
                if isinstance(value, Field):
                    cls._fields[key] = value

Our metaclass first attaches a _fields dictionary to the class itself. This data structure is where we will store Field definitions for later use. We then loop through the items in the namespace looking for Field instances, and finally we store them in our _fields dictionary.

Next up is the Enforcer base class itself. The Enforcer base class first needs to attach the EnforcerMeta metaclass we’ve just defined. This is a very common way for libraries to distribute their metaclasses, by defining a base class to inherit from, rather than requiring users to even know that a metaclass is being used, or how to attach the metaclass to their classes.

The second thing the Enforcer base class needs to do is to override the __setattr__ method. This is a special method on Python classes that allows you to override the default attribute setting behavior on your Python objects. The __setattr__ method takes in the name of the attribute being set, and the value being set.

    class Enforcer(object):
        # attach the metaclass
        __metaclass__ = EnforcerMeta

        def __setattr__(self, key, value):
            if key in self._fields:
                if not self._fields[key].is_valid(value):
                    raise TypeError('Invalid type for field!')
            super(Enforcer, self).__setattr__(key, value)

Our Enforcer class first attaches the metaclass. Then, it overrides the __setattr__ method so that it can watch for field assignments. First, we check to see if the attribute being set is one of our defined fields. Then, we ask the field definition if the value that is being passed is valid for the field definition. If it is not a valid type for the field, we raise a TypeError.

The last line of the Enforcer class is extremely important. This line is instructing the Python interpreter to call the __setattr__ implementation on the appropriate superclass definition, in this case object. Without this line, attribute setting would fail on all Enforcer subclasses.

Lets try out our new creation from the Python interactive prompt:

    >>> class Person(Enforcer):
    ...    name = Field(str)
    ...    age = Field(int)
    ... 
    >>> jonathan = Person()
    >>> jonathan.name = 3
    TypeError: Invalid type
    >>> jonathan.age = 'Old'
    TypeError: Invalid type
    >>> jonathan.name = 'Jonathan'
    >>> jonathan.age = 28

Our metaclass has completely changed the way that our class behaves, transforming it into a more rigid class definition that keeps our Java-loving friends happy. While such restrictive enforcement is atypical in Python, it certainly shows how powerful metaclasses can be.

This example also illustrates a common pattern for metaclasses, in which a class describes how it wants to behave in a “declarative” way without actually writing any code to implement that behavior. The metaclass subsequently takes that metadata, and uses it to “reprogram” the class. This is the essence of what metaprogramming brings to the table, and is the most popular use-case for metaclasses.

Example: Auto Decorator

Since being given a special syntax in Python 2.4, decorators have become a very commonly used feature. A decorator is essentially a wrapper around a function or method. In versions of Python prior to 2.4, decorators were applied by manually replacing the method definition with a “decorated” version of the method. Starting in Python 2.4, decorators could be applied using the now popular @decorator syntax.

Sometimes, I find myself in the situation where I need to decorate all of the methods of a class with the same decorator. Lets create a metaclass that will simplify this process by automatically applying a decorator that is declared in the namespace of the class:

    import inspect

    class AutoDecorateMeta(type):
        def __init__(cls, name, bases, ns):
            deco = ns.get('decorator', lambda f: f)
            for key, value in ns.items():
                # skip the decorator and constructor
                if key in ('decorator', '__init__'): continue

                # skip objects in the namespace that aren't methods
                if not inspect.isfunction(value): continue

                # apply the decorator
                setattr(cls, key, deco(value))

Our metaclass first looks up an attribute named decorator in the namespace of the class. This is the decorator that we will apply to every method of the class. Note that if such a decorator cannot be found in the namespace of the class, we define a fake decorator that will simply return the existing method definition.

Next, we loop through the namespace of the class looking for methods. If we encounter the decorator or the constructor, we skip them, as we don’t want to decorate our constructor or the decorator itself. We then make use of the inspect module to determine if the value is a method, and if it is, we replace the method definition with a decorated version of that method.

Lets put our metaclass to the test by creating a class that defines several properties. Typically, we’d have to decorate every method with the @property decorator, which could add a lot of verbosity to our class, depending on the number of methods we’d have to decorate.

    class Person(object):
        __metaclass__ = AutoDecorateMeta
        decorator = property

        def __init__(self, first, middle, last):
            self.first = first
            self.middle = middle
            self.last = last

        def name(self):
            return '%s %s' % (self.first, self.last)

        def full_name(self):
            return '%s %s %s' % (self.first, self.middle, self.last)

        def initials(self):
            return '%s%s%s' % (self.first[0], self.middle[0], self.last[0])

Now that we’ve created our class, we should be able to try it out from our Python interactive prompt to see if all of our methods were transformed into properties:

    
    >>> mlk = Person('Martin', 'Luther', 'King')
    >>> print mlk.name
    Martin King
    >>> print mlk.full_name
    Martin Luther King
    >>> print mlk.initials
    MLK

Not only is this example useful in practice, it also illustrates another use case for metaclasses and metaprogramming – automating repetitive tasks that occur in class definition. This metaclass could be used to wrap every method of a class in a transaction, to add debugging hooks, or even to synchronize all of the methods using a thread lock.

Cautionary Notes

Metaclasses are fantastically cool, and as we’ve seen, they can be pretty useful. The rise of object-relational mappers and web frameworks have put metaclasses into use by an increasing number of Python users, and has increased the visibility of metaclasses substantially. That being said, metaclasses are a feature of Python that must be used carefully. Because metaclasses do their work at class definition time, bugs in your metaclasses can result in errors that are triggered at import time. In addition, metaclasses are often hidden from the programmer behind a base class, which can cause confusion.

Applied judiciously, metaclasses can be a great tool for solving problems, and give Python programmers the ability to take advantage of metaprogramming techniques in their own code. I hope that this article has shed some light on what metaclasses are, and has taken some of the mystery out of metaclasses.