PEP 0593 added the ability to add arbitrary user-defined metadata to type annotations in Python.

At type-check time, such annotations are… inert. They don’t do anything. Annotated[int, X] just means int to the type-checker, regardless of the value of X. So the entire purpose of Annotated is to provide a run-time API to consume metadata, which integrates with the type checker syntactically, but does not otherwise disturb it.

Yet, the documentation for this central purpose seems, while not exactly absent, oddly incomplete.

The PEP itself simply says:

A tool or library encountering an Annotated type can scan through the annotations to determine if they are of interest (e.g., using isinstance()).

But it’s not clear where “the annotations” are, given that the PEP’s entire “consuming annotations” section does not even mention the __metadata__ attribute where the annotation’s arguments go, which was only even added to CPython’s documentation. Its list of examples just show the repr() of the relevant type.

There’s also a bit of an open question of what, exactly, we are supposed to isinstance()-ing here. If we want to find arguments to Annotated, presumably we need to be able to detect if an annotation is an Annotated. But isinstance(Annotated[int, "hello"], Annotated) is both False at runtime, and also a type-checking error, that looks like this:

1

Argument 2 to "isinstance" has incompatible type "<typing special form>"; expected "_ClassInfo"

The actual type of these objects, typing._AnnotatedAlias, does not seem to have a publicly available or documented alias, so that seems like the wrong route too.

Now, it certainly works to escape-hatch your way out of all of this with an Any, build some version-specific special-case hacks to dig around in the relevant namespaces, access __metadata__ and call it a day. But this solution is … unsatisfying.

What are you looking for?

Upon encountering these quirks, it is understandable to want to simply ask the question “is this annotation that I’m looking at an Annotated?” and to be frustrated that it seems so obscure to straightforwardly get an answer to that question without disabling all type-checking in your meta-programming code.

However, I think that this is a slight misframing of the problem. Code that is inspecting parameters for an annotation is going to do something with that annotation, which means that it must necessarily be looking for a specific set of annotations. Therefore the thing we want to pass to isinstance is not some obscure part of the annotations’ internals, but the actual interesting annotation type from your framework or application.

When consuming an Annotated parameter, there are 3 things you probably want to know:

1. What was the parameter itself? (type: The type you passed in.)
2. What was the name of the annotated object (i.e.: the parameter name, the attribute name) being passed the parameter? (type: str)
3. What was the actual type being annotated? (type: type)

And the things that we have are the type of the Annotated we’re querying for, and the object with annotations we are interrogating. So that gives us this function signature:

1
2
3
4
5

def annotated_by(
    annotated: object,
    kind: type[T],
) -> Iterable[tuple[str, T, type]]:
    ...

To extract this information, all we need are get_args and get_type_hints; no need for __metadata__ or get_origin or any other metaprogramming. Here’s a recipe:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

def annotated_by(
    annotated: object,
    kind: type[T],
) -> Iterable[tuple[str, T, type]]:
    for k, v in get_type_hints(annotated, include_extras=True).items():
        all_args = get_args(v)
        if not all_args:
            continue
        actual, *rest = all_args
        for arg in rest:
            if isinstance(arg, kind):
                yield k, arg, actual

It might seem a little odd to be blindly assuming that get_args(...)[0] will always be the relevant type, when that is not true of unions or generics. Note, however, that we are only yielding results when we have found the instance type in the argument list; our arbitrary user-defined instance isn’t valid as a type annotation argument in any other context. It can’t be part of a Union or a Generic, so we can rely on it to be an Annotated argument, and from there, we can make that assumption about the format of get_args(...).

This can give us back the annotations that we’re looking for in a handy format that’s easy to consume. Here’s a quick example of how you might use it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

@dataclass
class AnAnnotation:
    name: str

def a_function(
    a: str,
    b: Annotated[int, AnAnnotation("b")],
    c: Annotated[float, AnAnnotation("c")],
) -> None:
    ...

print(list(annotated_by(a_function, AnAnnotation)))

# [('b', AnAnnotation(name='b'), <class 'int'>),
#  ('c', AnAnnotation(name='c'), <class 'float'>)]

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor! I am also available for consulting work if you think your organization could benefit from expertise on topics like “how do I do Python metaprogramming, but, like, not super janky”.

Get it?
Quack quack quack quack
Quack quack quack quack

Weird Al Yancovic,
“I Want A New Duck”

Mypy makes most things better

Mypy is a static type checker for Python. If you’re not already familiar, you should check it out; it’s rapidly becoming a standard for Python projects. All the cool kids are doing it. With Mypy, you get all the benefits of high-level dynamic typing for rapid experimentation, and all the benefits of rigorous type checking to complement your tests and improve reliability.¹ The best of both worlds!

Mypy can change how you write Python code. In most cases, this is for the better. For example, I have opined on numerous occasions about how bad None is. But this can significantly change with Mypy. Now, when you return None, you can say -> Optional[str] and rest assured that all your callers will be quickly, statically checked for places where they might encounter an AttributeError on that None, which makes this a more appealing option than risking raising a runtime exception (which Mypy can’t check).

But sometimes, things can get worse

But in some cases, as you add type annotations, they can make your code more brittle, especially if you annotate with the most initially obvious types. Which is to say, you define some custom classes, and then say that the parameters to and return values from your functions and methods are simply instances of those classes you just defined.

Most Mypy tutorials give you a bunch of examples with str, int, List[int], maybe an Optional[float] or two, and then leave you to your own devices when it comes to defining your own classes; yet, huge amounts of real-world applications are custom classes.

So if you’re new to Mypy, particularly if you’re applying it to a large existing codebase, it’s quite natural to write a little code like this:

1
2
3
4
5
6

from dataclasses import dataclass
@dataclass
class Duck:
    quiet: bool = False
    def quack(self) -> None:
        print("Quack." if self.quiet else "QUACK!")

and then, later, write some code like this:

1
2
3
4

def duck_war(aggressor: Duck, defender: Duck) -> None:
    aggressor.quack()
    defender.quack()
    print("The only winning move is not to play.")

In untyped, pre-Mypy python, in addition to being a poignant message about the futility of escalating violence, duck_war is a very flexible function, regardless of where it’s defined. It can take anything with a quack() method.

But, while the strictness of the Mypy type-check here brings a level of safety — no None accidentally masquerading as a Duck here — it also adds a level of brittleness. Tests which use carefully-constructed fakes will now fail to type check, because duck_war technically insists upon only precisely instances of Duck and nothing else.

A few sub-optimal answers to this question

So when you want something else that has slightly different behavior — when you want a new Duck², so to speak — what do you do?

There are a couple of anti-patterns you might arrive at to work around this as you begin your Mypy journey:

1. add # type: ignore comments to all your tests, or remove their type signatures so they don’t get type checked. This solution throws the baby out with the bath water, as it eliminates any safety that any of these callers experience.
2. add calls to cast(Duck, ...) around any things which you know are “enough like” a Duck for your purposes. This is much more fine-grained and targeted (and can be a great hack for working with libraries who provide type stubs which are too specific) but this also trades off a bit too much safety, since nothing at the point of the cast verifies anything unless you build your own ad-hoc system to do so.
3. subclassing Duck. This can also be expedient, and not too bad if you have to; Mypy removes some of the sharpest edges from Python inheritance, providing some guard rails around overriding methods, but it remains a bad idea for all the usual reasons.

The good answer: typing.Protocol

Mypy has a feature, typing.Protocol , that provides a straightforward way to describe any object that has a quack() method.

You can do this like so:

1
2
3
4
5

from typing import Protocol

class Ducky(Protocol):
    def quack(self) -> None:
        "Quack."

Now, with only a small modification to its signature — while leaving the implementation the same — duck_war can now support anything sufficiently duck-like:

1
2

def duck_war(aggressor: Ducky, defender: Ducky) -> None:
    ...

In addition to making it possible for other code — for example, a unit test — to pass its own implementation of Ducky into duck_war without subclassing or tricking the type checker, this change also improves the safety of duck_war’s implementation itself. Previously, when it took a Duck, it would have been equally valid for duck_war to access the .quiet attribute of Duck as it would have been to access .quack, even though .quiet is ostensibly an internal implementation detail.

Now, we could add an underscore prefix to quiet to make it “private”, but the type checker will still happily let you access it. So a Protocol allows you to clearly reveal your intention about what types you expect your arguments to be.

Why isn’t everything like this?

Unfortunately, typing.Protocol began its life as typing_extensions.Protocol: a custom extended feature of the type system that wasn’t present in Mypy initially, and isn’t in the standard library until Python 3.8. Built-in types like Iterable and Sequence are type-checked as if they’re Protocols by being slightly special within Mypy, but it’s not clear to the casual user how this is happening.

However, other types, like io.TextIO, don’t quite behave this way, and some early-adopter projects for Mypy have types that are either too strict or too permissive because they predated this.

So I really wanted to write this post to highlight the Protocol style of describing types and encourage folks to use it.

In conclusion

The concept I’ve described above is not new in the world of type theory.

The way that typing works in Mypy with most types — builtins, custom classes, and abstract base classes — is known as nominal typing. Nominal as in “based on names”; if the object you have directly references the name of the type it’s being compared to, by being an instance of it, then it matches.

In other words: if it’s named “Duck”, it’s a duck. There are some advantages to nominal typing³, but this brittleness is not very Pythonic!

In contrast, the type of type-checking accomplished by Protocol is known as structural typing.⁴ Whether the caller matches a Protocol depends on the structure of your object — in other words, what methods and attributes it has.

In even other-er words - if it .quack()s like a duck, it is a duck.

If you’re just starting to use Mypy — particularly if you’re building a library that exports types that users are expected to implement — consider using Protocol to describe those types. With Protocol, while you get much-improved safety from type-checking, you don’t lose the wonderful flexibility and easy testability that duck typing has always given you in Python.