Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • fsl/pytreat-practicals-2020
  • mchiew/pytreat-practicals-2020
  • ndcn0236/pytreat-practicals-2020
  • nichols/pytreat-practicals-2020
4 results
Show changes
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 7638 additions and 120 deletions
advanced_topics/operator_overloading.ipynb → advanced_topics/04_operator_overloading.ipynb
View file @ 15d75bb5
%% Cell type:markdown id: tags:
# Operator overloading
> This practical assumes you are familiar with the basics of object-oriented
> programming in Python.
Operator overloading, in an object-oriented programming language, is the
process of customising the behaviour of _operators_ (e.g. `+`, `*`, `/` and
`-`) on user-defined types. This practical aims to show you that operator
overloading is __very__ easy to do in Python.
overloading is **very** easy to do in Python.
This practical gives a brief overview of the operators which you may be most
interested in implementing. However, there are many operators (and other
special methods) which you can support in your own classes - the [official
documentation](https://docs.python.org/3.5/reference/datamodel.html#basic-customization)
documentation](https://docs.python.org/3/reference/datamodel.html#basic-customization)
is the best reference if you are interested in learning more.
* [Overview](#overview)
* [Arithmetic operators](#arithmetic-operators)
* [Equality and comparison operators](#equality-and-comparison-operators)
* [The indexing operator `[]`](#the-indexing-operator)
* [The call operator `()`](#the-call-operator)
* [The dot operator `.`](#the-dot-operator)
<a class="anchor" id="overview"></a>
## Overview
In Python, when you add two numbers together:
%% Cell type:code id: tags:
```
a = 5
b = 10
r = a + b
print(r)
```
%% Cell type:markdown id: tags:
What actually goes on behind the scenes is this:
%% Cell type:code id: tags:
```
r = a.__add__(b)
print(r)
```
%% Cell type:markdown id: tags:
In other words, whenever you use the `+` operator on two variables (the
operands to the `+` operator), the Python interpreter calls the `__add__`
method of the first operand (`a`), and passes the second operand (`b`) as an
argument.
So it is very easy to use the `+` operator with our own classes - all we have
to do is implement a method called `__add__`.
<a class="anchor" id="arithmetic-operators"></a>
## Arithmetic operators
Let's play with an example - a class which represents a 2D vector:
%% Cell type:code id: tags:
```
class Vector2D(object):
def __init__(self, x, y):
self.x = x
self.y = y
def __str__(self):
return 'Vector2D({}, {})'.format(self.x, self.y)
```
%% Cell type:markdown id: tags:
> Note that we have implemented the special `__str__` method, which allows our
> `Vector2D` instances to be converted into strings.
If we try to use the `+` operator on this class, we are bound to get an error:
%% Cell type:code id: tags:
```
v1 = Vector2D(2, 3)
v2 = Vector2D(4, 5)
print(v1 + v2)
```
%% Cell type:markdown id: tags:
But all we need to do to support the `+` operator is to implement a method
called `__add__`:
%% Cell type:code id: tags:
```
class Vector2D(object):
def __init__(self, x, y):
self.x = x
self.y = y
def __str__(self):
return 'Vector2D({}, {})'.format(self.x, self.y)
def __add__(self, other):
return Vector2D(self.x + other.x,
self.y + other.y)
```
%% Cell type:markdown id: tags:
And now we can use `+` on `Vector2D` objects - it's that easy:
%% Cell type:code id: tags:
```
v1 = Vector2D(2, 3)
v2 = Vector2D(4, 5)
print('{} + {} = {}'.format(v1, v2, v1 + v2))
```
%% Cell type:markdown id: tags:
Our `__add__` method creates and returns a new `Vector2D` which contains the
sum of the `x` and `y` components of the `Vector2D` on which it is called, and
the `Vector2D` which is passed in. We could also make the `__add__` method
work with scalars, by extending its definition a bit:
%% Cell type:code id: tags:
```
class Vector2D(object):
def __init__(self, x, y):
self.x = x
self.y = y
def __add__(self, other):
if isinstance(other, Vector2D):
return Vector2D(self.x + other.x,
self.y + other.y)
else:
return Vector2D(self.x + other, self.y + other)
def __str__(self):
return 'Vector2D({}, {})'.format(self.x, self.y)
```
%% Cell type:markdown id: tags:
So now we can add both `Vector2D` instances and scalars numbers together:
%% Cell type:code id: tags:
```
v1 = Vector2D(2, 3)
v2 = Vector2D(4, 5)
n = 6
print('{} + {} = {}'.format(v1, v2, v1 + v2))
print('{} + {} = {}'.format(v1, n, v1 + n))
```
%% Cell type:markdown id: tags:
Other numeric and logical operators can be supported by implementing the
appropriate method, for example:
- Multiplication (`*`): `__mul__`
- Division (`/`): `__div__`
- Negation (`-`): `__neg__`
- In-place addition (`+=`): `__iadd__`
- Exclusive or (`^`): `__xor__`
Take a look at the [official
documentation](https://docs.python.org/3.5/reference/datamodel.html#emulating-numeric-types)
for a full list of the arithmetic and logical operators that your classes can
support.
When an operator is applied to operands of different types, a set of fall-back
rules are followed depending on the set of methods implemented on the
operands. For example, in the expression `a + b`, if `a.__add__` is not
implemented, but but `b.__radd__` is implemented, then the latter will be
called. Take a look at the [official
documentation](https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types)
for further details, including a full list of the arithmetic and logical
operators that your classes can support.
<a class="anchor" id="equality-and-comparison-operators"></a>
## Equality and comparison operators
Adding support for equality (`==`, `!=`) and comparison (e.g. `>=`) operators
is just as easy. Imagine that we have a class called `Label`, which represents
a label in a lookup table. Our `Label` has an integer label, a name, and an
RGB colour:
%% Cell type:code id: tags:
```
class Label(object):
def __init__(self, label, name, colour):
self.label = label
self.name = name
self.colour = colour
```
%% Cell type:markdown id: tags:
In order to ensure that a list of `Label` objects is ordered by their label
values, we can implement a set of functions, so that `Label` classes can be
compared using the standard comparison operators:
%% Cell type:code id: tags:
```
import functools
# Don't worry about this statement
# just yet - it is explained below
@functools.total_ordering
class Label(object):
def __init__(self, label, name, colour):
self.label = label
self.name = name
self.colour = colour
def __str__(self):
rgb = ''.join(['{:02x}'.format(c) for c in self.colour])
return 'Label({}, {}, #{})'.format(self.label, self.name, rgb)
def __repr__(self):
return str(self)
# implement Label == Label
def __eq__(self, other):
return self.label == other.label
# implement Label < Label
def __lt__(self, other):
return self.label < other.label
```
%% Cell type:markdown id: tags:
> We also added `__str__` and `__repr__` methods to the `Label` class so that
> `Label` instances will be printed nicely.
Now we can compare and sort our `Label` instances:
%% Cell type:code id: tags:
```
l1 = Label(1, 'Parietal', (255, 0, 0))
l2 = Label(2, 'Occipital', ( 0, 255, 0))
l3 = Label(3, 'Temporal', ( 0, 0, 255))
print('{} > {}: {}'.format(l1, l2, l1 > l2))
print('{} < {}: {}'.format(l1, l3, l1 < l3))
print('{} < {}: {}'.format(l1, l3, l1 <= l3))
print('{} != {}: {}'.format(l2, l3, l2 != l3))
print(sorted((l3, l1, l2)))
```
%% Cell type:markdown id: tags:
The
[`@functools.total_ordering`](https://docs.python.org/3.5/library/functools.html#functools.total_ordering)
[`@functools.total_ordering`](https://docs.python.org/3/library/functools.html#functools.total_ordering)
is a convenience
[decorator](https://docs.python.org/3.5/glossary.html#term-decorator) which,
[decorator](https://docs.python.org/3/glossary.html#term-decorator) which,
given a class that implements equality and a single comparison function
(`__lt__` in the above code), will "fill in" the remainder of the comparison
operators. If you need very specific or complicated behaviour, then you can
provide methods for _all_ of the comparison operators, e.g. `__gt__` for `>`,
`__ge__` for `>=`, etc.).
> Decorators are introduced in another practical.
But if you just want the operators to work in the conventional manner, you can
simply use the `@functools.total_ordering` decorator, and provide `__eq__`,
and just one of `__lt__`, `__le__`, `__gt__` or `__ge__`.
Refer to the [official
documentation](https://docs.python.org/3.5/reference/datamodel.html#object.__lt__)
documentation](https://docs.python.org/3/reference/datamodel.html#object.__lt__)
for all of the details on supporting comparison operators.
> You may see the `__cmp__` method in older code bases - this provides a
> C-style comparison function which returns `<0`, `0`, or `>0` based on
> comparing two items. This has been superseded by the rich comparison
> operators introduced here, and is no longer supported in Python 3.
<a class="anchor" id="the-indexing-operator"></a>
## The indexing operator `[]`
The indexing operator (`[]`) is generally used by "container" types, such as
the built-in `list` and `dict` classes.
At its essence, there are only three types of behaviours that are possible
with the `[]` operator. All that is needed to support them are to implement
three special methods in your class, regardless of whether your class will be
indexed by sequential integers (like a `list`) or by
[hashable](https://docs.python.org/3.5/glossary.html#term-hashable) values
[hashable](https://docs.python.org/3/glossary.html#term-hashable) values
(like a `dict`):
- __Retrieval__ is performed by the `__getitem__` method
- __Assignment__ is performed by the `__setitem__` method
- __Deletion__ is performed by the `__delitem__` method
- **Retrieval** is performed by the `__getitem__` method
- **Assignment** is performed by the `__setitem__` method
- **Deletion** is performed by the `__delitem__` method
Note that, if you implement these methods in your own class, there is no
requirement for them to actually provide any form of data storage or
retrieval. However if you don't, you will probably confuse users of your code
who are used to how the `list` and `dict` types work. Whenever you deviate
from conventional behaviour, make sure you explain it well in your
documentation!
The following contrived example demonstrates all three behaviours:
%% Cell type:code id: tags:
```
class TwoTimes(object):
def __init__(self):
self.__deleted = set()
self.__assigned = {}
def __getitem__(self, key):
if key in self.__deleted:
raise KeyError('{} has been deleted!'.format(key))
elif key in self.__assigned:
return self.__assigned[key]
else:
return key * 2
def __setitem__(self, key, value):
self.__assigned[key] = value
def __delitem__(self, key):
self.__deleted.add(key)
```
%% Cell type:markdown id: tags:
Guess what happens whenever we index a `TwoTimes` object:
%% Cell type:code id: tags:
```
tt = TwoTimes()
print('TwoTimes[{}] = {}'.format(2, tt[2]))
print('TwoTimes[{}] = {}'.format(6, tt[6]))
print('TwoTimes[{}] = {}'.format('abc', tt['abc']))
```
%% Cell type:markdown id: tags:
The `TwoTimes` class allows us to override the value for a specific key:
%% Cell type:code id: tags:
```
print(tt[4])
tt[4] = 'this is not 4 * 4'
print(tt[4])
```
%% Cell type:markdown id: tags:
And we can also "delete" keys:
%% Cell type:code id: tags:
```
print(tt['12345'])
del tt['12345']
# this is going to raise an error
print(tt['12345'])
```
%% Cell type:markdown id: tags:
If you wish to support the Python `start:stop:step` [slice
notation](https://docs.python.org/3.5/library/functions.html#slice), you
notation](https://docs.python.org/3/library/functions.html#slice), you
simply need to write your `__getitem__` and `__setitem__` methods so that they
can detect `slice` objects:
%% Cell type:code id: tags:
```
class TwoTimes(object):
def __init__(self, max):
self.__max = max
def __getitem__(self, key):
if isinstance(key, slice):
start = key.start or 0
stop = key.stop or self.__max
step = key.step or 1
else:
start = key
stop = key + 1
step = 1
return [i * 2 for i in range(start, stop, step)]
```
%% Cell type:markdown id: tags:
Now we can "slice" a `TwoTimes` instance:
%% Cell type:code id: tags:
```
tt = TwoTimes(10)
print(tt[5])
print(tt[3:7])
print(tt[::2])
```
%% Cell type:markdown id: tags:
> It is possible to sub-class the built-in `list` and `dict` classes if you
> wish to extend their functionality in some way. However, if you are writing
> a class that should mimic the one of the `list` or `dict` classes, but work
> in a different way internally (e.g. a `dict`-like object which uses a
> different hashing algorithm), the `Sequence` and `MutableMapping` classes
> are [a better choice](https://stackoverflow.com/a/7148602) - you can find
> them in the
> [`collections.abc`](https://docs.python.org/3.5/library/collections.abc.html)
> [`collections.abc`](https://docs.python.org/3/library/collections.abc.html)
> module.
<a class="anchor" id="the-call-operator"></a>
## The call operator `()`
Remember how everything in Python is an object, even functions? When you call
a function, a method called `__call__` is called on the function object. We can
implement the `__call__` method on our own class, which will allow us to "call"
objects as if they are functions.
For example, the `TimedFunction` class allows us to calculate the execution
time of any function:
%% Cell type:code id: tags:
```
import time
class TimedFunction(object):
def __init__(self, func):
self.func = func
def __call__(self, *args, **kwargs):
print('Timing {}...'.format(self.func.__name__))
start = time.time()
retval = self.func(*args, **kwargs)
end = time.time()
print('Elapsed time: {:0.2f} seconds'.format(end - start))
return retval
```
%% Cell type:markdown id: tags:
Let's see how the `TimedFunction` behaves:
%% Cell type:code id: tags:
```
import numpy as np
import numpy.linalg as npla
def inverse(data):
return npla.inv(data)
tf = TimedFunction(inverse)
data = np.random.random((5000, 5000))
# Wait a few seconds after
# running this code block!
inv = tf(data)
```
%% Cell type:markdown id: tags:
> The `TimedFunction` class is conceptually very similar to a
> [decorator](https://docs.python.org/3.5/glossary.html#term-decorator) -
> [decorator](https://docs.python.org/3/glossary.html#term-decorator) -
> decorators are covered in another practical.
<a class="anchor" id="the-dot-operator"></a>
## The dot operator `.`
Python allows us to override the `.` (dot) operator which is used to access
the attributes and methods of an object. This is very powerful, but is also
quite a niche feature, and it is easy to trip yourself up, so if you wish to
use this in your own project, make sure that you carefully read (and
understand) [the
documentation](https://docs.python.org/3.5/reference/datamodel.html#customizing-attribute-access),
documentation](https://docs.python.org/3/reference/datamodel.html#customizing-attribute-access),
and test your code comprehensively!
For this example, we need a little background information. OpenGL includes
the native data types `vec2`, `vec3`, and `vec4`, which can be used to
represent 2, 3, or 4 component vectors respectively. These data types have a
neat feature called [_swizzling_][glslref], which allows you to access any
component (`x`,`y`, `z`, `w` for vectors, or `r`, `g`, `b`, `a` for colours)
in any order, with a syntax similar to attribute access in Python.
[glslref]: https://www.khronos.org/opengl/wiki/Data_Type_(GLSL)#Swizzling
So here is an example which implements this swizzle-style attribute access on
a class called `Vector`, in which we have customised the behaviour of the `.`
operator:
%% Cell type:code id: tags:
```
class Vector(object):
def __init__(self, xyz):
self.__xyz = list(xyz)
def __str__(self):
return 'Vector({})'.format(self.__xyz)
def __getattr__(self, key):
# Swizzling behaviour only occurs when
# the attribute name is entirely comprised
# of 'x', 'y', and 'z'.
if not all([c in 'xyz' for c in key]):
raise AttributeError(key)
key = ['xyz'.index(c) for c in key]
return [self.__xyz[c] for c in key]
def __setattr__(self, key, value):
# Restrict swizzling behaviour as above
if not all([c in 'xyz' for c in key]):
return super().__setattr__(key, value)
if len(key) == 1:
value = (value,)
idxs = ['xyz'.index(c) for c in key]
for i, v in sorted(zip(idxs, value)):
self.__xyz[i] = v
```
%% Cell type:markdown id: tags:
And here it is in action:
%% Cell type:code id: tags:
```
v = Vector((1, 2, 3))
print('v: ', v)
print('xyz: ', v.xyz)
print('yz: ', v.zy)
print('zy: ', v.zy)
print('xx: ', v.xx)
v.xz = 10, 30
print(v)
v.y = 20
print(v)
```
......
advanced_topics/operator_overloading.md → advanced_topics/04_operator_overloading.md
View file @ 15d75bb5
......@@ -8,13 +8,13 @@
Operator overloading, in an object-oriented programming language, is the
process of customising the behaviour of _operators_ (e.g. `+`, `*`, `/` and
`-`) on user-defined types. This practical aims to show you that operator
overloading is __very__ easy to do in Python.
overloading is **very** easy to do in Python.
This practical gives a brief overview of the operators which you may be most
interested in implementing. However, there are many operators (and other
special methods) which you can support in your own classes - the [official
documentation](https://docs.python.org/3.5/reference/datamodel.html#basic-customization)
documentation](https://docs.python.org/3/reference/datamodel.html#basic-customization)
is the best reference if you are interested in learning more.
......@@ -168,10 +168,14 @@ appropriate method, for example:
- Exclusive or (`^`): `__xor__`
Take a look at the [official
documentation](https://docs.python.org/3.5/reference/datamodel.html#emulating-numeric-types)
for a full list of the arithmetic and logical operators that your classes can
support.
When an operator is applied to operands of different types, a set of fall-back
rules are followed depending on the set of methods implemented on the
operands. For example, in the expression `a + b`, if `a.__add__` is not
implemented, but but `b.__radd__` is implemented, then the latter will be
called. Take a look at the [official
documentation](https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types)
for further details, including a full list of the arithmetic and logical
operators that your classes can support.
<a class="anchor" id="equality-and-comparison-operators"></a>
......@@ -241,16 +245,16 @@ l2 = Label(2, 'Occipital', ( 0, 255, 0))
l3 = Label(3, 'Temporal', ( 0, 0, 255))
print('{} > {}: {}'.format(l1, l2, l1 > l2))
print('{} < {}: {}'.format(l1, l3, l1 < l3))
print('{} < {}: {}'.format(l1, l3, l1 <= l3))
print('{} != {}: {}'.format(l2, l3, l2 != l3))
print(sorted((l3, l1, l2)))
```
The
[`@functools.total_ordering`](https://docs.python.org/3.5/library/functools.html#functools.total_ordering)
[`@functools.total_ordering`](https://docs.python.org/3/library/functools.html#functools.total_ordering)
is a convenience
[decorator](https://docs.python.org/3.5/glossary.html#term-decorator) which,
[decorator](https://docs.python.org/3/glossary.html#term-decorator) which,
given a class that implements equality and a single comparison function
(`__lt__` in the above code), will "fill in" the remainder of the comparison
operators. If you need very specific or complicated behaviour, then you can
......@@ -267,7 +271,7 @@ and just one of `__lt__`, `__le__`, `__gt__` or `__ge__`.
Refer to the [official
documentation](https://docs.python.org/3.5/reference/datamodel.html#object.__lt__)
documentation](https://docs.python.org/3/reference/datamodel.html#object.__lt__)
for all of the details on supporting comparison operators.
......@@ -289,13 +293,13 @@ At its essence, there are only three types of behaviours that are possible
with the `[]` operator. All that is needed to support them are to implement
three special methods in your class, regardless of whether your class will be
indexed by sequential integers (like a `list`) or by
[hashable](https://docs.python.org/3.5/glossary.html#term-hashable) values
[hashable](https://docs.python.org/3/glossary.html#term-hashable) values
(like a `dict`):
- __Retrieval__ is performed by the `__getitem__` method
- __Assignment__ is performed by the `__setitem__` method
- __Deletion__ is performed by the `__delitem__` method
- **Retrieval** is performed by the `__getitem__` method
- **Assignment** is performed by the `__setitem__` method
- **Deletion** is performed by the `__delitem__` method
Note that, if you implement these methods in your own class, there is no
......@@ -366,7 +370,7 @@ print(tt['12345'])
If you wish to support the Python `start:stop:step` [slice
notation](https://docs.python.org/3.5/library/functions.html#slice), you
notation](https://docs.python.org/3/library/functions.html#slice), you
simply need to write your `__getitem__` and `__setitem__` methods so that they
can detect `slice` objects:
......@@ -410,7 +414,7 @@ print(tt[::2])
> different hashing algorithm), the `Sequence` and `MutableMapping` classes
> are [a better choice](https://stackoverflow.com/a/7148602) - you can find
> them in the
> [`collections.abc`](https://docs.python.org/3.5/library/collections.abc.html)
> [`collections.abc`](https://docs.python.org/3/library/collections.abc.html)
> module.
......@@ -468,7 +472,7 @@ inv = tf(data)
> The `TimedFunction` class is conceptually very similar to a
> [decorator](https://docs.python.org/3.5/glossary.html#term-decorator) -
> [decorator](https://docs.python.org/3/glossary.html#term-decorator) -
> decorators are covered in another practical.
......@@ -481,7 +485,7 @@ the attributes and methods of an object. This is very powerful, but is also
quite a niche feature, and it is easy to trip yourself up, so if you wish to
use this in your own project, make sure that you carefully read (and
understand) [the
documentation](https://docs.python.org/3.5/reference/datamodel.html#customizing-attribute-access),
documentation](https://docs.python.org/3/reference/datamodel.html#customizing-attribute-access),
and test your code comprehensively!
......@@ -545,7 +549,7 @@ v = Vector((1, 2, 3))
print('v: ', v)
print('xyz: ', v.xyz)
print('yz: ', v.zy)
print('zy: ', v.zy)
print('xx: ', v.xx)
v.xz = 10, 30
......
advanced_topics/context_managers.ipynb → advanced_topics/05_context_managers.ipynb
View file @ 15d75bb5
%% Cell type:markdown id: tags:
# Context managers
The recommended way to open a file in Python is via the `with` statement:
%% Cell type:code id: tags:
```
with open('context_managers.md', 'rt') as f:
with open('05_context_managers.md', 'rt') as f:
firstlines = f.readlines()[:4]
firstlines = [l.strip() for l in firstlines]
print('\n'.join(firstlines))
```
%% Cell type:markdown id: tags:
This is because the `with` statement ensures that the file will be closed
automatically, even if an error occurs inside the `with` statement.
The `with` statement is obviously hiding some internal details from us. But
these internals are in fact quite straightforward, and are known as [_context
managers_](https://docs.python.org/3.5/reference/datamodel.html#context-managers).
these internals are in fact quite straightforward, and are known as [*context
managers*](https://docs.python.org/3/reference/datamodel.html#context-managers).
* [Anatomy of a context manager](#anatomy-of-a-context-manager)
* [Why not just use `try ... finally`?](#why-not-just-use-try-finally)
* [Uses for context managers](#uses-for-context-managers)
* [Handling errors in `__exit__`](#handling-errors-in-exit)
* [Suppressing errors with `__exit__`](#suppressing-errors-with-exit)
* [Nesting context managers](#nesting-context-managers)
* [Functions as context managers](#functions-as-context-managers)
* [Methods as context managers](#methods-as-context-managers)
* [Useful references](#useful-references)
<a class="anchor" id="anatomy-of-a-context-manager"></a>
## Anatomy of a context manager
A _context manager_ is simply an object which has two specially named methods
A *context manager* is simply an object which has two specially named methods
`__enter__` and `__exit__`. Any object which has these methods can be used in
a `with` statement.
Let's define a context manager class that we can play with:
%% Cell type:code id: tags:
```
class MyContextManager(object):
def __enter__(self):
print('In enter')
def __exit__(self, *args):
print('In exit')
```
%% Cell type:markdown id: tags:
Now, what happens when we use `MyContextManager` in a `with` statement?
%% Cell type:code id: tags:
```
with MyContextManager():
print('In with block')
```
%% Cell type:markdown id: tags:
So the `__enter__` method is called before the statements in the `with` block,
and the `__exit__` method is called afterwards.
Context managers are that simple. What makes them really useful though, is
that the `__exit__` method will be called even if the code in the `with` block
raises an error. The error will be held, and only raised after the `__exit__`
method has finished:
%% Cell type:code id: tags:
```
with MyContextManager():
print('In with block')
assert 1 == 0
```
%% Cell type:markdown id: tags:
This means that we can use context managers to perform any sort of clean up or
finalisation logic that we always want to have executed.
<a class="anchor" id="why-not-just-use-try-finally"></a>
### Why not just use `try ... finally`?
Context managers do not provide anything that cannot be accomplished in other
ways. For example, we could accomplish very similar behaviour using
[`try` - `finally` logic](https://docs.python.org/3.5/tutorial/errors.html#handling-exceptions) -
[`try` - `finally` logic](https://docs.python.org/3/tutorial/errors.html#handling-exceptions) -
the statements in the `finally` clause will *always* be executed, whether an
error is raised or not:
%% Cell type:code id: tags:
```
print('Before try block')
try:
print('In try block')
assert 1 == 0
finally:
print('In finally block')
```
%% Cell type:markdown id: tags:
But context managers have the advantage that you can implement your clean-up
logic in one place, and re-use it as many times as you want.
<a class="anchor" id="uses-for-context-managers"></a>
## Uses for context managers
We have alraedy talked about how context managers can be used to perform any
task which requires some initialistion logic, and/or some clean-up logic. As an
example, here is a context manager which creates a temporary directory,
and then makes sure that it is deleted afterwards.
We have already talked about how context managers can be used to perform any
task which requires some initialistion and/or clean-up logic. As an example,
here is a context manager which creates a temporary directory, and then makes
sure that it is deleted afterwards.
%% Cell type:code id: tags:
```
import os
import shutil
import tempfile
class TempDir(object):
def __enter__(self):
self.tempDir = tempfile.mkdtemp()
self.prevDir = os.getcwd()
print('Changing to temp dir: {}'.format(self.tempDir))
print('Previous directory: {}'.format(self.prevDir))
os.chdir(self.tempDir)
def __exit__(self, *args):
print('Changing back to: {}'.format(self.prevDir))
print('Removing temp dir: {}'.format(self.tempDir))
os .chdir( self.prevDir)
shutil.rmtree(self.tempDir)
```
%% Cell type:markdown id: tags:
Now imagine that we have a function which loads data from a file, and performs
some calculation on it:
%% Cell type:code id: tags:
```
import numpy as np
def complexAlgorithm(infile):
data = np.loadtxt(infile)
return data.mean()
```
%% Cell type:markdown id: tags:
We could use the `TempDir` context manager to write a test case for this
function, and not have to worry about cleaning up the test data:
%% Cell type:code id: tags:
```
with TempDir():
print('Testing complex algorithm')
data = np.random.random((100, 100))
np.savetxt('data.txt', data)
result = complexAlgorithm('data.txt')
assert result > 0.1 and result < 0.9
print('Test passed (result: {})'.format(result))
```
%% Cell type:markdown id: tags:
<a class="anchor" id="handling-errors-in-exit"></a>
### Handling errors in `__exit__`
By now you must be [panicking](https://youtu.be/cSU_5MgtDc8?t=9) about why I
haven't mentioned those conspicuous `*args` that get passed to the`__exit__`
method. It turns out that a context manager's [`__exit__`
method](https://docs.python.org/3.5/reference/datamodel.html#object.__exit__)
method](https://docs.python.org/3/reference/datamodel.html#object.__exit__)
is always passed three arguments.
Let's adjust our `MyContextManager` class a little so we can see what these
arguments are for:
%% Cell type:code id: tags:
```
class MyContextManager(object):
def __enter__(self):
print('In enter')
def __exit__(self, arg1, arg2, arg3):
print('In exit')
print(' arg1: ', arg1)
print(' arg2: ', arg2)
print(' arg3: ', arg3)
```
%% Cell type:markdown id: tags:
If the code inside the `with` statement does not raise an error, these three
arguments will all be `None`.
%% Cell type:code id: tags:
```
with MyContextManager():
print('In with block')
```
%% Cell type:markdown id: tags:
However, if the code inside the `with` statement raises an error, things look
a little different:
%% Cell type:code id: tags:
```
with MyContextManager():
print('In with block')
raise ValueError('Oh no!')
```
%% Cell type:markdown id: tags:
So when an error occurs, the `__exit__` method is passed the following:
- The [`Exception`](https://docs.python.org/3.5/tutorial/errors.html)
- The [`Exception`](https://docs.python.org/3/tutorial/errors.html)
type that was raised.
- The `Exception` instance that was raised.
- A [`traceback`](https://docs.python.org/3.5/library/traceback.html) object
- A [`traceback`](https://docs.python.org/3/library/traceback.html) object
which can be used to get more information about the exception (e.g. line
number).
<a class="anchor" id="suppressing-errors-with-exit"></a>
### Suppressing errors with `__exit__`
The `__exit__` method is also capable of suppressing errors - if it returns a
value of `True`, then any error that was raised will be ignored. For example,
we could write a context manager which ignores any assertion errors, but
allows other errors to halt execution as normal:
%% Cell type:code id: tags:
```
class MyContextManager(object):
def __enter__(self):
print('In enter')
def __exit__(self, arg1, arg2, arg3):
print('In exit')
if issubclass(arg1, AssertionError):
return True
print(' arg1: ', arg1)
print(' arg2: ', arg2)
print(' arg3: ', arg3)
```
%% Cell type:markdown id: tags:
> Note that if a function or method does not explicitly return a value, its
> return value is `None` (which would evaluate to `False` when converted to a
> `bool`). Also note that we are using the built-in
> [`issubclass`](https://docs.python.org/3.5/library/functions.html#issubclass)
> [`issubclass`](https://docs.python.org/3/library/functions.html#issubclass)
> function, which allows us to test the type of a class.
Now, when we use `MyContextManager`, any assertion errors are suppressed,
whereas other errors will be raised as normal:
%% Cell type:code id: tags:
```
with MyContextManager():
assert 1 == 0
print('Continuing execution!')
with MyContextManager():
raise ValueError('Oh no!')
```
%% Cell type:markdown id: tags:
<a class="anchor" id="nesting-context-managers"></a>
## Nesting context managers
It is possible to nest `with` statements:
%% Cell type:code id: tags:
```
with open('05_context_managers.md', 'rt') as inf:
with TempDir():
with open('05_context_managers.md.copy', 'wt') as outf:
outf.write(inf.read())
```
%% Cell type:markdown id: tags:
You can also use multiple context managers in a single `with` statement:
%% Cell type:code id: tags:
```
with open('05_context_managers.md', 'rt') as inf, \
TempDir(), \
open('05_context_managers.md.copy', 'wt') as outf:
outf.write(inf.read())
```
%% Cell type:markdown id: tags:
<a class="anchor" id="functions-as-context-managers"></a>
## Functions as context managers
In fact, there is another method of defining a context manager in Python. The
In fact, there is another way to create context managers in Python. The
built-in [`contextlib`
module](https://docs.python.org/3.5/library/contextlib.html#contextlib.contextmanager)
has a decorator called `contextmanager`, which allows us to turn __any__
function into a context manager. The only requirement is that the function must
have a `yield` statement<sup>1</sup>. So we could rewrite our `TempDir` class
from above as a function:
module](https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager)
has a decorator called `@contextmanager`, which allows us to turn __any
function__ into a context manager. The only requirement is that the function
must have a `yield` statement<sup>1</sup>. So we could rewrite our `TempDir`
class from above as a function:
%% Cell type:code id: tags:
```
import os
import shutil
import tempfile
import contextlib
@contextlib.contextmanager
def tempdir():
testdir = tempfile.mkdtemp()
tdir = tempfile.mkdtemp()
prevdir = os.getcwd()
try:
os.chdir(testdir)
yield testdir
os.chdir(tdir)
yield tdir
finally:
os.chdir(prevdir)
shutil.rmtree(testdir)
shutil.rmtree(tdir)
```
%% Cell type:markdown id: tags:
This new `tempdir` function is used in exactly the same way as our `TempDir`
class:
%% Cell type:code id: tags:
```
print('In directory: {}'.format(os.getcwd()))
with tempdir():
with tempdir() as tmp:
print('Now in directory: {}'.format(os.getcwd()))
print('Back in directory: {}'.format(os.getcwd()))
```
%% Cell type:markdown id: tags:
> <sup>1</sup>
The `yield tdir` statement in our `tempdir` function causes the `tdir` value
to be passed to the `with` statement, so in the line `with tempdir() as tmp`,
the variable `tmp` will be given the value `tdir`.
> https://docs.python.org/3.5/howto/functional.html#generators
> <sup>1</sup> The `yield` keyword is used in *generator functions*.
> Functions which are used with the `@contextmanager` decorator must be
> generator functions which yield exactly one value.
> [Generators](https://www.python.org/dev/peps/pep-0289/) and [generator
> functions](https://docs.python.org/3/glossary.html#term-generator) are
> beyond the scope of this practical.
## Nesting context managers
<a class="anchor" id="methods-as-context-managers"></a>
## Methods as context managers
Since it is possible to write a function which is a context manager, it is of
course also possible to write a _method_ which is a context manager. Let's
play with another example. We have a `Notifier` class which can be used to
notify interested listeners when an event occurs. Listeners can be registered
for notification via the `register` method:
%% Cell type:code id: tags:
```
from collections import OrderedDict
class Notifier(object):
def __init__(self):
super().__init__()
self.listeners = OrderedDict()
def register(self, name, func):
self.listeners[name] = func
def notify(self):
for listener in self.listeners.values():
listener()
```
%% Cell type:markdown id: tags:
Now, let's build a little plotting application. First of all, we have a `Line`
class, which represents a line plot. The `Line` class is a sub-class of
`Notifier`, so whenever its display properties (`colour`, `width`, or `name`)
change, it emits a notification, and whatever is drawing it can refresh the
display:
%% Cell type:code id: tags:
```
import numpy as np
class Line(Notifier):
def __init__(self, data):
super().__init__()
self.__data = data
self.__colour = '#000000'
self.__width = 1
self.__name = 'line'
@property
def xdata(self):
return np.arange(len(self.__data))
@property
def ydata(self):
return np.copy(self.__data)
@property
def colour(self):
return self.__colour
@colour.setter
def colour(self, newColour):
self.__colour = newColour
print('Line: colour changed: {}'.format(newColour))
self.notify()
@property
def width(self):
return self.__width
@width.setter
def width(self, newWidth):
self.__width = newWidth
print('Line: width changed: {}'.format(newWidth))
self.notify()
@property
def name(self):
return self.__name
@name.setter
def name(self, newName):
self.__name = newName
print('Line: name changed: {}'.format(newName))
self.notify()
```
%% Cell type:markdown id: tags:
Now let's write a `Plotter` class, which can plot one or more `Line`
instances:
%% Cell type:code id: tags:
```
import matplotlib.pyplot as plt
class Plotter(object):
def __init__(self, axis):
self.__axis = axis
self.__lines = []
def addData(self, data):
line = Line(data)
self.__lines.append(line)
line.register('plot', self.lineChanged)
self.draw()
return line
def lineChanged(self):
self.draw()
def draw(self):
print('Plotter: redrawing plot')
ax = self.__axis
ax.clear()
for line in self.__lines:
ax.plot(line.xdata,
line.ydata,
color=line.colour,
linewidth=line.width,
label=line.name)
ax.legend()
```
%% Cell type:markdown id: tags:
Let's create a `Plotter` object, and add a couple of lines to it (note that
the `matplotlib` plot will open in a separate window):
%% Cell type:code id: tags:
```
# this line is only necessary when
# working in jupyer notebook/ipython
%matplotlib
fig = plt.figure()
ax = fig.add_subplot(111)
plotter = Plotter(ax)
l1 = plotter.addData(np.sin(np.linspace(0, 6 * np.pi, 50)))
l2 = plotter.addData(np.cos(np.linspace(0, 6 * np.pi, 50)))
fig.show()
```
%% Cell type:markdown id: tags:
Now, when we change the properties of our `Line` instances, the plot will be
automatically updated:
%% Cell type:code id: tags:
```
l1.colour = '#ff0000'
l2.colour = '#00ff00'
l1.width = 2
l2.width = 2
l1.name = 'sine'
l2.name = 'cosine'
```
%% Cell type:markdown id: tags:
Pretty cool! However, this seems very inefficient - every time we change the
properties of a `Line`, the `Plotter` will refresh the plot. If we were
plotting large amounts of data, this would be unacceptable, as plotting would
simply take too long.
Wouldn't it be nice if we were able to perform batch-updates of `Line`
properties, and only refresh the plot when we are done? Let's add an extra
method to the `Plotter` class:
%% Cell type:code id: tags:
```
import contextlib
class Plotter(object):
def __init__(self, axis):
self.__axis = axis
self.__lines = []
self.__holdUpdates = False
def addData(self, data):
line = Line(data)
self.__lines.append(line)
line.register('plot', self.lineChanged)
if not self.__holdUpdates:
self.draw()
return line
def lineChanged(self):
if not self.__holdUpdates:
self.draw()
def draw(self):
print('Plotter: redrawing plot')
ax = self.__axis
ax.clear()
for line in self.__lines:
ax.plot(line.xdata,
line.ydata,
color=line.colour,
linewidth=line.width,
label=line.name)
ax.legend()
@contextlib.contextmanager
def holdUpdates(self):
self.__holdUpdates = True
try:
yield
self.draw()
finally:
self.__holdUpdates = False
```
%% Cell type:markdown id: tags:
This new `holdUpdates` method allows us to temporarily suppress notifications
from all `Line` instances. Let's create a new plot:
%% Cell type:code id: tags:
```
fig = plt.figure()
ax = fig.add_subplot(111)
plotter = Plotter(ax)
l1 = plotter.addData(np.sin(np.linspace(0, 6 * np.pi, 50)))
l2 = plotter.addData(np.cos(np.linspace(0, 6 * np.pi, 50)))
plt.show()
```
%% Cell type:markdown id: tags:
Now, we can update many `Line` properties without performing any redundant
redraws:
%% Cell type:code id: tags:
```
with plotter.holdUpdates():
l1.colour = '#0000ff'
l2.colour = '#ffff00'
l1.width = 1
l2.width = 1
l1.name = '$sin(x)$'
l2.name = '$cos(x)$'
```
%% Cell type:markdown id: tags:
<a class="anchor" id="useful-references"></a>
## Useful references
Useful references
* [Context manager clases](https://docs.python.org/3.5/reference/datamodel.html#context-managers)
* The [`contextlib` module](https://docs.python.org/3.5/library/contextlib.html)
* [Context manager classes](https://docs.python.org/3/reference/datamodel.html#context-managers)
* The [`contextlib` module](https://docs.python.org/3/library/contextlib.html)
......
advanced_topics/context_managers.md → advanced_topics/05_context_managers.md
View file @ 15d75bb5
......@@ -5,7 +5,7 @@ The recommended way to open a file in Python is via the `with` statement:
```
with open('context_managers.md', 'rt') as f:
with open('05_context_managers.md', 'rt') as f:
firstlines = f.readlines()[:4]
firstlines = [l.strip() for l in firstlines]
print('\n'.join(firstlines))
......@@ -17,14 +17,26 @@ automatically, even if an error occurs inside the `with` statement.
The `with` statement is obviously hiding some internal details from us. But
these internals are in fact quite straightforward, and are known as [_context
managers_](https://docs.python.org/3.5/reference/datamodel.html#context-managers).
these internals are in fact quite straightforward, and are known as [*context
managers*](https://docs.python.org/3/reference/datamodel.html#context-managers).
* [Anatomy of a context manager](#anatomy-of-a-context-manager)
* [Why not just use `try ... finally`?](#why-not-just-use-try-finally)
* [Uses for context managers](#uses-for-context-managers)
* [Handling errors in `__exit__`](#handling-errors-in-exit)
* [Suppressing errors with `__exit__`](#suppressing-errors-with-exit)
* [Nesting context managers](#nesting-context-managers)
* [Functions as context managers](#functions-as-context-managers)
* [Methods as context managers](#methods-as-context-managers)
* [Useful references](#useful-references)
<a class="anchor" id="anatomy-of-a-context-manager"></a>
## Anatomy of a context manager
A _context manager_ is simply an object which has two specially named methods
A *context manager* is simply an object which has two specially named methods
`__enter__` and `__exit__`. Any object which has these methods can be used in
a `with` statement.
......@@ -71,12 +83,13 @@ This means that we can use context managers to perform any sort of clean up or
finalisation logic that we always want to have executed.
<a class="anchor" id="why-not-just-use-try-finally"></a>
### Why not just use `try ... finally`?
Context managers do not provide anything that cannot be accomplished in other
ways. For example, we could accomplish very similar behaviour using
[`try` - `finally` logic](https://docs.python.org/3.5/tutorial/errors.html#handling-exceptions) -
[`try` - `finally` logic](https://docs.python.org/3/tutorial/errors.html#handling-exceptions) -
the statements in the `finally` clause will *always* be executed, whether an
error is raised or not:
......@@ -95,13 +108,14 @@ But context managers have the advantage that you can implement your clean-up
logic in one place, and re-use it as many times as you want.
<a class="anchor" id="uses-for-context-managers"></a>
## Uses for context managers
We have alraedy talked about how context managers can be used to perform any
task which requires some initialistion logic, and/or some clean-up logic. As an
example, here is a context manager which creates a temporary directory,
and then makes sure that it is deleted afterwards.
We have already talked about how context managers can be used to perform any
task which requires some initialistion and/or clean-up logic. As an example,
here is a context manager which creates a temporary directory, and then makes
sure that it is deleted afterwards.
```
......@@ -162,13 +176,14 @@ with TempDir():
```
<a class="anchor" id="handling-errors-in-exit"></a>
### Handling errors in `__exit__`
By now you must be [panicking](https://youtu.be/cSU_5MgtDc8?t=9) about why I
haven't mentioned those conspicuous `*args` that get passed to the`__exit__`
method. It turns out that a context manager's [`__exit__`
method](https://docs.python.org/3.5/reference/datamodel.html#object.__exit__)
method](https://docs.python.org/3/reference/datamodel.html#object.__exit__)
is always passed three arguments.
......@@ -212,14 +227,15 @@ with MyContextManager():
So when an error occurs, the `__exit__` method is passed the following:
- The [`Exception`](https://docs.python.org/3.5/tutorial/errors.html)
- The [`Exception`](https://docs.python.org/3/tutorial/errors.html)
type that was raised.
- The `Exception` instance that was raised.
- A [`traceback`](https://docs.python.org/3.5/library/traceback.html) object
- A [`traceback`](https://docs.python.org/3/library/traceback.html) object
which can be used to get more information about the exception (e.g. line
number).
<a class="anchor" id="suppressing-errors-with-exit"></a>
### Suppressing errors with `__exit__`
......@@ -246,7 +262,7 @@ class MyContextManager(object):
> Note that if a function or method does not explicitly return a value, its
> return value is `None` (which would evaluate to `False` when converted to a
> `bool`). Also note that we are using the built-in
> [`issubclass`](https://docs.python.org/3.5/library/functions.html#issubclass)
> [`issubclass`](https://docs.python.org/3/library/functions.html#issubclass)
> function, which allows us to test the type of a class.
......@@ -265,16 +281,42 @@ with MyContextManager():
```
<a class="anchor" id="nesting-context-managers"></a>
## Nesting context managers
It is possible to nest `with` statements:
```
with open('05_context_managers.md', 'rt') as inf:
with TempDir():
with open('05_context_managers.md.copy', 'wt') as outf:
outf.write(inf.read())
```
You can also use multiple context managers in a single `with` statement:
```
with open('05_context_managers.md', 'rt') as inf, \
TempDir(), \
open('05_context_managers.md.copy', 'wt') as outf:
outf.write(inf.read())
```
<a class="anchor" id="functions-as-context-managers"></a>
## Functions as context managers
In fact, there is another method of defining a context manager in Python. The
In fact, there is another way to create context managers in Python. The
built-in [`contextlib`
module](https://docs.python.org/3.5/library/contextlib.html#contextlib.contextmanager)
has a decorator called `contextmanager`, which allows us to turn __any__
function into a context manager. The only requirement is that the function must
have a `yield` statement<sup>1</sup>. So we could rewrite our `TempDir` class
from above as a function:
module](https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager)
has a decorator called `@contextmanager`, which allows us to turn __any
function__ into a context manager. The only requirement is that the function
must have a `yield` statement<sup>1</sup>. So we could rewrite our `TempDir`
class from above as a function:
```
......@@ -285,18 +327,19 @@ import contextlib
@contextlib.contextmanager
def tempdir():
testdir = tempfile.mkdtemp()
tdir = tempfile.mkdtemp()
prevdir = os.getcwd()
try:
os.chdir(testdir)
yield testdir
os.chdir(tdir)
yield tdir
finally:
os.chdir(prevdir)
shutil.rmtree(testdir)
shutil.rmtree(tdir)
```
This new `tempdir` function is used in exactly the same way as our `TempDir`
class:
......@@ -304,27 +347,271 @@ class:
```
print('In directory: {}'.format(os.getcwd()))
with tempdir():
with tempdir() as tmp:
print('Now in directory: {}'.format(os.getcwd()))
print('Back in directory: {}'.format(os.getcwd()))
```
The `yield tdir` statement in our `tempdir` function causes the `tdir` value
to be passed to the `with` statement, so in the line `with tempdir() as tmp`,
the variable `tmp` will be given the value `tdir`.
> <sup>1</sup> The `yield` keyword is used in *generator functions*.
> Functions which are used with the `@contextmanager` decorator must be
> generator functions which yield exactly one value.
> [Generators](https://www.python.org/dev/peps/pep-0289/) and [generator
> functions](https://docs.python.org/3/glossary.html#term-generator) are
> beyond the scope of this practical.
> <sup>1</sup>
> https://docs.python.org/3.5/howto/functional.html#generators
<a class="anchor" id="methods-as-context-managers"></a>
## Methods as context managers
## Nesting context managers
Since it is possible to write a function which is a context manager, it is of
course also possible to write a _method_ which is a context manager. Let's
play with another example. We have a `Notifier` class which can be used to
notify interested listeners when an event occurs. Listeners can be registered
for notification via the `register` method:
```
from collections import OrderedDict
class Notifier(object):
def __init__(self):
super().__init__()
self.listeners = OrderedDict()
def register(self, name, func):
self.listeners[name] = func
def notify(self):
for listener in self.listeners.values():
listener()
```
Now, let's build a little plotting application. First of all, we have a `Line`
class, which represents a line plot. The `Line` class is a sub-class of
`Notifier`, so whenever its display properties (`colour`, `width`, or `name`)
change, it emits a notification, and whatever is drawing it can refresh the
display:
```
import numpy as np
class Line(Notifier):
def __init__(self, data):
super().__init__()
self.__data = data
self.__colour = '#000000'
self.__width = 1
self.__name = 'line'
@property
def xdata(self):
return np.arange(len(self.__data))
@property
def ydata(self):
return np.copy(self.__data)
@property
def colour(self):
return self.__colour
@colour.setter
def colour(self, newColour):
self.__colour = newColour
print('Line: colour changed: {}'.format(newColour))
self.notify()
@property
def width(self):
return self.__width
@width.setter
def width(self, newWidth):
self.__width = newWidth
print('Line: width changed: {}'.format(newWidth))
self.notify()
@property
def name(self):
return self.__name
@name.setter
def name(self, newName):
self.__name = newName
print('Line: name changed: {}'.format(newName))
self.notify()
```
Now let's write a `Plotter` class, which can plot one or more `Line`
instances:
```
import matplotlib.pyplot as plt
class Plotter(object):
def __init__(self, axis):
self.__axis = axis
self.__lines = []
def addData(self, data):
line = Line(data)
self.__lines.append(line)
line.register('plot', self.lineChanged)
self.draw()
return line
def lineChanged(self):
self.draw()
def draw(self):
print('Plotter: redrawing plot')
ax = self.__axis
ax.clear()
for line in self.__lines:
ax.plot(line.xdata,
line.ydata,
color=line.colour,
linewidth=line.width,
label=line.name)
ax.legend()
```
Let's create a `Plotter` object, and add a couple of lines to it (note that
the `matplotlib` plot will open in a separate window):
```
# this line is only necessary when
# working in jupyer notebook/ipython
%matplotlib
fig = plt.figure()
ax = fig.add_subplot(111)
plotter = Plotter(ax)
l1 = plotter.addData(np.sin(np.linspace(0, 6 * np.pi, 50)))
l2 = plotter.addData(np.cos(np.linspace(0, 6 * np.pi, 50)))
fig.show()
```
Now, when we change the properties of our `Line` instances, the plot will be
automatically updated:
```
l1.colour = '#ff0000'
l2.colour = '#00ff00'
l1.width = 2
l2.width = 2
l1.name = 'sine'
l2.name = 'cosine'
```
Pretty cool! However, this seems very inefficient - every time we change the
properties of a `Line`, the `Plotter` will refresh the plot. If we were
plotting large amounts of data, this would be unacceptable, as plotting would
simply take too long.
Wouldn't it be nice if we were able to perform batch-updates of `Line`
properties, and only refresh the plot when we are done? Let's add an extra
method to the `Plotter` class:
```
import contextlib
class Plotter(object):
def __init__(self, axis):
self.__axis = axis
self.__lines = []
self.__holdUpdates = False
def addData(self, data):
line = Line(data)
self.__lines.append(line)
line.register('plot', self.lineChanged)
if not self.__holdUpdates:
self.draw()
return line
def lineChanged(self):
if not self.__holdUpdates:
self.draw()
def draw(self):
print('Plotter: redrawing plot')
ax = self.__axis
ax.clear()
for line in self.__lines:
ax.plot(line.xdata,
line.ydata,
color=line.colour,
linewidth=line.width,
label=line.name)
ax.legend()
@contextlib.contextmanager
def holdUpdates(self):
self.__holdUpdates = True
try:
yield
self.draw()
finally:
self.__holdUpdates = False
```
This new `holdUpdates` method allows us to temporarily suppress notifications
from all `Line` instances. Let's create a new plot:
```
fig = plt.figure()
ax = fig.add_subplot(111)
plotter = Plotter(ax)
l1 = plotter.addData(np.sin(np.linspace(0, 6 * np.pi, 50)))
l2 = plotter.addData(np.cos(np.linspace(0, 6 * np.pi, 50)))
plt.show()
```
Now, we can update many `Line` properties without performing any redundant
redraws:
```
with plotter.holdUpdates():
l1.colour = '#0000ff'
l2.colour = '#ffff00'
l1.width = 1
l2.width = 1
l1.name = '$sin(x)$'
l2.name = '$cos(x)$'
```
<a class="anchor" id="useful-references"></a>
## Useful references
Useful references
* [Context manager clases](https://docs.python.org/3.5/reference/datamodel.html#context-managers)
* The [`contextlib` module](https://docs.python.org/3.5/library/contextlib.html)
\ No newline at end of file
* [Context manager classes](https://docs.python.org/3/reference/datamodel.html#context-managers)
* The [`contextlib` module](https://docs.python.org/3/library/contextlib.html)
advanced_topics/decorators.ipynb → advanced_topics/06_decorators.ipynb
View file @ 15d75bb5
%% Cell type:markdown id: tags:
# Decorators
Remember that in Python, everything is an object, including functions. This
means that we can do things like:
- Pass a function as an argument to another function.
- Create/define a function inside another function.
- Write a function which returns another function.
These abilities mean that we can do some neat things with functions in Python.
* [Overview](#overview)
* [Decorators on methods](#decorators-on-methods)
* [Example - memoization](#example-memoization)
* [Decorators with arguments](#decorators-with-arguments)
* [Chaining decorators](#chaining-decorators)
* [Decorator classes](#decorator-classes)
* [Appendix: Functions are not special](#appendix-functions-are-not-special)
* [Appendix: Closures](#appendix-closures)
* [Appendix: Decorators without arguments versus decorators with arguments](#appendix-decorators-without-arguments-versus-decorators-with-arguments)
* [Appendix: Per-instance decorators](#appendix-per-instance-decorators)
* [Appendix: Preserving function metadata](#appendix-preserving-function-metadata)
* [Appendix: Class decorators](#appendix-class-decorators)
* [Useful references](#useful-references)
<a class="anchor" id="overview"></a>
## Overview
Let's say that we want a way to calculate the execution time of any function
(this example might feel familiar to you if you have gone through the
practical on operator overloading).
Our first attempt at writing such a function might look like this:
%% Cell type:code id: tags:
```
import time
def timeFunc(func, *args, **kwargs):
start = time.time()
retval = func(*args, **kwargs)
end = time.time()
print('Ran {} in {:0.2f} seconds'.format(func.__name__, end - start))
return retval
```
%% Cell type:markdown id: tags:
The `timeFunc` function accepts another function, `func`, as its first
argument. It calls `func`, passing it all of the other arguments, and then
prints the time taken for `func` to complete:
%% Cell type:code id: tags:
```
import numpy as np
import numpy.linalg as npla
def inverse(a):
return npla.inv(a)
data = np.random.random((2000, 2000))
invdata = timeFunc(inverse, data)
```
%% Cell type:markdown id: tags:
But this means that whenever we want to time something, we have to call the
`timeFunc` function directly. Let's take advantage of the fact that we can
define a function inside another funciton. Look at the next block of code
carefully, and make sure you understand what our new `timeFunc` implementation
is doing.
%% Cell type:code id: tags:
```
import time
def timeFunc(func):
def wrapperFunc(*args, **kwargs):
start = time.time()
retval = func(*args, **kwargs)
end = time.time()
print('Ran {} in {:0.2f} seconds'.format(func.__name__, end - start))
return retval
return wrapperFunc
```
%% Cell type:markdown id: tags:
This new `timeFunc` function is again passed a function `func`, but this time
as its sole argument. It then creates and returns a new function,
`wrapperFunc`. This `wrapperFunc` function calls and times the function that
was passed to `timeFunc`. But note that when `timeFunc` is called,
`wrapperFunc` is _not_ called - it is only created and returned.
`wrapperFunc` is *not* called - it is only created and returned.
Let's use our new `timeFunc` implementation:
%% Cell type:code id: tags:
```
import numpy as np
import numpy.linalg as npla
def inverse(a):
return npla.inv(a)
data = np.random.random((2000, 2000))
inverse = timeFunc(inverse)
invdata = inverse(data)
```
%% Cell type:markdown id: tags:
Here, we did the following:
1. We defined a function called `inverse`:
> ```
> def inverse(a):
> return npla.inv(a)
> ```
2. We passed the `inverse` function to the `timeFunc` function, and
re-assigned the return value of `timeFunc` back to `inverse`:
> ```
> inverse = timeFunc(inverse)
> ```
3. We called the new `inverse` function:
> ```
> invdata = inverse(data)
> ```
So now the `inverse` variable refers to an instantiation of `wrapperFunc`,
which holds a reference to the original definition of `inverse`.
> If this is not clear, take a break now and read through the appendix on how
> [functions are not special](#appendix-functions-are-not-special).
Guess what? We have just created a __decorator__. A decorator is simply a
Guess what? We have just created a **decorator**. A decorator is simply a
function which accepts a function as its input, and returns another function
as its output. In the example above, we have _decorated_ the `inverse`
as its output. In the example above, we have *decorated* the `inverse`
function with the `timeFunc` decorator.
Python provides an alternative syntax for decorating one function with
another, using the `@` character. The approach that we used to decorate
`inverse` above:
%% Cell type:code id: tags:
```
def inverse(a):
return npla.inv(a)
inverse = timeFunc(inverse)
invdata = inverse(data)
```
%% Cell type:markdown id: tags:
is semantically equivalent to this:
%% Cell type:code id: tags:
```
@timeFunc
def inverse(a):
return npla.inv(a)
invdata = inverse(data)
```
%% Cell type:markdown id: tags:
<a class="anchor" id="decorators-on-methods"></a>
## Decorators on methods
Applying a decorator to the methods of a class works in the same way:
%% Cell type:code id: tags:
```
import numpy.linalg as npla
class MiscMaths(object):
@timeFunc
def inverse(self, a):
return npla.inv(a)
```
%% Cell type:markdown id: tags:
Now, the `inverse` method of all `MiscMaths` instances will be timed:
%% Cell type:code id: tags:
```
mm1 = MiscMaths()
mm2 = MiscMaths()
i1 = mm1.inverse(np.random.random((1000, 1000)))
i2 = mm2.inverse(np.random.random((1500, 1500)))
```
%% Cell type:markdown id: tags:
Note that only one `timeFunc` decorator was created here - the `timeFunc`
function was only called once - when the `MiscMaths` class was defined. This
might be clearer if we re-write the above code in the following (equivalent)
manner:
%% Cell type:code id: tags:
```
class MiscMaths(object):
def inverse(self, a):
return npla.inv(a)
MiscMaths.inverse = timeFunc(MiscMaths.inverse)
```
%% Cell type:markdown id: tags:
So only one `wrapperFunc` function exists, and this function is _shared_ by
So only one `wrapperFunc` function exists, and this function is *shared* by
all instances of the `MiscMaths` class - (such as the `mm1` and `mm2`
instances in the example above). In many cases this is not a problem, but
there can be situations where you need each instance of your class to have its
own unique decorator.
> If you are interested in solutions to this problem, take a look at the
> appendix on [per-instance decorators](#appendix-per-instance-decorators).
<a class="anchor" id="example-memoization"></a>
## Example - memoization
Let's move onto another example.
[Meowmoization](https://en.wikipedia.org/wiki/Memoization) is a common
performance optimisation technique used in cats. I mean software. Essentially,
memoization refers to the process of maintaining a cache for a function which
performs some expensive calculation. When the function is executed with a set
of inputs, the calculation is performed, and then a copy of the inputs and the
result are cached. If the function is called again with the same inputs, the
cached result can be returned.
This is a perfect problem to tackle with decorators:
%% Cell type:code id: tags:
```
def memoize(func):
cache = {}
def wrapper(*args):
# is there a value in the cache
# for this set of inputs?
cached = cache.get(args, None)
# If not, call the function,
# and cache the result.
if cached is None:
cached = func(*args)
cache[args] = cached
else:
print('Cached {}({}): {}'.format(func.__name__, args, cached))
return cached
return wrapper
```
%% Cell type:markdown id: tags:
We can now use our `memoize` decorator to add a memoization cache to any
function. Let's memoize a function which generates the $n^{th}$ number in the
[Fibonacci series](https://en.wikipedia.org/wiki/Fibonacci_number):
%% Cell type:code id: tags:
```
@memoize
def fib(n):
if n in (0, 1):
print('fib({}) = {}'.format(n, n))
return 1
return n
twoback = 1
oneback = 1
val = 0
val = 1
for _ in range(2, n):
val = oneback + twoback
twoback = oneback
oneback = val
print('fib({}) = {}'.format(n, val))
return val
```
%% Cell type:markdown id: tags:
For a given input, when `fib` is called the first time, it will calculate the
$n^{th}$ Fibonacci number:
%% Cell type:code id: tags:
```
for i in range(10):
fib(i)
```
%% Cell type:markdown id: tags:
However, on repeated calls with the same input, the calculation is skipped,
and instead the result is retrieved from the memoization cache:
%% Cell type:code id: tags:
```
for i in range(10):
fib(i)
```
%% Cell type:markdown id: tags:
> If you are wondering how the `wrapper` function is able to access the
> `cache` variable, refer to the [appendix on closures](#appendix-closures).
<a class="anchor" id="decorators-with-arguments"></a>
## Decorators with arguments
Continuing with our memoization example, let's say that we want to place a
limit on the maximum size that our cache can grow to. For example, the output
of our function might have large memory requirements, so we can only afford to
store a handful of pre-calculated results. It would be nice to be able to
specify the maximum cache size when we define our function to be memoized,
like so:
> ```
> # cache at most 10 results
> @limitedMemoize(10):
> def fib(n):
> ...
> ```
In order to support this, our `memoize` decorator function needs to be
modified - it is currently written to accept a function as its sole argument,
but we need it to accept a cache size limit.
%% Cell type:code id: tags:
```
from collections import OrderedDict
def limitedMemoize(maxSize):
cache = OrderedDict()
def decorator(func):
def wrapper(*args):
# is there a value in the cache
# for this set of inputs?
cached = cache.get(args, None)
# If not, call the function,
# and cache the result.
if cached is None:
cached = func(*args)
# If the cache has grown too big,
# remove the oldest item. In practice
# it would make more sense to remove
# the item with the oldest access
# time, but this is good enough for
# an introduction!
# time (or remove the least recently
# used item, as the built-in
# @functools.lru_cache does), but this
# is good enough for now!
if len(cache) >= maxSize:
cache.popitem(last=False)
cache[args] = cached
else:
print('Cached {}({}): {}'.format(func.__name__, args, cached))
return cached
return wrapper
return decorator
```
%% Cell type:markdown id: tags:
> We used the handy
> [`collections.OrderedDict`](https://docs.python.org/3.5/library/collections.html#collections.OrderedDict)
> [`collections.OrderedDict`](https://docs.python.org/3/library/collections.html#collections.OrderedDict)
> class here which preserves the insertion order of key-value pairs.
This is starting to look a little complicated - we now have _three_ layers of
This is starting to look a little complicated - we now have *three* layers of
functions. This is necessary when you wish to write a decorator which accepts
arguments (refer to the
[appendix](#appendix-decorators-without-arguments-versus-decorators-with-arguments)
for more details).
But this `limitedMemoize` decorator is used in essentially the same way as our
earlier `memoize` decorator:
%% Cell type:code id: tags:
```
@limitedMemoize(5)
def fib(n):
if n in (1, 2):
if n in (0, 1):
print('fib({}) = 1'.format(n))
return 1
return n
twoback = 1
oneback = 1
val = 0
val = 1
for _ in range(2, n):
val = oneback + twoback
twoback = oneback
oneback = val
print('fib({}) = {}'.format(n, val))
return val
```
%% Cell type:markdown id: tags:
Except that now, the `fib` function will only cache up to 5 values.
%% Cell type:code id: tags:
```
fib(10)
fib(11)
fib(12)
fib(13)
fib(14)
print('The result for 10 should come from the cache')
fib(10)
fib(15)
print('The result for 10 should no longer be cached')
fib(10)
```
%% Cell type:markdown id: tags:
<a class="anchor" id="chaining-decorators"></a>
## Chaining decorators
Decorators can easily be chained, or nested:
%% Cell type:code id: tags:
```
import time
@timeFunc
@memoize
def expensiveFunc(n):
time.sleep(n)
return n
```
%% Cell type:markdown id: tags:
> Remember that this is semantically equivalent to the following:
>
> ```
> def expensiveFunc(n):
> time.sleep(n)
> return n
>
> expensiveFunc = timeFunc(memoize(expensiveFunc))
> ```
Now we can see the effect of our memoization layer on performance:
%% Cell type:code id: tags:
```
expensiveFunc(0.5)
expensiveFunc(1)
expensiveFunc(1)
```
%% Cell type:markdown id: tags:
> Note that in Python 3.2 and newer you can use the
> [`functools.lru_cache`](https://docs.python.org/3/library/functools.html#functools.lru_cache)
> to memoize your functions.
<a class="anchor" id="decorator-classes"></a>
## Decorator classes
By now, you will have gained the impression that a decorator is a function
which _decorates_ another function. But if you went through the practical on
which *decorates* another function. But if you went through the practical on
operator overloading, you might remember the special `__call__` method, that
allows an object to be called as if it were a function.
This feature allows us to write our decorators as classes, instead of
functions. This can be handy if you are writing a decorator that has
complicated behaviour, and/or needs to maintain some sort of state which
cannot be easily or elegantly written using nested functions.
As an example, let's say we are writing a framework for unit testing. We want
to be able to "mark" our test functions like so, so they can be easily
identified and executed:
> ```
> @unitTest
> def testblerk():
> """tests the blerk algorithm."""
> ...
> ```
With a decorator like this, we wouldn't need to worry about where our tests
are located - they will all be detected because we have marked them as test
functions. What does this `unitTest` decorator look like?
%% Cell type:code id: tags:
```
class TestRegistry(object):
def __init__(self):
self.testFuncs = []
def __call__(self, func):
self.testFuncs.append(func)
def listTests(self):
print('All registered tests:')
for test in self.testFuncs:
print(' ', test.__name__)
def runTests(self):
for test in self.testFuncs:
print('Running test {:10s} ... '.format(test.__name__), end='')
try:
test()
print('passed!')
except Exception as e:
print('failed!')
# Create our test registry
registry = TestRegistry()
# Alias our registry to "unitTest"
# so that we can register tests
# with a "@unitTest" decorator.
unitTest = registry
```
%% Cell type:markdown id: tags:
So we've defined a class, `TestRegistry`, and created an instance of it,
`registry`, which will manage all of our unit tests. Now, in order to "mark"
any function as being a unit test, we just need to use the `unitTest`
decorator (which is simply a reference to our `TestRegistry` instance):
%% Cell type:code id: tags:
```
@unitTest
def testFoo():
assert 'a' in 'bcde'
@unitTest
def testBar():
assert 1 > 0
@unitTest
def testBlerk():
assert 9 % 2 == 0
```
%% Cell type:markdown id: tags:
Now that these functions have been registered with our `TestRegistry`
instance, we can run them all:
%% Cell type:code id: tags:
```
registry.listTests()
registry.runTests()
```
%% Cell type:markdown id: tags:
> Unit testing is something which you must do! This is __especially__
> Unit testing is something which you must do! This is **especially**
> important in an interpreted language such as Python, where there is no
> compiler to catch all of your mistakes.
>
> Python has a built-in
> [`unittest`](https://docs.python.org/3.5/library/unittest.html) module,
> [`unittest`](https://docs.python.org/3/library/unittest.html) module,
> however the third-party [`pytest`](https://docs.pytest.org/en/latest/) and
> [`nose`](http://nose2.readthedocs.io/en/latest/) are popular. It is also
> wise to combine your unit tests with
> [`coverage`](https://coverage.readthedocs.io/en/coverage-4.5.1/), which
> tells you how much of your code was executed, or _covered_ when your
> tells you how much of your code was executed, or *covered* when your
> tests were run.
<a class="anchor" id="appendix-functions-are-not-special"></a>
## Appendix: Functions are not special
When we write a statement like this:
%% Cell type:code id: tags:
```
a = [1, 2, 3]
```
%% Cell type:markdown id: tags:
the variable `a` is a reference to a `list`. We can create a new reference to
the same list, and delete `a`:
%% Cell type:code id: tags:
```
b = a
del a
```
%% Cell type:markdown id: tags:
Deleting `a` doesn't affect the list at all - the list still exists, and is
now referred to by a variable called `b`.
%% Cell type:code id: tags:
```
print('b: ', b)
```
%% Cell type:markdown id: tags:
`a` has, however, been deleted:
%% Cell type:code id: tags:
```
print('a: ', a)
```
%% Cell type:markdown id: tags:
The variables `a` and `b` are just references to a list that is sitting in
memory somewhere - renaming or removing a reference does not have any effect
upon the list<sup>2</sup>.
If you are familiar with C or C++, you can think of a variable in Python as
like a `void *` pointer - it is just a pointer of an unspecified type, which
is pointing to some item in memory (which does have a specific type). Deleting
the pointer does not have any effect upon the item to which it was pointing.
> <sup>2</sup> Until no more references to the list exist, at which point it
> will be
> [garbage-collected](https://www.quora.com/How-does-garbage-collection-in-Python-work-What-are-the-pros-and-cons).
Now, functions in Python work in _exactly_ the same way as variables. When we
define a function like this:
%% Cell type:code id: tags:
```
def inverse(a):
return npla.inv(a)
print(inverse)
```
%% Cell type:markdown id: tags:
there is nothing special about the name `inverse` - `inverse` is just a
reference to a function that resides somewhere in memory. We can create a new
reference to this function:
%% Cell type:code id: tags:
```
inv2 = inverse
```
%% Cell type:markdown id: tags:
And delete the old reference:
%% Cell type:code id: tags:
```
del inverse
```
%% Cell type:markdown id: tags:
But the function still exists, and is still callable, via our second
reference:
%% Cell type:code id: tags:
```
print(inv2)
data = np.random.random((10, 10))
invdata = inv2(data)
```
%% Cell type:markdown id: tags:
So there is nothing special about functions in Python - they are just items
that reside somewhere in memory, and to which we can create as many references
as we like.
> If it bothers you that `print(inv2)` resulted in
> `<function inverse at ...>`, and not `<function inv2 at ...>`, then refer to
> the appendix on
> [preserving function metdata](#appendix-preserving-function-metadata).
> [preserving function metadata](#appendix-preserving-function-metadata).
<a class="anchor" id="appendix-closures"></a>
## Appendix: Closures
Whenever we define or use a decorator, we are taking advantage of a concept
called a [_closure_][wiki-closure]. Take a second to re-familiarise yourself
called a [*closure*][wiki-closure]. Take a second to re-familiarise yourself
with our `memoize` decorator function from earlier - when `memoize` is called,
it creates and returns a function called `wrapper`:
[wiki-closure]: https://en.wikipedia.org/wiki/Closure_(computer_programming)
%% Cell type:code id: tags:
```
def memoize(func):
cache = {}
def wrapper(*args):
# is there a value in the cache
# for this set of inputs?
cached = cache.get(args, None)
# If not, call the function,
# and cache the result.
if cached is None:
cached = func(*args)
cache[args] = cached
else:
print('Cached {}({}): {}'.format(func.__name__, args, cached))
return cached
return wrapper
```
%% Cell type:markdown id: tags:
Then `wrapper` is executed at some arbitrary point in the future. But how does
it have access to `cache`, defined within the scope of the `memoize` function,
after the execution of `memoize` has ended?
%% Cell type:code id: tags:
```
def nby2(n):
return n * 2
# wrapper function is created here (and
# assigned back to the nby2 reference)
nby2 = memoize(nby2)
# wrapper function is executed here
print('nby2(2): ', nby2(2))
print('nby2(2): ', nby2(2))
```
%% Cell type:markdown id: tags:
The trick is that whenever a nested function is defined in Python, the scope
in which it is defined is preserved for that function's lifetime. So `wrapper`
has access to all of the variables within the `memoize` function's scope, that
were defined at the time that `wrapper` was created (which was when we called
`memoize`). This is why `wrapper` is able to access `cache`, even though at
the time that `wrapper` is called, the execution of `memoize` has long since
finished.
This is what is known as a
[_closure_](https://www.geeksforgeeks.org/python-closures/). Closures are a
[*closure*](https://www.geeksforgeeks.org/python-closures/). Closures are a
fundamental, and extremely powerful, aspect of Python and other high level
languages. So there's your answer,
[fishbulb](https://www.youtube.com/watch?v=CiAaEPcnlOg).
<a class="anchor" id="appendix-decorators-without-arguments-versus-decorators-with-arguments"></a>
## Appendix: Decorators without arguments versus decorators with arguments
There are three ways to invoke a decorator with the `@` notation:
1. Naming it, e.g. `@mydecorator`
2. Calling it, e.g. `@mydecorator()`
3. Calling it, and passing it arguments, e.g. `@mydecorator(1, 2, 3)`
Python expects a decorator function to behave differently in the second and
third scenarios, when compared to the first:
%% Cell type:code id: tags:
```
def decorator(*args):
print(' decorator({})'.format(args))
def wrapper(*args):
print(' wrapper({})'.format(args))
return wrapper
print('Scenario #1: @decorator')
@decorator
def noop():
pass
print('\nScenario #2: @decorator()')
@decorator()
def noop():
pass
print('\nScenario #3: @decorator(1, 2, 3)')
@decorator(1, 2, 3)
def noop():
pass
```
%% Cell type:markdown id: tags:
So if a decorator is "named" (scenario 1), only the decorator function
(`decorator` in the example above) is called, and is passed the decorated
function.
But if a decorator function is "called" (scenarios 2 or 3), both the decorator
function (`decorator`), __and its return value__ (`wrapper`) are called - the
function (`decorator`), **and its return value** (`wrapper`) are called - the
decorator function is passed the arguments that were provided, and its return
value is passed the decorated function.
This is why, if you are writing a decorator function which expects arguments,
you must use three layers of functions, like so:
%% Cell type:code id: tags:
```
def decorator(*args):
def realDecorator(func):
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
return realDecorator
```
%% Cell type:markdown id: tags:
> The author of this practical is angry about this, as he does not understand
> why the Python language designers couldn't allow a decorator function to be
> passed both the decorated function, and any arguments that were passed when
> the decorator was invoked, like so:
>
> ```
> def decorator(func, *args, **kwargs): # args/kwargs here contain
> # whatever is passed to the
> # decorator
>
> def wrapper(*args, **kwargs): # args/kwargs here contain
> # whatever is passed to the
> # decorated function
> return func(*args, **kwargs)
>
> return wrapper
> ```
<a class="anchor" id="appendix-per-instance-decorators"></a>
## Appendix: Per-instance decorators
In the section on [decorating methods](#decorators-on-methods), you saw
that when a decorator is applied to a method of a class, that decorator
is invoked just once, and shared by all instances of the class. Consider this
example:
%% Cell type:code id: tags:
```
def decorator(func):
print('Decorating {} function'.format(func.__name__))
def wrapper(*args, **kwargs):
print('Calling decorated function {}'.format(func.__name__))
return func(*args, **kwargs)
return wrapper
class MiscMaths(object):
@decorator
def add(self, a, b):
return a + b
```
%% Cell type:markdown id: tags:
Note that `decorator` was called at the time that the `MiscMaths` class was
defined. Now, all `MiscMaths` instances share the same `wrapper` function:
%% Cell type:code id: tags:
```
mm1 = MiscMaths()
mm2 = MiscMaths()
print('1 + 2 =', mm1.add(1, 2))
print('3 + 4 =', mm2.add(3, 4))
```
%% Cell type:markdown id: tags:
This is not an issue in many cases, but it can be problematic in some. Imagine
if we have a decorator called `ensureNumeric`, which makes sure that arguments
passed to a function are numbers:
%% Cell type:code id: tags:
```
def ensureNumeric(func):
def wrapper(*args):
args = tuple([float(a) for a in args])
return func(*args)
return wrapper
```
%% Cell type:markdown id: tags:
This all looks well and good - we can use it to decorate a numeric function,
allowing strings to be passed in as well:
%% Cell type:code id: tags:
```
@ensureNumeric
def mul(a, b):
return a * b
print(mul( 2, 3))
print(mul('5', '10'))
```
%% Cell type:markdown id: tags:
But what will happen when we try to decorate a method of a class?
%% Cell type:code id: tags:
```
class MiscMaths(object):
@ensureNumeric
def add(self, a, b):
return a + b
mm = MiscMaths()
print(mm.add('5', 10))
```
%% Cell type:markdown id: tags:
What happened here?? Remember that the first argument passed to any instance
method is the instance itself (the `self` argument). Well, the `MiscMaths`
instance was passed to the `wrapper` function, which then tried to convert it
into a `float`. So we can't actually apply the `ensureNumeric` function as a
decorator on a method in this way.
There are a few potential solutions here. We could modify the `ensureNumeric`
function, so that the `wrapper` ignores the first argument. But this would
mean that we couldn't use `ensureNumeric` with standalone functions.
But we _can_ manually apply the `ensureNumeric` decorator to `MiscMaths`
But we *can* manually apply the `ensureNumeric` decorator to `MiscMaths`
instances when they are initialised. We can't use the nice `@ensureNumeric`
syntax to apply our decorators, but this is a viable approach:
%% Cell type:code id: tags:
```
class MiscMaths(object):
def __init__(self):
self.add = ensureNumeric(self.add)
def add(self, a, b):
return a + b
mm = MiscMaths()
print(mm.add('5', 10))
```
%% Cell type:markdown id: tags:
Another approach is to use a second decorator, which dynamically creates the
real decorator when it is accessed on an instance. This requires the use of an
advanced Python technique called
[_descriptors_](https://docs.python.org/3.5/howto/descriptor.html), which is
[*descriptors*](https://docs.python.org/3/howto/descriptor.html), which is
beyond the scope of this practical. But if you are interested, you can see an
implementation of this approach
[here](https://git.fmrib.ox.ac.uk/fsl/fslpy/blob/1.6.8/fsl/utils/memoize.py#L249).
<a class="anchor" id="appendix-preserving-function-metadata"></a>
## Appendix: Preserving function metadata
You may have noticed that when we decorate a function, some of its properties
are lost. Consider this function:
%% Cell type:code id: tags:
```
def add2(a, b):
"""Adds two numbers together."""
return a + b
```
%% Cell type:markdown id: tags:
The `add2` function is an object which has some attributes, e.g.:
%% Cell type:code id: tags:
```
print('Name: ', add2.__name__)
print('Help: ', add2.__doc__)
```
%% Cell type:markdown id: tags:
However, when we apply a decorator to `add2`:
%% Cell type:code id: tags:
```
def decorator(func):
def wrapper(*args, **kwargs):
"""Internal wrapper function for decorator."""
print('Calling decorated function {}'.format(func.__name__))
return func(*args, **kwargs)
return wrapper
@decorator
def add2(a, b):
"""Adds two numbers together."""
return a + b
```
%% Cell type:markdown id: tags:
Those attributes are lost, and instead we get the attributes of the `wrapper`
function:
%% Cell type:code id: tags:
```
print('Name: ', add2.__name__)
print('Help: ', add2.__doc__)
```
%% Cell type:markdown id: tags:
While this may be inconsequential in most situations, it can be quite annoying
in some, such as when we are automatically [generating
documentation](http://www.sphinx-doc.org/) for our code.
Fortunately, there is a workaround, available in the built-in
[`functools`](https://docs.python.org/3.5/library/functools.html#functools.wraps)
[`functools`](https://docs.python.org/3/library/functools.html#functools.wraps)
module:
%% Cell type:code id: tags:
```
import functools
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
"""Internal wrapper function for decorator."""
print('Calling decorated function {}'.format(func.__name__))
return func(*args, **kwargs)
return wrapper
@decorator
def add2(a, b):
"""Adds two numbers together."""
return a + b
```
%% Cell type:markdown id: tags:
We have applied the `@functools.wraps` decorator to our internal `wrapper`
function - this will essentially replace the `wrapper` function metdata with
the metadata from our `func` function. So our `add2` name and documentation is
now preserved:
%% Cell type:code id: tags:
```
print('Name: ', add2.__name__)
print('Help: ', add2.__doc__)
```
%% Cell type:markdown id: tags:
<a class="anchor" id="appendix-class-decorators"></a>
## Appendix: Class decorators
> Not to be confused with [_decorator classes_](#decorator-classes)!
> Not to be confused with [*decorator classes*](#decorator-classes)!
In this practical, we have shown how decorators can be applied to functions
and methods. But decorators can in fact also be applied to _classes_. This is
and methods. But decorators can in fact also be applied to *classes*. This is
a fairly niche feature that you are probably not likely to need, so we will
only cover it briefly.
Imagine that we want all objects in our application to have a globally unique
(within the application) identifier. We could use a decorator which contains
the logic for generating unique IDs, and defines the interface that we can
use on an instance to obtain its ID:
%% Cell type:code id: tags:
```
import random
allIds = set()
def uniqueID(cls):
class subclass(cls):
def getUniqueID(self):
uid = getattr(self, '_uid', None)
if uid is not None:
return uid
while uid is None or uid in set():
uid = random.randint(1, 100)
self._uid = uid
return uid
return subclass
```
%% Cell type:markdown id: tags:
Now we can use the `@uniqueID` decorator on any class that we need to
have a unique ID:
%% Cell type:code id: tags:
```
@uniqueID
class Foo(object):
pass
@uniqueID
class Bar(object):
pass
```
%% Cell type:markdown id: tags:
All instances of these classes will have a `getUniqueID` method:
%% Cell type:code id: tags:
```
f1 = Foo()
f2 = Foo()
b1 = Bar()
b2 = Bar()
print('f1: ', f1.getUniqueID())
print('f2: ', f2.getUniqueID())
print('b1: ', b1.getUniqueID())
print('b2: ', b2.getUniqueID())
```
%% Cell type:markdown id: tags:
<a class="anchor" id="useful-references"></a>
## Useful references
* [Understanding decorators in 12 easy steps](http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/)
* [The decorators they won't tell you about](https://github.com/hchasestevens/hchasestevens.github.io/blob/master/notebooks/the-decorators-they-wont-tell-you-about.ipynb)
* [Closures - Wikipedia][wiki-closure]
* [Closures in Python](https://www.geeksforgeeks.org/python-closures/)
* [Garbage collection in Python](https://www.quora.com/How-does-garbage-collection-in-Python-work-What-are-the-pros-and-cons)
[wiki-closure]: https://en.wikipedia.org/wiki/Closure_(computer_programming)
......
advanced_topics/decorators.md → advanced_topics/06_decorators.md
View file @ 15d75bb5
......@@ -100,7 +100,7 @@ This new `timeFunc` function is again passed a function `func`, but this time
as its sole argument. It then creates and returns a new function,
`wrapperFunc`. This `wrapperFunc` function calls and times the function that
was passed to `timeFunc`. But note that when `timeFunc` is called,
`wrapperFunc` is _not_ called - it is only created and returned.
`wrapperFunc` is *not* called - it is only created and returned.
Let's use our new `timeFunc` implementation:
......@@ -151,9 +151,9 @@ which holds a reference to the original definition of `inverse`.
> [functions are not special](#appendix-functions-are-not-special).
Guess what? We have just created a __decorator__. A decorator is simply a
Guess what? We have just created a **decorator**. A decorator is simply a
function which accepts a function as its input, and returns another function
as its output. In the example above, we have _decorated_ the `inverse`
as its output. In the example above, we have *decorated* the `inverse`
function with the `timeFunc` decorator.
......@@ -228,7 +228,7 @@ MiscMaths.inverse = timeFunc(MiscMaths.inverse)
```
So only one `wrapperFunc` function exists, and this function is _shared_ by
So only one `wrapperFunc` function exists, and this function is *shared* by
all instances of the `MiscMaths` class - (such as the `mm1` and `mm2`
instances in the example above). In many cases this is not a problem, but
there can be situations where you need each instance of your class to have its
......@@ -292,11 +292,11 @@ def fib(n):
if n in (0, 1):
print('fib({}) = {}'.format(n, n))
return 1
return n
twoback = 1
oneback = 1
val = 0
val = 1
for _ in range(2, n):
......@@ -383,8 +383,10 @@ def limitedMemoize(maxSize):
# remove the oldest item. In practice
# it would make more sense to remove
# the item with the oldest access
# time, but this is good enough for
# an introduction!
# time (or remove the least recently
# used item, as the built-in
# @functools.lru_cache does), but this
# is good enough for now!
if len(cache) >= maxSize:
cache.popitem(last=False)
......@@ -398,11 +400,11 @@ def limitedMemoize(maxSize):
```
> We used the handy
> [`collections.OrderedDict`](https://docs.python.org/3.5/library/collections.html#collections.OrderedDict)
> [`collections.OrderedDict`](https://docs.python.org/3/library/collections.html#collections.OrderedDict)
> class here which preserves the insertion order of key-value pairs.
This is starting to look a little complicated - we now have _three_ layers of
This is starting to look a little complicated - we now have *three* layers of
functions. This is necessary when you wish to write a decorator which accepts
arguments (refer to the
[appendix](#appendix-decorators-without-arguments-versus-decorators-with-arguments)
......@@ -417,13 +419,13 @@ earlier `memoize` decorator:
@limitedMemoize(5)
def fib(n):
if n in (1, 2):
if n in (0, 1):
print('fib({}) = 1'.format(n))
return 1
return n
twoback = 1
oneback = 1
val = 0
val = 1
for _ in range(2, n):
......@@ -493,12 +495,17 @@ expensiveFunc(1)
```
> Note that in Python 3.2 and newer you can use the
> [`functools.lru_cache`](https://docs.python.org/3/library/functools.html#functools.lru_cache)
> to memoize your functions.
<a class="anchor" id="decorator-classes"></a>
## Decorator classes
By now, you will have gained the impression that a decorator is a function
which _decorates_ another function. But if you went through the practical on
which *decorates* another function. But if you went through the practical on
operator overloading, you might remember the special `__call__` method, that
allows an object to be called as if it were a function.
......@@ -589,17 +596,17 @@ registry.runTests()
```
> Unit testing is something which you must do! This is __especially__
> Unit testing is something which you must do! This is **especially**
> important in an interpreted language such as Python, where there is no
> compiler to catch all of your mistakes.
>
> Python has a built-in
> [`unittest`](https://docs.python.org/3.5/library/unittest.html) module,
> [`unittest`](https://docs.python.org/3/library/unittest.html) module,
> however the third-party [`pytest`](https://docs.pytest.org/en/latest/) and
> [`nose`](http://nose2.readthedocs.io/en/latest/) are popular. It is also
> wise to combine your unit tests with
> [`coverage`](https://coverage.readthedocs.io/en/coverage-4.5.1/), which
> tells you how much of your code was executed, or _covered_ when your
> tells you how much of your code was executed, or *covered* when your
> tests were run.
......@@ -706,7 +713,7 @@ as we like.
> If it bothers you that `print(inv2)` resulted in
> `<function inverse at ...>`, and not `<function inv2 at ...>`, then refer to
> the appendix on
> [preserving function metdata](#appendix-preserving-function-metadata).
> [preserving function metadata](#appendix-preserving-function-metadata).
<a class="anchor" id="appendix-closures"></a>
......@@ -714,7 +721,7 @@ as we like.
Whenever we define or use a decorator, we are taking advantage of a concept
called a [_closure_][wiki-closure]. Take a second to re-familiarise yourself
called a [*closure*][wiki-closure]. Take a second to re-familiarise yourself
with our `memoize` decorator function from earlier - when `memoize` is called,
it creates and returns a function called `wrapper`:
......@@ -776,7 +783,7 @@ finished.
This is what is known as a
[_closure_](https://www.geeksforgeeks.org/python-closures/). Closures are a
[*closure*](https://www.geeksforgeeks.org/python-closures/). Closures are a
fundamental, and extremely powerful, aspect of Python and other high level
languages. So there's your answer,
[fishbulb](https://www.youtube.com/watch?v=CiAaEPcnlOg).
......@@ -827,7 +834,7 @@ function.
But if a decorator function is "called" (scenarios 2 or 3), both the decorator
function (`decorator`), __and its return value__ (`wrapper`) are called - the
function (`decorator`), **and its return value** (`wrapper`) are called - the
decorator function is passed the arguments that were provided, and its return
value is passed the decorated function.
......@@ -959,7 +966,7 @@ function, so that the `wrapper` ignores the first argument. But this would
mean that we couldn't use `ensureNumeric` with standalone functions.
But we _can_ manually apply the `ensureNumeric` decorator to `MiscMaths`
But we *can* manually apply the `ensureNumeric` decorator to `MiscMaths`
instances when they are initialised. We can't use the nice `@ensureNumeric`
syntax to apply our decorators, but this is a viable approach:
......@@ -981,7 +988,7 @@ print(mm.add('5', 10))
Another approach is to use a second decorator, which dynamically creates the
real decorator when it is accessed on an instance. This requires the use of an
advanced Python technique called
[_descriptors_](https://docs.python.org/3.5/howto/descriptor.html), which is
[*descriptors*](https://docs.python.org/3/howto/descriptor.html), which is
beyond the scope of this practical. But if you are interested, you can see an
implementation of this approach
[here](https://git.fmrib.ox.ac.uk/fsl/fslpy/blob/1.6.8/fsl/utils/memoize.py#L249).
......@@ -1046,7 +1053,7 @@ documentation](http://www.sphinx-doc.org/) for our code.
Fortunately, there is a workaround, available in the built-in
[`functools`](https://docs.python.org/3.5/library/functools.html#functools.wraps)
[`functools`](https://docs.python.org/3/library/functools.html#functools.wraps)
module:
......@@ -1084,11 +1091,11 @@ print('Help: ', add2.__doc__)
## Appendix: Class decorators
> Not to be confused with [_decorator classes_](#decorator-classes)!
> Not to be confused with [*decorator classes*](#decorator-classes)!
In this practical, we have shown how decorators can be applied to functions
and methods. But decorators can in fact also be applied to _classes_. This is
and methods. But decorators can in fact also be applied to *classes*. This is
a fairly niche feature that you are probably not likely to need, so we will
only cover it briefly.
......
advanced_topics/07_threading.ipynb 0 → 100644
View file @ 15d75bb5
This diff is collapsed.
advanced_topics/07_threading.md 0 → 100644
View file @ 15d75bb5
This diff is collapsed.
advanced_topics/08_fslpy.ipynb 0 → 100644
View file @ 15d75bb5
This diff is collapsed.
advanced_topics/08_fslpy.md 0 → 100644
View file @ 15d75bb5
This diff is collapsed.
advanced_topics/08_fslpy/bighead.nii.gz 0 → 100755
View file @ 15d75bb5
File added
advanced_topics/08_fslpy/fmri.feat/reg/example_func.nii.gz 0 → 100755
View file @ 15d75bb5
File added
advanced_topics/08_fslpy/fmri.feat/reg/example_func2highres.mat 0 → 100644
View file @ 15d75bb5
0.99895165 0.0442250787 -0.01181694611 6.534548061
-0.0439203422 0.9987243849 0.02491016319 9.692178016
0.01290352651 -0.02436503913 0.9996197946 21.90296924
0 0 0 1
advanced_topics/08_fslpy/fmri.feat/reg/example_func2standard.mat 0 → 100755
View file @ 15d75bb5
1.057622308 0.05073972589 0.008769375553 -31.51452272
-0.0541050194 0.9680196522 0.1445326796 2.872941273
-0.01020603009 -0.2324151706 1.127557283 21.35031106
0 0 0 1
advanced_topics/08_fslpy/fmri.feat/reg/highres.nii.gz 0 → 100644
View file @ 15d75bb5
File added
advanced_topics/08_fslpy/fmri.feat/reg/highres2standard.mat 0 → 100644
View file @ 15d75bb5
1.056802026 -0.01924547726 0.02614687181 -36.51723948
0.009055463297 0.9745460053 0.09056277052 -8.771603455
-0.04315832679 -0.1680837227 1.136420957 -1.399839791
0 0 0 1
advanced_topics/08_fslpy/fmri.feat/reg/highres2standard_warp.nii.gz 0 → 100644
View file @ 15d75bb5
File added
advanced_topics/08_fslpy/fmri.feat/stats/tstat1.nii.gz 0 → 100755
View file @ 15d75bb5
File added
advanced_topics/09_structuring_projects.ipynb 0 → 100644
View file @ 15d75bb5
This diff is collapsed.
advanced_topics/09_structuring_projects.md 0 → 100644
View file @ 15d75bb5
This diff is collapsed.