Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • fsl/win-pytreat
  • ndcn0236/win-pytreat
  • shahdloo/win-pytreat
  • hossein/win-pytreat
  • pfr545/win-pytreat
5 results
Show changes
Commits on Source (11)
Showing
with 775 additions and 827 deletions
...@@ -17,7 +17,7 @@ contains the following: ...@@ -17,7 +17,7 @@ contains the following:
on using Python to accomplish specific tasks, and practicals which introduce on using Python to accomplish specific tasks, and practicals which introduce
a range of useful Python-based libraries. a range of useful Python-based libraries.
The practicals have been written under the assumption that FSL 6.0.4 is The practicals have been written under the assumption that FSL 6.0.7 or newer is
installed. installed.
...@@ -185,7 +185,7 @@ You may wish to install [`notedown`](https://github.com/aaren/notedown): ...@@ -185,7 +185,7 @@ You may wish to install [`notedown`](https://github.com/aaren/notedown):
``` ```
$FSLDIR/fslpython/bin/conda install -n fslpython -c conda-forge notedown $FSLDIR/fslpython/bin/conda install -n fslpython -c conda-forge notedown
ln -s $FSLDIR/fslpython/envs/fslpython/bin/notedown $FSLDIR/bin/fslnotedown ln -s $FSLDIR/bin/notedown $FSLDIR/share/fsl/bin/fslnotedown
``` ```
`notedown` is a handy tool which allows you to convert a markdown (`.md`) file `notedown` is a handy tool which allows you to convert a markdown (`.md`) file
......
%% Cell type:markdown id: tags: %% Cell type:markdown id:d1a921f7 tags:
# Decorators # Decorators
Remember that in Python, everything is an object, including functions. This Remember that in Python, everything is an object, including functions. This
means that we can do things like: means that we can do things like:
- Pass a function as an argument to another function. - Pass a function as an argument to another function.
- Create/define a function inside another function. - Create/define a function inside another function.
- Write a function which returns another function. - Write a function which returns another function.
These abilities mean that we can do some neat things with functions in Python. These abilities mean that we can do some neat things with functions in Python.
* [Overview](#overview) * [Overview](#overview)
* [Decorators on methods](#decorators-on-methods) * [Decorators on methods](#decorators-on-methods)
* [Example - memoization](#example-memoization) * [Example - memoization](#example-memoization)
* [Decorators with arguments](#decorators-with-arguments) * [Decorators with arguments](#decorators-with-arguments)
* [Chaining decorators](#chaining-decorators) * [Chaining decorators](#chaining-decorators)
* [Decorator classes](#decorator-classes) * [Decorator classes](#decorator-classes)
* [Appendix: Functions are not special](#appendix-functions-are-not-special) * [Appendix: Functions are not special](#appendix-functions-are-not-special)
* [Appendix: Closures](#appendix-closures) * [Appendix: Closures](#appendix-closures)
* [Appendix: Decorators without arguments versus decorators with arguments](#appendix-decorators-without-arguments-versus-decorators-with-arguments) * [Appendix: Decorators without arguments versus decorators with arguments](#appendix-decorators-without-arguments-versus-decorators-with-arguments)
* [Appendix: Per-instance decorators](#appendix-per-instance-decorators) * [Appendix: Per-instance decorators](#appendix-per-instance-decorators)
* [Appendix: Preserving function metadata](#appendix-preserving-function-metadata) * [Appendix: Preserving function metadata](#appendix-preserving-function-metadata)
* [Appendix: Class decorators](#appendix-class-decorators) * [Appendix: Class decorators](#appendix-class-decorators)
* [Useful references](#useful-references) * [Useful references](#useful-references)
<a class="anchor" id="overview"></a> <a class="anchor" id="overview"></a>
## Overview ## Overview
Let's say that we want a way to calculate the execution time of any function Let's say that we want a way to calculate the execution time of any function
(this example might feel familiar to you if you have gone through the (this example might feel familiar to you if you have gone through the
practical on operator overloading). practical on operator overloading).
Our first attempt at writing such a function might look like this: Our first attempt at writing such a function might look like this:
%% Cell type:code id: tags: %% Cell type:code id:c29789da tags:
``` ```
import time import time
def timeFunc(func, *args, **kwargs): def timeFunc(func, *args, **kwargs):
start = time.time() start = time.time()
retval = func(*args, **kwargs) retval = func(*args, **kwargs)
end = time.time() end = time.time()
print('Ran {} in {:0.2f} seconds'.format(func.__name__, end - start)) print('Ran {} in {:0.2f} seconds'.format(func.__name__, end - start))
return retval return retval
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:78c0c022 tags:
The `timeFunc` function accepts another function, `func`, as its first The `timeFunc` function accepts another function, `func`, as its first
argument. It calls `func`, passing it all of the other arguments, and then argument. It calls `func`, passing it all of the other arguments, and then
prints the time taken for `func` to complete: prints the time taken for `func` to complete:
%% Cell type:code id: tags: %% Cell type:code id:b0c736bb tags:
``` ```
import numpy as np import numpy as np
import numpy.linalg as npla import numpy.linalg as npla
def inverse(a): def inverse(a):
return npla.inv(a) return npla.inv(a)
data = np.random.random((2000, 2000)) data = np.random.random((2000, 2000))
invdata = timeFunc(inverse, data) invdata = timeFunc(inverse, data)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:6049b322 tags:
But this means that whenever we want to time something, we have to call the But this means that whenever we want to time something, we have to call the
`timeFunc` function directly. Let's take advantage of the fact that we can `timeFunc` function directly. Let's take advantage of the fact that we can
define a function inside another funciton. Look at the next block of code define a function inside another funciton. Look at the next block of code
carefully, and make sure you understand what our new `timeFunc` implementation carefully, and make sure you understand what our new `timeFunc` implementation
is doing. is doing.
%% Cell type:code id: tags: %% Cell type:code id:e919f62a tags:
``` ```
import time import time
def timeFunc(func): def timeFunc(func):
def wrapperFunc(*args, **kwargs): def wrapperFunc(*args, **kwargs):
start = time.time() start = time.time()
retval = func(*args, **kwargs) retval = func(*args, **kwargs)
end = time.time() end = time.time()
print('Ran {} in {:0.2f} seconds'.format(func.__name__, end - start)) print('Ran {} in {:0.2f} seconds'.format(func.__name__, end - start))
return retval return retval
return wrapperFunc return wrapperFunc
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:9b99cf0e tags:
This new `timeFunc` function is again passed a function `func`, but this time This new `timeFunc` function is again passed a function `func`, but this time
as its sole argument. It then creates and returns a new function, as its sole argument. It then creates and returns a new function,
`wrapperFunc`. This `wrapperFunc` function calls and times the function that `wrapperFunc`. This `wrapperFunc` function calls and times the function that
was passed to `timeFunc`. But note that when `timeFunc` is called, was passed to `timeFunc`. But note that when `timeFunc` is called,
`wrapperFunc` is *not* called - it is only created and returned. `wrapperFunc` is *not* called - it is only created and returned.
Let's use our new `timeFunc` implementation: Let's use our new `timeFunc` implementation:
%% Cell type:code id: tags: %% Cell type:code id:24abfd32 tags:
``` ```
import numpy as np import numpy as np
import numpy.linalg as npla import numpy.linalg as npla
def inverse(a): def inverse(a):
return npla.inv(a) return npla.inv(a)
data = np.random.random((2000, 2000)) data = np.random.random((2000, 2000))
inverse = timeFunc(inverse) inverse = timeFunc(inverse)
invdata = inverse(data) invdata = inverse(data)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:47a9062a tags:
Here, we did the following: Here, we did the following:
1. We defined a function called `inverse`: 1. We defined a function called `inverse`:
> ``` > ```
> def inverse(a): > def inverse(a):
> return npla.inv(a) > return npla.inv(a)
> ``` > ```
2. We passed the `inverse` function to the `timeFunc` function, and 2. We passed the `inverse` function to the `timeFunc` function, and
re-assigned the return value of `timeFunc` back to `inverse`: re-assigned the return value of `timeFunc` back to `inverse`:
> ``` > ```
> inverse = timeFunc(inverse) > inverse = timeFunc(inverse)
> ``` > ```
3. We called the new `inverse` function: 3. We called the new `inverse` function:
> ``` > ```
> invdata = inverse(data) > invdata = inverse(data)
> ``` > ```
So now the `inverse` variable refers to an instantiation of `wrapperFunc`, So now the `inverse` variable refers to an instantiation of `wrapperFunc`,
which holds a reference to the original definition of `inverse`. which holds a reference to the original definition of `inverse`.
> If this is not clear, take a break now and read through the appendix on how > If this is not clear, take a break now and read through the appendix on how
> [functions are not special](#appendix-functions-are-not-special). > [functions are not special](#appendix-functions-are-not-special).
Guess what? We have just created a **decorator**. A decorator is simply a Guess what? We have just created a **decorator**. A decorator is simply a
function which accepts a function as its input, and returns another function function which accepts a function as its input, and returns another function
as its output. In the example above, we have *decorated* the `inverse` as its output. In the example above, we have *decorated* the `inverse`
function with the `timeFunc` decorator. function with the `timeFunc` decorator.
Python provides an alternative syntax for decorating one function with Python provides an alternative syntax for decorating one function with
another, using the `@` character. The approach that we used to decorate another, using the `@` character. The approach that we used to decorate
`inverse` above: `inverse` above:
%% Cell type:code id: tags: %% Cell type:code id:b9f7e6ed tags:
``` ```
def inverse(a): def inverse(a):
return npla.inv(a) return npla.inv(a)
inverse = timeFunc(inverse) inverse = timeFunc(inverse)
invdata = inverse(data) invdata = inverse(data)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:53862dd0 tags:
is semantically equivalent to this: is semantically equivalent to this:
%% Cell type:code id: tags: %% Cell type:code id:ba2fcdf2 tags:
``` ```
@timeFunc @timeFunc
def inverse(a): def inverse(a):
return npla.inv(a) return npla.inv(a)
invdata = inverse(data) invdata = inverse(data)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:9e929f1e tags:
<a class="anchor" id="decorators-on-methods"></a> <a class="anchor" id="decorators-on-methods"></a>
## Decorators on methods ## Decorators on methods
Applying a decorator to the methods of a class works in the same way: Applying a decorator to the methods of a class works in the same way:
%% Cell type:code id: tags: %% Cell type:code id:9f6d03a9 tags:
``` ```
import numpy.linalg as npla import numpy.linalg as npla
class MiscMaths(object): class MiscMaths(object):
@timeFunc @timeFunc
def inverse(self, a): def inverse(self, a):
return npla.inv(a) return npla.inv(a)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:8d0db883 tags:
Now, the `inverse` method of all `MiscMaths` instances will be timed: Now, the `inverse` method of all `MiscMaths` instances will be timed:
%% Cell type:code id: tags: %% Cell type:code id:67aa37bd tags:
``` ```
mm1 = MiscMaths() mm1 = MiscMaths()
mm2 = MiscMaths() mm2 = MiscMaths()
i1 = mm1.inverse(np.random.random((1000, 1000))) i1 = mm1.inverse(np.random.random((1000, 1000)))
i2 = mm2.inverse(np.random.random((1500, 1500))) i2 = mm2.inverse(np.random.random((1500, 1500)))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:b4f3df41 tags:
Note that only one `timeFunc` decorator was created here - the `timeFunc` Note that only one `timeFunc` decorator was created here - the `timeFunc`
function was only called once - when the `MiscMaths` class was defined. This function was only called once - when the `MiscMaths` class was defined. This
might be clearer if we re-write the above code in the following (equivalent) might be clearer if we re-write the above code in the following (equivalent)
manner: manner:
%% Cell type:code id: tags: %% Cell type:code id:b11e0271 tags:
``` ```
class MiscMaths(object): class MiscMaths(object):
def inverse(self, a): def inverse(self, a):
return npla.inv(a) return npla.inv(a)
MiscMaths.inverse = timeFunc(MiscMaths.inverse) MiscMaths.inverse = timeFunc(MiscMaths.inverse)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:f8b882d1 tags:
So only one `wrapperFunc` function exists, and this function is *shared* by So only one `wrapperFunc` function exists, and this function is *shared* by
all instances of the `MiscMaths` class - (such as the `mm1` and `mm2` all instances of the `MiscMaths` class - (such as the `mm1` and `mm2`
instances in the example above). In many cases this is not a problem, but instances in the example above). In many cases this is not a problem, but
there can be situations where you need each instance of your class to have its there can be situations where you need each instance of your class to have its
own unique decorator. own unique decorator.
> If you are interested in solutions to this problem, take a look at the > If you are interested in solutions to this problem, take a look at the
> appendix on [per-instance decorators](#appendix-per-instance-decorators). > appendix on [per-instance decorators](#appendix-per-instance-decorators).
<a class="anchor" id="example-memoization"></a> <a class="anchor" id="example-memoization"></a>
## Example - memoization ## Example - memoization
Let's move onto another example. Let's move onto another example.
[Meowmoization](https://en.wikipedia.org/wiki/Memoization) is a common [Meowmoization](https://en.wikipedia.org/wiki/Memoization) is a common
performance optimisation technique used in cats. I mean software. Essentially, performance optimisation technique used in cats. I mean software. Essentially,
memoization refers to the process of maintaining a cache for a function which memoization refers to the process of maintaining a cache for a function which
performs some expensive calculation. When the function is executed with a set performs some expensive calculation. When the function is executed with a set
of inputs, the calculation is performed, and then a copy of the inputs and the of inputs, the calculation is performed, and then a copy of the inputs and the
result are cached. If the function is called again with the same inputs, the result are cached. If the function is called again with the same inputs, the
cached result can be returned. cached result can be returned.
This is a perfect problem to tackle with decorators: This is a perfect problem to tackle with decorators:
%% Cell type:code id: tags: %% Cell type:code id:b1901255 tags:
``` ```
def memoize(func): def memoize(func):
cache = {} cache = {}
def wrapper(*args): def wrapper(*args):
# is there a value in the cache # is there a value in the cache
# for this set of inputs? # for this set of inputs?
cached = cache.get(args, None) cached = cache.get(args, None)
# If not, call the function, # If not, call the function,
# and cache the result. # and cache the result.
if cached is None: if cached is None:
cached = func(*args) cached = func(*args)
cache[args] = cached cache[args] = cached
else: else:
print(f'Cached {func.__name__}({args}): {cached}') print(f'Cached {func.__name__}({args}): {cached}')
return cached return cached
return wrapper return wrapper
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:a903f1fb tags:
We can now use our `memoize` decorator to add a memoization cache to any We can now use our `memoize` decorator to add a memoization cache to any
function. Let's memoize a function which generates the $n^{th}$ number in the function. Let's memoize a function which generates the $n^{th}$ number in the
[Fibonacci series](https://en.wikipedia.org/wiki/Fibonacci_number): [Fibonacci series](https://en.wikipedia.org/wiki/Fibonacci_number):
%% Cell type:code id: tags: %% Cell type:code id:9e3c6654 tags:
``` ```
@memoize @memoize
def fib(n): def fib(n):
if n in (0, 1): if n in (0, 1):
print(f'fib({n}) = {n}') print(f'fib({n}) = {n}')
return n return n
twoback = 1 twoback = 1
oneback = 1 oneback = 1
val = 1 val = 1
for _ in range(2, n): for _ in range(2, n):
val = oneback + twoback val = oneback + twoback
twoback = oneback twoback = oneback
oneback = val oneback = val
print(f'fib({n}) = {val}') print(f'fib({n}) = {val}')
return val return val
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:68cdecaf tags:
For a given input, when `fib` is called the first time, it will calculate the For a given input, when `fib` is called the first time, it will calculate the
$n^{th}$ Fibonacci number: $n^{th}$ Fibonacci number:
%% Cell type:code id: tags: %% Cell type:code id:3f9165e8 tags:
``` ```
for i in range(10): for i in range(10):
fib(i) fib(i)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:5754fbd3 tags:
However, on repeated calls with the same input, the calculation is skipped, However, on repeated calls with the same input, the calculation is skipped,
and instead the result is retrieved from the memoization cache: and instead the result is retrieved from the memoization cache:
%% Cell type:code id: tags: %% Cell type:code id:a37f4fe9 tags:
``` ```
for i in range(10): for i in range(10):
fib(i) fib(i)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:68471ad2 tags:
> If you are wondering how the `wrapper` function is able to access the > If you are wondering how the `wrapper` function is able to access the
> `cache` variable, refer to the [appendix on closures](#appendix-closures). > `cache` variable, refer to the [appendix on closures](#appendix-closures).
<a class="anchor" id="decorators-with-arguments"></a> <a class="anchor" id="decorators-with-arguments"></a>
## Decorators with arguments ## Decorators with arguments
Continuing with our memoization example, let's say that we want to place a Continuing with our memoization example, let's say that we want to place a
limit on the maximum size that our cache can grow to. For example, the output limit on the maximum size that our cache can grow to. For example, the output
of our function might have large memory requirements, so we can only afford to of our function might have large memory requirements, so we can only afford to
store a handful of pre-calculated results. It would be nice to be able to store a handful of pre-calculated results. It would be nice to be able to
specify the maximum cache size when we define our function to be memoized, specify the maximum cache size when we define our function to be memoized,
like so: like so:
> ``` > ```
> # cache at most 10 results > # cache at most 10 results
> @limitedMemoize(10): > @limitedMemoize(10):
> def fib(n): > def fib(n):
> ... > ...
> ``` > ```
In order to support this, our `memoize` decorator function needs to be In order to support this, our `memoize` decorator function needs to be
modified - it is currently written to accept a function as its sole argument, modified - it is currently written to accept a function as its sole argument,
but we need it to accept a cache size limit. but we need it to accept a cache size limit.
%% Cell type:code id: tags: %% Cell type:code id:b0164df8 tags:
``` ```
from collections import OrderedDict from collections import OrderedDict
def limitedMemoize(maxSize): def limitedMemoize(maxSize):
cache = OrderedDict() cache = OrderedDict()
def decorator(func): def decorator(func):
def wrapper(*args): def wrapper(*args):
# is there a value in the cache # is there a value in the cache
# for this set of inputs? # for this set of inputs?
cached = cache.get(args, None) cached = cache.get(args, None)
# If not, call the function, # If not, call the function,
# and cache the result. # and cache the result.
if cached is None: if cached is None:
cached = func(*args) cached = func(*args)
# If the cache has grown too big, # If the cache has grown too big,
# remove the oldest item. In practice # remove the oldest item. In practice
# it would make more sense to remove # it would make more sense to remove
# the item with the oldest access # the item with the oldest access
# time (or remove the least recently # time (or remove the least recently
# used item, as the built-in # used item, as the built-in
# @functools.lru_cache does), but this # @functools.lru_cache does), but this
# is good enough for now! # is good enough for now!
if len(cache) >= maxSize: if len(cache) >= maxSize:
cache.popitem(last=False) cache.popitem(last=False)
cache[args] = cached cache[args] = cached
else: else:
print(f'Cached {func.__name__}({args}): {cached}') print(f'Cached {func.__name__}({args}): {cached}')
return cached return cached
return wrapper return wrapper
return decorator return decorator
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:15ff1d27 tags:
> We used the handy > We used the handy
> [`collections.OrderedDict`](https://docs.python.org/3/library/collections.html#collections.OrderedDict) > [`collections.OrderedDict`](https://docs.python.org/3/library/collections.html#collections.OrderedDict)
> class here which preserves the insertion order of key-value pairs. > class here which preserves the insertion order of key-value pairs.
This is starting to look a little complicated - we now have *three* layers of This is starting to look a little complicated - we now have *three* layers of
functions. This is necessary when you wish to write a decorator which accepts functions. This is necessary when you wish to write a decorator which accepts
arguments (refer to the arguments (refer to the
[appendix](#appendix-decorators-without-arguments-versus-decorators-with-arguments) [appendix](#appendix-decorators-without-arguments-versus-decorators-with-arguments)
for more details). for more details).
But this `limitedMemoize` decorator is used in essentially the same way as our But this `limitedMemoize` decorator is used in essentially the same way as our
earlier `memoize` decorator: earlier `memoize` decorator:
%% Cell type:code id: tags: %% Cell type:code id:665fd002 tags:
``` ```
@limitedMemoize(5) @limitedMemoize(5)
def fib(n): def fib(n):
if n in (0, 1): if n in (0, 1):
print(f'fib({n}) = 1') print(f'fib({n}) = 1')
return n return n
twoback = 1 twoback = 1
oneback = 1 oneback = 1
val = 1 val = 1
for _ in range(2, n): for _ in range(2, n):
val = oneback + twoback val = oneback + twoback
twoback = oneback twoback = oneback
oneback = val oneback = val
print(f'fib({n}) = {val}') print(f'fib({n}) = {val}')
return val return val
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:483f1117 tags:
Except that now, the `fib` function will only cache up to 5 values. Except that now, the `fib` function will only cache up to 5 values.
%% Cell type:code id: tags: %% Cell type:code id:efb2da47 tags:
``` ```
fib(10) fib(10)
fib(11) fib(11)
fib(12) fib(12)
fib(13) fib(13)
fib(14) fib(14)
print('The result for 10 should come from the cache') print('The result for 10 should come from the cache')
fib(10) fib(10)
fib(15) fib(15)
print('The result for 10 should no longer be cached') print('The result for 10 should no longer be cached')
fib(10) fib(10)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:d5e05bd3 tags:
<a class="anchor" id="chaining-decorators"></a> <a class="anchor" id="chaining-decorators"></a>
## Chaining decorators ## Chaining decorators
Decorators can easily be chained, or nested: Decorators can easily be chained, or nested:
%% Cell type:code id: tags: %% Cell type:code id:4ddeb6e0 tags:
``` ```
import time import time
@timeFunc @timeFunc
@memoize @memoize
def expensiveFunc(n): def expensiveFunc(n):
time.sleep(n) time.sleep(n)
return n return n
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:d1e30867 tags:
> Remember that this is semantically equivalent to the following: > Remember that this is semantically equivalent to the following:
> >
> ``` > ```
> def expensiveFunc(n): > def expensiveFunc(n):
> time.sleep(n) > time.sleep(n)
> return n > return n
> >
> expensiveFunc = timeFunc(memoize(expensiveFunc)) > expensiveFunc = timeFunc(memoize(expensiveFunc))
> ``` > ```
Now we can see the effect of our memoization layer on performance: Now we can see the effect of our memoization layer on performance:
%% Cell type:code id: tags: %% Cell type:code id:6c73a284 tags:
``` ```
expensiveFunc(0.5) expensiveFunc(0.5)
expensiveFunc(1) expensiveFunc(1)
expensiveFunc(1) expensiveFunc(1)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:41ec6be0 tags:
> Note that in Python 3.2 and newer you can use the > Note that in Python 3.2 and newer you can use the
> [`functools.lru_cache`](https://docs.python.org/3/library/functools.html#functools.lru_cache) > [`functools.lru_cache`](https://docs.python.org/3/library/functools.html#functools.lru_cache)
> to memoize your functions. > to memoize your functions.
<a class="anchor" id="decorator-classes"></a> <a class="anchor" id="decorator-classes"></a>
## Decorator classes ## Decorator classes
By now, you will have gained the impression that a decorator is a function By now, you will have gained the impression that a decorator is a function
which *decorates* another function. But if you went through the practical on which *decorates* another function. But if you went through the practical on
operator overloading, you might remember the special `__call__` method, that operator overloading, you might remember the special `__call__` method, that
allows an object to be called as if it were a function. allows an object to be called as if it were a function.
This feature allows us to write our decorators as classes, instead of This feature allows us to write our decorators as classes, instead of
functions. This can be handy if you are writing a decorator that has functions. This can be handy if you are writing a decorator that has
complicated behaviour, and/or needs to maintain some sort of state which complicated behaviour, and/or needs to maintain some sort of state which
cannot be easily or elegantly written using nested functions. cannot be easily or elegantly written using nested functions.
As an example, let's say we are writing a framework for unit testing. We want As an example, let's say we are writing a framework for unit testing. We want
to be able to "mark" our test functions like so, so they can be easily to be able to "mark" our test functions like so, so they can be easily
identified and executed: identified and executed:
> ``` > ```
> @unitTest > @unitTest
> def testblerk(): > def testblerk():
> """tests the blerk algorithm.""" > """tests the blerk algorithm."""
> ... > ...
> ``` > ```
With a decorator like this, we wouldn't need to worry about where our tests With a decorator like this, we wouldn't need to worry about where our tests
are located - they will all be detected because we have marked them as test are located - they will all be detected because we have marked them as test
functions. What does this `unitTest` decorator look like? functions. What does this `unitTest` decorator look like?
%% Cell type:code id: tags: %% Cell type:code id:154e7ece tags:
``` ```
class TestRegistry(object): class TestRegistry(object):
def __init__(self): def __init__(self):
self.testFuncs = [] self.testFuncs = []
def __call__(self, func): def __call__(self, func):
self.testFuncs.append(func) self.testFuncs.append(func)
def listTests(self): def listTests(self):
print('All registered tests:') print('All registered tests:')
for test in self.testFuncs: for test in self.testFuncs:
print(' ', test.__name__) print(' ', test.__name__)
def runTests(self): def runTests(self):
for test in self.testFuncs: for test in self.testFuncs:
print(f'Running test {test.__name__:10s} ... ', end='') print(f'Running test {test.__name__:10s} ... ', end='')
try: try:
test() test()
print('passed!') print('passed!')
except Exception as e: except Exception as e:
print('failed!') print('failed!')
# Create our test registry # Create our test registry
registry = TestRegistry() registry = TestRegistry()
# Alias our registry to "unitTest" # Alias our registry to "unitTest"
# so that we can register tests # so that we can register tests
# with a "@unitTest" decorator. # with a "@unitTest" decorator.
unitTest = registry unitTest = registry
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:37fc4213 tags:
So we've defined a class, `TestRegistry`, and created an instance of it, So we've defined a class, `TestRegistry`, and created an instance of it,
`registry`, which will manage all of our unit tests. Now, in order to "mark" `registry`, which will manage all of our unit tests. Now, in order to "mark"
any function as being a unit test, we just need to use the `unitTest` any function as being a unit test, we just need to use the `unitTest`
decorator (which is simply a reference to our `TestRegistry` instance): decorator (which is simply a reference to our `TestRegistry` instance):
%% Cell type:code id: tags: %% Cell type:code id:e3fea9b3 tags:
``` ```
@unitTest @unitTest
def testFoo(): def testFoo():
assert 'a' in 'bcde' assert 'a' in 'bcde'
@unitTest @unitTest
def testBar(): def testBar():
assert 1 > 0 assert 1 > 0
@unitTest @unitTest
def testBlerk(): def testBlerk():
assert 9 % 2 == 0 assert 9 % 2 == 0
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:1ce71a27 tags:
Now that these functions have been registered with our `TestRegistry` Now that these functions have been registered with our `TestRegistry`
instance, we can run them all: instance, we can run them all:
%% Cell type:code id: tags: %% Cell type:code id:677727a5 tags:
``` ```
registry.listTests() registry.listTests()
registry.runTests() registry.runTests()
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:556656cf tags:
> Unit testing is something which you must do! This is **especially** > Unit testing is something which you must do! This is **especially**
> important in an interpreted language such as Python, where there is no > important in an interpreted language such as Python, where there is no
> compiler to catch all of your mistakes. > compiler to catch all of your mistakes.
> >
> Python has a built-in > Python has a built-in
> [`unittest`](https://docs.python.org/3/library/unittest.html) module, > [`unittest`](https://docs.python.org/3/library/unittest.html) module,
> however the third-party [`pytest`](https://docs.pytest.org/en/latest/) and > however the third-party [`pytest`](https://docs.pytest.org/en/latest/) and
> [`nose`](http://nose2.readthedocs.io/en/latest/) are popular. It is also > [`nose`](http://nose2.readthedocs.io/en/latest/) are popular. It is also
> wise to combine your unit tests with > wise to combine your unit tests with
> [`coverage`](https://coverage.readthedocs.io/en/coverage-4.5.1/), which > [`coverage`](https://coverage.readthedocs.io/en/coverage-4.5.1/), which
> tells you how much of your code was executed, or *covered* when your > tells you how much of your code was executed, or *covered* when your
> tests were run. > tests were run.
<a class="anchor" id="appendix-functions-are-not-special"></a> <a class="anchor" id="appendix-functions-are-not-special"></a>
## Appendix: Functions are not special ## Appendix: Functions are not special
When we write a statement like this: When we write a statement like this:
%% Cell type:code id: tags: %% Cell type:code id:3da169bf tags:
``` ```
a = [1, 2, 3] a = [1, 2, 3]
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:d7e0a07e tags:
the variable `a` is a reference to a `list`. We can create a new reference to the variable `a` is a reference to a `list`. We can create a new reference to
the same list, and delete `a`: the same list, and delete `a`:
%% Cell type:code id: tags: %% Cell type:code id:51fa6194 tags:
``` ```
b = a b = a
del a del a
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:cd432339 tags:
Deleting `a` doesn't affect the list at all - the list still exists, and is Deleting `a` doesn't affect the list at all - the list still exists, and is
now referred to by a variable called `b`. now referred to by a variable called `b`.
%% Cell type:code id: tags: %% Cell type:code id:11a15463 tags:
``` ```
print('b: ', b) print('b: ', b)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:3a65c6a6 tags:
`a` has, however, been deleted: `a` has, however, been deleted:
%% Cell type:code id: tags: %% Cell type:code id:3238b196 tags:
``` ```
print('a: ', a) print('a: ', a)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:23794988 tags:
The variables `a` and `b` are just references to a list that is sitting in The variables `a` and `b` are just references to a list that is sitting in
memory somewhere - renaming or removing a reference does not have any effect memory somewhere - renaming or removing a reference does not have any effect
upon the list<sup>2</sup>. upon the list<sup>2</sup>.
If you are familiar with C or C++, you can think of a variable in Python as If you are familiar with C or C++, you can think of a variable in Python as
like a `void *` pointer - it is just a pointer of an unspecified type, which like a `void *` pointer - it is just a pointer of an unspecified type, which
is pointing to some item in memory (which does have a specific type). Deleting is pointing to some item in memory (which does have a specific type). Deleting
the pointer does not have any effect upon the item to which it was pointing. the pointer does not have any effect upon the item to which it was pointing.
> <sup>2</sup> Until no more references to the list exist, at which point it > <sup>2</sup> Until no more references to the list exist, at which point it
> will be > will be
> [garbage-collected](https://www.quora.com/How-does-garbage-collection-in-Python-work-What-are-the-pros-and-cons). > [garbage-collected](https://www.quora.com/How-does-garbage-collection-in-Python-work-What-are-the-pros-and-cons).
Now, functions in Python work in _exactly_ the same way as variables. When we Now, functions in Python work in _exactly_ the same way as variables. When we
define a function like this: define a function like this:
%% Cell type:code id: tags: %% Cell type:code id:beda8092 tags:
``` ```
def inverse(a): def inverse(a):
return npla.inv(a) return npla.inv(a)
print(inverse) print(inverse)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:f14ce63c tags:
there is nothing special about the name `inverse` - `inverse` is just a there is nothing special about the name `inverse` - `inverse` is just a
reference to a function that resides somewhere in memory. We can create a new reference to a function that resides somewhere in memory. We can create a new
reference to this function: reference to this function:
%% Cell type:code id: tags: %% Cell type:code id:4e7a2310 tags:
``` ```
inv2 = inverse inv2 = inverse
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:338c1f31 tags:
And delete the old reference: And delete the old reference:
%% Cell type:code id: tags: %% Cell type:code id:ced0239d tags:
``` ```
del inverse del inverse
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:dd29ce2c tags:
But the function still exists, and is still callable, via our second But the function still exists, and is still callable, via our second
reference: reference:
%% Cell type:code id: tags: %% Cell type:code id:a821c9ef tags:
``` ```
print(inv2) print(inv2)
data = np.random.random((10, 10)) data = np.random.random((10, 10))
invdata = inv2(data) invdata = inv2(data)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:aa2884e3 tags:
So there is nothing special about functions in Python - they are just items So there is nothing special about functions in Python - they are just items
that reside somewhere in memory, and to which we can create as many references that reside somewhere in memory, and to which we can create as many references
as we like. as we like.
> If it bothers you that `print(inv2)` resulted in > If it bothers you that `print(inv2)` resulted in
> `<function inverse at ...>`, and not `<function inv2 at ...>`, then refer to > `<function inverse at ...>`, and not `<function inv2 at ...>`, then refer to
> the appendix on > the appendix on
> [preserving function metadata](#appendix-preserving-function-metadata). > [preserving function metadata](#appendix-preserving-function-metadata).
<a class="anchor" id="appendix-closures"></a> <a class="anchor" id="appendix-closures"></a>
## Appendix: Closures ## Appendix: Closures
Whenever we define or use a decorator, we are taking advantage of a concept Whenever we define or use a decorator, we are taking advantage of a concept
called a [*closure*][wiki-closure]. Take a second to re-familiarise yourself called a [*closure*][wiki-closure]. Take a second to re-familiarise yourself
with our `memoize` decorator function from earlier - when `memoize` is called, with our `memoize` decorator function from earlier - when `memoize` is called,
it creates and returns a function called `wrapper`: it creates and returns a function called `wrapper`:
[wiki-closure]: https://en.wikipedia.org/wiki/Closure_(computer_programming) [wiki-closure]: https://en.wikipedia.org/wiki/Closure_(computer_programming)
%% Cell type:code id: tags: %% Cell type:code id:0e649258 tags:
``` ```
def memoize(func): def memoize(func):
cache = {} cache = {}
def wrapper(*args): def wrapper(*args):
# is there a value in the cache # is there a value in the cache
# for this set of inputs? # for this set of inputs?
cached = cache.get(args, None) cached = cache.get(args, None)
# If not, call the function, # If not, call the function,
# and cache the result. # and cache the result.
if cached is None: if cached is None:
cached = func(*args) cached = func(*args)
cache[args] = cached cache[args] = cached
else: else:
print(f'Cached {func.__name__}({args}): {cached}') print(f'Cached {func.__name__}({args}): {cached}')
return cached return cached
return wrapper return wrapper
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:738bf3e9 tags:
Then `wrapper` is executed at some arbitrary point in the future. But how does Then `wrapper` is executed at some arbitrary point in the future. But how does
it have access to `cache`, defined within the scope of the `memoize` function, it have access to `cache`, defined within the scope of the `memoize` function,
after the execution of `memoize` has ended? after the execution of `memoize` has ended?
%% Cell type:code id: tags: %% Cell type:code id:0076b1c2 tags:
``` ```
def nby2(n): def nby2(n):
return n * 2 return n * 2
# wrapper function is created here (and # wrapper function is created here (and
# assigned back to the nby2 reference) # assigned back to the nby2 reference)
nby2 = memoize(nby2) nby2 = memoize(nby2)
# wrapper function is executed here # wrapper function is executed here
print('nby2(2): ', nby2(2)) print('nby2(2): ', nby2(2))
print('nby2(2): ', nby2(2)) print('nby2(2): ', nby2(2))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:83ec0094 tags:
The trick is that whenever a nested function is defined in Python, the scope The trick is that whenever a nested function is defined in Python, the scope
in which it is defined is preserved for that function's lifetime. So `wrapper` in which it is defined is preserved for that function's lifetime. So `wrapper`
has access to all of the variables within the `memoize` function's scope, that has access to all of the variables within the `memoize` function's scope, that
were defined at the time that `wrapper` was created (which was when we called were defined at the time that `wrapper` was created (which was when we called
`memoize`). This is why `wrapper` is able to access `cache`, even though at `memoize`). This is why `wrapper` is able to access `cache`, even though at
the time that `wrapper` is called, the execution of `memoize` has long since the time that `wrapper` is called, the execution of `memoize` has long since
finished. finished.
This is what is known as a This is what is known as a
[*closure*](https://www.geeksforgeeks.org/python-closures/). Closures are a [*closure*](https://www.geeksforgeeks.org/python-closures/). Closures are a
fundamental, and extremely powerful, aspect of Python and other high level fundamental, and extremely powerful, aspect of Python and other high level
languages. So there's your answer, languages. So there's your answer,
[fishbulb](https://www.youtube.com/watch?v=CiAaEPcnlOg). [fishbulb](https://www.youtube.com/watch?v=aso4X_sOjuw).
<a class="anchor" id="appendix-decorators-without-arguments-versus-decorators-with-arguments"></a> <a class="anchor" id="appendix-decorators-without-arguments-versus-decorators-with-arguments"></a>
## Appendix: Decorators without arguments versus decorators with arguments ## Appendix: Decorators without arguments versus decorators with arguments
There are three ways to invoke a decorator with the `@` notation: There are three ways to invoke a decorator with the `@` notation:
1. Naming it, e.g. `@mydecorator` 1. Naming it, e.g. `@mydecorator`
2. Calling it, e.g. `@mydecorator()` 2. Calling it, e.g. `@mydecorator()`
3. Calling it, and passing it arguments, e.g. `@mydecorator(1, 2, 3)` 3. Calling it, and passing it arguments, e.g. `@mydecorator(1, 2, 3)`
Python expects a decorator function to behave differently in the second and Python expects a decorator function to behave differently in the second and
third scenarios, when compared to the first: third scenarios, when compared to the first:
%% Cell type:code id: tags: %% Cell type:code id:e3104a1f tags:
``` ```
def decorator(*args): def decorator(*args):
print(f' decorator({args})') print(f' decorator({args})')
def wrapper(*args): def wrapper(*args):
print(f' wrapper({args})') print(f' wrapper({args})')
return wrapper return wrapper
print('Scenario #1: @decorator') print('Scenario #1: @decorator')
@decorator @decorator
def noop(): def noop():
pass pass
print('\nScenario #2: @decorator()') print('\nScenario #2: @decorator()')
@decorator() @decorator()
def noop(): def noop():
pass pass
print('\nScenario #3: @decorator(1, 2, 3)') print('\nScenario #3: @decorator(1, 2, 3)')
@decorator(1, 2, 3) @decorator(1, 2, 3)
def noop(): def noop():
pass pass
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:c9f67d6f tags:
So if a decorator is "named" (scenario 1), only the decorator function So if a decorator is "named" (scenario 1), only the decorator function
(`decorator` in the example above) is called, and is passed the decorated (`decorator` in the example above) is called, and is passed the decorated
function. function.
But if a decorator function is "called" (scenarios 2 or 3), both the decorator But if a decorator function is "called" (scenarios 2 or 3), both the decorator
function (`decorator`), **and its return value** (`wrapper`) are called - the function (`decorator`), **and its return value** (`wrapper`) are called - the
decorator function is passed the arguments that were provided, and its return decorator function is passed the arguments that were provided, and its return
value is passed the decorated function. value is passed the decorated function.
This is why, if you are writing a decorator function which expects arguments, This is why, if you are writing a decorator function which expects arguments,
you must use three layers of functions, like so: you must use three layers of functions, like so:
%% Cell type:code id: tags: %% Cell type:code id:d0918102 tags:
``` ```
def decorator(*args): def decorator(*args):
def realDecorator(func): def realDecorator(func):
def wrapper(*args, **kwargs): def wrapper(*args, **kwargs):
return func(*args, **kwargs) return func(*args, **kwargs)
return wrapper return wrapper
return realDecorator return realDecorator
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:94d5777a tags:
> The author of this practical is angry about this, as he does not understand > The author of this practical is angry about this, as he does not understand
> why the Python language designers couldn't allow a decorator function to be > why the Python language designers couldn't allow a decorator function to be
> passed both the decorated function, and any arguments that were passed when > passed both the decorated function, and any arguments that were passed when
> the decorator was invoked, like so: > the decorator was invoked, like so:
> >
> ``` > ```
> def decorator(func, *args, **kwargs): # args/kwargs here contain > def decorator(func, *args, **kwargs): # args/kwargs here contain
> # whatever is passed to the > # whatever is passed to the
> # decorator > # decorator
> >
> def wrapper(*args, **kwargs): # args/kwargs here contain > def wrapper(*args, **kwargs): # args/kwargs here contain
> # whatever is passed to the > # whatever is passed to the
> # decorated function > # decorated function
> return func(*args, **kwargs) > return func(*args, **kwargs)
> >
> return wrapper > return wrapper
> ``` > ```
<a class="anchor" id="appendix-per-instance-decorators"></a> <a class="anchor" id="appendix-per-instance-decorators"></a>
## Appendix: Per-instance decorators ## Appendix: Per-instance decorators
In the section on [decorating methods](#decorators-on-methods), you saw In the section on [decorating methods](#decorators-on-methods), you saw
that when a decorator is applied to a method of a class, that decorator that when a decorator is applied to a method of a class, that decorator
is invoked just once, and shared by all instances of the class. Consider this is invoked just once, and shared by all instances of the class. Consider this
example: example:
%% Cell type:code id: tags: %% Cell type:code id:d5a7bcd8 tags:
``` ```
def decorator(func): def decorator(func):
print(f'Decorating {func.__name__} function') print(f'Decorating {func.__name__} function')
def wrapper(*args, **kwargs): def wrapper(*args, **kwargs):
print(f'Calling decorated function {func.__name__}') print(f'Calling decorated function {func.__name__}')
return func(*args, **kwargs) return func(*args, **kwargs)
return wrapper return wrapper
class MiscMaths(object): class MiscMaths(object):
@decorator @decorator
def add(self, a, b): def add(self, a, b):
return a + b return a + b
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:544d7f9f tags:
Note that `decorator` was called at the time that the `MiscMaths` class was Note that `decorator` was called at the time that the `MiscMaths` class was
defined. Now, all `MiscMaths` instances share the same `wrapper` function: defined. Now, all `MiscMaths` instances share the same `wrapper` function:
%% Cell type:code id: tags: %% Cell type:code id:4f88ffbc tags:
``` ```
mm1 = MiscMaths() mm1 = MiscMaths()
mm2 = MiscMaths() mm2 = MiscMaths()
print('1 + 2 =', mm1.add(1, 2)) print('1 + 2 =', mm1.add(1, 2))
print('3 + 4 =', mm2.add(3, 4)) print('3 + 4 =', mm2.add(3, 4))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:a4829c44 tags:
This is not an issue in many cases, but it can be problematic in some. Imagine This is not an issue in many cases, but it can be problematic in some. Imagine
if we have a decorator called `ensureNumeric`, which makes sure that arguments if we have a decorator called `ensureNumeric`, which makes sure that arguments
passed to a function are numbers: passed to a function are numbers:
%% Cell type:code id: tags: %% Cell type:code id:1e792905 tags:
``` ```
def ensureNumeric(func): def ensureNumeric(func):
def wrapper(*args): def wrapper(*args):
args = tuple([float(a) for a in args]) args = tuple([float(a) for a in args])
return func(*args) return func(*args)
return wrapper return wrapper
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:ba204929 tags:
This all looks well and good - we can use it to decorate a numeric function, This all looks well and good - we can use it to decorate a numeric function,
allowing strings to be passed in as well: allowing strings to be passed in as well:
%% Cell type:code id: tags: %% Cell type:code id:b9cdea22 tags:
``` ```
@ensureNumeric @ensureNumeric
def mul(a, b): def mul(a, b):
return a * b return a * b
print(mul( 2, 3)) print(mul( 2, 3))
print(mul('5', '10')) print(mul('5', '10'))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:3467c3fb tags:
But what will happen when we try to decorate a method of a class? But what will happen when we try to decorate a method of a class?
%% Cell type:code id: tags: %% Cell type:code id:8397c94f tags:
``` ```
class MiscMaths(object): class MiscMaths(object):
@ensureNumeric @ensureNumeric
def add(self, a, b): def add(self, a, b):
return a + b return a + b
mm = MiscMaths() mm = MiscMaths()
print(mm.add('5', 10)) print(mm.add('5', 10))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:019d27b2 tags:
What happened here?? Remember that the first argument passed to any instance What happened here?? Remember that the first argument passed to any instance
method is the instance itself (the `self` argument). Well, the `MiscMaths` method is the instance itself (the `self` argument). Well, the `MiscMaths`
instance was passed to the `wrapper` function, which then tried to convert it instance was passed to the `wrapper` function, which then tried to convert it
into a `float`. So we can't actually apply the `ensureNumeric` function as a into a `float`. So we can't actually apply the `ensureNumeric` function as a
decorator on a method in this way. decorator on a method in this way.
There are a few potential solutions here. We could modify the `ensureNumeric` There are a few potential solutions here. We could modify the `ensureNumeric`
function, so that the `wrapper` ignores the first argument. But this would function, so that the `wrapper` ignores the first argument. But this would
mean that we couldn't use `ensureNumeric` with standalone functions. mean that we couldn't use `ensureNumeric` with standalone functions.
But we *can* manually apply the `ensureNumeric` decorator to `MiscMaths` But we *can* manually apply the `ensureNumeric` decorator to `MiscMaths`
instances when they are initialised. We can't use the nice `@ensureNumeric` instances when they are initialised. We can't use the nice `@ensureNumeric`
syntax to apply our decorators, but this is a viable approach: syntax to apply our decorators, but this is a viable approach:
%% Cell type:code id: tags: %% Cell type:code id:24026aaa tags:
``` ```
class MiscMaths(object): class MiscMaths(object):
def __init__(self): def __init__(self):
self.add = ensureNumeric(self.add) self.add = ensureNumeric(self.add)
def add(self, a, b): def add(self, a, b):
return a + b return a + b
mm = MiscMaths() mm = MiscMaths()
print(mm.add('5', 10)) print(mm.add('5', 10))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:e566cd0c tags:
Another approach is to use a second decorator, which dynamically creates the Another approach is to use a second decorator, which dynamically creates the
real decorator when it is accessed on an instance. This requires the use of an real decorator when it is accessed on an instance. This requires the use of an
advanced Python technique called advanced Python technique called
[*descriptors*](https://docs.python.org/3/howto/descriptor.html), which is [*descriptors*](https://docs.python.org/3/howto/descriptor.html), which is
beyond the scope of this practical. But if you are interested, you can see an beyond the scope of this practical. But if you are interested, you can see an
implementation of this approach implementation of this approach
[here](https://git.fmrib.ox.ac.uk/fsl/fslpy/blob/1.6.8/fsl/utils/memoize.py#L249). [here](https://git.fmrib.ox.ac.uk/fsl/fslpy/blob/1.6.8/fsl/utils/memoize.py#L249).
<a class="anchor" id="appendix-preserving-function-metadata"></a> <a class="anchor" id="appendix-preserving-function-metadata"></a>
## Appendix: Preserving function metadata ## Appendix: Preserving function metadata
You may have noticed that when we decorate a function, some of its properties You may have noticed that when we decorate a function, some of its properties
are lost. Consider this function: are lost. Consider this function:
%% Cell type:code id: tags: %% Cell type:code id:1a5305ef tags:
``` ```
def add2(a, b): def add2(a, b):
"""Adds two numbers together.""" """Adds two numbers together."""
return a + b return a + b
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:e7f3eeaa tags:
The `add2` function is an object which has some attributes, e.g.: The `add2` function is an object which has some attributes, e.g.:
%% Cell type:code id: tags: %% Cell type:code id:910c13ab tags:
``` ```
print('Name: ', add2.__name__) print('Name: ', add2.__name__)
print('Help: ', add2.__doc__) print('Help: ', add2.__doc__)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:095d66ac tags:
However, when we apply a decorator to `add2`: However, when we apply a decorator to `add2`:
%% Cell type:code id: tags: %% Cell type:code id:cbb19013 tags:
``` ```
def decorator(func): def decorator(func):
def wrapper(*args, **kwargs): def wrapper(*args, **kwargs):
"""Internal wrapper function for decorator.""" """Internal wrapper function for decorator."""
print(f'Calling decorated function {func.__name__}') print(f'Calling decorated function {func.__name__}')
return func(*args, **kwargs) return func(*args, **kwargs)
return wrapper return wrapper
@decorator @decorator
def add2(a, b): def add2(a, b):
"""Adds two numbers together.""" """Adds two numbers together."""
return a + b return a + b
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:40efc69c tags:
Those attributes are lost, and instead we get the attributes of the `wrapper` Those attributes are lost, and instead we get the attributes of the `wrapper`
function: function:
%% Cell type:code id: tags: %% Cell type:code id:5e01729e tags:
``` ```
print('Name: ', add2.__name__) print('Name: ', add2.__name__)
print('Help: ', add2.__doc__) print('Help: ', add2.__doc__)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:60a81b0d tags:
While this may be inconsequential in most situations, it can be quite annoying While this may be inconsequential in most situations, it can be quite annoying
in some, such as when we are automatically [generating in some, such as when we are automatically [generating
documentation](http://www.sphinx-doc.org/) for our code. documentation](http://www.sphinx-doc.org/) for our code.
Fortunately, there is a workaround, available in the built-in Fortunately, there is a workaround, available in the built-in
[`functools`](https://docs.python.org/3/library/functools.html#functools.wraps) [`functools`](https://docs.python.org/3/library/functools.html#functools.wraps)
module: module:
%% Cell type:code id: tags: %% Cell type:code id:85852617 tags:
``` ```
import functools import functools
def decorator(func): def decorator(func):
@functools.wraps(func) @functools.wraps(func)
def wrapper(*args, **kwargs): def wrapper(*args, **kwargs):
"""Internal wrapper function for decorator.""" """Internal wrapper function for decorator."""
print(f'Calling decorated function {func.__name__}') print(f'Calling decorated function {func.__name__}')
return func(*args, **kwargs) return func(*args, **kwargs)
return wrapper return wrapper
@decorator @decorator
def add2(a, b): def add2(a, b):
"""Adds two numbers together.""" """Adds two numbers together."""
return a + b return a + b
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:8c11fedc tags:
We have applied the `@functools.wraps` decorator to our internal `wrapper` We have applied the `@functools.wraps` decorator to our internal `wrapper`
function - this will essentially replace the `wrapper` function metdata with function - this will essentially replace the `wrapper` function metdata with
the metadata from our `func` function. So our `add2` name and documentation is the metadata from our `func` function. So our `add2` name and documentation is
now preserved: now preserved:
%% Cell type:code id: tags: %% Cell type:code id:aec6d5b3 tags:
``` ```
print('Name: ', add2.__name__) print('Name: ', add2.__name__)
print('Help: ', add2.__doc__) print('Help: ', add2.__doc__)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:6122df0d tags:
<a class="anchor" id="appendix-class-decorators"></a> <a class="anchor" id="appendix-class-decorators"></a>
## Appendix: Class decorators ## Appendix: Class decorators
> Not to be confused with [*decorator classes*](#decorator-classes)! > Not to be confused with [*decorator classes*](#decorator-classes)!
In this practical, we have shown how decorators can be applied to functions In this practical, we have shown how decorators can be applied to functions
and methods. But decorators can in fact also be applied to *classes*. This is and methods. But decorators can in fact also be applied to *classes*. This is
a fairly niche feature that you are probably not likely to need, so we will a fairly niche feature that you are probably not likely to need, so we will
only cover it briefly. only cover it briefly.
Imagine that we want all objects in our application to have a globally unique Imagine that we want all objects in our application to have a globally unique
(within the application) identifier. We could use a decorator which contains (within the application) identifier. We could use a decorator which contains
the logic for generating unique IDs, and defines the interface that we can the logic for generating unique IDs, and defines the interface that we can
use on an instance to obtain its ID: use on an instance to obtain its ID:
%% Cell type:code id: tags: %% Cell type:code id:9fc9ddc5 tags:
``` ```
import random import random
allIds = set() allIds = set()
def uniqueID(cls): def uniqueID(cls):
class subclass(cls): class subclass(cls):
def getUniqueID(self): def getUniqueID(self):
uid = getattr(self, '_uid', None) uid = getattr(self, '_uid', None)
if uid is not None: if uid is not None:
return uid return uid
while uid is None or uid in set(): while uid is None or uid in set():
uid = random.randint(1, 100) uid = random.randint(1, 100)
self._uid = uid self._uid = uid
return uid return uid
return subclass return subclass
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:870f6f27 tags:
Now we can use the `@uniqueID` decorator on any class that we need to Now we can use the `@uniqueID` decorator on any class that we need to
have a unique ID: have a unique ID:
%% Cell type:code id: tags: %% Cell type:code id:211be2eb tags:
``` ```
@uniqueID @uniqueID
class Foo(object): class Foo(object):
pass pass
@uniqueID @uniqueID
class Bar(object): class Bar(object):
pass pass
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:82d6a89b tags:
All instances of these classes will have a `getUniqueID` method: All instances of these classes will have a `getUniqueID` method:
%% Cell type:code id: tags: %% Cell type:code id:48c7d0bd tags:
``` ```
f1 = Foo() f1 = Foo()
f2 = Foo() f2 = Foo()
b1 = Bar() b1 = Bar()
b2 = Bar() b2 = Bar()
print('f1: ', f1.getUniqueID()) print('f1: ', f1.getUniqueID())
print('f2: ', f2.getUniqueID()) print('f2: ', f2.getUniqueID())
print('b1: ', b1.getUniqueID()) print('b1: ', b1.getUniqueID())
print('b2: ', b2.getUniqueID()) print('b2: ', b2.getUniqueID())
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:417d7064 tags:
<a class="anchor" id="useful-references"></a> <a class="anchor" id="useful-references"></a>
## Useful references ## Useful references
* [Understanding decorators in 12 easy steps](http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/) * [Understanding decorators in 12 easy steps](http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/)
* [The decorators they won't tell you about](https://github.com/hchasestevens/hchasestevens.github.io/blob/master/notebooks/the-decorators-they-wont-tell-you-about.ipynb) * [The decorators they won't tell you about](https://github.com/hchasestevens/hchasestevens.github.io/blob/master/notebooks/the-decorators-they-wont-tell-you-about.ipynb)
* [Closures - Wikipedia][wiki-closure] * [Closures - Wikipedia][wiki-closure]
* [Closures in Python](https://www.geeksforgeeks.org/python-closures/) * [Closures in Python](https://www.geeksforgeeks.org/python-closures/)
* [Garbage collection in Python](https://www.quora.com/How-does-garbage-collection-in-Python-work-What-are-the-pros-and-cons) * [Garbage collection in Python](https://www.quora.com/How-does-garbage-collection-in-Python-work-What-are-the-pros-and-cons)
[wiki-closure]: https://en.wikipedia.org/wiki/Closure_(computer_programming) [wiki-closure]: https://en.wikipedia.org/wiki/Closure_(computer_programming)
......
...@@ -786,7 +786,7 @@ This is what is known as a ...@@ -786,7 +786,7 @@ This is what is known as a
[*closure*](https://www.geeksforgeeks.org/python-closures/). Closures are a [*closure*](https://www.geeksforgeeks.org/python-closures/). Closures are a
fundamental, and extremely powerful, aspect of Python and other high level fundamental, and extremely powerful, aspect of Python and other high level
languages. So there's your answer, languages. So there's your answer,
[fishbulb](https://www.youtube.com/watch?v=CiAaEPcnlOg). [fishbulb](https://www.youtube.com/watch?v=aso4X_sOjuw).
<a class="anchor" id="appendix-decorators-without-arguments-versus-decorators-with-arguments"></a> <a class="anchor" id="appendix-decorators-without-arguments-versus-decorators-with-arguments"></a>
......
#!/usr/bin/env python #!/usr/bin/env python
# See: https://fsl.fmrib.ox.ac.uk/fslcourse/lectures/scripting/_0200.fpd/cheat3 # See: https://open.win.ox.ac.uk/pages/fslcourse/lectures/scripting/_0200.fpd/cheat3
PI = 3.1417 PI = 3.1417
def add(a, b): def add(a, b):
......
%% Cell type:markdown id: tags: %% Cell type:markdown id:697e3cd9 tags:
# Object-oriented programming in Python # Object-oriented programming in Python
By now you might have realised that __everything__ in Python is an By now you might have realised that __everything__ in Python is an
object. Strings are objects, numbers are objects, functions are objects, object. Strings are objects, numbers are objects, functions are objects,
modules are objects - __everything__ is an object! modules are objects - __everything__ is an object!
But this does not mean that you have to use Python in an object-oriented But this does not mean that you have to use Python in an object-oriented
fashion. You can stick with functions and statements, and get quite a lot fashion. You can stick with functions and statements, and get quite a lot
done. But some problems are just easier to solve, and to reason about, when done. But some problems are just easier to solve, and to reason about, when
you use an object-oriented approach. you use an object-oriented approach.
* [Objects versus classes](#objects-versus-classes) * [Objects versus classes](#objects-versus-classes)
* [Defining a class](#defining-a-class) * [Defining a class](#defining-a-class)
* [Object creation - the `__init__` method](#object-creation-the-init-method) * [Object creation - the `__init__` method](#object-creation-the-init-method)
* [Our method is called `__init__`, but we didn't actually call the `__init__` method!](#our-method-is-called-init) * [Our method is called `__init__`, but we didn't actually call the `__init__` method!](#our-method-is-called-init)
* [We didn't specify the `self` argument - what gives?!?](#we-didnt-specify-the-self-argument) * [We didn't specify the `self` argument - what gives?!?](#we-didnt-specify-the-self-argument)
* [Attributes](#attributes) * [Attributes](#attributes)
* [Methods](#methods) * [Methods](#methods)
* [Method chaining](#method-chaining) * [Method chaining](#method-chaining)
* [Protecting attribute access](#protecting-attribute-access) * [Protecting attribute access](#protecting-attribute-access)
* [A better way - properties](#a-better-way-properties]) * [A better way - properties](#a-better-way-properties])
* [Inheritance](#inheritance) * [Inheritance](#inheritance)
* [The basics](#the-basics) * [The basics](#the-basics)
* [Code re-use and problem decomposition](#code-re-use-and-problem-decomposition) * [Code re-use and problem decomposition](#code-re-use-and-problem-decomposition)
* [Polymorphism](#polymorphism) * [Polymorphism](#polymorphism)
* [Multiple inheritance](#multiple-inheritance) * [Multiple inheritance](#multiple-inheritance)
* [Class attributes and methods](#class-attributes-and-methods) * [Class attributes and methods](#class-attributes-and-methods)
* [Class attributes](#class-attributes) * [Class attributes](#class-attributes)
* [Class methods](#class-methods) * [Class methods](#class-methods)
* [Appendix: The `object` base-class](#appendix-the-object-base-class) * [Appendix: The `object` base-class](#appendix-the-object-base-class)
* [Appendix: `__init__` versus `__new__`](#appendix-init-versus-new) * [Appendix: `__init__` versus `__new__`](#appendix-init-versus-new)
* [Appendix: Monkey-patching](#appendix-monkey-patching) * [Appendix: Monkey-patching](#appendix-monkey-patching)
* [Appendix: Method overloading](#appendix-method-overloading) * [Appendix: Method overloading](#appendix-method-overloading)
* [Useful references](#useful-references) * [Useful references](#useful-references)
<a class="anchor" id="objects-versus-classes"></a> <a class="anchor" id="objects-versus-classes"></a>
## Objects versus classes ## Objects versus classes
If you are versed in C++, Java, C#, or some other object-oriented language, If you are versed in C++, Java, C#, or some other object-oriented language,
then this should all hopefully sound familiar, and you can skip to the next then this should all hopefully sound familiar, and you can skip to the next
section. section.
If you have not done any object-oriented programming before, your first step If you have not done any object-oriented programming before, your first step
is to understand the difference between *objects* (also known as is to understand the difference between *objects* (also known as
*instances*) and *classes* (also known as *types*). *instances*) and *classes* (also known as *types*).
If you have some experience in C, then you can start off by thinking of a If you have some experience in C, then you can start off by thinking of a
class as like a `struct` definition - a `struct` is a specification for the class as like a `struct` definition - a `struct` is a specification for the
layout of a chunk of memory. For example, here is a typical struct definition: layout of a chunk of memory. For example, here is a typical struct definition:
> ``` > ```
> /** > /**
> * Struct representing a stack. > * Struct representing a stack.
> */ > */
> typedef struct __stack { > typedef struct __stack {
> uint8_t capacity; /**< the maximum capacity of this stack */ > uint8_t capacity; /**< the maximum capacity of this stack */
> uint8_t size; /**< the current size of this stack */ > uint8_t size; /**< the current size of this stack */
> void **top; /**< pointer to the top of this stack */ > void **top; /**< pointer to the top of this stack */
> } stack_t; > } stack_t;
> ``` > ```
Now, an *object* is not a definition, but rather a thing which resides in Now, an *object* is not a definition, but rather a thing which resides in
memory. An object can have *attributes* (pieces of information), and *methods* memory. An object can have *attributes* (pieces of information), and *methods*
(functions associated with the object). You can pass objects around your code, (functions associated with the object). You can pass objects around your code,
manipulate their attributes, and call their methods. manipulate their attributes, and call their methods.
Returning to our C metaphor, you can think of an object as like an Returning to our C metaphor, you can think of an object as like an
instantiation of a struct: instantiation of a struct:
> ``` > ```
> stack_t stack; > stack_t stack;
> stack.capacity = 10; > stack.capacity = 10;
> ``` > ```
One of the major differences between a `struct` in C, and a `class` in Python One of the major differences between a `struct` in C, and a `class` in Python
and other object oriented languages, is that you can't (easily) add functions and other object oriented languages, is that you can't (easily) add functions
to a `struct` - it is just a chunk of memory. Whereas in Python, you can add to a `struct` - it is just a chunk of memory. Whereas in Python, you can add
functions to your class definition, which will then be added as methods when functions to your class definition, which will then be added as methods when
you create an object from that class. you create an object from that class.
Of course there are many more differences between C structs and classes (most Of course there are many more differences between C structs and classes (most
notably [inheritance](todo), [polymorphism](todo), and [access notably [inheritance](todo), [polymorphism](todo), and [access
protection](todo)). But if you can understand the difference between a protection](todo)). But if you can understand the difference between a
*definition* of a C struct, and an *instantiation* of that struct, then you *definition* of a C struct, and an *instantiation* of that struct, then you
are most of the way towards understanding the difference between a *class*, are most of the way towards understanding the difference between a *class*,
and an *object*. and an *object*.
> But just to confuse you, remember that in Python, **everything** is an > But just to confuse you, remember that in Python, **everything** is an
> object - even classes! > object - even classes!
<a class="anchor" id="defining-a-class"></a> <a class="anchor" id="defining-a-class"></a>
## Defining a class ## Defining a class
Defining a class in Python is simple. Let's take on a small project, by Defining a class in Python is simple. Let's take on a small project, by
developing a class which can be used in place of the `fslmaths` shell command. developing a class which can be used in place of the `fslmaths` shell command.
%% Cell type:code id: tags: %% Cell type:code id:f5c3a9a7 tags:
``` ```
class FSLMaths(object): class FSLMaths:
pass pass
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:9f1c99bf tags:
In this statement, we defined a new class called `FSLMaths`, which inherits In this statement, we defined a new class called `FSLMaths`.
from the built-in `object` base-class (see [below](inheritance) for more
details on inheritance). > You may also often see code which looks like this:
> ```
> class FSLMaths(object):
> ...
> ```
>
> Here we have defined the `FSLMaths` class to inherit from the built-in
> `object` base-class. This is a throw-back to Python 2, and you can think of
> both patterns to be essentially equivalent. See [below](inheritance) for more
details on inheritance.
Now that we have defined our class, we can create objects - instances of that Now that we have defined our class, we can create objects - instances of that
class - by calling the class itself, as if it were a function: class - by calling the class itself, as if it were a function:
%% Cell type:code id: tags: %% Cell type:code id:f910530f tags:
``` ```
fm1 = FSLMaths() fm1 = FSLMaths()
fm2 = FSLMaths() fm2 = FSLMaths()
print(fm1) print(fm1)
print(fm2) print(fm2)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:a52f79db tags:
Although these objects are not of much use at this stage. Let's do some more Although these objects are not of much use at this stage. Let's do some more
work. work.
<a class="anchor" id="object-creation-the-init-method"></a> <a class="anchor" id="object-creation-the-init-method"></a>
## Object creation - the `__init__` method ## Object creation - the `__init__` method
The first thing that our `fslmaths` replacement will need is an input image. The first thing that our `fslmaths` replacement will need is an input image.
It makes sense to pass this in when we create an `FSLMaths` object: It makes sense to pass this in when we create an `FSLMaths` object:
%% Cell type:code id: tags: %% Cell type:code id:c9b2b073 tags:
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:151747eb tags:
Here we have added a _method_ called `__init__` to our class (remember that a Here we have added a _method_ called `__init__` to our class (remember that a
_method_ is just a function which is defined in a class, and which can be _method_ is just a function which is defined in a class, and which can be
called on instances of that class). This method expects two arguments - called on instances of that class). This method expects two arguments -
`self`, and `inimg`. So now, when we create an instance of the `FSLMaths` `self`, and `inimg`. So now, when we create an instance of the `FSLMaths`
class, we will need to provide an input image: class, we will need to provide an input image:
%% Cell type:code id: tags: %% Cell type:code id:4df00c18 tags:
``` ```
import nibabel as nib import nibabel as nib
import os.path as op import os.path as op
fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz') fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')
inimg = nib.load(fpath) inimg = nib.load(fpath)
fm = FSLMaths(inimg) fm = FSLMaths(inimg)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:de785025 tags:
There are a couple of things to note here... There are a couple of things to note here...
<a class="anchor" id="our-method-is-called-init"></a> <a class="anchor" id="our-method-is-called-init"></a>
### Our method is called `__init__`, but we didn't actually call the `__init__` method! ### Our method is called `__init__`, but we didn't actually call the `__init__` method!
`__init__` is a special method in Python - it is called when an instance of a `__init__` is a special method in Python - it is called when an instance of a
class is created. And recall that we can create an instance of a class by class is created. And recall that we can create an instance of a class by
calling the class in the same way that we call a function. calling the class in the same way that we call a function.
There are a number of "special" methods that you can add to a class in Python There are a number of "special" methods that you can add to a class in Python
to customise various aspects of how instances of the class behave. One of the to customise various aspects of how instances of the class behave. One of the
first ones you may come across is the `__str__` method, which defines how an first ones you may come across is the `__str__` method, which defines how an
object should be printed (more specifically, how an object gets converted into object should be printed (more specifically, how an object gets converted into
a string). For example, we could add a `__str__` method to our `FSLMaths` a string). For example, we could add a `__str__` method to our `FSLMaths`
class like so: class like so:
%% Cell type:code id: tags: %% Cell type:code id:512ee694 tags:
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
def __str__(self): def __str__(self):
return f'FSLMaths({self.img.get_filename()})' return f'FSLMaths({self.img.get_filename()})'
fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz') fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')
inimg = nib.load(fpath) inimg = nib.load(fpath)
fm = FSLMaths(inimg) fm = FSLMaths(inimg)
print(fm) print(fm)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:e7375369 tags:
Refer to the [official Refer to the [official
docs](https://docs.python.org/3/reference/datamodel.html#special-method-names) docs](https://docs.python.org/3/reference/datamodel.html#special-method-names)
for details on all of the special methods that can be defined in a class. And for details on all of the special methods that can be defined in a class. And
take a look at the appendix for some more details on [how Python objects get take a look at the appendix for some more details on [how Python objects get
created](appendix-init-versus-new). created](appendix-init-versus-new).
<a class="anchor" id="we-didnt-specify-the-self-argument"></a> <a class="anchor" id="we-didnt-specify-the-self-argument"></a>
### We didn't specify the `self` argument - what gives?!? ### We didn't specify the `self` argument - what gives?!?
The `self` argument is a special argument for methods in Python. If you are The `self` argument is a special argument for methods in Python. If you are
coming from C++, Java, C# or similar, `self` in Python is equivalent to `this` coming from C++, Java, C# or similar, `self` in Python is equivalent to `this`
in those languages. in those languages.
In a method, the `self` argument is a reference to the object that the method In a method, the `self` argument is a reference to the object that the method
was called on. So in this line of code: was called on. So in this line of code:
%% Cell type:code id: tags: %% Cell type:code id:fa4cb3b5 tags:
``` ```
fm = FSLMaths(inimg) fm = FSLMaths(inimg)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:b4828743 tags:
the `self` argument in `__init__` will be a reference to the `FSLMaths` object the `self` argument in `__init__` will be a reference to the `FSLMaths` object
that has been created (and is then assigned to the `fm` variable, after the that has been created (and is then assigned to the `fm` variable, after the
`__init__` method has finished). `__init__` method has finished).
But note that you __do not__ need to explicitly provide the `self` argument But note that you __do not__ need to explicitly provide the `self` argument
when you call a method on an object, or when you create a new object. The when you call a method on an object, or when you create a new object. The
Python runtime will take care of passing the instance to its method, as the Python runtime will take care of passing the instance to its method, as the
first argument to the method. first argument to the method.
But when you are writing a class, you __do__ need to explicitly list `self` as But when you are writing a class, you __do__ need to explicitly list `self` as
the first argument to all of the methods of the class. the first argument to all of the methods of the class.
<a class="anchor" id="attributes"></a> <a class="anchor" id="attributes"></a>
## Attributes ## Attributes
In Python, the term __attribute__ is used to refer to a piece of information In Python, the term __attribute__ is used to refer to a piece of information
that is associated with an object. An attribute is generally a reference to that is associated with an object. An attribute is generally a reference to
another object (which might be a string, a number, or a list, or some other another object (which might be a string, a number, or a list, or some other
more complicated object). more complicated object).
Remember that we modified our `FSLMaths` class so that it is passed an input Remember that we modified our `FSLMaths` class so that it is passed an input
image on creation: image on creation:
%% Cell type:code id: tags: %% Cell type:code id:564228d9 tags:
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz') fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')
fm = FSLMaths(nib.load(fpath)) fm = FSLMaths(nib.load(fpath))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:10858f3d tags:
Take a look at what is going on in the `__init__` method - we take the `inimg` Take a look at what is going on in the `__init__` method - we take the `inimg`
argument, and create a reference to it called `self.img`. We have added an argument, and create a reference to it called `self.img`. We have added an
_attribute_ to the `FSLMaths` instance, called `img`, and we can access that _attribute_ to the `FSLMaths` instance, called `img`, and we can access that
attribute like so: attribute like so:
%% Cell type:code id: tags: %% Cell type:code id:d4756303 tags:
``` ```
print('Input for our FSLMaths instance:', fm.img.get_filename()) print('Input for our FSLMaths instance:', fm.img.get_filename())
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:448f28df tags:
And that concludes the section on adding attributes to Python objects. And that concludes the section on adding attributes to Python objects.
Just kidding. But it really is that simple. This is one aspect of Python which Just kidding. But it really is that simple. This is one aspect of Python which
might be quite jarring to you if you are coming from a language with more might be quite jarring to you if you are coming from a language with more
rigid semantics, such as C++ or Java. In those languages, you must pre-specify rigid semantics, such as C++ or Java. In those languages, you must pre-specify
all of the attributes and methods that are a part of a class. But Python is all of the attributes and methods that are a part of a class. But Python is
much more flexible - you can simply add attributes to an object after it has much more flexible - you can simply add attributes to an object after it has
been created. In fact, you can even do this outside of the class been created. In fact, you can even do this outside of the class
definition<sup>1</sup>: definition<sup>1</sup>:
%% Cell type:code id: tags: %% Cell type:code id:d31da8c5 tags:
``` ```
fm = FSLMaths(inimg) fm = FSLMaths(inimg)
fm.another_attribute = 'Haha' fm.another_attribute = 'Haha'
print(fm.another_attribute) print(fm.another_attribute)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:3e395f1d tags:
__But ...__ while attributes can be added to a Python object at any time, it is __But ...__ while attributes can be added to a Python object at any time, it is
good practice (and makes for more readable and maintainable code) to add all good practice (and makes for more readable and maintainable code) to add all
of an object's attributes within the `__init__` method. of an object's attributes within the `__init__` method.
> <sup>1</sup>This not possible with many of the built-in types, such as > <sup>1</sup>This not possible with many of the built-in types, such as
> `list` and `dict` objects, nor with types that are defined in Python > `list` and `dict` objects, nor with types that are defined in Python
> extensions (Python modules that are written in C). > extensions (Python modules that are written in C).
<a class="anchor" id="methods"></a> <a class="anchor" id="methods"></a>
## Methods ## Methods
We've been dilly-dallying on this little `FSLMaths` project for a while now, We've been dilly-dallying on this little `FSLMaths` project for a while now,
but our class still can't actually do anything. Let's start adding some but our class still can't actually do anything. Let's start adding some
functionality: functionality:
%% Cell type:code id: tags: %% Cell type:code id:6e05364a tags:
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
self.operations = [] self.operations = []
def add(self, value): def add(self, value):
self.operations.append(('add', value)) self.operations.append(('add', value))
def mul(self, value): def mul(self, value):
self.operations.append(('mul', value)) self.operations.append(('mul', value))
def div(self, value): def div(self, value):
self.operations.append(('div', value)) self.operations.append(('div', value))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:6057ef19 tags:
Woah woah, [slow down egg-head!](https://www.youtube.com/watch?v=yz-TemWooa4) Woah woah, [slow down egg-head!](https://www.youtube.com/watch?v=yz-TemWooa4)
We've modified `__init__` so that a second attribute called `operations` is We've modified `__init__` so that a second attribute called `operations` is
added to our object - this `operations` attribute is simply a list. added to our object - this `operations` attribute is simply a list.
Then, we added a handful of methods - `add`, `mul`, and `div` - which each Then, we added a handful of methods - `add`, `mul`, and `div` - which each
append a tuple to that `operations` list. append a tuple to that `operations` list.
> Note that, just like in the `__init__` method, the first argument that will > Note that, just like in the `__init__` method, the first argument that will
> be passed to these methods is `self` - a reference to the object that the > be passed to these methods is `self` - a reference to the object that the
> method has been called on. > method has been called on.
The idea behind this design is that our `FSLMaths` class will not actually do The idea behind this design is that our `FSLMaths` class will not actually do
anything when we call the `add`, `mul` or `div` methods. Instead, it will anything when we call the `add`, `mul` or `div` methods. Instead, it will
*stage* each operation, and then perform them all in one go at a later point *stage* each operation, and then perform them all in one go at a later point
in time. So let's add another method, `run`, which actually does the work: in time. So let's add another method, `run`, which actually does the work:
%% Cell type:code id: tags: %% Cell type:code id:99953d05 tags:
``` ```
import numpy as np import numpy as np
import nibabel as nib import nibabel as nib
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
self.operations = [] self.operations = []
def add(self, value): def add(self, value):
self.operations.append(('add', value)) self.operations.append(('add', value))
def mul(self, value): def mul(self, value):
self.operations.append(('mul', value)) self.operations.append(('mul', value))
def div(self, value): def div(self, value):
self.operations.append(('div', value)) self.operations.append(('div', value))
def run(self, output=None): def run(self, output=None):
data = np.array(self.img.get_fdata()) data = np.array(self.img.get_fdata())
for oper, value in self.operations: for oper, value in self.operations:
# Value could be an image. # Value could be an image.
# If not, we assume that # If not, we assume that
# it is a scalar/numpy array. # it is a scalar/numpy array.
if isinstance(value, nib.nifti1.Nifti1Image): if isinstance(value, nib.nifti1.Nifti1Image):
value = value.get_fdata() value = value.get_fdata()
if oper == 'add': if oper == 'add':
data = data + value data = data + value
elif oper == 'mul': elif oper == 'mul':
data = data * value data = data * value
elif oper == 'div': elif oper == 'div':
data = data / value data = data / value
# turn final output into a nifti, # turn final output into a nifti,
# and save it to disk if an # and save it to disk if an
# 'output' has been specified. # 'output' has been specified.
outimg = nib.nifti1.Nifti1Image(data, inimg.affine) outimg = nib.nifti1.Nifti1Image(data, inimg.affine)
if output is not None: if output is not None:
nib.save(outimg, output) nib.save(outimg, output)
return outimg return outimg
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:953c519c tags:
We now have a useable (but not very useful) `FSLMaths` class! We now have a useable (but not very useful) `FSLMaths` class!
%% Cell type:code id: tags: %% Cell type:code id:c906152e tags:
``` ```
fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz') fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')
fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz') fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz')
inimg = nib.load(fpath) inimg = nib.load(fpath)
mask = nib.load(fmask) mask = nib.load(fmask)
fm = FSLMaths(inimg) fm = FSLMaths(inimg)
fm.mul(mask) fm.mul(mask)
fm.add(-10) fm.add(-10)
outimg = fm.run() outimg = fm.run()
norigvox = (inimg .get_fdata() > 0).sum() norigvox = (inimg .get_fdata() > 0).sum()
nmaskvox = (outimg.get_fdata() > 0).sum() nmaskvox = (outimg.get_fdata() > 0).sum()
print(f'Number of voxels >0 in original image: {norigvox}') print(f'Number of voxels >0 in original image: {norigvox}')
print(f'Number of voxels >0 in masked image: {nmaskvox}') print(f'Number of voxels >0 in masked image: {nmaskvox}')
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:6d7f078f tags:
<a class="anchor" id="method-chaining"></a> <a class="anchor" id="method-chaining"></a>
## Method chaining ## Method chaining
A neat trick, which is used by all the cool kids these days, is to write A neat trick, which is used by all the cool kids these days, is to write
classes that allow *method chaining* - writing one line of code which classes that allow *method chaining* - writing one line of code which
calls more than one method on an object, e.g.: calls more than one method on an object, e.g.:
> ``` > ```
> fm = FSLMaths(img) > fm = FSLMaths(img)
> result = fm.add(1).mul(10).run() > result = fm.add(1).mul(10).run()
> ``` > ```
Adding this feature to our budding `FSLMaths` class is easy - all we have Adding this feature to our budding `FSLMaths` class is easy - all we have
to do is return `self` from each method: to do is return `self` from each method:
%% Cell type:code id: tags: %% Cell type:code id:e68604ac tags:
``` ```
import numpy as np import numpy as np
import nibabel as nib import nibabel as nib
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
self.operations = [] self.operations = []
def add(self, value): def add(self, value):
self.operations.append(('add', value)) self.operations.append(('add', value))
return self return self
def mul(self, value): def mul(self, value):
self.operations.append(('mul', value)) self.operations.append(('mul', value))
return self return self
def div(self, value): def div(self, value):
self.operations.append(('div', value)) self.operations.append(('div', value))
return self return self
def run(self, output=None): def run(self, output=None):
data = np.array(self.img.get_fdata()) data = np.array(self.img.get_fdata())
for oper, value in self.operations: for oper, value in self.operations:
# Value could be an image. # Value could be an image.
# If not, we assume that # If not, we assume that
# it is a scalar/numpy array. # it is a scalar/numpy array.
if isinstance(value, nib.nifti1.Nifti1Image): if isinstance(value, nib.nifti1.Nifti1Image):
value = value.get_fdata() value = value.get_fdata()
if oper == 'add': if oper == 'add':
data = data + value data = data + value
elif oper == 'mul': elif oper == 'mul':
data = data * value data = data * value
elif oper == 'div': elif oper == 'div':
data = data / value data = data / value
# turn final output into a nifti, # turn final output into a nifti,
# and save it to disk if an # and save it to disk if an
# 'output' has been specified. # 'output' has been specified.
outimg = nib.nifti1.Nifti1Image(data, inimg.affine) outimg = nib.nifti1.Nifti1Image(data, inimg.affine)
if output is not None: if output is not None:
nib.save(outimg, output) nib.save(outimg, output)
return outimg return outimg
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:dbce91e6 tags:
Now we can chain all of our method calls, and even the creation of our Now we can chain all of our method calls, and even the creation of our
`FSLMaths` object, into a single line: `FSLMaths` object, into a single line:
%% Cell type:code id: tags: %% Cell type:code id:1cf1023a tags:
``` ```
fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz') fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')
fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz') fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz')
inimg = nib.load(fpath) inimg = nib.load(fpath)
mask = nib.load(fmask) mask = nib.load(fmask)
outimg = FSLMaths(inimg).mul(mask).add(-10).run() outimg = FSLMaths(inimg).mul(mask).add(-10).run()
norigvox = (inimg .get_fdata() > 0).sum() norigvox = (inimg .get_fdata() > 0).sum()
nmaskvox = (outimg.get_fdata() > 0).sum() nmaskvox = (outimg.get_fdata() > 0).sum()
print(f'Number of voxels >0 in original image: {norigvox}') print(f'Number of voxels >0 in original image: {norigvox}')
print(f'Number of voxels >0 in masked image: {nmaskvox}') print(f'Number of voxels >0 in masked image: {nmaskvox}')
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:81c627c0 tags:
> In fact, this is precisely how the > In fact, this is precisely how the
> [`fsl.wrappers.fslmaths`](https://users.fmrib.ox.ac.uk/~paulmc/fsleyes/fslpy/latest/fsl.wrappers.fslmaths.html) > [`fsl.wrappers.fslmaths`](https://users.fmrib.ox.ac.uk/~paulmc/fsleyes/fslpy/latest/fsl.wrappers.fslmaths.html)
> function works. > function works.
<a class="anchor" id="protecting-attribute-access"></a> <a class="anchor" id="protecting-attribute-access"></a>
## Protecting attribute access ## Protecting attribute access
In our `FSLMaths` class, the input image was added as an attribute called In our `FSLMaths` class, the input image was added as an attribute called
`img` to `FSLMaths` objects. We saw that it is easy to read the attributes `img` to `FSLMaths` objects. We saw that it is easy to read the attributes
of an object - if we have a `FSLMaths` instance called `fm`, we can read its of an object - if we have a `FSLMaths` instance called `fm`, we can read its
input image via `fm.img`. input image via `fm.img`.
But it is just as easy to write the attributes of an object. What's to stop But it is just as easy to write the attributes of an object. What's to stop
some sloppy research assistant from overwriting our `img` attribute? some sloppy research assistant from overwriting our `img` attribute?
%% Cell type:code id: tags: %% Cell type:code id:fb9c0900 tags:
``` ```
inimg = nib.load(op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')) inimg = nib.load(op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz'))
fm = FSLMaths(inimg) fm = FSLMaths(inimg)
fm.img = None fm.img = None
fm.run() fm.run()
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:e460cb1f tags:
Well, the scary answer is ... there is __nothing__ stopping you from doing Well, the scary answer is ... there is __nothing__ stopping you from doing
whatever you want to a Python object. You can add, remove, and modify whatever you want to a Python object. You can add, remove, and modify
attributes at will. You can even replace the methods of an existing object if attributes at will. You can even replace the methods of an existing object if
you like: you like:
%% Cell type:code id: tags: %% Cell type:code id:c4ded7fd tags:
``` ```
fm = FSLMaths(inimg) fm = FSLMaths(inimg)
def myadd(value): def myadd(value):
print('Oh no, I\'m not going to add {} to ' print('Oh no, I\'m not going to add {} to '
'your image. Go away!'.format(value)) 'your image. Go away!'.format(value))
fm.add = myadd fm.add = myadd
fm.add(123) fm.add(123)
fm.mul = None fm.mul = None
fm.mul(123) fm.mul(123)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:91af0a9d tags:
But you really shouldn't get into the habit of doing devious things like But you really shouldn't get into the habit of doing devious things like
this. Think of the poor souls who inherit your code years after you have left this. Think of the poor souls who inherit your code years after you have left
the lab - if you go around overwriting all of the methods and attributes of the lab - if you go around overwriting all of the methods and attributes of
your objects, they are not going to have a hope in hell of understanding what your objects, they are not going to have a hope in hell of understanding what
your code is actually doing, and they are not going to like you very your code is actually doing, and they are not going to like you very
much. Take a look at the appendix for a [brief discussion on this much. Take a look at the appendix for a [brief discussion on this
topic](appendix-monkey-patching). topic](appendix-monkey-patching).
Python tends to assume that programmers are "responsible adults", and hence Python tends to assume that programmers are "responsible adults", and hence
doesn't do much in the way of restricting access to the attributes or methods doesn't do much in the way of restricting access to the attributes or methods
of an object. This is in contrast to languages like C++ and Java, where the of an object. This is in contrast to languages like C++ and Java, where the
notion of a private attribute or method is strictly enforced by the language. notion of a private attribute or method is strictly enforced by the language.
However, there are a couple of conventions in Python that are However, there are a couple of conventions in Python that are
[universally adhered to](https://docs.python.org/3/tutorial/classes.html#private-variables): [universally adhered to](https://docs.python.org/3/tutorial/classes.html#private-variables):
* Class-level attributes and methods, and module-level attributes, functions, * Class-level attributes and methods, and module-level attributes, functions,
and classes, which begin with a single underscore (`_`), should be and classes, which begin with a single underscore (`_`), should be
considered __protected__ - they are intended for internal use only, and considered __protected__ - they are intended for internal use only, and
should not be considered part of the public API of a class or module. This should not be considered part of the public API of a class or module. This
is not enforced by the language in any way<sup>2</sup> - remember, we are is not enforced by the language in any way<sup>2</sup> - remember, we are
all responsible adults here! all responsible adults here!
* Class-level attributes and methods which begin with a double-underscore * Class-level attributes and methods which begin with a double-underscore
(`__`) should be considered __private__. Python provides a weak form of (`__`) should be considered __private__. Python provides a weak form of
enforcement for this rule - any attribute or method with such a name will enforcement for this rule - any attribute or method with such a name will
actually be _renamed_ (in a standardised manner) at runtime, so that it is actually be _renamed_ (in a standardised manner) at runtime, so that it is
not accessible through its original name (it is still accessible via its not accessible through its original name (it is still accessible via its
[mangled name](https://docs.python.org/3/tutorial/classes.html#private-variables) [mangled name](https://docs.python.org/3/tutorial/classes.html#private-variables)
though). though).
> <sup>2</sup> With the exception that module-level fields which begin with a > <sup>2</sup> With the exception that module-level fields which begin with a
> single underscore will not be imported into the local scope via the > single underscore will not be imported into the local scope via the
> `from [module] import *` technique. > `from [module] import *` technique.
So with all of this in mind, we can adjust our `FSLMaths` class to discourage So with all of this in mind, we can adjust our `FSLMaths` class to discourage
our sloppy research assistant from overwriting the `img` attribute: our sloppy research assistant from overwriting the `img` attribute:
%% Cell type:code id: tags: %% Cell type:code id:bb4fe666 tags:
``` ```
# remainder of definition omitted for brevity # remainder of definition omitted for brevity
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.__img = inimg self.__img = inimg
self.__operations = [] self.__operations = []
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:3f8e5ddc tags:
But now we have lost the ability to read our `__img` attribute: But now we have lost the ability to read our `__img` attribute:
%% Cell type:code id: tags: %% Cell type:code id:d86d565e tags:
``` ```
inimg = nib.load(op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')) inimg = nib.load(op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz'))
fm = FSLMaths(inimg) fm = FSLMaths(inimg)
print(fm.__img) print(fm.__img)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:bf2d7e8b tags:
<a class="anchor" id="a-better-way-properties"></a> <a class="anchor" id="a-better-way-properties"></a>
### A better way - properties ### A better way - properties
Python has a feature called Python has a feature called
[`properties`](https://docs.python.org/3/library/functions.html#property), [`properties`](https://docs.python.org/3/library/functions.html#property),
which is a nice way of controlling access to the attributes of an object. We which is a nice way of controlling access to the attributes of an object. We
can use properties by defining a "getter" method which can be used to access can use properties by defining a "getter" method which can be used to access
our attributes, and "decorating" them with the `@property` decorator (we will our attributes, and "decorating" them with the `@property` decorator (we will
cover decorators in a later practical). cover decorators in a later practical).
%% Cell type:code id: tags: %% Cell type:code id:6704b24b tags:
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.__img = inimg self.__img = inimg
self.__operations = [] self.__operations = []
@property @property
def img(self): def img(self):
return self.__img return self.__img
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:f5e0e4ce tags:
So we are still storing our input image as a private attribute, but now we So we are still storing our input image as a private attribute, but now we
have made it available in a read-only manner via the public `img` property: have made it available in a read-only manner via the public `img` property:
%% Cell type:code id: tags: %% Cell type:code id:6dd75ae0 tags:
``` ```
fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz') fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')
inimg = nib.load(fpath) inimg = nib.load(fpath)
fm = FSLMaths(inimg) fm = FSLMaths(inimg)
print(fm.img.get_filename()) print(fm.img.get_filename())
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:ca9123b9 tags:
Note that, even though we have defined `img` as a method, we can access it Note that, even though we have defined `img` as a method, we can access it
like an attribute - this is due to the magic behind the `@property` decorator. like an attribute - this is due to the magic behind the `@property` decorator.
We can also define "setter" methods for a property. For example, we might wish We can also define "setter" methods for a property. For example, we might wish
to add the ability for a user of our `FSLMaths` class to change the input to add the ability for a user of our `FSLMaths` class to change the input
image after creation. image after creation.
%% Cell type:code id: tags: %% Cell type:code id:4793497b tags:
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.__img = None self.__img = None
self.__operations = [] self.__operations = []
self.img = inimg self.img = inimg
@property @property
def img(self): def img(self):
return self.__img return self.__img
@img.setter @img.setter
def img(self, value): def img(self, value):
if not isinstance(value, nib.nifti1.Nifti1Image): if not isinstance(value, nib.nifti1.Nifti1Image):
raise ValueError('value must be a NIFTI image!') raise ValueError('value must be a NIFTI image!')
self.__img = value self.__img = value
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:368d7c7e tags:
> Note that we used the `img` setter method within `__init__` to validate the > Note that we used the `img` setter method within `__init__` to validate the
> initial `inimg` that was passed in during creation. > initial `inimg` that was passed in during creation.
Property setters are a nice way to add validation logic for when an attribute Property setters are a nice way to add validation logic for when an attribute
is assigned a value. In this example, an error will be raised if the new input is assigned a value. In this example, an error will be raised if the new input
is not a NIFTI image. is not a NIFTI image.
%% Cell type:code id: tags: %% Cell type:code id:c9d400bb tags:
``` ```
fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz') fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')
inimg = nib.load(fpath) inimg = nib.load(fpath)
fm = FSLMaths(inimg) fm = FSLMaths(inimg)
print('Input: ', fm.img.get_filename()) print('Input: ', fm.img.get_filename())
# let's change the input # let's change the input
# to a different image # to a different image
fpath2 = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain.nii.gz') fpath2 = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain.nii.gz')
inimg2 = nib.load(fpath2) inimg2 = nib.load(fpath2)
fm.img = inimg2 fm.img = inimg2
print('New input: ', fm.img.get_filename()) print('New input: ', fm.img.get_filename())
print('This is going to explode') print('This is going to explode')
fm.img = 'abcde' fm.img = 'abcde'
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:60058df5 tags:
<a class="anchor" id="inheritance"></a> <a class="anchor" id="inheritance"></a>
## Inheritance ## Inheritance
One of the major advantages of an object-oriented programming approach is One of the major advantages of an object-oriented programming approach is
_inheritance_ - the ability to define hierarchical relationships between _inheritance_ - the ability to define hierarchical relationships between
classes and instances. classes and instances.
<a class="anchor" id="the-basics"></a> <a class="anchor" id="the-basics"></a>
### The basics ### The basics
My local veterinary surgery runs some Python code which looks like the My local veterinary surgery runs some Python code which looks like the
following, to assist the nurses in identifying an animal when it arrives at following, to assist the nurses in identifying an animal when it arrives at
the surgery: the surgery:
%% Cell type:code id: tags: %% Cell type:code id:f6aabb7e tags:
``` ```
class Animal(object): class Animal:
def noiseMade(self): def noiseMade(self):
raise NotImplementedError('This method must be ' raise NotImplementedError('This method must be '
'implemented by sub-classes') 'implemented by sub-classes')
class Dog(Animal): class Dog(Animal):
def noiseMade(self): def noiseMade(self):
return 'Woof' return 'Woof'
class TalkingDog(Dog): class TalkingDog(Dog):
def noiseMade(self): def noiseMade(self):
return 'Hi Homer, find your soulmate!' return 'Hi Homer, find your soulmate!'
class Cat(Animal): class Cat(Animal):
def noiseMade(self): def noiseMade(self):
return 'Meow' return 'Meow'
class Labrador(Dog): class Labrador(Dog):
pass pass
class Chihuahua(Dog): class Chihuahua(Dog):
def noiseMade(self): def noiseMade(self):
return 'Yap yap yap' return 'Yap yap yap'
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:3ea5f303 tags:
Hopefully this example doesn't need much in the way of explanation - this Hopefully this example doesn't need much in the way of explanation - this
collection of classes represents a hierarchical relationship which exists in collection of classes represents a hierarchical relationship which exists in
the real world (and also represents the inherently annoying nature of the real world (and also represents the inherently annoying nature of
chihuahuas). For example, in the real world, all dogs are animals, but not all chihuahuas). For example, in the real world, all dogs are animals, but not all
animals are dogs. Therefore in our model, the `Dog` class has specified animals are dogs. Therefore in our model, the `Dog` class has specified
`Animal` as its base class. We say that the `Dog` class *extends*, *derives `Animal` as its base class. We say that the `Dog` class *extends*, *derives
from*, or *inherits from*, the `Animal` class, and that all `Dog` instances from*, or *inherits from*, the `Animal` class, and that all `Dog` instances
are also `Animal` instances (but not vice-versa). are also `Animal` instances (but not vice-versa).
What does that `noiseMade` method do? There is a `noiseMade` method defined What does that `noiseMade` method do? There is a `noiseMade` method defined
on the `Animal` class, but it has been re-implemented, or *overridden* in the on the `Animal` class, but it has been re-implemented, or *overridden* in the
`Dog`, `Dog`,
[`TalkingDog`](https://twitter.com/simpsonsqotd/status/427941665836630016?lang=en), [`TalkingDog`](https://twitter.com/simpsonsqotd/status/427941665836630016?lang=en),
`Cat`, and `Chihuahua` classes (but not on the `Labrador` class). We can call `Cat`, and `Chihuahua` classes (but not on the `Labrador` class). We can call
the `noiseMade` method on any `Animal` instance, but the specific behaviour the `noiseMade` method on any `Animal` instance, but the specific behaviour
that we get is dependent on the specific type of animal. that we get is dependent on the specific type of animal.
%% Cell type:code id: tags: %% Cell type:code id:5516a3fc tags:
``` ```
d = Dog() d = Dog()
l = Labrador() l = Labrador()
c = Cat() c = Cat()
ch = Chihuahua() ch = Chihuahua()
print('Noise made by dogs: ', d .noiseMade()) print('Noise made by dogs: ', d .noiseMade())
print('Noise made by labradors: ', l .noiseMade()) print('Noise made by labradors: ', l .noiseMade())
print('Noise made by cats: ', c .noiseMade()) print('Noise made by cats: ', c .noiseMade())
print('Noise made by chihuahuas:', ch.noiseMade()) print('Noise made by chihuahuas:', ch.noiseMade())
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:699b1cee tags:
Note that calling the `noiseMade` method on a `Labrador` instance resulted in Note that calling the `noiseMade` method on a `Labrador` instance resulted in
the `Dog.noiseMade` implementation being called. the `Dog.noiseMade` implementation being called.
<a class="anchor" id="code-re-use-and-problem-decomposition"></a> <a class="anchor" id="code-re-use-and-problem-decomposition"></a>
### Code re-use and problem decomposition ### Code re-use and problem decomposition
Inheritance allows us to split a problem into smaller problems, and to re-use Inheritance allows us to split a problem into smaller problems, and to re-use
code. Let's demonstrate this with a more involved (and even more contrived) code. Let's demonstrate this with a more involved (and even more contrived)
example. Imagine that a former colleague had written a class called example. Imagine that a former colleague had written a class called
`Operator`: `Operator`:
%% Cell type:code id: tags: %% Cell type:code id:cd3cbcd0 tags:
``` ```
class Operator(object): class Operator:
def __init__(self): def __init__(self):
super().__init__() # this line will be explained later super().__init__() # this line will be explained later
self.__operations = [] self.__operations = []
self.__opFuncs = {} self.__opFuncs = {}
@property @property
def operations(self): def operations(self):
return list(self.__operations) return list(self.__operations)
@property @property
def functions(self): def functions(self):
return dict(self.__opFuncs) return dict(self.__opFuncs)
def addFunction(self, name, func): def addFunction(self, name, func):
self.__opFuncs[name] = func self.__opFuncs[name] = func
def do(self, name, *values): def do(self, name, *values):
self.__operations.append((name, values)) self.__operations.append((name, values))
def preprocess(self, value): def preprocess(self, value):
return value return value
def run(self, input): def run(self, input):
data = self.preprocess(input) data = self.preprocess(input)
for oper, vals in self.__operations: for oper, vals in self.__operations:
func = self.__opFuncs[oper] func = self.__opFuncs[oper]
vals = [self.preprocess(v) for v in vals] vals = [self.preprocess(v) for v in vals]
data = func(data, *vals) data = func(data, *vals)
return data return data
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:3b2a3605 tags:
This `Operator` class provides an interface and logic to execute a chain of This `Operator` class provides an interface and logic to execute a chain of
operations - an operation is some function which accepts one or more inputs, operations - an operation is some function which accepts one or more inputs,
and produce one output. and produce one output.
But it stops short of defining any operations. Instead, we can create another But it stops short of defining any operations. Instead, we can create another
class - a sub-class - which derives from the `Operator` class. This sub-class class - a sub-class - which derives from the `Operator` class. This sub-class
will define the operations that will ultimately be executed by the `Operator` will define the operations that will ultimately be executed by the `Operator`
class. All that the `Operator` class does is: class. All that the `Operator` class does is:
- Allow functions to be registered with the `addFunction` method - all - Allow functions to be registered with the `addFunction` method - all
registered functions can be used via the `do` method. registered functions can be used via the `do` method.
- Stage an operation (using a registered function) via the `do` method. Note - Stage an operation (using a registered function) via the `do` method. Note
that `do` allows any number of values to be passed to it, as we used the `*` that `do` allows any number of values to be passed to it, as we used the `*`
operator when specifying the `values` argument. operator when specifying the `values` argument.
- Run all staged operations via the `run` method - it passes an input through - Run all staged operations via the `run` method - it passes an input through
all of the operations that have been staged, and then returns the final all of the operations that have been staged, and then returns the final
result. result.
Let's define a sub-class: Let's define a sub-class:
%% Cell type:code id: tags: %% Cell type:code id:141071fa tags:
``` ```
class NumberOperator(Operator): class NumberOperator(Operator):
def __init__(self): def __init__(self):
super().__init__() super().__init__()
self.addFunction('add', self.add) self.addFunction('add', self.add)
self.addFunction('mul', self.mul) self.addFunction('mul', self.mul)
self.addFunction('negate', self.negate) self.addFunction('negate', self.negate)
def preprocess(self, value): def preprocess(self, value):
return float(value) return float(value)
def add(self, a, b): def add(self, a, b):
return a + b return a + b
def mul(self, a, b): def mul(self, a, b):
return a * b return a * b
def negate(self, a): def negate(self, a):
return -a return -a
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:f71cf981 tags:
The `NumberOperator` is a sub-class of `Operator`, which we can use for basic The `NumberOperator` is a sub-class of `Operator`, which we can use for basic
numerical calculations. It provides a handful of simple numerical methods, but numerical calculations. It provides a handful of simple numerical methods, but
the most interesting stuff is inside `__init__`. the most interesting stuff is inside `__init__`.
> ``` > ```
> super().__init__() > super().__init__()
> ``` > ```
This line invokes `Operator.__init__` - the initialisation method for the This line invokes `Operator.__init__` - the initialisation method for the
`Operator` base-class. `Operator` base-class.
In Python, we can use the [built-in `super` In Python, we can use the [built-in `super`
method](https://docs.python.org/3/library/functions.html#super) to take care method](https://docs.python.org/3/library/functions.html#super) to take care
of correctly calling methods that are defined in an object's base-class (or of correctly calling methods that are defined in an object's base-class (or
classes, in the case of [multiple inheritance](multiple-inheritance)). classes, in the case of [multiple inheritance](multiple-inheritance)).
> The `super` function is one thing which changed between Python 2 and 3 - > The `super` function is one thing which changed between Python 2 and 3 -
> in Python 2, it was necessary to pass both the type and the instance > in Python 2, it was necessary to pass both the type and the instance
> to `super`. So it is common to see code that looks like this: > to `super`. So it is common to see code that looks like this:
> >
> ``` > ```
> def __init__(self): > def __init__(self):
> super(NumberOperator, self).__init__() > super(NumberOperator, self).__init__()
> ``` > ```
> >
> Fortunately things are a lot cleaner in Python 3. > Fortunately things are a lot cleaner in Python 3.
Let's move on to the next few lines in `__init__`: Let's move on to the next few lines in `__init__`:
> ``` > ```
> self.addFunction('add', self.add) > self.addFunction('add', self.add)
> self.addFunction('mul', self.mul) > self.addFunction('mul', self.mul)
> self.addFunction('negate', self.negate) > self.addFunction('negate', self.negate)
> ``` > ```
Here we are registering all of the functionality that is provided by the Here we are registering all of the functionality that is provided by the
`NumberOperator` class, via the `Operator.addFunction` method. `NumberOperator` class, via the `Operator.addFunction` method.
The `NumberOperator` class has also overridden the `preprocess` method, to The `NumberOperator` class has also overridden the `preprocess` method, to
ensure that all values handled by the `Operator` are numbers. This method gets ensure that all values handled by the `Operator` are numbers. This method gets
called within the `Operator.run` method - for a `NumberOperator` instance, the called within the `Operator.run` method - for a `NumberOperator` instance, the
`NumberOperator.preprocess` method will get called<sup>3</sup>. `NumberOperator.preprocess` method will get called<sup>3</sup>.
> <sup>3</sup> When a sub-class overrides a base-class method, it is still > <sup>3</sup> When a sub-class overrides a base-class method, it is still
> possible to access the base-class implementation [via the `super()` > possible to access the base-class implementation [via the `super()`
> function](https://stackoverflow.com/a/4747427) (the preferred method), or by > function](https://stackoverflow.com/a/4747427) (the preferred method), or by
> [explicitly calling the base-class > [explicitly calling the base-class
> implementation](https://stackoverflow.com/a/2421325). > implementation](https://stackoverflow.com/a/2421325).
Now let's see what our `NumberOperator` class does: Now let's see what our `NumberOperator` class does:
%% Cell type:code id: tags: %% Cell type:code id:c22efaf7 tags:
``` ```
no = NumberOperator() no = NumberOperator()
no.do('add', 10) no.do('add', 10)
no.do('mul', 2) no.do('mul', 2)
no.do('negate') no.do('negate')
print('Operations on {}: {}'.format(10, no.run(10))) print('Operations on {}: {}'.format(10, no.run(10)))
print('Operations on {}: {}'.format(2.5, no.run(5))) print('Operations on {}: {}'.format(2.5, no.run(5)))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:48651b8a tags:
It works! While this is a contrived example, hopefully you can see how It works! While this is a contrived example, hopefully you can see how
inheritance can be used to break a problem down into sub-problems: inheritance can be used to break a problem down into sub-problems:
- The `Operator` class provides all of the logic needed to manage and execute - The `Operator` class provides all of the logic needed to manage and execute
operations, without caring about what those operations are actually doing. operations, without caring about what those operations are actually doing.
- This leaves the `NumberOperator` class free to concentrate on implementing - This leaves the `NumberOperator` class free to concentrate on implementing
the functions that are specific to its task, and not having to worry about the functions that are specific to its task, and not having to worry about
how they are executed. how they are executed.
We could also easily implement other `Operator` sub-classes to work on We could also easily implement other `Operator` sub-classes to work on
different data types, such as arrays, images, or even non-numeric data such as different data types, such as arrays, images, or even non-numeric data such as
strings: strings:
%% Cell type:code id: tags: %% Cell type:code id:ef0836da tags:
``` ```
class StringOperator(Operator): class StringOperator(Operator):
def __init__(self): def __init__(self):
super().__init__() super().__init__()
self.addFunction('capitalise', self.capitalise) self.addFunction('capitalise', self.capitalise)
self.addFunction('concat', self.concat) self.addFunction('concat', self.concat)
def preprocess(self, value): def preprocess(self, value):
return str(value) return str(value)
def capitalise(self, s): def capitalise(self, s):
return ' '.join([w[0].upper() + w[1:] for w in s.split()]) return ' '.join([w[0].upper() + w[1:] for w in s.split()])
def concat(self, s1, s2): def concat(self, s1, s2):
return s1 + s2 return s1 + s2
so = StringOperator() so = StringOperator()
so.do('capitalise') so.do('capitalise')
so.do('concat', '!') so.do('concat', '!')
print(so.run('python is an ok language')) print(so.run('python is an ok language'))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:32a7a008 tags:
<a class="anchor" id="polymorphism"></a> <a class="anchor" id="polymorphism"></a>
### Polymorphism ### Polymorphism
Inheritance also allows us to take advantage of *polymorphism*, which refers Inheritance also allows us to take advantage of *polymorphism*, which refers
to the idea that, in an object-oriented language, we should be able to use an to the idea that, in an object-oriented language, we should be able to use an
object without having complete knowledge about the class, or type, of that object without having complete knowledge about the class, or type, of that
object. For example, we should be able to write a function which expects an object. For example, we should be able to write a function which expects an
`Operator` instance, but which will work on an instance of any `Operator` `Operator` instance, but which will work on an instance of any `Operator`
sub-classs. As an example, let's write a function which prints a summary of an sub-classs. As an example, let's write a function which prints a summary of an
`Operator` instance: `Operator` instance:
%% Cell type:code id: tags: %% Cell type:code id:8c1c604c tags:
``` ```
def operatorSummary(o): def operatorSummary(o):
print(type(o).__name__) print(type(o).__name__)
print(' All functions: ') print(' All functions: ')
for fname in o.functions.keys(): for fname in o.functions.keys():
print(' ', fname) print(' ', fname)
print(' Staged operations: ') print(' Staged operations: ')
for i, (fname, vals) in enumerate(o.operations): for i, (fname, vals) in enumerate(o.operations):
vals = ', '.join([str(v) for v in vals]) vals = ', '.join([str(v) for v in vals])
print(f' {i + 1}: {fname}({vals})') print(f' {i + 1}: {fname}({vals})')
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:9eaa76f5 tags:
Because the `operatorSummary` function only uses methods that are defined Because the `operatorSummary` function only uses methods that are defined
in the `Operator` base-class, we can use it on _any_ `Operator` instance, in the `Operator` base-class, we can use it on _any_ `Operator` instance,
regardless of its specific type: regardless of its specific type:
%% Cell type:code id: tags: %% Cell type:code id:cfde36c6 tags:
``` ```
operatorSummary(no) operatorSummary(no)
operatorSummary(so) operatorSummary(so)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:8bb37b38 tags:
<a class="anchor" id="multiple-inheritance"></a> <a class="anchor" id="multiple-inheritance"></a>
### Multiple inheritance ### Multiple inheritance
Python allows you to define a class which has multiple base classes - this is Python allows you to define a class which has multiple base classes - this is
known as _multiple inheritance_. For example, we might want to build a known as _multiple inheritance_. For example, we might want to build a
notification mechanisim into our `StringOperator` class, so that listeners can notification mechanisim into our `StringOperator` class, so that listeners can
be notified whenever the `capitalise` method gets called. It so happens that be notified whenever the `capitalise` method gets called. It so happens that
our old colleague of `Operator` class fame also wrote a `Notifier` class which our old colleague of `Operator` class fame also wrote a `Notifier` class which
allows listeners to register to be notified when an event occurs: allows listeners to register to be notified when an event occurs:
%% Cell type:code id: tags: %% Cell type:code id:594669f4 tags:
``` ```
class Notifier(object): class Notifier:
def __init__(self): def __init__(self):
super().__init__() super().__init__()
self.__listeners = {} self.__listeners = {}
def register(self, name, func): def register(self, name, func):
self.__listeners[name] = func self.__listeners[name] = func
def notify(self, *args, **kwargs): def notify(self, *args, **kwargs):
for func in self.__listeners.values(): for func in self.__listeners.values():
func(*args, **kwargs) func(*args, **kwargs)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:a9e8233d tags:
Let's modify the `StringOperator` class to use the functionality of the Let's modify the `StringOperator` class to use the functionality of the
`Notifier ` class: `Notifier ` class:
%% Cell type:code id: tags: %% Cell type:code id:5e84e72e tags:
``` ```
class StringOperator(Operator, Notifier): class StringOperator(Operator, Notifier):
def __init__(self): def __init__(self):
super().__init__() super().__init__()
self.addFunction('capitalise', self.capitalise) self.addFunction('capitalise', self.capitalise)
self.addFunction('concat', self.concat) self.addFunction('concat', self.concat)
def preprocess(self, value): def preprocess(self, value):
return str(value) return str(value)
def capitalise(self, s): def capitalise(self, s):
result = ' '.join([w[0].upper() + w[1:] for w in s.split()]) result = ' '.join([w[0].upper() + w[1:] for w in s.split()])
self.notify(result) self.notify(result)
return result return result
def concat(self, s1, s2): def concat(self, s1, s2):
return s1 + s2 return s1 + s2
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:bad6d952 tags:
Now, anything which is interested in uses of the `capitalise` method can Now, anything which is interested in uses of the `capitalise` method can
register as a listener on a `StringOperator` instance: register as a listener on a `StringOperator` instance:
%% Cell type:code id: tags: %% Cell type:code id:fe44243a tags:
``` ```
so = StringOperator() so = StringOperator()
def capitaliseCalled(result): def capitaliseCalled(result):
print('Capitalise operation called:', result) print('Capitalise operation called:', result)
so.register('mylistener', capitaliseCalled) so.register('mylistener', capitaliseCalled)
so.do('capitalise') so.do('capitalise')
so.do('concat', '?') so.do('concat', '?')
print(so.run('did you notice that functions are objects too')) print(so.run('did you notice that functions are objects too'))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:7d97afd0 tags:
> Simple classes such as the `Notifier` are sometimes referred to as > Simple classes such as the `Notifier` are sometimes referred to as
> [_mixins_](https://en.wikipedia.org/wiki/Mixin). > [_mixins_](https://en.wikipedia.org/wiki/Mixin).
If you wish to use multiple inheritance in your design, it is important to be If you wish to use multiple inheritance in your design, it is important to be
aware of the mechanism that Python uses to determine how base class methods aware of the mechanism that Python uses to determine how base class methods
are called (and which base class method will be called, in the case of naming are called (and which base class method will be called, in the case of naming
conflicts). This is referred to as the Method Resolution Order (MRO) - further conflicts). This is referred to as the Method Resolution Order (MRO) - further
details on the topic can be found details on the topic can be found
[here](https://www.python.org/download/releases/2.3/mro/), and a more concise [here](https://www.python.org/download/releases/2.3/mro/), and a more concise
summary summary
[here](http://python-history.blogspot.co.uk/2010/06/method-resolution-order.html). [here](http://python-history.blogspot.co.uk/2010/06/method-resolution-order.html).
Note also that for base class `__init__` methods to be correctly called in a Note also that for base class `__init__` methods to be correctly called in a
design which uses multiple inheritance, _all_ classes in the hierarchy must design which uses multiple inheritance, _all_ classes in the hierarchy must
invoke `super().__init__()`. This can become complicated when some base invoke `super().__init__()`. This can become complicated when some base
classes expect to be passed arguments to their `__init__` method. In scenarios classes expect to be passed arguments to their `__init__` method. In scenarios
like this it may be prefereable to manually invoke the base class `__init__` like this it may be prefereable to manually invoke the base class `__init__`
methods instead of using `super()`. For example: methods instead of using `super()`. For example:
> ``` > ```
> class StringOperator(Operator, Notifier): > class StringOperator(Operator, Notifier):
> def __init__(self): > def __init__(self):
> Operator.__init__(self) > Operator.__init__(self)
> Notifier.__init__(self) > Notifier.__init__(self)
> ``` > ```
This approach has the disadvantage that if the base classes change, you will This approach has the disadvantage that if the base classes change, you will
have to change these invocations. But the advantage is that you know exactly have to change these invocations. But the advantage is that you know exactly
how the class hierarchy will be initialised. In general though, doing how the class hierarchy will be initialised. In general though, doing
everything with `super()` will result in more maintainable code. everything with `super()` will result in more maintainable code.
<a class="anchor" id="class-attributes-and-methods"></a> <a class="anchor" id="class-attributes-and-methods"></a>
## Class attributes and methods ## Class attributes and methods
Up to this point we have been covering how to add attributes and methods to an Up to this point we have been covering how to add attributes and methods to an
_object_. But it is also possible to add methods and attributes to a _class_ _object_. But it is also possible to add methods and attributes to a _class_
(`static` methods and fields, for those of you familiar with C++ or Java). (`static` methods and fields, for those of you familiar with C++ or Java).
Class attributes and methods can be accessed without having to create an Class attributes and methods can be accessed without having to create an
instance of the class - they are not associated with individual objects, but instance of the class - they are not associated with individual objects, but
rather with the class itself. rather with the class itself.
Class methods and attributes can be useful in several scenarios - as a Class methods and attributes can be useful in several scenarios - as a
hypothetical, not very useful example, let's say that we want to gain usage hypothetical, not very useful example, let's say that we want to gain usage
statistics for how many times each type of operation is used on instances of statistics for how many times each type of operation is used on instances of
our `FSLMaths` class. We might, for example, use this information in a grant our `FSLMaths` class. We might, for example, use this information in a grant
application to show evidence that more research is needed to optimise the application to show evidence that more research is needed to optimise the
performance of the `add` operation. performance of the `add` operation.
<a class="anchor" id="class-attributes"></a> <a class="anchor" id="class-attributes"></a>
### Class attributes ### Class attributes
Let's add a `dict` called `opCounters` as a class attribute to the `FSLMaths` Let's add a `dict` called `opCounters` as a class attribute to the `FSLMaths`
class - whenever an operation is called on a `FSLMaths` instance, the counter class - whenever an operation is called on a `FSLMaths` instance, the counter
for that operation will be incremented: for that operation will be incremented:
%% Cell type:code id: tags: %% Cell type:code id:fa913af1 tags:
``` ```
import numpy as np import numpy as np
import nibabel as nib import nibabel as nib
class FSLMaths(object): class FSLMaths:
# It's this easy to add a class-level # It's this easy to add a class-level
# attribute. This dict is associated # attribute. This dict is associated
# with the FSLMaths *class*, not with # with the FSLMaths *class*, not with
# any individual FSLMaths instance. # any individual FSLMaths instance.
opCounters = {} opCounters = {}
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
self.operations = [] self.operations = []
def add(self, value): def add(self, value):
self.operations.append(('add', value)) self.operations.append(('add', value))
return self return self
def mul(self, value): def mul(self, value):
self.operations.append(('mul', value)) self.operations.append(('mul', value))
return self return self
def div(self, value): def div(self, value):
self.operations.append(('div', value)) self.operations.append(('div', value))
return self return self
def run(self, output=None): def run(self, output=None):
data = np.array(self.img.get_fdata()) data = np.array(self.img.get_fdata())
for oper, value in self.operations: for oper, value in self.operations:
# Code omitted for brevity # Code omitted for brevity
# Increment the usage counter for this operation. We can # Increment the usage counter for this operation. We can
# access class attributes (and methods) through the class # access class attributes (and methods) through the class
# itself, as shown here. # itself, as shown here.
FSLMaths.opCounters[oper] = FSLMaths.opCounters.get(oper, 0) + 1 FSLMaths.opCounters[oper] = FSLMaths.opCounters.get(oper, 0) + 1
# It is also possible to access class-level # It is also possible to access class-level
# attributes via instances of the class, e.g. # attributes via instances of the class, e.g.
# self.opCounters[oper] = self.opCounters.get(oper, 0) + 1 # self.opCounters[oper] = self.opCounters.get(oper, 0) + 1
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:b6772715 tags:
So let's see it in action: So let's see it in action:
%% Cell type:code id: tags: %% Cell type:code id:853b3401 tags:
``` ```
fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz') fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')
fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz') fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz')
inimg = nib.load(fpath) inimg = nib.load(fpath)
mask = nib.load(fmask) mask = nib.load(fmask)
FSLMaths(inimg).mul(mask).add(25).run() FSLMaths(inimg).mul(mask).add(25).run()
FSLMaths(inimg).add(15).div(1.5).run() FSLMaths(inimg).add(15).div(1.5).run()
print('FSLMaths usage statistics') print('FSLMaths usage statistics')
for oper in ('add', 'div', 'mul'): for oper in ('add', 'div', 'mul'):
print(' {} : {}'.format(oper, FSLMaths.opCounters.get(oper, 0))) print(' {} : {}'.format(oper, FSLMaths.opCounters.get(oper, 0)))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:f384f558 tags:
<a class="anchor" id="class-methods"></a> <a class="anchor" id="class-methods"></a>
### Class methods ### Class methods
It is just as easy to add a method to a class - let's take our reporting code It is just as easy to add a method to a class - let's take our reporting code
from above, and add it as a method to the `FSLMaths` class. from above, and add it as a method to the `FSLMaths` class.
A class method is denoted by the A class method is denoted by the
[`@classmethod`](https://docs.python.org/3.5/library/functions.html#classmethod) [`@classmethod`](https://docs.python.org/3.5/library/functions.html#classmethod)
decorator. Note that, where a regular method which is called on an instance decorator. Note that, where a regular method which is called on an instance
will be passed the instance as its first argument (`self`), a class method will be passed the instance as its first argument (`self`), a class method
will be passed the class itself as the first argument - the standard will be passed the class itself as the first argument - the standard
convention is to call this argument `cls`: convention is to call this argument `cls`:
%% Cell type:code id: tags: %% Cell type:code id:cab7d8eb tags:
``` ```
class FSLMaths(object): class FSLMaths:
opCounters = {} opCounters = {}
@classmethod @classmethod
def usage(cls): def usage(cls):
print('FSLMaths usage statistics') print('FSLMaths usage statistics')
for oper in ('add', 'div', 'mul'): for oper in ('add', 'div', 'mul'):
print(' {} : {}'.format(oper, FSLMaths.opCounters.get(oper, 0))) print(' {} : {}'.format(oper, FSLMaths.opCounters.get(oper, 0)))
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
self.operations = [] self.operations = []
def add(self, value): def add(self, value):
self.operations.append(('add', value)) self.operations.append(('add', value))
return self return self
def mul(self, value): def mul(self, value):
self.operations.append(('mul', value)) self.operations.append(('mul', value))
return self return self
def div(self, value): def div(self, value):
self.operations.append(('div', value)) self.operations.append(('div', value))
return self return self
def run(self, output=None): def run(self, output=None):
data = np.array(self.img.get_fdata()) data = np.array(self.img.get_fdata())
for oper, value in self.operations: for oper, value in self.operations:
FSLMaths.opCounters[oper] = self.opCounters.get(oper, 0) + 1 FSLMaths.opCounters[oper] = self.opCounters.get(oper, 0) + 1
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:7a306d6f tags:
> There is another decorator - > There is another decorator -
> [`@staticmethod`](https://docs.python.org/3.5/library/functions.html#staticmethod) - > [`@staticmethod`](https://docs.python.org/3.5/library/functions.html#staticmethod) -
> which can be used on methods defined within a class. The difference > which can be used on methods defined within a class. The difference
> between a `@classmethod` and a `@staticmethod` is that the latter will *not* > between a `@classmethod` and a `@staticmethod` is that the latter will *not*
> be passed the class (`cls`). > be passed the class (`cls`).
Calling a class method is the same as accessing a class attribute: Calling a class method is the same as accessing a class attribute:
%% Cell type:code id: tags: %% Cell type:code id:0b04e158 tags:
``` ```
fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz') fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')
fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz') fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz')
inimg = nib.load(fpath) inimg = nib.load(fpath)
mask = nib.load(fmask) mask = nib.load(fmask)
fm1 = FSLMaths(inimg).mul(mask).add(25) fm1 = FSLMaths(inimg).mul(mask).add(25)
fm2 = FSLMaths(inimg).add(15).div(1.5) fm2 = FSLMaths(inimg).add(15).div(1.5)
fm1.run() fm1.run()
fm2.run() fm2.run()
FSLMaths.usage() FSLMaths.usage()
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:4ad5de79 tags:
Note that it is also possible to access class attributes and methods through Note that it is also possible to access class attributes and methods through
instances: instances:
%% Cell type:code id: tags: %% Cell type:code id:4f09c8a3 tags:
``` ```
print(fm1.opCounters) print(fm1.opCounters)
fm1.usage() fm1.usage()
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:b7ba05c1 tags:
<a class="anchor" id="appendix-the-object-base-class"></a> <a class="anchor" id="appendix-the-object-base-class"></a>
## Appendix: The `object` base-class ## Appendix: The `object` base-class
When you are defining a class, you need to specify the base-class from which When you are defining a class, you need to specify the base-class from which
your class inherits. If your class does not inherit from a particular class, your class inherits. If your class does not inherit from a particular class,
then it should inherit from the built-in `object` class: then it implicitly inherits from the built-in `object` class:
> ``` > ```
> class MyClass(object): > class MyClass:
> ... > ...
> ``` > ```
However, in older code bases, you might see class definitions that look like In older code bases, you might see class definitions that look like this,
this, without explicitly inheriting from the `object` base class: which explicitly inherit from the `object` base class:
> ``` > ```
> class MyClass: > class MyClass(object):
> ... > ...
> ``` > ```
This syntax is a [throwback to older versions of In Python 3 there is actually no difference between these two approaches. The
Python](https://docs.python.org/2/reference/datamodel.html#new-style-and-classic-classes). latter syntax is a [throwback to older versions of
In Python 3 there is actually no difference in defining classes in the Python](https://docs.python.org/2/reference/datamodel.html#new-style-and-classic-classes); you can typically use either approach, and don't need to worry too much.
"new-style" way we have used throughout this tutorial, or the "old-style" way
mentioned in this appendix.
But if you are writing code which needs to run on both Python 2 and 3, you However, if you are writing code which needs to run on both Python 2 and 3, you
__must__ define your classes in the new-style convention, i.e. by explicitly __must__ define your classes in the new-style convention, i.e. by explicitly
inheriting from the `object` base class. Therefore, the safest approach is to inheriting from the `object` base class.
always use the new-style format.
<a class="anchor" id="appendix-init-versus-new"></a> <a class="anchor" id="appendix-init-versus-new"></a>
## Appendix: `__init__` versus `__new__` ## Appendix: `__init__` versus `__new__`
In Python, object creation is actually a two-stage process - *creation*, and In Python, object creation is actually a two-stage process - *creation*, and
then *initialisation*. The `__init__` method gets called during the then *initialisation*. The `__init__` method gets called during the
*initialisation* stage - its job is to initialise the state of the object. But *initialisation* stage - its job is to initialise the state of the object. But
note that, by the time `__init__` gets called, the object has already been note that, by the time `__init__` gets called, the object has already been
created. created.
You can also define a method called `__new__` if you need to control the You can also define a method called `__new__` if you need to control the
creation stage, although this is very rarely needed. One example of where you creation stage, although this is very rarely needed. One example of where you
might need to implement the `__new__` method is if you wish to create a might need to implement the `__new__` method is if you wish to create a
[subclass of a [subclass of a
`numpy.array`](https://docs.scipy.org/doc/numpy-1.14.0/user/basics.subclassing.html) `numpy.array`](https://docs.scipy.org/doc/numpy-1.14.0/user/basics.subclassing.html)
(although you might alternatively want to think about redefining your problem (although you might alternatively want to think about redefining your problem
so that this is not necessary). so that this is not necessary).
A brief explanation on A brief explanation on
the difference between `__new__` and `__init__` can be found the difference between `__new__` and `__init__` can be found
[here](https://www.reddit.com/r/learnpython/comments/2s3pms/what_is_the_difference_between_init_and_new/cnm186z/), [here](https://www.reddit.com/r/learnpython/comments/2s3pms/what_is_the_difference_between_init_and_new/cnm186z/),
and you may also wish to take a look at the [official Python and you may also wish to take a look at the [official Python
docs](https://docs.python.org/3/reference/datamodel.html#basic-customization). docs](https://docs.python.org/3/reference/datamodel.html#basic-customization).
<a class="anchor" id="appendix-monkey-patching"></a> <a class="anchor" id="appendix-monkey-patching"></a>
## Appendix: Monkey-patching ## Appendix: Monkey-patching
The act of run-time modification of objects or class definitions is referred The act of run-time modification of objects or class definitions is referred
to as [*monkey-patching*](https://en.wikipedia.org/wiki/Monkey_patch) and, to as [*monkey-patching*](https://en.wikipedia.org/wiki/Monkey_patch) and,
whilst it is allowed by the Python programming language, it is generally whilst it is allowed by the Python programming language, it is generally
considered quite bad practice. considered quite bad practice.
Just because you *can* do something doesn't mean that you *should*. Python Just because you *can* do something doesn't mean that you *should*. Python
gives you the flexibility to write your software in whatever manner you deem gives you the flexibility to write your software in whatever manner you deem
suitable. **But** if you want to write software that will be used, adopted, suitable. **But** if you want to write software that will be used, adopted,
maintained, and enjoyed by other people, you should be polite, write your code maintained, and enjoyed by other people, you should be polite, write your code
in a clear, readable fashion, and avoid the use of devious tactics such as in a clear, readable fashion, and avoid the use of devious tactics such as
monkey-patching. monkey-patching.
**However**, while monkey-patching may seem like a horrific programming **However**, while monkey-patching may seem like a horrific programming
practice to those of you coming from the realms of C++, Java, and the like, practice to those of you coming from the realms of C++, Java, and the like,
(and it is horrific in many cases), it can be *extremely* useful in certain (and it is horrific in many cases), it can be *extremely* useful in certain
circumstances. For instance, monkey-patching makes [unit testing a circumstances. For instance, monkey-patching makes [unit testing a
breeze in Python](https://docs.python.org/3/library/unittest.mock.html). breeze in Python](https://docs.python.org/3/library/unittest.mock.html).
As another example, consider the scenario where you are dependent on a third As another example, consider the scenario where you are dependent on a third
party library which has bugs in it. No problem - while you are waiting for the party library which has bugs in it. No problem - while you are waiting for the
library author to release a new version of the library, you can write your own library author to release a new version of the library, you can write your own
working implementation and [monkey-patch it working implementation and [monkey-patch it
in!](https://git.fmrib.ox.ac.uk/fsl/fsleyes/fsleyes/blob/0.21.0/fsleyes/views/viewpanel.py#L726) in!](https://git.fmrib.ox.ac.uk/fsl/fsleyes/fsleyes/blob/0.21.0/fsleyes/views/viewpanel.py#L726)
<a class="anchor" id="appendix-method-overloading"></a> <a class="anchor" id="appendix-method-overloading"></a>
## Appendix: Method overloading ## Appendix: Method overloading
Method overloading (defining multiple methods with the same name in a class, Method overloading (defining multiple methods with the same name in a class,
but each accepting different arguments) is one of the only object-oriented but each accepting different arguments) is one of the only object-oriented
features that is not present in Python. Becuase Python does not perform any features that is not present in Python. Becuase Python does not perform any
runtime checks on the types of arguments that are passed to a method, or the runtime checks on the types of arguments that are passed to a method, or the
compatibility of the method to accept the arguments, it would not be possible compatibility of the method to accept the arguments, it would not be possible
to determine which implementation of a method is to be called. In other words, to determine which implementation of a method is to be called. In other words,
in Python only the name of a method is used to identify that method, unlike in in Python only the name of a method is used to identify that method, unlike in
C++ and Java, where the full method signature (name, input types and return C++ and Java, where the full method signature (name, input types and return
types) is used. types) is used.
However, because a Python method can be written to accept any number or type However, because a Python method can be written to accept any number or type
of arguments, it is very easy to to build your own overloading logic by of arguments, it is very easy to to build your own overloading logic by
writing a "dispatch" method<sup>4</sup>. Here is YACE (Yet Another Contrived writing a "dispatch" method<sup>4</sup>. Here is YACE (Yet Another Contrived
Example): Example):
%% Cell type:code id: tags: %% Cell type:code id:49cbaa37 tags:
``` ```
class Adder(object): class Adder:
def add(self, *args): def add(self, *args):
if len(args) == 2: return self.__add2(*args) if len(args) == 2: return self.__add2(*args)
elif len(args) == 3: return self.__add3(*args) elif len(args) == 3: return self.__add3(*args)
elif len(args) == 4: return self.__add4(*args) elif len(args) == 4: return self.__add4(*args)
else: else:
raise AttributeError('No method available to accept {} ' raise AttributeError('No method available to accept {} '
'arguments'.format(len(args))) 'arguments'.format(len(args)))
def __add2(self, a, b): def __add2(self, a, b):
return a + b return a + b
def __add3(self, a, b, c): def __add3(self, a, b, c):
return a + b + c return a + b + c
def __add4(self, a, b, c, d): def __add4(self, a, b, c, d):
return a + b + c + d return a + b + c + d
a = Adder() a = Adder()
print('Add two: ', a.add(1, 2)) print('Add two: ', a.add(1, 2))
print('Add three:', a.add(1, 2, 3)) print('Add three:', a.add(1, 2, 3))
print('Add four: ', a.add(1, 2, 3, 4)) print('Add four: ', a.add(1, 2, 3, 4))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id:77f05681 tags:
> <sup>4</sup>Another option is the [`functools.singledispatch` > <sup>4</sup>Another option is the [`functools.singledispatch`
> decorator](https://docs.python.org/3/library/functools.html#functools.singledispatch), > decorator](https://docs.python.org/3/library/functools.html#functools.singledispatch),
> which is more complicated, but may allow you to write your dispatch logic in > which is more complicated, but may allow you to write your dispatch logic in
> a more concise manner. > a more concise manner.
<a class="anchor" id="useful-references"></a> <a class="anchor" id="useful-references"></a>
## Useful references ## Useful references
The official Python documentation has a wealth of information on the internal The official Python documentation has a wealth of information on the internal
workings of classes and objects, so these pages are worth a read: workings of classes and objects, so these pages are worth a read:
* https://docs.python.org/3/tutorial/classes.html * https://docs.python.org/3/tutorial/classes.html
* https://docs.python.org/3/reference/datamodel.html * https://docs.python.org/3/reference/datamodel.html
......
...@@ -111,14 +111,23 @@ developing a class which can be used in place of the `fslmaths` shell command. ...@@ -111,14 +111,23 @@ developing a class which can be used in place of the `fslmaths` shell command.
``` ```
class FSLMaths(object): class FSLMaths:
pass pass
``` ```
In this statement, we defined a new class called `FSLMaths`, which inherits In this statement, we defined a new class called `FSLMaths`.
from the built-in `object` base-class (see [below](inheritance) for more
details on inheritance). > You may also often see code which looks like this:
> ```
> class FSLMaths(object):
> ...
> ```
>
> Here we have defined the `FSLMaths` class to inherit from the built-in
> `object` base-class. This is a throw-back to Python 2, and you can think of
> both patterns to be essentially equivalent. See [below](inheritance) for more
details on inheritance.
Now that we have defined our class, we can create objects - instances of that Now that we have defined our class, we can create objects - instances of that
...@@ -146,7 +155,7 @@ It makes sense to pass this in when we create an `FSLMaths` object: ...@@ -146,7 +155,7 @@ It makes sense to pass this in when we create an `FSLMaths` object:
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
``` ```
...@@ -190,7 +199,7 @@ class like so: ...@@ -190,7 +199,7 @@ class like so:
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
...@@ -261,7 +270,7 @@ image on creation: ...@@ -261,7 +270,7 @@ image on creation:
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
...@@ -320,7 +329,7 @@ functionality: ...@@ -320,7 +329,7 @@ functionality:
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
...@@ -361,7 +370,7 @@ in time. So let's add another method, `run`, which actually does the work: ...@@ -361,7 +370,7 @@ in time. So let's add another method, `run`, which actually does the work:
import numpy as np import numpy as np
import nibabel as nib import nibabel as nib
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
...@@ -450,7 +459,7 @@ to do is return `self` from each method: ...@@ -450,7 +459,7 @@ to do is return `self` from each method:
import numpy as np import numpy as np
import nibabel as nib import nibabel as nib
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.img = inimg self.img = inimg
...@@ -611,7 +620,7 @@ our sloppy research assistant from overwriting the `img` attribute: ...@@ -611,7 +620,7 @@ our sloppy research assistant from overwriting the `img` attribute:
``` ```
# remainder of definition omitted for brevity # remainder of definition omitted for brevity
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.__img = inimg self.__img = inimg
self.__operations = [] self.__operations = []
...@@ -640,7 +649,7 @@ cover decorators in a later practical). ...@@ -640,7 +649,7 @@ cover decorators in a later practical).
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.__img = inimg self.__img = inimg
self.__operations = [] self.__operations = []
...@@ -674,7 +683,7 @@ image after creation. ...@@ -674,7 +683,7 @@ image after creation.
``` ```
class FSLMaths(object): class FSLMaths:
def __init__(self, inimg): def __init__(self, inimg):
self.__img = None self.__img = None
self.__operations = [] self.__operations = []
...@@ -740,7 +749,7 @@ the surgery: ...@@ -740,7 +749,7 @@ the surgery:
``` ```
class Animal(object): class Animal:
def noiseMade(self): def noiseMade(self):
raise NotImplementedError('This method must be ' raise NotImplementedError('This method must be '
'implemented by sub-classes') 'implemented by sub-classes')
...@@ -813,7 +822,7 @@ example. Imagine that a former colleague had written a class called ...@@ -813,7 +822,7 @@ example. Imagine that a former colleague had written a class called
``` ```
class Operator(object): class Operator:
def __init__(self): def __init__(self):
super().__init__() # this line will be explained later super().__init__() # this line will be explained later
...@@ -1057,7 +1066,7 @@ allows listeners to register to be notified when an event occurs: ...@@ -1057,7 +1066,7 @@ allows listeners to register to be notified when an event occurs:
``` ```
class Notifier(object): class Notifier:
def __init__(self): def __init__(self):
super().__init__() super().__init__()
...@@ -1187,7 +1196,7 @@ for that operation will be incremented: ...@@ -1187,7 +1196,7 @@ for that operation will be incremented:
import numpy as np import numpy as np
import nibabel as nib import nibabel as nib
class FSLMaths(object): class FSLMaths:
# It's this easy to add a class-level # It's this easy to add a class-level
# attribute. This dict is associated # attribute. This dict is associated
...@@ -1266,7 +1275,7 @@ convention is to call this argument `cls`: ...@@ -1266,7 +1275,7 @@ convention is to call this argument `cls`:
``` ```
class FSLMaths(object): class FSLMaths:
opCounters = {} opCounters = {}
...@@ -1343,36 +1352,33 @@ fm1.usage() ...@@ -1343,36 +1352,33 @@ fm1.usage()
When you are defining a class, you need to specify the base-class from which When you are defining a class, you need to specify the base-class from which
your class inherits. If your class does not inherit from a particular class, your class inherits. If your class does not inherit from a particular class,
then it should inherit from the built-in `object` class: then it implicitly inherits from the built-in `object` class:
> ``` > ```
> class MyClass(object): > class MyClass:
> ... > ...
> ``` > ```
However, in older code bases, you might see class definitions that look like In older code bases, you might see class definitions that look like this,
this, without explicitly inheriting from the `object` base class: which explicitly inherit from the `object` base class:
> ``` > ```
> class MyClass: > class MyClass(object):
> ... > ...
> ``` > ```
This syntax is a [throwback to older versions of In Python 3 there is actually no difference between these two approaches. The
Python](https://docs.python.org/2/reference/datamodel.html#new-style-and-classic-classes). latter syntax is a [throwback to older versions of
In Python 3 there is actually no difference in defining classes in the Python](https://docs.python.org/2/reference/datamodel.html#new-style-and-classic-classes); you can typically use either approach, and don't need to worry too much.
"new-style" way we have used throughout this tutorial, or the "old-style" way
mentioned in this appendix.
But if you are writing code which needs to run on both Python 2 and 3, you However, if you are writing code which needs to run on both Python 2 and 3, you
__must__ define your classes in the new-style convention, i.e. by explicitly __must__ define your classes in the new-style convention, i.e. by explicitly
inheriting from the `object` base class. Therefore, the safest approach is to inheriting from the `object` base class.
always use the new-style format.
<a class="anchor" id="appendix-init-versus-new"></a> <a class="anchor" id="appendix-init-versus-new"></a>
...@@ -1456,7 +1462,7 @@ Example): ...@@ -1456,7 +1462,7 @@ Example):
``` ```
class Adder(object): class Adder:
def add(self, *args): def add(self, *args):
if len(args) == 2: return self.__add2(*args) if len(args) == 2: return self.__add2(*args)
......
%% Cell type:markdown id:12ef343d tags: %% Cell type:markdown id:a19ed815 tags:
# Threading and parallel processing # Threading and parallel processing
The Python language has built-in support for multi-threading in the The Python language has built-in support for multi-threading in the
[`threading`](https://docs.python.org/3/library/threading.html) module, and [`threading`](https://docs.python.org/3/library/threading.html) module, and
true parallelism in the true parallelism in the
[`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html)
module. If you want to be impressed, skip straight to the section on module. If you want to be impressed, skip straight to the section on
[`multiprocessing`](multiprocessing). [`multiprocessing`](multiprocessing).
> *Note*: This notebook covers features that are built-in to the Python > *Note*: This notebook covers features that are built-in to the Python
> programming language. However, there are many other parallelisation options > programming language. However, there are many other parallelisation options
> available to you through third-party libraries - some of them are covered in `applications/parallel/parallel.ipynb`. > available to you through third-party libraries - some of them are covered in `applications/parallel/parallel.ipynb`.
> *Note*: If you are familiar with a "real" programming language such as C++ > *Note*: If you are familiar with a "real" programming language such as C++
> or Java, you might be disappointed with the native support for parallelism in > or Java, you might be disappointed with the native support for parallelism in
> Python. Python threads do not run in parallel because of the Global > Python. Python threads do not run in parallel because of the Global
> Interpreter Lock, and if you use `multiprocessing`, be prepared to either > Interpreter Lock, and if you use `multiprocessing`, be prepared to either
> bear the performance hit of copying data between processes, or jump through > bear the performance hit of copying data between processes, or jump through
> hoops order to share data between processes. > hoops order to share data between processes.
> >
> This limitation *might* be solved in a future Python release by way of > This limitation is being addressed in recent Python version, with
> [*sub-interpreters*](https://www.python.org/dev/peps/pep-0554/), but the > [_Free-threaded Python_ builds](https://docs.python.org/3/howto/free-threading-python.html),
> author of this practical is not holding his breath. > which will hopefully soon be the default behaviour.
* [Threading](#threading) * [Threading](#threading)
* [Subclassing `Thread`](#subclassing-thread) * [Subclassing `Thread`](#subclassing-thread)
* [Daemon threads](#daemon-threads) * [Daemon threads](#daemon-threads)
* [Thread synchronisation](#thread-synchronisation) * [Thread synchronisation](#thread-synchronisation)
* [`Lock`](#lock) * [`Lock`](#lock)
* [`Event`](#event) * [`Event`](#event)
* [The Global Interpreter Lock (GIL)](#the-global-interpreter-lock-gil) * [The Global Interpreter Lock (GIL)](#the-global-interpreter-lock-gil)
* [Multiprocessing](#multiprocessing) * [Multiprocessing](#multiprocessing)
* [`threading`-equivalent API](#threading-equivalent-api) * [`threading`-equivalent API](#threading-equivalent-api)
* [Higher-level API - the `multiprocessing.Pool`](#higher-level-api-the-multiprocessing-pool) * [Higher-level API - the `multiprocessing.Pool`](#higher-level-api-the-multiprocessing-pool)
* [`Pool.map`](#pool-map) * [`Pool.map`](#pool-map)
* [`Pool.apply_async`](#pool-apply-async) * [`Pool.apply_async`](#pool-apply-async)
* [Sharing data between processes](#sharing-data-between-processes) * [Sharing data between processes](#sharing-data-between-processes)
* [Memory-mapping](#memory-mapping) * [Memory-mapping](#memory-mapping)
* [Read-only sharing](#read-only-sharing) * [Read-only sharing](#read-only-sharing)
* [Read/write sharing](#read-write-sharing) * [Read/write sharing](#read-write-sharing)
<a class="anchor" id="threading"></a> <a class="anchor" id="threading"></a>
## Threading ## Threading
The [`threading`](https://docs.python.org/3/library/threading.html) module The [`threading`](https://docs.python.org/3/library/threading.html) module
provides a traditional multi-threading API that should be familiar to you if provides a traditional multi-threading API that should be familiar to you if
you have worked with threads in other languages. you have worked with threads in other languages.
Running a task in a separate thread in Python is easy - simply create a Running a task in a separate thread in Python is easy - simply create a
`Thread` object, and pass it the function or method that you want it to `Thread` object, and pass it the function or method that you want it to
run. Then call its `start` method: run. Then call its `start` method:
%% Cell type:code id:956c477f tags: %% Cell type:code id:50172fe8 tags:
``` ```
import time import time
import threading import threading
def longRunningTask(niters): def longRunningTask(niters):
for i in range(niters): for i in range(niters):
if i % 2 == 0: print('Tick') if i % 2 == 0: print('Tick')
else: print('Tock') else: print('Tock')
time.sleep(0.5) time.sleep(0.5)
t = threading.Thread(target=longRunningTask, args=(8,)) t = threading.Thread(target=longRunningTask, args=(8,))
t.start() t.start()
while t.is_alive(): while t.is_alive():
time.sleep(0.4) time.sleep(0.4)
print('Waiting for thread to finish...') print('Waiting for thread to finish...')
print('Finished!') print('Finished!')
``` ```
%% Cell type:markdown id:c7f0f9ad tags: %% Cell type:markdown id:f9f0d52d tags:
You can also `join` a thread, which will block execution in the current thread You can also `join` a thread, which will block execution in the current thread
until the thread that has been `join`ed has finished: until the thread that has been `join`ed has finished:
%% Cell type:code id:f6e3d5e6 tags: %% Cell type:code id:5389e92d tags:
``` ```
t = threading.Thread(target=longRunningTask, args=(6, )) t = threading.Thread(target=longRunningTask, args=(6, ))
t.start() t.start()
print('Joining thread ...') print('Joining thread ...')
t.join() t.join()
print('Finished!') print('Finished!')
``` ```
%% Cell type:markdown id:41def024 tags: %% Cell type:markdown id:2ca8ed21 tags:
<a class="anchor" id="subclassing-thread"></a> <a class="anchor" id="subclassing-thread"></a>
### Subclassing `Thread` ### Subclassing `Thread`
It is also possible to sub-class the `Thread` class, and override its `run` It is also possible to sub-class the `Thread` class, and override its `run`
method: method:
%% Cell type:code id:dbf8e4ff tags: %% Cell type:code id:19bf75de tags:
``` ```
class LongRunningThread(threading.Thread): class LongRunningThread(threading.Thread):
def __init__(self, niters, *args, **kwargs): def __init__(self, niters, *args, **kwargs):
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
self.niters = niters self.niters = niters
def run(self): def run(self):
for i in range(self.niters): for i in range(self.niters):
if i % 2 == 0: print('Tick') if i % 2 == 0: print('Tick')
else: print('Tock') else: print('Tock')
time.sleep(0.5) time.sleep(0.5)
t = LongRunningThread(6) t = LongRunningThread(6)
t.start() t.start()
t.join() t.join()
print('Done') print('Done')
``` ```
%% Cell type:markdown id:a3218617 tags: %% Cell type:markdown id:61391158 tags:
<a class="anchor" id="daemon-threads"></a> <a class="anchor" id="daemon-threads"></a>
### Daemon threads ### Daemon threads
By default, a Python application will not exit until _all_ active threads have By default, a Python application will not exit until _all_ active threads have
finished execution. If you want to run a task in the background for the finished execution. If you want to run a task in the background for the
duration of your application, you can mark it as a `daemon` thread - when all duration of your application, you can mark it as a `daemon` thread - when all
non-daemon threads in a Python application have finished, all daemon threads non-daemon threads in a Python application have finished, all daemon threads
will be halted, and the application will exit. will be halted, and the application will exit.
You can mark a thread as being a daemon by setting an attribute on it after You can mark a thread as being a daemon by setting an attribute on it after
creation: creation:
%% Cell type:code id:5a0b442b tags: %% Cell type:code id:6fcb1f65 tags:
``` ```
t = threading.Thread(target=longRunningTask) t = threading.Thread(target=longRunningTask)
t.daemon = True t.daemon = True
``` ```
%% Cell type:markdown id:f04828ff tags: %% Cell type:markdown id:69c1f604 tags:
See the [`Thread` See the [`Thread`
documentation](https://docs.python.org/3/library/threading.html#thread-objects) documentation](https://docs.python.org/3/library/threading.html#thread-objects)
for more details. for more details.
<a class="anchor" id="thread-synchronisation"></a> <a class="anchor" id="thread-synchronisation"></a>
### Thread synchronisation ### Thread synchronisation
The `threading` module provides some useful thread-synchronisation primitives The `threading` module provides some useful thread-synchronisation primitives
- the `Lock`, `RLock` (re-entrant `Lock`), and `Event` classes. The - the `Lock`, `RLock` (re-entrant `Lock`), and `Event` classes. The
`threading` module also provides `Condition` and `Semaphore` classes - refer `threading` module also provides `Condition` and `Semaphore` classes - refer
to the [documentation](https://docs.python.org/3/library/threading.html) for to the [documentation](https://docs.python.org/3/library/threading.html) for
more details. more details.
<a class="anchor" id="lock"></a> <a class="anchor" id="lock"></a>
#### `Lock` #### `Lock`
The [`Lock`](https://docs.python.org/3/library/threading.html#lock-objects) The [`Lock`](https://docs.python.org/3/library/threading.html#lock-objects)
class (and its re-entrant version, the class (and its re-entrant version, the
[`RLock`](https://docs.python.org/3/library/threading.html#rlock-objects)) [`RLock`](https://docs.python.org/3/library/threading.html#rlock-objects))
prevents a block of code from being accessed by more than one thread at a prevents a block of code from being accessed by more than one thread at a
time. For example, if we have multiple threads running this `task` function, time. For example, if we have multiple threads running this `task` function,
their [outputs](https://www.youtube.com/watch?v=F5fUFnfPpYU) will inevitably their [outputs](https://www.youtube.com/watch?v=F5fUFnfPpYU) will inevitably
become intertwined: become intertwined:
%% Cell type:code id:33d52d8b tags: %% Cell type:code id:7811d789 tags:
``` ```
def task(): def task():
for i in range(5): for i in range(5):
print(f'{i} Woozle ', end='') print(f'{i} Woozle ', end='')
time.sleep(0.1) time.sleep(0.1)
print('Wuzzle') print('Wuzzle')
threads = [threading.Thread(target=task) for i in range(5)] threads = [threading.Thread(target=task) for i in range(5)]
for t in threads: for t in threads:
t.start() t.start()
``` ```
%% Cell type:markdown id:281eb07a tags: %% Cell type:markdown id:1f3c088c tags:
But if we protect the critical section with a `Lock` object, the output will But if we protect the critical section with a `Lock` object, the output will
look more sensible: look more sensible:
%% Cell type:code id:40802e7f tags: %% Cell type:code id:20981b0e tags:
``` ```
lock = threading.Lock() lock = threading.Lock()
def task(): def task():
for i in range(5): for i in range(5):
with lock: with lock:
print(f'{i} Woozle ', end='') print(f'{i} Woozle ', end='')
time.sleep(0.1) time.sleep(0.1)
print('Wuzzle') print('Wuzzle')
threads = [threading.Thread(target=task) for i in range(5)] threads = [threading.Thread(target=task) for i in range(5)]
for t in threads: for t in threads:
t.start() t.start()
``` ```
%% Cell type:markdown id:88acba0e tags: %% Cell type:markdown id:f665943f tags:
> Instead of using a `Lock` object in a `with` statement, it is also possible > Instead of using a `Lock` object in a `with` statement, it is also possible
> to manually call its `acquire` and `release` methods: > to manually call its `acquire` and `release` methods:
> >
> def task(): > def task():
> for i in range(5): > for i in range(5):
> lock.acquire() > lock.acquire()
> print(f'{i} Woozle ', end='') > print(f'{i} Woozle ', end='')
> time.sleep(0.1) > time.sleep(0.1)
> print('Wuzzle') > print('Wuzzle')
> lock.release() > lock.release()
Python does not have any built-in constructs to implement `Lock`-based mutual Python does not have any built-in constructs to implement `Lock`-based mutual
exclusion across several functions or methods - each function/method must exclusion across several functions or methods - each function/method must
explicitly acquire/release a shared `Lock` instance. However, it is relatively explicitly acquire/release a shared `Lock` instance. However, it is relatively
straightforward to implement a decorator which does this for you: straightforward to implement a decorator which does this for you:
%% Cell type:code id:4c30c31a tags: %% Cell type:code id:ac049359 tags:
``` ```
def mutex(func, lock): def mutex(func, lock):
def wrapper(*args): def wrapper(*args):
with lock: with lock:
func(*args) func(*args)
return wrapper return wrapper
class MyClass(object): class MyClass(object):
def __init__(self): def __init__(self):
lock = threading.Lock() lock = threading.Lock()
self.safeFunc1 = mutex(self.safeFunc1, lock) self.safeFunc1 = mutex(self.safeFunc1, lock)
self.safeFunc2 = mutex(self.safeFunc2, lock) self.safeFunc2 = mutex(self.safeFunc2, lock)
def safeFunc1(self): def safeFunc1(self):
time.sleep(0.1) time.sleep(0.1)
print('safeFunc1 start') print('safeFunc1 start')
time.sleep(0.2) time.sleep(0.2)
print('safeFunc1 end') print('safeFunc1 end')
def safeFunc2(self): def safeFunc2(self):
time.sleep(0.1) time.sleep(0.1)
print('safeFunc2 start') print('safeFunc2 start')
time.sleep(0.2) time.sleep(0.2)
print('safeFunc2 end') print('safeFunc2 end')
mc = MyClass() mc = MyClass()
f1threads = [threading.Thread(target=mc.safeFunc1) for i in range(4)] f1threads = [threading.Thread(target=mc.safeFunc1) for i in range(4)]
f2threads = [threading.Thread(target=mc.safeFunc2) for i in range(4)] f2threads = [threading.Thread(target=mc.safeFunc2) for i in range(4)]
for t in f1threads + f2threads: for t in f1threads + f2threads:
t.start() t.start()
``` ```
%% Cell type:markdown id:c69dbe16 tags: %% Cell type:markdown id:4eb2c3a8 tags:
Try removing the `mutex` lock from the two methods in the above code, and see Try removing the `mutex` lock from the two methods in the above code, and see
what it does to the output. what it does to the output.
<a class="anchor" id="event"></a> <a class="anchor" id="event"></a>
#### `Event` #### `Event`
The The
[`Event`](https://docs.python.org/3/library/threading.html#event-objects) [`Event`](https://docs.python.org/3/library/threading.html#event-objects)
class is essentially a boolean [semaphore][semaphore-wiki]. It can be used to class is essentially a boolean [semaphore][semaphore-wiki]. It can be used to
signal events between threads. Threads can `wait` on the event, and be awoken signal events between threads. Threads can `wait` on the event, and be awoken
when the event is `set` by another thread: when the event is `set` by another thread:
[semaphore-wiki]: https://en.wikipedia.org/wiki/Semaphore_(programming) [semaphore-wiki]: https://en.wikipedia.org/wiki/Semaphore_(programming)
%% Cell type:code id:b0b933b6 tags: %% Cell type:code id:30b6989c tags:
``` ```
import numpy as np import numpy as np
processingFinished = threading.Event() processingFinished = threading.Event()
def processData(data): def processData(data):
print('Processing data ...') print('Processing data ...')
time.sleep(2) time.sleep(2)
print('Result:', data.mean()) print('Result:', data.mean())
processingFinished.set() processingFinished.set()
data = np.random.randint(1, 100, 100) data = np.random.randint(1, 100, 100)
t = threading.Thread(target=processData, args=(data,)) t = threading.Thread(target=processData, args=(data,))
t.start() t.start()
processingFinished.wait() processingFinished.wait()
print('Processing finished!') print('Processing finished!')
``` ```
%% Cell type:markdown id:2a6a36e2 tags: %% Cell type:markdown id:a1bb6596 tags:
<a class="anchor" id="the-global-interpreter-lock-gil"></a> <a class="anchor" id="the-global-interpreter-lock-gil"></a>
### The Global Interpreter Lock (GIL) ### The Global Interpreter Lock (GIL)
The [*Global Interpreter The [*Global Interpreter
Lock*](https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock) Lock*](https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock)
is an implementation detail of [CPython](https://github.com/python/cpython) is an implementation detail of [CPython](https://github.com/python/cpython)
(the official Python interpreter). The GIL means that a multi-threaded (the official Python interpreter). The GIL means that a multi-threaded
program written in pure Python is not able to take advantage of multiple program written in pure Python is not able to take advantage of multiple
cores - this essentially means that only one thread may be executing at any cores - this essentially means that only one thread may be executing at any
point in time. point in time.
> Note that this is likely to change in future Python releases, with the
> development of [_Free-threaded Python_](https://docs.python.org/3/howto/free-threading-python.html).
The `threading` module does still have its uses though, as this GIL problem The `threading` module does still have its uses though, as this GIL problem
does not affect tasks which involve calls to system or natively compiled does not affect tasks which involve calls to system or natively compiled
libraries (e.g. file and network I/O, Numpy operations, etc.). So you can, libraries (e.g. file and network I/O, Numpy operations, etc.). So you can,
for example, perform some expensive processing on a Numpy array in a thread for example, perform some expensive processing on a Numpy array in a thread
running on one core, whilst having another thread (e.g. user interaction) running on one core, whilst having another thread (e.g. user interaction)
running on another core. running on another core.
<a class="anchor" id="multiprocessing"></a> <a class="anchor" id="multiprocessing"></a>
## Multiprocessing ## Multiprocessing
For true parallelism, you should check out the For true parallelism, you should check out the
[`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html)
module. module.
The `multiprocessing` module spawns sub-processes, rather than threads, and so The `multiprocessing` module spawns sub-processes, rather than threads, and so
is not subject to the GIL constraints that the `threading` module suffers is not subject to the GIL constraints that the `threading` module suffers
from. It provides two APIs - a "traditional" equivalent to that provided by from. It provides two APIs - a "traditional" equivalent to that provided by
the `threading` module, and a powerful higher-level API. the `threading` module, and a powerful higher-level API.
> Python also provides the > Python also provides the
> [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) > [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html)
> module, which offers a simpler alternative API to `multiprocessing`. It > module, which offers a simpler alternative API to `multiprocessing`. It
> offers no functionality over `multiprocessing`, so is not covered here. > offers no functionality over `multiprocessing`, so is not covered here.
<a class="anchor" id="threading-equivalent-api"></a> <a class="anchor" id="threading-equivalent-api"></a>
### `threading`-equivalent API ### `threading`-equivalent API
The The
[`Process`](https://docs.python.org/3/library/multiprocessing.html#the-process-class) [`Process`](https://docs.python.org/3/library/multiprocessing.html#the-process-class)
class is the `multiprocessing` equivalent of the class is the `multiprocessing` equivalent of the
[`threading.Thread`](https://docs.python.org/3/library/threading.html#thread-objects) [`threading.Thread`](https://docs.python.org/3/library/threading.html#thread-objects)
class. `multprocessing` also has equivalents of the [`Lock` and `Event` class. `multprocessing` also has equivalents of the [`Lock` and `Event`
classes](https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processes), classes](https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processes),
and the other synchronisation primitives provided by `threading`. and the other synchronisation primitives provided by `threading`.
So you can simply replace `threading.Thread` with `multiprocessing.Process`, So you can simply replace `threading.Thread` with `multiprocessing.Process`,
and you will have true parallelism. and you will have true parallelism.
Because your "threads" are now independent processes, you need to be a little Because your "threads" are now independent processes, you need to be a little
careful about how to share information across them. If you only need to share careful about how to share information across them. If you only need to share
small amounts of data, you can use the [`Queue` and `Pipe` small amounts of data, you can use the [`Queue` and `Pipe`
classes](https://docs.python.org/3/library/multiprocessing.html#exchanging-objects-between-processes), classes](https://docs.python.org/3/library/multiprocessing.html#exchanging-objects-between-processes),
in the `multiprocessing` module. If you are working with large amounts of data in the `multiprocessing` module. If you are working with large amounts of data
where copying between processes is not feasible, things become more where copying between processes is not feasible, things become more
complicated, but read on... complicated, but read on...
<a class="anchor" id="higher-level-api-the-multiprocessing-pool"></a> <a class="anchor" id="higher-level-api-the-multiprocessing-pool"></a>
### Higher-level API - the `multiprocessing.Pool` ### Higher-level API - the `multiprocessing.Pool`
The real advantages of `multiprocessing` lie in its higher level API, centered The real advantages of `multiprocessing` lie in its higher level API, centered
around the [`Pool` around the [`Pool`
class](https://docs.python.org/3/library/multiprocessing.html#using-a-pool-of-workers). class](https://docs.python.org/3/library/multiprocessing.html#using-a-pool-of-workers).
Essentially, you create a `Pool` of worker processes - you specify the number Essentially, you create a `Pool` of worker processes - you specify the number
of processes when you create the pool. Once you have created a `Pool`, you can of processes when you create the pool. Once you have created a `Pool`, you can
use its methods to automatically parallelise tasks. The most useful are the use its methods to automatically parallelise tasks. The most useful are the
`map`, `starmap` and `apply_async` methods. `map`, `starmap` and `apply_async` methods.
The `Pool` class is a context manager, so can be used in a `with` statement, The `Pool` class is a context manager, so can be used in a `with` statement,
e.g.: e.g.:
> ``` > ```
> with mp.Pool(processes=16) as pool: > with mp.Pool(processes=16) as pool:
> # do stuff with the pool > # do stuff with the pool
> ``` > ```
It is possible to create a `Pool` outside of a `with` statement, but in this It is possible to create a `Pool` outside of a `with` statement, but in this
case you must ensure that you call its `close` method when you are finished. case you must ensure that you call its `close` method when you are finished.
Using a `Pool` in a `with` statement is therefore recommended, because you know Using a `Pool` in a `with` statement is therefore recommended, because you know
that it will be shut down correctly, even in the event of an error. that it will be shut down correctly, even in the event of an error.
> The best number of processes to use for a `Pool` will depend on the system > The best number of processes to use for a `Pool` will depend on the system
> you are running on (number of cores), and the tasks you are running (e.g. > you are running on (number of cores), and the tasks you are running (e.g.
> I/O bound or CPU bound). If you do not specify the number of processes when > I/O bound or CPU bound). If you do not specify the number of processes when
> creating a `Pool`, it will default to the number of cores on your machine. > creating a `Pool`, it will default to the number of cores on your machine.
<a class="anchor" id="pool-map"></a> <a class="anchor" id="pool-map"></a>
#### `Pool.map` #### `Pool.map`
The The
[`Pool.map`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map) [`Pool.map`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map)
method is the multiprocessing equivalent of the built-in method is the multiprocessing equivalent of the built-in
[`map`](https://docs.python.org/3/library/functions.html#map) function - it [`map`](https://docs.python.org/3/library/functions.html#map) function - it
is given a function, and a sequence, and it applies the function to each is given a function, and a sequence, and it applies the function to each
element in the sequence. element in the sequence.
%% Cell type:code id:daadc9c9 tags: %% Cell type:code id:025a74c5 tags:
``` ```
import time import time
import multiprocessing as mp import multiprocessing as mp
import numpy as np import numpy as np
def crunchImage(imgfile): def crunchImage(imgfile):
# Load a nifti image and calculate some # Load a nifti image and calculate some
# metric from the image. Use your # metric from the image. Use your
# imagination to fill in this function. # imagination to fill in this function.
time.sleep(2) time.sleep(2)
np.random.seed() np.random.seed()
result = np.random.randint(1, 100, 1)[0] result = np.random.randint(1, 100, 1)[0]
return result return result
imgfiles = [f'{i:02d}.nii.gz' for i in range(20)] imgfiles = [f'{i:02d}.nii.gz' for i in range(20)]
print(f'Crunching {len(imgfiles)} images...') print(f'Crunching {len(imgfiles)} images...')
start = time.time() start = time.time()
with mp.Pool(processes=16) as p: with mp.Pool(processes=16) as p:
results = p.map(crunchImage, imgfiles) results = p.map(crunchImage, imgfiles)
end = time.time() end = time.time()
for imgfile, result in zip(imgfiles, results): for imgfile, result in zip(imgfiles, results):
print(f'Result for {imgfile}: {result}') print(f'Result for {imgfile}: {result}')
print('Total execution time: {:0.2f} seconds'.format(end - start)) print('Total execution time: {:0.2f} seconds'.format(end - start))
``` ```
%% Cell type:markdown id:51d5ae8a tags: %% Cell type:markdown id:f363f85a tags:
The `Pool.map` method only works with functions that accept one argument, such The `Pool.map` method only works with functions that accept one argument, such
as our `crunchImage` function above. If you have a function which accepts as our `crunchImage` function above. If you have a function which accepts
multiple arguments, use the multiple arguments, use the
[`Pool.starmap`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.starmap) [`Pool.starmap`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.starmap)
method instead: method instead:
%% Cell type:code id:60ce3e5b tags: %% Cell type:code id:c23af106 tags:
``` ```
def crunchImage(imgfile, modality): def crunchImage(imgfile, modality):
time.sleep(2) time.sleep(2)
np.random.seed() np.random.seed()
if modality == 't1': if modality == 't1':
result = np.random.randint(1, 100, 1) result = np.random.randint(1, 100, 1)
elif modality == 't2': elif modality == 't2':
result = np.random.randint(100, 200, 1) result = np.random.randint(100, 200, 1)
return result[0] return result[0]
imgfiles = [f't1_{i:02d}.nii.gz' for i in range(10)] + \ imgfiles = [f't1_{i:02d}.nii.gz' for i in range(10)] + \
[f't2_{i:02d}.nii.gz' for i in range(10)] [f't2_{i:02d}.nii.gz' for i in range(10)]
modalities = ['t1'] * 10 + ['t2'] * 10 modalities = ['t1'] * 10 + ['t2'] * 10
args = [(f, m) for f, m in zip(imgfiles, modalities)] args = [(f, m) for f, m in zip(imgfiles, modalities)]
print('Crunching images...') print('Crunching images...')
start = time.time() start = time.time()
with mp.Pool(processes=16) as pool: with mp.Pool(processes=16) as pool:
results = pool.starmap(crunchImage, args) results = pool.starmap(crunchImage, args)
end = time.time() end = time.time()
for imgfile, modality, result in zip(imgfiles, modalities, results): for imgfile, modality, result in zip(imgfiles, modalities, results):
print(f'{imgfile} [{modality}]: {result}') print(f'{imgfile} [{modality}]: {result}')
print('Total execution time: {:0.2f} seconds'.format(end - start)) print('Total execution time: {:0.2f} seconds'.format(end - start))
``` ```
%% Cell type:markdown id:99d35451 tags: %% Cell type:markdown id:e91c1ea5 tags:
The `map` and `starmap` methods also have asynchronous equivalents `map_async` The `map` and `starmap` methods also have asynchronous equivalents `map_async`
and `starmap_async`, which return immediately. Refer to the and `starmap_async`, which return immediately. Refer to the
[`Pool`](https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool) [`Pool`](https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool)
documentation for more details. documentation for more details.
<a class="anchor" id="pool-apply-async"></a> <a class="anchor" id="pool-apply-async"></a>
#### `Pool.apply_async` #### `Pool.apply_async`
The The
[`Pool.apply`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.apply) [`Pool.apply`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.apply)
method will execute a function on one of the processes, and block until it has method will execute a function on one of the processes, and block until it has
finished. The finished. The
[`Pool.apply_async`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.apply_async) [`Pool.apply_async`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.apply_async)
method returns immediately, and is thus more suited to asynchronously method returns immediately, and is thus more suited to asynchronously
scheduling multiple jobs to run in parallel. scheduling multiple jobs to run in parallel.
`apply_async` returns an object of type `apply_async` returns an object of type
[`AsyncResult`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.AsyncResult). [`AsyncResult`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.AsyncResult).
An `AsyncResult` object has `wait` and `get` methods which will block until An `AsyncResult` object has `wait` and `get` methods which will block until
the job has completed. the job has completed.
%% Cell type:code id:b791805e tags: %% Cell type:code id:55e69074 tags:
``` ```
import time import time
import multiprocessing as mp import multiprocessing as mp
import numpy as np import numpy as np
def linear_registration(src, ref): def linear_registration(src, ref):
time.sleep(1) time.sleep(1)
return np.eye(4) return np.eye(4)
def nonlinear_registration(src, ref, affine): def nonlinear_registration(src, ref, affine):
time.sleep(3) time.sleep(3)
# this number represents a non-linear warp # this number represents a non-linear warp
# field - use your imagination people! # field - use your imagination people!
np.random.seed() np.random.seed()
return np.random.randint(1, 100, 1)[0] return np.random.randint(1, 100, 1)[0]
t1s = [f'{i:02d}_t1.nii.gz' for i in range(20)] t1s = [f'{i:02d}_t1.nii.gz' for i in range(20)]
std = 'MNI152_T1_2mm.nii.gz' std = 'MNI152_T1_2mm.nii.gz'
print('Running structural-to-standard registration ' print('Running structural-to-standard registration '
f'on {len(t1s)} subjects...') f'on {len(t1s)} subjects...')
# Run linear registration on all the T1s. # Run linear registration on all the T1s.
start = time.time() start = time.time()
with mp.Pool(processes=16) as pool: with mp.Pool(processes=16) as pool:
# We build a list of AsyncResult objects # We build a list of AsyncResult objects
linresults = [pool.apply_async(linear_registration, (t1, std)) linresults = [pool.apply_async(linear_registration, (t1, std))
for t1 in t1s] for t1 in t1s]
# Then we wait for each job to finish, # Then we wait for each job to finish,
# and replace its AsyncResult object # and replace its AsyncResult object
# with the actual result - an affine # with the actual result - an affine
# transformation matrix. # transformation matrix.
for i, r in enumerate(linresults): for i, r in enumerate(linresults):
linresults[i] = r.get() linresults[i] = r.get()
end = time.time() end = time.time()
print('Linear registrations completed in ' print('Linear registrations completed in '
f'{end - start:0.2f} seconds') f'{end - start:0.2f} seconds')
# Run non-linear registration on all the T1s, # Run non-linear registration on all the T1s,
# using the linear registrations to initialise. # using the linear registrations to initialise.
start = time.time() start = time.time()
with mp.Pool(processes=16) as pool: with mp.Pool(processes=16) as pool:
nlinresults = [pool.apply_async(nonlinear_registration, (t1, std, aff)) nlinresults = [pool.apply_async(nonlinear_registration, (t1, std, aff))
for (t1, aff) in zip(t1s, linresults)] for (t1, aff) in zip(t1s, linresults)]
# Wait for each non-linear reg to finish, # Wait for each non-linear reg to finish,
# and store the resulting warp field. # and store the resulting warp field.
for i, r in enumerate(nlinresults): for i, r in enumerate(nlinresults):
nlinresults[i] = r.get() nlinresults[i] = r.get()
end = time.time() end = time.time()
print('Non-linear registrations completed in ' print('Non-linear registrations completed in '
'{:0.2f} seconds'.format(end - start)) '{:0.2f} seconds'.format(end - start))
print('Non linear registrations:') print('Non linear registrations:')
for t1, result in zip(t1s, nlinresults): for t1, result in zip(t1s, nlinresults):
print(f'{t1} : {result}') print(f'{t1} : {result}')
``` ```
%% Cell type:markdown id:495c9c03 tags: %% Cell type:markdown id:3c855c8d tags:
<a class="anchor" id="sharing-data-between-processes"></a> <a class="anchor" id="sharing-data-between-processes"></a>
## Sharing data between processes ## Sharing data between processes
When you use the `Pool.map` method (or any of the other methods we have shown) When you use the `Pool.map` method (or any of the other methods we have shown)
to run a function on a sequence of items, those items must be copied into the to run a function on a sequence of items, those items must be copied into the
memory of each of the child processes. When the child processes are finished, memory of each of the child processes. When the child processes are finished,
the data that they return then has to be copied back to the parent process. the data that they return then has to be copied back to the parent process.
Any items which you wish to pass to a function that is executed by a `Pool` Any items which you wish to pass to a function that is executed by a `Pool`
must be *pickleable*<sup>1</sup> - the built-in must be *pickleable*<sup>1</sup> - the built-in
[`pickle`](https://docs.python.org/3/library/pickle.html) module is used by [`pickle`](https://docs.python.org/3/library/pickle.html) module is used by
`multiprocessing` to serialise and de-serialise the data passed to and `multiprocessing` to serialise and de-serialise the data passed to and
returned from a child process. The majority of standard Python types (`list`, returned from a child process. The majority of standard Python types (`list`,
`dict`, `str` etc), and Numpy arrays can be pickled and unpickled, so you only `dict`, `str` etc), and Numpy arrays can be pickled and unpickled, so you only
need to worry about this detail if you are passing objects of a custom type need to worry about this detail if you are passing objects of a custom type
(e.g. instances of classes that you have written, or that are defined in some (e.g. instances of classes that you have written, or that are defined in some
third-party library). third-party library).
> <sup>1</sup>*Pickleable* is the term used in the Python world to refer to > <sup>1</sup>*Pickleable* is the term used in the Python world to refer to
> something that is *serialisable* - basically, the process of converting an > something that is *serialisable* - basically, the process of converting an
> in-memory object into a binary form that can be stored and/or transmitted, > in-memory object into a binary form that can be stored and/or transmitted,
> and then loaded back into memory at some point in the future (in the same > and then loaded back into memory at some point in the future (in the same
> process, or in another process). > process, or in another process).
There is obviously some overhead in copying data back and forth between the There is obviously some overhead in copying data back and forth between the
main process and the worker processes; this may or may not be a problem. For main process and the worker processes; this may or may not be a problem. For
most computationally intensive tasks, this communication overhead is not most computationally intensive tasks, this communication overhead is not
important - the performance bottleneck is typically going to be the important - the performance bottleneck is typically going to be the
computation time, rather than I/O between the parent and child processes. computation time, rather than I/O between the parent and child processes.
However, if you are working with a large data set, where copying it between However, if you are working with a large data set, where copying it between
processes is not viable, you have a couple of options available to you. processes is not viable, you have a couple of options available to you.
<a class="anchor" id="memory-mapping"></a> <a class="anchor" id="memory-mapping"></a>
### Memory-mapping ### Memory-mapping
One method for sharing a large `numpy` array between multiple processes is to One method for sharing a large `numpy` array between multiple processes is to
use a _memory-mapped_ array. This is a feature built into `numpy` which use a _memory-mapped_ array. This is a feature built into `numpy` which
stores your data in a regular file, instead of in memory. This allows your stores your data in a regular file, instead of in memory. This allows your
data to be simultaneously read and written by multiple processes, and is fairly data to be simultaneously read and written by multiple processes, and is fairly
straightforward to accomplish. straightforward to accomplish.
For example, let's say you have some 4D fMRI data, and wish to fit a For example, let's say you have some 4D fMRI data, and wish to fit a
complicated model to the time series at each voxel. First we will load our 4D complicated model to the time series at each voxel. First we will load our 4D
data, and pre-allocate another array to store the fitted model parameters: data, and pre-allocate another array to store the fitted model parameters:
%% Cell type:code id:3384d747 tags: %% Cell type:code id:46938bac tags:
``` ```
import time import time
import functools as ft import functools as ft
import multiprocessing as mp import multiprocessing as mp
import numpy as np import numpy as np
# Store the parameters that are required # Store the parameters that are required
# to create our memory-mapped arrays, as # to create our memory-mapped arrays, as
# we need to re-use them a couple of times. # we need to re-use them a couple of times.
# #
# Note that in practice you would usually # Note that in practice you would usually
# want to store these files in a temporary # want to store these files in a temporary
# directory, and/or ensure that they are # directory, and/or ensure that they are
# deleted once you are finished. # deleted once you are finished.
data_params = dict(filename='data.mmap', shape=(91, 109, 91, 50), dtype=np.float32) data_params = dict(filename='data.mmap', shape=(91, 109, 91, 50), dtype=np.float32)
model_params = dict(filename='model.mmap', shape=(91, 109, 91), dtype=np.float32) model_params = dict(filename='model.mmap', shape=(91, 109, 91), dtype=np.float32)
# Load our data as a memory-mapped array (we # Load our data as a memory-mapped array (we
# are using random data for this example) # are using random data for this example)
data = np.memmap(**data_params, mode='w+') data = np.memmap(**data_params, mode='w+')
data[:] = np.random.random((91, 109, 91, 50)).astype(np.float32) data[:] = np.random.random((91, 109, 91, 50)).astype(np.float32)
data.flush() data.flush()
# Pre-allocate space to store the fitted # Pre-allocate space to store the fitted
# model parameters # model parameters
model = np.memmap(**model_params, mode='w+') model = np.memmap(**model_params, mode='w+')
``` ```
%% Cell type:markdown id:15ae0eb1 tags: %% Cell type:markdown id:89897973 tags:
> If your image files are uncompressed (i.e. `.nii` rather than `.nii.gz`), > If your image files are uncompressed (i.e. `.nii` rather than `.nii.gz`),
> you can instruct `nibabel` and `fslpy` to load them as a memory-map by > you can instruct `nibabel` and `fslpy` to load them as a memory-map by
> passing `mmap=True` to the `nibabel.load` function, and the > passing `mmap=True` to the `nibabel.load` function, and the
> `fsl.data.image.Image` constructor. > `fsl.data.image.Image` constructor.
Now we will write our model fitting function so that it works on one slice at Now we will write our model fitting function so that it works on one slice at
a time - this will allow us to process multiple slices in parallel. Note a time - this will allow us to process multiple slices in parallel. Note
that, within this function, we have to _re-load_ the memory-mapped arrays. In that, within this function, we have to _re-load_ the memory-mapped arrays. In
this example we have written this function so as to expect the arguments this example we have written this function so as to expect the arguments
required to create the two memory-maps to be passed in (the `data_params` and required to create the two memory-maps to be passed in (the `data_params` and
`model_params` dictionaries that we created above): `model_params` dictionaries that we created above):
%% Cell type:code id:2daf1f1b tags: %% Cell type:code id:037dbf6b tags:
``` ```
def fit_model(indata, outdata, sliceidx): def fit_model(indata, outdata, sliceidx):
indata = np.memmap(**indata, mode='r') indata = np.memmap(**indata, mode='r')
outdata = np.memmap(**outdata, mode='r+') outdata = np.memmap(**outdata, mode='r+')
# sleep to simulate expensive model fitting # sleep to simulate expensive model fitting
print(f'Fitting model at slice {sliceidx}') print(f'Fitting model at slice {sliceidx}')
time.sleep(1) time.sleep(1)
outdata[:, :, sliceidx] = indata[:, :, sliceidx, :].mean() + sliceidx outdata[:, :, sliceidx] = indata[:, :, sliceidx, :].mean() + sliceidx
``` ```
%% Cell type:markdown id:90b8f2e3 tags: %% Cell type:markdown id:eb4b7f8f tags:
Now we can use `multiprocessing` to fit the model in parallel across all of the Now we can use `multiprocessing` to fit the model in parallel across all of the
image slices: image slices:
%% Cell type:code id:ffb4c693 tags: %% Cell type:code id:ba54b93c tags:
``` ```
fit_function = ft.partial(fit_model, data_params, model_params) fit_function = ft.partial(fit_model, data_params, model_params)
slice_idxs = list(range(91)) slice_idxs = list(range(91))
with mp.Pool(processes=16) as pool: with mp.Pool(processes=16) as pool:
pool.map(fit_function, slice_idxs) pool.map(fit_function, slice_idxs)
print(model) print(model)
``` ```
%% Cell type:markdown id:dd0dd890 tags: %% Cell type:markdown id:145a22d2 tags:
<a class="anchor" id="read-only-sharing"></a> <a class="anchor" id="read-only-sharing"></a>
### Read-only sharing ### Read-only sharing
If you are working with a large dataset, you have determined that copying data If you are working with a large dataset, you have determined that copying data
between processes is having a substantial impact on your performance, and have between processes is having a substantial impact on your performance, and have
also decided that memory-mapping is not an option for you, and instead wish to also decided that memory-mapping is not an option for you, and instead wish to
*share* a single copy of the data between the processes, you will need to: *share* a single copy of the data between the processes, you will need to:
1. Structure your code so that the data you want to share is accessible at 1. Structure your code so that the data you want to share is accessible at
the *module level*. the *module level*.
2. Define/create/load the data *before* creating the `Pool`. 2. Define/create/load the data *before* creating the `Pool`.
This is because, when you create a `Pool`, what actually happens is that the This is because, when you create a `Pool`, what actually happens is that the
process your Python script is running in will [**fork**][wiki-fork] itself - process your Python script is running in will [**fork**][wiki-fork] itself -
the child processes that are created are used as the worker processes by the the child processes that are created are used as the worker processes by the
`Pool`. And if you create/load your data in your main process *before* this `Pool`. And if you create/load your data in your main process *before* this
fork occurs, all of the child processes will inherit the memory space of the fork occurs, all of the child processes will inherit the memory space of the
main process, and will therefore have (read-only) access to the data, without main process, and will therefore have (read-only) access to the data, without
any copying required. any copying required.
[wiki-fork]: https://en.wikipedia.org/wiki/Fork_(system_call) [wiki-fork]: https://en.wikipedia.org/wiki/Fork_(system_call)
Let's see this in action with a simple example. We'll start by defining a Let's see this in action with a simple example. We'll start by defining a
horrible little helper function which allows us to track the total memory horrible little helper function which allows us to track the total memory
usage: usage:
%% Cell type:code id:13fe8356 tags: %% Cell type:code id:cd36f0b7 tags:
``` ```
import sys import sys
import subprocess as sp import subprocess as sp
def memusage(msg): def memusage(msg):
if sys.platform == 'darwin': if sys.platform == 'darwin':
total = sp.run(['sysctl', 'hw.memsize'], capture_output=True).stdout.decode() total = sp.run(['sysctl', 'hw.memsize'], capture_output=True).stdout.decode()
total = int(total.split()[1]) // 1048576 total = int(total.split()[1]) // 1048576
usage = sp.run('vm_stat', capture_output=True).stdout.decode() usage = sp.run('vm_stat', capture_output=True).stdout.decode()
usage = usage.strip().split('\n') usage = usage.strip().split('\n')
usage = [l.split(':') for l in usage] usage = [l.split(':') for l in usage]
usage = {k.strip() : v.strip() for k, v in usage} usage = {k.strip() : v.strip() for k, v in usage}
usage = int(usage['Pages free'][:-1]) / 256.0 usage = int(usage['Pages free'][:-1]) / 256.0
usage = int(total - usage) usage = int(total - usage)
else: else:
stdout = sp.run(['free', '--mega'], capture_output=True).stdout.decode() stdout = sp.run(['free', '--mega'], capture_output=True).stdout.decode()
stdout = stdout.split('\n')[1].split() stdout = stdout.split('\n')[1].split()
total = int(stdout[1]) total = int(stdout[1])
usage = int(stdout[2]) usage = int(stdout[2])
print(f'Memory usage {msg}: {usage} / {total} MB') print(f'Memory usage {msg}: {usage} / {total} MB')
``` ```
%% Cell type:markdown id:398f7b19 tags: %% Cell type:markdown id:f50cbe16 tags:
Now our task is simply to calculate the sum of a large array of numbers. We're Now our task is simply to calculate the sum of a large array of numbers. We're
going to create a big chunk of data, and process it in chunks, keeping track going to create a big chunk of data, and process it in chunks, keeping track
of memory usage as the task progresses: of memory usage as the task progresses:
%% Cell type:code id:66b9917b tags: %% Cell type:code id:aa189b29 tags:
``` ```
import time import time
import multiprocessing as mp import multiprocessing as mp
import numpy as np import numpy as np
memusage('before creating data') memusage('before creating data')
# allocate 500MB of data # allocate 500MB of data
data = np.random.random(500 * (1048576 // 8)) data = np.random.random(500 * (1048576 // 8))
# Assign nelems values to each worker # Assign nelems values to each worker
# process (hard-coded so we need 12 # process (hard-coded so we need 12
# jobs to complete the task) # jobs to complete the task)
nelems = len(data) // 12 nelems = len(data) // 12
memusage('after creating data') memusage('after creating data')
# Each job process nelems values, # Each job process nelems values,
# starting from the specified offset # starting from the specified offset
def process_chunk(offset): def process_chunk(offset):
time.sleep(1) time.sleep(1)
return data[offset:offset + nelems].sum() return data[offset:offset + nelems].sum()
# Generate an offset into the data for each job - # Generate an offset into the data for each job -
# we will call process_chunk for each offset # we will call process_chunk for each offset
offsets = range(0, len(data), nelems) offsets = range(0, len(data), nelems)
# Create our worker process pool # Create our worker process pool
with mp.Pool(4) as pool: with mp.Pool(4) as pool:
results = pool.map_async(process_chunk, offsets) results = pool.map_async(process_chunk, offsets)
# Wait for all of the jobs to finish # Wait for all of the jobs to finish
elapsed = 0 elapsed = 0
while not results.ready(): while not results.ready():
memusage(f'after {elapsed} seconds') memusage(f'after {elapsed} seconds')
time.sleep(1) time.sleep(1)
elapsed += 1 elapsed += 1
results = results.get() results = results.get()
print('Total sum: ', sum(results)) print('Total sum: ', sum(results))
print('Sanity check:', data.sum()) print('Sanity check:', data.sum())
``` ```
%% Cell type:markdown id:9b06285f tags: %% Cell type:markdown id:23ab2138 tags:
You should be able to see that only one copy of `data` is created, and is You should be able to see that only one copy of `data` is created, and is
shared by all of the worker processes without any copying taking place. shared by all of the worker processes without any copying taking place.
So things are reasonably straightforward if you only need read-only acess to So things are reasonably straightforward if you only need read-only acess to
your data. But what if your worker processes need to be able to modify the your data. But what if your worker processes need to be able to modify the
data? Go back to the code block above and: data? Go back to the code block above and:
1. Modify the `process_chunk` function so that it modifies every element of 1. Modify the `process_chunk` function so that it modifies every element of
its assigned portion of the data before the call to `time.sleep`. For its assigned portion of the data before the call to `time.sleep`. For
example: example:
> ``` > ```
> data[offset:offset + nelems] += 1 > data[offset:offset + nelems] += 1
> ``` > ```
2. Restart the Jupyter notebook kernel (*Kernel -> Restart*) - this example is 2. Restart the Jupyter notebook kernel (*Kernel -> Restart*) - this example is
somewhat dependent on the behaviour of the Python garbage collector, so it somewhat dependent on the behaviour of the Python garbage collector, so it
helps to start afresh helps to start afresh
2. Re-run the two code blocks, and watch what happens to the memory usage. 2. Re-run the two code blocks, and watch what happens to the memory usage.
What happened? Well, you are seeing [copy-on-write][wiki-copy-on-write] in What happened? Well, you are seeing [copy-on-write][wiki-copy-on-write] in
action. When the `process_chunk` function is invoked, it is given a reference action. When the `process_chunk` function is invoked, it is given a reference
to the original data array in the memory space of the parent process. But as to the original data array in the memory space of the parent process. But as
soon as an attempt is made to modify it, a copy of the data, in the memory soon as an attempt is made to modify it, a copy of the data, in the memory
space of the child process, is created. The modifications are then applied to space of the child process, is created. The modifications are then applied to
this child process copy, and not to the original copy. So the total memory this child process copy, and not to the original copy. So the total memory
usage has blown out to twice as much as before, and the changes made by each usage has blown out to twice as much as before, and the changes made by each
child process are being lost! child process are being lost!
[wiki-copy-on-write]: https://en.wikipedia.org/wiki/Copy-on-write [wiki-copy-on-write]: https://en.wikipedia.org/wiki/Copy-on-write
<a class="anchor" id="read-write-sharing"></a> <a class="anchor" id="read-write-sharing"></a>
### Read/write sharing ### Read/write sharing
> If you have worked with a real programming language with true parallelism > If you have worked with a real programming language with true parallelism
> and shared memory via within-process multi-threading, feel free to take a > and shared memory via within-process multi-threading, feel free to take a
> break at this point. Breathe. Relax. Go punch a hole in a wall. I've been > break at this point. Breathe. Relax. Go punch a hole in a wall. I've been
> coding in Python for years, and this still makes me angry. Sometimes > coding in Python for years, and this still makes me angry. Sometimes
> ... don't tell anyone I said this ... I even find myself wishing I were > ... don't tell anyone I said this ... I even find myself wishing I were
> coding in *Java* instead of Python. Ugh. I need to take a shower. > coding in *Java* instead of Python. Ugh. I need to take a shower.
In order to truly share memory between multiple processes, the In order to truly share memory between multiple processes, the
`multiprocessing` module provides the [`Value`, `Array`, and `RawArray` `multiprocessing` module provides the [`Value`, `Array`, and `RawArray`
classes](https://docs.python.org/3/library/multiprocessing.html#shared-ctypes-objects), classes](https://docs.python.org/3/library/multiprocessing.html#shared-ctypes-objects),
which allow you to share individual values, or arrays of values, respectively. which allow you to share individual values, or arrays of values, respectively.
The `Array` and `RawArray` classes essentially wrap a typed pointer (from the The `Array` and `RawArray` classes essentially wrap a typed pointer (from the
built-in [`ctypes`](https://docs.python.org/3/library/ctypes.html) module) to built-in [`ctypes`](https://docs.python.org/3/library/ctypes.html) module) to
a block of memory. We can use the `Array` or `RawArray` class to share a Numpy a block of memory. We can use the `Array` or `RawArray` class to share a Numpy
array between our worker processes. The difference between an `Array` and a array between our worker processes. The difference between an `Array` and a
`RawArray` is that the former offers low-level synchronised `RawArray` is that the former offers low-level synchronised
(i.e. process-safe) access to the shared memory. This is necessary if your (i.e. process-safe) access to the shared memory. This is necessary if your
child processes will be modifying the same parts of your data. child processes will be modifying the same parts of your data.
> If you need fine-grained control over synchronising access to shared data by > If you need fine-grained control over synchronising access to shared data by
> multiple processes, all of the [synchronisation > multiple processes, all of the [synchronisation
> primitives](https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processes) > primitives](https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processes)
> from the `multiprocessing` module are at your disposal. > from the `multiprocessing` module are at your disposal.
The requirements for sharing memory between processes still apply here - we The requirements for sharing memory between processes still apply here - we
need to make our data accessible at the *module level*, and we need to create need to make our data accessible at the *module level*, and we need to create
our data before creating the `Pool`. And to achieve read and write capability, our data before creating the `Pool`. And to achieve read and write capability,
we also need to make sure that our input and output arrays are located in we also need to make sure that our input and output arrays are located in
shared memory - we can do this via the `Array` or `RawArray`. shared memory - we can do this via the `Array` or `RawArray`.
As an example, let's say we want to parallelise processing of an image by As an example, let's say we want to parallelise processing of an image by
having each worker process perform calculations on a chunk of the image. having each worker process perform calculations on a chunk of the image.
First, let's define a function which does the calculation on a specified set First, let's define a function which does the calculation on a specified set
of image coordinates: of image coordinates:
%% Cell type:code id:d7a8f363 tags: %% Cell type:code id:c1e1d718 tags:
``` ```
import multiprocessing as mp import multiprocessing as mp
import ctypes import ctypes
import numpy as np import numpy as np
np.set_printoptions(suppress=True) np.set_printoptions(suppress=True)
def process_chunk(shape, idxs): def process_chunk(shape, idxs):
# Get references to our # Get references to our
# input/output data, and # input/output data, and
# create Numpy array views # create Numpy array views
# into them. # into them.
sindata = process_chunk.input_data sindata = process_chunk.input_data
soutdata = process_chunk.output_data soutdata = process_chunk.output_data
indata = np.ctypeslib.as_array(sindata) .reshape(shape) indata = np.ctypeslib.as_array(sindata) .reshape(shape)
outdata = np.ctypeslib.as_array(soutdata).reshape(shape) outdata = np.ctypeslib.as_array(soutdata).reshape(shape)
# Do the calculation on # Do the calculation on
# the specified voxels # the specified voxels
outdata[idxs] = indata[idxs] ** 2 outdata[idxs] = indata[idxs] ** 2
``` ```
%% Cell type:markdown id:4f0cbe28 tags: %% Cell type:markdown id:f7a57523 tags:
Rather than passing the input and output data arrays in as arguments to the Rather than passing the input and output data arrays in as arguments to the
`process_chunk` function, we set them as attributes of the `process_chunk` `process_chunk` function, we set them as attributes of the `process_chunk`
function. This makes the input/output data accessible at the module level, function. This makes the input/output data accessible at the module level,
which is required in order to share the data between the main process and the which is required in order to share the data between the main process and the
child processes. child processes.
Now let's define a second function which processes an entire image. It does Now let's define a second function which processes an entire image. It does
the following: the following:
1. Initialises shared memory areas to store the input and output data. 1. Initialises shared memory areas to store the input and output data.
2. Copies the input data into shared memory. 2. Copies the input data into shared memory.
3. Sets the input and output data as attributes of the `process_chunk` function. 3. Sets the input and output data as attributes of the `process_chunk` function.
4. Creates sets of indices into the input data which, for each worker process, 4. Creates sets of indices into the input data which, for each worker process,
specifies the portion of the data that it is responsible for. specifies the portion of the data that it is responsible for.
5. Creates a worker pool, and runs the `process_chunk` function for each set 5. Creates a worker pool, and runs the `process_chunk` function for each set
of indices. of indices.
%% Cell type:code id:c3ff121f tags: %% Cell type:code id:c16c5587 tags:
``` ```
def process_dataset(data): def process_dataset(data):
nprocs = 8 nprocs = 8
origData = data origData = data
# Create arrays to store the # Create arrays to store the
# input and output data # input and output data
sindata = mp.RawArray(ctypes.c_double, data.size) sindata = mp.RawArray(ctypes.c_double, data.size)
soutdata = mp.RawArray(ctypes.c_double, data.size) soutdata = mp.RawArray(ctypes.c_double, data.size)
data = np.ctypeslib.as_array(sindata).reshape(data.shape) data = np.ctypeslib.as_array(sindata).reshape(data.shape)
outdata = np.ctypeslib.as_array(soutdata).reshape(data.shape) outdata = np.ctypeslib.as_array(soutdata).reshape(data.shape)
# Copy the input data # Copy the input data
# into shared memory # into shared memory
data[:] = origData data[:] = origData
# Make the input/output data # Make the input/output data
# accessible to the process_chunk # accessible to the process_chunk
# function. This must be done # function. This must be done
# *before* the worker pool is # *before* the worker pool is
# created - even though we are # created - even though we are
# doing things differently to the # doing things differently to the
# read-only example, we are still # read-only example, we are still
# making the data arrays accessible # making the data arrays accessible
# at the *module* level, so the # at the *module* level, so the
# memory they are stored in can be # memory they are stored in can be
# shared with the child processes. # shared with the child processes.
process_chunk.input_data = sindata process_chunk.input_data = sindata
process_chunk.output_data = soutdata process_chunk.output_data = soutdata
# number of voxels to be computed # number of voxels to be computed
# by each worker process. # by each worker process.
nvox = int(data.size / nprocs) nvox = int(data.size / nprocs)
# Generate coordinates for # Generate coordinates for
# every voxel in the image # every voxel in the image
xlen, ylen, zlen = data.shape xlen, ylen, zlen = data.shape
xs, ys, zs = np.meshgrid(np.arange(xlen), xs, ys, zs = np.meshgrid(np.arange(xlen),
np.arange(ylen), np.arange(ylen),
np.arange(zlen)) np.arange(zlen))
xs = xs.flatten() xs = xs.flatten()
ys = ys.flatten() ys = ys.flatten()
zs = zs.flatten() zs = zs.flatten()
# We're going to pass each worker # We're going to pass each worker
# process a list of indices, which # process a list of indices, which
# specify the data items which that # specify the data items which that
# worker process needs to compute. # worker process needs to compute.
xs = [xs[nvox * i:nvox * i + nvox] for i in range(nprocs)] + [xs[nvox * nprocs:]] xs = [xs[nvox * i:nvox * i + nvox] for i in range(nprocs)] + [xs[nvox * nprocs:]]
ys = [ys[nvox * i:nvox * i + nvox] for i in range(nprocs)] + [ys[nvox * nprocs:]] ys = [ys[nvox * i:nvox * i + nvox] for i in range(nprocs)] + [ys[nvox * nprocs:]]
zs = [zs[nvox * i:nvox * i + nvox] for i in range(nprocs)] + [zs[nvox * nprocs:]] zs = [zs[nvox * i:nvox * i + nvox] for i in range(nprocs)] + [zs[nvox * nprocs:]]
# Build the argument lists for # Build the argument lists for
# each worker process. # each worker process.
args = [(data.shape, (x, y, z)) for x, y, z in zip(xs, ys, zs)] args = [(data.shape, (x, y, z)) for x, y, z in zip(xs, ys, zs)]
# Create a pool of worker # Create a pool of worker
# processes and run the jobs. # processes and run the jobs.
with mp.Pool(processes=nprocs) as pool: with mp.Pool(processes=nprocs) as pool:
pool.starmap(process_chunk, args) pool.starmap(process_chunk, args)
return outdata return outdata
``` ```
%% Cell type:markdown id:09269b65 tags: %% Cell type:markdown id:dedc20ea tags:
Now we can call our `process_data` function just like any other function: Now we can call our `process_data` function just like any other function:
%% Cell type:code id:27a07401 tags: %% Cell type:code id:b3bfb596 tags:
``` ```
indata = np.array(np.arange(64).reshape((4, 4, 4)), dtype=np.float64) indata = np.array(np.arange(64).reshape((4, 4, 4)), dtype=np.float64)
outdata = process_dataset(indata) outdata = process_dataset(indata)
print('Input') print('Input')
print(indata) print(indata)
print('Output') print('Output')
print(outdata) print(outdata)
``` ```
......
...@@ -21,9 +21,9 @@ module. If you want to be impressed, skip straight to the section on ...@@ -21,9 +21,9 @@ module. If you want to be impressed, skip straight to the section on
> bear the performance hit of copying data between processes, or jump through > bear the performance hit of copying data between processes, or jump through
> hoops order to share data between processes. > hoops order to share data between processes.
> >
> This limitation *might* be solved in a future Python release by way of > This limitation is being addressed in recent Python version, with
> [*sub-interpreters*](https://www.python.org/dev/peps/pep-0554/), but the > [_Free-threaded Python_ builds](https://docs.python.org/3/howto/free-threading-python.html),
> author of this practical is not holding his breath. > which will hopefully soon be the default behaviour.
* [Threading](#threading) * [Threading](#threading)
...@@ -310,6 +310,10 @@ cores - this essentially means that only one thread may be executing at any ...@@ -310,6 +310,10 @@ cores - this essentially means that only one thread may be executing at any
point in time. point in time.
> Note that this is likely to change in future Python releases, with the
> development of [_Free-threaded Python_](https://docs.python.org/3/howto/free-threading-python.html).
The `threading` module does still have its uses though, as this GIL problem The `threading` module does still have its uses though, as this GIL problem
does not affect tasks which involve calls to system or natively compiled does not affect tasks which involve calls to system or natively compiled
libraries (e.g. file and network I/O, Numpy operations, etc.). So you can, libraries (e.g. file and network I/O, Numpy operations, etc.). So you can,
......
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# `bokeh` # `bokeh`
[`bokeh`](https://docs.bokeh.org/en/latest/index.html) is a Python library for creating interactive visualizations for modern web browsers. `bokeh` allows you to create these interactive web-based plots without having to code in javascript. [`bokeh`](https://docs.bokeh.org/en/latest/index.html) is a Python library for creating interactive visualizations for modern web browsers. `bokeh` allows you to create these interactive web-based plots without having to code in javascript.
`bokeh` has excellent documentation: https://docs.bokeh.org/en/latest/index.html `bokeh` has excellent documentation: https://docs.bokeh.org/en/latest/index.html
This notebook is not intended to instruct you how to use `bokeh`. Instead it pulls together interesting examples from the `bokeh` documentation into a single notebook to give you a taster of what can be done with `bokeh`. This notebook is not intended to instruct you how to use `bokeh`. Instead it pulls together interesting examples from the `bokeh` documentation into a single notebook to give you a taster of what can be done with `bokeh`.
## Install `bokeh` ## Install `bokeh`
`bokeh` is not installed in the `fslpython` environment so you will need to install it to run this notebook (e.g. `$FSLDIR/bin/conda install -p $FSLDIR bokeh`) `bokeh` is not installed in the `fslpython` environment so you will need to install it to run this notebook (e.g. `$FSLDIR/bin/conda install -p $FSLDIR bokeh bokeh_sampledata`)
Setup bokeh to work in this notebook: Setup bokeh to work in this notebook:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
from bokeh.plotting import figure, output_file, show from bokeh.plotting import figure, output_file, show
from bokeh.io import output_notebook from bokeh.io import output_notebook
output_notebook() output_notebook()
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Fetch some sampledata for the examples:
%% Cell type:code id: tags:
``` python
from bokeh import sampledata
sampledata.download()
```
%% Cell type:markdown id: tags:
## Scatter Plots ## Scatter Plots
https://docs.bokeh.org/en/latest/docs/gallery/iris.html https://docs.bokeh.org/en/latest/docs/gallery/iris.html
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
from bokeh.sampledata.iris import flowers from bokeh.sampledata.iris import flowers
colormap = {'setosa': 'red', 'versicolor': 'green', 'virginica': 'blue'} colormap = {'setosa': 'red', 'versicolor': 'green', 'virginica': 'blue'}
colors = [colormap[x] for x in flowers['species']] colors = [colormap[x] for x in flowers['species']]
p = figure(title = "Iris Morphology") p = figure(title = "Iris Morphology")
p.xaxis.axis_label = 'Petal Length' p.xaxis.axis_label = 'Petal Length'
p.yaxis.axis_label = 'Petal Width' p.yaxis.axis_label = 'Petal Width'
p.circle(flowers["petal_length"], flowers["petal_width"], p.circle(flowers["petal_length"], flowers["petal_width"],
color=colors, fill_alpha=0.2, size=10) color=colors, fill_alpha=0.2, size=10)
# output_file("iris.html", title="iris.py example") # output_file("iris.html", title="iris.py example")
show(p) show(p)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Line Plots ## Line Plots
https://docs.bokeh.org/en/latest/docs/gallery/box_annotation.html https://docs.bokeh.org/en/latest/docs/gallery/box_annotation.html
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
from bokeh.models import BoxAnnotation from bokeh.models import BoxAnnotation
from bokeh.sampledata.glucose import data from bokeh.sampledata.glucose import data
TOOLS = "pan,wheel_zoom,box_zoom,reset,save" TOOLS = "pan,wheel_zoom,box_zoom,reset,save"
data = data.loc['2010-10-04':'2010-10-04'] data = data.loc['2010-10-04':'2010-10-04']
p = figure(x_axis_type="datetime", tools=TOOLS, title="Glocose Readings, Oct 4th (Red = Outside Range)") p = figure(x_axis_type="datetime", tools=TOOLS, title="Glocose Readings, Oct 4th (Red = Outside Range)")
p.background_fill_color = "#efefef" p.background_fill_color = "#efefef"
p.xgrid.grid_line_color=None p.xgrid.grid_line_color=None
p.xaxis.axis_label = 'Time' p.xaxis.axis_label = 'Time'
p.yaxis.axis_label = 'Value' p.yaxis.axis_label = 'Value'
p.line(data.index, data.glucose, line_color='grey') p.line(data.index, data.glucose, line_color='grey')
p.circle(data.index, data.glucose, color='grey', size=1) p.circle(data.index, data.glucose, color='grey', size=1)
p.add_layout(BoxAnnotation(top=80, fill_alpha=0.1, fill_color='red', line_color='red')) p.add_layout(BoxAnnotation(top=80, fill_alpha=0.1, fill_color='red', line_color='red'))
p.add_layout(BoxAnnotation(bottom=180, fill_alpha=0.1, fill_color='red', line_color='red')) p.add_layout(BoxAnnotation(bottom=180, fill_alpha=0.1, fill_color='red', line_color='red'))
show(p) show(p)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Bar Charts # Bar Charts
https://docs.bokeh.org/en/latest/docs/gallery/bar_stacked.html https://docs.bokeh.org/en/latest/docs/gallery/bar_stacked.html
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
from bokeh.palettes import Spectral5 from bokeh.palettes import Spectral5
from bokeh.sampledata.autompg import autompg_clean as df from bokeh.sampledata.autompg import autompg_clean as df
from bokeh.transform import factor_cmap from bokeh.transform import factor_cmap
df.cyl = df.cyl.astype(str) df.cyl = df.cyl.astype(str)
df.yr = df.yr.astype(str) df.yr = df.yr.astype(str)
group = df.groupby(['cyl', 'mfr']) group = df.groupby(['cyl', 'mfr'])
index_cmap = factor_cmap('cyl_mfr', palette=Spectral5, factors=sorted(df.cyl.unique()), end=1) index_cmap = factor_cmap('cyl_mfr', palette=Spectral5, factors=sorted(df.cyl.unique()), end=1)
p = figure(width=800, height=500, title="Mean MPG by # Cylinders and Manufacturer", p = figure(width=800, height=500, title="Mean MPG by # Cylinders and Manufacturer",
x_range=group, toolbar_location=None, tooltips=[("MPG", "@mpg_mean"), ("Cyl, Mfr", "@cyl_mfr")]) x_range=group, toolbar_location=None, tooltips=[("MPG", "@mpg_mean"), ("Cyl, Mfr", "@cyl_mfr")])
p.vbar(x='cyl_mfr', top='mpg_mean', width=1, source=group, p.vbar(x='cyl_mfr', top='mpg_mean', width=1, source=group,
line_color="white", fill_color=index_cmap, ) line_color="white", fill_color=index_cmap, )
p.y_range.start = 0 p.y_range.start = 0
p.x_range.range_padding = 0.05 p.x_range.range_padding = 0.05
p.xgrid.grid_line_color = None p.xgrid.grid_line_color = None
p.xaxis.axis_label = "Manufacturer grouped by # Cylinders" p.xaxis.axis_label = "Manufacturer grouped by # Cylinders"
p.xaxis.major_label_orientation = 1.2 p.xaxis.major_label_orientation = 1.2
p.outline_line_color = None p.outline_line_color = None
show(p) show(p)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Distribution Plots # Distribution Plots
https://docs.bokeh.org/en/latest/docs/gallery/histogram.html https://docs.bokeh.org/en/latest/docs/gallery/histogram.html
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import numpy as np import numpy as np
import scipy.special import scipy.special
from bokeh.layouts import gridplot from bokeh.layouts import gridplot
def make_plot(title, hist, edges, x, pdf, cdf): def make_plot(title, hist, edges, x, pdf, cdf):
p = figure(title=title, tools='', background_fill_color="#fafafa") p = figure(title=title, tools='', background_fill_color="#fafafa")
p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:], p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:],
fill_color="navy", line_color="white", alpha=0.5) fill_color="navy", line_color="white", alpha=0.5)
p.line(x, pdf, line_color="#ff8888", line_width=4, alpha=0.7, legend_label="PDF") p.line(x, pdf, line_color="#ff8888", line_width=4, alpha=0.7, legend_label="PDF")
p.line(x, cdf, line_color="orange", line_width=2, alpha=0.7, legend_label="CDF") p.line(x, cdf, line_color="orange", line_width=2, alpha=0.7, legend_label="CDF")
p.y_range.start = 0 p.y_range.start = 0
p.legend.location = "center_right" p.legend.location = "center_right"
p.legend.background_fill_color = "#fefefe" p.legend.background_fill_color = "#fefefe"
p.xaxis.axis_label = 'x' p.xaxis.axis_label = 'x'
p.yaxis.axis_label = 'Pr(x)' p.yaxis.axis_label = 'Pr(x)'
p.grid.grid_line_color="white" p.grid.grid_line_color="white"
return p return p
# Normal Distribution # Normal Distribution
mu, sigma = 0, 0.5 mu, sigma = 0, 0.5
measured = np.random.normal(mu, sigma, 1000) measured = np.random.normal(mu, sigma, 1000)
hist, edges = np.histogram(measured, density=True, bins=50) hist, edges = np.histogram(measured, density=True, bins=50)
x = np.linspace(-2, 2, 1000) x = np.linspace(-2, 2, 1000)
pdf = 1/(sigma * np.sqrt(2*np.pi)) * np.exp(-(x-mu)**2 / (2*sigma**2)) pdf = 1/(sigma * np.sqrt(2*np.pi)) * np.exp(-(x-mu)**2 / (2*sigma**2))
cdf = (1+scipy.special.erf((x-mu)/np.sqrt(2*sigma**2)))/2 cdf = (1+scipy.special.erf((x-mu)/np.sqrt(2*sigma**2)))/2
p1 = make_plot("Normal Distribution (μ=0, σ=0.5)", hist, edges, x, pdf, cdf) p1 = make_plot("Normal Distribution (μ=0, σ=0.5)", hist, edges, x, pdf, cdf)
# Log-Normal Distribution # Log-Normal Distribution
mu, sigma = 0, 0.5 mu, sigma = 0, 0.5
measured = np.random.lognormal(mu, sigma, 1000) measured = np.random.lognormal(mu, sigma, 1000)
hist, edges = np.histogram(measured, density=True, bins=50) hist, edges = np.histogram(measured, density=True, bins=50)
x = np.linspace(0.0001, 8.0, 1000) x = np.linspace(0.0001, 8.0, 1000)
pdf = 1/(x* sigma * np.sqrt(2*np.pi)) * np.exp(-(np.log(x)-mu)**2 / (2*sigma**2)) pdf = 1/(x* sigma * np.sqrt(2*np.pi)) * np.exp(-(np.log(x)-mu)**2 / (2*sigma**2))
cdf = (1+scipy.special.erf((np.log(x)-mu)/(np.sqrt(2)*sigma)))/2 cdf = (1+scipy.special.erf((np.log(x)-mu)/(np.sqrt(2)*sigma)))/2
p2 = make_plot("Log Normal Distribution (μ=0, σ=0.5)", hist, edges, x, pdf, cdf) p2 = make_plot("Log Normal Distribution (μ=0, σ=0.5)", hist, edges, x, pdf, cdf)
# Gamma Distribution # Gamma Distribution
k, theta = 7.5, 1.0 k, theta = 7.5, 1.0
measured = np.random.gamma(k, theta, 1000) measured = np.random.gamma(k, theta, 1000)
hist, edges = np.histogram(measured, density=True, bins=50) hist, edges = np.histogram(measured, density=True, bins=50)
x = np.linspace(0.0001, 20.0, 1000) x = np.linspace(0.0001, 20.0, 1000)
pdf = x**(k-1) * np.exp(-x/theta) / (theta**k * scipy.special.gamma(k)) pdf = x**(k-1) * np.exp(-x/theta) / (theta**k * scipy.special.gamma(k))
cdf = scipy.special.gammainc(k, x/theta) cdf = scipy.special.gammainc(k, x/theta)
p3 = make_plot("Gamma Distribution (k=7.5, θ=1)", hist, edges, x, pdf, cdf) p3 = make_plot("Gamma Distribution (k=7.5, θ=1)", hist, edges, x, pdf, cdf)
# Weibull Distribution # Weibull Distribution
lam, k = 1, 1.25 lam, k = 1, 1.25
measured = lam*(-np.log(np.random.uniform(0, 1, 1000)))**(1/k) measured = lam*(-np.log(np.random.uniform(0, 1, 1000)))**(1/k)
hist, edges = np.histogram(measured, density=True, bins=50) hist, edges = np.histogram(measured, density=True, bins=50)
x = np.linspace(0.0001, 8, 1000) x = np.linspace(0.0001, 8, 1000)
pdf = (k/lam)*(x/lam)**(k-1) * np.exp(-(x/lam)**k) pdf = (k/lam)*(x/lam)**(k-1) * np.exp(-(x/lam)**k)
cdf = 1 - np.exp(-(x/lam)**k) cdf = 1 - np.exp(-(x/lam)**k)
p4 = make_plot("Weibull Distribution (λ=1, k=1.25)", hist, edges, x, pdf, cdf) p4 = make_plot("Weibull Distribution (λ=1, k=1.25)", hist, edges, x, pdf, cdf)
show(gridplot([p1,p2,p3,p4], ncols=2, width=400, height=400, toolbar_location=None)) show(gridplot([p1,p2,p3,p4], ncols=2, width=400, height=400, toolbar_location=None))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Boxplot # Boxplot
https://docs.bokeh.org/en/latest/docs/gallery/boxplot.html https://docs.bokeh.org/en/latest/docs/gallery/boxplot.html
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import numpy as np import numpy as np
import pandas as pd import pandas as pd
# generate some synthetic time series for six different categories # generate some synthetic time series for six different categories
cats = list("abcdef") cats = list("abcdef")
yy = np.random.randn(2000) yy = np.random.randn(2000)
g = np.random.choice(cats, 2000) g = np.random.choice(cats, 2000)
for i, l in enumerate(cats): for i, l in enumerate(cats):
yy[g == l] += i // 2 yy[g == l] += i // 2
df = pd.DataFrame(dict(score=yy, group=g)) df = pd.DataFrame(dict(score=yy, group=g))
# find the quartiles and IQR for each category # find the quartiles and IQR for each category
groups = df.groupby('group') groups = df.groupby('group')
q1 = groups.quantile(q=0.25) q1 = groups.quantile(q=0.25)
q2 = groups.quantile(q=0.5) q2 = groups.quantile(q=0.5)
q3 = groups.quantile(q=0.75) q3 = groups.quantile(q=0.75)
iqr = q3 - q1 iqr = q3 - q1
upper = q3 + 1.5*iqr upper = q3 + 1.5*iqr
lower = q1 - 1.5*iqr lower = q1 - 1.5*iqr
# find the outliers for each category # find the outliers for each category
def outliers(group): def outliers(group):
cat = group.name cat = group.name
return group[(group.score > upper.loc[cat]['score']) | (group.score < lower.loc[cat]['score'])]['score'] return group[(group.score > upper.loc[cat]['score']) | (group.score < lower.loc[cat]['score'])]['score']
out = groups.apply(outliers).dropna() out = groups.apply(outliers).dropna()
# prepare outlier data for plotting, we need coordinates for every outlier. # prepare outlier data for plotting, we need coordinates for every outlier.
if not out.empty: if not out.empty:
outx = list(out.index.get_level_values(0)) outx = list(out.index.get_level_values(0))
outy = list(out.values) outy = list(out.values)
p = figure(tools="", background_fill_color="#efefef", x_range=cats, toolbar_location=None) p = figure(tools="", background_fill_color="#efefef", x_range=cats, toolbar_location=None)
# if no outliers, shrink lengths of stems to be no longer than the minimums or maximums # if no outliers, shrink lengths of stems to be no longer than the minimums or maximums
qmin = groups.quantile(q=0.00) qmin = groups.quantile(q=0.00)
qmax = groups.quantile(q=1.00) qmax = groups.quantile(q=1.00)
upper.score = [min([x,y]) for (x,y) in zip(list(qmax.loc[:,'score']),upper.score)] upper.score = [min([x,y]) for (x,y) in zip(list(qmax.loc[:,'score']),upper.score)]
lower.score = [max([x,y]) for (x,y) in zip(list(qmin.loc[:,'score']),lower.score)] lower.score = [max([x,y]) for (x,y) in zip(list(qmin.loc[:,'score']),lower.score)]
# stems # stems
p.segment(cats, upper.score, cats, q3.score, line_color="black") p.segment(cats, upper.score, cats, q3.score, line_color="black")
p.segment(cats, lower.score, cats, q1.score, line_color="black") p.segment(cats, lower.score, cats, q1.score, line_color="black")
# boxes # boxes
p.vbar(cats, 0.7, q2.score, q3.score, fill_color="#E08E79", line_color="black") p.vbar(cats, 0.7, q2.score, q3.score, fill_color="#E08E79", line_color="black")
p.vbar(cats, 0.7, q1.score, q2.score, fill_color="#3B8686", line_color="black") p.vbar(cats, 0.7, q1.score, q2.score, fill_color="#3B8686", line_color="black")
# whiskers (almost-0 height rects simpler than segments) # whiskers (almost-0 height rects simpler than segments)
p.rect(cats, lower.score, 0.2, 0.01, line_color="black") p.rect(cats, lower.score, 0.2, 0.01, line_color="black")
p.rect(cats, upper.score, 0.2, 0.01, line_color="black") p.rect(cats, upper.score, 0.2, 0.01, line_color="black")
# outliers # outliers
if not out.empty: if not out.empty:
p.circle(outx, outy, size=6, color="#F38630", fill_alpha=0.6) p.circle(outx, outy, size=6, color="#F38630", fill_alpha=0.6)
p.xgrid.grid_line_color = None p.xgrid.grid_line_color = None
p.ygrid.grid_line_color = "white" p.ygrid.grid_line_color = "white"
p.grid.grid_line_width = 2 p.grid.grid_line_width = 2
p.xaxis.major_label_text_font_size="16px" p.xaxis.major_label_text_font_size="16px"
show(p) show(p)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Connectivity Matrix ## Connectivity Matrix
https://docs.bokeh.org/en/latest/docs/gallery/les_mis.html https://docs.bokeh.org/en/latest/docs/gallery/les_mis.html
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import numpy as np import numpy as np
from bokeh.sampledata.les_mis import data from bokeh.sampledata.les_mis import data
nodes = data['nodes'] nodes = data['nodes']
names = [node['name'] for node in sorted(data['nodes'], key=lambda x: x['group'])] names = [node['name'] for node in sorted(data['nodes'], key=lambda x: x['group'])]
N = len(nodes) N = len(nodes)
counts = np.zeros((N, N)) counts = np.zeros((N, N))
for link in data['links']: for link in data['links']:
counts[link['source'], link['target']] = link['value'] counts[link['source'], link['target']] = link['value']
counts[link['target'], link['source']] = link['value'] counts[link['target'], link['source']] = link['value']
colormap = ["#444444", "#a6cee3", "#1f78b4", "#b2df8a", "#33a02c", "#fb9a99", colormap = ["#444444", "#a6cee3", "#1f78b4", "#b2df8a", "#33a02c", "#fb9a99",
"#e31a1c", "#fdbf6f", "#ff7f00", "#cab2d6", "#6a3d9a"] "#e31a1c", "#fdbf6f", "#ff7f00", "#cab2d6", "#6a3d9a"]
xname = [] xname = []
yname = [] yname = []
color = [] color = []
alpha = [] alpha = []
for i, node1 in enumerate(nodes): for i, node1 in enumerate(nodes):
for j, node2 in enumerate(nodes): for j, node2 in enumerate(nodes):
xname.append(node1['name']) xname.append(node1['name'])
yname.append(node2['name']) yname.append(node2['name'])
alpha.append(min(counts[i,j]/4.0, 0.9) + 0.1) alpha.append(min(counts[i,j]/4.0, 0.9) + 0.1)
if node1['group'] == node2['group']: if node1['group'] == node2['group']:
color.append(colormap[node1['group']]) color.append(colormap[node1['group']])
else: else:
color.append('lightgrey') color.append('lightgrey')
data=dict( data=dict(
xname=xname, xname=xname,
yname=yname, yname=yname,
colors=color, colors=color,
alphas=alpha, alphas=alpha,
count=counts.flatten(), count=counts.flatten(),
) )
p = figure(title="Les Mis Occurrences", p = figure(title="Les Mis Occurrences",
x_axis_location="above", tools="hover,save", x_axis_location="above", tools="hover,save",
x_range=list(reversed(names)), y_range=names, x_range=list(reversed(names)), y_range=names,
tooltips = [('names', '@yname, @xname'), ('count', '@count')]) tooltips = [('names', '@yname, @xname'), ('count', '@count')])
p.width = 800 p.width = 800
p.height = 800 p.height = 800
p.grid.grid_line_color = None p.grid.grid_line_color = None
p.axis.axis_line_color = None p.axis.axis_line_color = None
p.axis.major_tick_line_color = None p.axis.major_tick_line_color = None
p.axis.major_label_text_font_size = "7px" p.axis.major_label_text_font_size = "7px"
p.axis.major_label_standoff = 0 p.axis.major_label_standoff = 0
p.xaxis.major_label_orientation = np.pi/3 p.xaxis.major_label_orientation = np.pi/3
p.rect('xname', 'yname', 0.9, 0.9, source=data, p.rect('xname', 'yname', 0.9, 0.9, source=data,
color='colors', alpha='alphas', line_color=None, color='colors', alpha='alphas', line_color=None,
hover_line_color='black', hover_color='colors') hover_line_color='black', hover_color='colors')
show(p) # show the plot show(p) # show the plot
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Sliders ## Sliders
https://docs.bokeh.org/en/latest/docs/gallery/slider.html https://docs.bokeh.org/en/latest/docs/gallery/slider.html
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import numpy as np import numpy as np
from bokeh.layouts import column, row from bokeh.layouts import column, row
from bokeh.models import CustomJS, Slider from bokeh.models import CustomJS, Slider
from bokeh.plotting import ColumnDataSource from bokeh.plotting import ColumnDataSource
x = np.linspace(0, 10, 500) x = np.linspace(0, 10, 500)
y = np.sin(x) y = np.sin(x)
source = ColumnDataSource(data=dict(x=x, y=y)) source = ColumnDataSource(data=dict(x=x, y=y))
plot = figure(y_range=(-10, 10), width=400, height=400) plot = figure(y_range=(-10, 10), width=400, height=400)
plot.line('x', 'y', source=source, line_width=3, line_alpha=0.6) plot.line('x', 'y', source=source, line_width=3, line_alpha=0.6)
amp_slider = Slider(start=0.1, end=10, value=1, step=.1, title="Amplitude") amp_slider = Slider(start=0.1, end=10, value=1, step=.1, title="Amplitude")
freq_slider = Slider(start=0.1, end=10, value=1, step=.1, title="Frequency") freq_slider = Slider(start=0.1, end=10, value=1, step=.1, title="Frequency")
phase_slider = Slider(start=0, end=6.4, value=0, step=.1, title="Phase") phase_slider = Slider(start=0, end=6.4, value=0, step=.1, title="Phase")
offset_slider = Slider(start=-5, end=5, value=0, step=.1, title="Offset") offset_slider = Slider(start=-5, end=5, value=0, step=.1, title="Offset")
callback = CustomJS(args=dict(source=source, amp=amp_slider, freq=freq_slider, phase=phase_slider, offset=offset_slider), callback = CustomJS(args=dict(source=source, amp=amp_slider, freq=freq_slider, phase=phase_slider, offset=offset_slider),
code=""" code="""
const data = source.data; const data = source.data;
const A = amp.value; const A = amp.value;
const k = freq.value; const k = freq.value;
const phi = phase.value; const phi = phase.value;
const B = offset.value; const B = offset.value;
const x = data['x'] const x = data['x']
const y = data['y'] const y = data['y']
for (var i = 0; i < x.length; i++) { for (var i = 0; i < x.length; i++) {
y[i] = B + A*Math.sin(k*x[i]+phi); y[i] = B + A*Math.sin(k*x[i]+phi);
} }
source.change.emit(); source.change.emit();
""") """)
amp_slider.js_on_change('value', callback) amp_slider.js_on_change('value', callback)
freq_slider.js_on_change('value', callback) freq_slider.js_on_change('value', callback)
phase_slider.js_on_change('value', callback) phase_slider.js_on_change('value', callback)
offset_slider.js_on_change('value', callback) offset_slider.js_on_change('value', callback)
layout = row( layout = row(
plot, plot,
column(amp_slider, freq_slider, phase_slider, offset_slider), column(amp_slider, freq_slider, phase_slider, offset_slider),
) )
show(layout) show(layout)
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
``` ```
......
This diff is collapsed.
...@@ -79,7 +79,7 @@ def ortho(data, voxel, fig=None, cursor=False, **kwargs): ...@@ -79,7 +79,7 @@ def ortho(data, voxel, fig=None, cursor=False, **kwargs):
voxel = [int(round(v)) for v in voxel] voxel = [int(round(v)) for v in voxel]
data = np.asanyarray(data, dtype=np.float) data = np.asanyarray(data, dtype=float)
data[data <= 0] = np.nan data[data <= 0] = np.nan
x, y, z = voxel x, y, z = voxel
......
This diff is collapsed.
...@@ -88,7 +88,7 @@ Each column in the pandas dataframe has its own data type, which can be: ...@@ -88,7 +88,7 @@ Each column in the pandas dataframe has its own data type, which can be:
- datetime for defining specific times (and timedelta for durations) - datetime for defining specific times (and timedelta for durations)
- categorical, where each element is selected from a finite list of text values - categorical, where each element is selected from a finite list of text values
- objects for anything else used for strings or columns with mixed elements - objects for anything else used for strings or columns with mixed elements
Each element in the column must match the type of the whole column. Each element in the column must match the type of the whole column.
When reading in a dataset pandas will try to assign the most specific type to each column. When reading in a dataset pandas will try to assign the most specific type to each column.
Every pandas datatype also has support for missing data (which we will look more at below). Every pandas datatype also has support for missing data (which we will look more at below).
...@@ -99,8 +99,8 @@ titanic.dtypes ...@@ -99,8 +99,8 @@ titanic.dtypes
Note that in much of python data types are referred to as dtypes. Note that in much of python data types are referred to as dtypes.
## Getting your data out ## Getting your data out
For some applications you might want to For some applications you might want to
extract your data as a numpy array, even though more and more projects extract your data as a numpy array, even though more and more projects
support pandas Dataframes directly (including "scikit-learn"). support pandas Dataframes directly (including "scikit-learn").
The underlying numpy array can be accessed using the `to_numpy` method The underlying numpy array can be accessed using the `to_numpy` method
``` ```
titanic.to_numpy() titanic.to_numpy()
...@@ -114,7 +114,7 @@ In this case this means that all data had to be converted ...@@ -114,7 +114,7 @@ In this case this means that all data had to be converted
to the generic "object" type, which is not particularly useful. to the generic "object" type, which is not particularly useful.
For most analyses, we would only be interested in the numeric columns. For most analyses, we would only be interested in the numeric columns.
Thise can be extracted using `select_dtypes`, which selects specific columns Thise can be extracted using `select_dtypes`, which selects specific columns
based on their data type (dtype): based on their data type (dtype):
``` ```
titanic.select_dtypes(include=np.number).to_numpy() titanic.select_dtypes(include=np.number).to_numpy()
...@@ -128,7 +128,7 @@ These are columns where each element has one of a finite list of possible values ...@@ -128,7 +128,7 @@ These are columns where each element has one of a finite list of possible values
(e.g., the "embark_town" column being "Southampton", "Cherbourg", or, "Queenstown, (e.g., the "embark_town" column being "Southampton", "Cherbourg", or, "Queenstown,
which are the three towns the Titanic docked to let on passengers). which are the three towns the Titanic docked to let on passengers).
As we will see below, `pandas` has extensive support for categorical values, As we will see below, `pandas` has extensive support for categorical values,
but many other tools do not. To support those tools, `pandas` allows you to but many other tools do not. To support those tools, `pandas` allows you to
replace such columns with dummy variables: replace such columns with dummy variables:
``` ```
pd.get_dummies(titanic) pd.get_dummies(titanic)
...@@ -160,7 +160,7 @@ titanic.embark_town ...@@ -160,7 +160,7 @@ titanic.embark_town
Note that this returns a single column is represented by a pandas Note that this returns a single column is represented by a pandas
[`Series`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html) [`Series`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html)
rather than a `DataFrame` object. A `Series` is a 1-dimensional array rather than a `DataFrame` object. A `Series` is a 1-dimensional array
representing a single column, while a `DataFrame` is a collection of `Series` representing a single column, while a `DataFrame` is a collection of `Series`
representing a table. Multiple columns can be returned by providing a representing a table. Multiple columns can be returned by providing a
list of columns names. This will return a `DataFrame`: list of columns names. This will return a `DataFrame`:
...@@ -236,9 +236,9 @@ titanic_sorted.tail(3) ...@@ -236,9 +236,9 @@ titanic_sorted.tail(3)
``` ```
Note that nearly all methods in pandas return a new `DataFrame`, which means Note that nearly all methods in pandas return a new `DataFrame`, which means
that we can "chain" methods. What we mean by "chaining" is that rather than that we can "chain" methods. What we mean by "chaining" is that rather than
storing a returned `DataFrame`, we instead call a different method on it. storing a returned `DataFrame`, we instead call a different method on it.
For example, below the `tail` method call returns a new `DataFrame` For example, below the `tail` method call returns a new `DataFrame`
on which we then call the `head` method: on which we then call the `head` method:
``` ```
...@@ -246,8 +246,8 @@ titanic_sorted.tail(10).head(5) # select the first 5 of the last 10 passengers ...@@ -246,8 +246,8 @@ titanic_sorted.tail(10).head(5) # select the first 5 of the last 10 passengers
``` ```
> This chaining is usually very efficient, because when creating a new `DataFrame` > This chaining is usually very efficient, because when creating a new `DataFrame`
> pandas minimizes the copying of the underlying data (similar to what `numpy` arrays do). > pandas minimizes the copying of the underlying data (similar to what `numpy` arrays do).
> While this allows for fast manipulation of the data, it does mean that if you update the > While this allows for fast manipulation of the data, it does mean that if you update the
> data in one of your `DataFrame` the data *might* also be changed in related `DataFrame`s > data in one of your `DataFrame` the data *might* also be changed in related `DataFrame`s
``` ```
...@@ -288,7 +288,7 @@ memory). ...@@ -288,7 +288,7 @@ memory).
titanic.query('(age > 60) & (embark_town == "Southampton")') titanic.query('(age > 60) & (embark_town == "Southampton")')
``` ```
When selecting a categorical multiple options from a categorical values you When selecting a categorical multiple options from a categorical values you
might want to use `isin`: might want to use `isin`:
``` ```
titanic[titanic['class'].isin(['First','Second'])] # select all passangers not in first or second class titanic[titanic['class'].isin(['First','Second'])] # select all passangers not in first or second class
...@@ -358,7 +358,7 @@ plotting functions expect to get a pandas `DataFrame` (although they will work ...@@ -358,7 +358,7 @@ plotting functions expect to get a pandas `DataFrame` (although they will work
with Numpy arrays as well). So we can plot age vs. fare like: with Numpy arrays as well). So we can plot age vs. fare like:
``` ```
sns.jointplot('age', 'fare', data=titanic) sns.jointplot(titanic, x='age', y='fare')
``` ```
**Exercise**: check the documentation from `sns.jointplot` (hover the mouse **Exercise**: check the documentation from `sns.jointplot` (hover the mouse
...@@ -366,7 +366,7 @@ over the text `jointplot` and press shift-tab) to find out how to turn the ...@@ -366,7 +366,7 @@ over the text `jointplot` and press shift-tab) to find out how to turn the
scatter plot into a density (kde) map scatter plot into a density (kde) map
``` ```
sns.jointplot('age', 'fare', data=titanic, ...) sns.jointplot(titanic, x='age', y='fare', ...)
``` ```
Here is just a brief example of how we can use multiple columns to illustrate Here is just a brief example of how we can use multiple columns to illustrate
...@@ -402,26 +402,26 @@ sns.violinplot(x='class', y='age', hue='sex', data=titanic, split=True, ...@@ -402,26 +402,26 @@ sns.violinplot(x='class', y='age', hue='sex', data=titanic, split=True,
There are a large number of built-in methods to summarize the observations in There are a large number of built-in methods to summarize the observations in
a Pandas `DataFrame`. Most of these will return a `Series` with the columns a Pandas `DataFrame`. Most of these will return a `Series` with the columns
names as index: names as the index. Some of these operations are only applicable to numeric
columns - we can use `select_dtypes(include='number')` to extract them:
``` ```
titanic.mean() titanic.select_dtypes(include='number').mean()
``` ```
``` ```
titanic.quantile(0.75) titanic.select_dtypes(include='number').quantile(0.75)
``` ```
One very useful one is `describe`, which gives an overview of many common One very useful one is `describe`, which gives an overview of many common
summary measures summary measures (note that non-numeric columns are ignored when summarizing
data in this way):
``` ```
titanic.describe() titanic.describe()
``` ```
Note that non-numeric columns are ignored when summarizing data in this way. For a more detailed exploration of the data, you might want to check
For a more detailed exploration of the data, you might want to check
[pandas_profiling](https://pandas-profiling.github.io/pandas-profiling/docs/) [pandas_profiling](https://pandas-profiling.github.io/pandas-profiling/docs/)
(not installed in fslpython, so the following will not run in fslpython): (not installed in fslpython, so the following will not run in fslpython):
...@@ -475,68 +475,27 @@ handbook](https://jakevdp.github.io/PythonDataScienceHandbook/06.00-figure-code. ...@@ -475,68 +475,27 @@ handbook](https://jakevdp.github.io/PythonDataScienceHandbook/06.00-figure-code.
![group by image](group_by.png) ![group by image](group_by.png)
``` ```
titanic.groupby('class').mean() titanic.groupby('class').mean(numeric_only=True)
``` ```
> Here we use the `numeric_only` argument to `mean()` to ignore non-numeric
> columns, instead of the `select_dtypes` function. The `numeric_only` argument
> is supported by some, but not all, pandas routines.
We can also group by multiple variables at once We can also group by multiple variables at once
``` ```
titanic.groupby(['class', 'survived']).mean() # as always in pandas supply multiple column names as lists, not tuples # as always in pandas supply multiple column names as lists, not tuples
titanic.groupby(['class', 'survived']).mean(numeric_only=True)
``` ```
When grouping it can help to use the `cut` method to split a continuous variable When grouping it can help to use the `cut` method to split a continuous variable
into a categorical one into a categorical one
``` ```
titanic.groupby(['class', pd.cut(titanic.age, bins=(0, 18, 50, np.inf))]).mean() titanic.groupby(['class', pd.cut(titanic.age, bins=(0, 18, 50, np.inf))]).mean(numeric_only=True)
```
We can use the `aggregate` method to apply a different function to each series
```
titanic.groupby(['class', 'survived']).aggregate((np.median, 'mad'))
```
Note that both the index (on the left) and the column names (on the top) now
have multiple levels. Such an index is referred to as `MultiIndex`.
This does complicate selecting specific columns/rows. You can read more of using
`MultiIndex` [here](http://pandas.pydata.org/pandas-docs/stable/advanced.html).
The short version is that columns can be selected using direct indexing (as
discussed above)
```
df_full = titanic.groupby(['class', 'survived']).aggregate((np.median, 'mad'))
```
```
df_full[('age', 'median')] # selects median age column; note that the round brackets are optional
```
```
df_full['age'] # selects both age columns
```
Remember that indexing based on the index was done through `loc`. The rest is
the same as for the columns above
```
df_full.loc[('First', 0)]
```
``` ```
df_full.loc['First']
```
More advanced use of the `MultiIndex` is possible through `xs`:
```
df_full.xs(0, level='survived') # selects all the zero's from the survived index
```
```
df_full.xs('mad', axis=1, level=1) # selects mad from the second level in the columns (i.e., axis=1)
```
## Reshaping tables ## Reshaping tables
...@@ -577,7 +536,7 @@ ax = sns.heatmap(titanic.groupby(['class', 'sex']).survived.mean().unstack('sex' ...@@ -577,7 +536,7 @@ ax = sns.heatmap(titanic.groupby(['class', 'sex']).survived.mean().unstack('sex'
ax.set_title('survival rate') ax.set_title('survival rate')
``` ```
> There are also many ways to produce prettier tables in pandas. > There are also many ways to produce prettier tables in pandas.
> This is documented [here](http://pandas.pydata.org/pandas-docs/stable/style.html). > This is documented [here](http://pandas.pydata.org/pandas-docs/stable/style.html).
Because this stacking/unstacking is fairly common after a groupby operation, Because this stacking/unstacking is fairly common after a groupby operation,
...@@ -586,7 +545,7 @@ there is a shortcut for it: `pivot_table` ...@@ -586,7 +545,7 @@ there is a shortcut for it: `pivot_table`
``` ```
titanic.pivot_table('survived', 'class', 'sex') titanic.pivot_table('survived', 'class', 'sex')
``` ```
The first argument is the numeric variable that will be summarised. The first argument is the numeric variable that will be summarised.
The next arguments indicates which categorical variable(s) should be The next arguments indicates which categorical variable(s) should be
used as respectively index or column. used as respectively index or column.
...@@ -668,14 +627,14 @@ to group the table based on tract (`groupby` or `pivot_table`). ...@@ -668,14 +627,14 @@ to group the table based on tract (`groupby` or `pivot_table`).
``` ```
In general pandas is better at handling long-form than wide-form data, because In general pandas is better at handling long-form than wide-form data, because
you can use `groupby` on long-form data. you can use `groupby` on long-form data.
For better visualization of the data an intermediate format is often preferred. One For better visualization of the data an intermediate format is often preferred. One
exception is calculating a covariance (`DataFrame.cov`) or correlation exception is calculating a covariance (`DataFrame.cov`) or correlation
(`DataFrame.corr`) matrices which require the variables that will be compared (`DataFrame.corr`) matrices which require the variables that will be compared
to be columns: to be columns:
``` ```
sns.heatmap(df_wide.corr(), cmap=sns.diverging_palette(240, 10, s=99, n=300), ) sns.heatmap(df_wide.corr(numeric_only=True), cmap=sns.diverging_palette(240, 10, s=99, n=300), )
``` ```
## Linear fitting (`statsmodels`) ## Linear fitting (`statsmodels`)
...@@ -690,7 +649,7 @@ more detailed description of the R-style formulas ...@@ -690,7 +649,7 @@ more detailed description of the R-style formulas
In short these functions describe the linear model as a string. For example, In short these functions describe the linear model as a string. For example,
`"y ~ x + a + x * a"` fits the variable `y` as a function of `x`, `a`, and the `"y ~ x + a + x * a"` fits the variable `y` as a function of `x`, `a`, and the
interaction between `x` and `a`. The intercept parameter is included by default interaction between `x` and `a`. The intercept parameter is included by default
(you can add `"+ 0"` to remove it). (you can add `"+ 0"` to remove it).
``` ```
...@@ -727,6 +686,6 @@ Not all data is well represented by a 2D table. If you want more dimensions to f ...@@ -727,6 +686,6 @@ Not all data is well represented by a 2D table. If you want more dimensions to f
Other useful features: Other useful features:
- [Concatenating and merging tables](https://pandas.pydata.org/pandas-docs/stable/getting_started/intro_tutorials/08_combine_dataframes.html) - [Concatenating and merging tables](https://pandas.pydata.org/pandas-docs/stable/getting_started/intro_tutorials/08_combine_dataframes.html)
- [Lots of time series support](https://pandas.pydata.org/pandas-docs/stable/getting_started/intro_tutorials/09_timeseries.html) - [Lots of time series support](https://pandas.pydata.org/pandas-docs/stable/getting_started/intro_tutorials/09_timeseries.html)
- [Rolling Window functions](http://pandas.pydata.org/pandas-docs/stable/computation.html#window-functions) - [Rolling Window functions](http://pandas.pydata.org/pandas-docs/stable/computation.html#window-functions)
for after you have meaningfully sorted your data for after you have meaningfully sorted your data
- and much, much more - and much, much more
\ No newline at end of file
This diff is collapsed.
...@@ -530,7 +530,7 @@ froth coming out of your mouth. I guess you're angry that `a * b` didn't give ...@@ -530,7 +530,7 @@ froth coming out of your mouth. I guess you're angry that `a * b` didn't give
you the matrix product, like it would have in Matlab. Well all I can say is you the matrix product, like it would have in Matlab. Well all I can say is
that Numpy is not Matlab. Matlab operations are typically consistent with that Numpy is not Matlab. Matlab operations are typically consistent with
linear algebra notation. This is not the case in Numpy. Get over it. linear algebra notation. This is not the case in Numpy. Get over it.
[Get yourself a calmative](https://youtu.be/M_w_n-8w3IQ?t=32). [Get yourself a calmative]h(https://youtu.be/W5R_jNLyPNw?t=32).
<a class="anchor" id="matrix-multiplication"></a> <a class="anchor" id="matrix-multiplication"></a>
......
%% Cell type:markdown id:ea6b9781 tags: %% Cell type:markdown id:d7eda4f4 tags:
# Plotting with python # Plotting with python
The main plotting module in python is `matplotlib`. There is a lot The main plotting module in python is `matplotlib`. There is a lot
that can be done with it - see the [webpage](https://matplotlib.org/gallery/index.html) that can be done with it - see the [webpage](https://matplotlib.org/gallery/index.html)
## Contents ## Contents
* [Running inside a notebook](#inside-notebook) * [Running inside a notebook](#inside-notebook)
* [2D plots](#2D-plots) * [2D plots](#2D-plots)
* [Histograms and Bar Plots](#histograms) * [Histograms and Bar Plots](#histograms)
* [Scatter plots](#scatter-plots) * [Scatter plots](#scatter-plots)
* [Subplots](#subplots) * [Subplots](#subplots)
* [Displaying Images](#displaying-images) * [Displaying Images](#displaying-images)
* [3D plots](#3D-plots) * [3D plots](#3D-plots)
* [Running in a standalone script](#plotting-in-scripts) * [Running in a standalone script](#plotting-in-scripts)
* [Exercise](#exercise) * [Exercise](#exercise)
--- ---
<a class="anchor" id="inside-notebook"></a> <a class="anchor" id="inside-notebook"></a>
## Inside a notebook ## Inside a notebook
Inside a jupyter notebook you get access to this in a slightly Inside a jupyter notebook you get access to this in a slightly
different way, compared to other modules: different way, compared to other modules:
%% Cell type:code id:35ff2bd1 tags: %% Cell type:code id:efa134cb tags:
``` ```
%matplotlib inline %matplotlib inline
``` ```
%% Cell type:markdown id:c3d1fa16 tags: %% Cell type:markdown id:7193841d tags:
This only needs to be done once in a notebook, like for standard imports. This only needs to be done once in a notebook, like for standard imports.
> There are also other alternatives, including interactive versions - see the practical on Jupyter notebooks for more information about this. > There are also other alternatives, including interactive versions - see the practical on Jupyter notebooks for more information about this.
The library works very similarly to plotting in matlab. Let's start The library works very similarly to plotting in matlab. Let's start
with some simple examples. with some simple examples.
<a class="anchor" id="2D-plots"></a> <a class="anchor" id="2D-plots"></a>
### 2D plots ### 2D plots
%% Cell type:code id:0963c7e2 tags: %% Cell type:code id:1176b4d3 tags:
``` ```
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import numpy as np import numpy as np
plt.style.use('bmh') plt.style.use('bmh')
x = np.linspace(-np.pi, np.pi, 256) x = np.linspace(-np.pi, np.pi, 256)
cosx, sinx = np.cos(x), np.sin(x) cosx, sinx = np.cos(x), np.sin(x)
plt.plot(x, cosx) plt.plot(x, cosx)
plt.plot(x, sinx, color='red', linewidth=4, linestyle='-.') plt.plot(x, sinx, color='red', linewidth=4, linestyle='-.')
plt.plot(x, sinx**2) plt.plot(x, sinx**2)
plt.xlim(-np.pi, np.pi) plt.xlim(-np.pi, np.pi)
plt.title('Our first plots') plt.title('Our first plots')
``` ```
%% Cell type:markdown id:801fad5d tags: %% Cell type:markdown id:1efbf859 tags:
> Note that the `plt.style.use('bmh')` command is not necessary, but it > Note that the `plt.style.use('bmh')` command is not necessary, but it
> does make nicer looking plots in general. You can use `ggplot` > does make nicer looking plots in general. You can use `ggplot`
> instead of `bmh` if you want something resembling plots made by R. > instead of `bmh` if you want something resembling plots made by R.
> For a list of options run: `print(plt.style.available)` > For a list of options run: `print(plt.style.available)`
You can also save the objects and interrogate/set their properties, as You can also save the objects and interrogate/set their properties, as
well as those for the general axes: well as those for the general axes:
%% Cell type:code id:da1d3278 tags: %% Cell type:code id:aa158675 tags:
``` ```
hdl = plt.plot(x, cosx) hdl = plt.plot(x, cosx)
print(hdl[0].get_color()) print(hdl[0].get_color())
hdl[0].set_color('#707010') hdl[0].set_color('#707010')
hdl[0].set_linewidth(0.5) hdl[0].set_linewidth(0.5)
plt.grid('off') plt.grid('off')
``` ```
%% Cell type:markdown id:26173cb4 tags: %% Cell type:markdown id:80251d8f tags:
Use `dir()` or `help()` or the online docs to get more info on what Use `dir()` or `help()` or the online docs to get more info on what
you can do with these. you can do with these.
<a class="anchor" id="histograms"></a> <a class="anchor" id="histograms"></a>
### Histograms and bar charts ### Histograms and bar charts
For a simple histogram you can do this: For a simple histogram you can do this:
%% Cell type:code id:7a7ef2b9 tags: %% Cell type:code id:627698de tags:
``` ```
r = np.random.rand(1000) r = np.random.rand(1000)
n,bins,_ = plt.hist((r-0.5)**2, bins=30) n,bins,_ = plt.hist((r-0.5)**2, bins=30)
``` ```
%% Cell type:markdown id:3ec49cd2 tags: %% Cell type:markdown id:28fdd0c5 tags:
where it also returns the number of elements in each bin, as `n`, and where it also returns the number of elements in each bin, as `n`, and
the bin centres, as `bins`. the bin centres, as `bins`.
> The `_` in the third part on the left > The `_` in the third part on the left
> hand side is a shorthand for just throwing away the corresponding part > hand side is a shorthand for just throwing away the corresponding part
> of the return structure. > of the return structure.
There is also a call for doing bar plots: There is also a call for doing bar plots:
%% Cell type:code id:69572b88 tags: %% Cell type:code id:73a6d691 tags:
``` ```
samp1 = r[0:10] samp1 = r[0:10]
samp2 = r[10:20] samp2 = r[10:20]
bwidth = 0.3 bwidth = 0.3
xcoord = np.arange(10) xcoord = np.arange(10)
plt.bar(xcoord-bwidth, samp1, width=bwidth, color='red', label='Sample 1') plt.bar(xcoord-bwidth, samp1, width=bwidth, color='red', label='Sample 1')
plt.bar(xcoord, samp2, width=bwidth, color='blue', label='Sample 2') plt.bar(xcoord, samp2, width=bwidth, color='blue', label='Sample 2')
plt.legend(loc='upper left') plt.legend(loc='upper left')
``` ```
%% Cell type:markdown id:94e33c3e tags: %% Cell type:markdown id:fb8ec42a tags:
<a class="anchor" id="scatter-plots"></a> <a class="anchor" id="scatter-plots"></a>
### Scatter plots ### Scatter plots
It would be possible to use `plot()` to create a scatter plot, but It would be possible to use `plot()` to create a scatter plot, but
there is also an alternative: `scatter()` there is also an alternative: `scatter()`
%% Cell type:code id:1ef6ef5a tags: %% Cell type:code id:4d5e359a tags:
``` ```
fig, ax = plt.subplots() fig, ax = plt.subplots()
# setup some sizes for each point (arbitrarily example here) # setup some sizes for each point (arbitrarily example here)
ssize = 100*abs(samp1-samp2) + 10 ssize = 100*abs(samp1-samp2) + 10
ax.scatter(samp1, samp2, s=ssize, alpha=0.5) ax.scatter(samp1, samp2, s=ssize, alpha=0.5)
# now add the y=x line # now add the y=x line
allsamps = np.hstack((samp1,samp2)) allsamps = np.hstack((samp1,samp2))
ax.plot([min(allsamps),max(allsamps)],[min(allsamps),max(allsamps)], color='red', linestyle='--') ax.plot([min(allsamps),max(allsamps)],[min(allsamps),max(allsamps)], color='red', linestyle='--')
plt.xlim(min(allsamps),max(allsamps)) plt.xlim(min(allsamps),max(allsamps))
plt.ylim(min(allsamps),max(allsamps)) plt.ylim(min(allsamps),max(allsamps))
``` ```
%% Cell type:markdown id:69710e7b tags: %% Cell type:markdown id:362901f1 tags:
> Note that in this case we use the first line return to get a handle to > Note that in this case we use the first line return to get a handle to
> the axis, `ax`, and the figure ,`fig`. The axis can be used instead of > the axis, `ax`, and the figure ,`fig`. The axis can be used instead of
> `plt` in most cases, although the `xlim()` and `ylim()` calls can only > `plt` in most cases, although the `xlim()` and `ylim()` calls can only
> be done through `plt`. > be done through `plt`.
> In general, figures and subplots can be created in matplotlib in a > In general, figures and subplots can be created in matplotlib in a
> similar fashion to matlab, but they do not have to be explicitly > similar fashion to matlab, but they do not have to be explicitly
> invoked as you can see from the earlier examples. > invoked as you can see from the earlier examples.
<a class="anchor" id="subplots"></a> <a class="anchor" id="subplots"></a>
### Subplots ### Subplots
These are very similar to matlab: These are very similar to matlab:
%% Cell type:code id:3ffabeb2 tags: %% Cell type:code id:38bd298d tags:
``` ```
plt.subplot(2, 1, 1) plt.subplot(2, 1, 1)
plt.plot(x,cosx, '.-') plt.plot(x,cosx, '.-')
plt.xlim(-np.pi, np.pi) plt.xlim(-np.pi, np.pi)
plt.ylabel('Full sampling') plt.ylabel('Full sampling')
plt.subplot(2, 1, 2) plt.subplot(2, 1, 2)
plt.plot(x[::30], cosx[::30], '.-') plt.plot(x[::30], cosx[::30], '.-')
plt.xlim(-np.pi, np.pi) plt.xlim(-np.pi, np.pi)
plt.ylabel('Subsampled') plt.ylabel('Subsampled')
``` ```
%% Cell type:markdown id:2c285f4e tags: %% Cell type:markdown id:cbc0ba9a tags:
<a class="anchor" id="displaying-images"></a> <a class="anchor" id="displaying-images"></a>
### Displaying images ### Displaying images
The main command for displaying images is `imshow()` The main command for displaying images is `imshow()`
%% Cell type:code id:9bfff4b7 tags: %% Cell type:code id:b9384347 tags:
``` ```
import nibabel as nib import nibabel as nib
import os.path as op import os.path as op
nim = nib.load(op.expandvars('${FSLDIR}/data/standard/MNI152_T1_1mm.nii.gz'), mmap=False) nim = nib.load(op.expandvars('${FSLDIR}/data/standard/MNI152_T1_1mm.nii.gz'), mmap=False)
imdat = nim.get_data().astype(float) imdat = nim.get_fdata()
imslc = imdat[:,:,70] imslc = imdat[:,:,70]
plt.imshow(imslc, cmap=plt.cm.gray) plt.imshow(imslc, cmap=plt.cm.gray)
plt.colorbar() plt.colorbar()
plt.axis('off') plt.axis('off')
``` ```
%% Cell type:markdown id:bd7cd76e tags: %% Cell type:markdown id:e5fcfef5 tags:
Note that matplotlib will use the **voxel data orientation**, and that Note that matplotlib will use the **voxel data orientation**, and that
configuring the plot orientation is **your responsibility**. To rotate a configuring the plot orientation is **your responsibility**. To rotate a
slice, simply transpose the data (`.T`). To invert the data along along an slice, simply transpose the data (`.T`). To invert the data along along an
axis, you don't need to modify the data - simply swap the axis limits around: axis, you don't need to modify the data - simply swap the axis limits around:
%% Cell type:code id:3623ebcc tags: %% Cell type:code id:57a78412 tags:
``` ```
plt.imshow(imslc.T, cmap=plt.cm.gray) plt.imshow(imslc.T, cmap=plt.cm.gray)
plt.xlim(reversed(plt.xlim())) plt.xlim(reversed(plt.xlim()))
plt.ylim(reversed(plt.ylim())) plt.ylim(reversed(plt.ylim()))
plt.colorbar() plt.colorbar()
plt.axis('off') plt.axis('off')
``` ```
%% Cell type:markdown id:a39985aa tags: %% Cell type:markdown id:27bee169 tags:
<a class="anchor" id="3D-plots"></a> <a class="anchor" id="3D-plots"></a>
### 3D plots ### 3D plots
%% Cell type:code id:fc6a81c9 tags: %% Cell type:code id:909cc557 tags:
``` ```
# Taken from https://matplotlib.org/gallery/mplot3d/wire3d.html#sphx-glr-gallery-mplot3d-wire3d-py # Taken from https://matplotlib.org/gallery/mplot3d/wire3d.html#sphx-glr-gallery-mplot3d-wire3d-py
from mpl_toolkits.mplot3d import axes3d from mpl_toolkits.mplot3d import axes3d
fig = plt.figure() fig = plt.figure()
ax = fig.add_subplot(111, projection='3d') ax = fig.add_subplot(111, projection='3d')
# Grab some test data. # Grab some test data.
X, Y, Z = axes3d.get_test_data(0.05) X, Y, Z = axes3d.get_test_data(0.05)
# Plot a basic wireframe. # Plot a basic wireframe.
ax.plot_wireframe(X, Y, Z, rstride=10, cstride=10) ax.plot_wireframe(X, Y, Z, rstride=10, cstride=10)
``` ```
%% Cell type:markdown id:ada68f3e tags: %% Cell type:markdown id:0b35d752 tags:
Surface renderings are many other plots are possible - see 3D examples on Surface renderings are many other plots are possible - see 3D examples on
the [matplotlib webpage](https://matplotlib.org/gallery/index.html#mplot3d-examples-index) the [matplotlib webpage](https://matplotlib.org/gallery/index.html#mplot3d-examples-index)
--- ---
<a class="anchor" id="plotting-in-scripts"></a> <a class="anchor" id="plotting-in-scripts"></a>
## Plotting from standalone scripts ## Plotting from standalone scripts
When running from a standalone script, the same `matplotlib` import is required, When running from a standalone script, the same `matplotlib` import is required,
but the line `%matplotlib <backend>` should *not* be used. but the line `%matplotlib <backend>` should *not* be used.
In a script it is also necessary to _finish_ with `plt.show()` as In a script it is also necessary to _finish_ with `plt.show()` as
otherwise nothing is actually displayed. For example, the above otherwise nothing is actually displayed. For example, the above
examples would setup a plot but the actual graphic would only appear examples would setup a plot but the actual graphic would only appear
after the `plt.show()` command was executed. Furthermore, control is after the `plt.show()` command was executed. Furthermore, control is
not returned to the script immediately as the plot is interactive by default. not returned to the script immediately as the plot is interactive by default.
--- ---
<a class="anchor" id="exercise"></a> <a class="anchor" id="exercise"></a>
## Exercise ## Exercise
Find a different type of plot (e.g., boxplot, violin plot, quiver Find a different type of plot (e.g., boxplot, violin plot, quiver
plot, pie chart, etc.), look up plot, pie chart, etc.), look up
the documentation and then write _your own code that calls this_ to create a plot the documentation and then write _your own code that calls this_ to create a plot
from some data that you create yourself (i.e., don't just blindly copy from some data that you create yourself (i.e., don't just blindly copy
example code from the docs). example code from the docs).
%% Cell type:code id:5e6f0017 tags: %% Cell type:code id:43c7f323 tags:
``` ```
# Make up some data and do the funky plot # Make up some data and do the funky plot
``` ```
%% Cell type:markdown id:b517e96f tags: %% Cell type:markdown id:e4f0aed8 tags:
If you want more plotting goodneess, have a look at the various tutorials available in the `applications/plottings` folder. If you want more plotting goodneess, have a look at the various tutorials available in the `applications/plottings` folder.
......
...@@ -152,7 +152,7 @@ The main command for displaying images is `imshow()` ...@@ -152,7 +152,7 @@ The main command for displaying images is `imshow()`
import nibabel as nib import nibabel as nib
import os.path as op import os.path as op
nim = nib.load(op.expandvars('${FSLDIR}/data/standard/MNI152_T1_1mm.nii.gz'), mmap=False) nim = nib.load(op.expandvars('${FSLDIR}/data/standard/MNI152_T1_1mm.nii.gz'), mmap=False)
imdat = nim.get_data().astype(float) imdat = nim.get_fdata()
imslc = imdat[:,:,70] imslc = imdat[:,:,70]
plt.imshow(imslc, cmap=plt.cm.gray) plt.imshow(imslc, cmap=plt.cm.gray)
plt.colorbar() plt.colorbar()
...@@ -227,4 +227,4 @@ example code from the docs). ...@@ -227,4 +227,4 @@ example code from the docs).
``` ```
If you want more plotting goodneess, have a look at the various tutorials available in the `applications/plottings` folder. If you want more plotting goodneess, have a look at the various tutorials available in the `applications/plottings` folder.
\ No newline at end of file