Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found
Select Git revision
  • instruction_updates
  • master
  • saad-edits
  • sean-migp
  • visualisation_rezvan
5 results

Target

Select target project
No results found
Select Git revision
  • instruction_updates
  • master
  • saad-edits
  • sean-migp
  • visualisation_rezvan
5 results
Show changes

Commits on Source 149

49 additional commits have been omitted to prevent performance issues.
85 files
+ 16003
− 3562
Compare changes
  • Side-by-side
  • Inline

Files

README.md

+108 −52
Original line number Diff line number Diff line 
# 2018 WIN PyTreat

# 2020 WIN PyTreat





This repository contains Jupyter notebooks and data for the 2018 WIN PyTreat.

This repository contains Jupyter notebooks and data for the 2020 WIN PyTreat.

It contains the following:



- The `talks` directory contains some (but not all) of the _Topyc_ talks that
@@ -17,44 +17,28 @@ It contains the following: 
  about the language.





The practicals have been written under the assumption that FSL 5.0.10 is

The practicals have been written under the assumption that FSL 6.0.3 is

installed.





## For attendees





To run these notebooks in the `fslpython` environment, you must first install

jupyter:

These notebooks can be run in the `fslpython` environment using:





```

# If your FSL installation requires administrative privileges to modify, then

# you MUST run these commands as root - don't just prefix each individual

# command with sudo, or you will probably install jupyter into the wrong

# location!

#

# One further complication - once you have become root, $FSLDIR may not be set,

# so either set it as we have done below, or make sure that it is set, before

# proceeding.

sudo su

export FSLDIR=/usr/local/fsl

source $FSLDIR/fslpython/bin/activate fslpython

conda install jupyter

source deactivate

ln -s $FSLDIR/fslpython/envs/fslpython/bin/jupyter $FSLDIR/bin/fsljupyter

git clone https://git.fmrib.ox.ac.uk/fsl/pytreat-practicals-2020.git

cd pytreat-practicals-2020

fslpython -m notebook

```





Then, clone this repository on your local machine, and run

`fsljupyter notebook`:





```

git clone git@git.fmrib.ox.ac.uk:fsl/pytreat-2018-practicals.git

cd pytreat-2018-practicals

fsljupyter notebook

```

A page should open in your web browser - to access the practicals, navigate

into one of the `getting_started` or `advanced_topics` directories, and click

on the `.ipynb` file you are interested in. Some of the talks are also

presented in notebook form - navigate to the talk you are interested in

(within the `talks` directory), and click on the `.ipynb` file to follow

along.





Throughout the week we might make changes to this repository. When this
@@ -63,67 +47,139 @@ following command: 




```

git reset --hard HEAD

git stash save

git pull origin master

git stash pop

```





> Note that this will overwrite any changes that you have made to the

> files in your repository!





Have fun!





## For contributors





The upstream repository can be found at:

The main repository can be found at:



https://git.fmrib.ox.ac.uk/fsl/pytreat-practicals-2020





Updates to the master branch should occur via merge requests. You can choose

to either work on a branch within this repository  (recommended), or on a fork of this

repository (advanced).



### Using a branch within this repository (recommended)



1. Make a local clone of the repository:



    ```

    git clone https://git.fmrib.ox.ac.uk/fsl/pytreat-practicals-2020.git

    ```



2. Create a branch for your work:



    ```

    git checkout -b my_cool_branch origin/master

    ```



3. Make your changes on this branch.



https://git.fmrib.ox.ac.uk/fsl/pytreat-2018-practicals

    ```

    git add <my_new_and_changed_files>

    git commit -m 'super cool updates'

    ```



4. Push your changes to the gitlab repository:



    ```

    git push origin my_cool_branch

    ```



5. In gitlab, submit a merge request from your branch onto the master

   branch.



To contribute to the practicals:

    https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html





### Using a fork of this repository (advanced)



1. Fork the upstream repository on gitlab



2. Make a local clone of your fork:



    ```

    git clone git@git.fmrib.ox.ac.uk:<username>/pytreat-2018-practicals

    git clone https://git.fmrib.ox.ac.uk/<your_username>/pytreat-practicals-2020.git

    ```



3. Add the upstream repository as a remote:



    ```

    git remote add upstream git@git.fmrib.ox.ac.uk:fsl/pytreat-2018-practicals.git

    git remote add upstream https://git.fmrib.ox.ac.uk/fsl/pytreat-practicals-2020.git

    ```



4. Make your changes on your local repository



5. Rebase onto the upstream repository, and push your changes to your fork:

    ```

    git add <my_new_and_changed_files>

    git commit -m 'super cool updates'

    ```



5. Push your changes to your fork:



    ```

    git fetch --all

    git rebase upstream/master

    git push --force origin master

    git push origin master

    ```



6. In gitlab, submit a merge request from your fork back to the upstream

   repository.

    

    https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html





### Updating your local repository



To bring in the changes that other people have contributed to the main

repository into your local repository:



```

git fetch --all



```



Then, do this if you are working on a branch within the main repository:



```

# make sure you are on the correct local branch:

git checkout my_cool_branch

git merge origin/master

```



Or, do this if you are working on a fork of the main repository:

```

git checkout master

git merge upstream/master

```





> Or, if you are comfortable with git, `rebase` is so much cooler:

>

> ```

> git fetch --all

>

> # replace <branch_name> with your local branch name

> git checkout <remote_name>/master

>

> # replace <remote_name> with the main repository name

> git rebase <remote_name>/master

> ```





### Writing your talk as a Jupyter notebook



When you install `jupyter` above, you may also wish to install

[`notedown`](https://github.com/aaren/notedown):

You may wish to install [`notedown`](https://github.com/aaren/notedown):



```

# .

# see instructions above

# .

conda install jupyter

pip install notedown

source deactivate

ln -s $FSLDIR/fslpython/envs/fslpython/bin/jupyter  $FSLDIR/bin/fsljupyter

$FSLDIR/fslpython/bin/conda install -n fslpython -c conda-forge notedown

ln -s $FSLDIR/fslpython/envs/fslpython/bin/notedown $FSLDIR/bin/fslnotedown

```

advanced_topics/02_modules_and_packages.ipynb

+6 −6
Original line number Diff line number Diff line 
%% Cell type:markdown id: tags:



# Modules and packages





Python gives you a lot of flexibility in how you organise your code. If you

want, you can write a Python program just as you would write a Bash script.

You don't _have_ to use functions, classes, modules or packages if you don't

want to, or if the script's task does not require them.

want to, or if the script task does not require them.





But when your code starts to grow beyond what can reasonably be defined in a

single file, you will (hopefully) want to start arranging it in a more

understandable manner.





For this practical we have prepared a handful of example files - you can find

them alongside this notebook file, in a directory called

`02_modules_and_packages/`.





## Contents



* [What is a module?](#what-is-a-module)

* [Importing modules](#importing-modules)

 * [Importing specific items from a module](#importing-specific-items-from-a-module)

 * [Importing everything from a module](#importing-everything-from-a-module)

 * [Module aliases](#module-aliases)

 * [What happens when I import a module?](#what-happens-when-i-import-a-module)

 * [How can I make my own modules importable?](#how-can-i-make-my-own-modules-importable)

* [Modules versus scripts](#modules-versus-scripts)

* [What is a package?](#what-is-a-package)

 * [`__init__.py`](#init-py)

* [Useful references](#useful-references)



%% Cell type:code id: tags:



``` 

import os

os.chdir('02_modules_and_packages')

```



%% Cell type:markdown id: tags:



<a class="anchor" id="what-is-a-module"></a>

## What is a module?





Any file ending with `.py` is considered to be a module in Python. Take a look

at `02_modules_and_packages/numfuncs.py` - either open it in your editor, or

run this code block:



%% Cell type:code id: tags:



``` 

with open('numfuncs.py', 'rt') as f:

    for line in f:

        print(line.rstrip())

```



%% Cell type:markdown id: tags:



This is a perfectly valid Python module, although not a particularly useful

one. It contains an attribute called `PI`, and a function `add`.





<a class="anchor" id="importing-modules"></a>

## Importing modules





Before we can use our module, we must `import` it. Importing a module in

Python will make its contents available in the local scope.  We can import the

contents of `numfuncs` like so:



%% Cell type:code id: tags:



``` 

import numfuncs

```



%% Cell type:markdown id: tags:



This imports `numfuncs` into the local scope - everything defined in the

`numfuncs` module can be accessed by prefixing it with `numfuncs.`:



%% Cell type:code id: tags:



``` 

print('PI:', numfuncs.PI)

print(numfuncs.add(1, 50))

```



%% Cell type:markdown id: tags:



There are a couple of other ways to import items from a module...





<a class="anchor" id="importing-specific-items-from-a-module"></a>

### Importing specific items from a module





If you only want to use one, or a few items from a module, you can import just

those items - a reference to just those items will be created in the local

scope:



%% Cell type:code id: tags:



``` 

from numfuncs import add

print(add(1, 3))

```



%% Cell type:markdown id: tags:



<a class="anchor" id="importing-everything-from-a-module"></a>

### Importing everything from a module





It is possible to import _everything_ that is defined in a module like so:



%% Cell type:code id: tags:



``` 

from numfuncs import *

print('PI: ', PI)

print(add(1, 5))

```



%% Cell type:markdown id: tags:



__PLEASE DON'T DO THIS!__ Because every time you do, somewhere in the world, a

software developer will will spontaneously stub his/her toe, and start crying.

software developer will spontaneously stub his/her toe, and start crying.

Using this approach can make more complicated programs very difficult to read,

because it is not possible to determine the origin of the functions and

attributes that are being used.





And naming collisions are inevitable when importing multiple modules in this

way, making it very difficult for somebody else to figure out what your code

is doing:



%% Cell type:code id: tags:



``` 

from numfuncs import *

from strfuncs import *



print(add(1, 5))

```



%% Cell type:markdown id: tags:



Instead, it is better to give modules a name when you import them.  While this

requires you to type more code, the benefits of doing this far outweigh the

hassle of typing a few extra characters - it becomes much easier to read and

trace through code when the functions you use are accessed through a namespace

for each module:



%% Cell type:code id: tags:



``` 

import numfuncs

import strfuncs

print('number add: ', numfuncs.add(1, 2))

print('string add: ', strfuncs.add(1, 2))

```



%% Cell type:markdown id: tags:



<a class="anchor" id="module-aliases"></a>

### Module aliases





And Python allows you to define an _alias_ for a module when you import it,

so you don't necessarily need to type out the full module name each time

you want to access something inside:



%% Cell type:code id: tags:



``` 

import numfuncs as nf

import strfuncs as sf

print('number add: ', nf.add(1, 2))

print('string add: ', sf.add(1, 2))

```



%% Cell type:markdown id: tags:



You have already seen this in the earlier practicals - here are a few

aliases which have become a de-facto standard for commonly used Python

modules:



%% Cell type:code id: tags:



``` 

import os.path           as op

import numpy             as np

import nibabel           as nib

import matplotlib        as mpl

import matplotlib.pyplot as plt

```



%% Cell type:markdown id: tags:



<a class="anchor" id="what-happens-when-i-import-a-module"></a>

### What happens when I import a module?





When you `import` a module, the contents of the module file are literally

executed by the Python runtime, exactly the same as if you had typed its

contents into `ipython`. Any attributes, functions, or classes which are

defined in the module will be bundled up into an object that represents the

module, and through which you can access the module's contents.





When we typed `import numfuncs` in the examples above, the following events

occurred:





1. Python created a `module` object to represent the module.



2. The `numfuncs.py` file was read and executed, and all of the items defined

   inside `numfuncs.py` (i.e. the `PI` attribute and the `add` function) were

   added to the `module` object.



3. A local variable called `numfuncs`, pointing to the `module` object,

   was added to the local scope.





Because module files are literally executed on import, any statements in the

module file which are not encapsulated inside a class or function will be

executed.  As an example, take a look at the file `sideeffects.py`. Let's

import it and see what happens:



%% Cell type:code id: tags:



``` 

import sideeffects

```



%% Cell type:markdown id: tags:



Ok, hopefully that wasn't too much of a surprise. Something which may be less

intuitive, however, is that a module's contents will only be executed on the

_first_ time that it is imported. After the first import, Python caches the

module's contents (all loaded modules are accessible through

[`sys.modules`](https://docs.python.org/3.5/library/sys.html#sys.modules)). On

subsequent imports, the cached version of the module is returned:



%% Cell type:code id: tags:



``` 

import sideeffects

```



%% Cell type:markdown id: tags:



<a class="anchor" id="how-can-i-make-my-own-modules-importable"></a>

### How can I make my own modules importable?





When you `import` a module, Python searches for it in the following locations,

in the following order:





1. Built-in modules (e.g. `os`, `sys`, etc.).

2. In the current directory or, if a script has been executed, in the directory

   containing that script.

3. In directories listed in the `$PYTHONPATH` environment variable.

4. In installed third-party libraries (e.g. `numpy`).





If you are experimenting or developing your program, the quickest and easiest

way to make your module(s) importable is to add their containing directory to

the `PYTHONPATH`. But if you are developing a larger piece of software, you

should probably organise your modules into _packages_, which are [described

should probably organise your modules into *packages*, which are [described

below](#what-is-a-package).





<a class="anchor" id="modules-versus-scripts"></a>

## Modules versus scripts





You now know that Python treats all files ending in `.py` as importable

modules. But all files ending in `.py` can also be treated as scripts. In

fact, there no difference between a _module_ and a _script_ - any `.py` file

can be executed as a script, or imported as a module, or both.





Have a look at the file `02_modules_and_packages/module_and_script.py`:



%% Cell type:code id: tags:



``` 

with open('module_and_script.py', 'rt') as f:

    for line in f:

        print(line.rstrip())

```



%% Cell type:markdown id: tags:



This file contains two functions `mul` and `main`.  The

`if __name__ == '__main__':` clause at the bottom is a standard trick in Python

that allows you to add code to a file that is _only executed when the module is

called as a script_. Try it in a terminal now:





> `python 02_modules_and_packages/module_and_script.py`





But if we `import` this module from another file, or from an interactive

session, the code within the `if __name__ == '__main__':` clause will not be

executed, and we can access its functions just like any other module that we

import.



%% Cell type:code id: tags:



``` 

import module_and_script as mas



a = 1.5

b = 3



print('mul({}, {}): {}'.format(a, b, mas.mul(a, b)))

print('calling main...')

mas.main([str(a), str(b)])

```



%% Cell type:markdown id: tags:



<a class="anchor" id="what-is-a-package"></a>

## What is a package?





You now know how to split your Python code up into separate files

(a.k.a. _modules_). When your code grows beyond a handful of files, you may

(a.k.a. *modules*). When your code grows beyond a handful of files, you may

wish for more fine-grained control over the namespaces in which your modules

live. Python has another feature which allows you to organise your modules

into _packages_.

into *packages*.





A package in Python is simply a directory which:





* Contains a special file called `__init__.py`

* May contain one or more module files (any other files ending in `*.py`)

* May contain other package directories.





For example, the [FSLeyes](https://git.fmrib.ox.ac.uk/fsl/fsleyes/fsleyes)

code is organised into packages and sub-packages as follows (abridged):





> ```

> fsleyes/

>     __init__.py

>     main.py

>     frame.py

>     views/

>         __init__.py

>         orthopanel.py

>         lightboxpanel.py

>     controls/

>         __init__.py

>         locationpanel.py

>         overlaylistpanel.py

> ```





Within a package structure, we will typically still import modules directly,

via their full path within the package:



%% Cell type:code id: tags:



``` 

import fsleyes.main as fmain

fmain.fsleyes_main()

```



%% Cell type:markdown id: tags:



<a class="anchor" id="init-py"></a>

### `__init__.py`





Every Python package must have an `__init__.py` file. In many cases, this will

actually be an empty file, and you don't need to worry about it any more, apart

from knowing that it is needed. But you can use `__init__.py` to perform some

package-specific initialisation, and/or to customise the package's namespace.





As an example, take a look the `02_modules_and_packages/fsleyes/__init__.py`

file in our mock FSLeyes package. We have imported the `fsleyes_main` function

from the `fsleyes.main` module, making it available at the package level. So

instead of importing the `fsleyes.main` module, we could instead just import

the `fsleyes` package:



%% Cell type:code id: tags:



``` 

import fsleyes

fsleyes.fsleyes_main()

```



%% Cell type:markdown id: tags:



<a class="anchor" id="useful-references"></a>

## Useful references



* [Modules and packages in Python](https://docs.python.org/3.5/tutorial/modules.html)

* [Modules and packages in Python](https://docs.python.org/3/tutorial/modules.html)

* [Using `__init__.py`](http://mikegrouchy.com/blog/2012/05/be-pythonic-__init__py.html)

advanced_topics/02_modules_and_packages.md

+7 −7
Original line number Diff line number Diff line
@@ -4,7 +4,7 @@ 
Python gives you a lot of flexibility in how you organise your code. If you

want, you can write a Python program just as you would write a Bash script.

You don't _have_ to use functions, classes, modules or packages if you don't

want to, or if the script's task does not require them.

want to, or if the script task does not require them.





But when your code starts to grow beyond what can reasonably be defined in a
@@ -113,7 +113,7 @@ print(add(1, 5)) 




__PLEASE DON'T DO THIS!__ Because every time you do, somewhere in the world, a

software developer will will spontaneously stub his/her toe, and start crying.

software developer will spontaneously stub his/her toe, and start crying.

Using this approach can make more complicated programs very difficult to read,

because it is not possible to determine the origin of the functions and

attributes that are being used.
@@ -242,7 +242,7 @@ in the following order: 
If you are experimenting or developing your program, the quickest and easiest

way to make your module(s) importable is to add their containing directory to

the `PYTHONPATH`. But if you are developing a larger piece of software, you

should probably organise your modules into _packages_, which are [described

should probably organise your modules into *packages*, which are [described

below](#what-is-a-package).
@@ -298,10 +298,10 @@ mas.main([str(a), str(b)]) 




You now know how to split your Python code up into separate files

(a.k.a. _modules_). When your code grows beyond a handful of files, you may

(a.k.a. *modules*). When your code grows beyond a handful of files, you may

wish for more fine-grained control over the namespaces in which your modules

live. Python has another feature which allows you to organise your modules

into _packages_.

into *packages*.





A package in Python is simply a directory which:
@@ -367,5 +367,5 @@ fsleyes.fsleyes_main() 
<a class="anchor" id="useful-references"></a>

## Useful references



* [Modules and packages in Python](https://docs.python.org/3.5/tutorial/modules.html)

* [Modules and packages in Python](https://docs.python.org/3/tutorial/modules.html)

* [Using `__init__.py`](http://mikegrouchy.com/blog/2012/05/be-pythonic-__init__py.html)

advanced_topics/03_object_oriented_programming.ipynb

+157 −67
Original line number Diff line number Diff line 
%% Cell type:markdown id: tags:



# Object-oriented programming in Python





By now you might have realised that __everything__ in Python is an

object. Strings are objects, numbers are objects, functions are objects,

modules are objects - __everything__ is an object!





But this does not mean that you have to use Python in an object-oriented

fashion. You can stick with functions and statements, and get quite a lot

done. But some problems are just easier to solve, and to reason about, when

you use an object-oriented approach.





* [Objects versus classes](#objects-versus-classes)

* [Defining a class](#defining-a-class)

* [Object creation - the `__init__` method](#object-creation-the-init-method)

 * [Our method is called `__init__`, but we didn't actually call the `__init__` method!](#our-method-is-called-init)

 * [We didn't specify the `self` argument - what gives?!?](#we-didnt-specify-the-self-argument)

* [Attributes](#attributes)

* [Methods](#methods)

* [Method chaining](#method-chaining)

* [Protecting attribute access](#protecting-attribute-access)

 * [A better way - properties](#a-better-way-properties])

* [Inheritance](#inheritance)

 * [The basics](#the-basics)

 * [Code re-use and problem decomposition](#code-re-use-and-problem-decomposition)

 * [Polymorphism](#polymorphism)

 * [Multiple inheritance](#multiple-inheritance)

* [Class attributes and methods](#class-attributes-and-methods)

 * [Class attributes](#class-attributes)

 * [Class methods](#class-methods)

* [Appendix: The `object` base-class](#appendix-the-object-base-class)

* [Appendix: `__init__` versus `__new__`](#appendix-init-versus-new)

* [Appendix: Monkey-patching](#appendix-monkey-patching)

* [Appendix: Method overloading](#appendix-method-overloading)

* [Useful references](#useful-references)





<a class="anchor" id="objects-versus-classes"></a>

## Objects versus classes





If you are versed in C++, Java, C#, or some other object-oriented language,

then this should all hopefully sound familiar, and you can skip to the next

section.





If you have not done any object-oriented programming before, your first step

is to understand the difference between _objects_ (also known as

_instances_) and _classes_ (also known as _types_).

is to understand the difference between *objects* (also known as

*instances*) and *classes* (also known as *types*).





If you have some experience in C, then you can start off by thinking of a

class as like a `struct` definition - a `struct` is a specification for the

layout of a chunk of memory. For example, here is a typical struct definition:



> ```

> /**

>  * Struct representing a stack.

>  */

> typedef struct __stack {

>   uint8_t capacity; /**< the maximum capacity of this stack */

>   uint8_t size;     /**< the current size of this stack     */

>   void  **top;      /**< pointer to the top of this stack   */

> } stack_t;

> ```





Now, an _object_ is not a definition, but rather a thing which resides in

memory. An object can have _attributes_ (pieces of information), and _methods_

Now, an *object* is not a definition, but rather a thing which resides in

memory. An object can have *attributes* (pieces of information), and *methods*

(functions associated with the object). You can pass objects around your code,

manipulate their attributes, and call their methods.





Returning to our C metaphor, you can think of an object as like an

instantiation of a struct:





> ```

> stack_t stack;

> stack.capacity = 10;

> ```





One of the major differences between a `struct` in C, and a `class` in Python

and other object oriented languages, is that you can't (easily) add functions

to a `struct` - it is just a chunk of memory. Whereas in Python, you can add

functions to your class definition, which will then be added as methods when

you create an object from that class.





Of course there are many more differences between C structs and classes (most

notably [inheritance](todo), [polymorphism](todo), and [access

protection](todo)). But if you can understand the difference between a

_definition_ of a C struct, and an _instantiation_ of that struct, then you

are most of the way towards understanding the difference between a _class_,

and an _object_.

*definition* of a C struct, and an *instantiation* of that struct, then you

are most of the way towards understanding the difference between a *class*,

and an *object*.





> But just to confuse you, remember that in Python, __everything__ is an

> But just to confuse you, remember that in Python, **everything** is an

> object - even classes!





<a class="anchor" id="defining-a-class"></a>

## Defining a class





Defining a class in Python is simple. Let's take on a small project, by

developing a class which can be used in place of the `fslmaths` shell command.



%% Cell type:code id: tags:



``` 

class FSLMaths(object):

    pass

```



%% Cell type:markdown id: tags:



In this statement, we defined a new class called `FSLMaths`, which inherits

from the built-in `object` base-class (see [below](inheritance) for more

details on inheritance).





Now that we have defined our class, we can create objects - instances of that

class - by calling the class itself, as if it were a function:



%% Cell type:code id: tags:



``` 

fm1 = FSLMaths()

fm2 = FSLMaths()

print(fm1)

print(fm2)

```



%% Cell type:markdown id: tags:



Although these objects are not of much use at this stage. Let's do some more

work.





<a class="anchor" id="object-creation-the-init-method"></a>

## Object creation - the `__init__` method





The first thing that our `fslmaths` replacement will need is an input image.

It makes sense to pass this in when we create an `FSLMaths` object:



%% Cell type:code id: tags:



``` 

class FSLMaths(object):

    def __init__(self, inimg):

        self.img = inimg

```



%% Cell type:markdown id: tags:



Here we have added a _method_ called `__init__` to our class (remember that a

_method_ is just a function which is defined in a class, and which can be

called on instances of that class).  This method expects two arguments -

`self`, and `inimg`. So now, when we create an instance of the `FSLMaths`

class, we will need to provide an input image:



%% Cell type:code id: tags:



``` 

import nibabel as nib

import os.path as op



fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')

inimg = nib.load(fpath)

fm    = FSLMaths(inimg)

```



%% Cell type:markdown id: tags:



There are a couple of things to note here...





<a class="anchor" id="our-method-is-called-init"></a>

### Our method is called `__init__`, but we didn't actually call the `__init__` method!





`__init__` is a special method in Python - it is called when an instance of a

class is created. And recall that we can create an instance of a class by

calling the class in the same way that we call a function.





There are a number of "special" methods that you can add to a class in Python

to customise various aspects of how instances of the class behave.  One of the

first ones you may come across is the `__str__` method, which defines how an

object should be printed (more specifically, how an object gets converted into

a string). For example, we could add a `__str__` method to our `FSLMaths`

class like so:



%% Cell type:code id: tags:



``` 

class FSLMaths(object):



    def __init__(self, inimg):

        self.img = inimg



    def __str__(self):

        return 'FSLMaths({})'.format(self.img.get_filename())



fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')

inimg = nib.load(fpath)

fm    = FSLMaths(inimg)



print(fm)

```



%% Cell type:markdown id: tags:



Refer to the [official

docs](https://docs.python.org/3.5/reference/datamodel.html#special-method-names)

docs](https://docs.python.org/3/reference/datamodel.html#special-method-names)

for details on all of the special methods that can be defined in a class. And

take a look at the appendix for some more details on [how Python objects get

created](appendix-init-versus-new).





<a class="anchor" id="we-didnt-specify-the-self-argument"></a>

### We didn't specify the `self` argument - what gives?!?





The `self` argument is a special argument for methods in Python. If you are

coming from C++, Java, C# or similar, `self` in Python is equivalent to `this`

in those languages.





In a method, the `self` argument is a reference to the object that the method

was called on. So in this line of code:



%% Cell type:code id: tags:



``` 

fm = FSLMaths(inimg)

```



%% Cell type:markdown id: tags:



the `self` argument in `__init__` will be a reference to the `FSLMaths` object

that has been created (and is then assigned to the `fm` variable, after the

`__init__` method has finished).





But note that you __do not__ need to explicitly provide the `self` argument

when you call a method on an object, or when you create a new object. The

Python runtime will take care of passing the instance to its method, as the

first argument to the method.





But when you are writing a class, you __do__ need to explicitly list `self` as

the first argument to all of the methods of the class.





<a class="anchor" id="attributes"></a>

## Attributes





In Python, the term __attribute__ is used to refer to a piece of information

that is associated with an object. An attribute is generally a reference to

another object (which might be a string, a number, or a list, or some other

more complicated object).





Remember that we modified our `FSLMaths` class so that it is passed an input

image on creation:



%% Cell type:code id: tags:



``` 

class FSLMaths(object):

    def __init__(self, inimg):

        self.img = inimg



fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')

fm    = FSLMaths(nib.load(fpath))

```



%% Cell type:markdown id: tags:



Take a look at what is going on in the `__init__` method - we take the `inimg`

argument, and create a reference to it called `self.img`. We have added an

_attribute_ to the `FSLMaths` instance, called `img`, and we can access that

attribute like so:



%% Cell type:code id: tags:



``` 

print('Input for our FSLMaths instance: {}'.format(fm.img.get_filename()))

```



%% Cell type:markdown id: tags:



And that concludes the section on adding attributes to Python objects.





Just kidding. But it really is that simple. This is one aspect of Python which

might be quite jarring to you if you are coming from a language with more

rigid semantics, such as C++ or Java. In those languages, you must pre-specify

all of the attributes and methods that are a part of a class. But Python is

much more flexible - you can simply add attributes to an object after it has

been created.  In fact, you can even do this outside of the class

definition<sup>1</sup>:



%% Cell type:code id: tags:



``` 

fm = FSLMaths(inimg)

fm.another_attribute = 'Haha'

print(fm.another_attribute)

```



%% Cell type:markdown id: tags:



__But ...__ while attributes can be added to a Python object at any time, it is

good practice (and makes for more readable and maintainable code) to add all

of an object's attributes within the `__init__` method.





> <sup>1</sup>This not possible with many of the built-in types, such as

> `list` and `dict` objects, nor with types that are defined in Python

> extensions (Python modules that are written in C).





<a class="anchor" id="methods"></a>

## Methods





We've been dilly-dallying on this little `FSLMaths` project for a while now,

but our class still can't actually do anything. Let's start adding some

functionality:



%% Cell type:code id: tags:



``` 

class FSLMaths(object):



    def __init__(self, inimg):

        self.img        = inimg

        self.operations = []



    def add(self, value):

        self.operations.append(('add', value))



    def mul(self, value):

        self.operations.append(('mul', value))



    def div(self, value):

        self.operations.append(('div', value))

```



%% Cell type:markdown id: tags:



Woah woah, [slow down egg-head!](https://www.youtube.com/watch?v=yz-TemWooa4)

We've modified `__init__` so that a second attribute called `operations` is

added to our object - this `operations` attribute is simply a list.





Then, we added a handful of methods - `add`, `mul`, and `div` - which each

append a tuple to that `operations` list.





> Note that, just like in the `__init__` method, the first argument that will

> be passed to these methods is `self` - a reference to the object that the

> method has been called on.





The idea behind this design is that our `FSLMaths` class will not actually do

anything when we call the `add`, `mul` or `div` methods. Instead, it will

"stage" each operation, and then perform them all in one go. So let's add

another method, `run`, which actually does the work:

*stage* each operation, and then perform them all in one go at a later point

in time. So let's add another method, `run`, which actually does the work:



%% Cell type:code id: tags:



``` 

import numpy   as np

import nibabel as nib



class FSLMaths(object):



    def __init__(self, inimg):

        self.img        = inimg

        self.operations = []



    def add(self, value):

        self.operations.append(('add', value))



    def mul(self, value):

        self.operations.append(('mul', value))



    def div(self, value):

        self.operations.append(('div', value))



    def run(self, output=None):



        data = np.array(self.img.get_data())



        for oper, value in self.operations:



            # Value could be an image.

            # If not, we assume that

            # it is a scalar/numpy array.

            if isinstance(value, nib.nifti1.Nifti1Image):

                value = value.get_data()





            if oper == 'add':

                data = data + value

            elif oper == 'mul':

                data = data * value

            elif oper == 'div':

                data = data / value



        # turn final output into a nifti,

        # and save it to disk if an

        # 'output' has been specified.

        outimg = nib.nifti1.Nifti1Image(data, inimg.affine)



        if output is not None:

            nib.save(outimg, output)



        return outimg

```



%% Cell type:markdown id: tags:



We now have a useable (but not very useful) `FSLMaths` class!



%% Cell type:code id: tags:



``` 

fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')

fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz')

inimg = nib.load(fpath)

mask  = nib.load(fmask)

fm    = FSLMaths(inimg)



fm.mul(mask)

fm.add(-10)



outimg = fm.run()



norigvox = (inimg .get_data() > 0).sum()

nmaskvox = (outimg.get_data() > 0).sum()



print('Number of voxels >0 in original image: {}'.format(norigvox))

print('Number of voxels >0 in masked image:   {}'.format(nmaskvox))

```



%% Cell type:markdown id: tags:



<a class="anchor" id="method-chaining"></a>

## Method chaining





A neat trick, which is used by all the cool kids these days, is to write

classes that allow *method chaining* - writing one line of code which

calls more than one method on an object, e.g.:



> ```

> fm = FSLMaths(img)

> result = fm.add(1).mul(10).run()

> ```



Adding this feature to our budding `FSLMaths` class is easy - all we have

to do is return `self` from each method:



%% Cell type:code id: tags:



``` 

import numpy   as np

import nibabel as nib



class FSLMaths(object):



    def __init__(self, inimg):

        self.img        = inimg

        self.operations = []



    def add(self, value):

        self.operations.append(('add', value))

        return self



    def mul(self, value):

        self.operations.append(('mul', value))

        return self



    def div(self, value):

        self.operations.append(('div', value))

        return self



    def run(self, output=None):



        data = np.array(self.img.get_data())



        for oper, value in self.operations:



            # Value could be an image.

            # If not, we assume that

            # it is a scalar/numpy array.

            if isinstance(value, nib.nifti1.Nifti1Image):

                value = value.get_data()



            if oper == 'add':

                data = data + value

            elif oper == 'mul':

                data = data * value

            elif oper == 'div':

                data = data / value



        # turn final output into a nifti,

        # and save it to disk if an

        # 'output' has been specified.

        outimg = nib.nifti1.Nifti1Image(data, inimg.affine)



        if output is not None:

            nib.save(outimg, output)



        return outimg

```



%% Cell type:markdown id: tags:



Now we can chain all of our method calls, and even the creation of our

`FSLMaths` object, into a single line:



%% Cell type:code id: tags:



``` 

fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')

fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz')

inimg = nib.load(fpath)

mask  = nib.load(fmask)



outimg = FSLMaths(inimg).mul(mask).add(-10).run()



norigvox = (inimg .get_data() > 0).sum()

nmaskvox = (outimg.get_data() > 0).sum()



print('Number of voxels >0 in original image: {}'.format(norigvox))

print('Number of voxels >0 in masked image:   {}'.format(nmaskvox))

```



%% Cell type:markdown id: tags:



> In fact, this is precisely how the

> [`fsl.wrappers.fslmaths`](https://users.fmrib.ox.ac.uk/~paulmc/fsleyes/fslpy/latest/fsl.wrappers.fslmaths.html)

> function works.





<a class="anchor" id="protecting-attribute-access"></a>

## Protecting attribute access





In our `FSLMaths` class, the input image was added as an attribute called

`img` to `FSLMaths` objects. We saw that it is easy to read the attributes

of an object - if we have a `FSLMaths` instance called `fm`, we can read its

input image via `fm.img`.





But it is just as easy to write the attributes of an object. What's to stop

some sloppy research assistant from overwriting our `img` attribute?



%% Cell type:code id: tags:



``` 

inimg = nib.load(op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz'))

fm = FSLMaths(inimg)

fm.img = None

fm.run()

```



%% Cell type:markdown id: tags:



Well, the scary answer is ... there is __nothing__ stopping you from doing

whatever you want to a Python object. You can add, remove, and modify

attributes at will. You can even replace the methods of an existing object if

you like:



%% Cell type:code id: tags:



``` 

fm = FSLMaths(inimg)



def myadd(value):

    print('Oh no, I\'m not going to add {} to '

          'your image. Go away!'.format(value))



fm.add = myadd

fm.add(123)



fm.mul = None

fm.mul(123)

```



%% Cell type:markdown id: tags:



But you really shouldn't get into the habit of doing devious things like

this. Think of the poor souls who inherit your code years after you have left

the lab - if you go around overwriting all of the methods and attributes of

your objects, they are not going to have a hope in hell of understanding what

your code is actually doing, and they are not going to like you very

much. Take a look at the appendix for a [brief discussion on this

topic](appendix-monkey-patching).





Python tends to assume that programmers are "responsible adults", and hence

doesn't do much in the way of restricting access to the attributes or methods

of an object. This is in contrast to languages like C++ and Java, where the

notion of a private attribute or method is strictly enforced by the language.





However, there are a couple of conventions in Python that are [universally

adhered

to](https://docs.python.org/3.5/tutorial/classes.html#private-variables):

However, there are a couple of conventions in Python that are

[universally adhered to](https://docs.python.org/3/tutorial/classes.html#private-variables):



* Class-level attributes and methods, and module-level attributes, functions,

  and classes, which begin with a single underscore (`_`), should be

  considered __protected__ - they are intended for internal use only, and

  should not be considered part of the public API of a class or module.  This

  is not enforced by the language in any way<sup>2</sup> - remember, we are

  all responsible adults here!



* Class-level attributes and methods which begin with a double-underscore

  (`__`) should be considered __private__. Python provides a weak form of

  enforcement for this rule - any attribute or method with such a name will

  actually be _renamed_ (in a standardised manner) at runtime, so that it is

  not accessible through its original name (it is still accessible via its

  [mangled

  name](https://docs.python.org/3.5/tutorial/classes.html#private-variables)

  [mangled name](https://docs.python.org/3/tutorial/classes.html#private-variables)

  though).





> <sup>2</sup> With the exception that module-level fields which begin with a

> single underscore will not be imported into the local scope via the

> `from [module] import *` techinque.

> `from [module] import *` technique.





So with all of this in mind, we can adjust our `FSLMaths` class to discourage

our sloppy research assistant from overwriting the `img` attribute:



%% Cell type:code id: tags:



``` 

# remainder of definition omitted for brevity

class FSLMaths(object):

    def __init__(self, inimg):

        self.__img        = inimg

        self.__operations = []

```



%% Cell type:markdown id: tags:



But now we have lost the ability to read our `__img` attribute:



%% Cell type:code id: tags:



``` 

inimg = nib.load(op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz'))

fm = FSLMaths(inimg)

print(fm.__img)

```



%% Cell type:markdown id: tags:



<a class="anchor" id="a-better-way-properties"></a>

### A better way - properties





Python has a feature called

[`properties`](https://docs.python.org/3.5/library/functions.html#property),

[`properties`](https://docs.python.org/3/library/functions.html#property),

which is a nice way of controlling access to the attributes of an object. We

can use properties by defining a "getter" method which can be used to access

our attributes, and "decorating" them with the `@property` decorator (we will

cover decorators in a later practical).



%% Cell type:code id: tags:



``` 

class FSLMaths(object):

    def __init__(self, inimg):

        self.__img        = inimg

        self.__operations = []



    @property

    def img(self):

        return self.__img

```



%% Cell type:markdown id: tags:



So we are still storing our input image as a private attribute, but now we

have made it available in a read-only manner via the public `img` property:



%% Cell type:code id: tags:



``` 

fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')

inimg = nib.load(fpath)

fm    = FSLMaths(inimg)



print(fm.img.get_filename())

```



%% Cell type:markdown id: tags:



Note that, even though we have defined `img` as a method, we can access it

like an attribute - this is due to the magic behind the `@property` decorator.





We can also define "setter" methods for a property. For example, we might wish

to add the ability for a user of our `FSLMaths` class to change the input

image after creation.



%% Cell type:code id: tags:



``` 

class FSLMaths(object):

    def __init__(self, inimg):

        self.__img        = None

        self.__operations = []

        self.img          = inimg



    @property

    def img(self):

        return self.__img



    @img.setter

    def img(self, value):

        if not isinstance(value, nib.nifti1.Nifti1Image):

            raise ValueError('value must be a NIFTI image!')

        self.__img = value

```



%% Cell type:markdown id: tags:



> Note that we used the `img` setter method within `__init__` to validate the

> initial `inimg` that was passed in during creation.





Property setters are a nice way to add validation logic for when an attribute

is assigned a value. In this example, an error will be raised if the new input

is not a NIFTI image.



%% Cell type:code id: tags:



``` 

fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')

inimg = nib.load(fpath)

fm    = FSLMaths(inimg)



print('Input:     ', fm.img.get_filename())



# let's change the input

# to a different image

fpath2 = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain.nii.gz')

inimg2 = nib.load(fpath2)

fm.img = inimg2



print('New input: ', fm.img.get_filename())



print('This is going to explode')

fm.img = 'abcde'

```



%% Cell type:markdown id: tags:



<a class="anchor" id="inheritance"></a>

## Inheritance





One of the major advantages of an object-oriented programming approach is

_inheritance_ - the ability to define hierarchical relationships between

classes and instances.





<a class="anchor" id="the-basics"></a>

### The basics





My local veterinary surgery runs some Python code which looks like the

following, to assist the nurses in identifying an animal when it arrives at

the surgery:



%% Cell type:code id: tags:



``` 

class Animal(object):

    def noiseMade(self):

        raise NotImplementedError('This method must be '

                                  'implemented by sub-classes')



class Dog(Animal):

    def noiseMade(self):

        return 'Woof'



class TalkingDog(Dog):

    def noiseMade(self):

        return 'Hi Homer, find your soulmate!'



class Cat(Animal):

    def noiseMade(self):

        return 'Meow'



class Labrador(Dog):

    pass



class Chihuahua(Dog):

    def noiseMade(self):

        return 'Yap yap yap'

```



%% Cell type:markdown id: tags:



Hopefully this example doesn't need much in the way of explanation - this

collection of classes captures a hierarchical relationship which exists in the

real world (and also captures the inherently annoying nature of

collection of classes represents a hierarchical relationship which exists in

the real world (and also represents the inherently annoying nature of

chihuahuas). For example, in the real world, all dogs are animals, but not all

animals are dogs.  Therefore in our model, the `Dog` class has specified

`Animal` as its base class. We say that the `Dog` class _extends_, _derives

from_, or _inherits from_, the `Animal` class, and that all `Dog` instances

`Animal` as its base class. We say that the `Dog` class *extends*, *derives

from*, or *inherits from*, the `Animal` class, and that all `Dog` instances

are also `Animal` instances (but not vice-versa).





What does that `noiseMade` method do?  There is a `noiseMade` method defined

on the `Animal` class, but it has been re-implemented, or _overridden_ in the

on the `Animal` class, but it has been re-implemented, or *overridden* in the

`Dog`,

[`TalkingDog`](https://twitter.com/simpsonsqotd/status/427941665836630016?lang=en),

`Cat`, and `Chihuahua` classes (but not on the `Labrador` class).  We can call

the `noiseMade` method on any `Animal` instance, but the specific behaviour

that we get is dependent on the specific type of animal.



%% Cell type:code id: tags:



``` 

d  = Dog()

l  = Labrador()

c  = Cat()

ch = Chihuahua()



print('Noise made by dogs:       {}'.format(d .noiseMade()))

print('Noise made by labradors:  {}'.format(l .noiseMade()))

print('Noise made by cats:       {}'.format(c .noiseMade()))

print('Noise made by chihuahuas: {}'.format(ch.noiseMade()))

```



%% Cell type:markdown id: tags:



Note that calling the `noiseMade` method on a `Labrador` instance resulted in

the `Dog.noiseMade` implementation being called.





<a class="anchor" id="code-re-use-and-problem-decomposition"></a>

### Code re-use and problem decomposition





Inheritance allows us to split a problem into smaller problems, and to re-use

code.  Let's demonstrate this with a more involved (and even more contrived)

example.  Imagine that a former colleague had written a class called

`Operator`:



%% Cell type:code id: tags:



``` 

class Operator(object):



    def __init__(self):

        super().__init__() # this line will be explained later

        self.__operations = []

        self.__opFuncs    = {}



    @property

    def operations(self):

        return list(self.__operations)



    @property

    def functions(self):

        return dict(self.__opFuncs)



    def addFunction(self, name, func):

        self.__opFuncs[name] = func



    def do(self, name, *values):

        self.__operations.append((name, values))



    def preprocess(self, value):

        return value



    def run(self, input):

        data = self.preprocess(input)

        for oper, vals in self.__operations:

            func = self.__opFuncs[oper]

            vals = [self.preprocess(v) for v in vals]

            data = func(data, *vals)

        return data

```



%% Cell type:markdown id: tags:



This `Operator` class provides an interface and logic to execute a chain of

operations - an operation is some function which accepts one or more inputs,

and produce one output.





But it stops short of defining any operations. Instead, we can create another

class - a sub-class - which derives from the `Operator` class. This sub-class

will define the operations that will ultimately be executed by the `Operator`

class. All that the `Operator` class does is:



- Allow functions to be registered with the `addFunction` method - all

  registered functions can be used via the `do` method.



- Stage an operation (using a registered function) via the `do` method. Note

  that `do` allows any number of values to be passed to it, as we used the `*`

  operator when specifying the `values` argument.



- Run all staged operations via the `run` method - it passes an input through

  all of the operations that have been staged, and then returns the final

  result.





Let's define a sub-class:



%% Cell type:code id: tags:



``` 

class NumberOperator(Operator):



    def __init__(self):

        super().__init__()

        self.addFunction('add',    self.add)

        self.addFunction('mul',    self.mul)

        self.addFunction('negate', self.negate)



    def preprocess(self, value):

        return float(value)



    def add(self, a, b):

        return a + b



    def mul(self, a, b):

        return a * b



    def negate(self, a):

        return -a

```



%% Cell type:markdown id: tags:



The `NumberOperator` is a sub-class of `Operator`, which we can use for basic

numerical calculations. It provides a handful of simple numerical methods, but

the most interesting stuff is inside `__init__`.





> ```

> super().__init__()

> ```





This line invokes `Operator.__init__` - the initialisation method for the

`Operator` base-class.





In Python, we can use the [built-in `super`

method](https://docs.python.org/3.5/library/functions.html#super) to take care

method](https://docs.python.org/3/library/functions.html#super) to take care

of correctly calling methods that are defined in an object's base-class (or

classes, in the case of [multiple inheritance](multiple-inheritance)).





> The `super` function is one thing which changed between Python 2 and 3 -

> in Python 2, it was necessary to pass both the type and the instance

> to `super`. So it is common to see code that looks like this:

>

> ```

> def __init__(self):

>     super(NumberOperator, self).__init__()

> ```

>

> Fortunately things are a lot cleaner in Python 3.





Let's move on to the next few lines in `__init__`:





> ```

> self.addFunction('add',    self.add)

> self.addFunction('mul',    self.mul)

> self.addFunction('negate', self.negate)

> ```





Here we are registering all of the functionality that is provided by the

`NumberOperator` class, via the `Operator.addFunction` method.





The `NumberOperator` class has also overridden the `preprocess` method, to

ensure that all values handled by the `Operator` are numbers. This method gets

called within the `Operator.run` method - for a `NumberOperator` instance, the

`NumberOperator.preprocess` method will get called<sup>3</sup>.





> <sup>3</sup> When a sub-class overrides a base-class method, it is still

> possible to access the base-class implementation [via the `super()`

> function](https://stackoverflow.com/a/4747427) (the preferred method), or by

> [explicitly calling the base-class

> implementation](https://stackoverflow.com/a/2421325).





Now let's see what our `NumberOperator` class does:



%% Cell type:code id: tags:



``` 

no = NumberOperator()

no.do('add', 10)

no.do('mul', 2)

no.do('negate')



print('Operations on {}: {}'.format(10,  no.run(10)))

print('Operations on {}: {}'.format(2.5, no.run(5)))

```



%% Cell type:markdown id: tags:



It works! While this is a contrived example, hopefully you can see how

inheritance can be used to break a problem down into sub-problems:



- The `Operator` class provides all of the logic needed to manage and execute

  operations, without caring about what those operations are actually doing.



- This leaves the `NumberOperator` class free to concentrate on implementing

  the functions that are specific to its task, and not having to worry about

  how they are executed.





We could also easily implement other `Operator` sub-classes to work on

different data types, such as arrays, images, or even non-numeric data such as

strings:



%% Cell type:code id: tags:



``` 

class StringOperator(Operator):

    def __init__(self):

        super().__init__()

        self.addFunction('capitalise', self.capitalise)

        self.addFunction('concat',     self.concat)



    def preprocess(self, value):

        return str(value)



    def capitalise(self, s):

        return ' '.join([w[0].upper() + w[1:] for w in s.split()])



    def concat(self, s1, s2):

        return s1 + s2



so = StringOperator()

so.do('capitalise')

so.do('concat', '!')



print(so.run('python is an ok language'))

```



%% Cell type:markdown id: tags:



<a class="anchor" id="polymorphism"></a>

### Polymorphism





Inheritance also allows us to take advantage of _polymorphism_, which refers

to idea that, in an object-oriented language, we should be able to use an

Inheritance also allows us to take advantage of *polymorphism*, which refers

to the idea that, in an object-oriented language, we should be able to use an

object without having complete knowledge about the class, or type, of that

object. For example, we should be able to write a function which expects an

`Operator` instance, but which will work on an instance of any `Operator`

sub-classs. As an example, let's write a function which prints a summary of an

`Operator` instance:



%% Cell type:code id: tags:



``` 

def operatorSummary(o):

    print(type(o).__name__)

    print('  All functions: ')

    for fname in o.functions.keys():

        print('    {}'.format(fname))

    print('  Staged operations: ')

    for i, (fname, vals) in enumerate(o.operations):

        vals = ', '.join([str(v) for v in vals])

        print('    {}: {}({})'.format(i + 1, fname, vals))

```



%% Cell type:markdown id: tags:



Because the `operatorSummary` function only uses methods that are defined

in the `Operator` base-class, we can use it on _any_ `Operator` instance,

regardless of its specific type:



%% Cell type:code id: tags:



``` 

operatorSummary(no)

operatorSummary(so)

```



%% Cell type:markdown id: tags:



<a class="anchor" id="multiple-inheritance"></a>

### Multiple inheritance





Python allows you to define a class which has multiple base classes - this is

known as _multiple inheritance_. For example, we might want to build a

notification mechanisim into our `StringOperator` class, so that listeners can

be notified whenever the `capitalise` method gets called. It so happens that

our old colleague of `Operator` class fame also wrote a `Notifier` class which

allows listeners to register to be notified when an event occurs:



%% Cell type:code id: tags:



``` 

class Notifier(object):



    def __init__(self):

        super().__init__()

        self.__listeners = {}



    def register(self, name, func):

        self.__listeners[name] = func



    def notify(self, *args, **kwargs):

        for func in self.__listeners.values():

            func(*args, **kwargs)

```



%% Cell type:markdown id: tags:



Let's modify the `StringOperator` class to use the functionality of the

`Notifier ` class:



%% Cell type:code id: tags:



``` 

class StringOperator(Operator, Notifier):



    def __init__(self):

        super().__init__()

        self.addFunction('capitalise', self.capitalise)

        self.addFunction('concat',     self.concat)



    def preprocess(self, value):

        return str(value)



    def capitalise(self, s):

        result = ' '.join([w[0].upper() + w[1:] for w in s.split()])

        self.notify(result)

        return result



    def concat(self, s1, s2):

        return s1 + s2

```



%% Cell type:markdown id: tags:



Now, anything which is interested in uses of the `capitalise` method can

register as a listener on a `StringOperator` instance:



%% Cell type:code id: tags:



``` 

so = StringOperator()



def capitaliseCalled(result):

    print('Capitalise operation called: {}'.format(result))



so.register('mylistener', capitaliseCalled)



so.do('capitalise')

so.do('concat', '?')



print(so.run('did you notice that functions are objects too'))

```



%% Cell type:markdown id: tags:



> Simple classes such as the `Notifier` are sometimes referred to as

> [_mixins_](https://en.wikipedia.org/wiki/Mixin).





If you wish to use multiple inheritance in your design, it is important to be

aware of the mechanism that Python uses to determine how base class methods

are called (and which base class method will be called, in the case of naming

conflicts). This is referred to as the Method Resolution Order (MRO) - further

details on the topic can be found

[here](https://www.python.org/download/releases/2.3/mro/), and a more concise

summary

[here](http://python-history.blogspot.co.uk/2010/06/method-resolution-order.html).





Note also that for base class `__init__` methods to be correctly called in a

design which uses multiple inheritance, _all_ classes in the hierarchy must

invoke `super().__init__()`. This can become complicated when some base

classes expect to be passed arguments to their `__init__` method. In scenarios

like this it may be prefereable to manually invoke the base class `__init__`

methods instead of using `super()`. For example:





> ```

> class StringOperator(Operator, Notifier):

>     def __init__(self):

>         Operator.__init__(self)

>         Notifier.__init__(self)

> ```





This approach has the disadvantage that if the base classes change, you will

have to change these invocations. But the advantage is that you know exactly

how the class hierarchy will be initialised. In general though, doing

everything with `super()` will result in more maintainable code.





<a class="anchor" id="class-attributes-and-methods"></a>

## Class attributes and methods





Up to this point we have been covering how to add attributes and methods to an

_object_. But it is also possible to add methods and attributes to a _class_

(`static` methods and fields, for those of you familiar with C++ or Java).





Class attributes and methods can be accessed without having to create an

instance of the class - they are not associated with individual objects, but

rather with the class itself.





Class methods and attributes can be useful in several scenarios - as a

hypothetical, not very useful example, let's say that we want to gain usage

statistics for how many times each type of operation is used on instances of

our `FSLMaths` class. We might, for example, use this information in a grant

application to show evidence that more research is needed to optimise the

performance of the `add` operation.





<a class="anchor" id="class-attributes"></a>

### Class attributes





Let's add a `dict` called `opCounters` as a class attribute to the `FSLMaths`

class - whenever an operation is called on a `FSLMaths` instance, the counter

for that operation will be incremented:



%% Cell type:code id: tags:



``` 

import numpy   as np

import nibabel as nib



class FSLMaths(object):



    # It's this easy to add a class-level

    # attribute. This dict is associated

    # with the FSLMaths *class*, not with

    # any individual FSLMaths instance.

    opCounters = {}



    def __init__(self, inimg):

        self.img        = inimg

        self.operations = []



    def add(self, value):

        self.operations.append(('add', value))

        return self



    def mul(self, value):

        self.operations.append(('mul', value))

        return self



    def div(self, value):

        self.operations.append(('div', value))

        return self



    def run(self, output=None):



        data = np.array(self.img.get_data())



        for oper, value in self.operations:



            # Code omitted for brevity



            # Increment the usage counter

            # for this operation. We can

            # access class attributes (and

            # methods) through the class

            # itself.

            FSLMaths.opCounters[oper] = self.opCounters.get(oper, 0) + 1

            # Increment the usage counter for this operation. We can

            # access class attributes (and methods) through the class

            # itself, as shown here.

            FSLMaths.opCounters[oper] = FSLMaths.opCounters.get(oper, 0) + 1



            # It is also possible to access class-level

            # attributes via instances of the class, e.g.

            # self.opCounters[oper] = self.opCounters.get(oper, 0) + 1

```



%% Cell type:markdown id: tags:



So let's see it in action:



%% Cell type:code id: tags:



``` 

fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')

fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz')

inimg = nib.load(fpath)

mask  = nib.load(fmask)



fm1 = FSLMaths(inimg)

fm2 = FSLMaths(inimg)



fm1.mul(mask)

fm1.add(15)



fm2.add(25)

fm1.div(1.5)



fm1.run()

fm2.run()

FSLMaths(inimg).mul(mask).add(25).run()

FSLMaths(inimg).add(15).div(1.5).run()



print('FSLMaths usage statistics')

for oper in ('add', 'div', 'mul'):

    print('  {} : {}'.format(oper, FSLMaths.opCounters.get(oper, 0)))

```



%% Cell type:markdown id: tags:



<a class="anchor" id="class-methods"></a>

### Class methods





It is just as easy to add a method to a class - let's take our reporting code

from above, and add it as a method to the `FSLMaths` class.





A class method is denoted by the

[`@classmethod`](https://docs.python.org/3.5/library/functions.html#classmethod)

decorator. Note that, where a regular method which is called on an instance

will be passed the instance as its first argument (`self`), a class method

will be passed the class itself as the first argument - the standard

convention is to call this argument `cls`:



%% Cell type:code id: tags:



``` 

class FSLMaths(object):



    opCounters = {}



    @classmethod

    def usage(cls):

        print('FSLMaths usage statistics')

        for oper in ('add', 'div', 'mul'):

            print('  {} : {}'.format(oper, FSLMaths.opCounters.get(oper, 0)))



    def __init__(self, inimg):

        self.img        = inimg

        self.operations = []



    def add(self, value):

        self.operations.append(('add', value))

        return self



    def mul(self, value):

        self.operations.append(('mul', value))

        return self



    def div(self, value):

        self.operations.append(('div', value))

        return self



    def run(self, output=None):



        data = np.array(self.img.get_data())



        for oper, value in self.operations:

            FSLMaths.opCounters[oper] = self.opCounters.get(oper, 0) + 1

```



%% Cell type:markdown id: tags:



> There is another decorator -

> [`@staticmethod`](https://docs.python.org/3.5/library/functions.html#staticmethod) -

> which can be used on methods defined within a class. The difference

> between a `@classmethod` and a `@staticmethod` is that the latter will _not_

> between a `@classmethod` and a `@staticmethod` is that the latter will *not*

> be passed the class (`cls`).





calling a class method is the same as accessing a class attribute:

Calling a class method is the same as accessing a class attribute:



%% Cell type:code id: tags:



``` 

fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')

fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz')

inimg = nib.load(fpath)

mask  = nib.load(fmask)



fm1 = FSLMaths(inimg)

fm2 = FSLMaths(inimg)



fm1.mul(mask)

fm1.add(15)



fm2.add(25)

fm1.div(1.5)

fm1 = FSLMaths(inimg).mul(mask).add(25)

fm2 = FSLMaths(inimg).add(15).div(1.5)



fm1.run()

fm2.run()



FSLMaths.usage()

```



%% Cell type:markdown id: tags:



Note that it is also possible to access class attributes and methods through

instances:



%% Cell type:code id: tags:



``` 

print(fm1.opCounters)

fm1.usage()

```



%% Cell type:markdown id: tags:



<a class="anchor" id="appendix-the-object-base-class"></a>

## Appendix: The `object` base-class





When you are defining a class, you need to specify the base-class from which

your class inherits. If your class does not inherit from a particular class,

then it should inherit from the built-in `object` class:





> ```

> class MyClass(object):

>     ...

> ```





However, in older code bases, you might see class definitions that look like

this, without explicitly inheriting from the `object` base class:





> ```

> class MyClass:

>     ...

> ```





This syntax is a [throwback to older versions of

Python](https://docs.python.org/2/reference/datamodel.html#new-style-and-classic-classes).

In Python 3 there is actually no difference in defining classes in the

"new-style" way we have used throughout this tutorial, or the "old-style" way

mentioned in this appendix.





But if you are writing code which needs to run on both Python 2 and 3, you

__must__ define your classes in the new-style convention, i.e. by explicitly

inheriting from the `object` base class. Therefore, the safest approach is to

always use the new-style format.





<a class="anchor" id="appendix-init-versus-new"></a>

## Appendix: `__init__` versus `__new__`





In Python, object creation is actually a two-stage process - _creation_, and

then _initialisation_. The `__init__` method gets called during the

_initialisation_ stage - its job is to initialise the state of the object. But

In Python, object creation is actually a two-stage process - *creation*, and

then *initialisation*. The `__init__` method gets called during the

*initialisation* stage - its job is to initialise the state of the object. But

note that, by the time `__init__` gets called, the object has already been

created.





You can also define a method called `__new__` if you need to control the

creation stage, although this is very rarely needed. One example of where you

might need to implement the `__new__` method is if you wish to create a

[subclass of a

`numpy.array`](https://docs.scipy.org/doc/numpy-1.14.0/user/basics.subclassing.html)

(although you might alternatively want to think about redefining your problem

so that this is not necessary).





A brief explanation on

the difference between `__new__` and `__init__` can be found

[here](https://www.reddit.com/r/learnpython/comments/2s3pms/what_is_the_difference_between_init_and_new/cnm186z/),

and you may also wish to take a look at the [official Python

docs](https://docs.python.org/3.5/reference/datamodel.html#basic-customization).

docs](https://docs.python.org/3/reference/datamodel.html#basic-customization).





<a class="anchor" id="appendix-monkey-patching"></a>

## Appendix: Monkey-patching





The act of run-time modification of objects or class definitions is referred

to as [_monkey-patching_](https://en.wikipedia.org/wiki/Monkey_patch) and,

to as [*monkey-patching*](https://en.wikipedia.org/wiki/Monkey_patch) and,

whilst it is allowed by the Python programming language, it is generally

considered quite bad practice.





Just because you _can_ do something doesn't mean that you _should_. Python

Just because you *can* do something doesn't mean that you *should*. Python

gives you the flexibility to write your software in whatever manner you deem

suitable.  __But__ if you want to write software that will be used, adopted,

suitable.  **But** if you want to write software that will be used, adopted,

maintained, and enjoyed by other people, you should be polite, write your code

in a clear, readable fashion, and avoid the use of devious tactics such as

monkey-patching.





__However__, while monkey-patching may seem like a horrific programming

**However**, while monkey-patching may seem like a horrific programming

practice to those of you coming from the realms of C++, Java, and the like,

(and it is horrific in many cases), it can be _extremely_ useful in certain

(and it is horrific in many cases), it can be *extremely* useful in certain

circumstances.  For instance, monkey-patching makes [unit testing a

breeze in Python](https://docs.python.org/3.5/library/unittest.mock.html).

breeze in Python](https://docs.python.org/3/library/unittest.mock.html).





As another example, consider the scenario where you are dependent on a third

party library which has bugs in it. No problem - while you are waiting for the

library author to release a new version of the library, you can write your own

working implementation and [monkey-patch it

in!](https://git.fmrib.ox.ac.uk/fsl/fsleyes/fsleyes/blob/0.21.0/fsleyes/views/viewpanel.py#L726)





<a class="anchor" id="appendix-method-overloading"></a>

## Appendix: Method overloading





Method overloading (defining multiple methods with the same name in a class,

but each accepting different arguments) is one of the only object-oriented

features that is not present in Python. Becuase Python does not perform any

runtime checks on the types of arguments that are passed to a method, or the

compatibility of the method to accept the arguments, it would not be possible

to determine which implementation of a method is to be called. In other words,

in Python only the name of a method is used to identify that method, unlike in

C++ and Java, where the full method signature (name, input types and return

types) is used.





However, because a Python method can be written to accept any number or type

of arguments, it is very easy to to build your own overloading logic by

writing a "dispatch" method<sup>4</sup>. Here is YACE (Yet Another Contrived

Example):



%% Cell type:code id: tags:



``` 

class Adder(object):



    def add(self, *args):

        if   len(args) == 2: return self.__add2(*args)

        elif len(args) == 3: return self.__add3(*args)

        elif len(args) == 4: return self.__add4(*args)

        else:

            raise AttributeError('No method available to accept {} '

                                 'arguments'.format(len(args)))



    def __add2(self, a, b):

        return a + b



    def __add3(self, a, b, c):

        return a + b + c



    def __add4(self, a, b, c, d):

        return a + b + c + d



a = Adder()



print('Add two:   {}'.format(a.add(1, 2)))

print('Add three: {}'.format(a.add(1, 2, 3)))

print('Add four:  {}'.format(a.add(1, 2, 3, 4)))

```



%% Cell type:markdown id: tags:



> <sup>4</sup>Another option is the [`functools.singledispatch`

> decorator](https://docs.python.org/3.5/library/functools.html#functools.singledispatch),

> decorator](https://docs.python.org/3/library/functools.html#functools.singledispatch),

> which is more complicated, but may allow you to write your dispatch logic in

> a more concise manner.





<a class="anchor" id="useful-references"></a>

## Useful references





The official Python documentation has a wealth of information on the internal

workings of classes and objects, so these pages are worth a read:





* https://docs.python.org/3.5/tutorial/classes.html

* https://docs.python.org/3.5/reference/datamodel.html
 
* https://docs.python.org/3/tutorial/classes.html

* https://docs.python.org/3/reference/datamodel.html

advanced_topics/03_object_oriented_programming.md

+152 −67
Original line number Diff line number Diff line
@@ -19,6 +19,7 @@ you use an object-oriented approach. 
 * [We didn't specify the `self` argument - what gives?!?](#we-didnt-specify-the-self-argument)

* [Attributes](#attributes)

* [Methods](#methods)

* [Method chaining](#method-chaining)

* [Protecting attribute access](#protecting-attribute-access)

 * [A better way - properties](#a-better-way-properties])

* [Inheritance](#inheritance)
@@ -46,8 +47,8 @@ section. 




If you have not done any object-oriented programming before, your first step

is to understand the difference between _objects_ (also known as

_instances_) and _classes_ (also known as _types_).

is to understand the difference between *objects* (also known as

*instances*) and *classes* (also known as *types*).





If you have some experience in C, then you can start off by thinking of a
@@ -66,8 +67,8 @@ layout of a chunk of memory. For example, here is a typical struct definition: 
> ```





Now, an _object_ is not a definition, but rather a thing which resides in

memory. An object can have _attributes_ (pieces of information), and _methods_

Now, an *object* is not a definition, but rather a thing which resides in

memory. An object can have *attributes* (pieces of information), and *methods*

(functions associated with the object). You can pass objects around your code,

manipulate their attributes, and call their methods.
@@ -92,12 +93,12 @@ you create an object from that class. 
Of course there are many more differences between C structs and classes (most

notably [inheritance](todo), [polymorphism](todo), and [access

protection](todo)). But if you can understand the difference between a

_definition_ of a C struct, and an _instantiation_ of that struct, then you

are most of the way towards understanding the difference between a _class_,

and an _object_.

*definition* of a C struct, and an *instantiation* of that struct, then you

are most of the way towards understanding the difference between a *class*,

and an *object*.





> But just to confuse you, remember that in Python, __everything__ is an

> But just to confuse you, remember that in Python, **everything** is an

> object - even classes!
@@ -206,7 +207,7 @@ print(fm) 




Refer to the [official

docs](https://docs.python.org/3.5/reference/datamodel.html#special-method-names)

docs](https://docs.python.org/3/reference/datamodel.html#special-method-names)

for details on all of the special methods that can be defined in a class. And

take a look at the appendix for some more details on [how Python objects get

created](appendix-init-versus-new).
@@ -352,8 +353,8 @@ append a tuple to that `operations` list. 


The idea behind this design is that our `FSLMaths` class will not actually do

anything when we call the `add`, `mul` or `div` methods. Instead, it will

"stage" each operation, and then perform them all in one go. So let's add

another method, `run`, which actually does the work:

*stage* each operation, and then perform them all in one go at a later point

in time. So let's add another method, `run`, which actually does the work:





```
@@ -387,7 +388,6 @@ class FSLMaths(object): 
            if isinstance(value, nib.nifti1.Nifti1Image):

                value = value.get_data()





            if oper == 'add':

                data = data + value

            elif oper == 'mul':
@@ -430,6 +430,99 @@ print('Number of voxels >0 in masked image: {}'.format(nmaskvox)) 
```





<a class="anchor" id="method-chaining"></a>

## Method chaining





A neat trick, which is used by all the cool kids these days, is to write

classes that allow *method chaining* - writing one line of code which

calls more than one method on an object, e.g.:



> ```

> fm = FSLMaths(img)

> result = fm.add(1).mul(10).run()

> ```



Adding this feature to our budding `FSLMaths` class is easy - all we have

to do is return `self` from each method:



```

import numpy   as np

import nibabel as nib



class FSLMaths(object):



    def __init__(self, inimg):

        self.img        = inimg

        self.operations = []



    def add(self, value):

        self.operations.append(('add', value))

        return self



    def mul(self, value):

        self.operations.append(('mul', value))

        return self



    def div(self, value):

        self.operations.append(('div', value))

        return self



    def run(self, output=None):



        data = np.array(self.img.get_data())



        for oper, value in self.operations:



            # Value could be an image.

            # If not, we assume that

            # it is a scalar/numpy array.

            if isinstance(value, nib.nifti1.Nifti1Image):

                value = value.get_data()



            if oper == 'add':

                data = data + value

            elif oper == 'mul':

                data = data * value

            elif oper == 'div':

                data = data / value



        # turn final output into a nifti,

        # and save it to disk if an

        # 'output' has been specified.

        outimg = nib.nifti1.Nifti1Image(data, inimg.affine)



        if output is not None:

            nib.save(outimg, output)



        return outimg

```





Now we can chain all of our method calls, and even the creation of our

`FSLMaths` object, into a single line:





```

fpath = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm.nii.gz')

fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz')

inimg = nib.load(fpath)

mask  = nib.load(fmask)



outimg = FSLMaths(inimg).mul(mask).add(-10).run()



norigvox = (inimg .get_data() > 0).sum()

nmaskvox = (outimg.get_data() > 0).sum()



print('Number of voxels >0 in original image: {}'.format(norigvox))

print('Number of voxels >0 in masked image:   {}'.format(nmaskvox))

```



> In fact, this is precisely how the

> [`fsl.wrappers.fslmaths`](https://users.fmrib.ox.ac.uk/~paulmc/fsleyes/fslpy/latest/fsl.wrappers.fslmaths.html)

> function works.





<a class="anchor" id="protecting-attribute-access"></a>

## Protecting attribute access
@@ -488,9 +581,8 @@ of an object. This is in contrast to languages like C++ and Java, where the 
notion of a private attribute or method is strictly enforced by the language.





However, there are a couple of conventions in Python that are [universally

adhered

to](https://docs.python.org/3.5/tutorial/classes.html#private-variables):

However, there are a couple of conventions in Python that are

[universally adhered to](https://docs.python.org/3/tutorial/classes.html#private-variables):



* Class-level attributes and methods, and module-level attributes, functions,

  and classes, which begin with a single underscore (`_`), should be
@@ -504,14 +596,13 @@ to](https://docs.python.org/3.5/tutorial/classes.html#private-variables): 
  enforcement for this rule - any attribute or method with such a name will

  actually be _renamed_ (in a standardised manner) at runtime, so that it is

  not accessible through its original name (it is still accessible via its

  [mangled

  name](https://docs.python.org/3.5/tutorial/classes.html#private-variables)

  [mangled name](https://docs.python.org/3/tutorial/classes.html#private-variables)

  though).





> <sup>2</sup> With the exception that module-level fields which begin with a

> single underscore will not be imported into the local scope via the

> `from [module] import *` techinque.

> `from [module] import *` technique.





So with all of this in mind, we can adjust our `FSLMaths` class to discourage
@@ -541,7 +632,7 @@ print(fm.__img) 




Python has a feature called

[`properties`](https://docs.python.org/3.5/library/functions.html#property),

[`properties`](https://docs.python.org/3/library/functions.html#property),

which is a nice way of controlling access to the attributes of an object. We

can use properties by defining a "getter" method which can be used to access

our attributes, and "decorating" them with the `@property` decorator (we will
@@ -676,17 +767,17 @@ class Chihuahua(Dog): 




Hopefully this example doesn't need much in the way of explanation - this

collection of classes captures a hierarchical relationship which exists in the

real world (and also captures the inherently annoying nature of

collection of classes represents a hierarchical relationship which exists in

the real world (and also represents the inherently annoying nature of

chihuahuas). For example, in the real world, all dogs are animals, but not all

animals are dogs.  Therefore in our model, the `Dog` class has specified

`Animal` as its base class. We say that the `Dog` class _extends_, _derives

from_, or _inherits from_, the `Animal` class, and that all `Dog` instances

`Animal` as its base class. We say that the `Dog` class *extends*, *derives

from*, or *inherits from*, the `Animal` class, and that all `Dog` instances

are also `Animal` instances (but not vice-versa).





What does that `noiseMade` method do?  There is a `noiseMade` method defined

on the `Animal` class, but it has been re-implemented, or _overridden_ in the

on the `Animal` class, but it has been re-implemented, or *overridden* in the

`Dog`,

[`TalkingDog`](https://twitter.com/simpsonsqotd/status/427941665836630016?lang=en),

`Cat`, and `Chihuahua` classes (but not on the `Labrador` class).  We can call
@@ -819,7 +910,7 @@ This line invokes `Operator.__init__` - the initialisation method for the 




In Python, we can use the [built-in `super`

method](https://docs.python.org/3.5/library/functions.html#super) to take care

method](https://docs.python.org/3/library/functions.html#super) to take care

of correctly calling methods that are defined in an object's base-class (or

classes, in the case of [multiple inheritance](multiple-inheritance)).
@@ -920,8 +1011,8 @@ print(so.run('python is an ok language')) 
### Polymorphism





Inheritance also allows us to take advantage of _polymorphism_, which refers

to idea that, in an object-oriented language, we should be able to use an

Inheritance also allows us to take advantage of *polymorphism*, which refers

to the idea that, in an object-oriented language, we should be able to use an

object without having complete knowledge about the class, or type, of that

object. For example, we should be able to write a function which expects an

`Operator` instance, but which will work on an instance of any `Operator`
@@ -1110,12 +1201,15 @@ class FSLMaths(object): 


    def add(self, value):

        self.operations.append(('add', value))

        return self



    def mul(self, value):

        self.operations.append(('mul', value))

        return self



    def div(self, value):

        self.operations.append(('div', value))

        return self



    def run(self, output=None):
@@ -1125,12 +1219,15 @@ class FSLMaths(object): 


            # Code omitted for brevity



            # Increment the usage counter

            # for this operation. We can

            # access class attributes (and

            # methods) through the class

            # itself.

            FSLMaths.opCounters[oper] = self.opCounters.get(oper, 0) + 1

            # Increment the usage counter for this operation. We can

            # access class attributes (and methods) through the class

            # itself, as shown here.

            FSLMaths.opCounters[oper] = FSLMaths.opCounters.get(oper, 0) + 1



            # It is also possible to access class-level

            # attributes via instances of the class, e.g.

            # self.opCounters[oper] = self.opCounters.get(oper, 0) + 1



```
@@ -1143,17 +1240,8 @@ fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz') 
inimg = nib.load(fpath)

mask  = nib.load(fmask)



fm1 = FSLMaths(inimg)

fm2 = FSLMaths(inimg)



fm1.mul(mask)

fm1.add(15)



fm2.add(25)

fm1.div(1.5)



fm1.run()

fm2.run()

FSLMaths(inimg).mul(mask).add(25).run()

FSLMaths(inimg).add(15).div(1.5).run()



print('FSLMaths usage statistics')

for oper in ('add', 'div', 'mul'):
@@ -1194,12 +1282,15 @@ class FSLMaths(object): 


    def add(self, value):

        self.operations.append(('add', value))

        return self



    def mul(self, value):

        self.operations.append(('mul', value))

        return self



    def div(self, value):

        self.operations.append(('div', value))

        return self



    def run(self, output=None):
@@ -1213,11 +1304,11 @@ class FSLMaths(object): 
> There is another decorator -

> [`@staticmethod`](https://docs.python.org/3.5/library/functions.html#staticmethod) -

> which can be used on methods defined within a class. The difference

> between a `@classmethod` and a `@staticmethod` is that the latter will _not_

> between a `@classmethod` and a `@staticmethod` is that the latter will *not*

> be passed the class (`cls`).





calling a class method is the same as accessing a class attribute:

Calling a class method is the same as accessing a class attribute:





```
@@ -1226,14 +1317,8 @@ fmask = op.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz') 
inimg = nib.load(fpath)

mask  = nib.load(fmask)



fm1 = FSLMaths(inimg)

fm2 = FSLMaths(inimg)



fm1.mul(mask)

fm1.add(15)



fm2.add(25)

fm1.div(1.5)

fm1 = FSLMaths(inimg).mul(mask).add(25)

fm2 = FSLMaths(inimg).add(15).div(1.5)



fm1.run()

fm2.run()
@@ -1294,9 +1379,9 @@ always use the new-style format. 
## Appendix: `__init__` versus `__new__`





In Python, object creation is actually a two-stage process - _creation_, and

then _initialisation_. The `__init__` method gets called during the

_initialisation_ stage - its job is to initialise the state of the object. But

In Python, object creation is actually a two-stage process - *creation*, and

then *initialisation*. The `__init__` method gets called during the

*initialisation* stage - its job is to initialise the state of the object. But

note that, by the time `__init__` gets called, the object has already been

created.
@@ -1314,7 +1399,7 @@ A brief explanation on 
the difference between `__new__` and `__init__` can be found

[here](https://www.reddit.com/r/learnpython/comments/2s3pms/what_is_the_difference_between_init_and_new/cnm186z/),

and you may also wish to take a look at the [official Python

docs](https://docs.python.org/3.5/reference/datamodel.html#basic-customization).

docs](https://docs.python.org/3/reference/datamodel.html#basic-customization).





<a class="anchor" id="appendix-monkey-patching"></a>
@@ -1322,24 +1407,24 @@ docs](https://docs.python.org/3.5/reference/datamodel.html#basic-customization). 




The act of run-time modification of objects or class definitions is referred

to as [_monkey-patching_](https://en.wikipedia.org/wiki/Monkey_patch) and,

to as [*monkey-patching*](https://en.wikipedia.org/wiki/Monkey_patch) and,

whilst it is allowed by the Python programming language, it is generally

considered quite bad practice.





Just because you _can_ do something doesn't mean that you _should_. Python

Just because you *can* do something doesn't mean that you *should*. Python

gives you the flexibility to write your software in whatever manner you deem

suitable.  __But__ if you want to write software that will be used, adopted,

suitable.  **But** if you want to write software that will be used, adopted,

maintained, and enjoyed by other people, you should be polite, write your code

in a clear, readable fashion, and avoid the use of devious tactics such as

monkey-patching.





__However__, while monkey-patching may seem like a horrific programming

**However**, while monkey-patching may seem like a horrific programming

practice to those of you coming from the realms of C++, Java, and the like,

(and it is horrific in many cases), it can be _extremely_ useful in certain

(and it is horrific in many cases), it can be *extremely* useful in certain

circumstances.  For instance, monkey-patching makes [unit testing a

breeze in Python](https://docs.python.org/3.5/library/unittest.mock.html).

breeze in Python](https://docs.python.org/3/library/unittest.mock.html).





As another example, consider the scenario where you are dependent on a third
@@ -1398,7 +1483,7 @@ print('Add four: {}'.format(a.add(1, 2, 3, 4))) 
```



> <sup>4</sup>Another option is the [`functools.singledispatch`

> decorator](https://docs.python.org/3.5/library/functools.html#functools.singledispatch),

> decorator](https://docs.python.org/3/library/functools.html#functools.singledispatch),

> which is more complicated, but may allow you to write your dispatch logic in

> a more concise manner.
@@ -1411,5 +1496,5 @@ The official Python documentation has a wealth of information on the internal 
workings of classes and objects, so these pages are worth a read:





* https://docs.python.org/3.5/tutorial/classes.html

* https://docs.python.org/3.5/reference/datamodel.html
 
* https://docs.python.org/3/tutorial/classes.html

* https://docs.python.org/3/reference/datamodel.html

advanced_topics/04_operator_overloading.ipynb

+14 −14
Original line number Diff line number Diff line 
%% Cell type:markdown id: tags:



# Operator overloading





> This practical assumes you are familiar with the basics of object-oriented

> programming in Python.





Operator overloading, in an object-oriented programming language, is the

process of customising the behaviour of _operators_ (e.g. `+`, `*`, `/` and

`-`) on user-defined types. This practical aims to show you that operator

overloading is __very__ easy to do in Python.

overloading is **very** easy to do in Python.





This practical gives a brief overview of the operators which you may be most

interested in implementing. However, there are many operators (and other

special methods) which you can support in your own classes - the [official

documentation](https://docs.python.org/3.5/reference/datamodel.html#basic-customization)

documentation](https://docs.python.org/3/reference/datamodel.html#basic-customization)

is the best reference if you are interested in learning more.





* [Overview](#overview)

* [Arithmetic operators](#arithmetic-operators)

* [Equality and comparison operators](#equality-and-comparison-operators)

* [The indexing operator `[]`](#the-indexing-operator)

* [The call operator `()`](#the-call-operator)

* [The dot operator `.`](#the-dot-operator)





<a class="anchor" id="overview"></a>

## Overview





In Python, when you add two numbers together:



%% Cell type:code id: tags:



``` 

a = 5

b = 10

r = a + b

print(r)

```



%% Cell type:markdown id: tags:



What actually goes on behind the scenes is this:



%% Cell type:code id: tags:



``` 

r = a.__add__(b)

print(r)

```



%% Cell type:markdown id: tags:



In other words, whenever you use the `+` operator on two variables (the

operands to the `+` operator), the Python interpreter calls the `__add__`

method of the first operand (`a`), and passes the second operand (`b`) as an

argument.





So it is very easy to use the `+` operator with our own classes - all we have

to do is implement a method called `__add__`.





<a class="anchor" id="arithmetic-operators"></a>

## Arithmetic operators





Let's play with an example - a class which represents a 2D vector:



%% Cell type:code id: tags:



``` 

class Vector2D(object):

    def __init__(self, x, y):

        self.x = x

        self.y = y



    def __str__(self):

        return 'Vector2D({}, {})'.format(self.x, self.y)

```



%% Cell type:markdown id: tags:



> Note that we have implemented the special `__str__` method, which allows our

> `Vector2D` instances to be converted into strings.





If we try to use the `+` operator on this class, we are bound to get an error:



%% Cell type:code id: tags:



``` 

v1 = Vector2D(2, 3)

v2 = Vector2D(4, 5)

print(v1 + v2)

```



%% Cell type:markdown id: tags:



But all we need to do to support the `+` operator is to implement a method

called `__add__`:



%% Cell type:code id: tags:



``` 

class Vector2D(object):

    def __init__(self, x, y):

        self.x = x

        self.y = y



    def __str__(self):

        return 'Vector2D({}, {})'.format(self.x, self.y)



    def __add__(self, other):

        return Vector2D(self.x + other.x,

                        self.y + other.y)

```



%% Cell type:markdown id: tags:



And now we can use `+` on `Vector2D` objects - it's that easy:



%% Cell type:code id: tags:



``` 

v1 = Vector2D(2, 3)

v2 = Vector2D(4, 5)

print('{} + {} = {}'.format(v1, v2, v1 + v2))

```



%% Cell type:markdown id: tags:



Our `__add__` method creates and returns a new `Vector2D` which contains the

sum of the `x` and `y` components of the `Vector2D` on which it is called, and

the `Vector2D` which is passed in.  We could also make the `__add__` method

work with scalars, by extending its definition a bit:



%% Cell type:code id: tags:



``` 

class Vector2D(object):

    def __init__(self, x, y):

        self.x = x

        self.y = y



    def __add__(self, other):

        if isinstance(other, Vector2D):

            return Vector2D(self.x + other.x,

                            self.y + other.y)

        else:

            return Vector2D(self.x + other, self.y + other)



    def __str__(self):

        return 'Vector2D({}, {})'.format(self.x, self.y)

```



%% Cell type:markdown id: tags:



So now we can add both `Vector2D` instances and scalars numbers together:



%% Cell type:code id: tags:



``` 

v1 = Vector2D(2, 3)

v2 = Vector2D(4, 5)

n  = 6



print('{} + {} = {}'.format(v1, v2, v1 + v2))

print('{} + {} = {}'.format(v1, n,  v1 + n))

```



%% Cell type:markdown id: tags:



Other numeric and logical operators can be supported by implementing the

appropriate method, for example:



- Multiplication (`*`): `__mul__`

- Division (`/`): `__div__`

- Negation (`-`): `__neg__`

- In-place addition (`+=`): `__iadd__`

- Exclusive or (`^`): `__xor__`





When an operator is applied to operands of different types, a set of fall-back

rules are followed depending on the set of methods implemented on the

operands. For example, in the expression `a + b`, if `a.__add__` is not

implemented, but but `b.__radd__` is implemented, then the latter will be

called.  Take a look at the [official

documentation](https://docs.python.org/3.5/reference/datamodel.html#emulating-numeric-types)

documentation](https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types)

for further details, including a full list of the arithmetic and logical

operators that your classes can support.





<a class="anchor" id="equality-and-comparison-operators"></a>

## Equality and comparison operators





Adding support for equality (`==`, `!=`) and comparison (e.g. `>=`) operators

is just as easy. Imagine that we have a class called `Label`, which represents

a label in a lookup table. Our `Label` has an integer label, a name, and an

RGB colour:



%% Cell type:code id: tags:



``` 

class Label(object):

    def __init__(self, label, name, colour):

        self.label  = label

        self.name   = name

        self.colour = colour

```



%% Cell type:markdown id: tags:



In order to ensure that a list of `Label` objects is ordered by their label

values, we can implement a set of functions, so that `Label` classes can be

compared using the standard comparison operators:



%% Cell type:code id: tags:



``` 

import functools



# Don't worry about this statement

# just yet - it is explained below

@functools.total_ordering



class Label(object):

    def __init__(self, label, name, colour):

        self.label  = label

        self.name   = name

        self.colour = colour



    def __str__(self):

        rgb = ''.join(['{:02x}'.format(c) for c in self.colour])

        return 'Label({}, {}, #{})'.format(self.label, self.name, rgb)



    def __repr__(self):

        return str(self)



    # implement Label == Label

    def __eq__(self, other):

        return self.label == other.label



    # implement Label < Label

    def __lt__(self, other):

        return self.label < other.label

```



%% Cell type:markdown id: tags:



> We also added `__str__` and `__repr__` methods to the `Label` class so that

> `Label` instances will be printed nicely.





Now we can compare and sort our `Label` instances:



%% Cell type:code id: tags:



``` 

l1 = Label(1, 'Parietal',  (255,   0,   0))

l2 = Label(2, 'Occipital', (  0, 255,   0))

l3 = Label(3, 'Temporal',  (  0,   0, 255))



print('{} >  {}: {}'.format(l1, l2, l1  > l2))

print('{} <  {}: {}'.format(l1, l3, l1 <= l3))

print('{} != {}: {}'.format(l2, l3, l2 != l3))

print(sorted((l3, l1, l2)))

```



%% Cell type:markdown id: tags:



The

[`@functools.total_ordering`](https://docs.python.org/3.5/library/functools.html#functools.total_ordering)

[`@functools.total_ordering`](https://docs.python.org/3/library/functools.html#functools.total_ordering)

is a convenience

[decorator](https://docs.python.org/3.5/glossary.html#term-decorator) which,

[decorator](https://docs.python.org/3/glossary.html#term-decorator) which,

given a class that implements equality and a single comparison function

(`__lt__` in the above code), will "fill in" the remainder of the comparison

operators.  If you need very specific or complicated behaviour, then you can

provide methods for _all_ of the comparison operators, e.g. `__gt__` for `>`,

`__ge__` for `>=`, etc.).





> Decorators are introduced in another practical.





But if you just want the operators to work in the conventional manner, you can

simply use the `@functools.total_ordering` decorator, and provide `__eq__`,

and just one of `__lt__`, `__le__`, `__gt__` or `__ge__`.





Refer to the [official

documentation](https://docs.python.org/3.5/reference/datamodel.html#object.__lt__)

documentation](https://docs.python.org/3/reference/datamodel.html#object.__lt__)

for all of the details on supporting comparison operators.





> You may see the `__cmp__` method in older code bases - this provides a

> C-style comparison function which returns `<0`, `0`, or `>0` based on

> comparing two items. This has been superseded by the rich comparison

> operators introduced here, and is no longer supported in Python 3.





<a class="anchor" id="the-indexing-operator"></a>

## The indexing operator `[]`





The indexing operator (`[]`) is generally used by "container" types, such as

the built-in `list` and `dict` classes.





At its essence, there are only three types of behaviours that are possible

with the `[]` operator. All that is needed to support them are to implement

three special methods in your class, regardless of whether your class will be

indexed by sequential integers (like a `list`) or by

[hashable](https://docs.python.org/3.5/glossary.html#term-hashable) values

[hashable](https://docs.python.org/3/glossary.html#term-hashable) values

(like a `dict`):





- __Retrieval__ is performed by the `__getitem__` method

- __Assignment__ is performed by the `__setitem__` method

- __Deletion__ is performed by the `__delitem__` method

- **Retrieval** is performed by the `__getitem__` method

- **Assignment** is performed by the `__setitem__` method

- **Deletion** is performed by the `__delitem__` method





Note that, if you implement these methods in your own class, there is no

requirement for them to actually provide any form of data storage or

retrieval. However if you don't, you will probably confuse users of your code

who are used to how the `list` and `dict` types work. Whenever you deviate

from conventional behaviour, make sure you explain it well in your

documentation!





The following contrived example demonstrates all three behaviours:



%% Cell type:code id: tags:



``` 

class TwoTimes(object):



    def __init__(self):

        self.__deleted  = set()

        self.__assigned = {}



    def __getitem__(self, key):

        if key in self.__deleted:

            raise KeyError('{} has been deleted!'.format(key))

        elif key in self.__assigned:

            return self.__assigned[key]

        else:

            return key * 2



    def __setitem__(self, key, value):

        self.__assigned[key] = value



    def __delitem__(self, key):

        self.__deleted.add(key)

```



%% Cell type:markdown id: tags:



Guess what happens whenever we index a `TwoTimes` object:



%% Cell type:code id: tags:



``` 

tt = TwoTimes()

print('TwoTimes[{}] = {}'.format(2,     tt[2]))

print('TwoTimes[{}] = {}'.format(6,     tt[6]))

print('TwoTimes[{}] = {}'.format('abc', tt['abc']))

```



%% Cell type:markdown id: tags:



The `TwoTimes` class allows us to override the value for a specific key:



%% Cell type:code id: tags:



``` 

print(tt[4])

tt[4] = 'this is not 4 * 4'

print(tt[4])

```



%% Cell type:markdown id: tags:



And we can also "delete" keys:



%% Cell type:code id: tags:



``` 

print(tt['12345'])

del tt['12345']



# this is going to raise an error

print(tt['12345'])

```



%% Cell type:markdown id: tags:



If you wish to support the Python `start:stop:step` [slice

notation](https://docs.python.org/3.5/library/functions.html#slice), you

notation](https://docs.python.org/3/library/functions.html#slice), you

simply need to write your `__getitem__` and `__setitem__` methods so that they

can detect `slice` objects:



%% Cell type:code id: tags:



``` 

class TwoTimes(object):



    def __init__(self, max):

        self.__max = max



    def __getitem__(self, key):

        if isinstance(key, slice):

            start = key.start or 0

            stop  = key.stop  or self.__max

            step  = key.step  or 1

        else:

            start = key

            stop  = key + 1

            step  = 1



        return [i * 2 for i in range(start, stop, step)]

```



%% Cell type:markdown id: tags:



Now we can "slice" a `TwoTimes` instance:



%% Cell type:code id: tags:



``` 

tt = TwoTimes(10)



print(tt[5])

print(tt[3:7])

print(tt[::2])

```



%% Cell type:markdown id: tags:



> It is possible to sub-class the built-in `list` and `dict` classes if you

> wish to extend their functionality in some way. However, if you are writing

> a class that should mimic the one of the `list` or `dict` classes, but work

> in a different way internally (e.g. a `dict`-like object which uses a

> different hashing algorithm), the `Sequence` and `MutableMapping` classes

> are [a better choice](https://stackoverflow.com/a/7148602) - you can find

> them in the

> [`collections.abc`](https://docs.python.org/3.5/library/collections.abc.html)

> [`collections.abc`](https://docs.python.org/3/library/collections.abc.html)

> module.





<a class="anchor" id="the-call-operator"></a>

## The call operator `()`





Remember how everything in Python is an object, even functions? When you call

a function, a method called `__call__` is called on the function object. We can

implement the `__call__` method on our own class, which will allow us to "call"

objects as if they are functions.





For example, the `TimedFunction` class allows us to calculate the execution

time of any function:



%% Cell type:code id: tags:



``` 

import time



class TimedFunction(object):



    def __init__(self, func):

        self.func = func



    def __call__(self, *args, **kwargs):

        print('Timing {}...'.format(self.func.__name__))



        start  = time.time()

        retval = self.func(*args, **kwargs)

        end    = time.time()



        print('Elapsed time: {:0.2f} seconds'.format(end - start))

        return retval

```



%% Cell type:markdown id: tags:



Let's see how the `TimedFunction` behaves:



%% Cell type:code id: tags:



``` 

import numpy        as np

import numpy.linalg as npla



def inverse(data):

    return npla.inv(data)



tf   = TimedFunction(inverse)

data = np.random.random((5000, 5000))



# Wait a few seconds after

# running this code block!

inv = tf(data)

```



%% Cell type:markdown id: tags:



> The `TimedFunction` class is conceptually very similar to a

> [decorator](https://docs.python.org/3.5/glossary.html#term-decorator) -

> [decorator](https://docs.python.org/3/glossary.html#term-decorator) -

> decorators are covered in another practical.





<a class="anchor" id="the-dot-operator"></a>

## The dot operator `.`





Python allows us to override the `.` (dot) operator which is used to access

the attributes and methods of an object.  This is very powerful, but is also

quite a niche feature, and it is easy to trip yourself up, so if you wish to

use this in your own project, make sure that you carefully read (and

understand) [the

documentation](https://docs.python.org/3.5/reference/datamodel.html#customizing-attribute-access),

documentation](https://docs.python.org/3/reference/datamodel.html#customizing-attribute-access),

and test your code comprehensively!





For this example, we need a little background information.  OpenGL includes

the native data types `vec2`, `vec3`, and `vec4`, which can be used to

represent 2, 3, or 4 component vectors respectively. These data types have a

neat feature called [_swizzling_][glslref], which allows you to access any

component (`x`,`y`, `z`, `w` for vectors, or `r`, `g`, `b`, `a` for colours)

in any order, with a syntax similar to attribute access in Python.





[glslref]: https://www.khronos.org/opengl/wiki/Data_Type_(GLSL)#Swizzling





So here is an example which implements this swizzle-style attribute access on

a class called `Vector`, in which we have customised the behaviour of the `.`

operator:



%% Cell type:code id: tags:



``` 

class Vector(object):

    def __init__(self, xyz):

        self.__xyz = list(xyz)



    def __str__(self):

        return 'Vector({})'.format(self.__xyz)



    def __getattr__(self, key):



        # Swizzling behaviour only occurs when

        # the attribute name is entirely comprised

        # of 'x', 'y', and 'z'.

        if not all([c in 'xyz' for c in key]):

            raise AttributeError(key)



        key = ['xyz'.index(c) for c in key]

        return [self.__xyz[c] for c in key]



    def __setattr__(self, key, value):



        # Restrict swizzling behaviour as above

        if not all([c in 'xyz' for c in key]):

            return super().__setattr__(key, value)



        if len(key) == 1:

            value = (value,)



        idxs = ['xyz'.index(c) for c in key]



        for i, v in sorted(zip(idxs, value)):

            self.__xyz[i] = v

```



%% Cell type:markdown id: tags:



And here it is in action:



%% Cell type:code id: tags:



``` 

v = Vector((1, 2, 3))



print('v:   ', v)

print('xyz: ', v.xyz)

print('zy:  ', v.zy)

print('xx:  ', v.xx)



v.xz = 10, 30

print(v)

v.y = 20

print(v)

```

advanced_topics/04_operator_overloading.md

+14 −14
Original line number Diff line number Diff line
@@ -8,13 +8,13 @@ 
Operator overloading, in an object-oriented programming language, is the

process of customising the behaviour of _operators_ (e.g. `+`, `*`, `/` and

`-`) on user-defined types. This practical aims to show you that operator

overloading is __very__ easy to do in Python.

overloading is **very** easy to do in Python.





This practical gives a brief overview of the operators which you may be most

interested in implementing. However, there are many operators (and other

special methods) which you can support in your own classes - the [official

documentation](https://docs.python.org/3.5/reference/datamodel.html#basic-customization)

documentation](https://docs.python.org/3/reference/datamodel.html#basic-customization)

is the best reference if you are interested in learning more.
@@ -173,7 +173,7 @@ rules are followed depending on the set of methods implemented on the 
operands. For example, in the expression `a + b`, if `a.__add__` is not

implemented, but but `b.__radd__` is implemented, then the latter will be

called.  Take a look at the [official

documentation](https://docs.python.org/3.5/reference/datamodel.html#emulating-numeric-types)

documentation](https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types)

for further details, including a full list of the arithmetic and logical

operators that your classes can support.
@@ -252,9 +252,9 @@ print(sorted((l3, l1, l2))) 




The

[`@functools.total_ordering`](https://docs.python.org/3.5/library/functools.html#functools.total_ordering)

[`@functools.total_ordering`](https://docs.python.org/3/library/functools.html#functools.total_ordering)

is a convenience

[decorator](https://docs.python.org/3.5/glossary.html#term-decorator) which,

[decorator](https://docs.python.org/3/glossary.html#term-decorator) which,

given a class that implements equality and a single comparison function

(`__lt__` in the above code), will "fill in" the remainder of the comparison

operators.  If you need very specific or complicated behaviour, then you can
@@ -271,7 +271,7 @@ and just one of `__lt__`, `__le__`, `__gt__` or `__ge__`. 




Refer to the [official

documentation](https://docs.python.org/3.5/reference/datamodel.html#object.__lt__)

documentation](https://docs.python.org/3/reference/datamodel.html#object.__lt__)

for all of the details on supporting comparison operators.
@@ -293,13 +293,13 @@ At its essence, there are only three types of behaviours that are possible 
with the `[]` operator. All that is needed to support them are to implement

three special methods in your class, regardless of whether your class will be

indexed by sequential integers (like a `list`) or by

[hashable](https://docs.python.org/3.5/glossary.html#term-hashable) values

[hashable](https://docs.python.org/3/glossary.html#term-hashable) values

(like a `dict`):





- __Retrieval__ is performed by the `__getitem__` method

- __Assignment__ is performed by the `__setitem__` method

- __Deletion__ is performed by the `__delitem__` method

- **Retrieval** is performed by the `__getitem__` method

- **Assignment** is performed by the `__setitem__` method

- **Deletion** is performed by the `__delitem__` method





Note that, if you implement these methods in your own class, there is no
@@ -370,7 +370,7 @@ print(tt['12345']) 




If you wish to support the Python `start:stop:step` [slice

notation](https://docs.python.org/3.5/library/functions.html#slice), you

notation](https://docs.python.org/3/library/functions.html#slice), you

simply need to write your `__getitem__` and `__setitem__` methods so that they

can detect `slice` objects:
@@ -414,7 +414,7 @@ print(tt[::2]) 
> different hashing algorithm), the `Sequence` and `MutableMapping` classes

> are [a better choice](https://stackoverflow.com/a/7148602) - you can find

> them in the

> [`collections.abc`](https://docs.python.org/3.5/library/collections.abc.html)

> [`collections.abc`](https://docs.python.org/3/library/collections.abc.html)

> module.
@@ -472,7 +472,7 @@ inv = tf(data) 




> The `TimedFunction` class is conceptually very similar to a

> [decorator](https://docs.python.org/3.5/glossary.html#term-decorator) -

> [decorator](https://docs.python.org/3/glossary.html#term-decorator) -

> decorators are covered in another practical.
@@ -485,7 +485,7 @@ the attributes and methods of an object. This is very powerful, but is also 
quite a niche feature, and it is easy to trip yourself up, so if you wish to

use this in your own project, make sure that you carefully read (and

understand) [the

documentation](https://docs.python.org/3.5/reference/datamodel.html#customizing-attribute-access),

documentation](https://docs.python.org/3/reference/datamodel.html#customizing-attribute-access),

and test your code comprehensively!

advanced_topics/05_context_managers.ipynb

+25 −17
Original line number Diff line number Diff line 
%% Cell type:markdown id: tags:



# Context managers





The recommended way to open a file in Python is via the `with` statement:



%% Cell type:code id: tags:



``` 

with open('05_context_managers.md', 'rt') as f:

    firstlines = f.readlines()[:4]

    firstlines = [l.strip() for l in firstlines]

    print('\n'.join(firstlines))

```



%% Cell type:markdown id: tags:



This is because the `with` statement ensures that the file will be closed

automatically, even if an error occurs inside the `with` statement.





The `with` statement is obviously hiding some internal details from us. But

these internals are in fact quite straightforward, and are known as [_context

managers_](https://docs.python.org/3.5/reference/datamodel.html#context-managers).

these internals are in fact quite straightforward, and are known as [*context

managers*](https://docs.python.org/3/reference/datamodel.html#context-managers).





* [Anatomy of a context manager](#anatomy-of-a-context-manager)

 * [Why not just use `try ... finally`?](#why-not-just-use-try-finally)

* [Uses for context managers](#uses-for-context-managers)

 * [Handling errors in `__exit__`](#handling-errors-in-exit)

 * [Suppressing errors with `__exit__`](#suppressing-errors-with-exit)

* [Nesting context managers](#nesting-context-managers)

* [Functions as context managers](#functions-as-context-managers)

* [Methods as context managers](#methods-as-context-managers)

* [Useful references](#useful-references)





<a class="anchor" id="anatomy-of-a-context-manager"></a>

## Anatomy of a context manager





A _context manager_ is simply an object which has two specially named methods

A *context manager* is simply an object which has two specially named methods

`__enter__` and `__exit__`. Any object which has these methods can be used in

a `with` statement.





Let's define a context manager class that we can play with:



%% Cell type:code id: tags:



``` 

class MyContextManager(object):

    def __enter__(self):

        print('In enter')

    def __exit__(self, *args):

        print('In exit')

```



%% Cell type:markdown id: tags:



Now, what happens when we use `MyContextManager` in a `with` statement?



%% Cell type:code id: tags:



``` 

with MyContextManager():

    print('In with block')

```



%% Cell type:markdown id: tags:



So the `__enter__` method is called before the statements in the `with` block,

and the `__exit__` method is called afterwards.





Context managers are that simple. What makes them really useful though, is

that the `__exit__` method will be called even if the code in the `with` block

raises an error. The error will be held, and only raised after the `__exit__`

method has finished:



%% Cell type:code id: tags:



``` 

with MyContextManager():

    print('In with block')

    assert 1 == 0

```



%% Cell type:markdown id: tags:



This means that we can use context managers to perform any sort of clean up or

finalisation logic that we always want to have executed.





<a class="anchor" id="why-not-just-use-try-finally"></a>

### Why not just use `try ... finally`?





Context managers do not provide anything that cannot be accomplished in other

ways.  For example, we could accomplish very similar behaviour using

[`try` - `finally` logic](https://docs.python.org/3.5/tutorial/errors.html#handling-exceptions) -

[`try` - `finally` logic](https://docs.python.org/3/tutorial/errors.html#handling-exceptions) -

the statements in the `finally` clause will *always* be executed, whether an

error is raised or not:



%% Cell type:code id: tags:



``` 

print('Before try block')

try:

    print('In try block')

    assert 1 == 0

finally:

    print('In finally block')

```



%% Cell type:markdown id: tags:



But context managers have the advantage that you can implement your clean-up

logic in one place, and re-use it as many times as you want.





<a class="anchor" id="uses-for-context-managers"></a>

## Uses for context managers





We have already talked about how context managers can be used to perform any

task which requires some initialistion and/or clean-up logic. As an example,

here is a context manager which creates a temporary directory, and then makes

sure that it is deleted afterwards.



%% Cell type:code id: tags:



``` 

import os

import shutil

import tempfile



class TempDir(object):



    def __enter__(self):



        self.tempDir = tempfile.mkdtemp()

        self.prevDir = os.getcwd()



        print('Changing to temp dir: {}'.format(self.tempDir))

        print('Previous directory:   {}'.format(self.prevDir))



        os.chdir(self.tempDir)



    def __exit__(self, *args):



        print('Changing back to:  {}'.format(self.prevDir))

        print('Removing temp dir: {}'.format(self.tempDir))



        os    .chdir( self.prevDir)

        shutil.rmtree(self.tempDir)

```



%% Cell type:markdown id: tags:



Now imagine that we have a function which loads data from a file, and performs

some calculation on it:



%% Cell type:code id: tags:



``` 

import numpy as np



def complexAlgorithm(infile):

    data = np.loadtxt(infile)

    return data.mean()

```



%% Cell type:markdown id: tags:



We could use the `TempDir` context manager to write a test case for this

function,  and not have to worry about cleaning up the test data:



%% Cell type:code id: tags:



``` 

with TempDir():

    print('Testing complex algorithm')



    data = np.random.random((100, 100))

    np.savetxt('data.txt', data)



    result = complexAlgorithm('data.txt')



    assert result > 0.1 and result < 0.9

    print('Test passed (result: {})'.format(result))

```



%% Cell type:markdown id: tags:



<a class="anchor" id="handling-errors-in-exit"></a>

### Handling errors in `__exit__`





By now you must be [panicking](https://youtu.be/cSU_5MgtDc8?t=9) about why I

haven't mentioned those conspicuous `*args` that get passed to the`__exit__`

method.  It turns out that a context manager's [`__exit__`

method](https://docs.python.org/3.5/reference/datamodel.html#object.__exit__)

method](https://docs.python.org/3/reference/datamodel.html#object.__exit__)

is always passed three arguments.





Let's adjust our `MyContextManager` class a little so we can see what these

arguments are for:



%% Cell type:code id: tags:



``` 

class MyContextManager(object):

    def __enter__(self):

        print('In enter')



    def __exit__(self, arg1, arg2, arg3):

        print('In exit')

        print('  arg1: ', arg1)

        print('  arg2: ', arg2)

        print('  arg3: ', arg3)

```



%% Cell type:markdown id: tags:



If the code inside the `with` statement does not raise an error, these three

arguments will all be `None`.



%% Cell type:code id: tags:



``` 

with MyContextManager():

    print('In with block')

```



%% Cell type:markdown id: tags:



However, if the code inside the `with` statement raises an error, things look

a little different:



%% Cell type:code id: tags:



``` 

with MyContextManager():

    print('In with block')

    raise ValueError('Oh no!')

```



%% Cell type:markdown id: tags:



So when an error occurs, the `__exit__` method is passed the following:



- The [`Exception`](https://docs.python.org/3.5/tutorial/errors.html)

- The [`Exception`](https://docs.python.org/3/tutorial/errors.html)

  type that was raised.

- The `Exception` instance that was raised.

- A [`traceback`](https://docs.python.org/3.5/library/traceback.html) object

- A [`traceback`](https://docs.python.org/3/library/traceback.html) object

  which can be used to get more information about the exception (e.g. line

  number).





<a class="anchor" id="suppressing-errors-with-exit"></a>

### Suppressing errors with `__exit__`





The `__exit__` method is also capable of suppressing errors - if it returns a

value of `True`, then any error that was raised will be ignored. For example,

we could write a context manager which ignores any assertion errors, but

allows other errors to halt execution as normal:



%% Cell type:code id: tags:



``` 

class MyContextManager(object):

    def __enter__(self):

        print('In enter')



    def __exit__(self, arg1, arg2, arg3):

        print('In exit')

        if issubclass(arg1, AssertionError):

            return True

        print('  arg1: ', arg1)

        print('  arg2: ', arg2)

        print('  arg3: ', arg3)

```



%% Cell type:markdown id: tags:



> Note that if a function or method does not explicitly return a value, its

> return value is `None` (which would evaluate to `False` when converted to a

> `bool`).  Also note that we are using the built-in

> [`issubclass`](https://docs.python.org/3.5/library/functions.html#issubclass)

> [`issubclass`](https://docs.python.org/3/library/functions.html#issubclass)

> function, which allows us to test the type of a class.





Now, when we use `MyContextManager`, any assertion errors are suppressed,

whereas other errors will be raised as normal:



%% Cell type:code id: tags:



``` 

with MyContextManager():

    assert 1 == 0



print('Continuing execution!')



with MyContextManager():

    raise ValueError('Oh no!')

```



%% Cell type:markdown id: tags:



<a class="anchor" id="nesting-context-managers"></a>

## Nesting context managers





It is possible to nest `with` statements:



%% Cell type:code id: tags:



``` 

with open('05_context_managers.md', 'rt') as inf:

    with TempDir():

        with open('05_context_managers.md.copy', 'wt') as outf:

            outf.write(inf.read())

```



%% Cell type:markdown id: tags:



You can also use multiple context managers in a single `with` statement:



%% Cell type:code id: tags:



``` 

with open('05_context_managers.md', 'rt') as inf, \

     TempDir(), \

     open('05_context_managers.md.copy', 'wt') as outf:

    outf.write(inf.read())

```



%% Cell type:markdown id: tags:



<a class="anchor" id="functions-as-context-managers"></a>

## Functions as context managers





In fact, there is another way to create context managers in Python. The

built-in [`contextlib`

module](https://docs.python.org/3.5/library/contextlib.html#contextlib.contextmanager)

module](https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager)

has a decorator called `@contextmanager`, which allows us to turn __any

function__ into a context manager.  The only requirement is that the function

must have a `yield` statement<sup>1</sup>. So we could rewrite our `TempDir`

class from above as a function:



%% Cell type:code id: tags:



``` 

import os

import shutil

import tempfile

import contextlib



@contextlib.contextmanager

def tempdir():

    tdir    = tempfile.mkdtemp()

    prevdir = os.getcwd()

    try:



        os.chdir(tdir)

        yield tdir



    finally:

        os.chdir(prevdir)

        shutil.rmtree(tdir)

```



%% Cell type:markdown id: tags:



This new `tempdir` function is used in exactly the same way as our `TempDir`

class:



%% Cell type:code id: tags:



``` 

print('In directory:      {}'.format(os.getcwd()))



with tempdir() as tmp:

    print('Now in directory:  {}'.format(os.getcwd()))



print('Back in directory: {}'.format(os.getcwd()))

```



%% Cell type:markdown id: tags:



The `yield tdir` statement in our `tempdir` function causes the `tdir` value

to be passed to the `with` statement, so in the line `with tempdir() as tmp`,

the variable `tmp` will be given the value `tdir`.





> <sup>1</sup> The `yield` keyword is used in _generator functions_.

> <sup>1</sup> The `yield` keyword is used in *generator functions*.

> Functions which are used with the `@contextmanager` decorator must be

> generator functions which yield exactly one value.

> [Generators](https://www.python.org/dev/peps/pep-0289/) and [generator

> functions](https://docs.python.org/3.5/glossary.html#term-generator) are

> functions](https://docs.python.org/3/glossary.html#term-generator) are

> beyond the scope of this practical.





<a class="anchor" id="methods-as-context-managers"></a>

## Methods as context managers





Since it is possible to write a function which is a context manager, it is of

course also possible to write a _method_ which is a context manager. Let's

play with another example. We have a `Notifier` class which can be used to

notify interested listeners when an event occurs. Listeners can be registered

for notification via the `register` method:



%% Cell type:code id: tags:



``` 

from collections import OrderedDict



class Notifier(object):

    def __init__(self):

        super().__init__()

        self.listeners = OrderedDict()



    def register(self, name, func):

        self.listeners[name] = func



    def notify(self):

        for listener in self.listeners.values():

            listener()

```



%% Cell type:markdown id: tags:



Now, let's build a little plotting application. First of all, we have a `Line`

class, which represents a line plot. The `Line` class is a sub-class of

`Notifier`, so whenever its display properties (`colour`, `width`, or `name`)

change, it emits a notification, and whatever is drawing it can refresh the

display:



%% Cell type:code id: tags:



``` 

import numpy as np



class Line(Notifier):



    def __init__(self, data):

        super().__init__()

        self.__data   = data

        self.__colour = '#000000'

        self.__width  = 1

        self.__name   = 'line'



    @property

    def xdata(self):

        return np.arange(len(self.__data))



    @property

    def ydata(self):

        return np.copy(self.__data)



    @property

    def colour(self):

        return self.__colour



    @colour.setter

    def colour(self, newColour):

        self.__colour = newColour

        print('Line: colour changed: {}'.format(newColour))

        self.notify()



    @property

    def width(self):

        return self.__width



    @width.setter

    def width(self, newWidth):

        self.__width = newWidth

        print('Line: width changed: {}'.format(newWidth))

        self.notify()



    @property

    def name(self):

        return self.__name



    @name.setter

    def name(self, newName):

        self.__name = newName

        print('Line: name changed: {}'.format(newName))

        self.notify()

```



%% Cell type:markdown id: tags:



Now let's write a `Plotter` class, which can plot one or more `Line`

instances:



%% Cell type:code id: tags:



``` 

import matplotlib.pyplot as plt



class Plotter(object):

    def __init__(self, axis):

        self.__axis   = axis

        self.__lines  = []



    def addData(self, data):

        line = Line(data)

        self.__lines.append(line)

        line.register('plot', self.lineChanged)

        self.draw()

        return line



    def lineChanged(self):

        self.draw()



    def draw(self):

        print('Plotter: redrawing plot')



        ax = self.__axis

        ax.clear()

        for line in self.__lines:

            ax.plot(line.xdata,

                    line.ydata,

                    color=line.colour,

                    linewidth=line.width,

                    label=line.name)

        ax.legend()

```



%% Cell type:markdown id: tags:



Let's create a `Plotter` object, and add a couple of lines to it (note that

the `matplotlib` plot will open in a separate window):



%% Cell type:code id: tags:



``` 

# this line is only necessary when

# working in jupyer notebook/ipython

%matplotlib



fig     = plt.figure()

ax      = fig.add_subplot(111)

plotter = Plotter(ax)

l1      = plotter.addData(np.sin(np.linspace(0, 6 * np.pi, 50)))

l2      = plotter.addData(np.cos(np.linspace(0, 6 * np.pi, 50)))



fig.show()

```



%% Cell type:markdown id: tags:



Now, when we change the properties of our `Line` instances, the plot will be

automatically updated:



%% Cell type:code id: tags:



``` 

l1.colour = '#ff0000'

l2.colour = '#00ff00'

l1.width  = 2

l2.width  = 2

l1.name   = 'sine'

l2.name   = 'cosine'

```



%% Cell type:markdown id: tags:



Pretty cool! However, this seems very inefficient - every time we change the

properties of a `Line`, the `Plotter` will refresh the plot. If we were

plotting large amounts of data, this would be unacceptable, as plotting would

simply take too long.





Wouldn't it be nice if we were able to perform batch-updates of `Line`

properties, and only refresh the plot when we are done? Let's add an extra

method to the `Plotter` class:



%% Cell type:code id: tags:



``` 

import contextlib



class Plotter(object):

    def __init__(self, axis):

        self.__axis        = axis

        self.__lines       = []

        self.__holdUpdates = False



    def addData(self, data):

        line = Line(data)

        self.__lines.append(line)

        line.register('plot', self.lineChanged)



        if not self.__holdUpdates:

            self.draw()

        return line



    def lineChanged(self):

        if not self.__holdUpdates:

            self.draw()



    def draw(self):

        print('Plotter: redrawing plot')



        ax = self.__axis

        ax.clear()

        for line in self.__lines:

            ax.plot(line.xdata,

                    line.ydata,

                    color=line.colour,

                    linewidth=line.width,

                    label=line.name)

        ax.legend()



    @contextlib.contextmanager

    def holdUpdates(self):

        self.__holdUpdates = True

        try:

            yield

            self.draw()

        finally:

            self.__holdUpdates = False

```



%% Cell type:markdown id: tags:



This new `holdUpdates` method allows us to temporarily suppress notifications

from all `Line` instances. So now, we can update many `Line` properties

without performing any redundant redraws:

from all `Line` instances.  Let's create a new plot:



%% Cell type:code id: tags:



``` 

fig     = plt.figure()

ax      = fig.add_subplot(111)

plotter = Plotter(ax)

l1      = plotter.addData(np.sin(np.linspace(0, 6 * np.pi, 50)))

l2      = plotter.addData(np.cos(np.linspace(0, 6 * np.pi, 50)))



plt.show()

```



%% Cell type:markdown id: tags:



Now, we can update many `Line` properties without performing any redundant

redraws:



%% Cell type:code id: tags:



``` 

with plotter.holdUpdates():

    l1        = plotter.addData(np.sin(np.linspace(0, 6 * np.pi, 50)))

    l2        = plotter.addData(np.cos(np.linspace(0, 6 * np.pi, 50)))

    l1.colour = '#0000ff'

    l2.colour = '#ffff00'

    l1.width  = 1

    l2.width  = 1

    l1.name   = '$sin(x)$'

    l2.name   = '$cos(x)$'

```



%% Cell type:markdown id: tags:



<a class="anchor" id="useful-references"></a>

## Useful references





* [Context manager classes](https://docs.python.org/3.5/reference/datamodel.html#context-managers)

* The [`contextlib` module](https://docs.python.org/3.5/library/contextlib.html)
 
* [Context manager classes](https://docs.python.org/3/reference/datamodel.html#context-managers)

* The [`contextlib` module](https://docs.python.org/3/library/contextlib.html)

advanced_topics/05_context_managers.md

+21 −17
Original line number Diff line number Diff line
@@ -17,8 +17,8 @@ automatically, even if an error occurs inside the `with` statement. 




The `with` statement is obviously hiding some internal details from us. But

these internals are in fact quite straightforward, and are known as [_context

managers_](https://docs.python.org/3.5/reference/datamodel.html#context-managers).

these internals are in fact quite straightforward, and are known as [*context

managers*](https://docs.python.org/3/reference/datamodel.html#context-managers).





* [Anatomy of a context manager](#anatomy-of-a-context-manager)
@@ -36,7 +36,7 @@ managers_](https://docs.python.org/3.5/reference/datamodel.html#context-managers 
## Anatomy of a context manager





A _context manager_ is simply an object which has two specially named methods

A *context manager* is simply an object which has two specially named methods

`__enter__` and `__exit__`. Any object which has these methods can be used in

a `with` statement.
@@ -89,7 +89,7 @@ finalisation logic that we always want to have executed. 


Context managers do not provide anything that cannot be accomplished in other

ways.  For example, we could accomplish very similar behaviour using

[`try` - `finally` logic](https://docs.python.org/3.5/tutorial/errors.html#handling-exceptions) -

[`try` - `finally` logic](https://docs.python.org/3/tutorial/errors.html#handling-exceptions) -

the statements in the `finally` clause will *always* be executed, whether an

error is raised or not:
@@ -183,7 +183,7 @@ with TempDir(): 
By now you must be [panicking](https://youtu.be/cSU_5MgtDc8?t=9) about why I

haven't mentioned those conspicuous `*args` that get passed to the`__exit__`

method.  It turns out that a context manager's [`__exit__`

method](https://docs.python.org/3.5/reference/datamodel.html#object.__exit__)

method](https://docs.python.org/3/reference/datamodel.html#object.__exit__)

is always passed three arguments.
@@ -227,10 +227,10 @@ with MyContextManager(): 


So when an error occurs, the `__exit__` method is passed the following:



- The [`Exception`](https://docs.python.org/3.5/tutorial/errors.html)

- The [`Exception`](https://docs.python.org/3/tutorial/errors.html)

  type that was raised.

- The `Exception` instance that was raised.

- A [`traceback`](https://docs.python.org/3.5/library/traceback.html) object

- A [`traceback`](https://docs.python.org/3/library/traceback.html) object

  which can be used to get more information about the exception (e.g. line

  number).
@@ -262,7 +262,7 @@ class MyContextManager(object): 
> Note that if a function or method does not explicitly return a value, its

> return value is `None` (which would evaluate to `False` when converted to a

> `bool`).  Also note that we are using the built-in

> [`issubclass`](https://docs.python.org/3.5/library/functions.html#issubclass)

> [`issubclass`](https://docs.python.org/3/library/functions.html#issubclass)

> function, which allows us to test the type of a class.
@@ -312,7 +312,7 @@ with open('05_context_managers.md', 'rt') as inf, \ 


In fact, there is another way to create context managers in Python. The

built-in [`contextlib`

module](https://docs.python.org/3.5/library/contextlib.html#contextlib.contextmanager)

module](https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager)

has a decorator called `@contextmanager`, which allows us to turn __any

function__ into a context manager.  The only requirement is that the function

must have a `yield` statement<sup>1</sup>. So we could rewrite our `TempDir`
@@ -359,11 +359,11 @@ to be passed to the `with` statement, so in the line `with tempdir() as tmp`, 
the variable `tmp` will be given the value `tdir`.





> <sup>1</sup> The `yield` keyword is used in _generator functions_.

> <sup>1</sup> The `yield` keyword is used in *generator functions*.

> Functions which are used with the `@contextmanager` decorator must be

> generator functions which yield exactly one value.

> [Generators](https://www.python.org/dev/peps/pep-0289/) and [generator

> functions](https://docs.python.org/3.5/glossary.html#term-generator) are

> functions](https://docs.python.org/3/glossary.html#term-generator) are

> beyond the scope of this practical.
@@ -582,20 +582,24 @@ class Plotter(object): 




This new `holdUpdates` method allows us to temporarily suppress notifications

from all `Line` instances. So now, we can update many `Line` properties

without performing any redundant redraws:

from all `Line` instances.  Let's create a new plot:





```

fig     = plt.figure()

ax      = fig.add_subplot(111)

plotter = Plotter(ax)

l1      = plotter.addData(np.sin(np.linspace(0, 6 * np.pi, 50)))

l2      = plotter.addData(np.cos(np.linspace(0, 6 * np.pi, 50)))



plt.show()

```



Now, we can update many `Line` properties without performing any redundant

redraws:



```

with plotter.holdUpdates():

    l1        = plotter.addData(np.sin(np.linspace(0, 6 * np.pi, 50)))

    l2        = plotter.addData(np.cos(np.linspace(0, 6 * np.pi, 50)))

    l1.colour = '#0000ff'

    l2.colour = '#ffff00'

    l1.width  = 1
@@ -609,5 +613,5 @@ with plotter.holdUpdates(): 
## Useful references





* [Context manager classes](https://docs.python.org/3.5/reference/datamodel.html#context-managers)

* The [`contextlib` module](https://docs.python.org/3.5/library/contextlib.html)
 
* [Context manager classes](https://docs.python.org/3/reference/datamodel.html#context-managers)

* The [`contextlib` module](https://docs.python.org/3/library/contextlib.html)

advanced_topics/06_decorators.ipynb

+23 −21
Original line number Diff line number Diff line 
%% Cell type:markdown id: tags:



# Decorators





Remember that in Python, everything is an object, including functions. This

means that we can do things like:





- Pass a function as an argument to another function.

- Create/define a function inside another function.

- Write a function which returns another function.





These abilities mean that we can do some neat things with functions in Python.





* [Overview](#overview)

* [Decorators on methods](#decorators-on-methods)

* [Example - memoization](#example-memoization)

* [Decorators with arguments](#decorators-with-arguments)

* [Chaining decorators](#chaining-decorators)

* [Decorator classes](#decorator-classes)

* [Appendix: Functions are not special](#appendix-functions-are-not-special)

* [Appendix: Closures](#appendix-closures)

* [Appendix: Decorators without arguments versus decorators with arguments](#appendix-decorators-without-arguments-versus-decorators-with-arguments)

* [Appendix: Per-instance decorators](#appendix-per-instance-decorators)

* [Appendix: Preserving function metadata](#appendix-preserving-function-metadata)

* [Appendix: Class decorators](#appendix-class-decorators)

* [Useful references](#useful-references)





<a class="anchor" id="overview"></a>

## Overview





Let's say that we want a way to calculate the execution time of any function

(this example might feel familiar to you if you have gone through the

practical on operator overloading).





Our first attempt at writing such a function might look like this:



%% Cell type:code id: tags:



``` 

import time

def timeFunc(func, *args, **kwargs):



    start  = time.time()

    retval = func(*args, **kwargs)

    end    = time.time()



    print('Ran {} in {:0.2f} seconds'.format(func.__name__, end - start))



    return retval

```



%% Cell type:markdown id: tags:



The `timeFunc` function accepts another function, `func`, as its first

argument. It calls `func`, passing it all of the other arguments, and then

prints the time taken for `func` to complete:



%% Cell type:code id: tags:



``` 

import numpy        as np

import numpy.linalg as npla



def inverse(a):

    return npla.inv(a)



data    = np.random.random((2000, 2000))

invdata = timeFunc(inverse, data)

```



%% Cell type:markdown id: tags:



But this means that whenever we want to time something, we have to call the

`timeFunc` function directly. Let's take advantage of the fact that we can

define a function inside another funciton. Look at the next block of code

carefully, and make sure you understand what our new `timeFunc` implementation

is doing.



%% Cell type:code id: tags:



``` 

import time

def timeFunc(func):



    def wrapperFunc(*args, **kwargs):



        start  = time.time()

        retval = func(*args, **kwargs)

        end    = time.time()



        print('Ran {} in {:0.2f} seconds'.format(func.__name__, end - start))



        return retval



    return wrapperFunc

```



%% Cell type:markdown id: tags:



This new `timeFunc` function is again passed a function `func`, but this time

as its sole argument. It then creates and returns a new function,

`wrapperFunc`. This `wrapperFunc` function calls and times the function that

was passed to `timeFunc`.  But note that when `timeFunc` is called,

`wrapperFunc` is _not_ called - it is only created and returned.

`wrapperFunc` is *not* called - it is only created and returned.





Let's use our new `timeFunc` implementation:



%% Cell type:code id: tags:



``` 

import numpy        as np

import numpy.linalg as npla



def inverse(a):

    return npla.inv(a)



data    = np.random.random((2000, 2000))

inverse = timeFunc(inverse)

invdata = inverse(data)

```



%% Cell type:markdown id: tags:



Here, we did the following:





1. We defined a function called `inverse`:



  > ```

  > def inverse(a):

  >     return npla.inv(a)

  > ```



2. We passed the `inverse` function to the `timeFunc` function, and

   re-assigned the return value of `timeFunc` back to `inverse`:



  > ```

  > inverse = timeFunc(inverse)

  > ```



3. We called the new `inverse` function:



  > ```

  > invdata = inverse(data)

  > ```





So now the `inverse` variable refers to an instantiation of `wrapperFunc`,

which holds a reference to the original definition of `inverse`.





> If this is not clear, take a break now and read through the appendix on how

> [functions are not special](#appendix-functions-are-not-special).





Guess what? We have just created a __decorator__. A decorator is simply a

Guess what? We have just created a **decorator**. A decorator is simply a

function which accepts a function as its input, and returns another function

as its output. In the example above, we have _decorated_ the `inverse`

as its output. In the example above, we have *decorated* the `inverse`

function with the `timeFunc` decorator.





Python provides an alternative syntax for decorating one function with

another, using the `@` character. The approach that we used to decorate

`inverse` above:



%% Cell type:code id: tags:



``` 

def inverse(a):

    return npla.inv(a)



inverse = timeFunc(inverse)

invdata = inverse(data)

```



%% Cell type:markdown id: tags:



is semantically equivalent to this:



%% Cell type:code id: tags:



``` 

@timeFunc

def inverse(a):

    return npla.inv(a)



invdata = inverse(data)

```



%% Cell type:markdown id: tags:



<a class="anchor" id="decorators-on-methods"></a>

## Decorators on methods





Applying a decorator to the methods of a class works in the same way:



%% Cell type:code id: tags:



``` 

import numpy.linalg as npla



class MiscMaths(object):



    @timeFunc

    def inverse(self, a):

        return npla.inv(a)

```



%% Cell type:markdown id: tags:



Now, the `inverse` method of all `MiscMaths` instances will be timed:



%% Cell type:code id: tags:



``` 

mm1 = MiscMaths()

mm2 = MiscMaths()



i1 = mm1.inverse(np.random.random((1000, 1000)))

i2 = mm2.inverse(np.random.random((1500, 1500)))

```



%% Cell type:markdown id: tags:



Note that only one `timeFunc` decorator was created here - the `timeFunc`

function was only called once - when the `MiscMaths` class was defined.  This

might be clearer if we re-write the above code in the following (equivalent)

manner:



%% Cell type:code id: tags:



``` 

class MiscMaths(object):

    def inverse(self, a):

        return npla.inv(a)



MiscMaths.inverse = timeFunc(MiscMaths.inverse)

```



%% Cell type:markdown id: tags:



So only one `wrapperFunc` function exists, and this function is _shared_ by

So only one `wrapperFunc` function exists, and this function is *shared* by

all instances of the `MiscMaths` class - (such as the `mm1` and `mm2`

instances in the example above). In many cases this is not a problem, but

there can be situations where you need each instance of your class to have its

own unique decorator.





> If you are interested in solutions to this problem, take a look at the

> appendix on [per-instance decorators](#appendix-per-instance-decorators).





<a class="anchor" id="example-memoization"></a>

## Example - memoization





Let's move onto another example.

[Meowmoization](https://en.wikipedia.org/wiki/Memoization) is a common

performance optimisation technique used in cats. I mean software. Essentially,

memoization refers to the process of maintaining a cache for a function which

performs some expensive calculation. When the function is executed with a set

of inputs, the calculation is performed, and then a copy of the inputs and the

result are cached. If the function is called again with the same inputs, the

cached result can be returned.





This is a perfect problem to tackle with decorators:



%% Cell type:code id: tags:



``` 

def memoize(func):



    cache = {}



    def wrapper(*args):



        # is there a value in the cache

        # for this set of inputs?

        cached = cache.get(args, None)



        # If not, call the function,

        # and cache the result.

        if cached is None:

            cached      = func(*args)

            cache[args] = cached

        else:

            print('Cached {}({}): {}'.format(func.__name__, args, cached))



        return cached



    return wrapper

```



%% Cell type:markdown id: tags:



We can now use our `memoize` decorator to add a memoization cache to any

function.  Let's memoize a function which generates the $n^{th}$ number in the

[Fibonacci series](https://en.wikipedia.org/wiki/Fibonacci_number):



%% Cell type:code id: tags:



``` 

@memoize

def fib(n):



    if n in (0, 1):

        print('fib({}) = {}'.format(n, n))

        return n



    twoback = 1

    oneback = 1

    val     = 1



    for _ in range(2, n):



        val     = oneback + twoback

        twoback = oneback

        oneback = val



    print('fib({}) = {}'.format(n, val))



    return val

```



%% Cell type:markdown id: tags:



For a given input, when `fib` is called the first time, it will calculate the

$n^{th}$ Fibonacci number:



%% Cell type:code id: tags:



``` 

for i in range(10):

    fib(i)

```



%% Cell type:markdown id: tags:



However, on repeated calls with the same input, the calculation is skipped,

and instead the result is retrieved from the memoization cache:



%% Cell type:code id: tags:



``` 

for i in range(10):

    fib(i)

```



%% Cell type:markdown id: tags:



> If you are wondering how the `wrapper` function is able to access the

> `cache` variable, refer to the [appendix on closures](#appendix-closures).





<a class="anchor" id="decorators-with-arguments"></a>

## Decorators with arguments





Continuing with our memoization example, let's say that we want to place a

limit on the maximum size that our cache can grow to. For example, the output

of our function might have large memory requirements, so we can only afford to

store a handful of pre-calculated results. It would be nice to be able to

specify the maximum cache size when we define our function to be memoized,

like so:





> ```

> # cache at most 10 results

> @limitedMemoize(10):

> def fib(n):

>     ...

> ```





In order to support this, our `memoize` decorator function needs to be

modified - it is currently written to accept a function as its sole argument,

but we need it to accept a cache size limit.



%% Cell type:code id: tags:



``` 

from collections import OrderedDict



def limitedMemoize(maxSize):



    cache = OrderedDict()



    def decorator(func):

        def wrapper(*args):



            # is there a value in the cache

            # for this set of inputs?

            cached = cache.get(args, None)



            # If not, call the function,

            # and cache the result.

            if cached is None:



                cached = func(*args)



                # If the cache has grown too big,

                # remove the oldest item. In practice

                # it would make more sense to remove

                # the item with the oldest access

                # time, but this is good enough for

                # an introduction!

                # time (or remove the least recently

                # used item, as the built-in

                # @functools.lru_cache does), but this

                # is good enough for now!

                if len(cache) >= maxSize:

                    cache.popitem(last=False)



                cache[args] = cached

            else:

                print('Cached {}({}): {}'.format(func.__name__, args, cached))



            return cached

        return wrapper

    return decorator

```



%% Cell type:markdown id: tags:



> We used the handy

> [`collections.OrderedDict`](https://docs.python.org/3.5/library/collections.html#collections.OrderedDict)

> [`collections.OrderedDict`](https://docs.python.org/3/library/collections.html#collections.OrderedDict)

> class here which preserves the insertion order of key-value pairs.





This is starting to look a little complicated - we now have _three_ layers of

This is starting to look a little complicated - we now have *three* layers of

functions. This is necessary when you wish to write a decorator which accepts

arguments (refer to the

[appendix](#appendix-decorators-without-arguments-versus-decorators-with-arguments)

for more details).





But this `limitedMemoize` decorator is used in essentially the same way as our

earlier `memoize` decorator:



%% Cell type:code id: tags:



``` 

@limitedMemoize(5)

def fib(n):



    if n in (0, 1):

        print('fib({}) = 1'.format(n))

        return n



    twoback = 1

    oneback = 1

    val     = 1



    for _ in range(2, n):



        val     = oneback + twoback

        twoback = oneback

        oneback = val



    print('fib({}) = {}'.format(n, val))



    return val

```



%% Cell type:markdown id: tags:



Except that now, the `fib` function will only cache up to 5 values.



%% Cell type:code id: tags:



``` 

fib(10)

fib(11)

fib(12)

fib(13)

fib(14)

print('The result for 10 should come from the cache')

fib(10)

fib(15)

print('The result for 10 should no longer be cached')

fib(10)

```



%% Cell type:markdown id: tags:



<a class="anchor" id="chaining-decorators"></a>

## Chaining decorators





Decorators can easily be chained, or nested:



%% Cell type:code id: tags:



``` 

import time



@timeFunc

@memoize

def expensiveFunc(n):

    time.sleep(n)

    return n

```



%% Cell type:markdown id: tags:



> Remember that this is semantically equivalent to the following:

>

> ```

> def expensiveFunc(n):

>     time.sleep(n)

>     return n

>

> expensiveFunc = timeFunc(memoize(expensiveFunc))

> ```





Now we can see the effect of our memoization layer on performance:



%% Cell type:code id: tags:



``` 

expensiveFunc(0.5)

expensiveFunc(1)

expensiveFunc(1)

```



%% Cell type:markdown id: tags:



> Note that in Python 3.2 and newer you can use the

> [`functools.lru_cache`](https://docs.python.org/3/library/functools.html#functools.lru_cache)

> to memoize your functions.





<a class="anchor" id="decorator-classes"></a>

## Decorator classes





By now, you will have gained the impression that a decorator is a function

which _decorates_ another function. But if you went through the practical on

which *decorates* another function. But if you went through the practical on

operator overloading, you might remember the special `__call__` method, that

allows an object to be called as if it were a function.





This feature allows us to write our decorators as classes, instead of

functions. This can be handy if you are writing a decorator that has

complicated behaviour, and/or needs to maintain some sort of state which

cannot be easily or elegantly written using nested functions.





As an example, let's say we are writing a framework for unit testing. We want

to be able to "mark" our test functions like so, so they can be easily

identified and executed:





> ```

> @unitTest

> def testblerk():

>     """tests the blerk algorithm."""

>     ...

> ```





With a decorator like this, we wouldn't need to worry about where our tests

are located - they will all be detected because we have marked them as test

functions. What does this `unitTest` decorator look like?



%% Cell type:code id: tags:



``` 

class TestRegistry(object):



    def __init__(self):

        self.testFuncs = []



    def __call__(self, func):

        self.testFuncs.append(func)



    def listTests(self):

        print('All registered tests:')

        for test in self.testFuncs:

            print(' ', test.__name__)



    def runTests(self):

        for test in self.testFuncs:

            print('Running test {:10s} ... '.format(test.__name__), end='')

            try:

                test()

                print('passed!')

            except Exception as e:

                print('failed!')



# Create our test registry

registry = TestRegistry()



# Alias our registry to "unitTest"

# so that we can register tests

# with a "@unitTest" decorator.

unitTest = registry

```



%% Cell type:markdown id: tags:



So we've defined a class, `TestRegistry`, and created an instance of it,

`registry`, which will manage all of our unit tests. Now, in order to "mark"

any function as being a unit test, we just need to use the `unitTest`

decorator (which is simply a reference to our `TestRegistry` instance):



%% Cell type:code id: tags:



``` 

@unitTest

def testFoo():

    assert 'a' in 'bcde'



@unitTest

def testBar():

    assert 1 > 0



@unitTest

def testBlerk():

    assert 9 % 2 == 0

```



%% Cell type:markdown id: tags:



Now that these functions have been registered with our `TestRegistry`

instance, we can run them all:



%% Cell type:code id: tags:



``` 

registry.listTests()

registry.runTests()

```



%% Cell type:markdown id: tags:



> Unit testing is something which you must do! This is __especially__

> Unit testing is something which you must do! This is **especially**

> important in an interpreted language such as Python, where there is no

> compiler to catch all of your mistakes.

>

> Python has a built-in

> [`unittest`](https://docs.python.org/3.5/library/unittest.html) module,

> [`unittest`](https://docs.python.org/3/library/unittest.html) module,

> however the third-party [`pytest`](https://docs.pytest.org/en/latest/) and

> [`nose`](http://nose2.readthedocs.io/en/latest/) are popular.  It is also

> wise to combine your unit tests with

> [`coverage`](https://coverage.readthedocs.io/en/coverage-4.5.1/), which

> tells you how much of your code was executed, or _covered_ when your

> tells you how much of your code was executed, or *covered* when your

> tests were run.





<a class="anchor" id="appendix-functions-are-not-special"></a>

## Appendix: Functions are not special





When we write a statement like this:



%% Cell type:code id: tags:



``` 

a = [1, 2, 3]

```



%% Cell type:markdown id: tags:



the variable `a` is a reference to a `list`. We can create a new reference to

the same list, and delete `a`:



%% Cell type:code id: tags:



``` 

b = a

del a

```



%% Cell type:markdown id: tags:



Deleting `a` doesn't affect the list at all - the list still exists, and is

now referred to by a variable called `b`.



%% Cell type:code id: tags:



``` 

print('b: ', b)

```



%% Cell type:markdown id: tags:



`a` has, however, been deleted:



%% Cell type:code id: tags:



``` 

print('a: ', a)

```



%% Cell type:markdown id: tags:



The variables `a` and `b` are just references to a list that is sitting in

memory somewhere - renaming or removing a reference does not have any effect

upon the list<sup>2</sup>.





If you are familiar with C or C++, you can think of a variable in Python as

like a `void *` pointer - it is just a pointer of an unspecified type, which

is pointing to some item in memory (which does have a specific type). Deleting

the pointer does not have any effect upon the item to which it was pointing.





> <sup>2</sup> Until no more references to the list exist, at which point it

> will be

> [garbage-collected](https://www.quora.com/How-does-garbage-collection-in-Python-work-What-are-the-pros-and-cons).





Now, functions in Python work in _exactly_ the same way as variables.  When we

define a function like this:



%% Cell type:code id: tags:



``` 

def inverse(a):

    return npla.inv(a)



print(inverse)

```



%% Cell type:markdown id: tags:



there is nothing special about the name `inverse` - `inverse` is just a

reference to a function that resides somewhere in memory. We can create a new

reference to this function:



%% Cell type:code id: tags:



``` 

inv2 = inverse

```



%% Cell type:markdown id: tags:



And delete the old reference:



%% Cell type:code id: tags:



``` 

del inverse

```



%% Cell type:markdown id: tags:



But the function still exists, and is still callable, via our second

reference:



%% Cell type:code id: tags:



``` 

print(inv2)

data    = np.random.random((10, 10))

invdata = inv2(data)

```



%% Cell type:markdown id: tags:



So there is nothing special about functions in Python - they are just items

that reside somewhere in memory, and to which we can create as many references

as we like.





> If it bothers you that `print(inv2)` resulted in

> `<function inverse at ...>`, and not `<function inv2 at ...>`, then refer to

> the appendix on

> [preserving function metdata](#appendix-preserving-function-metadata).

> [preserving function metadata](#appendix-preserving-function-metadata).





<a class="anchor" id="appendix-closures"></a>

## Appendix: Closures





Whenever we define or use a decorator, we are taking advantage of a concept

called a [_closure_][wiki-closure]. Take a second to re-familiarise yourself

called a [*closure*][wiki-closure]. Take a second to re-familiarise yourself

with our `memoize` decorator function from earlier - when `memoize` is called,

it creates and returns a function called `wrapper`:





[wiki-closure]: https://en.wikipedia.org/wiki/Closure_(computer_programming)



%% Cell type:code id: tags:



``` 

def memoize(func):



    cache = {}



    def wrapper(*args):



        # is there a value in the cache

        # for this set of inputs?

        cached = cache.get(args, None)



        # If not, call the function,

        # and cache the result.

        if cached is None:

            cached      = func(*args)

            cache[args] = cached

        else:

            print('Cached {}({}): {}'.format(func.__name__, args, cached))



        return cached



    return wrapper

```



%% Cell type:markdown id: tags:



Then `wrapper` is executed at some arbitrary point in the future. But how does

it have access to `cache`, defined within the scope of the `memoize` function,

after the execution of `memoize` has ended?



%% Cell type:code id: tags:



``` 

def nby2(n):

    return n * 2



# wrapper function is created here (and

# assigned back to the nby2 reference)

nby2 = memoize(nby2)



# wrapper function is executed here

print('nby2(2): ', nby2(2))

print('nby2(2): ', nby2(2))

```



%% Cell type:markdown id: tags:



The trick is that whenever a nested function is defined in Python, the scope

in which it is defined is preserved for that function's lifetime. So `wrapper`

has access to all of the variables within the `memoize` function's scope, that

were defined at the time that `wrapper` was created (which was when we called

`memoize`).  This is why `wrapper` is able to access `cache`, even though at

the time that `wrapper` is called, the execution of `memoize` has long since

finished.





This is what is known as a

[_closure_](https://www.geeksforgeeks.org/python-closures/). Closures are a

[*closure*](https://www.geeksforgeeks.org/python-closures/). Closures are a

fundamental, and extremely powerful, aspect of Python and other high level

languages. So there's your answer,

[fishbulb](https://www.youtube.com/watch?v=CiAaEPcnlOg).





<a class="anchor" id="appendix-decorators-without-arguments-versus-decorators-with-arguments"></a>

## Appendix: Decorators without arguments versus decorators with arguments





There are three ways to invoke a decorator with the `@` notation:





1. Naming it, e.g. `@mydecorator`

2. Calling it, e.g. `@mydecorator()`

3. Calling it, and passing it arguments, e.g. `@mydecorator(1, 2, 3)`





Python expects a decorator function to behave differently in the second and

third scenarios, when compared to the first:



%% Cell type:code id: tags:



``` 

def decorator(*args):

    print('  decorator({})'.format(args))

    def wrapper(*args):

        print('    wrapper({})'.format(args))

    return wrapper



print('Scenario #1: @decorator')

@decorator

def noop():

    pass



print('\nScenario #2: @decorator()')

@decorator()

def noop():

    pass



print('\nScenario #3: @decorator(1, 2, 3)')

@decorator(1, 2, 3)

def noop():

    pass

```



%% Cell type:markdown id: tags:



So if a decorator is "named" (scenario 1), only the decorator function

(`decorator` in the example above) is called, and is passed the decorated

function.





But if a decorator function is "called" (scenarios 2 or 3), both the decorator

function (`decorator`), __and its return value__ (`wrapper`) are called - the

function (`decorator`), **and its return value** (`wrapper`) are called - the

decorator function is passed the arguments that were provided, and its return

value is passed the decorated function.





This is why, if you are writing a decorator function which expects arguments,

you must use three layers of functions, like so:



%% Cell type:code id: tags:



``` 

def decorator(*args):

    def realDecorator(func):

        def wrapper(*args, **kwargs):

            return func(*args, **kwargs)

        return wrapper

    return realDecorator

```



%% Cell type:markdown id: tags:



> The author of this practical is angry about this, as he does not understand

> why the Python language designers couldn't allow a decorator function to be

> passed both the decorated function, and any arguments that were passed when

> the decorator was invoked, like so:

>

> ```

> def decorator(func, *args, **kwargs): # args/kwargs here contain

>                                       # whatever is passed to the

>                                       # decorator

>

>     def wrapper(*args, **kwargs):     # args/kwargs here contain

>                                       # whatever is passed to the

>                                       # decorated function

>          return func(*args, **kwargs)

>

>     return wrapper

> ```





<a class="anchor" id="appendix-per-instance-decorators"></a>

## Appendix: Per-instance decorators





In the section on [decorating methods](#decorators-on-methods), you saw

that when a decorator is applied to a method of a class,  that decorator

is invoked just once, and shared by all instances of the class. Consider this

example:



%% Cell type:code id: tags:



``` 

def decorator(func):

    print('Decorating {} function'.format(func.__name__))

    def wrapper(*args, **kwargs):

        print('Calling decorated function {}'.format(func.__name__))

        return func(*args, **kwargs)

    return wrapper



class MiscMaths(object):



    @decorator

    def add(self, a, b):

        return a + b

```



%% Cell type:markdown id: tags:



Note that `decorator` was called at the time that the `MiscMaths` class was

defined. Now, all `MiscMaths` instances share the same `wrapper` function:



%% Cell type:code id: tags:



``` 

mm1 = MiscMaths()

mm2 = MiscMaths()



print('1 + 2 =', mm1.add(1, 2))

print('3 + 4 =', mm2.add(3, 4))

```



%% Cell type:markdown id: tags:



This is not an issue in many cases, but it can be problematic in some. Imagine

if we have a decorator called `ensureNumeric`, which makes sure that arguments

passed to a function are numbers:



%% Cell type:code id: tags:



``` 

def ensureNumeric(func):

    def wrapper(*args):

        args = tuple([float(a) for a in args])

        return func(*args)

    return wrapper

```



%% Cell type:markdown id: tags:



This all looks well and good - we can use it to decorate a numeric function,

allowing strings to be passed in as well:



%% Cell type:code id: tags:



``` 

@ensureNumeric

def mul(a, b):

    return a * b



print(mul( 2,   3))

print(mul('5', '10'))

```



%% Cell type:markdown id: tags:



But what will happen when we try to decorate a method of a class?



%% Cell type:code id: tags:



``` 

class MiscMaths(object):



    @ensureNumeric

    def add(self, a, b):

        return a + b



mm = MiscMaths()

print(mm.add('5', 10))

```



%% Cell type:markdown id: tags:



What happened here?? Remember that the first argument passed to any instance

method is the instance itself (the `self` argument). Well, the `MiscMaths`

instance was passed to the `wrapper` function, which then tried to convert it

into a `float`.  So we can't actually apply the `ensureNumeric` function as a

decorator on a method in this way.





There are a few potential solutions here. We could modify the `ensureNumeric`

function, so that the `wrapper` ignores the first argument. But this would

mean that we couldn't use `ensureNumeric` with standalone functions.





But we _can_ manually apply the `ensureNumeric` decorator to `MiscMaths`

But we *can* manually apply the `ensureNumeric` decorator to `MiscMaths`

instances when they are initialised.  We can't use the nice `@ensureNumeric`

syntax to apply our decorators, but this is a viable approach:



%% Cell type:code id: tags:



``` 

class MiscMaths(object):



    def __init__(self):

        self.add = ensureNumeric(self.add)



    def add(self, a, b):

        return a + b



mm = MiscMaths()

print(mm.add('5', 10))

```



%% Cell type:markdown id: tags:



Another approach is to use a second decorator, which dynamically creates the

real decorator when it is accessed on an instance. This requires the use of an

advanced Python technique called

[_descriptors_](https://docs.python.org/3.5/howto/descriptor.html), which is

[*descriptors*](https://docs.python.org/3/howto/descriptor.html), which is

beyond the scope of this practical. But if you are interested, you can see an

implementation of this approach

[here](https://git.fmrib.ox.ac.uk/fsl/fslpy/blob/1.6.8/fsl/utils/memoize.py#L249).





<a class="anchor" id="appendix-preserving-function-metadata"></a>

## Appendix: Preserving function metadata





You may have noticed that when we decorate a function, some of its properties

are lost. Consider this function:



%% Cell type:code id: tags:



``` 

def add2(a, b):

    """Adds two numbers together."""

    return a + b

```



%% Cell type:markdown id: tags:



The `add2` function is an object which has some attributes, e.g.:



%% Cell type:code id: tags:



``` 

print('Name: ', add2.__name__)

print('Help: ', add2.__doc__)

```



%% Cell type:markdown id: tags:



However, when we apply a decorator to `add2`:



%% Cell type:code id: tags:



``` 

def decorator(func):

    def wrapper(*args, **kwargs):

        """Internal wrapper function for decorator."""

        print('Calling decorated function {}'.format(func.__name__))

        return func(*args, **kwargs)

    return wrapper





@decorator

def add2(a, b):

    """Adds two numbers together."""

    return a + b

```



%% Cell type:markdown id: tags:



Those attributes are lost, and instead we get the attributes of the `wrapper`

function:



%% Cell type:code id: tags:



``` 

print('Name: ', add2.__name__)

print('Help: ', add2.__doc__)

```



%% Cell type:markdown id: tags:



While this may be inconsequential in most situations, it can be quite annoying

in some, such as when we are automatically [generating

documentation](http://www.sphinx-doc.org/) for our code.





Fortunately, there is a workaround, available in the built-in

[`functools`](https://docs.python.org/3.5/library/functools.html#functools.wraps)

[`functools`](https://docs.python.org/3/library/functools.html#functools.wraps)

module:



%% Cell type:code id: tags:



``` 

import functools



def decorator(func):

    @functools.wraps(func)

    def wrapper(*args, **kwargs):

        """Internal wrapper function for decorator."""

        print('Calling decorated function {}'.format(func.__name__))

        return func(*args, **kwargs)

    return wrapper



@decorator

def add2(a, b):

    """Adds two numbers together."""

    return a + b

```



%% Cell type:markdown id: tags:



We have applied the `@functools.wraps` decorator to our internal `wrapper`

function - this will essentially replace the `wrapper` function metdata with

the metadata from our `func` function. So our `add2` name and documentation is

now preserved:



%% Cell type:code id: tags:



``` 

print('Name: ', add2.__name__)

print('Help: ', add2.__doc__)

```



%% Cell type:markdown id: tags:



<a class="anchor" id="appendix-class-decorators"></a>

## Appendix: Class decorators





> Not to be confused with [_decorator classes_](#decorator-classes)!

> Not to be confused with [*decorator classes*](#decorator-classes)!





In this practical, we have shown how decorators can be applied to functions

and methods. But decorators can in fact also be applied to _classes_. This is

and methods. But decorators can in fact also be applied to *classes*. This is

a fairly niche feature that you are probably not likely to need, so we will

only cover it briefly.





Imagine that we want all objects in our application to have a globally unique

(within the application) identifier. We could use a decorator which contains

the logic for generating unique IDs, and defines the interface that we can

use on an instance to obtain its ID:



%% Cell type:code id: tags:



``` 

import random



allIds = set()



def uniqueID(cls):

    class subclass(cls):

        def getUniqueID(self):



            uid = getattr(self, '_uid', None)



            if uid is not None:

                return uid



            while uid is None or uid in set():

                uid = random.randint(1, 100)



            self._uid = uid

            return uid



    return subclass

```



%% Cell type:markdown id: tags:



Now we can use the `@uniqueID` decorator on any class that we need to

have a unique ID:



%% Cell type:code id: tags:



``` 

@uniqueID

class Foo(object):

    pass



@uniqueID

class Bar(object):

    pass

```



%% Cell type:markdown id: tags:



All instances of these classes will have a `getUniqueID` method:



%% Cell type:code id: tags:



``` 

f1 = Foo()

f2 = Foo()

b1 = Bar()

b2 = Bar()



print('f1: ', f1.getUniqueID())

print('f2: ', f2.getUniqueID())

print('b1: ', b1.getUniqueID())

print('b2: ', b2.getUniqueID())

```



%% Cell type:markdown id: tags:



<a class="anchor" id="useful-references"></a>

## Useful references





* [Understanding decorators in 12 easy steps](http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/)

* [The decorators they won't tell you about](https://github.com/hchasestevens/hchasestevens.github.io/blob/master/notebooks/the-decorators-they-wont-tell-you-about.ipynb)

* [Closures - Wikipedia][wiki-closure]

* [Closures in Python](https://www.geeksforgeeks.org/python-closures/)

* [Garbage collection in Python](https://www.quora.com/How-does-garbage-collection-in-Python-work-What-are-the-pros-and-cons)



[wiki-closure]: https://en.wikipedia.org/wiki/Closure_(computer_programming)

advanced_topics/06_decorators.md

+23 −21
Original line number Diff line number Diff line
@@ -100,7 +100,7 @@ This new `timeFunc` function is again passed a function `func`, but this time 
as its sole argument. It then creates and returns a new function,

`wrapperFunc`. This `wrapperFunc` function calls and times the function that

was passed to `timeFunc`.  But note that when `timeFunc` is called,

`wrapperFunc` is _not_ called - it is only created and returned.

`wrapperFunc` is *not* called - it is only created and returned.





Let's use our new `timeFunc` implementation:
@@ -151,9 +151,9 @@ which holds a reference to the original definition of `inverse`. 
> [functions are not special](#appendix-functions-are-not-special).





Guess what? We have just created a __decorator__. A decorator is simply a

Guess what? We have just created a **decorator**. A decorator is simply a

function which accepts a function as its input, and returns another function

as its output. In the example above, we have _decorated_ the `inverse`

as its output. In the example above, we have *decorated* the `inverse`

function with the `timeFunc` decorator.
@@ -228,7 +228,7 @@ MiscMaths.inverse = timeFunc(MiscMaths.inverse) 
```





So only one `wrapperFunc` function exists, and this function is _shared_ by

So only one `wrapperFunc` function exists, and this function is *shared* by

all instances of the `MiscMaths` class - (such as the `mm1` and `mm2`

instances in the example above). In many cases this is not a problem, but

there can be situations where you need each instance of your class to have its
@@ -383,8 +383,10 @@ def limitedMemoize(maxSize): 
                # remove the oldest item. In practice

                # it would make more sense to remove

                # the item with the oldest access

                # time, but this is good enough for

                # an introduction!

                # time (or remove the least recently

                # used item, as the built-in

                # @functools.lru_cache does), but this

                # is good enough for now!

                if len(cache) >= maxSize:

                    cache.popitem(last=False)
@@ -398,11 +400,11 @@ def limitedMemoize(maxSize): 
```



> We used the handy

> [`collections.OrderedDict`](https://docs.python.org/3.5/library/collections.html#collections.OrderedDict)

> [`collections.OrderedDict`](https://docs.python.org/3/library/collections.html#collections.OrderedDict)

> class here which preserves the insertion order of key-value pairs.





This is starting to look a little complicated - we now have _three_ layers of

This is starting to look a little complicated - we now have *three* layers of

functions. This is necessary when you wish to write a decorator which accepts

arguments (refer to the

[appendix](#appendix-decorators-without-arguments-versus-decorators-with-arguments)
@@ -503,7 +505,7 @@ expensiveFunc(1) 




By now, you will have gained the impression that a decorator is a function

which _decorates_ another function. But if you went through the practical on

which *decorates* another function. But if you went through the practical on

operator overloading, you might remember the special `__call__` method, that

allows an object to be called as if it were a function.
@@ -594,17 +596,17 @@ registry.runTests() 
```





> Unit testing is something which you must do! This is __especially__

> Unit testing is something which you must do! This is **especially**

> important in an interpreted language such as Python, where there is no

> compiler to catch all of your mistakes.

>

> Python has a built-in

> [`unittest`](https://docs.python.org/3.5/library/unittest.html) module,

> [`unittest`](https://docs.python.org/3/library/unittest.html) module,

> however the third-party [`pytest`](https://docs.pytest.org/en/latest/) and

> [`nose`](http://nose2.readthedocs.io/en/latest/) are popular.  It is also

> wise to combine your unit tests with

> [`coverage`](https://coverage.readthedocs.io/en/coverage-4.5.1/), which

> tells you how much of your code was executed, or _covered_ when your

> tells you how much of your code was executed, or *covered* when your

> tests were run.
@@ -711,7 +713,7 @@ as we like. 
> If it bothers you that `print(inv2)` resulted in

> `<function inverse at ...>`, and not `<function inv2 at ...>`, then refer to

> the appendix on

> [preserving function metdata](#appendix-preserving-function-metadata).

> [preserving function metadata](#appendix-preserving-function-metadata).





<a class="anchor" id="appendix-closures"></a>
@@ -719,7 +721,7 @@ as we like. 




Whenever we define or use a decorator, we are taking advantage of a concept

called a [_closure_][wiki-closure]. Take a second to re-familiarise yourself

called a [*closure*][wiki-closure]. Take a second to re-familiarise yourself

with our `memoize` decorator function from earlier - when `memoize` is called,

it creates and returns a function called `wrapper`:
@@ -781,7 +783,7 @@ finished. 




This is what is known as a

[_closure_](https://www.geeksforgeeks.org/python-closures/). Closures are a

[*closure*](https://www.geeksforgeeks.org/python-closures/). Closures are a

fundamental, and extremely powerful, aspect of Python and other high level

languages. So there's your answer,

[fishbulb](https://www.youtube.com/watch?v=CiAaEPcnlOg).
@@ -832,7 +834,7 @@ function. 




But if a decorator function is "called" (scenarios 2 or 3), both the decorator

function (`decorator`), __and its return value__ (`wrapper`) are called - the

function (`decorator`), **and its return value** (`wrapper`) are called - the

decorator function is passed the arguments that were provided, and its return

value is passed the decorated function.
@@ -964,7 +966,7 @@ function, so that the `wrapper` ignores the first argument. But this would 
mean that we couldn't use `ensureNumeric` with standalone functions.





But we _can_ manually apply the `ensureNumeric` decorator to `MiscMaths`

But we *can* manually apply the `ensureNumeric` decorator to `MiscMaths`

instances when they are initialised.  We can't use the nice `@ensureNumeric`

syntax to apply our decorators, but this is a viable approach:
@@ -986,7 +988,7 @@ print(mm.add('5', 10)) 
Another approach is to use a second decorator, which dynamically creates the

real decorator when it is accessed on an instance. This requires the use of an

advanced Python technique called

[_descriptors_](https://docs.python.org/3.5/howto/descriptor.html), which is

[*descriptors*](https://docs.python.org/3/howto/descriptor.html), which is

beyond the scope of this practical. But if you are interested, you can see an

implementation of this approach

[here](https://git.fmrib.ox.ac.uk/fsl/fslpy/blob/1.6.8/fsl/utils/memoize.py#L249).
@@ -1051,7 +1053,7 @@ documentation](http://www.sphinx-doc.org/) for our code. 




Fortunately, there is a workaround, available in the built-in

[`functools`](https://docs.python.org/3.5/library/functools.html#functools.wraps)

[`functools`](https://docs.python.org/3/library/functools.html#functools.wraps)

module:
@@ -1089,11 +1091,11 @@ print('Help: ', add2.__doc__) 
## Appendix: Class decorators





> Not to be confused with [_decorator classes_](#decorator-classes)!

> Not to be confused with [*decorator classes*](#decorator-classes)!





In this practical, we have shown how decorators can be applied to functions

and methods. But decorators can in fact also be applied to _classes_. This is

and methods. But decorators can in fact also be applied to *classes*. This is

a fairly niche feature that you are probably not likely to need, so we will

only cover it briefly.

advanced_topics/08_fslpy/fmri.feat/reg/highres2standard.mat

0 → 100644
+4 −0
Original line number Diff line number Diff line 
1.056802026  -0.01924547726  0.02614687181  -36.51723948  

0.009055463297  0.9745460053  0.09056277052  -8.771603455  

-0.04315832679  -0.1680837227  1.136420957  -1.399839791  

0  0  0  1