Merge branch 'MJ-fixes' into 'master'

Implemented MJ's comments See merge request fsl/pytreat-2018-practicals!19

Merge branch 'MJ-fixes' into 'master'
09a1c0de · Paul McCarthy · 9406de2f · a1c97f14 · 09a1c0de · 09a1c0de
Commit 09a1c0de authored 7 years ago by Paul McCarthy
--- a/getting_started/02_text_io.ipynb
+++ b/getting_started/02_text_io.ipynb
@@ -110,7 +110,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "> Any lines starting with `!` will be interpreted as shell commands by ipython. It is great when playing around in the ipython notebook or in the ipython terminal, however it is an ipython-only feature and hence is not available when writing python scripts. How to call shell commands from python will be discusses in the `scripts` practical.\n",
+    "> Any lines starting with `!` will be interpreted as shell commands by ipython. It is great when playing around in the ipython notebook or in the ipython terminal, however it is an ipython-only feature and hence is not available when writing python scripts. How to call shell commands from python will be discussed in the `scripts` practical.\n",
    "\n",
    "If we want to add to the existing file we can open it in the append mode:"
   ]
@@ -375,8 +375,8 @@
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
-    "print(str(datetime.now())\n",
+    "print('str(): ', str(datetime.now()))\n",
-    "print(repr(datetime.now())"
+    "print('repr(): ', repr(datetime.now()))"
   ]
  },
  {
@@ -492,7 +492,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "Note that the variable `:` delimiter separates the variable identifies on the left from the formatting rules on the right.\n",
+    "Note that the variable `:` delimiter separates the variable identifiers on the left from the formatting rules on the right.\n",
    "\n",
    "Finally the new, fancy formatted string literals (only available in python 3.6+).\n",
    "This new format is very similar to the recommended style, except that all placeholders are automatically evaluated in the local environment at the time the template is defined.\n",
@@ -596,7 +596,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "> We use the syntax `[<expr> for <element> in <sequence>]` here which applies the `expr` to each `element` in the `sequence` and returns the resulting list. This is a convenient form in python to create a new list from the old one.\n",
+    "> We use the syntax `[<expr> for <element> in <sequence>]` here which applies the `expr` to each `element` in the `sequence` and returns the resulting list. This is a list comprehension - a convenient form in python to create a new list from the old one.\n",
    "\n",
    "<a class=\"anchor\" id=\"converting-strings-to-numbers\"></a>\n",
    "### Converting strings to numbers\n",

 %% Cell type:markdown id: tags:
 # Text input/output
 In this section we will explore how to write and/or retrieve our data from text files.
 Most of the functionality for reading/writing files and manipulating strings is available without any imports. However, you can find some additional functionality in the [`string`](https://docs.python.org/3.6/library/string.html) module.
 Most of the string functions are available as methods on string objects. This means that you can use the ipython autocomplete to check for them.
 %% Cell type:code id: tags:
 ``` 
 empty_string = ''
 ```
 %% Cell type:code id: tags:
 ``` 
 empty_string.    # after running the code block above, put your cursor behind the dot and press tab to get a list of methods
 ```
 %% Cell type:markdown id: tags:
 * [Reading/writing files](#reading-writing-files)
 * [Creating new strings](#creating-new-strings)
 * [String syntax](#string-syntax)
  * [Unicode versus bytes](#unicode-versus-bytes)
 * [Converting objects into strings](#converting-objects-into-strings)
 * [Combining strings](#combining-strings)
 * [String formattings](#string-formatting)
 * [Extracting information from strings](#extracting-information-from-strings)
 * [Splitting strings](#splitting-strings)
 * [Converting strings to numbers](#converting-strings-to-numbers)
 * [Regular expressions](#regular-expressions)
 * [Exercises](#exercises)
 <a class="anchor" id="reading-writing-files"></a>
 ## Reading/writing files
 The syntax to open a file in python is `with open(<filename>, <mode>) as <file_object>: <block of code>`, where
 * `filename` is a string with the name of the file
 * `mode` is one of 'r' (for read-only access), 'w' (for writing a file, this wipes out any existing content), 'a' (for appending to an existing file).
 * `file_object` is a variable name which will be used within the `block of code` to access the opened file.
 For example the following will read all the text in `README.md` and print it.
 %% Cell type:code id: tags:
 ``` 
 with open('README.md', 'r') as readme_file:
    print(readme_file.read())
 ```
 %% Cell type:markdown id: tags:
 > The `with` statement is an advanced python feature, however you will probably only encounter it when opening files. In that context it merely ensures that the file will be properly closed as soon as the program leaves the `with` statement (even if an error is raised within the `with` statement).
 You could also use the `readlines()` method to get a list of all the lines.
 A very similar syntax is used to write files:
 %% Cell type:code id: tags:
 ``` 
 with open('02_text_io/my_file', 'w') as my_file:
    my_file.write('This is my first line\n')
    my_file.writelines(['Second line\n', 'and the third\n'])
 ```
 %% Cell type:markdown id: tags:
 Note that no new line characters get added automatically. We can investigate the resulting file using
 %% Cell type:code id: tags:
 ``` 
 !cat 02_text_io/my_file
 ```
 %% Cell type:markdown id: tags:
-> Any lines starting with `!` will be interpreted as shell commands by ipython. It is great when playing around in the ipython notebook or in the ipython terminal, however it is an ipython-only feature and hence is not available when writing python scripts. How to call shell commands from python will be discusses in the `scripts` practical.
+> Any lines starting with `!` will be interpreted as shell commands by ipython. It is great when playing around in the ipython notebook or in the ipython terminal, however it is an ipython-only feature and hence is not available when writing python scripts. How to call shell commands from python will be discussed in the `scripts` practical.
 If we want to add to the existing file we can open it in the append mode:
 %% Cell type:code id: tags:
 ``` 
 with open('02_text_io/my_file', 'a') as my_file:
    my_file.write('More lines is always better\n')
 !cat 02_text_io/my_file
 ```
 %% Cell type:markdown id: tags:
 Below we will discuss how we can convert python objects to strings to store in these files and how to extract those python objects from strings again.
 <a class="anchor" id="creating-new-strings"></a>
 ## Creating new strings
 <a class="anchor" id="string-syntax"></a>
 ### String syntax
 Single-line strings can be created in python using either single or double quotes
 %% Cell type:code id: tags:
 ``` 
 a_string = 'To be or not to be'
 same_string = "To be or not to be"
 print(a_string == same_string)
 ```
 %% Cell type:markdown id: tags:
 The main rationale for choosing between single or double quotes, is whether the string itself will contain any quotes. You can include a single quote in a string surrounded by single quotes by escaping it with the `\` character, however in such a case it would be more convenient to use double quotes:
 %% Cell type:code id: tags:
 ``` 
 a_string = "That's the question"
 same_string = 'That\'s the question'
 print(a_string == same_string)
 ```
 %% Cell type:markdown id: tags:
 New-lines (`\n`), tabs (`\t`) and many other special characters are supported
 %% Cell type:code id: tags:
 ``` 
 a_string = "This is the first line.\nAnd here is the second.\n\tThe third starts with a tab."
 print(a_string)
 ```
 %% Cell type:markdown id: tags:
 However, the easiest way to create multi-line strings is to use a triple quote (again single or double quotes can be used). Triple quotes allow your string to span multiple lines:
 %% Cell type:code id: tags:
 ``` 
 multi_line_string = """This is the first line.
 And here is the second.
 \tThird line starts with a tab."""
 print(multi_line_string)
 ```
 %% Cell type:markdown id: tags:
 If you don't want python to reintepret your `\n`, `\t`, etc. in your strings, you can prepend the quotes enclosing the string with an `r`. This will lead to python interpreting the following string as raw text.
 %% Cell type:code id: tags:
 ``` 
 single_line_string = r"This string is not multiline.\nEven though it contains the \n character"
 print(single_line_string)
 ```
 %% Cell type:markdown id: tags:
 One pitfall when creating a list of strings is that python automatically concatenates string literals, which are only separated by white space:
 %% Cell type:code id: tags:
 ``` 
 my_list_of_strings = ['a', 'b', 'c' 'd', 'e']
 print("The 'c' and 'd' got concatenated, because we forgot the comma:", my_list_of_strings)
 ```
 %% Cell type:markdown id: tags:
 <a class="anchor" id="unicode-versus-bytes"></a>
 #### unicode versus bytes
 To encourage the spread of python around the world, python 3 switched to using unicode as the default for strings and code (which is one of the main reasons for the incompatibility between python 2 and 3).
 This means that each element in a string is a unicode character (using [UTF-8 encoding](https://docs.python.org/3/howto/unicode.html)), which can consist of one or more bytes.
 The advantage is that any unicode characters can now be used in strings or in the code itself:
 %% Cell type:code id: tags:
 ``` 
 Δ = "café"
 print(Δ)
 ```
 %% Cell type:markdown id: tags:
 In python 2 each element in a string was a single byte rather than a potentially multi-byte character. You can convert back to interpreting your sequence as a unicode string or a byte array using:
 * `encode()` called on a string converts it into a bytes array (`bytes` object)
 * `decode()` called on a `bytes` array converts it into a unicode string.
 %% Cell type:code id: tags:
 ``` 
 delta = "Δ"
 print('The character', delta, 'consists of the following 2 bytes', delta.encode())
 ```
 %% Cell type:markdown id: tags:
 These byte arrays can be created directly be prepending the quotes enclosing the string with a `b`, which tells python 3 to interpret the following as a byte array:
 %% Cell type:code id: tags:
 ``` 
 a_byte_array = b'\xce\xa9'
 print('The two bytes ', a_byte_array, ' become single unicode character (', a_byte_array.decode(), ') with UTF-8 encoding')
 ```
 %% Cell type:markdown id: tags:
 Especially in code dealing with strings (e.g., reading/writing of files) many of the errors arising of running python 2 code in python 3 arise from the mixing of unicode strings with byte arrays. Decoding and/or encoding some of these objects can often fix these issues.
 By default any file opened in python will be interpreted as unicode. If you want to treat a file as raw bytes, you have to include a 'b' in the `mode` when calling the `open()` function:
 %% Cell type:code id: tags:
 ``` 
 import os.path as op
 with open(op.expandvars('${FSLDIR}/data/standard/MNI152_T1_1mm.nii.gz'), 'rb') as gzipped_nifti:
    print('First few bytes of gzipped NIFTI file:', gzipped_nifti.read(10))
 ```
 %% Cell type:markdown id: tags:
 > We use the `expandvars()` function here to insert the FSLDIR environmental variable into our string. This function will be presented in the file management practical.
 <a class="anchor" id="converting-objects-into-strings"></a>
 ### converting objects into strings
 There are two functions to convert python objects into strings, `repr()` and `str()`.
 All other functions that rely on string-representations of python objects will use one of these two (for example the `print()` function will call `str()` on the object).
 The goal of the `str()` function is to be readable, while the goal of `repr()` is to be unambiguous. Compare
 %% Cell type:code id: tags:
 ``` 
 print(str("3"))
 print(str(3))
 ```
 %% Cell type:markdown id: tags:
 with
 %% Cell type:code id: tags:
 ``` 
 print(repr("3"))
 print(repr(3))
 ```
 %% Cell type:markdown id: tags:
 In both cases you get the value of the object (3), but only the `repr` returns enough information to actually know the type of the object.
 Perhaps the difference is clearer with a more advanced object.
 The `datetime` module contains various classes and functions to work with dates (there is also a `time` module).
 Here we will look at the alternative string representations of the `datetime` object itself:
 %% Cell type:code id: tags:
 ``` 
 from datetime import datetime
-print(str(datetime.now())
+print('str(): ', str(datetime.now()))
-print(repr(datetime.now())
+print('repr(): ', repr(datetime.now()))
 ```
 %% Cell type:markdown id: tags:
 Note that the result from `str()` is human-readable as a date, while the result from `repr()` is more useful if you wanted to recreate the `datetime` object.
 <a class="anchor" id="combining-strings"></a>
 ### Combining strings
 The simplest way to concatenate strings is to simply add them together:
 %% Cell type:code id: tags:
 ``` 
 a_string = "Part 1"
 other_string = "Part 2"
 full_string = a_string + ", " + other_string
 print(full_string)
 ```
 %% Cell type:markdown id: tags:
 Given a whole sequence of strings, you can concatenate them together using the `join()` method:
 %% Cell type:code id: tags:
 ``` 
 list_of_strings = ['first', 'second', 'third', 'fourth']
 full_string = ', '.join(list_of_strings)
 print(full_string)
 ```
 %% Cell type:markdown id: tags:
 Note that the string on which the `join()` method is called (`', '` in this case) is used as a delimiter to separate the different strings. If you just want to concatenate the strings you can call `join()` on the empty string:
 %% Cell type:code id: tags:
 ``` 
 list_of_strings = ['first', 'second', 'third', 'fourth']
 full_string = ''.join(list_of_strings)
 print(full_string)
 ```
 %% Cell type:markdown id: tags:
 <a class="anchor" id="string-formatting"></a>
 ### String formatting
 Using the techniques in [Combining strings](#combining-strings) we can build simple strings. For longer strings it is often useful to first write a template strings with some placeholders, where variables are later inserted. Built into python are currently 4 different ways of doing this (with many packages providing similar capabilities):
 * the recommended [new-style formatting](https://docs.python.org/3.6/library/string.html#format-string-syntax).
 * printf-like [old-style formatting](https://docs.python.org/3/library/stdtypes.html#old-string-formatting)
 * [formatted string literals](https://docs.python.org/3.6/reference/lexical_analysis.html#f-strings) (these are only available in python 3.6+)
 * bash-like [template-strings](https://docs.python.org/3.6/library/string.html#template-strings)
 Here we provide a single example using the first three methods, so you can recognize them in the future.
 First the old print-f style. Note that this style is invoked by using the modulo (`%`) operator on the string. Every placeholder (starting with the `%`) is then replaced by one of the values provided.
 %% Cell type:code id: tags:
 ``` 
 a = 3
 b = 1 / 3
 print('%.3f = %i + %.3f' % (a + b, a, b))
 print('%(total).3f = %(a)i + %(b).3f' % {'a': a, 'b': b, 'total': a + b})
 ```
 %% Cell type:markdown id: tags:
 Then the recommended new style formatting (You can find a nice tutorial [here](https://www.digitalocean.com/community/tutorials/how-to-use-string-formatters-in-python-3)). Note that this style is invoked by calling the `format()` method on the string and the placeholders are marked by the curly braces `{}`.
 %% Cell type:code id: tags:
 ``` 
 a = 3
 b = 1 / 3
 print('{:.3f} = {} + {:.3f}'.format(a + b, a, b))
 print('{total:.3f} = {a} + {b:.3f}'.format(a=a, b=b, total=a+b))
 ```
 %% Cell type:markdown id: tags:
-Note that the variable `:` delimiter separates the variable identifies on the left from the formatting rules on the right.
+Note that the variable `:` delimiter separates the variable identifiers on the left from the formatting rules on the right.
 Finally the new, fancy formatted string literals (only available in python 3.6+).
 This new format is very similar to the recommended style, except that all placeholders are automatically evaluated in the local environment at the time the template is defined.
 This means that we do not have to explicitly provide the parameters (and we can evaluate the sum inside the string!), although it does mean we also can not re-use the template.
 %% Cell type:code id: tags:
 ``` 
 a = 3
 b = 1/3
 print(f'{a + b:.3f} = {a} + {b:.3f}')
 ```
 %% Cell type:markdown id: tags:
 This code block will fail in fslpython, since it uses python 3.5.
 <a class="anchor" id="extracting-information-from-strings"></a>
 ## Extracting information from strings
 <a class="anchor" id="splitting-strings"></a>
 ### Splitting strings
 The simplest way to extract a sub-string is to use slicing
 %% Cell type:code id: tags:
 ``` 
 a_string = 'abcdefghijklmnopqrstuvwxyz'
 print(a_string[10])  # create a string containing only the 10th character
 print(a_string[20:])  # create a string containing the 20th character onward
 print(a_string[::-1])  # creating the reverse string
 ```
 %% Cell type:markdown id: tags:
 If you are not sure, where to cut into a string, you can use the `find()` method to find the first occurrence of a sub-string or `findall()` to find all occurrences.
 %% Cell type:code id: tags:
 ``` 
 a_string = 'abcdefghijklmnopqrstuvwxyz'
 index = a_string.find('fgh')
 print(a_string[:index])  # extracts the sub-string up to the first occurence of 'fgh'
 print('index for non-existent sub-string', a_string.find('cats'))  # note that find returns -1 when it can not find the sub-string rather than raising an error.
 ```
 %% Cell type:markdown id: tags:
 You can automate this process of splitting a string at a sub-string using the `split()` method. By default it will split a string at the white space.
 %% Cell type:code id: tags:
 ``` 
 print('The split() method\trecognizes a wide variety\nof white space'.split())
 ```
 %% Cell type:markdown id: tags:
 To separate a comma separated list we will need to supply the delimiter to the `split()` method. We can then use the `strip()` method to remove any whitespace at the beginning or end of the string:
 %% Cell type:code id: tags:
 ``` 
 scientific_packages_string = "numpy, scipy, pandas, matplotlib, nibabel"
 list_with_whitespace = scientific_packages_string.split(',')
 print(list_with_whitespace)
 list_without_whitespace = [individual_string.strip() for individual_string in list_with_whitespace]
 print(list_without_whitespace)
 ```
 %% Cell type:markdown id: tags:
-> We use the syntax `[<expr> for <element> in <sequence>]` here which applies the `expr` to each `element` in the `sequence` and returns the resulting list. This is a convenient form in python to create a new list from the old one.
+> We use the syntax `[<expr> for <element> in <sequence>]` here which applies the `expr` to each `element` in the `sequence` and returns the resulting list. This is a list comprehension - a convenient form in python to create a new list from the old one.
 <a class="anchor" id="converting-strings-to-numbers"></a>
 ### Converting strings to numbers
 Once you have extracted a number from a string, you can convert it into an actual integer or float by calling respectively `int()` or `float()` on it. `float()` understands a wide variety of different ways to write numbers:
 %% Cell type:code id: tags:
 ``` 
 print(int("3"))
 print(float("3"))
 print(float("3.213"))
 print(float("3.213e5"))
 print(float("3.213E-25"))
 ```
 %% Cell type:markdown id: tags:
 <a class="anchor" id="regular-expressions"></a>
 ### Regular expressions
 Regular expressions are used for looking for specific patterns in a longer string. This can be used to extract specific information from a well-formatted string or to modify a string. In python regular expressions are available in the [re](https://docs.python.org/3/library/re.html#re-syntax) module.
 A full discussion of regular expression goes far beyond this practical. If you are interested, have a look [here](https://docs.python.org/3/howto/regex.html).
 <a class="anchor" id="exercises"></a>
 ## Exercises
 ### Joining/splitting strings
 The file 02_text_io/input.txt contains integers in a 2-column format (separated by spaces). Read in this file and write it back out in 2-rows separated by comma's.
 %% Cell type:code id: tags:
 ``` 
 input_filename = '02_text_io/input.txt'
 out_filename = '02_text_io/output.txt'
 with open(input_filename, 'r') as input_file:
    ...
 with open(output_filename, 'w') as output_file:
    ...
 ```
 %% Cell type:markdown id: tags:
 ### String formatting and regular expressions
 Given a template for MRI files:
 s<subject_id>/<modality>_<res>mm.nii.gz
 where <subject_id> is a 6-digit subject-id, <modality> is one of T1w, T2w, or PD, and <res> is the resolution of the image (up to one digits behind the dot, e.g. 1.5)
 Write a function that takes the subject_id (as an integer), the modality (as a string), and the resolution (as a float) and returns the complete filename (Hint: use one of the formatting techniques mentioned in [String formatting](#string-formatting)).
 %% Cell type:code id: tags:
 ``` 
 def get_filename(subject_id, modality, resolution):
    ...
 ```
 %% Cell type:markdown id: tags:
 For a more difficult exercise, write a function that extracts the subject id, modality, and resolution from a filename name (using a regular expression or by using `find` and `split` to access relevant parts of the string)
 %% Cell type:code id: tags:
 ``` 
 def get_parameters(filename):
    ...
    return subject_id, modality, resolution
 ```

--- a/getting_started/02_text_io.md
+++ b/getting_started/02_text_io.md
@@ -52,7 +52,7 @@ Note that no new line characters get added automatically. We can investigate the
 ```
 !cat 02_text_io/my_file
 ```
-> Any lines starting with `!` will be interpreted as shell commands by ipython. It is great when playing around in the ipython notebook or in the ipython terminal, however it is an ipython-only feature and hence is not available when writing python scripts. How to call shell commands from python will be discusses in the `scripts` practical.
+> Any lines starting with `!` will be interpreted as shell commands by ipython. It is great when playing around in the ipython notebook or in the ipython terminal, however it is an ipython-only feature and hence is not available when writing python scripts. How to call shell commands from python will be discussed in the `scripts` practical.
 If we want to add to the existing file we can open it in the append mode:
 ```
@@ -166,8 +166,8 @@ The `datetime` module contains various classes and functions to work with dates
 Here we will look at the alternative string representations of the `datetime` object itself:
 ```
 from datetime import datetime
-print(str(datetime.now())
+print('str(): ', str(datetime.now()))
-print(repr(datetime.now())
+print('repr(): ', repr(datetime.now()))
 ```
 Note that the result from `str()` is human-readable as a date, while the result from `repr()` is more useful if you wanted to recreate the `datetime` object.
@@ -222,7 +222,7 @@ b = 1 / 3
 print('{:.3f} = {} + {:.3f}'.format(a + b, a, b))
 print('{total:.3f} = {a} + {b:.3f}'.format(a=a, b=b, total=a+b))
 ```
-Note that the variable `:` delimiter separates the variable identifies on the left from the formatting rules on the right.
+Note that the variable `:` delimiter separates the variable identifiers on the left from the formatting rules on the right.
 Finally the new, fancy formatted string literals (only available in python 3.6+).
 This new format is very similar to the recommended style, except that all placeholders are automatically evaluated in the local environment at the time the template is defined.
@@ -269,7 +269,7 @@ print(list_with_whitespace)
 list_without_whitespace = [individual_string.strip() for individual_string in list_with_whitespace]
 print(list_without_whitespace)
 ```
-> We use the syntax `[<expr> for <element> in <sequence>]` here which applies the `expr` to each `element` in the `sequence` and returns the resulting list. This is a convenient form in python to create a new list from the old one.
+> We use the syntax `[<expr> for <element> in <sequence>]` here which applies the `expr` to each `element` in the `sequence` and returns the resulting list. This is a list comprehension - a convenient form in python to create a new list from the old one.
 <a class="anchor" id="converting-strings-to-numbers"></a>
 ### Converting strings to numbers

--- a/getting_started/07_jupyter.ipynb
+++ b/getting_started/07_jupyter.ipynb
@@ -38,6 +38,22 @@
    "string.capwords?"
   ]
  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This also works for any of the magic commands discussed below"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%run?"
+   ]
+  },
  {
   "cell_type": "markdown",
   "metadata": {},
@@ -91,13 +107,25 @@
   "source": [
    "r = !fslstats ${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz -r\n",
    "r_lower, r_upper = [float(element) for element in r[0].split()]\n",
-    "print('Bounds are ({:d}, {:d})'.format(r_lower, r_upper"
+    "print('Bounds are ({:.0f}, {:.0f})'.format(r_lower, r_upper))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
+    "---\n",
+    "\n",
+    "## Running python scripts\n",
+    "We could run a python script as a shell command above. However, it will often be more convenient to use `%run` instead.\n",
+    "> ```\n",
+    "> %run <python script> <arguments...>\n",
+    "> ```\n",
+    "Arguments are provided in exactly the same way as if you called `python` in the shell. The main advantages are:\n",
+    "- Any top-level variables will be made available to you after the script finishes\n",
+    "- All the debugging/timing/profiling tools discussed below will be available to you\n",
+    "A common workflow, when writing a python script is to have an Ipython terminal open next to your text editor and regularly use %run to test the script\n",
+    "\n",
    "---\n",
    "\n",
    "## Running other programming languages\n",
@@ -122,7 +150,7 @@
   "source": [
    "---\n",
    "\n",
-    "## Timing code\n",
+    "## Timing/profiling code\n",
    "We can time a line of code with `%time` or a whole code block using `%%time`.\n",
    "To get the time needed to calculate the sine of a million random numbers:"
   ]
@@ -156,6 +184,24 @@
    "%timeit np.sin(numbers)  # this will take a few seconds to run"
   ]
  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, if you want to figure out what part of the code is actually slowing you down you can use `%prun`, which gives you an overview of how long the interpreter spent in each method:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import nibabel as nib\n",
+    "import os.path as op\n",
+    "%prun nib.load(op.expandvars('${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz'))"
+   ]
+  },
  {
   "cell_type": "markdown",
   "metadata": {},
@@ -174,6 +220,10 @@
   "source": [
    "import numpy as np\n",
    "def total(a_list):\n",
+    "    \"\"\"Calculate the total of a list.\n",
+    "\n",
+    "    This is a very naive (not recommended) and bugged implementation\n",
+    "    \"\"\"\n",
    "    # create local copy befor changing the input\n",
    "    local_list = list(a_list)\n",
    "    total = 0.\n",
@@ -208,7 +258,7 @@
    "\n",
    "> WARNING: you need to quit the debugger before any further commands will run (type `q` into the prompt)!\n",
    "\n",
-    "If you always want to enter the debugger when an error is raised you can call `%pdb on` at any time (call `%pdf off` to rever this)\n",
+    "If you always want to enter the debugger when an error is raised you can call `%pdb on` at any time (call `%pdf off` to reverse this)\n",
    "\n",
    "---\n",
    "\n",
@@ -233,7 +283,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "> Keep in mind that as soon as you have started plotting you can no longer change your backend without restarting python.\n",
+    "> Keep in mind that as soon as you have started plotting you can no longer change your backend without exiting the python interpreter and restarting `python` (note that in the jupyter notebook you can just press `Restart` in the `Kernel` menu).\n",
    "\n",
    "To do the equivalent in a python script would look like\n",
    "> ```\n",
@@ -264,6 +314,7 @@
    "> mpl.use(<backend>)\n",
    "> from matplotlib.pylab import *\n",
    "> ```\n",
+    "> The last line imports everything from the matplotlib.pylab module into the namespace.\n",
    "\n",
    "I start most of my notebooks or terminals with the `%pylab` command, because afterwards I can just do stuff like:"
   ]
@@ -283,6 +334,9 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
+    "The main disadvantage is that it will not be obvious to the naive reader of this code, whether functions like `linspace`, `sin`, or `plot` are originate from numpy, matplotlib, or are built-in.\n",
+    "This is why we dont recommend `from <module> import *` statements in any longer code or code you intend to share.\n",
+    "\n",
    "---\n",
    "\n",
    "## Exporting code from the Jupyter notebook\n",
@@ -333,7 +387,7 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "!python script_from_notebook.py"
+    "%run script_from_notebook.py"
   ]
  },
  {
@@ -344,7 +398,7 @@
    "\n",
    "## Exporting code from the Ipython terminal\n",
    "You can access the full history of your session using `%history`.\n",
-    "To save the history to a file use `%history -f <filename>`\n",
+    "To save the history to a file use `%history -f <filename>`.\n",
    "You will probably have to clean a lot of erroneous commands you typed from that file before you are able to run it as a script."
   ]
  }

 %% Cell type:markdown id: tags:
 # Jupyter notebook and IPython
 Our main interaction with python so far has been through the [Jupyter notebook](http://jupyter.org/).
 These notebooks are extremely popular these days within the python scientific community, however they support many more languages, such as R and octave (and even matlab with the right [plugin](https://github.com/Calysto/matlab_kernel)).
 They allow for interactive analysis of your data interspersed by explanatory notes (including LaTeX) with inline plotting.
 However, they can not be called as scripts on the command line or be imported from other python code, which makes them rather stand-alone.
 This makes them more useful for analysis that needs to be reproducible, but does not need to be replicated on different datasets (e.g., making a plot for a paper).
 For more ad-hoc analysis it can be useful to just use the command line (i.e., a REPL).
 We strongly recommend to use the IPython (available as `ipython` or `fslipython`) rather than default python REPL (available through `python` or `fslpython`)
 Both Ipython and the jupyter notebook offer a whole range of magic commands, which all start with a `%` sign.
 * A magic command starting with a single `%` sign will only affect the single line.
 * A magic command starting with two '%' signs will affect the whole block of code.
 Note that the normal python interpreter will not understand these magic commands, so you will have to take them out when writing a python script or library.
 Here we will discuss some of the many features available to you in Ipython and the Jupyter notebook
 ---
 ## Getting help
 To get the documentation for any object or method simply append a question mark
 %% Cell type:code id: tags:
 ``` 
 import string
 string.capwords?
 ```
 %% Cell type:markdown id: tags:
+This also works for any of the magic commands discussed below
+%% Cell type:code id: tags:
+``` 
+%run?
+```
+%% Cell type:markdown id: tags:
 Alternatively you can put two questions marks to get the complete code for the method or object class
 %% Cell type:code id: tags:
 ``` 
 import string
 string.capwords??
 ```
 %% Cell type:markdown id: tags:
 Both Ipython and Jupyter also come with autocomplete, which is available at any time by pressing the tab key
 ---
 ## Running shell commands
 Commands starting with a `!` will be sent to the shell rather than the python interpreter.
 %% Cell type:code id: tags:
 ``` 
 !fslstats ${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz -r
 ```
 %% Cell type:markdown id: tags:
 You can even capture the output from the shell command in a variable:
 %% Cell type:code id: tags:
 ``` 
 r = !fslstats ${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz -r
 r_lower, r_upper = [float(element) for element in r[0].split()]
-print('Bounds are ({:d}, {:d})'.format(r_lower, r_upper
+print('Bounds are ({:.0f}, {:.0f})'.format(r_lower, r_upper))
 ```
 %% Cell type:markdown id: tags:
 ---
+## Running python scripts
+We could run a python script as a shell command above. However, it will often be more convenient to use `%run` instead.
+> ```
+> %run <python script> <arguments...>
+> ```
+Arguments are provided in exactly the same way as if you called `python` in the shell. The main advantages are:
+- Any top-level variables will be made available to you after the script finishes
+- All the debugging/timing/profiling tools discussed below will be available to you
+A common workflow, when writing a python script is to have an Ipython terminal open next to your text editor and regularly use %run to test the script
+---
 ## Running other programming languages
 In the notebook you can include a whole code block using another language by using `%%<language>` (for many languages you will have to install a toolkit first, just google your favorite language besides python)
 %% Cell type:code id: tags:
 ``` 
 %%bash
 for filename in `ls *.md` ; do
    head -n 1 ${filename}
 done
 ```
 %% Cell type:markdown id: tags:
 ---
-## Timing code
+## Timing/profiling code
 We can time a line of code with `%time` or a whole code block using `%%time`.
 To get the time needed to calculate the sine of a million random numbers:
 %% Cell type:code id: tags:
 ``` 
 import numpy as np
 numbers = np.random.rand(int(1e6))
 %time np.sin(numbers)
 ```
 %% Cell type:markdown id: tags:
 For very fast evaluation, you might need to run it multiple times to get an accurate estimate. The `%timeit` (or `%%timeit` for a code block) takes care of this for you.
 %% Cell type:code id: tags:
 ``` 
 import numpy as np
 numbers = np.random.rand(10)
 %timeit np.sin(numbers)  # this will take a few seconds to run
 ```
 %% Cell type:markdown id: tags:
+Finally, if you want to figure out what part of the code is actually slowing you down you can use `%prun`, which gives you an overview of how long the interpreter spent in each method:
+%% Cell type:code id: tags:
+``` 
+import nibabel as nib
+import os.path as op
+%prun nib.load(op.expandvars('${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz'))
+```
+%% Cell type:markdown id: tags:
 ---
 ## Debugging
 Despite your best efforts in many cases some error will crop up
 %% Cell type:code id: tags:
 ``` 
 import numpy as np
 def total(a_list):
+    """Calculate the total of a list.
+    This is a very naive (not recommended) and bugged implementation
+    """
    # create local copy befor changing the input
    local_list = list(a_list)
    total = 0.
    while len(local_list) > 0:
        total += local_list.pop(1)  # returns element at index=1 and removes it
    return total
 print(total([2, 3, 4]))
 ```
 %% Cell type:markdown id: tags:
 You can always open a debugger at the location of the last error by using the `%debug` magic command. You can find a list of commands available in the debugger [here](http://www.georgejhunt.com/olpc/pydebug/pydebug/ipdb.html)
 %% Cell type:code id: tags:
 ``` 
 %debug
 ```
 %% Cell type:markdown id: tags:
 Try to check the value of `a_list` and `local_list` from within the debugger.
 > WARNING: you need to quit the debugger before any further commands will run (type `q` into the prompt)!
-If you always want to enter the debugger when an error is raised you can call `%pdb on` at any time (call `%pdf off` to rever this)
+If you always want to enter the debugger when an error is raised you can call `%pdb on` at any time (call `%pdf off` to reverse this)
 ---
 ## Enabling plotting
 By far the most popular scientific plotting library is [matplotlib](https://matplotlib.org/).
 You can enable plotting in Ipython or the jupyter notebook using `%matplotlib <backend>`, where [backend](https://matplotlib.org/faq/usage_faq.html#what-is-a-backend) is the system that will be used to display the plots.
 When failing to provide a backend it will simply use the default (which is usually fine).
 * In the jupyter notebook use the `nbagg` backend for interactive plots or the `inline` backend for non-interactive plots
 * Otherwise on Mac OSx use the `macosx` backend
 %% Cell type:code id: tags:
 ``` 
 %matplotlib nbagg
 ```
 %% Cell type:markdown id: tags:
-> Keep in mind that as soon as you have started plotting you can no longer change your backend without restarting python.
+> Keep in mind that as soon as you have started plotting you can no longer change your backend without exiting the python interpreter and restarting `python` (note that in the jupyter notebook you can just press `Restart` in the `Kernel` menu).
 To do the equivalent in a python script would look like
 > ```
 > import matplotlib as mpl
 > mpl.use(<backend>)
 > ```
 For interactive use it can be handy to have all the `numpy` numeric functions and `matplotlib` plotting functions directly available without importing them explicitly.
 This can be achieved using the `%pylab <backend>` magic command.
 %% Cell type:code id: tags:
 ``` 
 %pylab nbagg
 ```
 %% Cell type:markdown id: tags:
 This is equivalent in python code to:
 > ```
 > import matplotlib as mpl
 > mpl.use(<backend>)
 > from matplotlib.pylab import *
 > ```
+> The last line imports everything from the matplotlib.pylab module into the namespace.
 I start most of my notebooks or terminals with the `%pylab` command, because afterwards I can just do stuff like:
 %% Cell type:code id: tags:
 ``` 
 x = linspace(0, pi, 301)
 y = sin(x)
 plot(x, y, 'r-')
 ```
 %% Cell type:markdown id: tags:
+The main disadvantage is that it will not be obvious to the naive reader of this code, whether functions like `linspace`, `sin`, or `plot` are originate from numpy, matplotlib, or are built-in.
+This is why we dont recommend `from <module> import *` statements in any longer code or code you intend to share.
 ---
 ## Exporting code from the Jupyter notebook
 If you have a code cell in the jupyter notebook, that you want to convert into a script, you can use the %%writefile
 %% Cell type:code id: tags:
 ``` 
 %%writefile script_from_notebook.py
 # a bunch of imports
 import numpy as np
 from datetime import datetime
 ```
 %% Cell type:markdown id: tags:
 Any additional code cells need to contain the `-a` flag to stop jupyter from overwriting the original code
 %% Cell type:code id: tags:
 ``` 
 %%writefile -a script_from_notebook.py
 print('today is ', datetime.now())
 print('sin(3) is ', np.sin(3))
 ```
 %% Cell type:markdown id: tags:
 We can now run this script
 %% Cell type:code id: tags:
 ``` 
-!python script_from_notebook.py
+%run script_from_notebook.py
 ```
 %% Cell type:markdown id: tags:
 ---
 ## Exporting code from the Ipython terminal
 You can access the full history of your session using `%history`.
-To save the history to a file use `%history -f <filename>`
+To save the history to a file use `%history -f <filename>`.
 You will probably have to clean a lot of erroneous commands you typed from that file before you are able to run it as a script.

--- a/getting_started/07_jupyter.md
+++ b/getting_started/07_jupyter.md
@@ -130,7 +130,7 @@ Try to check the value of `a_list` and `local_list` from within the debugger.
 > WARNING: you need to quit the debugger before any further commands will run (type `q` into the prompt)!
-If you always want to enter the debugger when an error is raised you can call `%pdb on` at any time (call `%pdf off` to rever this)
+If you always want to enter the debugger when an error is raised you can call `%pdb on` at any time (call `%pdf off` to reverse this)
 ---
@@ -143,7 +143,7 @@ When failing to provide a backend it will simply use the default (which is usual
 ```
 %matplotlib nbagg
 ```
-> Keep in mind that as soon as you have started plotting you can no longer change your backend without restarting python.
+> Keep in mind that as soon as you have started plotting you can no longer change your backend without exiting the python interpreter and restarting `python` (note that in the jupyter notebook you can just press `Restart` in the `Kernel` menu).
 To do the equivalent in a python script would look like
 > ```
@@ -163,6 +163,7 @@ This is equivalent in python code to:
 > mpl.use(<backend>)
 > from matplotlib.pylab import *
 > ```
+> The last line imports everything from the matplotlib.pylab module into the namespace.
 I start most of my notebooks or terminals with the `%pylab` command, because afterwards I can just do stuff like:
 ```
@@ -170,6 +171,8 @@ x = linspace(0, pi, 301)
 y = sin(x)
 plot(x, y, 'r-')
 ```
+The main disadvantage is that it will not be obvious to the naive reader of this code, whether functions like `linspace`, `sin`, or `plot` are originate from numpy, matplotlib, or are built-in.
+This is why we dont recommend `from <module> import *` statements in any longer code or code you intend to share.
 ---