Skip to content
Snippets Groups Projects
Commit 3b8b24ea authored by Paul McCarthy's avatar Paul McCarthy :mountain_bicyclist:
Browse files

notebook prac

parent 162ce137
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
getting_started/07_jupyter.ipynb
+ 20
− 7
View file @ 3b8b24ea
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Jupyter notebook and IPython # Jupyter notebook and IPython
Our main interaction with python so far has been through the [Jupyter notebook](http://jupyter.org/).
These notebooks are extremely popular these days within the python scientific community, however they support many more languages, such as R and octave (and even matlab with the right [plugin](https://github.com/Calysto/matlab_kernel)).
They allow for interactive analysis of your data interspersed by explanatory notes (including LaTeX) with inline plotting.
However, they can not be called as scripts on the command line or be imported from other python code, which makes them rather stand-alone.
This makes them more useful for analysis that needs to be reproducible, but does not need to be replicated on different datasets (e.g., making a plot for a paper).
For more ad-hoc analysis it can be useful to just use the command line (i.e., a REPL). Our main interaction with python so far has been through the [Jupyter
We strongly recommend to use the IPython (available as `ipython` or `fslipython`) rather than default python REPL (available through `python` or `fslpython`) notebook](http://jupyter.org/). These notebooks are extremely popular these
days within the python scientific community, however they support many more
languages, such as R and octave (and even matlab with the right
[plugin](https://github.com/Calysto/matlab_kernel)). They allow for
interactive analysis of your data interspersed by explanatory notes (including
LaTeX) with inline plotting. However, they can not be called as scripts on
the command line or be imported from other python code, which makes them
rather stand-alone. This makes them more useful for analysis that needs to be
reproducible, but does not need to be replicated on different datasets (e.g.,
making a plot for a paper).
For more ad-hoc analysis it can be useful to just use the command line (i.e.,
a REPL<sup>*</sup>). We strongly recommend to use the IPython (available as
`fslipython` or `ipython`) rather than default python REPL (available through
`fslpython` or `python`), as IPython is much more user-friendly.
> <sup>*</sup>REPL = **R**ead-**E**val-**P**rint-**L**oop - the geeky term for
> an interactive prompt. You may hear younger generations using the term
> [ESRR](https://www.youtube.com/watch?v=wBoRkg5-Ieg) instead.
Both Ipython and the jupyter notebook offer a whole range of magic commands, which all start with a `%` sign. Both Ipython and the jupyter notebook offer a whole range of magic commands, which all start with a `%` sign.
* A magic command starting with a single `%` sign will only affect the single line. * A magic command starting with a single `%` sign will only affect the single line.
* A magic command starting with two '%' signs will affect the whole block of code. * A magic command starting with two '%' signs will affect the whole block of code.
Note that the normal python interpreter will not understand these magic commands, so you will have to take them out when writing a python script or library. Note that the normal python interpreter will not understand these magic commands, so you will have to take them out when writing a python script or library.
Here we will discuss some of the many features available to you in Ipython and the Jupyter notebook Here we will discuss some of the many features available to you in Ipython and the Jupyter notebook
--- ---
## Getting help ## Getting help
To get the documentation for any object or method simply append a question mark To get the documentation for any object or method simply append a question mark
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
import string import string
string.capwords? string.capwords?
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
This also works for any of the magic commands discussed below This also works for any of the magic commands discussed below
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
%run? %run?
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Alternatively you can put two questions marks to get the complete code for the method or object class Alternatively you can put two questions marks to get the complete code for the method or object class
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
import string import string
string.capwords?? string.capwords??
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Both Ipython and Jupyter also come with autocomplete, which is available at any time by pressing the tab key Both Ipython and Jupyter also come with autocomplete, which is available at any time by pressing the tab key
--- ---
## Running shell commands ## Running shell commands
Commands starting with a `!` will be sent to the shell rather than the python interpreter. Commands starting with a `!` will be sent to the shell rather than the python interpreter.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
!fslstats ${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz -r !fslstats ${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz -r
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
You can even capture the output from the shell command in a variable: You can even capture the output from the shell command in a variable:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
r = !fslstats ${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz -r r = !fslstats ${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz -r
r_lower, r_upper = [float(element) for element in r[0].split()] r_lower, r_upper = [float(element) for element in r[0].split()]
print('Bounds are ({:.0f}, {:.0f})'.format(r_lower, r_upper)) print('Bounds are ({:.0f}, {:.0f})'.format(r_lower, r_upper))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
## Running python scripts ## Running python scripts
We could run a python script as a shell command above. However, it will often be more convenient to use `%run` instead. We could run a python script as a shell command above. However, it will often be more convenient to use `%run` instead.
> ``` > ```
> %run <python script> <arguments...> > %run <python script> <arguments...>
> ``` > ```
Arguments are provided in exactly the same way as if you called `python` in the shell. The main advantages are: Arguments are provided in exactly the same way as if you called `python` in the shell. The main advantages are:
- Any top-level variables will be made available to you after the script finishes - Any top-level variables will be made available to you after the script finishes
- All the debugging/timing/profiling tools discussed below will be available to you - All the debugging/timing/profiling tools discussed below will be available to you
A common workflow, when writing a python script is to have an Ipython terminal open next to your text editor and regularly use %run to test the script A common workflow, when writing a python script is to have an Ipython terminal open next to your text editor and regularly use %run to test the script
--- ---
## Running other programming languages ## Running other programming languages
In the notebook you can include a whole code block using another language by using `%%<language>` (for many languages you will have to install a toolkit first, just google your favorite language besides python) In the notebook you can include a whole code block using another language by using `%%<language>` (for many languages you will have to install a toolkit first, just google your favorite language besides python)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
%%bash %%bash
for filename in `ls *.md` ; do for filename in `ls *.md` ; do
head -n 1 ${filename} head -n 1 ${filename}
done done
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
## Timing/profiling code ## Timing/profiling code
We can time a line of code with `%time` or a whole code block using `%%time`. We can time a line of code with `%time` or a whole code block using `%%time`.
To get the time needed to calculate the sine of a million random numbers: To get the time needed to calculate the sine of a million random numbers:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
import numpy as np import numpy as np
numbers = np.random.rand(int(1e6)) numbers = np.random.rand(int(1e6))
%time np.sin(numbers) %time np.sin(numbers)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
For very fast evaluation, you might need to run it multiple times to get an accurate estimate. The `%timeit` (or `%%timeit` for a code block) takes care of this for you. For very fast evaluation, you might need to run it multiple times to get an accurate estimate. The `%timeit` (or `%%timeit` for a code block) takes care of this for you.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
import numpy as np import numpy as np
numbers = np.random.rand(10) numbers = np.random.rand(10)
%timeit np.sin(numbers) # this will take a few seconds to run %timeit np.sin(numbers) # this will take a few seconds to run
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Finally, if you want to figure out what part of the code is actually slowing you down you can use `%prun`, which gives you an overview of how long the interpreter spent in each method: Finally, if you want to figure out what part of the code is actually slowing you down you can use `%prun`, which gives you an overview of how long the interpreter spent in each method:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
import nibabel as nib import nibabel as nib
import os.path as op import os.path as op
%prun nib.load(op.expandvars('${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz')) %prun nib.load(op.expandvars('${FSLDIR}/data/standard/FMRIB58_FA_1mm.nii.gz'))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
## Debugging ## Debugging
Despite your best efforts in many cases some error will crop up Despite your best efforts in many cases some error will crop up
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
import numpy as np import numpy as np
def total(a_list): def total(a_list):
"""Calculate the total of a list. """Calculate the total of a list.
This is a very naive (not recommended) and bugged implementation This is a very naive (not recommended) and bugged implementation
""" """
# create local copy befor changing the input # create local copy befor changing the input
local_list = list(a_list) local_list = list(a_list)
total = 0. total = 0.
while len(local_list) > 0: while len(local_list) > 0:
total += local_list.pop(1) # returns element at index=1 and removes it total += local_list.pop(1) # returns element at index=1 and removes it
return total return total
print(total([2, 3, 4])) print(total([2, 3, 4]))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
You can always open a debugger at the location of the last error by using the `%debug` magic command. You can find a list of commands available in the debugger [here](http://www.georgejhunt.com/olpc/pydebug/pydebug/ipdb.html) You can always open a debugger at the location of the last error by using the `%debug` magic command. You can find a list of commands available in the debugger [here](http://www.georgejhunt.com/olpc/pydebug/pydebug/ipdb.html)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
%debug %debug
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Try to check the value of `a_list` and `local_list` from within the debugger. Try to check the value of `a_list` and `local_list` from within the debugger.
> WARNING: you need to quit the debugger before any further commands will run (type `q` into the prompt)! > WARNING: you need to quit the debugger before any further commands will run (type `q` into the prompt)!
If you always want to enter the debugger when an error is raised you can call `%pdb on` at any time (call `%pdf off` to reverse this) If you always want to enter the debugger when an error is raised you can call `%pdb on` at any time (call `%pdf off` to reverse this)
--- ---
## Enabling plotting ## Enabling plotting
By far the most popular scientific plotting library is [matplotlib](https://matplotlib.org/). By far the most popular scientific plotting library is [matplotlib](https://matplotlib.org/).
You can enable plotting in Ipython or the jupyter notebook using `%matplotlib <backend>`, where [backend](https://matplotlib.org/faq/usage_faq.html#what-is-a-backend) is the system that will be used to display the plots. You can enable plotting in Ipython or the jupyter notebook using `%matplotlib <backend>`, where [backend](https://matplotlib.org/faq/usage_faq.html#what-is-a-backend) is the system that will be used to display the plots.
When failing to provide a backend it will simply use the default (which is usually fine). When failing to provide a backend it will simply use the default (which is usually fine).
* In the jupyter notebook use the `nbagg` backend for interactive plots or the `inline` backend for non-interactive plots * In the jupyter notebook use the `nbagg` backend for interactive plots or the `inline` backend for non-interactive plots
* Otherwise on Mac OSx use the `macosx` backend * Otherwise on Mac OSx use the `macosx` backend
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
%matplotlib nbagg %matplotlib nbagg
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
> Keep in mind that as soon as you have started plotting you can no longer change your backend without exiting the python interpreter and restarting `python` (note that in the jupyter notebook you can just press `Restart` in the `Kernel` menu). > Keep in mind that as soon as you have started plotting you can no longer change your backend without exiting the python interpreter and restarting `python` (note that in the jupyter notebook you can just press `Restart` in the `Kernel` menu).
To do the equivalent in a python script would look like To do the equivalent in a python script would look like
> ``` > ```
> import matplotlib as mpl > import matplotlib as mpl
> mpl.use(<backend>) > mpl.use(<backend>)
> ``` > ```
For interactive use it can be handy to have all the `numpy` numeric functions and `matplotlib` plotting functions directly available without importing them explicitly. For interactive use it can be handy to have all the `numpy` numeric functions and `matplotlib` plotting functions directly available without importing them explicitly.
This can be achieved using the `%pylab <backend>` magic command. This can be achieved using the `%pylab <backend>` magic command.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
%pylab nbagg %pylab nbagg
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
This is equivalent in python code to: This is equivalent in python code to:
> ``` > ```
> import matplotlib as mpl > import matplotlib as mpl
> mpl.use(<backend>) > mpl.use(<backend>)
> from matplotlib.pylab import * > from matplotlib.pylab import *
> ``` > ```
> The last line imports everything from the matplotlib.pylab module into the namespace. > The last line imports everything from the matplotlib.pylab module into the namespace.
I start most of my notebooks or terminals with the `%pylab` command, because afterwards I can just do stuff like: I start most of my notebooks or terminals with the `%pylab` command, because afterwards I can just do stuff like:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
x = linspace(0, pi, 301) x = linspace(0, pi, 301)
y = sin(x) y = sin(x)
plot(x, y, 'r-') plot(x, y, 'r-')
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The main disadvantage is that it will not be obvious to the naive reader of this code, whether functions like `linspace`, `sin`, or `plot` are originate from numpy, matplotlib, or are built-in. The main disadvantage is that it will not be obvious to the naive reader of this code, whether functions like `linspace`, `sin`, or `plot` are originate from numpy, matplotlib, or are built-in.
This is why we dont recommend `from <module> import *` statements in any longer code or code you intend to share. This is why we dont recommend `from <module> import *` statements in any longer code or code you intend to share.
--- ---
## Exporting code from the Jupyter notebook ## Exporting code from the Jupyter notebook
If you have a code cell in the jupyter notebook, that you want to convert into a script, you can use the %%writefile If you have a code cell in the jupyter notebook, that you want to convert into a script, you can use the %%writefile
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
%%writefile script_from_notebook.py %%writefile script_from_notebook.py
# a bunch of imports # a bunch of imports
import numpy as np import numpy as np
from datetime import datetime from datetime import datetime
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Any additional code cells need to contain the `-a` flag to stop jupyter from overwriting the original code Any additional code cells need to contain the `-a` flag to stop jupyter from overwriting the original code
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
%%writefile -a script_from_notebook.py %%writefile -a script_from_notebook.py
print('today is ', datetime.now()) print('today is ', datetime.now())
print('sin(3) is ', np.sin(3)) print('sin(3) is ', np.sin(3))
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
We can now run this script We can now run this script
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` ```
%run script_from_notebook.py %run script_from_notebook.py
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
## Exporting code from the Ipython terminal ## Exporting code from the Ipython terminal
You can access the full history of your session using `%history`. You can access the full history of your session using `%history`.
To save the history to a file use `%history -f <filename>`. To save the history to a file use `%history -f <filename>`.
You will probably have to clean a lot of erroneous commands you typed from that file before you are able to run it as a script. You will probably have to clean a lot of erroneous commands you typed from that file before you are able to run it as a script.
......
getting_started/07_jupyter.md
+ 20
− 8
View file @ 3b8b24ea
# Jupyter notebook and IPython # Jupyter notebook and IPython
Our main interaction with python so far has been through the [Jupyter notebook](http://jupyter.org/).
These notebooks are extremely popular these days within the python scientific community, however they support many more languages, such as R and octave (and even matlab with the right [plugin](https://github.com/Calysto/matlab_kernel)).
They allow for interactive analysis of your data interspersed by explanatory notes (including LaTeX) with inline plotting.
However, they can not be called as scripts on the command line or be imported from other python code, which makes them rather stand-alone.
This makes them more useful for analysis that needs to be reproducible, but does not need to be replicated on different datasets (e.g., making a plot for a paper).
For more ad-hoc analysis it can be useful to just use the command line (i.e., a REPL). Our main interaction with python so far has been through the [Jupyter
We strongly recommend to use the IPython (available as `ipython` or `fslipython`) rather than default python REPL (available through `python` or `fslpython`) notebook](http://jupyter.org/). These notebooks are extremely popular these
days within the python scientific community, however they support many more
languages, such as R and octave (and even matlab with the right
[plugin](https://github.com/Calysto/matlab_kernel)). They allow for
interactive analysis of your data interspersed by explanatory notes (including
LaTeX) with inline plotting. However, they can not be called as scripts on
the command line or be imported from other python code, which makes them
rather stand-alone. This makes them more useful for analysis that needs to be
reproducible, but does not need to be replicated on different datasets (e.g.,
making a plot for a paper).
For more ad-hoc analysis it can be useful to just use the command line (i.e.,
a REPL<sup>*</sup>). We strongly recommend to use the IPython (available as
`fslipython` or `ipython`) rather than default python REPL (available through
`fslpython` or `python`), as IPython is much more user-friendly.
> <sup>*</sup>REPL = **R**ead-**E**val-**P**rint-**L**oop - the geeky term for
> an interactive prompt. You may hear younger generations using the term
> [ESRR](https://www.youtube.com/watch?v=wBoRkg5-Ieg) instead.
Both Ipython and the jupyter notebook offer a whole range of magic commands, which all start with a `%` sign. Both Ipython and the jupyter notebook offer a whole range of magic commands, which all start with a `%` sign.
* A magic command starting with a single `%` sign will only affect the single line. * A magic command starting with a single `%` sign will only affect the single line.
...@@ -206,4 +219,3 @@ We can now run this script ...@@ -206,4 +219,3 @@ We can now run this script
You can access the full history of your session using `%history`. You can access the full history of your session using `%history`.
To save the history to a file use `%history -f <filename>`. To save the history to a file use `%history -f <filename>`.
You will probably have to clean a lot of erroneous commands you typed from that file before you are able to run it as a script. You will probably have to clean a lot of erroneous commands you typed from that file before you are able to run it as a script.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment