03_file_management.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# File management\n",
    "\n",
    "\n",
    "In this section we will introduce you to file management - how do we find and\n",
    "manage files, directories and paths in Python?\n",
    "\n",
    "\n",
    "Most of Python's built-in functionality for managing files and paths is spread\n",
    "across the following modules:\n",
    "\n",
    "\n",
    " - [`os`](https://docs.python.org/3.5/library/os.html)\n",
    " - [`shutil`](https://docs.python.org/3.5/library/shutil.html)\n",
    " - [`os.path`](https://docs.python.org/3.5/library/os.path.html)\n",
    " - [`glob`](https://docs.python.org/3.5/library/glob.html)\n",
    " - [`fnmatch`](https://docs.python.org/3.5/library/fnmatch.html)\n",
    "\n",
    "\n",
    "The `os` and `shutil` modules have functions allowing you to manage _files and\n",
    "directories_. The `os.path`, `glob` and `fnmatch` modules have functions for\n",
    "managing file and directory _paths_.\n",
    "\n",
    "\n",
    "> Another standard library -\n",
    "> [`pathlib`](https://docs.python.org/3.5/library/pathlib.html) - was added in\n",
    "> Python 3.4, and provides an object-oriented interface to path management. We\n",
    "> aren't going to cover `pathlib` here, but feel free to take a look at it if\n",
    "> you are into that sort of thing.\n",
    "\n",
    "\n",
    "## Contents\n",
    "\n",
    "\n",
    "If you are impatient, feel free to dive straight in to the exercises, and use the\n",
    "other sections as a reference. You might miss out on some neat tricks though.\n",
    "\n",
    "\n",
    "* [Managing files and directories](#managing-files-and-directories)\n",
    " * [Querying and changing the current directory](#querying-and-changing-the-current-directory)\n",
    " * [Directory listings](#directory-listings)\n",
    " * [Creating and removing directories](#creating-and-removing-directories)\n",
    " * [Moving and removing files](#moving-and-removing-files)\n",
    " * [Walking a directory tree](#walking-a-directory-tree)\n",
    " * [Copying, moving, and removing directory trees](#copying-moving-and-removing-directory-trees)\n",
    "* [Managing file paths](#managing-file-paths)\n",
    " * [File and directory tests](#file-and-directory-tests)\n",
    " * [Deconstructing paths](#deconstructing-paths)\n",
    " * [Absolute and relative paths](#absolute-and-relative-paths)\n",
    " * [Wildcard matching a.k.a. globbing](#wildcard-matching-aka-globbing)\n",
    " * [Expanding the home directory and environment variables](#expanding-the-home-directory-and-environment-variables)\n",
    " * [Cross-platform compatibility](#cross-platform-compatbility)\n",
    "* [Exercises](#exercises)\n",
    " * [Re-name subject directories](#re-name-subject-directories)\n",
    " * [Re-organise a data set](#re-organise-a-data-set)\n",
    " * [Solutions](#solutions)\n",
    "\n",
    "\n",
    "<a class=\"anchor\" id=\"managing-files-and-directories\"></a>\n",
    "## Managing files and directories\n",
    "\n",
    "\n",
    "The `os` module contains functions for querying and changing the current\n",
    "working directory, moving and removing individual files, and for listing,\n",
    "creating, removing, and traversing directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import os.path as op"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> If you are using a library with a long name, you can create an alias for it\n",
    "> using the `as` keyword, as we have done here for the `os.path` module.\n",
    "\n",
    "\n",
    "<a class=\"anchor\" id=\"querying-and-changing-the-current-directory\"></a>\n",
    "### Querying and changing the current directory\n",
    "\n",
    "\n",
    "You can query and change the current directory with the `os.getcwd` and\n",
    "`os.chdir` functions.\n",
    "\n",
    "\n",
    "> Here we're also going to use the `expanduser` function from the `os.path`\n",
    "> module, which allows us to expand the tilde character to the user's home\n",
    "> directory This is [covered in more detail\n",
    "> below](#expanding-the-home-directory-and-environment-variables)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cwd = os.getcwd()\n",
    "print('Current directory: {}'.format(cwd))\n",
    "\n",
    "os.chdir(op.expanduser('~'))\n",
    "print('Changed to: {}'.format(os.getcwd()))\n",
    "\n",
    "os.chdir(cwd)\n",
    "print('Changed back to: {}'.format(cwd))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the rest of this practical, we're going to use a little data set that has\n",
    "been pre-generated, and is located in a sub-directory called\n",
    "`03_file_management`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir('03_file_management')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"directory-listings\"></a>\n",
    "### Directory listings\n",
    "\n",
    "\n",
    "Use the `os.listdir` function to get a directory listing (a.k.a. the Unix `ls`\n",
    "command):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cwd = os.getcwd()\n",
    "listing = os.listdir(cwd)\n",
    "print('Directory listing: {}'.format(cwd))\n",
    "print('\\n'.join(listing))\n",
    "print()\n",
    "\n",
    "datadir = 'raw_mri_data'\n",
    "listing = os.listdir(datadir)\n",
    "print('Directory listing: {}'.format(datadir))\n",
    "print('\\n'.join(listing))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Check out the `os.scandir` function as an alternative to `os.listdir`, if\n",
    "> you have performance problems on large data sets.\n",
    "\n",
    "\n",
    "> In the code above, we used the string `join` method to print each path in\n",
    "> our directory listing on a new line. If you have a list of strings, the\n",
    "> `join` method is a handy way to insert a delimiting character or string\n",
    "> (e.g. newline, space, tab, comma - any string you want), between each string\n",
    "> in the list.\n",
    "\n",
    "\n",
    "<a class=\"anchor\" id=\"creating-and-removing-directories\"></a>\n",
    "### Creating and removing directories\n",
    "\n",
    "\n",
    "You can, not surprisingly, use the `os.mkdir` function to make a\n",
    "directory. The `os.makedirs` function is also handy - it is equivalent to\n",
    "`mkdir -p` in Unix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(os.listdir('.'))\n",
    "os.mkdir('onedir')\n",
    "os.makedirs('a/big/tree/of/directories')\n",
    "print(os.listdir('.'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `os.rmdir` and `os.removedirs` functions perform the reverse\n",
    "operations. The `os.removedirs` function will only remove empty directories,\n",
    "and you must pass it the _leaf_ directory, just like `rmdir -p` in Unix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.rmdir('onedir')\n",
    "os.removedirs('a/big/tree/of/directories')\n",
    "print(os.listdir('.'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"moving-and-removing-files\"></a>\n",
    "### Moving and removing files\n",
    "\n",
    "\n",
    "The `os.remove` and `os.rename` functions perform the equivalent of the Unix\n",
    "`rm` and `mv` commands for files. Just like in Unix, if the destination file\n",
    "you pass to `os.rename` already exists, it will be silently overwritten!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('file.txt', 'wt') as f:\n",
    "    f.write('This file contains nothing of interest')\n",
    "\n",
    "print(os.listdir())\n",
    "os.rename('file.txt', 'file2.txt')\n",
    "print(os.listdir())\n",
    "os.remove('file2.txt')\n",
    "print(os.listdir())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `os.rename` function will also work on directories, but the `shutil.move`\n",
    "function (covered below) is more flexible.\n",
    "\n",
    "\n",
    "<a class=\"anchor\" id=\"walking-a-directory-tree\"></a>\n",
    "### Walking a directory tree\n",
    "\n",
    "\n",
    "The `os.walk` function is a useful one to know about. It is a bit fiddly to\n",
    "use, but it is the best option if you need to traverse a directory tree.  It\n",
    "will recursively iterate over all of the files in a directory tree - by\n",
    "default it will traverse the tree in a breadth-first manner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# On each iteration of the loop, we get:\n",
    "#   - root:  the current directory\n",
    "#   - dirs:  a list of all sub-directories in the root\n",
    "#   - files: a list of all files in the root\n",
    "for root, dirs, files in os.walk('raw_mri_data'):\n",
    "    print('Current directory: {}'.format(root))\n",
    "    print('  Sub-directories:')\n",
    "    print('\\n'.join(['    {}'.format(d) for d in dirs]))\n",
    "    print('  Files:')\n",
    "    print('\\n'.join(['    {}'.format(f) for f in files]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note that `os.walk` does not guarantee a specific ordering in the lists of\n",
    "> files and sub-directories that it returns. However, you can force an\n",
    "> ordering quite easily - see its\n",
    "> [documentation](https://docs.python.org/3.5/library/os.html#os.walk) for\n",
    "> more details.\n",
    "\n",
    "\n",
    "If you need to traverse the directory depth-first, you can use the `topdown`\n",
    "parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for root, dirs, files in os.walk('raw_mri_data', topdown=False):\n",
    "    print('Current directory: {}'.format(root))\n",
    "    print('  Sub-directories:')\n",
    "    print('\\n'.join(['    {}'.format(d) for d in dirs]))\n",
    "    print('  Files:')\n",
    "    print('\\n'.join(['    {}'.format(f) for f in files]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Here we have explicitly named the `topdown` argument when passing it to the\n",
    "> `os.walk` function. This is referred to as a a _keyword argument_ - unnamed\n",
    "> arguments are referred to as _positional arguments_. We'll give some more\n",
    "> examples of positional and keyword arguments below.\n",
    "\n",
    "\n",
    "<a class=\"anchor\" id=\"copying-moving-and-removing-directory-trees\"></a>\n",
    "### Copying, moving, and removing directory trees\n",
    "\n",
    "\n",
    "The `shutil` module contains some higher level functions for copying and\n",
    "moving files and directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `shutil.copy` and `shutil.move` functions work just like the Unix `cp` and\n",
    "`mv` commands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copy the source file to a destination file\n",
    "src = 'raw_mri_data/subj_1/t1.nii'\n",
    "shutil.copy(src, 'subj_1_t1.nii')\n",
    "\n",
    "print(os.listdir('.'))\n",
    "\n",
    "# copy the source file to a destination directory\n",
    "os.mkdir('data_backup')\n",
    "shutil.copy('subj_1_t1.nii', 'data_backup')\n",
    "\n",
    "print(os.listdir('.'))\n",
    "print(os.listdir('data_backup'))\n",
    "\n",
    "# Move the file copy into that destination directory\n",
    "shutil.move('subj_1_t1.nii', 'data_backup/subj_1_t1_backup.nii')\n",
    "\n",
    "print(os.listdir('.'))\n",
    "print(os.listdir('data_backup'))\n",
    "\n",
    "# Move that destination directory into another directory\n",
    "os.mkdir('data_backup_backup')\n",
    "shutil.move('data_backup', 'data_backup_backup')\n",
    "\n",
    "print(os.listdir('.'))\n",
    "print(os.listdir('data_backup_backup'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `shutil.copytree` function allows you to copy entire directory trees - it\n",
    "is the equivalent of the Unix `cp -r` command. The reverse operation is provided\n",
    "by the `shutil.rmtree` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shutil.copytree('raw_mri_data', 'raw_mri_data_backup')\n",
    "print(os.listdir('.'))\n",
    "shutil.rmtree('raw_mri_data_backup')\n",
    "shutil.rmtree('data_backup_backup')\n",
    "print(os.listdir('.'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"managing-file-paths\"></a>\n",
    "## Managing file paths\n",
    "\n",
    "\n",
    "The `os.path` module contains functions for creating and manipulating file and\n",
    "directory paths, such as stripping directory prefixes and suffixes, and\n",
    "joining directory paths in a cross-platform manner. In this code, we are using\n",
    "`op` to refer to `os.path` - remember that we [created an alias\n",
    "earlier](#managing-files-and-directories).\n",
    "\n",
    "\n",
    "> Note that many of the functions in the `os.path` module do not care if your\n",
    "> path actually refers to a real file or directory - they are just\n",
    "> manipulating the path string, and will happily generate invalid or\n",
    "> non-existent paths for you.\n",
    "\n",
    "\n",
    "<a class=\"anchor\" id=\"file-and-directory-tests\"></a>\n",
    "### File and directory tests\n",
    "\n",
    "\n",
    "If you want to know whether a given path is a file, or a directory, or whether\n",
    "it exists at all, then the `os.path` module has got your back with its\n",
    "`isfile`, `isdir`, and `exists` functions. Let's define a silly function which\n",
    "will tell us what a path is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def whatisit(path, existonly=False):\n",
    "\n",
    "    print('Does {} exist? {}'.format(path, op.exists(path)))\n",
    "\n",
    "    if not existonly:\n",
    "        print('Is {} a file? {}'     .format(path, op.isfile(path)))\n",
    "        print('Is {} a directory? {}'.format(path, op.isdir( path)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This is the first time in this series of practicals that we have defined our\n",
    "> own function, [hooray!](https://www.youtube.com/watch?v=zQiibNVIvK4) All\n",
    "> function definitions in Python begin with the `def` keyword:\n",
    ">\n",
    "> ```\n",
    "> def myfunction():\n",
    ">     function_body\n",
    "> ```\n",
    ">\n",
    "> Just like with other control flow tools, such as `if`, `for`, and `while`\n",
    "> statements, the body of a function must be indented (with four spaces\n",
    "> please!).\n",
    ">\n",
    "> Python functions can be written to accept any number of arguments:\n",
    ">\n",
    "> ```\n",
    "> def myfunction(arg1, arg2, arg3):\n",
    ">     function_body\n",
    "> ```\n",
    ">\n",
    "> Arguments can also be given default values:\n",
    ">\n",
    "> ```\n",
    "> def myfunction(arg1, arg2, arg3=False):\n",
    ">     function_body\n",
    "> ```\n",
    ">\n",
    "> In our `whatisit` function above, we gave the `existonly` argument (which\n",
    "> controls whether the path is only tested for existence) a default value.\n",
    "> This makes the `existonly` argument optional - we can call `whatisit` either\n",
    "> with or without this argument.\n",
    ">\n",
    "> To return a value from a function, use the `return` keyword:\n",
    ">\n",
    "> ```\n",
    "> def add(n1, n2):\n",
    ">     return n1 + n2\n",
    "> ```\n",
    ">\n",
    "> Take a look at the [official Python\n",
    "> tutorial](https://docs.python.org/3.5/tutorial/controlflow.html#defining-functions)\n",
    "> for more details on defining your own functions.\n",
    "\n",
    "\n",
    "Now let's use that function to test some paths. Here we are using the\n",
    "`op.join` function to construct paths - it is [covered\n",
    "below](#cross-platform-compatbility):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dirname  = op.join('raw_mri_data')\n",
    "filename = op.join('raw_mri_data', 'subj_1', 't1.nii')\n",
    "nonexist = op.join('very', 'unlikely', 'to', 'exist')\n",
    "\n",
    "whatisit(dirname)\n",
    "whatisit(filename)\n",
    "whatisit(nonexist)\n",
    "whatisit(nonexist, existonly=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"deconstructing-paths\"></a>\n",
    "### Deconstructing paths\n",
    "\n",
    "\n",
    "If you are only interested in the directory or file component of a path then\n",
    "the `os.path` module has the `dirname`, `basename`, and `split` functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = '/path/to/my/image.nii'\n",
    "\n",
    "print('Directory name:           {}'.format(op.dirname( path)))\n",
    "print('Base name:                {}'.format(op.basename(path)))\n",
    "print('Directory and base names: {}'.format(op.split(   path)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note here that `op.split` returns both the directory and base names - it is\n",
    "> super easy to define a Python function that returns multiple values, simply by\n",
    "> having it return a tuple. For example, the implementation of `op.split` might\n",
    "> look something like this:\n",
    ">\n",
    ">\n",
    "> ```\n",
    "> def mysplit(path):\n",
    ">     dirname  = op.dirname(path)\n",
    ">     basename = op.basename(path)\n",
    ">\n",
    ">     # It is not necessary to use round brackets here\n",
    ">     # to denote the tuple - the return values will\n",
    ">     # be implicitly grouped into a tuple for us.\n",
    ">     return dirname, basename\n",
    "> ```\n",
    ">\n",
    ">\n",
    "> When calling a function which returns multiple values, you can _unpack_ those\n",
    "> values in a single statement like so:\n",
    ">\n",
    ">\n",
    "> ```\n",
    "> dirname, basename = mysplit(path)\n",
    ">\n",
    "> print('Directory name: {}'.format(dirname))\n",
    "> print('Base name:      {}'.format(basename))\n",
    "> ```\n",
    "\n",
    "\n",
    "If you want to extract the prefix or suffix of a file, you can use `splitext`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prefix, suffix = op.splitext('image.nii')\n",
    "\n",
    "print('Prefix: {}'.format(prefix))\n",
    "print('Suffix: {}'.format(suffix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Double-barrelled file suffixes (e.g. `.nii.gz`) are the work of the devil.\n",
    "> Correct handling of them is an open problem in Computer Science, and is\n",
    "> considered by many to be unsolvable.  For `imglob`, `imcp`, and `immv`-like\n",
    "> functionality, check out the `fsl.utils.path` and `fsl.utils.imcp` modules,\n",
    "> part of the [`fslpy` project](https://pypi.python.org/pypi/fslpy).\n",
    "\n",
    "\n",
    "<a class=\"anchor\" id=\"absolute-and-relative-paths\"></a>\n",
    "### Absolute and relative paths\n",
    "\n",
    "\n",
    "The `os.path` module has three useful functions for converting between\n",
    "absolute and relative paths. The `op.abspath` and `op.relpath` functions will\n",
    "respectively turn the provided path into an equivalent absolute or relative\n",
    "path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = op.abspath('relative/path/to/some/file.txt')\n",
    "\n",
    "print('Absolutised: {}'.format(path))\n",
    "print('Relativised: {}'.format(op.relpath(path)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the `op.abspath` and `op.relpath` functions work relative to the\n",
    "current working directory. The `op.relpath` function allows you to specify a\n",
    "different directory to work from, and another function - `op.normpath` -\n",
    "allows you create absolute paths with a different starting\n",
    "point. `op.normpath` will take care of removing duplicate back-slashes,\n",
    "and resolving references to `\".\"` and `\"..\"`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = 'relative/path/to/some/file.txt'\n",
    "root = '/vols/Data/'\n",
    "abspath = op.normpath(op.join(root, path))\n",
    "\n",
    "print('Absolute path: {}'.format(abspath))\n",
    "print('Relative path: {}'.format(op.relpath(abspath, root)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"wildcard-matching-aka-globbing\"></a>\n",
    "### Wildcard matching a.k.a. globbing\n",
    "\n",
    "\n",
    "The `glob` module has a function, also called `glob`, which allows you to find\n",
    "files, based on unix-style wildcard pattern matching."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glob import glob\n",
    "\n",
    "root = 'raw_mri_data'\n",
    "\n",
    "# find all niftis for subject 1\n",
    "images = glob(op.join(root, 'subj_1', '*.nii*'))\n",
    "\n",
    "print('Subject #1 images:')\n",
    "print('\\n'.join(['  {}'.format(i) for i in images]))\n",
    "\n",
    "# find all subject directories\n",
    "subjdirs = glob(op.join(root, 'subj_*'))\n",
    "\n",
    "print('Subject directories:')\n",
    "print('\\n'.join(['  {}'.format(d) for d in subjdirs]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with [`os.walk`](walking-a-directory-tree), the order of the results\n",
    "returned by `glob` is arbitrary. Unfortunately the undergraduate who\n",
    "acquired this specific data set did not think to use zero-padded subject IDs\n",
    "(you'll be pleased to know that this student was immediately kicked out of his\n",
    "college and banned from ever returning), so we can't simply sort the paths\n",
    "alphabetically. Instead, let's use some trickery to sort the subject\n",
    "directories numerically by ID:\n",
    "\n",
    "\n",
    "Let's define a function which, given a subject directory, returns the numeric\n",
    "subject ID:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_subject_id(subjdir):\n",
    "\n",
    "    # Remove any leading directories (e.g. \"raw_mri_data/\")\n",
    "    subjdir = op.basename(subjdir)\n",
    "\n",
    "    # Split \"subj_[id]\" into two words\n",
    "    prefix, sid = subjdir.split('_')\n",
    "\n",
    "    # return the subject ID as an integer\n",
    "    return int(sid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function works like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(get_subject_id('raw_mri_data/subj_9'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have this function, we can sort the directories in one line of\n",
    "code, via the built-in\n",
    "[`sorted`](https://docs.python.org/3.5/library/functions.html#sorted)\n",
    "function.  The directories will be sorted according to the `key` function that\n",
    "we specify, which provides a mapping from each directory to a sortable\n",
    "&quot;key&quot;:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "subjdirs = sorted(subjdirs, key=get_subject_id)\n",
    "print('Subject directories, sorted by ID:')\n",
    "print('\\n'.join(['  {}'.format(d) for d in subjdirs]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note that in Python, we can pass a function around just like any other\n",
    "> variable - we passed the `get_subject_id` function as an argument to the\n",
    "> `sorted` function. This is possible (and normal) because functions are\n",
    "> [first class citizens](https://en.wikipedia.org/wiki/First-class_citizen) in\n",
    "> Python!\n",
    "\n",
    "\n",
    "As of Python 3.5, `glob` also supports recursive pattern matching via the\n",
    "`recursive` flag. Let's say we want a list of all resting-state scans in our\n",
    "data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rscans = glob('raw_mri_data/**/rest.nii.gz', recursive=True)\n",
    "\n",
    "print('Resting state scans:')\n",
    "print('\\n'.join(rscans))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Internally, the `glob` module uses the `fnmatch` module, which implements the\n",
    "pattern matching logic.\n",
    "\n",
    "* If you are searching your file system for files and directory, use\n",
    "  `glob.glob`.\n",
    "\n",
    "* But if you already have a set of paths, you can use the `fnmatch.fnmatch`\n",
    "  and `fnmatch.filter` functions to identify which paths match your pattern.\n",
    "\n",
    "\n",
    "Note that the syntax used by `glob` and `fnmatch` is similar, but __not__\n",
    "identical to the syntax that you are used to from `bash`. Refer to the\n",
    "[`fnmatch` module](https://docs.python.org/3.5/library/fnmatch.html)\n",
    "documentation for details. If you need more complicated pattern matching, you\n",
    "can use regular expressions, available via the [`re`\n",
    "module](https://docs.python.org/3.5/library/re.html).\n",
    "\n",
    "\n",
    "For example, let's retrieve all images that are in our data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "allimages = glob(op.join('raw_mri_data', '**', '*.nii*'), recursive=True)\n",
    "print('All images in experiment:')\n",
    "\n",
    "# Let's just print the first and last few\n",
    "print('\\n'.join(['  {}'.format(i) for i in allimages[:3]]))\n",
    "print('  .')\n",
    "print('  .')\n",
    "print('  .')\n",
    "print('\\n'.join(['  {}'.format(i) for i in allimages[-3:]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's reduce that list to only those images which are uncompressed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fnmatch\n",
    "\n",
    "# filter a list of images\n",
    "uncompressed = fnmatch.filter(allimages, '*.nii')\n",
    "print('All uncompressed images:')\n",
    "print('\\n'.join(['  {}'.format(i) for i in uncompressed]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And further reduce the list by identifying which of these images are T1 scans,\n",
    "and are from our fictional patient group, made up of subjects 1, 4, 7, 8, and\n",
    "9:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients = [1, 4, 7, 8, 9]\n",
    "\n",
    "print('All uncompressed T1 images from patient group:')\n",
    "for filename in uncompressed:\n",
    "\n",
    "    fullfile = filename\n",
    "    dirname, filename = op.split(fullfile)\n",
    "    subjid = get_subject_id(dirname)\n",
    "\n",
    "    if subjid in patients and fnmatch.fnmatch(filename, 't1.*'):\n",
    "        print('  {}'.format(fullfile))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"expanding-the-home-directory-and-environment-variables\"></a>\n",
    "### Expanding the home directory and environment variables\n",
    "\n",
    "\n",
    "You have [already been\n",
    "introduced](#querying-and-changing-the-current-directory) to the\n",
    "`op.expanduser` function. Another handy function is the `op.expandvars`\n",
    "function, which will expand expand any environment variables in a path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(op.expanduser('~'))\n",
    "print(op.expandvars('$HOME'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"cross-platform-compatbility\"></a>\n",
    "### Cross-platform compatibility\n",
    "\n",
    "\n",
    "If you are the type of person who likes running code on both Windows and Unix\n",
    "machines, you will be delighted to learn that the `os`  module has a couple\n",
    "of useful attributes:\n",
    "\n",
    "\n",
    " - `os.sep` contains the separator character that is used in file paths on\n",
    "   your platform (i.e. &#47; on Unix, &#92; on Windows).\n",
    " - `os.pathsep` contains the separator character that is used when creating\n",
    "   path lists (e.g. on your `$PATH`  environment variable - &#58; on Unix,\n",
    "   and &#58; on Windows).\n",
    "\n",
    "\n",
    "You will also find the `op.join` function handy. Given a set of directory\n",
    "and/or file names, it will construct a path suited to the platform that you\n",
    "are running on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = op.join('home', 'fsluser', '.bash_profile')\n",
    "\n",
    "# if you are on Unix, you will get 'home/fsluser/.bash_profile'.\n",
    "# On windows, you will get 'home\\\\fsluser\\\\.bash_profile'\n",
    "print(path)\n",
    "\n",
    "# Tn create an absolute path from\n",
    "# the file system root, use os.sep:\n",
    "print(op.join(op.sep, 'home', 'fsluser', '.bash_profile'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"exercises\"></a>\n",
    "## Exercises\n",
    "\n",
    "\n",
    "<a class=\"anchor\" id=\"re-name-subject-directories\"></a>\n",
    "### Re-name subject directories\n",
    "\n",
    "\n",
    "Write a function which can rename the subject directories in `raw_mri_data` so\n",
    "that the subject IDs are padded with zeros, and thus will be able to be sorted\n",
    "alphabetically. This function:\n",
    "\n",
    "\n",
    "  - Should accept the path to the parent directory of the data set\n",
    "    (`raw_mri_data` in this case).\n",
    "  - Should be able to handle any number of subjects\n",
    "    > Hint: `numpy.log10`\n",
    "\n",
    "  - May assume that the subject directory names follow the pattern\n",
    "    `subj_[id]`, where `[id]` is the integer subject ID.\n",
    "\n",
    "\n",
    "<a class=\"anchor\" id=\"re-organise-a-data-set\"></a>\n",
    "### Re-organise a data set\n",
    "\n",
    "\n",
    "Write a function which can be used to separate the data for each group\n",
    "(patients: 1, 4, 7, 8, 9, and controls: 2, 3, 5, 6, 10) into sub-directories\n",
    "`CON` and `PAT`.\n",
    "\n",
    "This function should work with any number of groups, and should accept three\n",
    "parameters:\n",
    "\n",
    " - The root directory of the data set (e.g. `raw_mri_data`).\n",
    " - A list of strings, the labels for each group.\n",
    " - A list of lists, with each list containing the subject IDs for one group.\n",
    "\n",
    "\n",
    "__Extra exercises:__ If you are looking for something more to do, you can find\n",
    "some more exercises in the file `03_file_management_extra.md`.\n",
    "\n",
    "\n",
    "<a class=\"anchor\" id=\"solutions\"></a>\n",
    "### Solutions\n",
    "\n",
    "\n",