diff --git a/advanced_topics/context_managers.ipynb b/advanced_topics/context_managers.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..0642b866b5bbd8dbe0cccbff56e81647f123b511 --- /dev/null +++ b/advanced_topics/context_managers.ipynb @@ -0,0 +1,462 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Context managers\n", + "\n", + "\n", + "The recommended way to open a file in Python is via the `with` statement:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with open('context_managers.md', 'rt') as f:\n", + " firstlines = f.readlines()[:4]\n", + " firstlines = [l.strip() for l in firstlines]\n", + " print('\\n'.join(firstlines))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This is because the `with` statement ensures that the file will be closed\n", + "automatically, even if an error occurs inside the `with` statement.\n", + "\n", + "\n", + "The `with` statement is obviously hiding some internal details from us. But\n", + "these internals are in fact quite straightforward, and are known as [_context\n", + "managers_](https://docs.python.org/3.5/reference/datamodel.html#context-managers).\n", + "\n", + "\n", + "## Anatomy of a context manager\n", + "\n", + "\n", + "A _context manager_ is simply an object which has two specially named methods\n", + "`__enter__` and `__exit__`. Any object which has these methods can be used in\n", + "a `with` statement.\n", + "\n", + "\n", + "Let's define a context manager class that we can play with:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "class MyContextManager(object):\n", + " def __enter__(self):\n", + " print('In enter')\n", + " def __exit__(self, *args):\n", + " print('In exit')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now, what happens when we use `MyContextManager` in a `with` statement?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with MyContextManager():\n", + " print('In with block')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So the `__enter__` method is called before the statements in the `with` block,\n", + "and the `__exit__` method is called afterwards.\n", + "\n", + "\n", + "Context managers are that simple. What makes them really useful though, is\n", + "that the `__exit__` method will be called even if the code in the `with` block\n", + "raises an error. The error will be held, and only raised after the `__exit__`\n", + "method has finished:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with MyContextManager():\n", + " print('In with block')\n", + " assert 1 == 0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This means that we can use context managers to perform any sort of clean up or\n", + "finalisation logic that we always want to have executed.\n", + "\n", + "\n", + "### Why not just use `try ... finally`?\n", + "\n", + "\n", + "Context managers do not provide anything that cannot be accomplished in other\n", + "ways. For example, we could accomplish very similar behaviour using\n", + "[`try` - `finally` logic](https://docs.python.org/3.5/tutorial/errors.html#handling-exceptions) -\n", + "the statements in the `finally` clause will *always* be executed, whether an\n", + "error is raised or not:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print('Before try block')\n", + "try:\n", + " print('In try block')\n", + " assert 1 == 0\n", + "finally:\n", + " print('In finally block')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "But context managers have the advantage that you can implement your clean-up\n", + "logic in one place, and re-use it as many times as you want.\n", + "\n", + "\n", + "## Uses for context managers\n", + "\n", + "\n", + "We have alraedy talked about how context managers can be used to perform any\n", + "task which requires some initialistion logic, and/or some clean-up logic. As an\n", + "example, here is a context manager which creates a temporary directory,\n", + "and then makes sure that it is deleted afterwards." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import shutil\n", + "import tempfile\n", + "\n", + "class TempDir(object):\n", + "\n", + " def __enter__(self):\n", + "\n", + " self.tempDir = tempfile.mkdtemp()\n", + " self.prevDir = os.getcwd()\n", + "\n", + " print('Changing to temp dir: {}'.format(self.tempDir))\n", + " print('Previous directory: {}'.format(self.prevDir))\n", + "\n", + " os.chdir(self.tempDir)\n", + "\n", + " def __exit__(self, *args):\n", + "\n", + " print('Changing back to: {}'.format(self.prevDir))\n", + " print('Removing temp dir: {}'.format(self.tempDir))\n", + "\n", + " os .chdir( self.prevDir)\n", + " shutil.rmtree(self.tempDir)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now imagine that we have a function which loads data from a file, and performs\n", + "some calculation on it:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "def complexAlgorithm(infile):\n", + " data = np.loadtxt(infile)\n", + " return data.mean()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We could use the `TempDir` context manager to write a test case for this\n", + "function, and not have to worry about cleaning up the test data:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with TempDir():\n", + " print('Testing complex algorithm')\n", + "\n", + " data = np.random.random((100, 100))\n", + " np.savetxt('data.txt', data)\n", + "\n", + " result = complexAlgorithm('data.txt')\n", + "\n", + " assert result > 0.1 and result < 0.9\n", + " print('Test passed (result: {})'.format(result))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Handling errors in `__exit__`\n", + "\n", + "\n", + "By now you must be [panicking](https://youtu.be/cSU_5MgtDc8?t=9) about why I\n", + "haven't mentioned those conspicuous `*args` that get passed to the`__exit__`\n", + "method. It turns out that a context manager's [`__exit__`\n", + "method](https://docs.python.org/3.5/reference/datamodel.html#object.__exit__)\n", + "is always passed three arguments.\n", + "\n", + "\n", + "Let's adjust our `MyContextManager` class a little so we can see what these\n", + "arguments are for:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "class MyContextManager(object):\n", + " def __enter__(self):\n", + " print('In enter')\n", + "\n", + " def __exit__(self, arg1, arg2, arg3):\n", + " print('In exit')\n", + " print(' arg1: ', arg1)\n", + " print(' arg2: ', arg2)\n", + " print(' arg3: ', arg3)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If the code inside the `with` statement does not raise an error, these three\n", + "arguments will all be `None`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with MyContextManager():\n", + " print('In with block')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "However, if the code inside the `with` statement raises an error, things look\n", + "a little different:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with MyContextManager():\n", + " print('In with block')\n", + " raise ValueError('Oh no!')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So when an error occurs, the `__exit__` method is passed the following:\n", + "\n", + "- The [`Exception`](https://docs.python.org/3.5/tutorial/errors.html)\n", + " type that was raised.\n", + "- The `Exception` instance that was raised.\n", + "- A [`traceback`](https://docs.python.org/3.5/library/traceback.html) object\n", + " which can be used to get more information about the exception (e.g. line\n", + " number).\n", + "\n", + "\n", + "### Suppressing errors with `__exit__`\n", + "\n", + "\n", + "The `__exit__` method is also capable of suppressing errors - if it returns a\n", + "value of `True`, then any error that was raised will be ignored. For example,\n", + "we could write a context manager which ignores any assertion errors, but\n", + "allows other errors to halt execution as normal:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "class MyContextManager(object):\n", + " def __enter__(self):\n", + " print('In enter')\n", + "\n", + " def __exit__(self, arg1, arg2, arg3):\n", + " print('In exit')\n", + " if issubclass(arg1, AssertionError):\n", + " return True\n", + " print(' arg1: ', arg1)\n", + " print(' arg2: ', arg2)\n", + " print(' arg3: ', arg3)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> Note that if a function or method does not explicitly return a value, its\n", + "> return value is `None` (which would evaluate to `False` when converted to a\n", + "> `bool`). Also note that we are using the built-in\n", + "> [`issubclass`](https://docs.python.org/3.5/library/functions.html#issubclass)\n", + "> function, which allows us to test the type of a class.\n", + "\n", + "\n", + "Now, when we use `MyContextManager`, any assertion errors are suppressed,\n", + "whereas other errors will be raised as normal:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with MyContextManager():\n", + " assert 1 == 0\n", + "\n", + "print('Continuing execution!')\n", + "\n", + "with MyContextManager():\n", + " raise ValueError('Oh no!')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Functions as context managers\n", + "\n", + "\n", + "In fact, there is another method of defining a context manager in Python. The\n", + "built-in [`contextlib`\n", + "module](https://docs.python.org/3.5/library/contextlib.html#contextlib.contextmanager)\n", + "has a decorator called `contextmanager`, which allows us to turn __any__\n", + "function into a context manager. The only requirement is that the function must\n", + "have a `yield` statement<sup>1</sup>. So we could rewrite our `TempDir` class\n", + "from above as a function:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import shutil\n", + "import tempfile\n", + "import contextlib\n", + "\n", + "@contextlib.contextmanager\n", + "def tempdir():\n", + " testdir = tempfile.mkdtemp()\n", + " prevdir = os.getcwd()\n", + " try:\n", + "\n", + " os.chdir(testdir)\n", + " yield testdir\n", + "\n", + " finally:\n", + " os.chdir(prevdir)\n", + " shutil.rmtree(testdir)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This new `tempdir` function is used in exactly the same way as our `TempDir`\n", + "class:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print('In directory: {}'.format(os.getcwd()))\n", + "\n", + "with tempdir():\n", + " print('Now in directory: {}'.format(os.getcwd()))\n", + "\n", + "print('Back in directory: {}'.format(os.getcwd()))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> <sup>1</sup>\n", + "\n", + "> https://docs.python.org/3.5/howto/functional.html#generators\n", + "\n", + "\n", + "## Nesting context managers\n", + "\n", + "\n", + "\n", + "\n", + "Useful references\n", + "\n", + "* [Context manager clases](https://docs.python.org/3.5/reference/datamodel.html#context-managers)\n", + "* The [`contextlib` module](https://docs.python.org/3.5/library/contextlib.html)" + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/advanced_topics/context_managers.md b/advanced_topics/context_managers.md new file mode 100644 index 0000000000000000000000000000000000000000..5b3585a943d600339c7d705d8e796efe4a8b5190 --- /dev/null +++ b/advanced_topics/context_managers.md @@ -0,0 +1,330 @@ +# Context managers + + +The recommended way to open a file in Python is via the `with` statement: + + +``` +with open('context_managers.md', 'rt') as f: + firstlines = f.readlines()[:4] + firstlines = [l.strip() for l in firstlines] + print('\n'.join(firstlines)) +``` + + +This is because the `with` statement ensures that the file will be closed +automatically, even if an error occurs inside the `with` statement. + + +The `with` statement is obviously hiding some internal details from us. But +these internals are in fact quite straightforward, and are known as [_context +managers_](https://docs.python.org/3.5/reference/datamodel.html#context-managers). + + +## Anatomy of a context manager + + +A _context manager_ is simply an object which has two specially named methods +`__enter__` and `__exit__`. Any object which has these methods can be used in +a `with` statement. + + +Let's define a context manager class that we can play with: + + +``` +class MyContextManager(object): + def __enter__(self): + print('In enter') + def __exit__(self, *args): + print('In exit') +``` + + +Now, what happens when we use `MyContextManager` in a `with` statement? + + +``` +with MyContextManager(): + print('In with block') +``` + + +So the `__enter__` method is called before the statements in the `with` block, +and the `__exit__` method is called afterwards. + + +Context managers are that simple. What makes them really useful though, is +that the `__exit__` method will be called even if the code in the `with` block +raises an error. The error will be held, and only raised after the `__exit__` +method has finished: + + +``` +with MyContextManager(): + print('In with block') + assert 1 == 0 +``` + + +This means that we can use context managers to perform any sort of clean up or +finalisation logic that we always want to have executed. + + +### Why not just use `try ... finally`? + + +Context managers do not provide anything that cannot be accomplished in other +ways. For example, we could accomplish very similar behaviour using +[`try` - `finally` logic](https://docs.python.org/3.5/tutorial/errors.html#handling-exceptions) - +the statements in the `finally` clause will *always* be executed, whether an +error is raised or not: + + +``` +print('Before try block') +try: + print('In try block') + assert 1 == 0 +finally: + print('In finally block') +``` + + +But context managers have the advantage that you can implement your clean-up +logic in one place, and re-use it as many times as you want. + + +## Uses for context managers + + +We have alraedy talked about how context managers can be used to perform any +task which requires some initialistion logic, and/or some clean-up logic. As an +example, here is a context manager which creates a temporary directory, +and then makes sure that it is deleted afterwards. + + +``` +import os +import shutil +import tempfile + +class TempDir(object): + + def __enter__(self): + + self.tempDir = tempfile.mkdtemp() + self.prevDir = os.getcwd() + + print('Changing to temp dir: {}'.format(self.tempDir)) + print('Previous directory: {}'.format(self.prevDir)) + + os.chdir(self.tempDir) + + def __exit__(self, *args): + + print('Changing back to: {}'.format(self.prevDir)) + print('Removing temp dir: {}'.format(self.tempDir)) + + os .chdir( self.prevDir) + shutil.rmtree(self.tempDir) +``` + + +Now imagine that we have a function which loads data from a file, and performs +some calculation on it: + + +``` +import numpy as np + +def complexAlgorithm(infile): + data = np.loadtxt(infile) + return data.mean() +``` + + +We could use the `TempDir` context manager to write a test case for this +function, and not have to worry about cleaning up the test data: + + +``` +with TempDir(): + print('Testing complex algorithm') + + data = np.random.random((100, 100)) + np.savetxt('data.txt', data) + + result = complexAlgorithm('data.txt') + + assert result > 0.1 and result < 0.9 + print('Test passed (result: {})'.format(result)) +``` + + +### Handling errors in `__exit__` + + +By now you must be [panicking](https://youtu.be/cSU_5MgtDc8?t=9) about why I +haven't mentioned those conspicuous `*args` that get passed to the`__exit__` +method. It turns out that a context manager's [`__exit__` +method](https://docs.python.org/3.5/reference/datamodel.html#object.__exit__) +is always passed three arguments. + + +Let's adjust our `MyContextManager` class a little so we can see what these +arguments are for: + + +``` +class MyContextManager(object): + def __enter__(self): + print('In enter') + + def __exit__(self, arg1, arg2, arg3): + print('In exit') + print(' arg1: ', arg1) + print(' arg2: ', arg2) + print(' arg3: ', arg3) +``` + + +If the code inside the `with` statement does not raise an error, these three +arguments will all be `None`. + + +``` +with MyContextManager(): + print('In with block') +``` + + +However, if the code inside the `with` statement raises an error, things look +a little different: + + +``` +with MyContextManager(): + print('In with block') + raise ValueError('Oh no!') +``` + + +So when an error occurs, the `__exit__` method is passed the following: + +- The [`Exception`](https://docs.python.org/3.5/tutorial/errors.html) + type that was raised. +- The `Exception` instance that was raised. +- A [`traceback`](https://docs.python.org/3.5/library/traceback.html) object + which can be used to get more information about the exception (e.g. line + number). + + +### Suppressing errors with `__exit__` + + +The `__exit__` method is also capable of suppressing errors - if it returns a +value of `True`, then any error that was raised will be ignored. For example, +we could write a context manager which ignores any assertion errors, but +allows other errors to halt execution as normal: + + +``` +class MyContextManager(object): + def __enter__(self): + print('In enter') + + def __exit__(self, arg1, arg2, arg3): + print('In exit') + if issubclass(arg1, AssertionError): + return True + print(' arg1: ', arg1) + print(' arg2: ', arg2) + print(' arg3: ', arg3) +``` + +> Note that if a function or method does not explicitly return a value, its +> return value is `None` (which would evaluate to `False` when converted to a +> `bool`). Also note that we are using the built-in +> [`issubclass`](https://docs.python.org/3.5/library/functions.html#issubclass) +> function, which allows us to test the type of a class. + + +Now, when we use `MyContextManager`, any assertion errors are suppressed, +whereas other errors will be raised as normal: + + +``` +with MyContextManager(): + assert 1 == 0 + +print('Continuing execution!') + +with MyContextManager(): + raise ValueError('Oh no!') +``` + + +## Functions as context managers + + +In fact, there is another method of defining a context manager in Python. The +built-in [`contextlib` +module](https://docs.python.org/3.5/library/contextlib.html#contextlib.contextmanager) +has a decorator called `contextmanager`, which allows us to turn __any__ +function into a context manager. The only requirement is that the function must +have a `yield` statement<sup>1</sup>. So we could rewrite our `TempDir` class +from above as a function: + + +``` +import os +import shutil +import tempfile +import contextlib + +@contextlib.contextmanager +def tempdir(): + testdir = tempfile.mkdtemp() + prevdir = os.getcwd() + try: + + os.chdir(testdir) + yield testdir + + finally: + os.chdir(prevdir) + shutil.rmtree(testdir) +``` + +This new `tempdir` function is used in exactly the same way as our `TempDir` +class: + + +``` +print('In directory: {}'.format(os.getcwd())) + +with tempdir(): + print('Now in directory: {}'.format(os.getcwd())) + +print('Back in directory: {}'.format(os.getcwd())) +``` + + + + + +> <sup>1</sup> + +> https://docs.python.org/3.5/howto/functional.html#generators + + +## Nesting context managers + + + + +Useful references + +* [Context manager clases](https://docs.python.org/3.5/reference/datamodel.html#context-managers) +* The [`contextlib` module](https://docs.python.org/3.5/library/contextlib.html) \ No newline at end of file