Newer
Older
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Structuring a Python project\n",
"\n",
"\n",
"If you are writing code that you are sure will never be seen or used by\n",
"anybody else, then you can structure your project however you want, and you\n",
"can stop reading now.\n",
"\n",
"\n",
"However, if you are intending to make your code available for others to use,\n",
"either as end users, or as a dependency of their own code, you will make their\n",
"lives much easier if you spend a little time organising your project\n",
"directory.\n",
"\n",
"\n",
"* [Recommended project structure](#recommended-project-structure)\n",
" * [The `mypackage/` directory](#the-mypackage-directory)\n",
" * [`README`](#readme)\n",
" * [`LICENSE`](#license)\n",
" * [`requirements.txt`](#requirements-txt)\n",
" * [`setup.py`](#setup-py)\n",
"* [Appendix: Tests](#appendix-tests)\n",
"* [Appendix: Versioning](#appendix-versioning)\n",
" * [Include the version in your code](#include-the-version-in-your-code)\n",
" * [Deprecate, don't remove!](#deprecate-dont-remove)\n",
"* [Appendix: Cookiecutter](#appendix-cookiecutter)\n",
"\n",
"\n",
"Official documentation:\n",
"https://packaging.python.org/tutorials/distributing-packages/\n",
"\n",
"\n",
"<a class=\"anchor\" id=\"recommended-project-structure\"></a>\n",
"## Recommended project structure\n",
"\n",
"\n",
"A Python project directory should, at the very least, have a structure that\n",
"resembles the following:\n",
"\n",
"\n",
"> ```\n",
"> myproject/\n",
"> mypackage/\n",
"> __init__.py\n",
"> mymodule.py\n",
"> README\n",
"> LICENSE\n",
"> requirements.txt\n",
"> setup.py\n",
"> ```\n",
"\n",
"\n",
"This example structure is in the `example_project/` sub-directory - have a\n",
"look through it if you like.\n",
"\n",
"\n",
"<a class=\"anchor\" id=\"the-mypackage-directory\"></a>\n",
"### The `mypackage/` directory\n",
"\n",
"\n",
"The first thing you should do is make sure that all of your python code is\n",
"organised into a sensibly-named\n",
"[_package_](https://docs.python.org/3.5/tutorial/modules.html#packages). This\n",
"is important, because it greatly reduces the possibility of naming collisions\n",
"when people install your library alongside other libraries. Hands up those of\n",
"you who have ever written a file called `utils.[py|m|c|cpp]`!\n",
"\n",
"\n",
"Check out the `advanced_topics/02_modules_and_packages.ipynb` practical for\n",
"more details on packages in Python.\n",
"\n",
"\n",
"<a class=\"anchor\" id=\"readme\"></a>\n",
"### `README`\n",
"\n",
"\n",
"Every project should have a README file. This is simply a plain text file\n",
"which describes your project and how to use it. It is common and acceptable\n",
"for a README file to be written in plain text,\n",
"[reStructuredText](http://www.sphinx-doc.org/en/stable/rest.html)\n",
"(`README.rst`), or\n",
"[markdown](https://guides.github.com/features/mastering-markdown/)\n",
"(`README.md`).\n",
"\n",
"\n",
"<a class=\"anchor\" id=\"license\"></a>\n",
"### `LICENSE`\n",
"\n",
"\n",
"Having a LICENSE file makes it easy for people to understand the constraints\n",
"under which your code can be used.\n",
"\n",
"\n",
"<a class=\"anchor\" id=\"requirements-txt\"></a>\n",
"### `requirements.txt`\n",
"\n",
"\n",
"This file is not strictly necessary, but is very common in Python projects.\n",
"It contains a list of the Python-based dependencies of your project, in a\n",
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
"standardised syntax. You can specify the exact version, or range of versions,\n",
"that your project requires. For example:\n",
"\n",
"\n",
"> six==1.*\n",
"> numpy==1.*\n",
"> scipy>=0.18,<2\n",
"> nibabel==2.*\n",
"\n",
"\n",
"If your project has optional dependencies, i.e. libraries which are not\n",
"critical but, if present, will allow your project to offer some extra\n",
"features, you can list them in a separate requirements file called, for\n",
"example, `requirements-extra.txt`.\n",
"\n",
"\n",
"Having all your dependencies listed in a file in this way makes it easy for\n",
"others to install the dependencies needed by your project, simply by running:\n",
"\n",
"\n",
"> pip install -r requirements.txt\n",
"\n",
"\n",
"<a class=\"anchor\" id=\"setup-py\"></a>\n",
"### `setup.py`\n",
"\n",
"\n",
"This is the most important file (apart from your code, of course). Python\n",
"projects are installed using\n",
"[`setuptools`](https://setuptools.readthedocs.io/en/latest/), which is used\n",
"internally during both the creation of, and installation of Python libraries.\n",
"\n",
"\n",
"The `setup.py` file in a Python project is akin to a `Makefile` in a C/C++\n",
"project. But `setup.py` is also the location where you can define project\n",
"metadata (e.g. name, author, URL, etc) in a standardised format and, if\n",
"necessary, customise aspects of the build process for your library.\n",
"\n",
"\n",
"You generally don't need to worry about, or interact with `setuptools` at all.\n",
"With one exception - `setup.py` is a Python script, and its main job is to\n",
"call the `setuptools.setup` function, passing it information about your\n",
"project.\n",
"The `setup.py` for our example project might look like this:\n",
"> ```\n",
"> #!/usr/bin/env python\n",
">\n",
"> from setuptools import setup\n",
">\n",
"> # Import version number from\n",
"> # the project package (see\n",
"> # the section on versioning).\n",
"> from mypackage import __version__\n",
">\n",
"> # Read in requirements from\n",
"> # the requirements.txt file.\n",
"> with open('requirements.txt', 'rt') as f:\n",
"> requirements = [l.strip() for l in f.readlines()]\n",
">\n",
"> # Generate a list of all of the\n",
"> # packages that are in your project.\n",
"> packages = find_packages()\n",
">\n",
"> setup(\n",
">\n",
"> name='Example project',\n",
"> description='Example Python project for PyTreat',\n",
"> url='https://git.fmrib.ox.ac.uk/fsl/pytreat-2018-practicals/',\n",
"> author='Paul McCarthy',\n",
"> author_email='pauldmccarthy@gmail.com',\n",
"> license='Apache License Version 2.0',\n",
">\n",
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
"> version=__version__,\n",
">\n",
"> install_requires=requirements,\n",
">\n",
"> classifiers=[\n",
"> 'Development Status :: 3 - Alpha',\n",
"> 'Intended Audience :: Developers',\n",
"> 'License :: OSI Approved :: Apache Software License',\n",
"> 'Programming Language :: Python :: 2.7',\n",
"> 'Programming Language :: Python :: 3.4',\n",
"> 'Programming Language :: Python :: 3.5',\n",
"> 'Programming Language :: Python :: 3.6',\n",
"> 'Topic :: Software Development :: Libraries :: Python Modules'],\n",
"> )\n",
"> ```\n",
"\n",
"\n",
"The `setup` function gets passed all of your project's metadata, including its\n",
"version number, depedencies, and licensing information. The `classifiers`\n",
"argument should contain a list of\n",
"[classifiers](https://pypi.python.org/pypi?%3Aaction=list_classifiers) which\n",
"are applicable to your project. Classifiers are purely for descriptive\n",
"purposes - they can be used to aid people in finding your project on\n",
"[`PyPI`](https://pypi.python.org/pypi), if you release it there.\n",
"\n",
"\n",
"See\n",
"[here](https://packaging.python.org/tutorials/distributing-packages/#setup-args)\n",
"for more details on `setup.py` and the `setup` function.\n",
"\n",
"\n",
"<a class=\"anchor\" id=\"appendix-tests\"></a>\n",
"## Appendix: Tests\n",
"\n",
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
"\n",
"There are no strict rules for where to put your tests (you have tests,\n",
"right?). There are two main conventions:\n",
"\n",
"\n",
"You can store your test files _inside_ your package directory:\n",
"\n",
"\n",
"> ```\n",
"> myproject/\n",
"> mypackage/\n",
"> __init__.py\n",
"> mymodule.py\n",
"> tests/\n",
"> __init__.py\n",
"> test_mymodule.py\n",
"> ```\n",
"\n",
"\n",
"\n",
"Or, you can store your test files _alongside_ your package directory:\n",
"\n",
"\n",
"> ```\n",
"> myproject/\n",
"> mypackage/\n",
"> __init__.py\n",
"> mymodule.py\n",
"> tests/\n",
"> test_mymodule.py\n",
"> ```\n",
"\n",
"\n",
"If you want your test code to be completely independent of your project's\n",
"code, then go with the second option. However, if you would like your test\n",
"code to be distributed as part of your project (e.g. so that end users can run\n",
"them), then the first option is probably the best.\n",
"\n",
"\n",
"But in the end, the standard Python unit testing frameworks\n",
"([`pytest`](https://docs.pytest.org/en/latest/) and\n",
"[`nose`](http://nose2.readthedocs.io/en/latest/)) are pretty good at finding\n",
"your test functions no matter where you've hidden them, so the choice is\n",
"really up to you.\n",
"<a class=\"anchor\" id=\"appendix-versioning\"></a>\n",
"## Appendix: Versioning\n",
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
"\n",
"\n",
"If you are intending to make your project available for public use (e.g. on\n",
"[PyPI](https://pypi.python.org/pypi) and/or\n",
"[conda](https://anaconda.org/anaconda/repo)), it is __very important__ to\n",
"manage the version number of your project. If somebody decides to build their\n",
"software on top of your project, they are not going to be very happy with you\n",
"if you make substantial, API-breaking changes without changing your version\n",
"number in an appropriate manner.\n",
"\n",
"\n",
"Python has [official standards](https://www.python.org/dev/peps/pep-0440/) on\n",
"what constitutes a valid version number. These standards can be quite\n",
"complicated but, in the vast majority of cases, a simple three-number\n",
"versioning scheme comprising _major_, _minor_, and _patch_ release\n",
"numbers should suffice. Such a version number has the form:\n",
"\n",
"\n",
"> major.minor.patch\n",
"\n",
"\n",
"For example, a version number of `1.3.2` has a _major_ release of 1, _minor_\n",
"release of 3, and a _patch_ release of 2.\n",
"\n",
"\n",
"If you follow some simple and rational guidelines for versioning\n",
"`your_project`, then people who use your project can, for instance, specify\n",
"that they depend on `your_project==1.*`, and be sure that their code will work\n",
"for _any_ version of `your_project` with a major release of 1. Following these\n",
"simple guidelines greatly improves software interoperability, and makes\n",
"everybody (i.e. developers of other projects, and end users) much happier!\n",
"\n",
"\n",
"Many modern Python projects use some form of [_semantic\n",
"versioning_](https://semver.org/). Semantic versioning is simply a set of\n",
"guidelines on how to manage your version number:\n",
"\n",
"\n",
" - The _major_ release number should be incremented whenever you introduce any\n",
" backwards-incompatible changes. In other words, if you change your code\n",
" such that some other code which uses your code would break, you should\n",
" increment the major release number.\n",
"\n",
" - The _minor_ release number should be incremented whenever you add any new\n",
" (backwards-compatible) features to your project.\n",
"\n",
" - The _patch_ release number should be incremented for backwards-compatible\n",
" bug-fixes and other minor changes.\n",
"\n",
"\n",
"If you like to automate things,\n",
"[`bumpversion`](https://github.com/peritus/bumpversion) is a simple tool that\n",
"you can use to help manage your version number.\n",
"\n",
"\n",
"<a class=\"anchor\" id=\"include-the-version-in-your-code\"></a>\n",
"### Include the version in your code\n",
"\n",
"\n",
"While the version of a library is ultimately defined in `setup.py`, it is\n",
"standard practice for a Python library to contain a version string called\n",
"`__version__` in the `__init__.py` file of the top-level package. For example,\n",
"our `example_project/mypackage/__init__.py` file contains this line:\n",
"\n",
"\n",
"> __version__ = '0.1.0'\n",
"\n",
"\n",
"This makes a library's version number programmatically accessible and\n",
"queryable.\n",
"\n",
"\n",
"<a class=\"anchor\" id=\"deprecate-dont-remove\"></a>\n",
"### Deprecate, don't remove!\n",
"\n",
"\n",
"If you really want to change your API, but can't bring yourself to increment\n",
"your major release number, consider\n",
"[_deprecating_](https://en.wikipedia.org/wiki/Deprecation#Software_deprecation)\n",
"the old API, and postponing its removal until you are ready for a major\n",
"release. This will allow you to change your API, but retain\n",
"backwards-compatilbiity with the old API until it can safely be removed at the\n",
"next major release.\n",
"\n",
"\n",
"You can use the built-in\n",
"[`warnings`](https://docs.python.org/3.5/library/exceptions.html#DeprecationWarning)\n",
"module to warn about uses of deprecated items. There are also some\n",
"[third-party libraries](https://github.com/briancurtin/deprecation) which make\n",
"it easy to mark a function, method or class as being deprecated.\n",
"\n",
"\n",
"<a class=\"anchor\" id=\"appendix-cookiecutter\"></a>\n",
"## Appendix: Cookiecutter\n",
"\n",
"\n",
"It is worth mentioning\n",
"[Cookiecutter](https://github.com/audreyr/cookiecutter), a little utility\n",
"program which you can use to generate a skeleton file/directory structure for\n",
"a new Python project.\n",
"\n",
"\n",
"You need to give it a template (there are many available templates, including\n",
"for projects in languages other than Python) - a couple of useful templates\n",
"are the [minimal Python package\n",
"template](https://github.com/kragniz/cookiecutter-pypackage-minimal), and the\n",
"[full Python package\n",
"template](https://github.com/audreyr/cookiecutter-pypackage) (although the\n",
"latter is probably overkill for most).\n",
"\n",
"\n",
"Here is how to create a skeleton project directory based off the minimal\n",
"> ```\n",
"> pip install cookiecutter\n",
">\n",
"> # tell cookiecutter to create a directory\n",
"> # from the pypackage-minimal template\n",
"> cookiecutter https://github.com/kragniz/cookiecutter-pypackage-minimal.git\n",
">\n",
"> # cookiecutter will then prompt you for\n",
"> # basic information (e.g. projectname,\n",
"> # author name/email), and then create a\n",
"> # new directory containing the project\n",
"> # skeleton.\n",
"> ```"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 2
}