creating_fsl_conda_recipes.md 13.9 KB
Newer Older
1
2
3
4
5
6
7
8
# Writing a conda recipe for a FSL project


This document describes how to write a conda recipe for a FSL project. All FSL
project repositories have an associated recipe repository, which contains
metadata describing how to build a conda recipe from the project repository.


9
10
11
You can use a script to automatically generate an initial version of a
repository for your project (recommended), or you can create a conda recipe by
hand.
12
13


14
15
16
17
18
Conda recipes for all FSL projects, and some internally packaged/managed
third-party dependencies, are hosted in the GitLab
[fsl/conda](https://git.fmrib.ox.ac.uk/fsl/conda/) namespace. The name of a
FSL conda recipe repository is the same as the name of the FSL conda package -
for example, the conda package for the
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
[`fsl/avwutils`](https://git.fmrib.ox.ac.uk/fsl/avwutils/) project is named
`fsl-avwutils`, and is hosted at
[`fsl/conda/fsl-avwutils`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-avwutils/).


## FSL conda package naming conventions


FSL conda package names must follow the [conda package naming
conventions](https://conda.io/projects/conda-build/en/latest/concepts/package-naming-conv.html#index-0),
and be comprised solely of _lowercase alpha characters, numeric digits,
underscores, hyphens, or dots_.


Furthermore, all FSL conda packages are prefixed with `fsl-`. An FSL project
with name `<project>` will have a corresponding conda package name of
`fsl-<project>`. For FSL projects with a name that begins with `fsl`
(e.g. `fslvbm`, `fsl_deface`), the leading `fsl` will be dropped in the
construction of the corresponding conda-package name. For example:

39

40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
| **FSL project name** | **Conda package name** |
| -------------------- | ---------------------- |
| `avwutils`           | `fsl-avwutils`         |
| `fslvbm`             | `fsl-vbm`              |
| `fsl_deface`         | `fsl-deface`           |
| `fsl-mrs`            | `fsl-mrs`              |
| `NewNifti`           | `fsl-newnifti`         |


## Automatically generate a conda recipe for a FSL project


The fsl/fsl-ci-rules> project contains a utility script called
`create_conda_recipe` that can be used to automatically generate an initial
version of a recipe for a FSL project that is hosted on GitLab. This is a
useful way to start, as the `create_conda_recipe` script will generate a
recipe with a standardised structure, and with most information already
populated.


To generate an initial version of a conda recipe for a new project:


1. Make sure you have FSL installed and active in your shell environment.

2. Create a [GitLab personal access
   token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#personal-access-tokens),
67
   which will allow you to programmatically interact with GitLab.
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

3. Clone the `fsl/fsl-ci-rules` repository, and configure your environment
   so you can use `create_conda_recipe`:

    ```
    git clone https://git.fmrib.ox.ac.uk/fsl/fsl-ci-rules.git
    export PYTHONPATH=$(pwd)/fsl-ci-rules/utils/
    ```

4. Call the script to generate an initial version of the recipe:

   ```
   fslpython -m create_conda_recipe -t <token> <project_path>
   ```

   where:
    - `<token>` is you GitLab personal access token
    - `<project_path>` is the GitLab repository path of your project (e.g.
      `fsl/avwutils`, `paulmc/fsleyes`, etc), or the full URL to a git
      repository.


The `create_conda_recipe` script will automatically create a conda recipe
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
repository on GitLab, within the
[fsl/conda](https://git.fmrib.ox.ac.uk/fsl/conda) namespace, and named
according to the FSL conda package naming conventions outlined above. For
example, this command command:


```
fslpython -m create_conda_recipe -t <token> paulmc/my_new_project
```


will create a conda recipe for a GitLab project repository at
`https://git.fmrib.ox.ac.uk/paulmc/my_cool_project`, for a conda package
called `fsl-my_cool_project`, and will upload the recipe to a GitLab
repository at `https://git.fmrib.ox.ac.uk/fsl/conda/fsl-my_cool_project`.
106
107
108
109
110


The `create_conda_recipe` script has a few options allowing you to control its
behaviour (e.g. create a recipe repository locally insteaad of on GitLab,
create the recipe from a branch other than `master` from the project
111
repository); run `fslpython -m create_conda_recipe -h` for help on the
112
113
114
115
116
117
118
available options.


## Create a conda recipe for a FSL project by hand


We recommend using the `create_conda_recipe` script as outlined above, to
119
generate an initial version of the recipe for your FSL project. This has the
120
advantage that recipes for all FSL projects will follow a few simple
121
122
123
conventions and standards. However, it is possible to create a conda recipe by
hand.

124

125
126
127
128
The purpose of this section is not to provide full instructions on writing a
conda recipe, but rather on the specific issues that need to be considered
when writing a conda recipe for a FSL project. More general information on
creating conda recipes can be found at these websites:
129
130
131
132
133

 - https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
 - https://python-packaging-tutorial.readthedocs.io/en/latest/conda.html


134
A FSL conda recipe is simply a flat directory containing the following files:
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153

 - `meta.yaml`: A YAML file which contains a description of the conda package,
   including its name, current version, URL to the project repository, and
   list of dependencies.

 - `build.sh`: A bash script which is executed in order to build the
   package. For FSL C++ projects, this script essentially just calls
   `$FSLDIR/etc/fslconf/fsl-devel.sh`, then runs `make` and `make
   install`. For FSL `Makefile`-based projects, `build.sh` is required, but
   for Python-based projects, `build.sh` may or may not be necessary.

 - `post-link.sh`: For projects which provide executables that are installed
   into `$FSLDIR/bin/`, this script is used to conditionally create wrapper
   scripts in `$FSLDIR/share/fsl/bin/`, as outlined in
   [`official_fsl_installations.md`](official_fsl_installations.md). This
   script is not required for projects which do not provide any executables.

 - `pre-unlink.sh`: This script is called when a package is *uninstalled* -
   its job is to remove any wrapper scripts that were created by
154
155
   `post-link.sh`. This script is not required for projects which do not
   provide any executables.
156
157


158
FSL projects are broadly divided into one of the following categories:
159
160
161
162
163
164
165
166
167
168

  - **`Makefile`-based project**: FSL projects which use a FSL-style
    `Makefile` to compile and install their deliverables (shared libraries,
    scripts, and compiled executables).

  - **`setup.py`-based project**: FSL projects which are written in Python,
    and which have a `setup.py` file which is used to build the project as
    a Python package.


169
170
171
172
173
174
The mechanisms by which projects in these categories are built are slightly
different and, therefore, the conda recipe for projects from different
category will look slightly different. Examples of `Makefile`-based and
`setup.py`-based FSL projects, and associated conda recipes, can be
respectively found in the `examples/cpp` and `examples/python`
sub-directories. Some important details are highlighted below.
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211


### Writing the `meta.yaml` file


The `meta.yaml` file contains metadata about your project, including:
 - The conda package name
 - The URL of the project git repository
 - The version number
 - The build number (used in case the conda package for a single version
   needs to be re-built for some reason).
 - A list of the package build- and run-time dependencies.


**Essential metadata** In order to allow for automatic maintenance of FSL
conda recipes, you should define the essential metadata (name, version, etc)
as `jinja2` variables using the following syntax:


```
{% set name       = <conda-package-name> %}
{% set version    = <version-number>     %}
{% set repository = <repository-url>     %}
{% set build      = '0'                  %}
```


Then use these variables within the recipe YAML, e.g.:


```
package:
  name:    {{ name }}
  version: {{ version }}
```


212
213
214
215
By following this convention, FSL conda recipes can be automatically updated
when a new version of a project is released.


216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
**Project repository and git revision** The automated CI rules defined in
`fsl-ci-rules` allow the project repositoy and git revision (tag, branch, etc)
used to build a conda package to be overridden via the `FSLCONDA_REPOSITORY`
and `FSLCONDA_REVISION` environment variables. To facilitate this, the project
source repository and git revision must be specified in the `meta.yaml` like
so (remembering that we have defined `repository` and `version` as `jinja2
variables, above):


```
source:
  # the FSLCONDA_REPOSITORY and FSLCONDA_REVISION
  # environment variables can be used to override
  # the repository/revision for development purposes.
  git_url: {{ os.environ.get("FSLCONDA_REPOSITORY", repository) }}
  git_rev: {{ os.environ.get("FSLCONDA_REVISION",   version)    }}
```


235
236
237
238
239
Following this convention allows conda packages for development and testing to
be built, both automatically, and when developing/testing locally, simply by
setting the `FSLCONDA_REPOSITORY` and `FSLCONDA_REVISION` variables.


240
241
242
243
244
245
246
247
**run_exports (C/C++ projects only)** When compiling a C/C++ project, any
shared library dependencies of the project must be present at the time of
compilation, and at run time. This means that the dependencies of your project
may need to be listed twice within the `requirements` section - once under
`host` (or `build`), and again under `run`.  This can be avoided by specifying
`run_exports` in the recipes of the dependencies.


248
249
250
251
252
253
254
255
The
[`run_exports`](https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#export-runtime-requirements)
section is a trick which can be used within `meta.yaml`, which essentially
allows us to define a dependency as a build-time dependency, and have it
automatically propagated as a run-time dependency.  It is used simply to allow
us to avoid having to list shared library dependencies twice.


256
So if you are writing a conda recipe for a C/C++ project which provides shared
257
258
library files that will be used by other projects, add a `run_exports` section
to the `build` section like so:
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274


```
build:
  number: {{ build }}
  run_exports:
    strong:
      - {{ name }}
```


**`noarch` and `script` (`setup.py`-based projects only)** Conda recipes for
Python `setup.py`-based projects are built slightly differently to native
projects, as the Python `setuptools` machinery is integrated into the conda
build process. For `setup.py`-based projects, the build command is usually
specified within the `build` section of the `meta.yaml` file. Furthermore, if
275
you are packging a *pure* Python project, with no natively compiled code or
276
extensions, you must label your recipe as being of type `noarch: python` -
277
278
this is also specified within the `build` section. So a typical `build`
section for a `setup.py`-based project will resemble the following:
279
280
281
282
283
284
285
286
287
288
289
290
291
292


```
build:
  number: {{ build }}
  noarch: python
  script: {{ PYTHON }} -m pip install . --no-deps --ignore-installed --no-cache-dir -vvv
```


**Requirements** The [`fsl/base`](https://git.fmrib.ox.ac.uk/fsl/base) project
provides the fundamental elements of the contents of a FSL installation,
including FSL initialisation scripts and the `Makefile` machinery. As such it
must be installed in order to build `Makefile`-based FSL projects, and must be
293
present at run time for most FSL commands to function. All FSL conda recipes
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
must therefore list `fsl-base` as a requirement. C/C++ projects will also need
to have a C++ compiler installed at build time - the convention within conda
recipes is to list compilers as `host` dependencies. For example:


```
requirements:
  build:
    - {{ compiler('cxx') }}
  host:
    - fsl-base >=2101.0
```


### Defining the build proceess


If you are writing a recipe for a `Makefile`-based FSL project, you need to
write a `build.sh` script which uses the FSL `Makefile` machinery to build
your project. A typical `build.sh` script will resemble the following:


```
#!/usr/bin/env bash

# $PREFIX is the installation destination
# when a conda package is being built
export FSLDIR=$PREFIX

# Install project source code into
# $PREFIX/src/ (a.k.a. $FSLDIR/src/)
mkdir -p $PREFIX/src/
cp -r $(pwd) $PREFIX/src/$PKG_NAME

# Configure the build environment
. $FSLDIR/etc/fslconf/fsl-devel.sh

# Build and install the project
make
make install
```


If you are writing a recipe for a `setup.py`-based FSL project, a `build.sh`
script is generally not required.


### `post-link.sh` and `pre-unlink.sh` scripts.
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384


Official/full FSL installations contain wrapper scripts for every FSL
executable in `$FSLDIR/share/fsl/bin/` - this is so that a user can add FSL
commands to their `$PATH` via this directory, rather than the `$FSLDIR/bin/`
directory, and avoid adding all of the other executables in `$FSLDIR/bin/` to
their `$PATH` (e.g. `python`).


These wrapper scripts are created/removed by two utility commands which are
provided by the `fsl-base` package - `createFSLWrapper` and
`removeFSLWrapper`. The conda recipes for any FSL projects which provide
executables need to call these scripts at the time of
installation/uninstallation to ensure that wrapper scripts for the executables
are created/removed.


This can be achieved by using [`post-link.sh` and
`pre-unlink.sh`](https://docs.conda.io/projects/conda-build/en/latest/resources/link-scripts.html)
scripts, which are conda-specific mechanisms allowing custom logic to be
executed when a conda package is installed or uninstalled.


For a FSL project which provides executables called `fsl_command1`,
`fsl_command2` and `fsl_command3`, the `post-link.sh` for the project recipe
should contain:


```
if [ -e ${FSLDIR}/share/fsl/sbin/createFSLWrapper ]; then
    ${FSLDIR}/share/fsl/sbin/createFSLWrapper fsl_command1 fsl_command2 fsl_command3
fi
```


The corresponding `pre-unlink.sh` script should contain:


```
if [ -e ${FSLDIR}/share/fsl/sbin/removeFSLWrapper ]; then
    ${FSLDIR}/share/fsl/sbin/removeFSLWrapper fsl_command1 fsl_command2 fsl_command3
fi
```