creating_fsl_conda_recipes.md 18.8 KB
Newer Older
1
2
3
4
5
6
7
8
# Writing a conda recipe for a FSL project


This document describes how to write a conda recipe for a FSL project. All FSL
project repositories have an associated recipe repository, which contains
metadata describing how to build a conda recipe from the project repository.


9
10
11
You can use a script to automatically generate an initial version of a
repository for your project (recommended), or you can create a conda recipe by
hand.
12
13


14
15
16
17
18
Conda recipes for all FSL projects, and some internally packaged/managed
third-party dependencies, are hosted in the GitLab
[fsl/conda](https://git.fmrib.ox.ac.uk/fsl/conda/) namespace. The name of a
FSL conda recipe repository is the same as the name of the FSL conda package -
for example, the conda package for the
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
[`fsl/avwutils`](https://git.fmrib.ox.ac.uk/fsl/avwutils/) project is named
`fsl-avwutils`, and is hosted at
[`fsl/conda/fsl-avwutils`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-avwutils/).


## FSL conda package naming conventions


FSL conda package names must follow the [conda package naming
conventions](https://conda.io/projects/conda-build/en/latest/concepts/package-naming-conv.html#index-0),
and be comprised solely of _lowercase alpha characters, numeric digits,
underscores, hyphens, or dots_.


Furthermore, all FSL conda packages are prefixed with `fsl-`. An FSL project
with name `<project>` will have a corresponding conda package name of
`fsl-<project>`. For FSL projects with a name that begins with `fsl`
(e.g. `fslvbm`, `fsl_deface`), the leading `fsl` will be dropped in the
construction of the corresponding conda-package name. For example:

39

40
41
42
43
44
45
46
47
48
| **FSL project name** | **Conda package name** |
| -------------------- | ---------------------- |
| `avwutils`           | `fsl-avwutils`         |
| `fslvbm`             | `fsl-vbm`              |
| `fsl_deface`         | `fsl-deface`           |
| `fsl-mrs`            | `fsl-mrs`              |
| `NewNifti`           | `fsl-newnifti`         |


Paul McCarthy's avatar
Paul McCarthy committed
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
There are a small number of exceptions to the above conventions. For example,
the [`fdt`](https://git.fmrib.ox.ac.uk/fsl/fdt) project is built into two
conda packages - the [`fsl-fdt`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-fdt)
package, providing CPU-only executables, and the
[`fsl-fdt-gpu`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-fdt-gpu) package,
providing GPU/CUDA-enabled executables.


## A note on FSL CUDA projects


Most FSL CUDA projects usually provide both GPU-enabled and CPU-only
executables. For example, the [`fsl/fdt`](https://git.fmrib.ox.ac.uk/fsl/fdt)
provides a range of CPU-only executables, including `dtifit` and `vecreg`, in
addition to providing GPU-enabled executables such as `xfibres_gpu`.


To accommodate this convention, multiple conda recipes are used for these
"hybrid" projects. For example, the `fsl/fdt` project is built from two
separate recipes:


 - [`fsl/conda/fsl-fdt`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-fdt), which
   builds the CPU-only executables - these recipes are built as `linux` and
   `macos` packages.
 - [`fsl/conda/fsl-fdt-gpu`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-fdt),
   which builds the GPU-enabled executables - these recipes are built as
   `linux-cuda-X.Y` and `macos-cuda-X.Y` packages.


79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
## Automatically generate a conda recipe for a FSL project


The fsl/fsl-ci-rules> project contains a utility script called
`create_conda_recipe` that can be used to automatically generate an initial
version of a recipe for a FSL project that is hosted on GitLab. This is a
useful way to start, as the `create_conda_recipe` script will generate a
recipe with a standardised structure, and with most information already
populated.


To generate an initial version of a conda recipe for a new project:


1. Make sure you have FSL installed and active in your shell environment.

2. Create a [GitLab personal access
   token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#personal-access-tokens),
97
   which will allow you to programmatically interact with GitLab.
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120

3. Clone the `fsl/fsl-ci-rules` repository, and configure your environment
   so you can use `create_conda_recipe`:

    ```
    git clone https://git.fmrib.ox.ac.uk/fsl/fsl-ci-rules.git
    export PYTHONPATH=$(pwd)/fsl-ci-rules/utils/
    ```

4. Call the script to generate an initial version of the recipe:

   ```
   fslpython -m create_conda_recipe -t <token> <project_path>
   ```

   where:
    - `<token>` is you GitLab personal access token
    - `<project_path>` is the GitLab repository path of your project (e.g.
      `fsl/avwutils`, `paulmc/fsleyes`, etc), or the full URL to a git
      repository.


The `create_conda_recipe` script will automatically create a conda recipe
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
repository on GitLab, within the
[fsl/conda](https://git.fmrib.ox.ac.uk/fsl/conda) namespace, and named
according to the FSL conda package naming conventions outlined above. For
example, this command command:


```
fslpython -m create_conda_recipe -t <token> paulmc/my_new_project
```


will create a conda recipe for a GitLab project repository at
`https://git.fmrib.ox.ac.uk/paulmc/my_cool_project`, for a conda package
called `fsl-my_cool_project`, and will upload the recipe to a GitLab
repository at `https://git.fmrib.ox.ac.uk/fsl/conda/fsl-my_cool_project`.
136
137
138
139
140


The `create_conda_recipe` script has a few options allowing you to control its
behaviour (e.g. create a recipe repository locally insteaad of on GitLab,
create the recipe from a branch other than `master` from the project
141
repository); run `fslpython -m create_conda_recipe -h` for help on the
142
143
144
145
146
147
148
available options.


## Create a conda recipe for a FSL project by hand


We recommend using the `create_conda_recipe` script as outlined above, to
149
generate an initial version of the recipe for your FSL project. This has the
150
advantage that recipes for all FSL projects will follow a few simple
151
152
153
conventions and standards. However, it is possible to create a conda recipe by
hand.

154

155
156
157
158
The purpose of this section is not to provide full instructions on writing a
conda recipe, but rather on the specific issues that need to be considered
when writing a conda recipe for a FSL project. More general information on
creating conda recipes can be found at these websites:
159

Paul McCarthy's avatar
Paul McCarthy committed
160

161
162
163
164
 - https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
 - https://python-packaging-tutorial.readthedocs.io/en/latest/conda.html


165
A FSL conda recipe is simply a flat directory containing the following files:
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184

 - `meta.yaml`: A YAML file which contains a description of the conda package,
   including its name, current version, URL to the project repository, and
   list of dependencies.

 - `build.sh`: A bash script which is executed in order to build the
   package. For FSL C++ projects, this script essentially just calls
   `$FSLDIR/etc/fslconf/fsl-devel.sh`, then runs `make` and `make
   install`. For FSL `Makefile`-based projects, `build.sh` is required, but
   for Python-based projects, `build.sh` may or may not be necessary.

 - `post-link.sh`: For projects which provide executables that are installed
   into `$FSLDIR/bin/`, this script is used to conditionally create wrapper
   scripts in `$FSLDIR/share/fsl/bin/`, as outlined in
   [`official_fsl_installations.md`](official_fsl_installations.md). This
   script is not required for projects which do not provide any executables.

 - `pre-unlink.sh`: This script is called when a package is *uninstalled* -
   its job is to remove any wrapper scripts that were created by
185
186
   `post-link.sh`. This script is not required for projects which do not
   provide any executables.
187
188


189
FSL projects are broadly divided into one of the following categories:
190
191
192
193
194

  - **`Makefile`-based project**: FSL projects which use a FSL-style
    `Makefile` to compile and install their deliverables (shared libraries,
    scripts, and compiled executables).

Paul McCarthy's avatar
Paul McCarthy committed
195
196
197
198
  - **`Makefile`-based CUDA project**: FSL projects which use a FSL-style
    `Makefile`, and which provide GPU-accelerated executables linked against
    CUDA.

199
200
201
202
203
  - **`setup.py`-based project**: FSL projects which are written in Python,
    and which have a `setup.py` file which is used to build the project as
    a Python package.


204
205
206
207
The mechanisms by which projects in these categories are built are slightly
different and, therefore, the conda recipe for projects from different
category will look slightly different. Examples of `Makefile`-based and
`setup.py`-based FSL projects, and associated conda recipes, can be
Paul McCarthy's avatar
Paul McCarthy committed
208
209
210
respectively found in the `examples/cpp`, `examples/cuda`, and
`examples/python` sub-directories. Some important details are highlighted
below.
211
212
213
214
215
216


### Writing the `meta.yaml` file


The `meta.yaml` file contains metadata about your project, including:
Paul McCarthy's avatar
Paul McCarthy committed
217

218
219
220
221
222
223
224
225
226
227
228
229
230
231
 - The conda package name
 - The URL of the project git repository
 - The version number
 - The build number (used in case the conda package for a single version
   needs to be re-built for some reason).
 - A list of the package build- and run-time dependencies.


**Essential metadata** In order to allow for automatic maintenance of FSL
conda recipes, you should define the essential metadata (name, version, etc)
as `jinja2` variables using the following syntax:


```
Paul McCarthy's avatar
Paul McCarthy committed
232
233
234
235
{% set name       = '<conda-package-name>' %}
{% set version    = '<version-number>'     %}
{% set repository = '<repository-url>'     %}
{% set build      = '0'                    %}
236
237
238
239
240
241
242
243
```


Then use these variables within the recipe YAML, e.g.:


```
package:
Paul McCarthy's avatar
Paul McCarthy committed
244
  name:    {{ name    }}
245
246
247
248
  version: {{ version }}
```


249
250
251
252
By following this convention, FSL conda recipes can be automatically updated
when a new version of a project is released.


253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
**Project repository and git revision** The automated CI rules defined in
`fsl-ci-rules` allow the project repositoy and git revision (tag, branch, etc)
used to build a conda package to be overridden via the `FSLCONDA_REPOSITORY`
and `FSLCONDA_REVISION` environment variables. To facilitate this, the project
source repository and git revision must be specified in the `meta.yaml` like
so (remembering that we have defined `repository` and `version` as `jinja2
variables, above):


```
source:
  # the FSLCONDA_REPOSITORY and FSLCONDA_REVISION
  # environment variables can be used to override
  # the repository/revision for development purposes.
  git_url: {{ os.environ.get("FSLCONDA_REPOSITORY", repository) }}
  git_rev: {{ os.environ.get("FSLCONDA_REVISION",   version)    }}
```


272
273
274
275
276
Following this convention allows conda packages for development and testing to
be built, both automatically, and when developing/testing locally, simply by
setting the `FSLCONDA_REPOSITORY` and `FSLCONDA_REVISION` variables.


277
278
279
280
**run_exports (C/C++ projects only)** When compiling a C/C++ project, any
shared library dependencies of the project must be present at the time of
compilation, and at run time. This means that the dependencies of your project
may need to be listed twice within the `requirements` section - once under
Paul McCarthy's avatar
Paul McCarthy committed
281
282
`host` (or `build`), and again under `run`.  We can avoid having to list
dependencies twice by specifying `run_exports` in the dependency recipes.
283
284


285
286
287
288
The
[`run_exports`](https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#export-runtime-requirements)
section is a trick which can be used within `meta.yaml`, which essentially
allows us to define a dependency as a build-time dependency, and have it
Paul McCarthy's avatar
Paul McCarthy committed
289
automatically propagated as a run-time dependency.
290
291


292
So if you are writing a conda recipe for a C/C++ project which provides shared
293
294
library files that will be used by other projects, add a `run_exports` section
to the `build` section like so:
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310


```
build:
  number: {{ build }}
  run_exports:
    strong:
      - {{ name }}
```


**`noarch` and `script` (`setup.py`-based projects only)** Conda recipes for
Python `setup.py`-based projects are built slightly differently to native
projects, as the Python `setuptools` machinery is integrated into the conda
build process. For `setup.py`-based projects, the build command is usually
specified within the `build` section of the `meta.yaml` file. Furthermore, if
311
you are packging a *pure* Python project, with no natively compiled code or
312
extensions, you must label your recipe as being of type `noarch: python` -
313
314
this is also specified within the `build` section. So a typical `build`
section for a `setup.py`-based project will resemble the following:
315
316
317
318
319
320
321
322
323
324
325
326
327
328


```
build:
  number: {{ build }}
  noarch: python
  script: {{ PYTHON }} -m pip install . --no-deps --ignore-installed --no-cache-dir -vvv
```


**Requirements** The [`fsl/base`](https://git.fmrib.ox.ac.uk/fsl/base) project
provides the fundamental elements of the contents of a FSL installation,
including FSL initialisation scripts and the `Makefile` machinery. As such it
must be installed in order to build `Makefile`-based FSL projects, and must be
329
present at run time for most FSL commands to function. All FSL conda recipes
330
must therefore list `fsl-base` as a requirement. C/C++ projects will also need
Paul McCarthy's avatar
Paul McCarthy committed
331
to have a C++ compiler installed at build time. For example:
332
333
334
335
336
337


```
requirements:
  build:
    - {{ compiler('cxx') }}
Paul McCarthy's avatar
Paul McCarthy committed
338
    - make
339
340
341
342
343
  host:
    - fsl-base >=2101.0
```


Paul McCarthy's avatar
Paul McCarthy committed
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
### Recipes for CUDA projects


> **Note:** Mechanisms for building CUDA packages on macOS do not currently
> exist.


Separate packages are created from `Makefile`-based CUDA projects for
different versions of CUDA.  To enable this, the `meta.yaml` file for a CUDA
project needs to contain a couple of additional elements. First, when a CUDA
package is built, an environment variable called `CUDA_VER` will be set in the
environment, containing the `major.minor` CUDA version the package is being
built against.  The `meta.yaml` file can read this environment variable in as
a `jinja2` variable like so:


```
{{ '{% set cuda_version = os.environ["CUDA_VER"] %}' }}
```


Built CUDA packages must be labelled with the version of CUDA they were built
against.  This is accomplished by adding the CUDA version to the package
build string, like so:


```
build:
  number: {{ '{{ build }}' }}
  string: {{ 'h{{ PKG_HASH }}_cuda{{ cuda_version }}_{{ PKG_BUILDNUM }}' }}
  run_exports:
    strong:
      - {{ '{{ name }}' }}
```


When a CUDA package is built, the `nvcc` compiler must be installed in the
build environment, as it cannot be installed through `conda`. However, the
CUDA toolkit *can* be installed through conda, in addition to a "shim" `nvcc`
compiler package which allows `conda` to integrate itself with the externally
installed `nvcc` compiler. A further complication is that different versions
of `nvcc` are compatible with different versions of `gcc`, so the version of
`gcc` that should be installed depends on the version of CUDA against which
the package is being built.


To handle all of these complicationes, a template `requirements` section for
a CUDA project needs to look something like this:


```
requirements:
  host:
    - fsl-base >=2101.0
  build:
    # Different versions of nvcc need
    # different versions of gcc
    {{ '{% if   cuda_version in ("9.2", "10.0") %}' }}
    - {{ '{{ compiler("cxx") }} 7.*' }}  # [linux]
    {{ '{% elif cuda_version in ("10.1", "10.2") %}' }}
    - {{ '{{ compiler("cxx") }} 8.*' }}  # [linux]
    {{ '{% elif cuda_version in ("11.0", "11.1") %}' }}
    - {{ '{{ compiler("cxx") }} 9.*' }}  # [linux]
    {{ '{% endif %}' }}
    - make
    # The nvcc_linux-64 package is a shim
    # which will use the system-provided
    # nvcc, but in a manner that is
    # integrated with conda.
    - nvcc_linux-64 {{ '{{ cuda_version }}' }}  # [linux]
```


417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
### Defining the build proceess


If you are writing a recipe for a `Makefile`-based FSL project, you need to
write a `build.sh` script which uses the FSL `Makefile` machinery to build
your project. A typical `build.sh` script will resemble the following:


```
#!/usr/bin/env bash

# $PREFIX is the installation destination
# when a conda package is being built
export FSLDIR=$PREFIX

# Install project source code into
# $PREFIX/src/ (a.k.a. $FSLDIR/src/)
mkdir -p $PREFIX/src/
cp -r $(pwd) $PREFIX/src/$PKG_NAME

# Configure the build environment
. $FSLDIR/etc/fslconf/fsl-devel.sh

# Build and install the project
make
make install
```


If you are writing a recipe for a `setup.py`-based FSL project, a `build.sh`
script is generally not required.


Paul McCarthy's avatar
Paul McCarthy committed
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
A critical requirement of "hybrid" FSL CUDA projects, which provide both
CPU-only and GPU-capable executables, is that the project `Makefile` must be
able to conditionally compile only the CPU component, **or** the GPU
components, provided by the project.


For example, the [`fdt`](https://git.fmrib.ox.ac.uk/fsl/fdt) `Makefile` allows
`cpu` and `gpu` flags to be specified to control which parts of the project
are compiled - `make cpu=1` will compile only the CPU components of the `fdt`
project, whereas `make cpu=0 gpu=1` will compile only the GPU components.


> The specific `make` invocations that need to be made depend on how the
> project `Makefile` is written.


The `build.sh` scripts for the
[`fsl-fdt`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-fdt) and
[`fsl-fdt-gpu`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-fdt-gpu) take
advantage of this mechanism, so that the `fsl-fdt` recipe will only compile
the CPU components, and the `fsl-fdt-gpu` recipe will only compile the GPU
components

The `build.sh` script for the `fsl-fdt` recipe therefore looks the same as the
template `build.sh` script above, except the `make` invocations are as
follows:


```
make cpu=1
make cpu=1 install
```


And the `make` invocations in the `build.sh` script for the `fsl-fdt-gpu`
recipe are as follows:


```
make cpu=0 gpu=1
make cpu=0 gpu=1 install
```


494
### `post-link.sh` and `pre-unlink.sh` scripts.
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537


Official/full FSL installations contain wrapper scripts for every FSL
executable in `$FSLDIR/share/fsl/bin/` - this is so that a user can add FSL
commands to their `$PATH` via this directory, rather than the `$FSLDIR/bin/`
directory, and avoid adding all of the other executables in `$FSLDIR/bin/` to
their `$PATH` (e.g. `python`).


These wrapper scripts are created/removed by two utility commands which are
provided by the `fsl-base` package - `createFSLWrapper` and
`removeFSLWrapper`. The conda recipes for any FSL projects which provide
executables need to call these scripts at the time of
installation/uninstallation to ensure that wrapper scripts for the executables
are created/removed.


This can be achieved by using [`post-link.sh` and
`pre-unlink.sh`](https://docs.conda.io/projects/conda-build/en/latest/resources/link-scripts.html)
scripts, which are conda-specific mechanisms allowing custom logic to be
executed when a conda package is installed or uninstalled.


For a FSL project which provides executables called `fsl_command1`,
`fsl_command2` and `fsl_command3`, the `post-link.sh` for the project recipe
should contain:


```
if [ -e ${FSLDIR}/share/fsl/sbin/createFSLWrapper ]; then
    ${FSLDIR}/share/fsl/sbin/createFSLWrapper fsl_command1 fsl_command2 fsl_command3
fi
```


The corresponding `pre-unlink.sh` script should contain:


```
if [ -e ${FSLDIR}/share/fsl/sbin/removeFSLWrapper ]; then
    ${FSLDIR}/share/fsl/sbin/removeFSLWrapper fsl_command1 fsl_command2 fsl_command3
fi
```