Commit 97a7a7d7 authored by Paul McCarthy's avatar Paul McCarthy 🚵
Browse files

DOC: Details on creating recipes. Not finished

parent 31352187
# Writing a conda recipe for a FSL project
This document describes how to write a conda recipe for a FSL project. All FSL
project repositories have an associated recipe repository, which contains
metadata describing how to build a conda recipe from the project repository.
You can create a conda recipe by hand, or you can use a script to
automatically generate an initial version of a repository for your project.
Conda recipes for all FSL projects, and some internally packaged dependencies,
are hosted in the [fsl/conda](https://git.fmrib.ox.ac.uk/fsl/conda/) gitlab
namespace. The name of a FSL conda recipe repository is the same as the name
of the FSL conda package - for example, the conda package for the
[`fsl/avwutils`](https://git.fmrib.ox.ac.uk/fsl/avwutils/) project is named
`fsl-avwutils`, and is hosted at
[`fsl/conda/fsl-avwutils`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-avwutils/).
## FSL conda package naming conventions
FSL conda package names must follow the [conda package naming
conventions](https://conda.io/projects/conda-build/en/latest/concepts/package-naming-conv.html#index-0),
and be comprised solely of _lowercase alpha characters, numeric digits,
underscores, hyphens, or dots_.
Furthermore, all FSL conda packages are prefixed with `fsl-`. An FSL project
with name `<project>` will have a corresponding conda package name of
`fsl-<project>`. For FSL projects with a name that begins with `fsl`
(e.g. `fslvbm`, `fsl_deface`), the leading `fsl` will be dropped in the
construction of the corresponding conda-package name. For example:
| **FSL project name** | **Conda package name** |
| -------------------- | ---------------------- |
| `avwutils` | `fsl-avwutils` |
| `fslvbm` | `fsl-vbm` |
| `fsl_deface` | `fsl-deface` |
| `fsl-mrs` | `fsl-mrs` |
| `NewNifti` | `fsl-newnifti` |
## Automatically generate a conda recipe for a FSL project
The fsl/fsl-ci-rules> project contains a utility script called
`create_conda_recipe` that can be used to automatically generate an initial
version of a recipe for a FSL project that is hosted on GitLab. This is a
useful way to start, as the `create_conda_recipe` script will generate a
recipe with a standardised structure, and with most information already
populated.
To generate an initial version of a conda recipe for a new project:
1. Make sure you have FSL installed and active in your shell environment.
2. Create a [GitLab personal access
token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#personal-access-tokens),
which will allow you to use the GitLab API.
3. Clone the `fsl/fsl-ci-rules` repository, and configure your environment
so you can use `create_conda_recipe`:
```
git clone https://git.fmrib.ox.ac.uk/fsl/fsl-ci-rules.git
export PYTHONPATH=$(pwd)/fsl-ci-rules/utils/
```
4. Call the script to generate an initial version of the recipe:
```
fslpython -m create_conda_recipe -t <token> <project_path>
```
where:
- `<token>` is you GitLab personal access token
- `<project_path>` is the GitLab repository path of your project (e.g.
`fsl/avwutils`, `paulmc/fsleyes`, etc), or the full URL to a git
repository.
The `create_conda_recipe` script will automatically create a conda recipe
repository on GitLab, named according to the FSL conda package naming
conventions outlined above.
The `create_conda_recipe` script has a few options allowing you to control its
behaviour (e.g. create a recipe repository locally insteaad of on GitLab,
create the recipe from a branch other than `master` from the project
repository) - run `fslpython -m create_conda_recipe -h` for help on the
available options.
## Create a conda recipe for a FSL project by hand
We recommend using the `create_conda_recipe` script as outlined above, to
generate an initial version of the recipe for your FSL projec. This has the
advantage that recipes for all FSL projects will follow a few simple
conventions and standards. However, it is possible to create a conda recipe
by hand.
Useful information on creating conda recipes can be found at these websites:
- https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
- https://python-packaging-tutorial.readthedocs.io/en/latest/conda.html
A FSL conda recipe contains the following:
- `meta.yaml`: A YAML file which contains a description of the conda package,
including its name, current version, URL to the project repository, and
list of dependencies.
- `build.sh`: A bash script which is executed in order to build the
package. For FSL C++ projects, this script essentially just calls
`$FSLDIR/etc/fslconf/fsl-devel.sh`, then runs `make` and `make
install`. For FSL `Makefile`-based projects, `build.sh` is required, but
for Python-based projects, `build.sh` may or may not be necessary.
- `post-link.sh`: For projects which provide executables that are installed
into `$FSLDIR/bin/`, this script is used to conditionally create wrapper
scripts in `$FSLDIR/share/fsl/bin/`, as outlined in
[`official_fsl_installations.md`](official_fsl_installations.md). This
script is not required for projects which do not provide any executables.
- `pre-unlink.sh`: This script is called when a package is *uninstalled* -
its job is to remove any wrapper scripts that were created by
`post-link.sh`. script is not required for projects which do not provide
any executables.
A FSL conda recipe is simply a flat directory containing the above files. FSL
projects are broadly divided into one of the following types:
- **`Makefile`-based project**: FSL projects which use a FSL-style
`Makefile` to compile and install their deliverables (shared libraries,
scripts, and compiled executables).
- **`setup.py`-based project**: FSL projects which are written in Python,
and which have a `setup.py` file which is used to build the project as
a Python package.
The mechanisms by which projects of these types are built are slightly
different and, therefore, the conda recipe for projects of different types
will look slightly different. Examples of `Makefile`-based and
`setup.py`-based FSL projects, and associated conda recipes, can be found in
the `examples/cpp` and `examples/python` sub-directories. Some important
details are highlighted below.
### Writing the `meta.yaml` file
The `meta.yaml` file contains metadata about your project, including:
- The conda package name
- The URL of the project git repository
- The version number
- The build number (used in case the conda package for a single version
needs to be re-built for some reason).
- A list of the package build- and run-time dependencies.
**Essential metadata** In order to allow for automatic maintenance of FSL
conda recipes, you should define the essential metadata (name, version, etc)
as `jinja2` variables using the following syntax:
```
{% set name = <conda-package-name> %}
{% set version = <version-number> %}
{% set repository = <repository-url> %}
{% set build = '0' %}
```
Then use these variables within the recipe YAML, e.g.:
```
package:
name: {{ name }}
version: {{ version }}
```
**Project repository and git revision** The automated CI rules defined in
`fsl-ci-rules` allow the project repositoy and git revision (tag, branch, etc)
used to build a conda package to be overridden via the `FSLCONDA_REPOSITORY`
and `FSLCONDA_REVISION` environment variables. To facilitate this, the project
source repository and git revision must be specified in the `meta.yaml` like
so (remembering that we have defined `repository` and `version` as `jinja2
variables, above):
```
source:
# the FSLCONDA_REPOSITORY and FSLCONDA_REVISION
# environment variables can be used to override
# the repository/revision for development purposes.
git_url: {{ os.environ.get("FSLCONDA_REPOSITORY", repository) }}
git_rev: {{ os.environ.get("FSLCONDA_REVISION", version) }}
```
**run_exports (C/C++ projects only)** When compiling a C/C++ project, any
shared library dependencies of the project must be present at the time of
compilation, and at run time. This means that the dependencies of your project
may need to be listed twice within the `requirements` section - once under
`host` (or `build`), and again under `run`. This can be avoided by specifying
`run_exports` in the recipes of the dependencies.
So if you are writing a conda recipe for a C/C++ project which provides shared
library files that will be used by other projects, add a `run_exports` section to
the `build` section like so:
```
build:
number: {{ build }}
run_exports:
strong:
- {{ name }}
```
The
[`run_exports`](https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#export-runtime-requirements)
section is a trick which can be used within `meta.yaml`, which essentially
allows us to define a dependency as a build-time dependency, and have it
automatically propagated as a run-time dependency. It is used simply to allow
us to avoid having to list shared library dependencies twice.
**`noarch` and `script` (`setup.py`-based projects only)** Conda recipes for
Python `setup.py`-based projects are built slightly differently to native
projects, as the Python `setuptools` machinery is integrated into the conda
build process. For `setup.py`-based projects, the build command is usually
specified within the `build` section of the `meta.yaml` file. Furthermore, if
you are packging a "pure" Python project, with no natively compiled code or
extensions, you must label your recipe as being of type `noarch: python` -
this is also done within the `build` section. So a typical `build` section for
a `setup.py`-based project will resemble the following:
```
build:
number: {{ build }}
noarch: python
script: {{ PYTHON }} -m pip install . --no-deps --ignore-installed --no-cache-dir -vvv
```
**Requirements** The [`fsl/base`](https://git.fmrib.ox.ac.uk/fsl/base) project
provides the fundamental elements of the contents of a FSL installation,
including FSL initialisation scripts and the `Makefile` machinery. As such it
must be installed in order to build `Makefile`-based FSL projects, and must be
present at run time for many FSL commands to function. All FSL conda recipes
must therefore list `fsl-base` as a requirement. C/C++ projects will also need
to have a C++ compiler installed at build time - the convention within conda
recipes is to list compilers as `host` dependencies. For example:
```
requirements:
build:
- {{ compiler('cxx') }}
host:
- fsl-base >=2101.0
```
### Defining the build proceess
If you are writing a recipe for a `Makefile`-based FSL project, you need to
write a `build.sh` script which uses the FSL `Makefile` machinery to build
your project. A typical `build.sh` script will resemble the following:
```
#!/usr/bin/env bash
# $PREFIX is the installation destination
# when a conda package is being built
export FSLDIR=$PREFIX
# Install project source code into
# $PREFIX/src/ (a.k.a. $FSLDIR/src/)
mkdir -p $PREFIX/src/
cp -r $(pwd) $PREFIX/src/$PKG_NAME
# Configure the build environment
. $FSLDIR/etc/fslconf/fsl-devel.sh
# Build and install the project
make
make install
```
If you are writing a recipe for a `setup.py`-based FSL project, a `build.sh`
script is generally not required.
### `post-link.sh` and `pre-unlink.sh` scripts.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment