README.md 7.13 KB
Newer Older
1
2
3
4
5
6
# FSL conda package infrastructure


_This documentation is a work in progress_


Paul McCarthy's avatar
Paul McCarthy committed
7
8
This repository hosts information and documentation regarding the FSL conda
package infrastructure.
9

10

Paul McCarthy's avatar
Paul McCarthy committed
11
12
This `README` file contains a general overview. More detailed information on
different topics can be found in the folowing files:
13

14

Paul McCarthy's avatar
Paul McCarthy committed
15
16
17
18
19
20
21
22
23
24
 - [`development_workflow.md`](development_workflow.md): Details on
   development and release workflows and processes.
 - [`official_fsl_installations.md`](official_fsl_installations.md): Details
   on official/full conda-based FSL installations.
 - [`local_development.md`](local_development.md): Setting up an FSL
   development environment on your own machine.
 - [`building_fsl_conda_packages.md`](building_fsl_conda_packages.md): Details
   on how to build a FSL conda package locally.
 - [`creating_fsl_conda_recipes.md`](creating_fsl_conda_recipes.md): Details
   on how to create a FSL conda recipe.
25
26


27
28
29
30
31
32
## Quick summary


FSL is organised into a number of repositories hosted at
https://git.fmrib.ox.ac.uk.

Paul McCarthy's avatar
Paul McCarthy committed
33
 - fsl/conda/docs> (this repository): High level documentation
Paul McCarthy's avatar
Paul McCarthy committed
34
35
36
37
38
39
 - fsl/conda/manifest>: Metadata for managing public and internal FSL
   releases
 - fsl/conda/manifest-rules>: CI configuration used by the
   fsl/conda/manifest> repository
 - fsl/conda/installer>: Home of the `fslinstaller.py` script, for
   installing FSL
Paul McCarthy's avatar
Paul McCarthy committed
40
 - fsl/conda/fsl-ci-rules>: CI rules for automated building and publishing
41
   of FSL conda packages
42
 - `fsl/<project>`: Source code repository for FSL project `<project>`
Paul McCarthy's avatar
Paul McCarthy committed
43
   (e.g. fsl/avwutils>)
44
 - `fsl/conda/fsl-<project>`: Conda recipe repository for FSL project
Paul McCarthy's avatar
Paul McCarthy committed
45
   `<project>` (e.g. fsl/conda/fsl-avwutils>)
46
47


48
49
## Overview

Paul McCarthy's avatar
Paul McCarthy committed
50
51
52
FSL is installed using the [**conda**](https://conda.io/) package and
environment management system. Each FSL project is built as a conda package,
and published to an internally hosted conda channel.
53

54

55
Prior to FSL 6.0.6, FSL was released and installed as a monolithic archive
Paul McCarthy's avatar
Paul McCarthy committed
56
57
58
59
file, containing most FSL tools, and a separate conda environment called
`fslpython`, configured after the archive file was installed. Some of the
reasons for transitioning to a purely conda-based release model are as
follows:
60
61
62

 - Simplify FSL installation procedure
 - Reduce the size of `$FSLDIR`
Paul McCarthy's avatar
Paul McCarthy committed
63
64
65
 - Allow "minimal" FSL installations, with only the utilities that are
   required
 - Improve build reproducibility
66
 - Improve support for container-based FSL installations
Paul McCarthy's avatar
Paul McCarthy committed
67
68
 - Simplify cross-platform management of external dependencies (e.g. VTK,
   OpenBLAS)
69
70


71
## FSL releases
72

73
74
75
76
77
78
79
80
81
82
83

Public and internal FSL release are defined by a set of conda
[`environment.yml`
files](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-file-manually),
one for each supported platform and CUDA version.  Each `environment.yml` file
contains a list of the packages which are included in the FSL release.


FSL releases are described by a `manifest.json` file, which is an index of all
available `environment.yml` files, and which contains all of the information
needed to install a particular FSL release.
84
85


86
87
88
89
The fsl/conda/manifest> repository is where new FSL releases, both public and
internal, are created. The `environment.yml` and `manifest.json` files for a
new release are automatically generated from information stored in the
`fsl-release.yml` file in that repository.
90

91

Paul McCarthy's avatar
Paul McCarthy committed
92
The `manifest.json` and `environment.yml` files for public and internal
Paul McCarthy's avatar
Paul McCarthy committed
93
94
release are currently published to
https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/releases/.
Paul McCarthy's avatar
Paul McCarthy committed
95
96


97
98
99
100
101
## The `fslinstaller.py` script

FSL releases are installed via a script called `fslinstaller.py`, which is
maintained at the fsl/conda/installer> repository.

102
The `fslinstaller.py` script performs the following tasks:
103

104
105
106
107
108
109
1. Downloads the FSL release `manifest.json` file, and the appropriate
   `environment.yml` file for the platform.
2. Downloads a `miniconda` installer.
3. Installs a base `miniconda` environment into `$FSLDIR` (where `$FSLDIR`
   defaults to `/usr/local/fsl/`, or is explicitly specified by the user).
4. Installs all FSL packages into the `$FSLDIR` base conda environment.
Paul McCarthy's avatar
Paul McCarthy committed
110
111
5. Sets up the user shell profile so that FSL is configured in their shell
   environment by default.
112
113
114
115
116


## Partial FSL installations


Paul McCarthy's avatar
Paul McCarthy committed
117
118
119
Users may choose to install specific FSL packages by installing the desired
FSL packages directly from the FSL conda channel into their own conda
environment.
120
121


Paul McCarthy's avatar
Paul McCarthy committed
122
123
124
125
126
127
> **Note:** All FSL conda packages list the dependencies required for their
> execution, so when a particular package is installed, all of its run-time
> dependencies will also be installed. However, as we transition away from
> monolithic installations and towards conda-based installations, there are
> likely to be omissions and errors in the dependency hierarchy. These issues
> will be resolved on a case-by-case basis.
128
129


130
131
## FSL conda packages

132

133
134
135
All FSL projects and their dependencies are published as conda packages. Most
FSL conda packages are published to an internally maintained channel, but some
packages are published to the `conda-forge` channel at https://anaconda.org.
136

137

Paul McCarthy's avatar
Paul McCarthy committed
138
139
140
141
142
All FSL projects, and any dependencies which are not available on
https://anaconda.org, are built according to a conda *recipe*, which describes
how the project can be built as a conda package. All FSL conda recipes are
hosted as git repositories on the FMRIB gitlab server, under the `fsl/conda/`
group.
143

144

Paul McCarthy's avatar
Paul McCarthy committed
145
146
To clarify, each FSL project, and internally managed dependency, comprises
**two** git repositories<sup>*</sup>:
147

Paul McCarthy's avatar
Paul McCarthy committed
148
149
150
151
* The **project** repository contains the project source code/resources, and
  is typically hosted at `https://git.fmrib.ox.ac.uk/fsl/<project>`
* The **recipe** repository contains a conda recipe for the project, and is
  hosted at `https://git.fmrib.ox.ac.uk/fsl/conda/fsl-<project>`.
152
153


Paul McCarthy's avatar
Paul McCarthy committed
154
155
156
FSL conda packages are built automatically using Gitlab CI - the CI
infrastructure for automatically building conda packages is hosted in the
fsl/fsl-ci-rules> repository.
157
158


Paul McCarthy's avatar
Paul McCarthy committed
159
160
161
162
163
> <sup>*</sup>A small number of FSL projects have more than one recipe
> repository associated with them - for example, the
> [`fsl/fdt`](https://git.fmrib.ox.ac.uk/fsl/fdt) project has two recipes -
> the [`fsl/conda/fsl-fdt`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-fdt)
> recipe provides CPU executables, and the
164
> [`fsl/conda/fsl-fdt-cuda`](https://git.fmrib.ox.ac.uk/fsl/conda/fsl-fdt-cuda)
Paul McCarthy's avatar
Paul McCarthy committed
165
> recipe provides GPU/CUDA executables.
166
167


168
169
## Public and internal channels

170

171
172
FSL projects are published to one of two conda internally managed conda
channels:
173

174
175
176
177
178
  - The **public** channel contains stable and development packages for most
    FSL projects.  Every time a new tag is added to a project repository, a
    stable conda package is built and published to the public channel. Every
    time commits are merged into the master branch of a repository, a
    development conda package is built and published to the public channel.
179

180
181
182
  - The **internal** channel contains stable and development packages for FSL
    projects which are not part of official FSL releases, and are only part of
    internal release.
183

Paul McCarthy's avatar
Paul McCarthy committed
184
These channels are currently hosted online at the following URLs:
185

Paul McCarthy's avatar
Paul McCarthy committed
186
187
 - https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/
 - https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/internal/