README.md 11.3 KB
Newer Older
Paul McCarthy's avatar
Paul McCarthy committed
1
2
3
4
# FSL release manifest


This repository is used to manage FSL releases. The `fsl-release.yml` file
5
defines the packages that are installed as part of a FSL release.
6
7
8
9

FSL release `manifest.json` and `environment.yml` files are automatically
generated from information stored in this repository, using CI rules and
scripts in the fsl/conda/manifest-rules> repository.
Paul McCarthy's avatar
Paul McCarthy committed
10
11


12
13
14
15
16
17
18
19
20
This repository contains two branches, `master` and `external`. These two
branches are used to create three different types of FSL release:

 - Public FSL releases (e.g. FSL 6.0.6) are created from **tags** on the
   `external` branch.
 - Development releases (development versions of public releases) are created
   from **commits** on the `external` branch.
 - Internal FSL releases (for internal use only) are created from **commits**
   on the `master` branch
21
22


Paul McCarthy's avatar
Paul McCarthy committed
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
## The `fslinstaller.py` script


FSL is installed using a Python 2/3 script called
[fslinstaller.py](https://git.fmrib.ox.ac.uk/fsl/conda/installer).


The `fslinstaller` script starts by downloading a _manifest_ file (referred
to as `manifest.json`), a JSON file which contains information about:

  - available FSL releases (past and present)
  - miniconda installer releases, upon which FSL installations are based
  - The latest available version of the fslinstaller script itself (used
    for self-updating)


Once the appropriate FSL release has been determined, the `fslinstaller`
downloads a FSL release file - A FSL release is defined by a conda
environment specification file (referred to as `environment.yml`), which
is a YAML file that contains a list of the individual packages and their
respective versions that comprise a specific FSL release. Separate
`environment.yml` files exist for each supported platform and CUDA version.


## Generating the `manifest.json` and `environment.yml` files


The `manifest.json` and `environment.yml` files are automatically generated
from information stored in this repository:

 - The `fsl-release.yml` file contains information about the `fslinstaller`,
   miniconda installers, and packages for the latest FSL release.
 - Information about past FSL releases is obtained from the most recently
   generated `manifest.json` file.


The `fsl-release.yml` file is a YAML file with the following structure:

```yaml
# URL to FSL release information. The fslinstaller.py
# script, and previously generated manifest.json
# file is assumed to be downloadable from this URL.
# URLs for newly generated environment files are
# constructed using this URL.
Paul McCarthy's avatar
Paul McCarthy committed
67
release-url: https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/releases/
Paul McCarthy's avatar
Paul McCarthy committed
68
69
70
71
72
73
74
75
76

# Latest fslinstaller version
installer: 1.2.3

# URLs to miniconda installers to
# use for all supported platforms
miniconda:
  linux-64: https://github.com/conda-forge/miniforge/releases/download/4.10.3-1/Miniforge3-4.10.3-1-Linux-x86_64.sh
  macos-64: https://github.com/conda-forge/miniforge/releases/download/4.10.3-1/Miniforge3-4.10.3-1-MacOSX-x86_64.sh
Paul McCarthy's avatar
Paul McCarthy committed
77
  macos-M1: https://github.com/conda-forge/miniforge/releases/download/4.10.3-1/Miniforge3-4.10.3-1-MacOSX-arm64.sh
Paul McCarthy's avatar
Paul McCarthy committed
78
79
80
81
82
83
84
85
86
87

# Versions of CUDA for which FSL packages are
# built (currently only available on Linux).
cuda:
 - 9.2
 - 10.2
 - 11.0

# Conda channels to source packages from
channels:
88
 - https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/
Paul McCarthy's avatar
Paul McCarthy committed
89
90
91
 - conda-forge
 - defaults

92
93
94
# Private channel containing packages that are
# only released internally. This is prepended
# to the channels list for internal releases.
Paul McCarthy's avatar
Paul McCarthy committed
95
internal-channel: http://${FSLCONDA_USERNAME}:${FSLCONDA_PASSWORD}@https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/internal/
Paul McCarthy's avatar
Paul McCarthy committed
96

Paul McCarthy's avatar
Paul McCarthy committed
97
98
# The FSL installation is created in two stages.
# First of all, the base_packages are installed.
Paul McCarthy's avatar
Paul McCarthy committed
99
100
# Versions of these packages are specified in
# the packages list below
Paul McCarthy's avatar
Paul McCarthy committed
101
base-packages:
Paul McCarthy's avatar
Paul McCarthy committed
102
103
104
105
 - fsl-base
 - fslpy
 - fsl-sub
 - libopenblas
Paul McCarthy's avatar
Paul McCarthy committed
106
107
 ...

Paul McCarthy's avatar
Paul McCarthy committed
108
109
110
111
# List of all packages that comprise a FSL
# installation. All packages not installed
# in the first stage (see base_packages
# above) are installed in the second stage.
Paul McCarthy's avatar
Paul McCarthy committed
112
packages:
Paul McCarthy's avatar
Paul McCarthy committed
113
114
 - ciftilib 1.5.3
 - fsleyes 1.3.0
Paul McCarthy's avatar
Paul McCarthy committed
115
 - fslpy 3.7.0
Paul McCarthy's avatar
Paul McCarthy committed
116
 - fsl-avwutils 2007.0
Paul McCarthy's avatar
Paul McCarthy committed
117
 - fsl-base 2109.1
Paul McCarthy's avatar
Paul McCarthy committed
118
 - fsl-eddy 2106.0
Paul McCarthy's avatar
Paul McCarthy committed
119
120
 - fsl-sub 2.5.8
 - libopenblas 0.3.18 *openmp*
Paul McCarthy's avatar
Paul McCarthy committed
121
 ...
Paul McCarthy's avatar
Paul McCarthy committed
122

Paul McCarthy's avatar
Paul McCarthy committed
123
124
# List of additional platform-specific
# packages to be installed.
Paul McCarthy's avatar
Paul McCarthy committed
125
linux-64-packages: []
Paul McCarthy's avatar
Paul McCarthy committed
126
macos-64-packages: []
Paul McCarthy's avatar
Paul McCarthy committed
127
macos-M1-packages: []
Paul McCarthy's avatar
Paul McCarthy committed
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
```


When changes are made to the `fsl-release.yml` file, new versions of the
`manifest.json` and `environment.yml` files are generated and made available
for download.


Note that there are two types of manifest file:

  - The official release manifest, containing information about past and
    present public FSL releases - this is always called `manifest.json`, and
    may be re-generated on new public releases, and on updates to e.g. the
    fslinstaller script.

143
144
145
146
147
  - Development manifests, containing information about one internal or
    development FSL release. Development manifests are generated for every
    internal or development release. Development manifests are named according
    to a standard convention described in the **Internal/development FSL
    releases** section, below.
Paul McCarthy's avatar
Paul McCarthy committed
148
149
150
151


Manifest and environment files can only be published when they are generated
from a new tag, or when they are generated from specific branches (currently
152
153
`master` or `external`). However, manifest and environment files can be
downloaded as GitLab artifacts from any branch.
Paul McCarthy's avatar
Paul McCarthy committed
154
155
156
157
158


## New FSL releases


159
_Summary:_
Paul McCarthy's avatar
Paul McCarthy committed
160
 - New public FSL release are created by adding a tag to the `external` branch
161
162
163
164
165
166
   of this repository
 - The `deploy-release-manifest` and `deploy-environment-files` jobs, on the
   CI pipeline for the new tag, must be manually started in order to publish
   the new release.


167
168
169
New public FSL releases are denoted by adding a tag to the `external` branch
of this repository. The tag is used as the FSL version identifier, which must
be of the form `X.Y.Z[.W]`, where X, Y, Z, and W are integers.
Paul McCarthy's avatar
Paul McCarthy committed
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184

When a new tag is added to this repository, the `manifest.json` and
`environment.yml` files are generated from the contents of `fsl-release.yml`,
and the contents of the previous `manifest.json` (used for obtaining
information about past FSL releases). These files are named according to these
conventions:

  - `manifest.json` - the release manifest
  - `fsl-<tag>_<platform>.yml` - FSL version `<tag>` for platform `<platform>`
    (e.g. `fsl-6.1.0_macos-64.yml`).
  - `fsl-<tag>_<platform>_cuda<cudaver>.yml` - FSL version `<tag>` for
    platform `<platform>` and CUDA version `<cudaver>`
    (e.g. `fsl-6.1.0_linux-64_cuda11.0.yml`).


185
186
187
188
189
190
After they have been generated, a test installation of the new release on each
supported platform is performed using the `fslinstaller.py` script, and then
the manifest and environment files are published to the FSL release web
server.  This publishing step must be manually triggered through the GitLab
web interface, via the `deploy-environment-files` and
`deploy-release-manifest` jobs.
Paul McCarthy's avatar
Paul McCarthy committed
191
192
193
194
195


## Internal/development FSL releases


196
_Summary:_
197
 - New internal FSL release may be created by adding commits to the `master`
198
   branch of this repository
199
 - New development releases may be created by adding commits to the `external`
200
201
202
203
204
205
   branch of this repository
 - In both cases, the `deploy-environment-files` and
   `deploy-development-manifest` jobs on the associated CI pipelines must be
   started manually.


206
When commits are added to the `master` or `external` branch of this
207
208
209
210
repository, separate `manifest.json` and `environment.yml` files are generated
from the contents of `fsl-release.yml` and made available for download.  These
files are assigned a version identifier of the form
`<tag>.<date>.<commit>.<branch>`, where:
Paul McCarthy's avatar
Paul McCarthy committed
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241


where:
  - `<tag>` is the _most recent_ public FSL release version identifier (e.g
    `6.1.0`).
  - `<date>` is the date of the commit that the files are generated from,
    in the form `YYYYMMDD`.
  - `<commit>` is the short hash of the git commit that the files are
    generated from.
  - `<branchname>` is the name of the branch on this repository.


The released files are then named as follows, where:

  - `manifest-<tag>.<date>.<commit>.<branch>.json` - the development release
    manifest file
  - `fsl-<tag>.<date>.<commit>.<branch>_<platform>.yml` - FSL environment file
    for platform `<platform>`.
  - `fsl-<tag>.<date>.<commit>.<branch>_<platform>_cuda<cudaver>.yml` - FSL
    environment file for platform `<platform>` (e.g. `linux-64`) and CUDA
    version `<cudaver>` (e.g. `11.0`).

After they have been generated, the manifest and environment files may be
published. As with public releases, the process of publishing the manifest and
environment files must be manually triggered through the GitLab web interface,
via the `deploy-environment-files` and `deploy-development-manifest` jobs.


## Updates to `manifest.json`


242
243
_Summary:_
 - The public release `manifest.json` file can be updated by adding commits to
244
   the `master` or `external` branches of this repository
245
246
247
248
 - The `deploy-release-manifest` job on the associated CI pipeline must be
   started manually.


Paul McCarthy's avatar
Paul McCarthy committed
249
250
251
252
253
254
255
256
257
258
Sometimes the contents of the official release `manifest.json` file may need
to be changed independently of a public FSL release - for example, a new
version of the `fslinstaller.py` script may be released, or the `miniconda`
installer may need to be changed.


Normally, `manifest.json` files that are not associated with a tag are named
`manifest-<tag>.<date>.<commit>.<branch>.json` as outlined above. However, a
new release `manifest.json` file is always re-generated, and can be deployed
through the Gitlab web interface, via the `deploy-release-manifest` job (with
259
the caveat that only files generated from the `master` or `external` branches
Paul McCarthy's avatar
Paul McCarthy committed
260
261
262
263
264
may be published).


Release `manifest.json` files which are generated from branches will only
contain information about past official FSL releases. Release `manifest.json`
265
files which are generated from new tags will contain information about past
Paul McCarthy's avatar
Paul McCarthy committed
266
official FSL release, and also about the new FSL release.
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294


## Development workflow


The `fsl-release.yml` file on the `master` branch should be updated as new
versions of FSL projects are released. This will result in new manifest and
environment files for internal installations to be created.


In order to create manifest and environment files for a new public FSL
release, the `fsl-release.yml` file on the `external` branch must be updated.
This can be accomplished using the following process:

1. Clone the `fsl/conda/manifest` repository locally:
   ```
   git clone git@git.fmrib.ox.ac.uk:fsl/conda/manifest.git
   cd manifest
   ```
2. Create a new branch off the `external` branch:
   ```
   git checkout -b rel/6.0.7 origin/external
   ```
3. Merge all changes from the `master` branch onto this branch:
   ```
   git merge origin/master
   ```
4. Resolve any conflicts that arise, making sure that internal-only FSL
295
296
297
298
299
300
   packages are not included in the package lists. A good idea is to compare
   your branch with the `master` branch - usually the only difference should
   be that your branch does not contain any internal-only packages:
   ```
   git diff origin/master HEAD
   ```
301
302
303
304
305
306
307
5. Merge your branch into the `external` branch on the `fsl/conda/manifest`
   GitLab repository via a merge request. This will result in new manifest
   and environment files for a new development release to be created.

6. If this is a public FSL release, create a new tag on the `external` branch.
   This will result in a new `manifest.json`, and environment files for the
   new FSL release to be created.