README.md 7.25 KB
Newer Older
Paul McCarthy's avatar
Paul McCarthy committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
# FSL release manifest


This repository is used to manage FSL releases. The `fsl-release.yml` file
defines the packages that are installed as part of a FSL release.


## The `fslinstaller.py` script


FSL is installed using a Python 2/3 script called
[fslinstaller.py](https://git.fmrib.ox.ac.uk/fsl/conda/installer).


The `fslinstaller` script starts by downloading a _manifest_ file (referred
to as `manifest.json`), a JSON file which contains information about:

  - available FSL releases (past and present)
  - miniconda installer releases, upon which FSL installations are based
  - The latest available version of the fslinstaller script itself (used
    for self-updating)


Once the appropriate FSL release has been determined, the `fslinstaller`
downloads a FSL release file - A FSL release is defined by a conda
environment specification file (referred to as `environment.yml`), which
is a YAML file that contains a list of the individual packages and their
respective versions that comprise a specific FSL release. Separate
`environment.yml` files exist for each supported platform and CUDA version.


## Generating the `manifest.json` and `environment.yml` files


The `manifest.json` and `environment.yml` files are automatically generated
from information stored in this repository:

 - The `fsl-release.yml` file contains information about the `fslinstaller`,
   miniconda installers, and packages for the latest FSL release.
 - Information about past FSL releases is obtained from the most recently
   generated `manifest.json` file.


The `fsl-release.yml` file is a YAML file with the following structure:

```yaml
# URL to FSL release information. The fslinstaller.py
# script, and previously generated manifest.json
# file is assumed to be downloadable from this URL.
# URLs for newly generated environment files are
# constructed using this URL.
release_url: http://18.133.213.73/releases/

# Latest fslinstaller version
installer: 1.2.3

# URLs to miniconda installers to
# use for all supported platforms
miniconda:
  linux-64: https://github.com/conda-forge/miniforge/releases/download/4.10.3-1/Miniforge3-4.10.3-1-Linux-x86_64.sh
  macos-64: https://github.com/conda-forge/miniforge/releases/download/4.10.3-1/Miniforge3-4.10.3-1-MacOSX-x86_64.sh

# Versions of CUDA for which FSL packages are
# built (currently only available on Linux).
cuda:
 - 9.2
 - 10.2
 - 11.0

# Conda channels to source packages from
channels:
 - http://18.133.213.73/public/
 - conda-forge
 - defaults

# Private internal channel for development and
# testing releases. This is prepended to the
# channels list for development releases.
internal_channel: http://${FSLCONDA_USERNAME}:${FSLCONDA_PASSWORD}@18.133.213.73/internal/

# List of FSL conda packages and their
# dependencies which comprise the FSL
# release.
packages:
 - fsl-avwutils=2007.0
 - fsl-eddy-cpu=2106.0
 - python=3.9.6
 ...
```


When changes are made to the `fsl-release.yml` file, new versions of the
`manifest.json` and `environment.yml` files are generated and made available
for download.


Note that there are two types of manifest file:

  - The official release manifest, containing information about past and
    present public FSL releases - this is always called `manifest.json`, and
    may be re-generated on new public releases, and on updates to e.g. the
    fslinstaller script.

  - Development manifests, containing information about one
    internal/development FSL release. Development manifests are generated for
    every internal release. Development manifests are named according to a
    standard convention described in the **Internal/development FSL releases**
    section, below.


Manifest and environment files can only be published when they are generated
from a new tag, or when they are generated from specific branches (currently
`master` or `internal`).


## New FSL releases


New public FSL releases are denoted by adding a tag to this repository. The
tag is used as the FSL version identifier, which must be of the form
`X.Y.Z[.W]`, where X, Y, Z, and W are integers.

When a new tag is added to this repository, the `manifest.json` and
`environment.yml` files are generated from the contents of `fsl-release.yml`,
and the contents of the previous `manifest.json` (used for obtaining
information about past FSL releases). These files are named according to these
conventions:

  - `manifest.json` - the release manifest
  - `fsl-<tag>_<platform>.yml` - FSL version `<tag>` for platform `<platform>`
    (e.g. `fsl-6.1.0_macos-64.yml`).
  - `fsl-<tag>_<platform>_cuda<cudaver>.yml` - FSL version `<tag>` for
    platform `<platform>` and CUDA version `<cudaver>`
    (e.g. `fsl-6.1.0_linux-64_cuda11.0.yml`).


After they have been generated, the manifest and environment files are
published to the FSL release web server.  This publishing step must be manually
triggered through the GitLab web interface, via the `deploy-environment-files`
and `deploy-release-manifest` jobs.


## Internal/development FSL releases


When commits are added to _any_ branch of this repository, separate
`manifest.json` and `environment.yml` files are generated from the contents of
`fsl-release.yml` and made available for download.  These files are assigned a
version identifier of the form `<tag>.<date>.<commit>.<branch>`, where:


where:
  - `<tag>` is the _most recent_ public FSL release version identifier (e.g
    `6.1.0`).
  - `<date>` is the date of the commit that the files are generated from,
    in the form `YYYYMMDD`.
  - `<commit>` is the short hash of the git commit that the files are
    generated from.
  - `<branchname>` is the name of the branch on this repository.


The released files are then named as follows, where:

  - `manifest-<tag>.<date>.<commit>.<branch>.json` - the development release
    manifest file
  - `fsl-<tag>.<date>.<commit>.<branch>_<platform>.yml` - FSL environment file
    for platform `<platform>`.
  - `fsl-<tag>.<date>.<commit>.<branch>_<platform>_cuda<cudaver>.yml` - FSL
    environment file for platform `<platform>` (e.g. `linux-64`) and CUDA
    version `<cudaver>` (e.g. `11.0`).

After they have been generated, the manifest and environment files may be
published. As with public releases, the process of publishing the manifest and
environment files must be manually triggered through the GitLab web interface,
via the `deploy-environment-files` and `deploy-development-manifest` jobs.


## Updates to `manifest.json`


Sometimes the contents of the official release `manifest.json` file may need
to be changed independently of a public FSL release - for example, a new
version of the `fslinstaller.py` script may be released, or the `miniconda`
installer may need to be changed.


Normally, `manifest.json` files that are not associated with a tag are named
`manifest-<tag>.<date>.<commit>.<branch>.json` as outlined above. However, a
new release `manifest.json` file is always re-generated, and can be deployed
through the Gitlab web interface, via the `deploy-release-manifest` job (with
the caveat that only files generated from the `master` or `internal` branches
may be published).


Release `manifest.json` files which are generated from branches will only
contain information about past official FSL releases. Release `manifest.json`
files which are generated from tags will contain information about past
official FSL release, and also about the new FSL release.