ENH: Allow tests to be run on cluster via fsl_sub

This MR makes several changes to the behaviour of pyfeeds, and adds the capability to submit tests to a HPC cluster via fsl_sub. This can be achieved with the new --fslsub option. There is also a new --fslsubOptions option which allows additional per-test submission options to be specified (e.g. jobram=4, coprocessor=cuda, etc). These can be added to a pyfeeds configuration file under a [fslsubOptions] section, e.g.:

[fslsubOptions]
*eddy* = coprocessor=cuda
*feat* = jobram=4 jobtime=120

Note that when using pyfeeds run --fslsub on the FMRIB cluster, you must specify a --sandboxDir location which is on the shared file system, i.e. which is accessible to both the submission node and cluster nodes.

This MR also fixes a race condition which can occur when running nested tests in parallel. For example, with the following test structure:

test1/feedsRun
test1/test2/feedsRun

The output directory structure will also be nested, i.e.:

test1/feedsRun.log
test1/test2/feedsRun.log

If these tests are run in parallel, test2 may be executed before test1, meaning that the test1/test2/ directory may already exist.

A number of other changes have been made:

  • Changes the method used to run tests in parallel from a ThreadPool to a multiprocessing.Pool - this is so that evaluation of test outputs can be run in parallel. A ThreadPool is still used when using the new --fslsub option.
  • The final pyfeeds results table now distinguishes between tests which fail due to a non-zero exit code (reported as Errored), and tests which fail due to their outputs not matching their benchmark data (reported as Failed).
  • If pyfeeds is called with the --overwrite flag, the test sandbox directory already exists, it is deleted.
  • Migrates the packaging configuration from using setup.py to pyproject.toml.
  • Changes the version number management strategy to use setuptools-scm.
Edited by Paul McCarthy

Merge request reports

Loading