Merge branch 'master' into 'master'

V1.1.0 NIfTI-MRS

See merge request fsl/fsl_mrs!16

.gitlab-ci.yml

@@ -49,31 +49,15 @@ stages:
 # when and where jobs are executed.
 ####################################
 
-
-.only_upstream: &only_upstream
-  only:
-    - branches@fsl/fsl_mrs
-
-
-.only_master: &only_master
-  only:
-    - master
-
-
-.only_releases: &only_releases
-  only:
-    - tags@fsl/fsl_mrs
-
-
 .except_releases: &except_releases
   except:
     - tags
 
-
 # ############
 # # 1. style 
 # ############
 flake8:
+  <<: *except_releases
   image: python:3.7-slim-buster
   stage: style
   before_script:
@@ -81,6 +65,7 @@ flake8:
   - pip install flake8  
   script:
   - flake8 fsl_mrs
+  - flake8 fsl_mrs/scripts/*
   allow_failure: true
 
 ############
@@ -88,6 +73,7 @@ flake8:
 ############
 
 pytest:
+  <<: *except_releases
   image: wtclarke/fsl_mrs_tests:1.0
   stage: test
   variables:
@@ -108,10 +94,16 @@ pytest:
 ############
 
 pages:
-  <<: *only_master
+  variables:
+    GIT_LFS_SKIP_SMUDGE: "1"
+    GIT_STRATEGY: clone
+  only:
+    - master@wclarke/fsl_mrs
+    - tags@fsl/fsl_mrs
   image: python:3.7
   stage: doc
   script:
+  - git describe --tag --dirty
   - pip install -U sphinx sphinx_rtd_theme
   - pip install .
   - sphinx-build -b html ./docs/user_docs public

CHANGELOG.rst

 This document contains the FSL-MRS release history in reverse chronological order.
 
+1.1.0 (Thursday 18th February 2021)
+-----------------------------------
+- Support for NIfTI-MRS format.
+- Preprocessing scripts reoriented around NIfTI-MRS framework
+- New script results_to_spectrum for generating full fits in NIfTI-MRS format from fsl_mrs results.
+- Documentation and example data updated for move to NIfTI-MRS.
+- Added mrs_info command to give quick text summary of NIfTI-MRS file contents.
+- Updates to the WIP dynamic fitting module.
+
 1.0.6 (Tuesday 12th January 2021)
 ---------------------------------
 - Internal changes to core MRS class.
@@ -10,7 +19,7 @@ This document contains the FSL-MRS release history in reverse chronological orde
 - Synthetic spectra now use fitting model directly.
 - Bug fixes in the fsl_Mrs commandline interface. Thanks to Alex Craig-Craven.
 - WIP: Dynamic fitting model and dynamic experiment simulation.
-- spec2nii requirement pinned to 0.2.11 during NIFTI-MRS development.
+- spec2nii requirement pinned to 0.2.11 during NIfTI-MRS development.
 
 1.0.5 (Friday 9th October 2020)
 -------------------------------

README.md

@@ -52,6 +52,8 @@ After installation see the [quick start guide](https://open.win.ox.ac.uk/pages/f
 : simulate basis spectra
 - **mrs_vis**
 : quick visualisation of the spectra or basis spectra
+- **mrs_info**
+: quick information on NIfTI MRS file
 - **svs_segment & mrsi_segment**
 : Run tissue segmentation for SVS/MRSI from T1 image.
 

docs/user_docs/conf.py

@@ -26,6 +26,7 @@ copyright = f'{date.year}, Will Clarke & Saad Jbabdi, University of Oxford, Oxfo
 author = 'William Clarke'
 
 # The full version, including alpha/beta/rc tags
+print(f'fsl_mrs version: {fsl_mrs.__version__}')
 version = fsl_mrs.__version__
 release = version
 

docs/user_docs/data_conversion.rst

@@ -2,21 +2,23 @@
 
 Data Conversion
 ===============
-There is a plethora of spectroscopy data formats in existence. Many are vendor specific and proprietary. The fitting capabilities of FSL-MRS can be used with either NIfTI or an ASCII file format, but to access the full features of the post-processing tools data must be in the NIfTI + JSON format. To facilitate the conversion of data to this format FSL-MRS is distributed with an additional conversion tool :code:`spec2nii`. Spec2nii currently converts SVS and MRSI data to NIfTI from the following formats. 
+There is a plethora of spectroscopy data formats in existence. Many are vendor specific and proprietary. The fitting capabilities of FSL-MRS can be used with either NIfTI or an ASCII file format, but to access the full features of the post-processing tools data must be in the NIfTI-MRS format. To facilitate the conversion of data to this format FSL-MRS is distributed with an additional conversion tool :code:`spec2nii`. Spec2nii currently converts SVS and MRSI data to NIfTI-MRS from the following formats. 
 
 =============== ================ ===== ===== =======================
  Format          File extension   SVS   CSI   Automatic orientation  
 =============== ================ ===== ===== ======================= 
  Siemens Twix    .dat             Yes   No    Yes                    
  Siemens DICOM   .ima / .dcm      Yes   Yes   Yes                    
- Philips         .SPAR/.SDAT      Yes   No    No                     
- GE              .7 (pfile)       Yes   No    No                     
+ Philips         .SPAR/.SDAT      Yes   No    Yes                     
+ GE              .7 (pfile)       Yes   No    Yes
+ UIH DICOM       .dcm             Yes   Yes   Yes 
  LCModel         .RAW             Yes   No    No                     
- jMRUI           .txt             Yes   No    No                     
+ jMRUI           .txt             Yes   No    No
+ jMRUI           .mrui            Yes   No    No
  ASCII           .txt             Yes   No    No                     
 =============== ================ ===== ===== =======================
 
-The authors of the tool are happy to provide additional conversion routines if sample data and a thorough description of the format is provided. Please see the spec2nii `project page on Github <https://github.com/wexeee/spec2nii>`_.
+Bruker format conversion is currently under development. The authors of the tool are happy to provide additional conversion routines if sample data and a thorough description of the format is provided. Please see the spec2nii `project page on Github <https://github.com/wexeee/spec2nii>`_.
 
 Use of spec2nii 
 ---------------
@@ -44,6 +46,8 @@ Spec2nii can then be run to convert all data from a single ‘evalinfo’ flag.
 
 ‘Image’ is the most typically used for the main dataset. Other flags might be used for noise data, water reference data or any other use specified by the sequence programmer. 
 
+Twix format loop variables (e.g. `Ave` or `ida`) can be assigned to specific NIfTI-MRS dimensions using the :code:`-d{5,6,7}` command line options. NIfTI MRS dimension tags (e.g. `DIM_COIL`) can be specified using the :code:`-t{5,6,7}` command line options.
+
 DICOM
 ~~~~~
 
@@ -51,17 +55,28 @@ Spec2nii can be passed a single file or directory of DICOM files for conversion.
 ::
 
     spec2nii dicom <file_or_dir>
- 
+
+NIfTI-MRS dimension tags (e.g. `DIM_COIL`) can be specified using the `-t` command line argument.
+
+UIH DICOM
+~~~~~~~~~
+Conversion for UIH DICOM format SVS and CSI spectroscopy.
+::
+
+    spec2nii uih DCM_FILE_or_DIR
+
+NIfTI MRS dimension tags (e.g. `DIM_COIL`) can be specified using the `-t` command line argument.
+
 Philips
 ~~~~~~~
-Conversion for Philips SDAT/SPAR files. Orientation information not validated.
+Conversion for Philips SDAT/SPAR files.
 ::
 
     spec2nii philips <SDAT> <SPAR>
 
 GE
 ~~
-Conversion for GE pfiles (.7). Orientation information not validated.
+Conversion for GE pfiles (.7).
 ::
 
     spec2nii GE <file>

docs/user_docs/introduction.rst

@@ -18,4 +18,4 @@ Citing FSL-MRS
 --------------
 If you use FSL-MRS please cite: 
 
-    Clarke WT, Stagg CJ, Jbabdi S. FSL-MRS: An end-to-end spectroscopy analysis package. Biorxiv 2020 
\ No newline at end of file
+    Clarke WT, Stagg CJ, Jbabdi S. FSL-MRS: An end-to-end spectroscopy analysis package. MRM https://doi.org/10.1002/mrm.28630
\ No newline at end of file

docs/user_docs/processing.rst

@@ -4,13 +4,13 @@ Processing
 **For SVS**
 
 
-Processed SVS data comprises a time domain signal localised to a single spatial volume. This is Fourier transformed to give a frequency domain spectrum. In an un-processed state, the data might not be coil-combined, might have multiple transients needing averaging and might have other acquisition loops requiring specific processing. Typically, SVS data requires no “reconstruction” per-se, but several steps must be followed to achieve the highest quality data possible from an acquisition. 
+Processed SVS data comprises a time domain signal localised to a single spatial volume. This is Fourier transformed to give a frequency domain spectrum. In an un-processed state, the data might not be coil-combined, might have multiple transients needing averaging, and might have other acquisition loops requiring specific processing. Typically, SVS data requires no “reconstruction” per-se, but several steps must be followed to achieve the highest quality data possible from an acquisition. 
 
-For a complete overview of pre-processing we recommend [NEAR20]_. In short, the data must be coil combined, processed to remove small frequency drifts and corrupted transients, averaged, corrected for eddy currents and phased. Optionally large residual water signals can be removed. This is summarised in the table below.
+For a complete overview of pre-processing we recommend [NEAR20]_. In short, the data must be coil combined, processed to remove small frequency drifts and corrupted transients, averaged, corrected for eddy currents, and finally phased. Optionally, large residual water signals can be removed. This is summarised in the table below.
 
 **For MRSI**
 
-MRSI comprises an “image” of spectroscopy data, each voxel contains time or frequency domain data. MRSI data will require reconstruction from the raw (k-space) data collected by the scanner. This may be carried out either online (on the scanner) or offline. Typically, this reconstruction incorporates some of the steps described above for SVS data (e.g. coil combination or averaging). Other steps used for SVS processing would not be commonly used for MRSI data (e.g. Bad average removal). However, the majority of the fsl_mrs_proc commands can be run on MRSI data stored in NIfTI format where the processing will be applied independently per-voxel. 
+MRSI comprises an “image” of spectroscopy data, each voxel contains time or frequency domain data. MRSI data will require reconstruction from the raw (k-space) data collected by the scanner. This may be carried out either online (on the scanner) or offline. Typically, this reconstruction incorporates some of the steps described above for SVS data (e.g. coil combination or averaging). Other steps used for SVS processing would not be commonly used for MRSI data (e.g. bad average removal). However, the majority of the fsl_mrs_proc commands can be run on MRSI data stored in NIfTI format where the processing will be applied independently per-voxel. 
 
 Due to the complexity and specialism of MRSI reconstruction FSL-MRS does not provide MRSI reconstruction. Nor do we advise application of pre-processing beyond that the data should be coil-combined and repetitions averaged before fitting. 
 
@@ -25,10 +25,10 @@ Subcommands
 ======================= ==============================================================
 fsl_mrs_proc operation	 Description	
 ======================= ==============================================================
-coilcombine	             Combine individual coils of receiver phased array.
+coilcombine	         Combine individual coils of receiver phased array.
 average             	 Average FIDs, with optional complex weighting.	
 align               	 Phase and frequency align FIDs using spectral registration.
-align-diff	             Phase and frequency align sub-spectra for differencing.
+align-diff	         Phase and frequency align sub-spectra for differencing.
 ecc  	                 Eddy current correction using a water phase reference scan.
 remove	                 Remove peak (typically residual water) using HLSVD.
 tshift	                 shift/re-sample in time domain.	
@@ -68,7 +68,7 @@ fsl_mrs_preproc
 ---------------
 
 :code:`fsl_mrs_preproc` combines a number of processing steps to provide a one step processing of non-edited SVS data.
-The script requires a list of transients to be averaged (--data), water reference data (--reference) and an output location (--output). The data can be coil combined or un-combined but must be consistent. 
+The script requires the user to provide unsuppressed data (--data), water reference data (--reference) and an output location (--output). The data can be coil combined or un-combined but must be consistent. 
 ::
 
     fsl_mrs_preproc --output my_subj --data metab*.nii.gz --reference wref*.nii.gz --report 
@@ -88,9 +88,9 @@ Python & Interactive Interface
 To access the processing methods in either a python or interactive python enviroment load the `preproc` module
 ::
 
-    import fsl_mrs.utils.preproc
+    from fsl_mrs.utils.preproc import nifti_mrs_proc
 
-Reports can be generated using the associated [subcmd]_report functions in the preproc submodules.
+Reports and figures can be generated using the :code:`figure` and :code:`report` keyword arguments.
 
 fsl_mrs_proc subcommand specifics
 ---------------------------------
@@ -99,7 +99,7 @@ fsl_mrs_proc subcommand specifics
         Takes a list of files (:code:`--file`) and runs a weighted SVD [RODG10]_ coil combination on them optionally using a single water reference dataset (:code:`--reference`) to calculate the complex weightings of each coil. The function expects data to be stored as 5D data, with the last dimension storing individual coil data. Each file is treated separately. Pre-whitening can be disabled (:code:`--noprewhiten`). 
 
 2. average (averaging) 
-        Takes either a single file or list of files (:code:`--file`) and takes the mean across the list of files (:code:`--avgfiles`) or across a certain dimension (:code:`--dim`, indexes from 0). 
+        Takes a file as input (:code:`--file`) and takes the mean across across a certain dimension (:code:`--dim`, either a NIfTI-MRS tag or dim index (5, 6, 7).
 
 3. align (phase-frequency alignment) 
         Takes a list of files (:code:`--file`) and aligns each FID to the FID nearest to the mean, or to a single passed reference FID (:code:`--reference`). The ppm range can be defined (:code:`--ppm`, default = 0.2->4.2 ppm). 

docs/user_docs/quick_start.rst

@@ -11,41 +11,54 @@ Below we describe an end-to-end pipeline that includes data conversion, processi
 
 1. Convert your data
 ~~~~~~~~~~~~~~~~~~~~
-Before running FSL-MRS, time domain data must be prepared in a complex 4D-NIFTI + json format. You can do this by running the accompanying spec2nii tool (see :ref:`Data Conversion <data_conversion>`).
+Before running FSL-MRS, time domain data must be prepared in a complex NIfTI-MRS format. You can do this by running the accompanying spec2nii tool (see :ref:`Data Conversion <data_conversion>`).
 
 Example conversion::
 
-    spec2nii dicom -f my_metab_file -j metab.dcm
+    spec2nii dicom -f my_metab_file metab.dcm
 
-This will convert the dicom file (metab.dcm) to a NIfTI file named my_metab_file.nii and because the -j option was specified, a JSON file called my_metab_file.json. Conversion of other formats is possible by changing the first argument "dicom" to another option. For a list of supported formats see :ref:`Data Conversion <data_conversion>`.
+This will convert the dicom file (metab.dcm) to a NIfTI file named my_metab_file.nii.gz. Conversion of other formats is possible by changing the first argument "dicom" to another option. For a list of supported formats see :ref:`Data Conversion <data_conversion>`.
 
 A directory of DICOM data can also be converted::
 
-    spec2nii dicom -f my_metab_file -j ./dcm_metab_dir/
+    spec2nii dicom -f my_metab_file ./dcm_metab_dir/
 
-The output of the above command will be `my_metab_file_000.nii.gz, *_001.nii.gz, *_002.nii.gz, ...` continuing up to the number of DICOM instances in the directory. This number should match the number of transients acquired in an SVS sequence. There will also be matching JSON files: `my_metab_file_000.json, *_001.json, *_002.json, ...`
+The output of the above command will also be `my_metab_file.nii.gz` but the converter will append dynamics into higher NIfTI-MRS dimensions (continuing up to the number of DICOM instances in the directory).
 
 You might need to covert multiple files to process a single spectroscopy acquisition. A typical single voxel dataset will have both water suppressed and water unsuppressed data. Depending on format these might be contained in a single raw data file or be spread over two or more. Some protocols may acquire additional data to be used in pre-processing, e.g. for eddy-current correction. These files should be converted with a different name::
 
-    spec2nii dicom -f my_wref_file -j ./dcm_wref_dir/
-    spec2nii dicom -f my_ecc_file -j ./dcm_ecc_dir/
+    spec2nii dicom -f my_wref_file ./dcm_wref_dir/
+    spec2nii dicom -f my_ecc_file ./dcm_ecc_dir/
 
 But note that there are frequently multiple calibration scans for e.g. shimming and water suppression calibration acquired before the actual MRS acquisition. These files aren't used for analysis and can be safely ignored.
 
 1.1 Take a look at your data
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-You can use :code:`mrs_vis` on the command line to view your data at any stage of the process::
+You can use :code:`mrs_info` and :code:`mrs_vis` on the command line to view your data at any stage of the process. First mrs_info to see the dimensionality of the data::
 
-    mrs_vis my_metab_file_000.nii.gz
+    mrs_info my_metab_file.nii.gz
 
-You should see some **noisy** data
+    Read file my_metab_file.nii.gz (/path_to_file).
+    NIfTI-MRS version 0.2
+    Data shape (1, 1, 1, 4096, 32, 64)
+    Dimension tags: ['DIM_COIL', 'DIM_DYN', None]
+    Spectrometer Frequency: 297.219948 MHz
+    Dwelltime (Bandwidth): 8.330E-05s (12005 Hz)
+    Nucleus: 1H
+    Field Strength: 6.98 T
+
+Then mrs_vis to visualise the data::
+
+    mrs_vis my_metab_file.nii.gz
+
+:code:`mrs_vis` will automatically perform coil combination and averaging down to a single spectrum for display purposes only.
 
 .. image:: data/raw_conv.png
     :width: 600
 
-You can also see all the converted data in a directory::
+You can also quickly view data across one of the NIfTI-MRS higher dimensions (those containing uncombined coils, or averages etc.) In this case we plot all the transients stored in the dimension tagged *DIM_DYN* (i.e. averages)::
 
-    mrs_vis ./my_conv_data/
+    mrs_vis my_metab_file.nii.gz --display_dim DIM_DYN
 
 .. image:: data/mrs_vis_dir.png
     :width: 600
@@ -56,15 +69,15 @@ Have a look at the :ref:`Visualisation <visualisation>` page for more informatio
 
 2. Process your raw data
 ~~~~~~~~~~~~~~~~~~~~~~~~
-Some data requires pre-processing. Often MRSI data will have gone through appropriate pre-processing during reconstruction, if so skip to step 3. For unprocessed single voxel (SVS) data, read on.
+Some data requires pre-processing. Often MRSI data will have gone through appropriate pre-processing during reconstruction, if so skip to step 3. For unprocessed single-voxel (SVS) data, read on.
 
-Use the :code:`fsl_mrs_proc` commands to pre-process your raw data. :code:`fsl_mrs_proc` contains routines for many common steps (e.g. coil combination, phase-frequency alignment, residual water removal). E.g.::
+Use the :code:`fsl_mrs_proc` commands to pre-process your raw data. :code:`fsl_mrs_proc` contains routines for many common processing steps (e.g. coil combination, phase-frequency alignment, residual water removal). For example::
 
-    fsl_mrs_proc coilcombine --file my_metab_file*.nii.gz --reference my_wref_file.nii.gz --output combined -r
-    fsl_mrs_proc align       --file combined*.nii.gz --ppm 1.8 3.5                        --output aligned -r
-    fsl_mrs_proc average     --file aligned*.nii.gz --avgfiles                            --output avg -r
-    fsl_mrs_proc remove      --file avg.nii.gz                                            --output water_removed -r
-    fsl_mrs_proc phase       --file water_removed.nii.gz                                  --output metab -r
+    fsl_mrs_proc coilcombine --file my_metab_file.nii.gz --reference my_wref_file.nii.gz --output combined -r
+    fsl_mrs_proc align       --file combined.nii.gz --ppm 1.8 3.5                        --output aligned -r
+    fsl_mrs_proc average     --file aligned.nii.gz  --dim DIM_DYN                        --output avg -r
+    fsl_mrs_proc remove      --file avg.nii.gz                                           --output water_removed -r
+    fsl_mrs_proc phase       --file water_removed.nii.gz                                 --output metab -r
 
 The -r requests a HTML report to be generated for each stage of the processing. The different HTML reports can be merged using::
 
@@ -74,7 +87,7 @@ If your data is unedited single voxel (SVS) try out the prepackaged processing p
 
 ::
 
-    fsl_mrs_preproc --output processed --data my_metab_file*.nii.gz --reference my_wref_file*.nii.gz --report 
+    fsl_mrs_preproc --output processed --data my_metab_file.nii.gz --reference my_wref_file.nii.gz --report 
 
 Have a look at the source code for :code:`fsl_mrs_preproc` to see how you can construct your own python script using the processing modules. You can always prototype using Jupyter/IPython (see :ref:`Demos <demos>`)
 
@@ -128,9 +141,9 @@ For visualising MRSI data, fits, and fitting results, `FSLeyes
 
 Demos
 -----
-Two demo Jupyter notebooks are provided alongside some sample data in the `example_usage` directory. These notebooks show an example processing pipeline implemented both on the command-line and in interactive python.
+Demo Jupyter notebooks are provided alongside some sample data in the `example_usage` directory. These notebooks show an example processing pipeline implemented both on the command-line and in interactive python.
 
-To access these clone the |fslmrs_gitlab|_ with `Git LFS <https://git-lfs.github.com/>`_ installed, or download directly from |fslmrs_pkg_data|_.
+To access these clone the |fslmrs_gitlab|_ with `Git LFS <https://git-lfs.github.com/>`_ installed.
 
 You will need to have jupyter notebook installed::
 

docs/user_docs/visualisation.rst

@@ -15,7 +15,7 @@ There are 4 ways of visualising/interacting with MRS data in FSL-MRS:
 1. Quick glance
 ---------------
 
-The first thing one might want to do when given a FID file or simulated spectra is to have a quick look at the FFTs to see if they look like one would expect. FSL-MRS provides a light-weight script (:code:`mrs_vis`) to quickly glance at the MRS or MRSI data. For example, running :code:`mrs_vis` on the provided example SVS data:
+The first thing one might want to do when given a FID file or simulated spectra is to have a quick look at the spectra to see if they look like one would expect. To get a sense of the dimensionality and basic status of the data run :code:`mrs_info` for a quick text summary. FSL-MRS then provides a light-weight script (:code:`mrs_vis`) to quickly visualise the MRS or MRSI data. For example, running :code:`mrs_vis` on the provided example SVS data:
 
 ::
 
@@ -26,7 +26,7 @@ gives the following basic plot:
 .. image:: data/mrs_vis_svs.png
   :width: 400
 
-Note that the reason :code:`mrs_vis` "knows" how to scale the x-axis is that the relevant information is stored in the JSON sidecar (namely the *dwell time* and the *central frequency*).
+Note that the reason :code:`mrs_vis` "knows" how to scale the x-axis is that the relevant information is stored in the NIfTI-MRS MRS header extension (namely the *dwell time* and the *central frequency*).
 
 :code:`mrs_vis` can also visualise a folder of mrs data::
 

example_usage/Example MRSI fitting.ipynb

@@ -46,20 +46,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[]"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "outputs": [],
    "source": [
     "#%%capture\n",
     "%sx mkdir -p MRSI\n",
@@ -79,20 +68,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[]"
-      ]
-     },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "outputs": [],
    "source": [
     "%sx fsl_mrsi --data example_data/example_mrsi/mrsi.nii.gz \\\n",
     "             --basis example_data/example_mrsi/3T_slaser_32vespa_1250.BASIS \\\n",
@@ -123,26 +101,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "['total 272',\n",
-       " 'drwxr-xr-x   6 saad  staff     192 15 Jun 21:45 concs',\n",
-       " 'drwxr-xr-x   8 saad  staff     256 15 Jun 21:45 fit',\n",
-       " '-rw-r--r--   1 saad  staff  134462 15 Jun 21:43 fit_avg.png',\n",
-       " '-rw-r--r--   1 saad  staff    1388 15 Jun 21:43 options.txt',\n",
-       " 'drwxr-xr-x  50 saad  staff    1600 15 Jun 21:45 qc',\n",
-       " 'drwxr-xr-x  31 saad  staff     992 15 Jun 21:45 uncertainties']"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "outputs": [],
    "source": [
     "%sx ls -l MRSI/example_mrsi_fit"
    ]
@@ -163,7 +124,9 @@
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "%sx fsleyes example_data/example_mrsi/T1.anat/T1.nii.gz example_data/example_mrsi/mrsi.nii.gz MRSI/example_mrsi_fit/fit/fit.nii.gz MRSI/example_mrsi_fit/fit/baseline.nii.gz MRSI/example_mrsi_fit/concs/internal/NAA+NAAG.nii.gz"
+   ]
   }
  ],
  "metadata": {
@@ -182,7 +145,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.6"
+   "version": "3.7.7"
   }
  },
  "nbformat": 4,
 %% Cell type:markdown id: tags:
 
 # Example of MRSI fitting on the command line
 
 This notebook demos the process of fitting an MRSI scan using the command line scripts included in FSL-MRS.
 ### Contents:
 - [1. Reconstruction, Processing and Data Conversion](#1.-Reconstruction,-processing-and-data-conversion)
 - [2. Segmentation](#2.-Tissue-segmentation)
 - [3. Fitting](#3.-Fitting)
 - [4. Visualisation of fit](#4.-Visualisation-of-fit)
 
 Will Clarke
 June 2020
 University of Oxford
 
 %% Cell type:markdown id: tags:
 
 ## 1. Reconstruction, processing and data conversion
 
 MRSI reconstruction (from k-space data) can be highly specialised depending on the sequence. For example reconstruction of non-cartesian trajectories (e.g. spirals or concentric-rings) requires regridding or use of the NUFFT. Due to the specialised nature of MRSI reconstruction FSL-MRS does not contain tools for this step.
 
 Simmilarly post-processing of MRSI data is commonly done as part of the reconstruction process. Though many of the processing tools in FSL-MRS can be run on MRSI data they have not been created with MRSI in mind.
 
 This example therefore assumes that you are able to provide reconstructed and processed data ready for fitting. Data can be converted to NIfTI from the Siemens DICOM format using spec2nii. The authors of FSL-MRS and spec2nii are happy to add supported formats if example data and interpretation is provided.
 
 %% Cell type:markdown id: tags:
 
 ## 2. Tissue segmentation
 
 Tissue segmentation is required to have meaningful scaling of water referenced metabolite concentrations. mrsi_segment will produce three files, corresponding to the GM, WM and CSF FAST tissue segmentations registered to the MRSI volume.
 
 Run tissue segmentation on the packaged T1 data and mask using the MRSI data file. Here we provide a (partial) .anat file produced by [fsl_anat](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/fsl_anat) to speed up execution.
 
 This step requires an installation of FSL compatible with fslpy.
 
 %% Cell type:code id: tags:
 
 ``` python
 #%%capture
 %sx mkdir -p MRSI
 %sx mrsi_segment -o MRSI -a example_data/example_mrsi/T1.anat example_data/example_mrsi/mrsi.nii.gz
 ```
 
-%%%% Output: execute_result
-
-    []
-
 %% Cell type:markdown id: tags:
 
 ## 3. Fitting
 
 The below is a call to the main MRSI wrapper script. Not the use of the :code:`--add_MM` flag as this metabolite basis does not contain a macromolecule basis. Also note that the mrsi dataset provided is accompagnied by a JSON sidecar file which contains some useful information such as the echo time, the central frequency, and the dwell time.
 
 The script will by default run in parallel on the available CPU cores. Depending on hardware this should take a few minutes.
 
 %% Cell type:code id: tags:
 
 ``` python
 %sx fsl_mrsi --data example_data/example_mrsi/mrsi.nii.gz \
              --basis example_data/example_mrsi/3T_slaser_32vespa_1250.BASIS \
              --output MRSI/example_mrsi_fit \
              --mask example_data/example_mrsi/mask.nii.gz \
              --h2o example_data/example_mrsi/wref.nii.gz \
              --tissue_frac MRSI/mrsi_seg_wm.nii.gz MRSI/mrsi_seg_gm.nii.gz MRSI/mrsi_seg_csf.nii.gz \
              --add_MM \
              --baseline_order 2 \
              --combine PCho GPC --combine Cr PCr --combine NAA NAAG --combine Glu Gln --combine Glc Tau \
              --ignore Gly HG \
              --overwrite
 ```
 
-%%%% Output: execute_result
-
-    []
-
 %% Cell type:markdown id: tags:
 
 ## 4. Visualisation of fit
 
 Now take a look at the outputs. A PNG of the fit to the average of all voxels is provided for a quick sanity check. The folders contain the following:
 
 - concs         : concentration for each metabolite or combined metabolites (subfolders contain different types of referencing)
 - fit           : the model prediction FID, the residual FID, and the baseline (also in the time domain).
 - qc            : QC parameters split per metabolite
 - uncertainties : the std of the fit per metabolite
 
 %% Cell type:code id: tags:
 
 ``` python
 %sx ls -l MRSI/example_mrsi_fit
 ```
 
-%%%% Output: execute_result
-
-    ['total 272',
-     'drwxr-xr-x   6 saad  staff     192 15 Jun 21:45 concs',
-     'drwxr-xr-x   8 saad  staff     256 15 Jun 21:45 fit',
-     '-rw-r--r--   1 saad  staff  134462 15 Jun 21:43 fit_avg.png',
-     '-rw-r--r--   1 saad  staff    1388 15 Jun 21:43 options.txt',
-     'drwxr-xr-x  50 saad  staff    1600 15 Jun 21:45 qc',
-     'drwxr-xr-x  31 saad  staff     992 15 Jun 21:45 uncertainties']
-
 %% Cell type:markdown id: tags:
 
 Now run the command below in a terminal in order to load the data in FSLeyes. Follow the instructions [here](https://users.fmrib.ox.ac.uk/~saad/fsl_mrs/html/visualisation.html#fsleyes) to set it up in such a way that you can explore each voxel's individual fit.
 
 ```
 fsleyes example_data/example_mrsi/T1.anat/T1.nii.gz example_data/example_mrsi/mrsi.nii.gz MRSI/example_mrsi_fit/fit/fit.nii.gz MRSI/example_mrsi_fit/fit/baseline.nii.gz MRSI/example_mrsi_fit/concs/internal/NAA+NAAG.nii.gz
 ```
 
 %% Cell type:code id: tags:
 
 ``` python
+%sx fsleyes example_data/example_mrsi/T1.anat/T1.nii.gz example_data/example_mrsi/mrsi.nii.gz MRSI/example_mrsi_fit/fit/fit.nii.gz MRSI/example_mrsi_fit/fit/baseline.nii.gz MRSI/example_mrsi_fit/concs/internal/NAA+NAAG.nii.gz
 ```

example_usage/Example SVS processing - command-line.ipynb

@@ -32,11 +32,9 @@
    "source": [
     "## 1. File conversion using spec2nii\n",
     "\n",
-    "Run spec2nii twix -v to establish the file contents, then using this information run spec2nii twix -e with the appropriate flags to extract the required scans.\n",
+    "Run spec2nii twix -v to establish the file contents, then using this information run spec2nii twix -e with the appropriate flags to extract the required scans. The -q flag suppresses text output.\n",
     "\n",
-    "The -j option causes a JSON side-car file to be generated. The -q flag suppresses text output.\n",
-    "\n",
-    "This dataset uses a modified versions of the cmrr spectro package sequences on a siemens scanner. It has three sets of water reference scans. The first is taged as a phase correction, iscollected at the start of the main suppressed water scan and will be used for eddy current correction.\n",
+    "This dataset uses a modified versions of the CMRR spectro package sequences on a Siemens scanner. It has three sets of water reference scans. The first is tagged as a phase correction, ans is collected at the start of the main suppressed water scan and will be used for eddy current correction.\n",
     "\n",
     "The second is collected in a separate scan with only the RF portion of the water suppression disabled. This is used for coil combination (and could be used for eddy current correction). The third is collected with the OVS and all aspects of the water suppression disabled. It therefore experiences eddy currents unlike all the other scans. It will be used for final concentration scaling."
    ]
@@ -60,7 +58,7 @@
     "The first is tagged as \"image\" and contains 64 repetitions (sets) of 4096 point FIDs collected on 32 channels.  \n",
     "The second has a single FID on each channel.\n",
     "\n",
-    "We now call spec2nii again specifying the group we want to extract each time. spec2nii will generate a NIfTI file for each repetition (set) with a size of 1x1x1x4096x32.  \n",
+    "We now call spec2nii again specifying the group we want to extract each time. Each call to spec2nii will generate a NIfTI MRS file with a size of 1x1x1x4096x32xNdyn, where Ndyn is the number of dynamics (repeats).\n",
     "  \n",
     "We repeat this for the water reference scans, extracting just the image data."
    ]
@@ -72,11 +70,11 @@
    "outputs": [],
    "source": [
     "%%bash\n",
-    "spec2nii twix -e image    -f steam_metab_raw -o data -j -q example_data/meas_MID310_STEAM_metab_FID115673.dat\n",
-    "spec2nii twix -e phasecor -f steam_ecc_raw   -o data -j -q example_data/meas_MID310_STEAM_metab_FID115673.dat\n",
+    "spec2nii twix -e image    -f steam_metab_raw -o data -q example_data/meas_MID310_STEAM_metab_FID115673.dat\n",
+    "spec2nii twix -e phasecor -f steam_ecc_raw   -o data -q example_data/meas_MID310_STEAM_metab_FID115673.dat\n",
     "\n",
-    "spec2nii twix -e image    -f steam_wref_comb_raw  -o data -j -q example_data/meas_MID311_STEAM_wref1_FID115674.dat\n",
-    "spec2nii twix -e image    -f steam_wref_quant_raw -o data -j -q example_data/meas_MID312_STEAM_wref3_FID115675.dat"
+    "spec2nii twix -e image    -f steam_wref_comb_raw  -o data -q example_data/meas_MID311_STEAM_wref1_FID115674.dat\n",
+    "spec2nii twix -e image    -f steam_wref_quant_raw -o data -q example_data/meas_MID312_STEAM_wref3_FID115675.dat"
    ]
   },
   {
@@ -96,11 +94,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "After conversion we have 64 averages from the suppressed water scan. The \"Set\" descriptor at the end of the file name arises from the indexing loop used at acquisition time to store the data. This is likely to vary from sequence to sequence, and is the choice of the pulse sequence programmer. Multiple looping indicies might be used for more complex data acquisition strategies.  \n",
+    "After conversion we have four files. steam_metab_raw is the the suppressed water data. \n",
     "  \n",
-    "We also have one ecc scan, and two \"Reps\" (another looping index) of each of the water reference scans. In this case both Set and Rep are used to indicate transients that should be averaged toghether.\n",
-    "\n",
-    "Notice that each NIfTI file is paired witht a JSON sidecar file which contains useful information for further processing."
+    "We also have one ecc scan, and two water reference scans. The former contains a single dynamic, whilst the latter two contain two dynamics. All have data from 32 uncombined coils."
    ]
   },
   {
@@ -117,7 +113,7 @@
     "1. Take averages of water references used for combination across files\n",
     "2. Coil combine the metab data, the ecc data and the quantification data using the \"comb\" data as the reference.\n",
     "3. Phase and frequency align the data where there are multiple transients.\n",
-    "4. Combine data across those transients (one in each file) by taking the mean.\n",
+    "4. Combine data across those transients by taking the mean.\n",
     "5. Run eddy current correction using the appropriate reference.\n",
     "6. In this data an additional FID point is collected before the echo centre. Remove this.\n",
     "7. Run HLSVD on the data to remove the residual water in the water suppressed data.\n",
@@ -136,21 +132,21 @@
     "if not os.path.isdir('fsl_mrs_proc'):\n",
     "    os.mkdir('fsl_mrs_proc')\n",
     "\n",
-    "#1. Combine comb water data across files\n",
-    "%sx fsl_mrs_proc average --file data/steam_wref_comb_raw*.nii.gz --avgfiles --output fsl_mrs_proc --filename steam_wref_comb\n",
+    "#1. Combine water reference data for combination across dynamics\n",
+    "%sx fsl_mrs_proc average --file data/steam_wref_comb_raw.nii.gz --dim DIM_DYN --output fsl_mrs_proc --filename steam_wref_comb\n",
     "\n",
     "#2. Run coil combination on the three sets of data\n",
-    "%sx fsl_mrs_proc coilcombine --file data/steam_metab_raw*.nii.gz --reference fsl_mrs_proc/steam_wref_comb.nii.gz --output fsl_mrs_proc -r \n",
-    "%sx fsl_mrs_proc coilcombine --file data/steam_wref_quant_raw*.nii.gz --reference fsl_mrs_proc/steam_wref_comb.nii.gz --output fsl_mrs_proc \n",
+    "%sx fsl_mrs_proc coilcombine --file data/steam_metab_raw.nii.gz --reference fsl_mrs_proc/steam_wref_comb.nii.gz --output fsl_mrs_proc -r \n",
+    "%sx fsl_mrs_proc coilcombine --file data/steam_wref_quant_raw.nii.gz --reference fsl_mrs_proc/steam_wref_comb.nii.gz --output fsl_mrs_proc \n",
     "%sx fsl_mrs_proc coilcombine --file data/steam_ecc_raw.nii.gz --reference fsl_mrs_proc/steam_wref_comb.nii.gz --output fsl_mrs_proc \n",
     "\n",
     "#3. Align averages of water ref and metab data\n",
-    "%sx fsl_mrs_proc align --file fsl_mrs_proc/steam_metab_raw*.nii.gz --ppm 1.8 3.5 --output fsl_mrs_proc -r --filename steam_metab_align --apod 50 \n",
-    "%sx fsl_mrs_proc align --file fsl_mrs_proc/steam_wref_quant_raw*.nii.gz --ppm 4 6 --output fsl_mrs_proc --filename steam_water_align \n",
+    "%sx fsl_mrs_proc align --file fsl_mrs_proc/steam_metab_raw.nii.gz --ppm 1.8 3.5 --output fsl_mrs_proc -r --filename steam_metab_align --apod 50 \n",
+    "%sx fsl_mrs_proc align --file fsl_mrs_proc/steam_wref_quant_raw.nii.gz --ppm 4 6 --output fsl_mrs_proc --filename steam_water_align \n",
     "\n",
     "#3. Combine data across averages\n",
-    "%sx fsl_mrs_proc average --file fsl_mrs_proc/steam_metab_align*.nii.gz --avgfiles  --output fsl_mrs_proc -r --filename steam_metab_comb \n",
-    "%sx fsl_mrs_proc average --file fsl_mrs_proc/steam_water_align*.nii.gz --avgfiles --output fsl_mrs_proc --filename steam_wquant_comb \n",
+    "%sx fsl_mrs_proc average --file fsl_mrs_proc/steam_metab_align.nii.gz --dim DIM_DYN  --output fsl_mrs_proc -r --filename steam_metab_comb \n",
+    "%sx fsl_mrs_proc average --file fsl_mrs_proc/steam_water_align.nii.gz --dim DIM_DYN --output fsl_mrs_proc --filename steam_wquant_comb \n",
     "\n",
     "#5. Run the eddy current correction on the data\n",
     "%sx fsl_mrs_proc ecc --file fsl_mrs_proc/steam_metab_comb.nii.gz --reference fsl_mrs_proc/steam_ecc_raw.nii.gz  --output fsl_mrs_proc -r --filename steam_metab_comb_ecc \n",
@@ -195,15 +191,13 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "scrolled": false
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "%sx fsl_mrs_preproc --output fsl_mrs_preproc --data data/steam_metab_raw*.nii.gz --reference data/steam_wref_comb_raw*.nii.gz --quant data/steam_wref_quant_raw*.nii.gz --ecc data/steam_ecc_raw.nii.gz --hlsvd --leftshift 1 --t1 example_data/T1_biascorr.nii.gz --overwrite --report \n",
+    "%sx fsl_mrs_preproc --output fsl_mrs_preproc --data data/steam_metab_raw.nii.gz --reference data/steam_wref_comb_raw.nii.gz --quant data/steam_wref_quant_raw.nii.gz --ecc data/steam_ecc_raw.nii.gz --hlsvd --leftshift 1 --t1 example_data/T1.anat/T1_biascorr.nii.gz --overwrite --report \n",
     "\n",
     "# You can also run this script without independent quant and ecc data (just 1 water reference)\n",
-    "%sx fsl_mrs_preproc --output fsl_mrs_preproc_simple --data data/steam_metab_raw*.nii.gz --reference data/steam_wref_comb_raw*.nii.gz --hlsvd --leftshift 1 --t1 example_data/T1_biascorr.nii.gz --overwrite --report "
+    "%sx fsl_mrs_preproc --output fsl_mrs_preproc_simple --data data/steam_metab_raw.nii.gz --reference data/steam_wref_comb_raw.nii.gz --hlsvd --leftshift 1 --t1 example_data/T1.anat/T1_biascorr.nii.gz --overwrite --report "
    ]