Initial commit

LICENSE

+Authors: Fidel Alfaro-Almagro, Stephen M. Smith, Tom Nichols, & Paul McCarthy
+
+Copyright 2020 University of Oxford
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

README.md

+# Generation, processing, and analysis of brain imaging confounds for UK Biobank
+
+This repository has the whole toolset used for the work for the Generation, processing, and analysis of brain imaging confounds for UK Biobank. 
+
+The contents of the directories are:
+
+* `data`: Text files with confound names and IDP groupings. This directory will be fully populated once the scripts in `generate_initial_data` are run.
+* `bash_scripts`: Command line bash scripts that can be useful for general purpose.
+* `generate_initial_data`: Scripts to generate the confounds, IDPs, and non-IDPs data files that will be used in the main processing stages. Instructions on how to run them can be found in the [`README.md`](https://git.fmrib.ox.ac.uk/falmagro/ukb_unconfound/blob/master/generate_initial_data/README.md) of that directory.
+* `confound_processing`: Main set of scripts (mostly in matlab) that perform the processing and analysis of the confounds as well as generate the main set of figures.
+    * `functions`: Matlab functions that will be called in an SGE cluster.
+    * `scripts`: Matlab / python / bash scripts that will perform the analysis. They are numbered by the order of execution.
+    * `common_matlab`: Utilities to be used by matlab code. This includes libraries such as ([FSLNETS](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FSLNets#Installing_FSLNets)).
+
+An interactive version of the Figures can be found [in this link](https://users.fmrib.ox.ac.uk/~falmagro/confounds/). 
+
+These html files are also generated with the scripts in `confound_processing`.
+
+# Instructions
+* Download the [FBP (Biobank Pipeline)](https://git.fmrib.ox.ac.uk/falmagro/UK_biobank_pipeline_v_1)
+* Activate [bb_python](https://git.fmrib.ox.ac.uk/falmagro/UK_biobank_pipeline_v_1/blob/master/bb_python/python_installation/install_bb_python.sh) virtual env. 
+* Adapt the script [init_vars](https://git.fmrib.ox.ac.uk/falmagro/UK_biobank_pipeline_v_1/blob/master/init_vars) to your system configuration and source it.
+* Add `sh_utils` directory to `$PATH`
+* Change `$subjDir` with the absolute path to the directory of all UKB subjects (a file `subj.txt` is needed as an index in that directory).
+* Each subject in `$subjDir` needs to be in UKB directory format. A DICOM directory is needed for each subject with the `*.txt` and `*.dcm` generated by FBP
+* Change $MATLAB_BIN to the appropriate Matlab binary
+
+# Dependencies
+* [FBP (UK Biobank Pipeline)](https://git.fmrib.ox.ac.uk/falmagro/UK_biobank_pipeline_v_1). 
+* [bb_python](https://git.fmrib.ox.ac.uk/falmagro/UK_biobank_pipeline_v_1/blob/master/bb_python/python_installation/install_bb_python.sh) 
+* Some functions in `confound_processing` run in parallel using SGE (SUN Grid Engine). 
+* Matlab code was run with version 2017a.
+* R code was run with version 3.6.1
+* Python scripts will need the following libraries not included in `bb_python`:
+    * `plotly==3.10.0`
+    * `fmrib-unpack==1.4.1`
+

bash_scripts/bb_wait_SGE_jobs

+#!/bin/bash
+#
+# Authors: Fidel Alfaro-Almagro
+#
+# Copyright 2017 University of Oxford
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+for jobID in $(echo $1 | sed "s/,/ /g") ; do
+    while [ ! "$(qstat | awk '{print $1}' | grep "^$jobID" | wc -l)" == "0" ] ; do
+        sleep 30 ; 
+    done
+done

bash_scripts/calc_perc

+#!/bin/env python
+#
+# Authors: Fidel Alfaro-Almagro
+#
+# Copyright 2017 University of Oxford
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import numpy as np
+import sys,argparse,os.path
+
+class MyParser(argparse.ArgumentParser):
+    def error(self, message):
+        sys.stderr.write('error: %s\n' % message)
+        self.print_help()
+        sys.exit(2)
+
+class Usage(Exception):
+    def __init__(self, msg):
+        self.msg = msg
+
+def main(): 
+    
+    parser = MyParser(description='Percentile Calculator')
+    parser.add_argument("dataFile", nargs=1, help='File Name with the data')
+    parser.add_argument("percentile", nargs="?", default=-1, help='Percentile to calculate out of the data (If no value given, calc 2nd, 25th, 5th, 75th and 98th percentile)')
+
+    argsa = parser.parse_args()
+
+    dataFile = argsa.dataFile[0]
+    dataFile = dataFile.strip()
+
+    if dataFile == "":
+        print('No file specified')
+        exit(1)
+
+    if not os.path.isfile(dataFile):
+        print('Specified file does not exist')
+        exit(1)
+
+    data = np.loadtxt(dataFile)
+
+    percentile = argsa.percentile
+
+    if percentile == -1:
+        p2 =np.percentile(data,2 )
+        p25=np.percentile(data,25)
+        p50=np.percentile(data,50)
+        p75=np.percentile(data,75)
+        p98=np.percentile(data,98)
+
+        print("P2: " + str(p2) + "   P25: " + str(p25) + "   P50: " + str(p50) + "   P75: " + str(p75) + "   P98: " + str(p98))
+       
+    else:
+        perc=np.percentile(data,percentile)
+        print(str(np.percentile(data,percentile)))
+
+if __name__ == "__main__":
+    main()
+

bash_scripts/column_to_row

+#!/bin/bash
+
+###################################################
+#                                                 #
+# Script to turn a column from a file into a row  #
+# Arguments:                                      #
+#    $1) File with the column to turn into row    #
+#    $2) Number of column. Default: 1 (Optional)  #
+#                                                 #
+# By: Fidel Alfaro Almagro (2015)                 #
+#                                                 #
+###################################################
+
+fil="$1"
+
+if [ "$2" == "" ] ; then
+    col_num="1"
+else
+    col_num="$2"
+fi
+
+result=""
+
+for elem in `cat $fil | awk -v val=$col_num '{print $val}' `; do
+	result="$result $elem"
+done
+
+echo $result
+

bash_scripts/generate_NaNs

+#!/bin/bash
+
+#################################################
+#                                               #
+# Script to generate a string of NaNs.          #
+# Arguments:                                    #
+#    $1) Number of NaNs to generate             #
+#    $2) Initial value of the string (Optional) #
+#                                               #
+# By Fidel Alfaro Almagro (2017)                #
+#                                               #
+#################################################
+
+num_NaNs="$1"
+
+if [ "$2" == "" ] ; then
+    result=""
+else
+    result="$2"
+fi
+
+for (( i=0 ; i<$num_NaNs ; i ++ )) ; do
+    result="$result NaN"
+done
+
+echo $result

bash_scripts/run_mcr

+#!/bin/bash
+
+# Wrapper for MATLAB Compiled applications that sets up the MCR_CACHE_ROOT
+
+if [ $# -eq 0 ]; then
+    echo "Must be called with an appropriate MCR task command line"
+    exit 100
+fi
+
+MCR_CR="mcr_${JOB_ID}"
+if [ "${SGE_TASK_ID}" != "undefined" ]; then
+    MCR_CR="${MCR_CR}-${SGE_TASK_ID}"
+fi
+MCR_CACHE_ROOT="/tmp/${MCR_CR}"
+export MCR_CACHE_ROOT
+
+"$@"
+status=$?
+rm -rf "/tmp/${MCR_CR}"
+exit $status

bash_scripts/shape

+#!/bin/bash
+#
+# Authors: Fidel Alfaro-Almagro, Stephen M. Smith, Tom Nichols, & Paul McCarthy
+#
+# Copyright 2017 University of Oxford
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+cat $1 | awk '{print NF}' | sort | uniq -c

conf_processing/common_matlab/FDR.m

+function [pID,pN] = FDR(p,q)
+% FORMAT [pID,pN] = FDR(p,q)
+% 
+% p   - vector of p-values
+% q   - False Discovery Rate level
+%
+% pID - p-value threshold based on independence or positive dependence
+% pN  - Nonparametric p-value threshold
+%______________________________________________________________________________
+% $Id: FDR.m,v 2.1 2010/08/05 14:34:19 nichols Exp $
+
+
+p = p(isfinite(p));  % Toss NaN's
+p = sort(p(:));
+V = length(p);
+I = (1:V)';
+
+cVID = 1;
+cVN = sum(1./(1:V));
+
+pID = p(max(find(p<=I/V*q/cVID)));
+if isempty(pID), pID=0; end
+pN = p(max(find(p<=I/V*q/cVN)));
+if isempty(pN), pN=0; end
+

conf_processing/common_matlab/FSLNets/README

+
+%%%  FSLNets v0.6.3
+%%%  FMRIB Analysis Group
+%%%  Copyright (C) 2012-2016 University of Oxford
+%%%  Licence is same as FSL licence
+%%%  See documentation at  http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FSLNets
+
+%%%%%%%%%% list of backwards incompatibilities from v0.4 to v0.5
+nets_pics renamed to nets_nodepics
+nets_consistency renamed to nets_groupmean
+nets_makemats and nets_r2z/nets_r2z_icov have all been merged to form a new function nets_netmats
+

conf_processing/common_matlab/FSLNets/call_fsl.m

+function [status,output] = call_fsl(cmd)
+% [status, output] = call_fsl(cmd)
+% 
+% Wrapper around calls to FSL binaries
+% clears LD_LIBRARY_PATH and ensures
+% the FSL envrionment variables have been
+% set up
+% Debian/Ubuntu users should uncomment as
+% indicated
+
+fsldir=getenv('FSLDIR');
+
+% Debian/Ubuntu - uncomment the following
+%fsllibdir=sprintf('%s/%s', fsldir, 'bin');
+
+if ismac
+  dylibpath=getenv('DYLD_LIBRARY_PATH');
+  setenv('DYLD_LIBRARY_PATH');
+else
+  ldlibpath=getenv('LD_LIBRARY_PATH');
+  setenv('LD_LIBRARY_PATH');
+  % Debian/Ubuntu - uncomment the following
+  % setenv('LD_LIBRARY_PATH',fsllibdir);
+end
+
+command = sprintf('/bin/sh -c ''. %s/etc/fslconf/fsl.sh; %s''', fsldir, cmd);
+[status,output] = system(command);
+
+if ismac
+  setenv('DYLD_LIBRARY_PATH', dylibpath);
+else
+    setenv('LD_LIBRARY_PATH', ldlibpath);
+end
+
+if status
+    error('FSL call (%s) failed, %s', command, output)
+end
\ No newline at end of file

conf_processing/common_matlab/FSLNets/netjs/index.html

+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<HTML><HEAD><meta http-equiv="Content-Type" content="text/html;charset=utf-8">
+<TITLE>FSLNets interactive netmat visualisation - Paul McCarthy, FMRIB</TITLE></HEAD><BODY>
+<STYLE TYPE="text/css">BODY { background: white ; font-family: 'Arial'}</STYLE>
+
+<a href="../index.html">... back to main index</a> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<b>FSLNets interactive netmat visualisation</b> <p>
+
+  <div id="networkCtrl" style="display: inline-block; vertical-align: top"></div>
+  <div id="fullNetwork" style="display: inline"></div>
+  <div id="subNetwork"  style="display: inline"></div>
+
+  <script data-main="js/main" src="js/lib/require.js"></script>
+</body>
+</html>

conf_processing/common_matlab/FSLNets/netjs/js/lib/LICENSE.d3.js

+Copyright (c) 2010-2014, Michael Bostock
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* The name Michael Bostock may not be used to endorse or promote products
+  derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL MICHAEL BOSTOCK BE LIABLE FOR ANY DIRECT,
+INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
+EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

conf_processing/common_matlab/FSLNets/netjs/js/lib/mustache.js

+/*!
+ * mustache.js - Logic-less {{mustache}} templates with JavaScript
+ * http://github.com/janl/mustache.js
+ */
+
+/*global define: false*/
+
+(function (root, factory) {
+  if (typeof exports === "object" && exports) {
+    factory(exports); // CommonJS
+  } else {
+    var mustache = {};
+    factory(mustache);
+    if (typeof define === "function" && define.amd) {
+      define(mustache); // AMD
+    } else {
+      root.Mustache = mustache; // <script>
+    }
+  }
+}(this, function (mustache) {
+
+  var whiteRe = /\s*/;
+  var spaceRe = /\s+/;
+  var nonSpaceRe = /\S/;
+  var eqRe = /\s*=/;
+  var curlyRe = /\s*\}/;
+  var tagRe = /#|\^|\/|>|\{|&|=|!/;
+
+  // Workaround for https://issues.apache.org/jira/browse/COUCHDB-577
+  // See https://github.com/janl/mustache.js/issues/189
+  var RegExp_test = RegExp.prototype.test;
+  function testRegExp(re, string) {
+    return RegExp_test.call(re, string);
+  }
+
+  function isWhitespace(string) {
+    return !testRegExp(nonSpaceRe, string);
+  }
+
+  var Object_toString = Object.prototype.toString;
+  var isArray = Array.isArray || function (object) {
+    return Object_toString.call(object) === '[object Array]';
+  };
+
+  function isFunction(object) {
+    return typeof object === 'function';
+  }
+
+  function escapeRegExp(string) {
+    return string.replace(/[\-\[\]{}()*+?.,\\\^$|#\s]/g, "\\$&");
+  }
+
+  var entityMap = {
+    "&": "&amp;",
+    "<": "&lt;",
+    ">": "&gt;",
+    '"': '&quot;',
+    "'": '&#39;',
+    "/": '&#x2F;'
+  };
+
+  function escapeHtml(string) {
+    return String(string).replace(/[&<>"'\/]/g, function (s) {
+      return entityMap[s];
+    });
+  }
+
+  function escapeTags(tags) {
+    if (!isArray(tags) || tags.length !== 2) {
+      throw new Error('Invalid tags: ' + tags);
+    }
+
+    return [
+      new RegExp(escapeRegExp(tags[0]) + "\\s*"),
+      new RegExp("\\s*" + escapeRegExp(tags[1]))
+    ];
+  }
+
+  /**
+   * Breaks up the given `template` string into a tree of tokens. If the `tags`
+   * argument is given here it must be an array with two string values: the
+   * opening and closing tags used in the template (e.g. [ "<%", "%>" ]). Of
+   * course, the default is to use mustaches (i.e. mustache.tags).
+   *
+   * A token is an array with at least 4 elements. The first element is the
+   * mustache symbol that was used inside the tag, e.g. "#" or "&". If the tag
+   * did not contain a symbol (i.e. {{myValue}}) this element is "name". For
+   * all template text that appears outside a symbol this element is "text".
+   *
+   * The second element of a token is its "value". For mustache tags this is
+   * whatever else was inside the tag besides the opening symbol. For text tokens
+   * this is the text itself.
+   *
+   * The third and fourth elements of the token are the start and end indices
+   * in the original template of the token, respectively.
+   *
+   * Tokens that are the root node of a subtree contain two more elements: an
+   * array of tokens in the subtree and the index in the original template at which
+   * the closing tag for that section begins.
+   */
+  function parseTemplate(template, tags) {
+    tags = tags || mustache.tags;
+    template = template || '';
+
+    if (typeof tags === 'string') {
+      tags = tags.split(spaceRe);
+    }
+
+    var tagRes = escapeTags(tags);
+    var scanner = new Scanner(template);
+
+    var sections = [];     // Stack to hold section tokens
+    var tokens = [];       // Buffer to hold the tokens
+    var spaces = [];       // Indices of whitespace tokens on the current line
+    var hasTag = false;    // Is there a {{tag}} on the current line?
+    var nonSpace = false;  // Is there a non-space char on the current line?
+
+    // Strips all whitespace tokens array for the current line
+    // if there was a {{#tag}} on it and otherwise only space.
+    function stripSpace() {
+      if (hasTag && !nonSpace) {
+        while (spaces.length) {
+          delete tokens[spaces.pop()];
+        }
+      } else {
+        spaces = [];
+      }
+
+      hasTag = false;
+      nonSpace = false;
+    }
+
+    var start, type, value, chr, token, openSection;
+    while (!scanner.eos()) {
+      start = scanner.pos;
+
+      // Match any text between tags.
+      value = scanner.scanUntil(tagRes[0]);
+      if (value) {
+        for (var i = 0, len = value.length; i < len; ++i) {
+          chr = value.charAt(i);
+
+          if (isWhitespace(chr)) {
+            spaces.push(tokens.length);
+          } else {
+            nonSpace = true;
+          }
+
+          tokens.push(['text', chr, start, start + 1]);
+          start += 1;
+
+          // Check for whitespace on the current line.
+          if (chr === '\n') {
+            stripSpace();
+          }
+        }
+      }
+
+      // Match the opening tag.
+      if (!scanner.scan(tagRes[0])) break;
+      hasTag = true;
+
+      // Get the tag type.
+      type = scanner.scan(tagRe) || 'name';
+      scanner.scan(whiteRe);
+
+      // Get the tag value.
+      if (type === '=') {
+        value = scanner.scanUntil(eqRe);
+        scanner.scan(eqRe);
+        scanner.scanUntil(tagRes[1]);
+      } else if (type === '{') {
+        value = scanner.scanUntil(new RegExp('\\s*' + escapeRegExp('}' + tags[1])));
+        scanner.scan(curlyRe);
+        scanner.scanUntil(tagRes[1]);
+        type = '&';
+      } else {
+        value = scanner.scanUntil(tagRes[1]);
+      }
+
+      // Match the closing tag.
+      if (!scanner.scan(tagRes[1])) {
+        throw new Error('Unclosed tag at ' + scanner.pos);
+      }
+
+      token = [ type, value, start, scanner.pos ];
+      tokens.push(token);
+
+      if (type === '#' || type === '^') {
+        sections.push(token);
+      } else if (type === '/') {
+        // Check section nesting.
+        openSection = sections.pop();
+
+        if (!openSection) {
+          throw new Error('Unopened section "' + value + '" at ' + start);
+        }
+        if (openSection[1] !== value) {
+          throw new Error('Unclosed section "' + openSection[1] + '" at ' + start);
+        }
+      } else if (type === 'name' || type === '{' || type === '&') {
+        nonSpace = true;
+      } else if (type === '=') {
+        // Set the tags for the next time around.
+        tagRes = escapeTags(tags = value.split(spaceRe));
+      }
+    }
+
+    // Make sure there are no open sections when we're done.
+    openSection = sections.pop();
+    if (openSection) {
+      throw new Error('Unclosed section "' + openSection[1] + '" at ' + scanner.pos);
+    }
+
+    return nestTokens(squashTokens(tokens));
+  }
+
+  /**