Commit d4ca21a6 authored by Fidel Alfaro Almagro's avatar Fidel Alfaro Almagro
Browse files

Initial commit

parents
Authors: Fidel Alfaro-Almagro, Stephen M. Smith, Tom Nichols, & Paul McCarthy
Copyright 2020 University of Oxford
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
# Generation, processing, and analysis of brain imaging confounds for UK Biobank
This repository has the whole toolset used for the work for the Generation, processing, and analysis of brain imaging confounds for UK Biobank.
The contents of the directories are:
* `data`: Text files with confound names and IDP groupings. This directory will be fully populated once the scripts in `generate_initial_data` are run.
* `bash_scripts`: Command line bash scripts that can be useful for general purpose.
* `generate_initial_data`: Scripts to generate the confounds, IDPs, and non-IDPs data files that will be used in the main processing stages. Instructions on how to run them can be found in the [`README.md`](https://git.fmrib.ox.ac.uk/falmagro/ukb_unconfound/blob/master/generate_initial_data/README.md) of that directory.
* `confound_processing`: Main set of scripts (mostly in matlab) that perform the processing and analysis of the confounds as well as generate the main set of figures.
* `functions`: Matlab functions that will be called in an SGE cluster.
* `scripts`: Matlab / python / bash scripts that will perform the analysis. They are numbered by the order of execution.
* `common_matlab`: Utilities to be used by matlab code. This includes libraries such as ([FSLNETS](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FSLNets#Installing_FSLNets)).
An interactive version of the Figures can be found [in this link](https://users.fmrib.ox.ac.uk/~falmagro/confounds/).
These html files are also generated with the scripts in `confound_processing`.
# Instructions
* Download the [FBP (Biobank Pipeline)](https://git.fmrib.ox.ac.uk/falmagro/UK_biobank_pipeline_v_1)
* Activate [bb_python](https://git.fmrib.ox.ac.uk/falmagro/UK_biobank_pipeline_v_1/blob/master/bb_python/python_installation/install_bb_python.sh) virtual env.
* Adapt the script [init_vars](https://git.fmrib.ox.ac.uk/falmagro/UK_biobank_pipeline_v_1/blob/master/init_vars) to your system configuration and source it.
* Add `sh_utils` directory to `$PATH`
* Change `$subjDir` with the absolute path to the directory of all UKB subjects (a file `subj.txt` is needed as an index in that directory).
* Each subject in `$subjDir` needs to be in UKB directory format. A DICOM directory is needed for each subject with the `*.txt` and `*.dcm` generated by FBP
* Change $MATLAB_BIN to the appropriate Matlab binary
# Dependencies
* [FBP (UK Biobank Pipeline)](https://git.fmrib.ox.ac.uk/falmagro/UK_biobank_pipeline_v_1).
* [bb_python](https://git.fmrib.ox.ac.uk/falmagro/UK_biobank_pipeline_v_1/blob/master/bb_python/python_installation/install_bb_python.sh)
* Some functions in `confound_processing` run in parallel using SGE (SUN Grid Engine).
* Matlab code was run with version 2017a.
* R code was run with version 3.6.1
* Python scripts will need the following libraries not included in `bb_python`:
* `plotly==3.10.0`
* `fmrib-unpack==1.4.1`
#!/bin/bash
#
# Authors: Fidel Alfaro-Almagro
#
# Copyright 2017 University of Oxford
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
for jobID in $(echo $1 | sed "s/,/ /g") ; do
while [ ! "$(qstat | awk '{print $1}' | grep "^$jobID" | wc -l)" == "0" ] ; do
sleep 30 ;
done
done
#!/bin/env python
#
# Authors: Fidel Alfaro-Almagro
#
# Copyright 2017 University of Oxford
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import numpy as np
import sys,argparse,os.path
class MyParser(argparse.ArgumentParser):
def error(self, message):
sys.stderr.write('error: %s\n' % message)
self.print_help()
sys.exit(2)
class Usage(Exception):
def __init__(self, msg):
self.msg = msg
def main():
parser = MyParser(description='Percentile Calculator')
parser.add_argument("dataFile", nargs=1, help='File Name with the data')
parser.add_argument("percentile", nargs="?", default=-1, help='Percentile to calculate out of the data (If no value given, calc 2nd, 25th, 5th, 75th and 98th percentile)')
argsa = parser.parse_args()
dataFile = argsa.dataFile[0]
dataFile = dataFile.strip()
if dataFile == "":
print('No file specified')
exit(1)
if not os.path.isfile(dataFile):
print('Specified file does not exist')
exit(1)
data = np.loadtxt(dataFile)
percentile = argsa.percentile
if percentile == -1:
p2 =np.percentile(data,2 )
p25=np.percentile(data,25)
p50=np.percentile(data,50)
p75=np.percentile(data,75)
p98=np.percentile(data,98)
print("P2: " + str(p2) + " P25: " + str(p25) + " P50: " + str(p50) + " P75: " + str(p75) + " P98: " + str(p98))
else:
perc=np.percentile(data,percentile)
print(str(np.percentile(data,percentile)))
if __name__ == "__main__":
main()
#!/bin/bash
###################################################
# #
# Script to turn a column from a file into a row #
# Arguments: #
# $1) File with the column to turn into row #
# $2) Number of column. Default: 1 (Optional) #
# #
# By: Fidel Alfaro Almagro (2015) #
# #
###################################################
fil="$1"
if [ "$2" == "" ] ; then
col_num="1"
else
col_num="$2"
fi
result=""
for elem in `cat $fil | awk -v val=$col_num '{print $val}' `; do
result="$result $elem"
done
echo $result
#!/bin/bash
#################################################
# #
# Script to generate a string of NaNs. #
# Arguments: #
# $1) Number of NaNs to generate #
# $2) Initial value of the string (Optional) #
# #
# By Fidel Alfaro Almagro (2017) #
# #
#################################################
num_NaNs="$1"
if [ "$2" == "" ] ; then
result=""
else
result="$2"
fi
for (( i=0 ; i<$num_NaNs ; i ++ )) ; do
result="$result NaN"
done
echo $result
#!/bin/bash
# Wrapper for MATLAB Compiled applications that sets up the MCR_CACHE_ROOT
if [ $# -eq 0 ]; then
echo "Must be called with an appropriate MCR task command line"
exit 100
fi
MCR_CR="mcr_${JOB_ID}"
if [ "${SGE_TASK_ID}" != "undefined" ]; then
MCR_CR="${MCR_CR}-${SGE_TASK_ID}"
fi
MCR_CACHE_ROOT="/tmp/${MCR_CR}"
export MCR_CACHE_ROOT
"$@"
status=$?
rm -rf "/tmp/${MCR_CR}"
exit $status
#!/bin/bash
#
# Authors: Fidel Alfaro-Almagro, Stephen M. Smith, Tom Nichols, & Paul McCarthy
#
# Copyright 2017 University of Oxford
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
cat $1 | awk '{print NF}' | sort | uniq -c
function [pID,pN] = FDR(p,q)
% FORMAT [pID,pN] = FDR(p,q)
%
% p - vector of p-values
% q - False Discovery Rate level
%
% pID - p-value threshold based on independence or positive dependence
% pN - Nonparametric p-value threshold
%______________________________________________________________________________
% $Id: FDR.m,v 2.1 2010/08/05 14:34:19 nichols Exp $
p = p(isfinite(p)); % Toss NaN's
p = sort(p(:));
V = length(p);
I = (1:V)';
cVID = 1;
cVN = sum(1./(1:V));
pID = p(max(find(p<=I/V*q/cVID)));
if isempty(pID), pID=0; end
pN = p(max(find(p<=I/V*q/cVN)));
if isempty(pN), pN=0; end
%%% FSLNets v0.6.3
%%% FMRIB Analysis Group
%%% Copyright (C) 2012-2016 University of Oxford
%%% Licence is same as FSL licence
%%% See documentation at http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FSLNets
%%%%%%%%%% list of backwards incompatibilities from v0.4 to v0.5
nets_pics renamed to nets_nodepics
nets_consistency renamed to nets_groupmean
nets_makemats and nets_r2z/nets_r2z_icov have all been merged to form a new function nets_netmats
function [status,output] = call_fsl(cmd)
% [status, output] = call_fsl(cmd)
%
% Wrapper around calls to FSL binaries
% clears LD_LIBRARY_PATH and ensures
% the FSL envrionment variables have been
% set up
% Debian/Ubuntu users should uncomment as
% indicated
fsldir=getenv('FSLDIR');
% Debian/Ubuntu - uncomment the following
%fsllibdir=sprintf('%s/%s', fsldir, 'bin');
if ismac
dylibpath=getenv('DYLD_LIBRARY_PATH');
setenv('DYLD_LIBRARY_PATH');
else
ldlibpath=getenv('LD_LIBRARY_PATH');
setenv('LD_LIBRARY_PATH');
% Debian/Ubuntu - uncomment the following
% setenv('LD_LIBRARY_PATH',fsllibdir);
end
command = sprintf('/bin/sh -c ''. %s/etc/fslconf/fsl.sh; %s''', fsldir, cmd);
[status,output] = system(command);
if ismac
setenv('DYLD_LIBRARY_PATH', dylibpath);
else
setenv('LD_LIBRARY_PATH', ldlibpath);
end
if status
error('FSL call (%s) failed, %s', command, output)
end
\ No newline at end of file
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<TITLE>FSLNets interactive netmat visualisation - Paul McCarthy, FMRIB</TITLE></HEAD><BODY>
<STYLE TYPE="text/css">BODY { background: white ; font-family: 'Arial'}</STYLE>
<a href="../index.html">... back to main index</a> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<b>FSLNets interactive netmat visualisation</b> <p>
<div id="networkCtrl" style="display: inline-block; vertical-align: top"></div>
<div id="fullNetwork" style="display: inline"></div>
<div id="subNetwork" style="display: inline"></div>
<script data-main="js/main" src="js/lib/require.js"></script>
</body>
</html>
Copyright (c) 2010-2014, Michael Bostock
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* The name Michael Bostock may not be used to endorse or promote products
derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL MICHAEL BOSTOCK BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
This diff is collapsed.
/*!
* mustache.js - Logic-less {{mustache}} templates with JavaScript
* http://github.com/janl/mustache.js
*/
/*global define: false*/
(function (root, factory) {
if (typeof exports === "object" && exports) {
factory(exports); // CommonJS
} else {
var mustache = {};
factory(mustache);
if (typeof define === "function" && define.amd) {
define(mustache); // AMD
} else {
root.Mustache = mustache; // <script>
}
}
}(this, function (mustache) {
var whiteRe = /\s*/;
var spaceRe = /\s+/;
var nonSpaceRe = /\S/;
var eqRe = /\s*=/;
var curlyRe = /\s*\}/;
var tagRe = /#|\^|\/|>|\{|&|=|!/;
// Workaround for https://issues.apache.org/jira/browse/COUCHDB-577
// See https://github.com/janl/mustache.js/issues/189
var RegExp_test = RegExp.prototype.test;
function testRegExp(re, string) {
return RegExp_test.call(re, string);
}
function isWhitespace(string) {
return !testRegExp(nonSpaceRe, string);
}
var Object_toString = Object.prototype.toString;
var isArray = Array.isArray || function (object) {
return Object_toString.call(object) === '[object Array]';
};
function isFunction(object) {
return typeof object === 'function';
}
function escapeRegExp(string) {
return string.replace(/[\-\[\]{}()*+?.,\\\^$|#\s]/g, "\\$&");
}
var entityMap = {
"&": "&amp;",
"<": "&lt;",
">": "&gt;",
'"': '&quot;',
"'": '&#39;',
"/": '&#x2F;'
};
function escapeHtml(string) {
return String(string).replace(/[&<>"'\/]/g, function (s) {
return entityMap[s];
});
}
function escapeTags(tags) {
if (!isArray(tags) || tags.length !== 2) {
throw new Error('Invalid tags: ' + tags);
}
return [
new RegExp(escapeRegExp(tags[0]) + "\\s*"),
new RegExp("\\s*" + escapeRegExp(tags[1]))
];
}
/**
* Breaks up the given `template` string into a tree of tokens. If the `tags`
* argument is given here it must be an array with two string values: the
* opening and closing tags used in the template (e.g. [ "<%", "%>" ]). Of
* course, the default is to use mustaches (i.e. mustache.tags).
*
* A token is an array with at least 4 elements. The first element is the
* mustache symbol that was used inside the tag, e.g. "#" or "&". If the tag
* did not contain a symbol (i.e. {{myValue}}) this element is "name". For
* all template text that appears outside a symbol this element is "text".
*
* The second element of a token is its "value". For mustache tags this is
* whatever else was inside the tag besides the opening symbol. For text tokens
* this is the text itself.
*
* The third and fourth elements of the token are the start and end indices
* in the original template of the token, respectively.
*
* Tokens that are the root node of a subtree contain two more elements: an
* array of tokens in the subtree and the index in the original template at which
* the closing tag for that section begins.
*/
function parseTemplate(template, tags) {
tags = tags || mustache.tags;
template = template || '';
if (typeof tags === 'string') {
tags = tags.split(spaceRe);
}
var tagRes = escapeTags(tags);
var scanner = new Scanner(template);
var sections = []; // Stack to hold section tokens
var tokens = []; // Buffer to hold the tokens
var spaces = []; // Indices of whitespace tokens on the current line
var hasTag = false; // Is there a {{tag}} on the current line?
var nonSpace = false; // Is there a non-space char on the current line?
// Strips all whitespace tokens array for the current line
// if there was a {{#tag}} on it and otherwise only space.
function stripSpace() {
if (hasTag && !nonSpace) {
while (spaces.length) {
delete tokens[spaces.pop()];
}
} else {
spaces = [];
}
hasTag = false;
nonSpace = false;
}
var start, type, value, chr, token, openSection;
while (!scanner.eos()) {
start = scanner.pos;
// Match any text between tags.
value = scanner.scanUntil(tagRes[0]);
if (value) {
for (var i = 0, len = value.length; i < len; ++i) {
chr = value.charAt(i);
if (isWhitespace(chr)) {
spaces.push(tokens.length);
} else {
nonSpace = true;
}
tokens.push(['text', chr, start, start + 1]);
start += 1;
// Check for whitespace on the current line.
if (chr === '\n') {
stripSpace();
}
}
}
// Match the opening tag.
if (!scanner.scan(tagRes[0])) break;
hasTag = true;
// Get the tag type.
type = scanner.scan(tagRe) || 'name';
scanner.scan(whiteRe);
// Get the tag value.
if (type === '=') {
value = scanner.scanUntil(eqRe);
scanner.scan(eqRe);
scanner.scanUntil(tagRes[1]);
} else if (type === '{') {
value = scanner.scanUntil(new RegExp('\\s*' + escapeRegExp('}' + tags[1])));
scanner.scan(curlyRe);
scanner.scanUntil(tagRes[1]);
type = '&';
} else {
value = scanner.scanUntil(tagRes[1]);
}
// Match the closing tag.
if (!scanner.scan(tagRes[1])) {
throw new Error('Unclosed tag at ' + scanner.pos);
}
token = [ type, value, start, scanner.pos ];
tokens.push(token);
if (type === '#' || type === '^') {
sections.push(token);
} else if (type === '/') {
// Check section nesting.
openSection = sections.pop();
if (!openSection) {
throw new Error('Unopened section "' + value + '" at ' + start);
}
if (openSection[1] !== value) {
throw new Error('Unclosed section "' + openSection[1] + '" at ' + start);
}
} else if (type === 'name' || type === '{' || type === '&') {
nonSpace = true;
} else if (type === '=') {
// Set the tags for the next time around.
tagRes = escapeTags(tags = value.split(spaceRe));
}
}
// Make sure there are no open sections when we're done.
openSection = sections.pop();
if (openSection) {
throw new Error('Unclosed section "' + openSection[1] + '" at ' + scanner.pos);
}
return nestTokens(squashTokens(tokens));
}
/**
* Combines the values of consecutive text tokens in the given `tokens` array
* to a single token.