ENH: Update FUNPACK to work with CSV files generated on the RAP

A range of changes updating funpack to work with CSV files generated on the UK Biobank Research Analysis Platform (RAP), and for compatibility with new versions of pandas and pyparsing.

  • funpack can now read input files which use the p<datafield>[_i<instance>][_a<index>] column naming convention.
  • funpack can now read input files which contain "compressed" multi-valued columns - this is described below.
  • New --normalise_names option, which converts column names to the <datafield>-<instance>.<index> format.
  • Migrate to setuptools-scm for version number management.
  • Various updates to address changes and deprecations in recent versions of pandas and pyparsing.

The most important change in this release is the new ability of funpack to load "compressed" multi-valued columns, such as data field 41202 (ICD10 diagnosis codes). CSV files generated on the RAP store these datafields in a different format to how they are stored in CSV files downloaded from the UKB Showcase.

Traditionally, these multi-valued data fields are stored with a separate column for each value, for example:

eid,  41202-0.0, 41202-0.1, 41202-0.2, ..., 41202-0.<N-1>
1001, C751,      C792,      ,          ...,
1002, S051,      ,          ,          ...,
1003, E091,      C751,      Q123,      ..., X998
1004, ,          ,          ,          ...,
1005, G443,      G451,      G459,      ...,

i.e. there are as many columns as needed to store the maximum number of values (N) for any individual.

In files generated on the RAP, some of these multi-valued datafields have been "compressed" into a single column, with values separated by pipe (|) characters, e.g.:

eid,  41202-0.0
1001, C751|C792
1002, S051
1003, E091|C751|Q123|...|X998
1004,
1005, G443|G451|G459

funpack has been updated to automatically detect columns of this type, and convert them to the previous format, so that the values are stored across multiple columns. This expansion is performed by default, but can be inhibited via the new --no_expand_compressed option.

CSV files generated on the RAP may also use an alternative column naming scheme of the form:

p<datafield>[_i<instance>][_a<index>]

where the instance and array index sections are only present for datafields with multiple instances or values, e.g.:

  • p31 for sex
  • p53_i0, for date of attending assessment centre (first visit)
  • p22600_a1 for job title (second several values)
  • p20003_i1_a2 for verbally reported treatments/medications (second visit, third of several values)

funpack has been updated to correctly interpret files which use this column naming convention, and to optionally convert them to the more conventional <datafield>-<instance>.<index> format via the new --normalise_names option (which would convert the above column names to 31-0.0, 53-0.0, 22600-0.1 and 20003-1.2).

Edited by Paul McCarthy

Merge request reports

Loading