GitLab

  Jobs can fail in a way that the epilog doesn't run.
  This now checks lock file against running sge shepherd processes and
  auto-removes locks that don't make sense

Hard code rm path
Update to grep -E
Switch to $() notation

Remove lock message from .o file to prevent user confusion

Improve security of cuda epilog
Ensure device mode is such that only the cluster can access.
Only assign GPUs designated for cluster use
Correct behaviour when first GPU is already allocated - don't fail the script prematurely
Improve clean up of temporary files
Mail admins when there is a fatal error which causes the queue to go into the error state
Move dev special file ownership states and exit states to constants defined at script start

Unfortunately this doesn't prevent the job from 'disappearing' when an exec fails

Adds GPU locking and notification of tasks about cards available to them for use.