Setting Up an HPC

Environment Variables

Admins will need to setup the below env vars.

1. $PROJECTS_HOME
2. $WORKDIR
3. $BC_* (Baseline Configuration Env Vars)

Directories

Admins will need to create the below directories

1. $PROJECTS_HOME/singularity/containers
2. $PROJECTS_HOME/datools

Utils

mkdir $PROJECTS_HOME/datools

cd $PROJECTS_HOME/datools

git clone git@gitlab.infinitetactics.com:scale/ilauncher-utils.git - (requires ssh keys)

OR

git clone http://gitlab.infinitetactics.com/scale/ilauncher-utils.git

mv iLauncher-utils utils

chgrp -R datools utils

chmod -R 775 utils

Containers

Select an HPC with all desired containers (preferably at the same DSRC as the HPC you are setting up). Archive the desired containers into a single file (tar). SCP files to HPC at $PROJECTS_HOME/singularity/containers/datools. The archive file should be quite large so a help desk ticket may be required to increase your disk quota.

mkdir $PROJECTS_HOME/singularity/containers/datools

chgrp datools datools && chmod 775 datools

cd $PROJECTS_HOME/singularity/containers/datools

Example:

scp onyx.erd.hpc.mil:/p/app/unsupported/singularity/containers/datools/containers.tar .

tar xzvf containers.tar

chgrp datools * && chmod 775 *

Conda

Requires the below images in at *$PROJECTS_HOME/singularity/containers/datools

• conda-cpu27.simg
• conda-cpu36.simg
• conda-cpu37.simg
• conda-gpu_cuda8027.simg
• conda-gpu_cuda9027.simg
• conda-gpu_cuda11036.simg
• conda-gpu_cuda11037.simg
• conda-gpu_cuda11038.simg

Copy files from another HPC and create sym links

cd $PROJECTS_HOME/datools

mkdir conda-bin2.7 conda-bin3.6 conda-bin3.7 logs

ln -s conda-bin2.7 conda-bin2 && ln -s anconda-bin3.7 conda-bin3

scp onyx.erdc.hpc.mil:/p/app/unsupported/datools/conda-bin2.7/conda conda-bin2.7/

scp onyx.erdc.hpc.mil:/p/app/unsupported/datools/conda-bin3.6/conda conda-bin3.6/

scp onyx.erdc.hpc.mil:/p/app/unsupported/datools/conda-bin3.7/conda conda-bin3.7/

cd $PROJECTS_HOME/datools/conda-bin2.7 && Create sym links as described below

cd $PROJECTS_HOME/datools/conda-bin3.6 && Create sym links as described below

cd $PROJECTS_HOME/datools/conda-bin3.7 && Create sym links as described below

cd $PROJECTS_HOME/datools && chgrp -R datools conda-bin* logs && chown -R 775 conda-bin* logs

Conda Sym Links

Command

bash

for cmd in python python3 dask-worker dask-scheduler \

jupyter pip pip3 tensorboard  conda-bash conda ipengine ipython ipcontroller; do

ln -s conda "$cmd"

done

Result

ls -ltr

total 4

-rwxrwxr-x. 1 rlarson datools 1556 Apr 21 11:40 conda

lrwxrwxrwx. 1 rlarson datools 8 Apr 21 11:51 python -> conda

lrwxrwxrwx. 1 rlarson datools 8 Apr 21 11:51 python3 -> conda

lrwxrwxrwx. 1 rlarson datools 8 Apr 21 11:51 dask-worker -> conda

lrwxrwxrwx. 1 rlarson datools 8 Apr 21 11:51 dask-scheduler -> conda

lrwxrwxrwx. 1 rlarson datools 8 Apr 21 11:51 jupyter -> conda

lrwxrwxrwx. 1 rlarson datools 8 Apr 21 11:51 pip -> conda

lrwxrwxrwx. 1 rlarson datools 8 Apr 21 11:51 pip3 -> conda

lrwxrwxrwx. 1 rlarson datools 8 Apr 21 11:51 tensorboard -> conda

lrwxrwxrwx. 1 rlarson datools 8 Apr 21 11:51 conda-bash -> conda

Cuda Stubs

Needs Updated

Misc

Copy misc directories and files from another HPC that has already been setup. Choose an HPC and replace $HPC with the fully qualified domain name in the below commands. Replace $PH with the $PROJECTS_HOME on you selected HPC in the below commands.

cd $PROJECTS_HOME/datools

scp -r $HPC:$PH/datools/utils/VM .

scp -r $HPC:$PH/datools/utils/VirtualGL .

scp $HPC:$PH/datools/utils/zeppelin.tgz .

chmod and chgrp all files and directories