DCCUserGuide

Workshops are regularly held during the semester to help you get started with the DCC. View the schedule and enroll. PDF slides

  • Additional Resources
  • The Duke Compute Cluster (DCC) Overview

    • Formerly known as the “DSCR” (Duke Shared Cluster Resource)
    • 628 compute nodes with 15667 CPU cores (as of April 2019)
    • Most nodes are purchased by labs and depts
    • Some are provided by the University
    • 1000 TB of primary storage (Isilon X-series)
    • Red Hat Enterprise Linux 7
    • Uses the Slurm job queueing system

    NEW See: Overview of cluster storage and Update Paths to Use New DCC Storage Options

    Getting Started

    From on-campus (University or DUHS networks):

    ssh NetID@dcc-slogin.oit.duke.edu

    From off campus, first , use the Duke VPN
    To move files to (or from) the DCC use scp or rsync for Linux or Mac workstations, Use winscp for Windows

    Copying a file to the DCC (“push”) % scp data001.txt netid@dcc-slogin.oit.duke.edu:.

    Copying a file from the DCC (“pull”): % scp netid@dcc-slogin.oit.duke.edu:output.txt .

    Use either scp -r (small files) or rsync –av (large files)

    Pushing a directory:

    rsync –av dir1/ netid@dcc-slogin-02.oit.duke.edu:. scp -r dir1/ netid@dcc-slogin-02.oit.duke.edu:.

    Pulling a directory:

    rsync –av netid@dcc-slogin.oit.duke.edu:~/dir1 . scp -r netid@dcc-slogin.oit.duke.edu:~/dir1 .

    DCC Software

    Run module avail to see the list of installed applications

    Run module load (module name) to use the application

    You can add the module load command to your job scripts, or to the end of the .bash_profile file in your home directory.

    Applications can be installed on request: Send an email to rescomputing@duke.edu

    Introduction to SLURM

    Most DCC partitions are lab-owned machines. These can only be used by members of the group. Submitting to a group partition gives “high-priority”.

    Submit to partitions with

    #SBATCH –p (partition name) --account=(account name)

    (in a job script) or

    srun –p (partition name) --account=(account name) --pty bash –i

    (interactively) In general, the partition name and account name will be the same for most lab-owned machines.

    Common DCC Partitions

    There are four different DCC partitions to which batch jobs and interactive sessions can be directed by all DCC users:

    • common for jobs that will run on the DCC core nodes (up to 64 GB RAM).
    • gpu-common for jobs that will run on DCC GPU nodes.

    • scavenger for jobs that will run on lab-owned nodes in “low priority” (kill and requeue preemption).

    • scavenger-gpu for GPU jobs that will run on lab-owned nodes in “low priority” (kill and requeue preemption).

    Note: These partitions do not require an “–account=” flag. (i.e., they accept jobs from any account.) If a partition is not specified, the default partition is the common partition.

    Running an interactive job

    Reserve a compute node by typing srun –pty bash -i

    tm103@dscr-slogin-02 ~ $ srun --pty bash -i   
    srun: job 186535 queued and waiting for resources   
    srun: job 186535 has been allocated resources   
    tm103@dcc-core-11 ~ $
    tm103@dcc-core-11 ~ $ squeue -u tm103  
    
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 
    186535 common bash tm103 R 0:14 1 dcc-core-11 

    I now have an interactive session in the common partition on node dcc-core-11

    SLURM commands

    sbatch – Submit a batch job

    #SBATCH – Specify job parameters in a job script

    squeue – Show lists of jobs

    scancel – Delete one or more batch jobs

    sinfo – Show info about machines

    scontrol – Show cluster configuration information

    sbatch

    Use sbatch (all lower case) to submit text file job scripts, e.g. test.sh

        sbatch test.sh

    Use #SBATCH (upper case) in your scripts for scheduler directives, e.g.

    #SBATCH --mem=1G
    #SBATCH --output=matlab.out

    All SLURM directives can be given on the command line instead of the script. https://slurm.schedmd.com/sbatch.html

    Job script example

    #!/bin/bash  
    
#SBATCH --output=test.out
    hostname # print hostname

    This prints the name of the compute node in the file “test.out”

    tm103@dscr-slogin-02 ~/slurm $ sbatch simple.sh
    Submitted batch job 186554
tm103@dscr-slogin-02 ~/slurm $ cat test.out 
    dcc-core-14 

    Long-form commands example

    #SBATCH --output=slurm.out
    #SBATCH --error=slurm.err  
    #SBATCH --mem=100 # 100 MB RAM 
    #SBATCH -–partition=scavenger# 
    hostname 1>&2  #prints hostname to the error file

    This job will run in low priority on a lab node in the “scavenger” partition

    Short-form commands example.

    SLURM short commands don’t use “=“ signs

    #!/bin/bash
    #SBATCH -o slurm.out
    #SBATCH -e slurm.err
    #SBATCH -–mem=4G # 4 GBs RAM 
    #SBATCH –p scavenger
    hostname 1>&2  #prints hostname to the error file

    R example script

    #!/bin/bash
    #SBATCH –e slurm.err
    #SBATCH --mem=4G # 4 GB RAM 
    module load R/3.6.0
    R CMD BATCH Rcode.R

    This loads the environment module for R/3.6.0 and runs a single R script (“Rcode.R”)

    The “#SBATCH –mem=4G” requests additional RAM

    Slurm memory directives

    The default memory request (allocation) is 2 GB RAM. This is a hard limit, always request a little more. To request a total amount of memory for the job, use

    --mem=<MB>

    or

    --mem=<Gigabyte>G

    the amount of memory required per node, or

    --mem-per-cpu=<MB>

    The amount of memory per CPU core, for multi-threaded jobs

    Note: –mem and –mem-per-cpu are mutually exclusive

    Slurm parallel directives

    All parallel directives have defaults of 1

    -N <number> How many nodes (machines)

    -n <number> or --ntasks=<number> How many parallel jobs (“tasks”)

    -c <number> or --cpus-per-task=<number>

    Use -n and -N for multi-node jobs (e.g. MPI)

    Use -c (–cpus-per-task) for multi-threaded jobs

    Multi-threaded (multi-core) example

    !/bin/bash
    #SBATCH –J test 
    #SBATCH –o test.out
    #SBATCH –c 4
    #SBATCH –-mem-per-cpu=500 #(500 MB) 
    myApplication –n $SLURM_CPUS_PER_TASK

    The value of $SLURM_CPUS_PER_TASK is the number after “-c” This example starts a single, multi-threaded job that uses 4 CPU cores and 2 GB (4x500MB) of RAM

    OpenMP multicore example

    !/bin/bash
    #SBATCH –J openmp-test 
    #SBATCH –o slurm.out
    #SBATCH –c 4
    export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
    myOpenMPapp # will run on 4 CPU cores

    This sets $OMP_NUM_THREADS to the value of $SLURM_CPUS_PER_TASK

    Slurm job arrays

    Slurm job arrays are a mechanism for submitting and managing collections of similar jobs using one job script and one application program

    Add --array or -a option to the job script

    Each job task will inherit a SLURM_ARRAY_TASK_ID environment variable with a different integer value.

    Each job array can be up 100,000 job tasks on the DCC

    Job arrays are only supported for batch jobs

    Job array “tasks” must be independent: http://slurm.schedmd.com/job_array.html

    For example, in a job script, add the line

    #SBATCH --array=1-30

    or, alternatively,

    #SBATCH -a 1-30

    to submit 30 job tasks. The job array indices can also be specified on the command line, e.g.

    sbatch -a 1-30 myjob.sh

    The index values can be continuous, e.g.

    -a 0-31 (32 tasks, numbered from 0,1,2,…,31)

    or discontinuous, e.g.

    -a 3,5,7-9,12 (6 tasks, numbers 3,5,7,8,9,12)

    It can also be a single job task, e.g.

    -a 7

    The discontinous notation is useful for resubmitting specific job tasks that had previously failed

    Each job task is assigned the enviromental variable

    $SLURM_ARRAY_TASK_ID set to it’s index value.

    tm103@dcc-slogin-02  ~/misc/jobarrays $ cat array-test.sh 
    #!/bin/bash
    echo $SLURM_ARRAY_TASK_ID
    tm103@dcc-slogin-02  ~/misc/jobarrays $ sbatch -a 1-3 array-test.sh 
    Submitted batch job 24845830
    tm103@dcc-slogin-02  ~/misc/jobarrays $ ls slurm-24845830*
    slurm-24845830_1.out  slurm-24845830_2.out  slurm-24845830_3.out
    tm103@dcc-slogin-02  ~/misc/jobarrays $ cat slurm-24845830*
    1
    2
    3
    tm103@dcc-slogin-02  ~/misc/jobarrays $

    Python job array example

    #!/bin/bash
    #SBATCH -e slurm_%A_%a.err
    #SBATCH -o slurm_%A_%a.out 
    #SBATCH --array=1-5000 
    python myCode.py
    $ cat test.py
    import os
    taskID=int(os.environ['SLURM_ARRAY_TASK_ID'])
    ...

    Start 5000 Python jobs, each with a different “taskID”, initialized from $SLURM_ARRAY_TASK_ID

    Importing Environmental Variables

    Python:

    numCPUs=int(os.environ['SLURM_CPUS_PER_TASK'])
    taskID=int(os.environ['SLURM_ARRAY_TASK_ID'])

    R:

    numCPUs <- as.integer(Sys.getenv(SLURM_CPUS_PER_TASK)) 
    taskID <- as.integer(Sys.getenv(SLURM_ARRAY_TASK_ID)) 
    

    MATLAB:

    numCPUs = str2num(getenv('SLURM_CPUS_PER_TASK'))
    taskID = str2num(getenv('SLURM_ARRAY_TASK_ID'))

    Processing separate input files

    Process an existing file list, e.g. files.txt

    #!/bin/bash
    readarray -t FILES < files.txt
    FILENAME=${FILES[(($SLURM_ARRAY_TASK_ID - 1))]}
    myapp $FILENAME

    Dynamically generate a file list from “ls”

    #!/bin/bash
    export FILES=($(ls -1 myfile*))
    FILENAME=${FILES[(($SLURM_ARRAY_TASK_ID - 1))]}
    myapp $FILENAME

    Example: Using the taskID as part of the file name and output directory

    For the case with input file names of the form input1,input2,…,inputN for -a 1-N, e.g.

    '#!/bin/bash
    '#SBATCH -e slurm_%A_%a.err
    '#SBATCH -o slurm_%A_%a.out      
    mkdir out_${SLURM_ARRAY_TASK_ID}     
    cd out_${SLURM_ARRAY_TASK_ID}
    myapp ../input_${SLURM_ARRAY_TASK_ID}.txt

    where output directories out1, out2, … are created for input files input1.txt, input2.txt,…

    “Unrolling” for loops example

    Original “serial” code (Python)

    fibonacci = [0,1,1,2,3,5,8,13,21]
    for i in range(len(fibonacci)):
        print(i,fibonacci[i])

    Job array version

    import os
    i=int(os.environ['SLURM_ARRAY_TASK_ID'])
    fibonacci = [0,1,1,2,3,5,8,13,21]
    #for i in range(len(fibonacci)):
        print(i,fibonacci[i])

    where the for loop is commented-out and each job task is doing a single “iteration”

    tm103@dcc-slogin-02  ~/misc/jobarrays $ cat fib-array.sh 
    #!/bin/bash
    #SBATCH -e slurm.err
    module load Python/2.7.11
    python fibonacci.py
    tm103@dcc-slogin-02  ~/misc/jobarrays $ sbatch –a 1-8 fib-array.sh 
    Submitted batch job 24856052
    tm103@dcc-slogin-02  ~/misc/jobarrays $ ls slurm-24856052_*
    slurm-24856052_1.out  slurm-24856052_3.out  slurm-24856052_5.out  slurm-24856052_7.out
    slurm-24856052_2.out  slurm-24856052_4.out  slurm-24856052_6.out  slurm-24856052_8.out
    tm103@dcc-slogin-02  ~/misc/jobarrays $ cat slurm-24856052*
    (1, 1)
    (2, 1)
    (3, 2)
    (4, 3)
    (5, 5)
    (6, 8)
    (7, 13)
    (8, 21)
    tm103@dcc-slogin-02  ~/misc/jobarrays $

    Running MPI jobs

    Supported MPI versions are Intel MPI and OpenMPI

    Compiling with OpenMPI

    tm103@dcc-slogin-02  ~ $ module load OpenMPI/2.1.0
    OpenMPI 2.1.0
    tm103@dcc-slogin-02  ~ $ which mpicc
    tm103@dcc-slogin-03  ~ $ cd /dscrhome/tm103/misc/slurm/openmpi
    tm103@dcc-slogin-03  ~/misc/slurm/openmpi $ mpicc -o openhello hello.c
    tm103@dscr-slogin-02  ~/misc/slurm/openmpi  ls -l openhello 
    -rwxr-xr-x. 1 tm103 scsc 9184 Sep  1 16:08 openhello`

    OpenMPI job script

    #!/bin/bash
    #SBATCH -o openhello.out
    #SBATCH -e slurm.err
    #SBATCH -n 20
    module load OpenMPI/2.1.0
    mpirun -n $SLURM_NTASKS openhello

    OpenMPI example output

    tm103@dscr-slogin-02 ~/misc/slurm/openmpi $ cat openhello.out  
    dscr-core-01, rank 0 out of 20 processors  
    dscr-core-01, rank 1 out of 20 processors  
    dscr-core-01, rank 2 out of 20 processors  
    dscr-core-01, rank 3 out of 20 processors  
    dscr-core-01, rank 4 out of 20 processors   
    dscr-core-03, rank 13 out of 20 processors 
    dscr-core-03, rank 14 out of 20 processors 
    dscr-core-03, rank 10 out of 20 processors 
    dscr-core-03, rank 11 out of 20 processors 
    dscr-core-03, rank 12 out of 20 processors 
    dscr-core-02, rank 8 out of 20 processors 
    dscr-core-02, rank 9 out of 20 processors 
    dscr-core-02, rank 5 out of 20 processors

    Intel MPI example

    #!/bin/bash
    #SBATCH --ntasks=20
    #SBATCH --output=intelhello.out
    export I_MPI_PMI_LIBRARY=/opt/slurm/lib64/libpmi.so
    source /opt/apps/intel/intelvars.sh srun -n $SLURM_NTASKS intelhello

    GPU jobs

    To run a GPU batch job, add the job script lines

        #SBATCH -p gpu-common --gres=gpu:1 
        #SBATCH -c 6 

    To get an interactive GPU node session, type the command line

    srun -p gpu-common --gres=gpu:1 --pty bash –i

    as below.

    tm103@dscr-slogin-02  ~ $ srun -p gpu-common --gres=gpu:1 --pty bash -i 
    tm103@dscr-gpu-01  ~ $ /usr/local/cuda-7.5/samples/1_Utilities/deviceQuery/deviceQuery 
    ...
    Detected 1 CUDA Capable device(s)
    
    Device 0: "Tesla K80"
      CUDA Driver Version / Runtime Version          7.5 / 7.5
      CUDA Capability Major/Minor version number:    3.7
      Total amount of global memory:                 11520 MBytes (12079136768 bytes)
      (13) Multiprocessors, (192) CUDA Cores/MP:     2496 CUDA Cores
    ...

    Job dependencies

    https://hcc-docs.unl.edu/display/HCCDOC/Job+Dependencies

    Submit a job that waits for another job to finish.

    $ sbatch dep1.q
    Submitted batch job 666898

    Make a note of the assigned job ID of dep1

    $ sbatch --dependency=afterok:666898 dep2.q

    Job dep2 will not start until dep1 finishes

    Job dependencies with arrays

    Wait for specific job array elements

    sbatch --depend=after:123_4 my.job sbatch --depend=afterok:123_4:123_8 my.job2

    Wait for entire job array to complete

    sbatch --depend=afterany:123 my.job

    Wait for entire job array to complete successfully

    sbatch --depend=afterok:123 my.job

    Wait for entire job array to complete and at least one task fails

    sbatch --depend=afternotok:123 my.job

    Additional Resources

    http://schedmd.com/slurmdocs