Job arrays

Job arrays offer a mechanism for submitting and managing collections of similar jobs quickly and easily. Instead of submitting N jobs independently, you can submit one array job unifying N tasks.

A job array can be submitted simply by adding

#SBATCH --array=x-y

to the job script where x and y are the array bounds. A job array can also be specified at the command line with

sbatch --array=x-y job_script.sub

A job array will then be created with a number of independent jobs, known as arrays tasks.

Comma-separated list of task numbers can be provided, if for example a user wants to rerun specific jobs after a previously completed job array.

sbatch --array=4,6,12,16  job_script.sub

A user can limit the number of active array tasks at a time using the %N option

#SBATCH --array=1-100%5

The above will create an array of 200 tasks, running only 5 tasks at a time.

Each sub-job in a job array will have a SLURM_ARRAY_JOB_ID that includes both the parent SLURM_ARRAY_JOB_ID and a unique SLURM_ARRAY_TASK_ID after the character underscore "_".

101300_1
101300_2
101300_3
101300_4
101300_5

To cancel a job array or a sub-job use the scancel command:

scancel 101300 # will cancel the whole array
scancel 101300_1 # will cancel only the first sub-job

The $SLURM_ARRAY_TASK_ID can be used inside the job script to handle input and output files for that task.

For example, for a job arrays with input files named input_1.txt, input_2.txt etc one can refer the the input files as input_${SLURM_ARRAY_TASK_ID}.txt. The output files can he handled the same way.