Parallel Jobs

Pegasus IV Cluster

Parallel jobs

Pegasus IV compute nodes and their attributes

32 Intel i7-8700, 6 CPU cores, 32 GByte of memory, Torque attributes "hexa", "i7", and "gen8"
139 Intel i5, 4 CPU cores, 16 GByte of memory, Torque attributes "quad" and "i5"

75 i5-6600, extra Torque attribute "gen6"
16 i5-4690, extra Torque attribute "gen4"
48 i5-3570, extra Torque attribute "gen3"

20 Intel i5-2500, 4 CPU cores, 16 Gbytes of memory (Wilemski/Hale subcluster), Torque attributes "wilemski"

Pegasus IV cluster is set up for parallel programming using MPI (message passing interface). We have installed OpenMPI 1.10.1 for use with the Intel compiler suite.

Compiling parallel code

The compilers are invoked via wrapping scripts that specify the appropriate compiler switches and link to the parallel libraries. In order to compile a parallel program written in Fortran you should type:

mpifort <file_name>

If you want to change the executable name or optimize your code, mpifort will take all standard Intel Fortran compiler switches (see Serial jobs). (Note: Instead of mpifort, you can also use mpif77 and mpif90, but these commands are deprecated and maybe removed in future versions of OpenMPI.) Parallel C code is compiled using:

mpicc <file_name>

while parallel C++ code is compiled by invoking

mpiCC <file_name>

As in the case of mpifort, both mpicc and mpiCC pass standard compiler switches to the original compilers, i.e., icc and icpc, respectively.

Submitting a parallel job

Similar to serial jobs, parallel jobs can be started interactively or submitted to the batch queue. To start a parallel job interactively, you can use "qsub -I" but you need to specify how many processors you wish to use. Typing

qsub -I -l nodes=4

requests an interactive job with 4 processors. Once your interactive job has started, type

mpirun ./a.out

at the command prompt to start execution of the MPI program "a.out". mpirun will automatically use all the processors assigned to the job by Torque.

To submit a parallel batch job, you need an appropriate Torque script file. A miniminal example looks like:

#!/bin/tcsh
#PBS -l nodes=2:ppn=4:quad,walltime=400:00:00
#PBS -m ae
#PBS -M user@mst.edu
#PBS -q qsNormal

cd $PBS_O_WORKDIR
mpirun ./a.out

This script is very similar to the Torque script for a serial job. However, the second line:

#PBS -l nodes=2:ppn=4:quad,walltime=400:00:00

now requests a total of 8 processors (2 compute nodes with 4 processors each) and in the last line the executable is started via mpirun (which will use all processors allocated to the job by Torque). You can also combine resources of different types. For example,

#PBS -l nodes=2:ppn=4:quad+2:ppn=6:hexa,walltime=400:00:00

requests a total of 20 processors (2 nodes of type quad with 4 processors each and 2 nodes of type hexa with 6 processors each).

Note: If ppn is not specified in the resource request, a "node" is interpreted as a single processor rather than a full physical compute node. If both nodes and ppn are specified, "node" refers to compute nodes (machines) and ppn refers to processors.