Using Cray MPI

User-Specified Cray MPI Location

By default Simcenter STAR-CCM+ attempts to locate the native MPI libraries in default locations. When set, the environment variable MPICH_DIR provides the default location of the installed MPI. This variable is typically set by the cray-mpich Linux module (which is either loaded explicitly by the user or by default by the system). The location of the native MPI libraries can also be specified manually using the CRAY_MPI_HOME environment variable, for example:

setenv CRAY_MPI_HOME /opt/cray/pe/mpich/8.1.14/ofi/cray/10.0

Ensure that the version of Cray MPI specified is compatible with Simcenter STAR-CCM+ and the selected -mpi option.

Cray MPI supports applications built with different compilers. Simcenter STAR-CCM+ must use the Cray MPI libraries built using the GNU compilers. Thefore, when specifying the location of the installed MPI using the Linux module functionality, make sure that the GNU compiler build of Cray MPI is selected. For example, using module load PrgEnv-gnu.

Selecting the System libfabric on Cray EX Systems

The Slingshot 11 (Cassini) network on Cray EX systems requires a custom libfabric provided by Cray on their systems, which which is usually loaded by default via the module system. This libfabric version must be used in order to run Simcenter STAR-CCM+ on Cray EX systems. If Simcenter STAR-CCM+ cannot detect the system libfabric automatically it prints a warning message. In this case, the path to the system libfabric must be specified manually using the expert command-line options -xsystemlibfabric and -ldlibpath.

Process Placement and Distribution

Cray MPI 8 on SLURM Systems

Process placement can be achieved via the -machinefile, -on, or -batchsystem command line options.

On SLURM systems the default process distribution strategy is set to arbitrary, meaning it follows the process order given by the batch system, host file, or host list. This strategy can be overwritten by setting --distribution or -m to the desired value via the -mpiflags command line option.

Cray MPI 7 and Cray MPI 8 on Non-SLURM Systems

Because Cray MPI is tightly coupled with the job scheduler, when launching Simcenter STAR-CCM+ you must specify the number of processes using the -np command line option. The list of hosts is automatically determined by the Cray system; the list may not be specified via the -machinefile, -on, nor -batchsystem command line options.

If special process distribution is required, control this distribution using the job scheduler (usually PBS or SLURM) when submitting a job. Refer to the documentation of your cluster and job scheduler for details.

Specifying Environment Variables

On non-SLURM Cray systems, user environment variables (via the -mpiflags option) can be passed to Cray MPI using either -e or --environment-override. When the same environment variable is passed twice, -e takes precedence over --environment-override, if they are passed using different command line flags, regardless of their order. Otherwise the last value is used.