This package contains Mpi applications and packages for testing and 
evaluation.



NOTE TO RPM USERS
-------------------------------------------------------------------
If you have installed the eth-tools-fastfabric package through your distro
you likely only have a subset of the applications required for full
testing. Certain run scripts, builds, and functionality will be 
limited. To enable full functionality, the eth-mpi-apps package
must be installed. This package is available from Intel and contained
in the Intel Ethernet Fabric Suite (IEFS). 


SELECTING THE MPI TO USE
------------------------

If the oneAPI setvars.sh or the Intel MPI env/vars.sh have been sourced,
make will build these programs for the selected Intel MPI (I_MPI_ROOT).

Otherwise, Make will build these programs with the MPI installed in
 /usr/mpi/gcc/openmpi* 

You can override this by exporting MPICH_PREFIX.

If the MPI itself supports binary compatibility across MPI versions
(such as Intel MPI), a binary compatible, but different, version of MPI
may be selected at runtime by exporting MPICH_PREFIX.  Otherwise, the
MPI selected at build time will be used.

COMPILING THE TEST PROGRAMS:
----------------------------

The top level Makefile will build all the applications and benchmarks,
using the default (described above) or the value of MPICH_PREFIX (also
described above).

If a GPU enabled build is desired, do one of the following:
- export MPI_APPS_ONEAPI=y - this will build for Intel GPUs.
- export MPI_APPS_CUDA=y - this will build for NVIDIA GPUs.
     as needed, CUDA_DIR may be used to indicate the directory where CUDA
     is installed.  The default is /usr/local/cuda

Some useful make targets include:

	clean          - remove all binaries. Useful when changing MPIs.
	all/full/quick - currently synonyms. All build all targets.

COMPILING HPL
-------------

A version of the High Performance Computing Linpack Benchmark (HPL) is
provided, but requires that the build environment have either the Intel Math
Kernel Library (MKL) or OpenBLAS available. When compiling the MPI test
programs, if MPICH_PREFIX selects Intel MPI and MKL is also available in
the oneAPI tree where MPICH_PREFIX is getting Intel MPI from, then
MKL will be used.  Alternately when using Intel MPI, the environment variable
MKLROOT may be set to tell the Makefile to use MKL. (MKLROOT will be
automatically set if . /opt/intel/oneapi/setvars.sh is run.)
Otherwise the Makefile will choose OPENBLAS if it's rpm is installed.

OpenBLAS can be found at https://openblas.net. MKL is part of the oneAPI
package and can be found at
https://www.intel.com/content/www/us/en/developer/tools/oneapi/overview.html

If neither BLAS library is found, compilation will fail with an error. If
you wish to use a different BLAS library, please see the documentation in the
HPL source for information on manually configuring and building HPL.

RUNNING THE TEST PROGRAMS:
--------------------------

A run_ script is provided for each test application.
These scripts assume the existence of a local "mpi_hosts" file
[an alternate file name can be exported in MPI_HOSTS if desired].
For example, to run the bandwith test, simply type:

# ./run_bw

The scripts log the output of the MPI programs to the 
$PWD/logs directory.

If using SLURM, the host list can be autogenerated by specifying
MPI_HOSTS=slurm or creating an mpi_hosts file with slurm as the first host name.

For example, within a SLURM salloc session:
# MPI_HOSTS=slurm ./run_bw
or you may use sbatch directly against a run_ script such as:
# echo "slurm" > mpi_hosts
# sbatch -N 2 SBATCH_ARGS ./run_bw
where -N 2 requests 2 hosts for the job and SBATCH_ARGS should be replaced with
any additional required sbatch arguments, such as selecting a SLURM partition.

Each script has a --help option which can provide some more information.

For more information on all the scripts and their options, see
the Intel Ethernet Fabric Suite Fast Fabric User Guide.  

RUNNING HPL
-----------

When running the provided version of HPL, an additional script has been
provided for configuring it with common sample problem sizes. Use the
provided "config_hpl2" script to select from one of several sample HPL
configuration files before running HPL with the provided "run_hpl2" script.

TUNING THE TEST PROGRAMS:
-------------------------

PSM3, Intel MPI, oneCCL and OpenMPI are all extremely tunable. A complete
discussion of all their settings is far beyond the scope of this README.
When using IEFS, you can easily tune any of their settings. See the files
psm3.params, intelmpi.params, oneccl.params, and openmpi.params for details.
Each file describes the syntax of the given parameter file, explains precedence
rules and includes samples for some of the more commonly used parameters.
