When running a full-node MPI job with MVAPICH 3.0 , you may encounter the following warning message:
[][mvp_generate_implicit_cpu_mapping] WARNING: You appear to be running at full subscription for this job. UCX spawns an additional thread for each process which may result in oversubscribed cores and poor performance. Please consider reserving at least 2 cores per node for the additional threads, enabling SMT, or setting MVP_THREADS_PER_PROCESS=2 to ensure that sufficient resources are available.
This message appears whenever you allocate a full subscription job with MVAPICH 3.0. It is based on the observed behavior of UCX, where MVAPICH developers have noted that an extra progress thread may be spawned for each process that calls ucx_init. This may not occur in every case, as it can depend on the UCX version. If you do not observe any performance impacts or signs of oversubscription, you may disregard the warning and rely on your current observations.
Affected versions
mvapich/3.0 and above
Workaround
If you do not observe any performance impacts or signs of oversubscription, you may disregard the warning and rely on your current observations.
