Ctrlk
Contact support
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Validation Tests

Container images

These tests use NVIDIA NGC container images:

  • nvcr.io/nvidia/nv-mission-control/nvbandwidth — bandwidth tests. This image is part of the NVIDIA Mission Control software stack and requires a valid NVIDIA Mission Control entitlement. Pulling it needs an NGC API key configured as an image pull secret in the cluster. The bandwidth manifests below reference this secret as ngc-nvcr; create it before applying them (this step is also included in the post-wizard deployment steps).

  • nvcr.io/nvidia/pytorch — NCCL tests. This image is publicly pullable from NGC without authentication and ships the NCCL test binaries (for example /usr/local/bin/all_reduce_perf_mpi).

To create the ngc-nvcr image pull secret, generate an NGC API key from the NGC dashboard (Setup → Generate API Key), then create a docker-registry secret in the namespace where the tests run. NGC uses the literal username $oauthtoken and the API key as the password (quote $oauthtoken so the shell does not expand it):

kubectl create secret docker-registry ngc-nvcr \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password='<NGC_API_KEY>' \
  -n default

The NCCL tests use the public pytorch image and do not need this secret.

Adjust image tags to match the CUDA and driver versions supported by your cluster.

Air-gapped clusters

The MPI-based tests below start sshd in the worker pods by installing openssh-server at runtime with apt-get, which requires the pods to reach a package repository. In an air-gapped cluster, either point the nodes at a reachable local apt mirror, or substitute a container image that already includes openssh-server (and, for the NCCL tests, the NCCL test binaries) so that no runtime package installation is needed.

Multi-Node NVLink (MNNVL) ComputeDomain Bandwidth Test (dra-computedomain-test.yaml)

This test exercises the NVIDIA DRA driver's ComputeDomain (the Multi-Node NVLink / IMEX channel) with a multi-node nvbandwidth MPIJob. It applies to MNNVL-capable platforms — for example DGX GB200 and GB300, and future MNNVL systems — and is named for the capability it validates rather than a specific platform SKU.

---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: nvbandwidth-test-compute-domain
spec:
  numNodes: 2
  channel:
    resourceClaimTemplate:
      name: nvbandwidth-test-compute-domain-channel

---
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nvbandwidth-test
spec:
  slotsPerWorker: 4
  launcherCreationPolicy: WaitForWorkersReady
  runPolicy:
    cleanPodPolicy: Running
  # The NGC nvbandwidth image runs as root and does not ship an SSH server,
  # so keys are mounted under /root/.ssh and the worker installs openssh-server
  # at runtime (see the Worker spec below).
  sshAuthMountPath: /root/.ssh
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        metadata:
          labels:
            mpi-memcpy-dra-test-replica: mpi-launcher
        spec:
          restartPolicy: OnFailure
          # The launcher pins to a control-plane node; tolerate its taint.
          tolerations:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists
            effect: NoSchedule
          containers:
          - image: nvcr.io/nvidia/nv-mission-control/nvbandwidth:1.8.0
            name: mpi-launcher
            securityContext:
              runAsUser: 0
            env:
            - name: OMPI_ALLOW_RUN_AS_ROOT
              value: "1"
            - name: OMPI_ALLOW_RUN_AS_ROOT_CONFIRM
              value: "1"
            command: ["/bin/bash", "-lc"]
            args:
            - >
              mpirun --allow-run-as-root
              --bind-to core --map-by ppr:4:node -np 8
              --report-bindings -q
              -mca plm_rsh_args "-p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /root/.ssh/id_rsa"
              nvbandwidth -t multinode_device_to_device_memcpy_read_ce
          imagePullSecrets:
          - name: ngc-nvcr
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: node-role.kubernetes.io/control-plane
                    operator: Exists
    Worker:
      replicas: 2
      template:
        metadata:
          labels:
            mpi-memcpy-dra-test-replica: mpi-worker
        spec:
          restartPolicy: OnFailure
          containers:
          - image: nvcr.io/nvidia/nv-mission-control/nvbandwidth:1.8.0
            name: mpi-worker
            securityContext:
              runAsUser: 0
            # The NGC image has no SSH server; install openssh-server at runtime
            # and start sshd on port 2222 for the MPI launcher to connect.
            command: ["/bin/bash", "-lc"]
            args:
            - >
              apt-get update -qq &&
              apt-get install -y -q openssh-server &&
              mkdir -p /run/sshd &&
              ssh-keygen -A &&
              exec /usr/sbin/sshd -De -p 2222 -o StrictModes=no
            resources:
              limits:
                nvidia.com/gpu: 4
              claims:
              - name: compute-domain-channel
          imagePullSecrets:
          - name: ngc-nvcr
          resourceClaims:
          - name: compute-domain-channel
            resourceClaimTemplateName: nvbandwidth-test-compute-domain-channel
          # GB200/GB300 only: co-locate all worker pods within the same NVL
          # clique so the multi-node nvbandwidth test exercises NVLink rather
          # than the scale-out fabric. Omit (or change topologyKey) on
          # non-NVL accelerators.
          affinity:
            podAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: mpi-memcpy-dra-test-replica
                    operator: In
                    values:
                    - mpi-worker
                topologyKey: nvidia.com/gpu.clique

InfiniBand (SR-IOV) Bandwidth Tests (ib-bandwidth-test.yaml)

nvbandwidth over the SR-IOV InfiniBand fabric. This test applies to InfiniBand-fabric deployments — for example GB200 / GB300 SuperPOD systems configured for InfiniBand.

InfiniBand (SR-IOV) NCCL Tests (ib-nccl-test.yaml)

NCCL all-reduce over the SR-IOV InfiniBand fabric. This test applies to InfiniBand-fabric deployments — DGX B200, and B-series / GB-series systems configured for InfiniBand.

Spectrum-X (RoCE) deployments

This test targets the InfiniBand fabric. DGX systems configured for NVIDIA Spectrum-X Ethernet (RoCE) — for example DGX B300 SuperPOD deployments using Spectrum-X — use a different NCCL transport and a different secondary-network and resource model, and therefore require a separate RoCE NCCL validation test. Do not use the InfiniBand test below on a Spectrum-X / RoCE-configured cluster; use the Spectrum-X (RoCE) NCCL Tests section that follows.

Spectrum-X (RoCE) NCCL Tests (roce-nccl-test.yaml)

NCCL all-reduce over the NVIDIA Spectrum-X Ethernet (RoCE) fabric. This test applies to Spectrum-X / RoCE deployments — for example DGX B300 SuperPOD systems configured for Spectrum-X. It is the RoCE counterpart to the InfiniBand NCCL test above: it engages the NVIDIA Spectrum-X NCCL plugin (NCCL_NET_PLUGIN=spcx) and the twin-planar RoCE secondary networks, rather than the SR-IOV InfiniBand transport.

Fabric and resource names are site-specific

The nvidia.com/rX-pY resources and the k8s.v1.cni.cncf.io/networks names below are the twin-planar layout (8 rails × 2 planes = 16 RoCE VFs) provisioned by the NVIDIA Network Operator for Spectrum-X. Align them to your site's NetworkAttachmentDefinition / NicClusterPolicy names before applying. The manifest sets NCCL_DEBUG=INFO so the launcher log shows which transport NCCL selected; confirm a successful run by checking the launcher log for the Spectrum-X plugin (lines such as NCCL INFO Assigned NET plugin SPCX to comm and channels routed via NET/SPCX/<n>/GDRDMA) together with a non-zero average bus bandwidth (# Avg bus bandwidth).

Last updated