> For the complete documentation index, see [llms.txt](https://run-ai-docs.nvidia.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://run-ai-docs.nvidia.com/self-hosted/2.25/getting-started/installation/bcm-install/airgapped-deployment.md). # Air-Gapped Deployment This guide covers end-to-end deployment of NVIDIA Run:ai on a cluster with no internet access, including air-gapped-specific steps for preparing and transferring artifacts offline. From BCM 11.33.1, the installation is decoupled into two wizards: `cm-kubernetes-setup` provisions Kubernetes and the NVIDIA Run:ai dependencies, and `cm-runai-setup` then deploys NVIDIA Run:ai itself. Complete the [Preparations](/self-hosted/2.25/getting-started/installation/bcm-install/preparations.md) checklist before starting. All infrastructure requirements (IP addresses, DNS records, TLS certificates, credentials) are the same for both connected and air-gapped deployments. {% hint style="info" %} **Note** These instructions require **BCM 11.33.1** or later. {% endhint %} ## Prerequisites and Requirements * **BCM version**: 11.33.1 or later on the air-gapped cluster * **Kubernetes version**: v1.34.7 * **Operating system**: Ubuntu 22.04, Ubuntu 24.04, Rocky Linux 9u3, or RHEL 9u3 * **Architecture**: All nodes in the cluster must share the same architecture (x86\_64 or arm64). Mixed-architecture clusters require additional manual steps not covered by this guide. The initial steps must be performed on a separate **internet-connected host** with: * The same OS and architecture as the air-gapped cluster * BCM package repositories configured * The `airgap-scripts` directory from BCM 11.33.1 or later * Internet access {% hint style="info" %} **Note** A BCM 11.33.1 virtual machine is the recommended internet-connected host for convenience, as it comes with the required package repositories and `airgap-scripts` already configured. {% endhint %} ## Prepare Air-Gapped Requirements All steps in this section are performed on the **internet-connected host**. 1. Add the airgap scripts to `PATH`: * If the `cm-setup` package is installed, the scripts are located at: ```bash /cm/local/apps/cm-setup/lib/python3.12/site-packages/cmsetup/plugins/kubernetes/airgap-scripts ``` * When the `cm-setup` module is loaded (enabled by default), this path is available via the `K8S_AG_SCRIPTS` environment variable: ```bash export PATH=$PATH:$K8S_AG_SCRIPTS ``` * If `cm-setup` is not available, copy the scripts to the host manually before proceeding. 2. Install Helm to `/usr/local/bin/helm`: * Run the included helper script: ```bash download_helm_binary.sh ``` * Verify the installation: ```bash root@internet-host:~# helm version version.BuildInfo{Version:"v3.20.2", GitCommit:"8fb76d6ab555577e98e23b7500009537a471feee", GitTreeState:"clean", GoVersion:"go1.25.9"} ``` 3. Install Docker and skopeo. On a BCM head node: * Run `apt update` if needed, then install skopeo: ```bash apt install skopeo ``` * Run `cm-docker-setup` to configure Docker, then load the Docker module: ```bash cm-docker-setup module load docker ```

Note

During the cm-docker-setup installation, make sure to select the head node that will be running the Kubernetes wizard later.

* Otherwise, follow the [Docker Engine installation instructions](https://docs.docker.com/engine/install/) for your OS. * Verify Docker is running: ```bash docker ps ```

Note

BCM uses skopeo for most image handling, but Docker is required to support multi-arch container images for Run:ai. To avoid Docker Hub pull rate limits (100 unauthenticated pulls per 6 hours), authenticate before downloading:

skopeo login docker.io
   docker login

Authenticated pulls are limited to 200 per 6 hours, which is sufficient for preparing the air-gapped tarball. These credentials are not included in the tarball or the air-gapped environment.

4. Create a working directory and run the download scripts from within it: * Create the directory and navigate into it: ```bash mkdir -p airgapped cd airgapped ``` * Run the OS-appropriate package download script, then download the container images, Helm charts, and CRDs: ```bash KUBE_VERSION="1.34" # For Ubuntu Linux: download_ubuntu_packages.sh --kube-version=$KUBE_VERSION # For Rocky Linux or RHEL 9u3: download_r9_packages.sh --kube-version=$KUBE_VERSION # Then, for either OS: download_container_images.sh --kube-version=$KUBE_VERSION download_helm_charts.sh download_crds.sh ```

Note

Run each script individually to verify it succeeds before proceeding to the next. All scripts write output to the current directory. Adjust --kube-version if deploying a different Kubernetes version.

Multi-architecture clusters

download_ubuntu_packages.sh, download_r9_packages.sh, and download_container_images.sh accept --arch to override the architecture of the host performing the download. Helm charts and CRDs are not architecture-specific. For example, to download container images for more than one architecture:

download_container_images.sh --arch amd64 --arch arm64
5. Download the NVIDIA Run:ai air-gapped tarball to the same working directory. Air-gapped packages are published as an NGC Resource; browse the [NGC Resource](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/runai/resources/runai-airgapp-package/version?) page to select the required version. This step requires an NGC Personal Key (beginning with `nvapi-`); see [NGC Personal Keys](https://org.ngc.nvidia.com/setup/personal-keys) for more information. * Before running the `curl` command, set the following variables. To avoid writing `NGC_CLI_API_KEY` to `.bash_history`, use `read -rs -p "NGC API key: " NGC_CLI_API_KEY` instead of a plain assignment: ```shellscript NGC_CLI_API_KEY="nvapi-REDACTED" RUNAI_VERSION="" ``` * Run the following command: ```bash curl -LO --request GET \ "https://api.ngc.nvidia.com/v2/org/nvidia/team/runai/resources/runai-airgapp-package/versions/$RUNAI_VERSION/files/runai-airgapped-package-$RUNAI_VERSION.tar.gz" \ -H "Authorization: Bearer ${NGC_CLI_API_KEY}" \ -H "Content-Type: application/json" ``` * Verify the directory contents resemble the following: ```bash root@internet-host:~/airgapped# ls -alh total 14G drwxr-xr-x 9 root root 4.0K Jun 10 04:53 . drwx------ 10 root root 4.0K Jun 10 04:21 .. drwxr-xr-x 2 root root 40 Jun 10 03:30 crds drwxr-xr-x 2 root root 4.0K Jun 10 01:25 helm-charts drwxr-xr-x 2 root root 88 Jun 10 00:56 k8s-images drwxr-xr-x 4 root root 30 Jun 10 00:44 packages drwxr-xr-x 3 root root 19 Jun 10 00:45 packages-cuda drwxr-xr-x 4 root root 30 Jun 10 00:45 packages-head drwxr-xr-x 3 root root 19 Jun 10 00:45 packages-non-dgx -rw-r--r-- 1 root root 13G Jun 10 04:32 runai-airgapped-package-2.25.x.tar.gz ``` * Verify file integrity with `sha256sum` and confirm the checksum matches the value provided by NVIDIA Run:ai Support. 6. From the parent directory, create a single archive and transfer it to the **active head node** of the air-gapped cluster (for example, via USB or a secure file transfer): ```bash cd .. tar -czf airgapped.tar.gz airgapped ``` ## Generate the Deployment Configuration All remaining steps are performed on the **air-gapped cluster head node**. 1. Access the active BCM head node via ssh: ```sh ssh root@ ``` 2. Verify the BCM version: ```sh cm-package-release-info -f cm-setup,cmdaemon Name Version Release(s) -------- --------- ------------ cm-setup 11.33.1 cmdaemon 11.33.1 ``` 3. Create the configuration and validation files in `/cm/shared/runai/` exactly as described in steps 3–4 of the [Connected deployment](/self-hosted/2.25/getting-started/installation/bcm-install/deployment.md#deploy-kubernetes-using-the-wizard) guide (DGX platform configuration files, certificates, credentials, and validation test files). 4. Start the Kubernetes wizard with the air-gap options so it pre-configures air-gapped deployment mode: ```sh cm-kubernetes-setup --airgap-registry master.cm.cluster:5000 --airgap-registry-as-helm ``` Replace `master.cm.cluster:5000` with your container registry address if different. `master.cm.cluster:5000` is the default for the BCM container registry installed later in this guide. 5. Proceed through the wizard screens exactly as described in [Connected deployment](/self-hosted/2.25/getting-started/installation/bcm-install/deployment.md#deploy-kubernetes-using-the-wizard) (steps 6–38), with one difference: at the summary screen (step 39) select **Save config & Exit** to write the configuration to `/root/cm-kubernetes-setup.conf`.

Note

The setup does not execute automatically at this stage. The wizard displays the execution command, but the air-gapped package, registry, and image-push steps below must be completed before running the deployment.

## Install Air-Gapped Package Requirements 1. Extract the archive transferred from the internet-connected host: ```bash tar -xvf airgapped.tar.gz ``` 2. Navigate to the extracted directory: * On **Ubuntu**, move it to `/tmp/airgapped` first. This is required because `apt` runs as the `_apt` user, which lacks permissions to access files under `/root`: ```bash mv airgapped /tmp/airgapped cd /tmp/airgapped ``` * On **Rocky Linux or RHEL**, change into the directory in place: ```bash cd airgapped ``` 3. Add the airgap scripts to `PATH`: ```bash export PATH=$PATH:$K8S_AG_SCRIPTS ``` 4. Install the packages on the head node and into the required software images. The required software images are determined automatically from the configuration file: ```bash cm-kubernetes-setup --airgap-install-system-packages -c /root/cm-kubernetes-setup.conf ``` For **BCM HA clusters**, copy the files to the secondary head node and repeat this step, adding the `--skip-headnode-check` flag (a normal installation is not allowed from a secondary head node).

Multi-architecture clusters

The package source must include every CPU architecture referenced by software images in the Kubernetes cluster. Package installation automatically follows each software image's architecture.

## Install Docker and Docker Registry 1. Run the container registry wizard: ```bash cm-container-registry-setup --skip-packages ``` * Select the active head node. * Ensure the container registry hostname is added, and use a custom domain name or additional Subject Alternative Name: `master.cm.cluster`. * Click **Save & Deploy**. 2. On the head node, Helm must talk to this registry without TLS issues. Copy the registry CA and update the system trust store: ```bash cp /cm/local/apps/docker-registry/var/etc/$(hostname):5000-ca.crt \ /usr/local/share/ca-certificates/bcm-docker-registry.crt update-ca-certificates ``` 3. Run the Docker wizard: ```bash cm-docker-setup --skip-packages ``` * Select the active head node and click **Save & Deploy**. 4. Verify Docker is running: ```bash module load docker docker ps ```

Note

docker login is not required in this step, as images will not be pulled from Docker Hub.

## Push Container Images and Helm Charts 1. Push all container images to the local registry: ```bash cm-kubernetes-setup --airgap-push-container-images -c /root/cm-kubernetes-setup.conf ``` 2. Push the Helm charts: ```bash cm-kubernetes-setup --airgap-push-helm-charts -c /root/cm-kubernetes-setup.conf ``` 3. Extract the downloaded CRD files: ```bash $K8S_AG_SCRIPTS/extract_crds.sh ```

Multi-architecture clusters

All images for the architectures you downloaded are pushed to the registry, irrespective of the cluster's own architecture.

## Deploy Kubernetes With the packages installed, the registry populated, and the CRDs extracted, deploy Kubernetes from the saved configuration. Use `screen` or `tmux` to prevent interruptions: ```bash screen -S install_runai cm-kubernetes-setup -c /root/cm-kubernetes-setup.conf ``` {% hint style="info" %} **Note** During the deployment process all nodes that are members of the new Kubernetes cluster will be rebooted. {% endhint %} ## Deploy NVIDIA Run:ai Using the Wizard Once `cm-kubernetes-setup` has completed, deploy NVIDIA Run:ai with the dedicated `cm-runai-setup` wizard. This wizard is provided by the `cm-setup-runai-bcm11` package, which is pre-installed on the BCM head node(s). 1. Run the wizard with `--skip-packages` (required for air-gapped deployments so the wizard does not attempt to auto-update over the internet): ```bash cm-runai-setup --skip-packages ``` 2. Select **Deploy** and click **Ok** to proceed:
Kubernetes operations
3. Select the **self-hosted** NVIDIA Run:ai deployment mode and click **Ok** to proceed:
Kubernetes operations
4. The wizard detects that it is deploying onto an air-gapped Kubernetes cluster and prompts for the NVIDIA Run:ai air-gapped tarball location. Enter the path to `runai-airgapped-package-2.25.x.tar.gz` and click **Ok** to proceed:
Kubernetes operations
5. Provide the NVIDIA Run:ai configuration (registry credentials, control plane FQDN, and certificate/key paths) as described in step 4 of the [Connected deployment](/self-hosted/2.25/getting-started/installation/bcm-install/deployment.md#deploy-nvidia-run-ai-using-the-wizard) `cm-runai-setup` section. Click **Ok** to proceed:
Kubernetes operations
{% hint style="info" %} **Note** It's recommended to save all certificates, configuration files, and deployment artifacts into a persistent and accessible location in case of redeployment. The `/cm/shared/runai/` directory referred to in this guide resides on a shared mount point and would be a suitable location. See the [TLS certificates](/self-hosted/2.25/getting-started/installation/bcm-install/preparations.md#tls-certificates) section for additional clarification. {% endhint %} 6. Select **yes** to deploy the NVIDIA Run:ai cluster. Click **Ok** to proceed:
Kubernetes operations
7. Select the node category for the NVIDIA Run:ai control plane nodes. Select `k8s-system-user`, the same category used for the Kubernetes control plane nodes. Click **Ok** to proceed:
Kubernetes operations
8. Select the required **NVIDIA Run:ai version** (v2.25.x). The `x` is replaced with the latest patch version available for the chosen minor version. Click **Ok** to proceed:
Kubernetes operations
9. The wizard validates the control plane DNS configuration. Confirm the DNS check passes and click **Ok** to proceed:
Kubernetes operations
10. Specify the load balancer IP address for NVIDIA Run:ai inference (Kourier), prepared as documented in the [Pre-installation checklist](/self-hosted/2.25/getting-started/installation/bcm-install/preparations.md#pre-installation-checklist) section. Click **Ok** to proceed:
Kubernetes operations
11. Select **Save & Exit** at the summary screen, then click **Ok**.
Kubernetes operations
{% hint style="info" %} **Note** In BCM 11.33.1, always select **Save & Exit** here. A regression currently prevents going through the questions and continuing to the deployment in a single run. {% endhint %} 12. Run the deployment from the saved configuration (keep `--skip-packages` for air-gapped): ```bash cm-runai-setup --skip-packages -c /root/cm-runai-setup.conf ``` When the setup completes, NVIDIA Run:ai should be up and running: ```bash root@bcm11-headnode:~# curl -s https://runai.example.com | grep title Run.ai ``` ## Connect to NVIDIA Run:ai User Interface Upon completion of `cm-runai-setup`, access NVIDIA Run:ai at the ingress IP or hostname specified earlier (e.g. [runai.example.com](http://runai.example.com)). The NVIDIA Run:ai credentials required for login were entered down in step 5: On first access, administrators are presented with an **optional onboarding wizard** that helps with initial setup tasks. The onboarding wizard can guide you through: * Configuring single sign-on (SSO) * Inviting the first research team You can choose to **complete or skip** the onboarding wizard and perform these actions later. After the BCM installation assistant completes, additional steps are required. If multiple Kubernetes clusters are configured in this instance of BCM, load the correct Kubernetes module before running all post-wizard commands: ```bash module unload kubernetes module load kubernetes/k8s-user ``` ## NVIDIA Dynamic Resource Allocation (DRA) Driver The [NVIDIA DRA Driver for GPUs](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-cds.html) extends how NVIDIA GPUs are consumed within Kubernetes. This is required to enable secure Internode Memory Exchange (IMEX) on Multi-Node NVLink (MNNVL) systems (e.g. GB200, GB300) for Kubernetes workloads and should be included with all NVIDIA GPU systems. 1. Install using Helm: ```bash helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ && helm repo update helm install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \ --version="25.8.0" \ --create-namespace \ --namespace nvidia-dra-driver-gpu \ --set nvidiaDriverRoot=/ \ --set resources.gpus.enabled=false ``` {% hint style="warning" %} **NGC image pull secret (bandwidth tests only)** The bandwidth tests — `dra-computedomain-test.yaml` and `ib-bandwidth-test.yaml` — pull the entitled `nvcr.io/nvidia/nv-mission-control/nvbandwidth` image and reference an image pull secret named `ngc-nvcr`. Before applying either, create that secret in the `default` namespace (replace `` with a valid NGC key): ```bash kubectl create secret docker-registry ngc-nvcr \ --docker-server=nvcr.io \ --docker-username='$oauthtoken' \ --docker-password='' \ -n default ``` The NCCL tests (`ib-nccl-test.yaml`, `roce-nccl-test.yaml`) use the public `nvcr.io/nvidia/pytorch` image and do not need this secret. {% endhint %} 2. **Multi-Node NVLink (MNNVL) platforms (e.g. GB200, GB300) only** - Create a `dra-computedomain-test.yaml` file in `/cm/shared/runai` from the [Validation tests](/self-hosted/2.25/getting-started/installation/bcm-install/validation-tests.md). The test co-locates its worker pods within a single NVL clique automatically via `podAffinity` with `topologyKey: nvidia.com/gpu.clique`, so there is no clique ID to edit for a single-rack NVL72 cluster. For multi-rack systems, adjust the `podAffinity` `topologyKey` to match your topology: ```bash # Optional: inspect the NVL clique labels on the GPU nodes kubectl describe nodes | grep nvidia.com/gpu.clique= > nvidia.com/gpu.clique=f84d133c-bbc9-55fd-b1ff-ffffc7ef6783.23322 # For MNNVL platforms (e.g. GB200 & GB300) kubectl apply -f /cm/shared/runai/dra-computedomain-test.yaml ``` 3. **Multi-Node NVLink (MNNVL) platforms (e.g. GB200, GB300) only** - Validate the test successfully completed and inspect the logs of the launcher: ```bash # For MNNVL platforms (e.g. GB200 & GB300) kubectl get pods > NAME READY STATUS RESTARTS AGE > nvbandwidth-test-launcher-snb82 0/1 Completed 0 72s kubectl logs nvbandwidth-test-launcher-snb82 ``` 4. **Multi-Node NVLink (MNNVL) platforms (e.g. GB200, GB300) only** - Cleanup test: ```bash # For MNNVL platforms (e.g. GB200 & GB300) kubectl delete -f dra-computedomain-test.yaml ``` ### Enable DRA and Multi-Node NVLink The default NVIDIA Run:ai configuration does not expose DRA features. After installing the DRA components, this can be enabled by modifying the `runaiconfig` in the cluster. See [Advanced cluster configurations](/self-hosted/2.25/infrastructure-setup/advanced-setup/cluster-config.md) for more details: ```yaml # Edit the runaiconfig object to toggle GPUNetworkAccelerationEnabled # to true and adjust tolerations for the Kubernetes control plane kubectl patch runaiconfig runai \ -n runai \ --type='merge' \ -p '{ "spec": { "workload-controller": { "GPUNetworkAccelerationEnabled": true }, "global": { "tolerations": [ { "key": "node-role.kubernetes.io/control-plane", "operator": "Exists", "effect": "NoSchedule" } ] } } }' ``` Instructions for validating the change and reverting if necessary: ```yaml # Validate the patch was applied successfully kubectl get runaiconfig runai \ -n runai \ -o custom-columns=GPUAccelEnabled:.spec.workload-controller.GPUNetworkAccelerationEnabled,Tolerations:.spec.global.tolerations # To revert the runaiconfig object change kubectl patch runaiconfig runai -n runai --type='merge' -p '{ "spec": { "workload-controller": { "GPUNetworkAccelerationEnabled": false }, "global": { "tolerations": null } } }' ``` ## Configure the Network Operator In version 11.33.1 of the BCM installation assistant, the Network Operator requires additional configuration on DGX B200 / GB200 & B300 / GB300 SuperPOD / BasePOD systems. While the operator is installed in a preceding step, it does not automatically initialize or configure SR-IOV and secondary network plugins. The following CRD resources have to be created in the exact order as below: * SR-IOV Network Policies for each NVIDIA InfiniBand NIC * An nvIPAM IP address pool * SR-IOV InfiniBand networks 1. Create SR-IOV network node policies using the `nic-cluster-policy.yaml` that was created in an earlier step: ```bash # DGX GB300 Example - substitute policy name as appropriate kubectl apply -f /cm/shared/runai/nic-cluster-policy-gb300.yaml ``` 2. Create an IPAM IP Pool using the `combined-ippools.yaml` that was created in an earlier step: ```bash # DGX GB300 Example - substitute policy name as appropriate kubectl apply -f /cm/shared/runai/combined-ippools-gb300.yaml ``` 3. Create the SR-IOV IB networks using the `combined-sriovbnet.yaml` that was created in an earlier step: ```bash # DGX GB300 Example - substitute policy name as appropriate kubectl apply -f /cm/shared/runai/combined-sriovibnet-gb300.yaml ``` 4. Create the SR-IOV Node Pool configuration using the `sriov-node-pool-config.yaml` appropriate for the DGX platform: ```bash kubectl apply -f /cm/shared/runai/sriov-node-pool-config.yaml ``` {% hint style="info" %} **Note** This will typically reconfigure NICs and may result in a node reboot. The supplied YAML sets the maxUnavailable field to 20%. This value should be adjusted to align with your operational requirements. A value of 1 would have the effect of serializing the upgrade and would result in blocking upon a single node failure. It may be appropriate for a small lab deployment to set it to 100%. This would prevent any single machine failure from blocking the remaining nodes from upgrading. For larger clusters, setting the value to a lower percentage means that the upgrade process will be effectively split into batches. {% endhint %} 5. Validate by describing one of the DGX nodes and checking for SRIOV devices: ```bash # Describe a DGX worker node kubectl describe node --context=kubernetes-admin@k8s-user | grep sriovib # Example output nvidia.com/sriovib_resource_a: 16 nvidia.com/sriovib_resource_b: 16 nvidia.com/sriovib_resource_c: 16 nvidia.com/sriovib_resource_d: 16 # Check the state of SR-IOV Nodes kubectl get -n network-operator sriovnetworknodestate --context=kubernetes-admin@k8s-user # Example Output NAME SYNC STATUS Succeeded ``` {% hint style="info" %} **Note** It might take several minutes for these settings to take effect. If the `sriovnetworkconfig` daemon changes the NIC config, then a node reboot will occur. {% endhint %} 6. Create the test file matching your platform's fabric in `/cm/shared/runai` from the [Validation tests](/self-hosted/2.25/getting-started/installation/bcm-install/validation-tests.md) page, then validate by running it. Apply only the test that matches the cluster's fabric (InfiniBand **or** Spectrum-X / RoCE), not both: * For GB200 & GB300 (InfiniBand fabric) - [ib-bandwidth-test.yaml](https://run-ai-docs.nvidia.com/self-hosted/2.25/getting-started/installation/bcm-install/pages/xKlnlrgoNacyUdLGB7GF#infiniband-sr-iov-bandwidth-tests-ib-bandwidth-test.yaml) (requires the `ngc-nvcr` image pull secret created in the DRA driver section above): ```bash # DGX GB200 & GB300 kubectl apply -f /cm/shared/runai/ib-bandwidth-test.yaml -n default ``` * For B200 & B300 (InfiniBand fabric) - [ib-nccl-test.yaml](https://run-ai-docs.nvidia.com/self-hosted/2.25/getting-started/installation/bcm-install/pages/xKlnlrgoNacyUdLGB7GF#infiniband-sr-iov-nccl-tests-ib-nccl-test.yaml): ```bash # DGX B200 & B300 configured for InfiniBand kubectl apply -f /cm/shared/runai/ib-nccl-test.yaml -n default # Clean up after validating kubectl delete -f /cm/shared/runai/ib-nccl-test.yaml -n default ``` * For B300 SuperPOD with Spectrum-X (RoCE Ethernet fabric) - [roce-nccl-test.yaml](https://run-ai-docs.nvidia.com/self-hosted/2.25/getting-started/installation/bcm-install/pages/xKlnlrgoNacyUdLGB7GF#spectrum-x-roce-nccl-tests-roce-nccl-test.yaml): ```bash # DGX B300 SuperPOD configured for Spectrum-X (RoCE) kubectl apply -f /cm/shared/runai/roce-nccl-test.yaml -n default # Clean up after validating kubectl delete -f /cm/shared/runai/roce-nccl-test.yaml -n default ``` {% hint style="info" %} **Spectrum-X (RoCE) deployments** The InfiniBand tests above target the InfiniBand fabric. DGX systems configured for NVIDIA Spectrum-X Ethernet (RoCE) — for example DGX B300 SuperPOD deployments using Spectrum-X — use a different fabric: run the [roce-nccl-test.yaml](https://run-ai-docs.nvidia.com/self-hosted/2.25/getting-started/installation/bcm-install/pages/xKlnlrgoNacyUdLGB7GF#spectrum-x-roce-nccl-tests-roce-nccl-test.yaml) test (the Spectrum-X bullet above) instead of the InfiniBand tests. Do not run the InfiniBand tests on a Spectrum-X / RoCE-configured cluster, and do not run the RoCE test on an InfiniBand cluster. {% endhint %} {% hint style="info" %} **Note** The Network Operator will restart the DGX nodes if the number of Virtual Functions in the SR-IOV Network Policy file does not match the NVIDIA/Mellanox firmware configuration. {% endhint %} ### (Optional) Apply Security Policies By default, BCM Kubernetes deployment has permissive security policies to ease in development environments. For production clusters or in secure environments, it's recommended to take additional steps to harden the cluster. This includes steps such as configuring permission manager, applying Kyverno policies, and applying Calico policies. For deployments of NVIDIA Run:ai as a part of NVIDIA Mission Control, please reach out to your NVIDIA representative for the latest example configurations and suggested policies. The Mission Control software installation guide's [Kubernetes Security Hardening](https://docs.nvidia.com/mission-control/docs/nmc-software-installation-guide/2.0.0/nmc-kube-security-guide.html) documentation provides guidance for application and links for obtaining the latest policy manifests. ### (Optional) Create Node Pools See [Node pools](/self-hosted/2.25/platform-management/aiinitiatives/resources/node-pools.md) to create and manage groups of nodes (either by predefined node label or administrator-defined node labels). This optional configuration step can be used for advanced deployment scenarios to allocate different resources across teams or projects. ### (Optional) Add Additional Users See [Users](/self-hosted/2.25/infrastructure-setup/authentication/users.md) for steps on adding additional users beyond the initially created account or configuring SSO authentication. ### (Optional) Install the NVIDIA Run:ai Command Line Tool To obtain the command line binary, see the [Install and configure CLI](/self-hosted/2.25/reference/cli/install-cli.md) section. #### Test the Command Line Tool Installation Validate the installation by running the following command: ```bash runai version ``` {% hint style="info" %} **Note** If NVIDIA Run:ai had previously been installed via BCM, it may be necessary to update the command line version. {% endhint %} #### Set the Control Plane URL The following step is required for Windows users only. Linux and Mac clients are configured via the installation script. Run the following command (substituting the NVIDIA Run:ai control plane FQDN value specified in previous steps) to create the `config.json` file in the default path: ```bash runai config set --cp-url runai.example.com ``` To validate the installation, please refer to the quick start guides for deploying single-GPU training jobs, multi-node training jobs, single-GPU inference jobs, and multi-GPU inference jobs. Certain NGC workloads may require adding an NGC API key and docker credentials into the cluster as an image pull secret; see the image pull secret instructions in [Validation tests](/self-hosted/2.25/getting-started/installation/bcm-install/validation-tests.md). 1. Validate the ingress IP for NVIDIA Run:ai inference is configured. `EXTERNAL-IP` should have the value configured in the prior MetalLB steps: ```bash kubectl get svc -n knative-serving kourier -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kourier LoadBalancer x.x.x.x 10.1.1.26 80:31038/TCP,443:30783/TCP 8h ``` 2. Validate distributed training workloads, see [Run your first distributed training workload](https://github.com/run-ai/runai-product-docs/tree/v2.25/workloads-in-nvidia-run-ai/using-training/distributed-training/quick-starts/distributed-training-quickstart.md). 3. Validate distributed inference workloads, see [Run your first custom inference workload](/self-hosted/2.25/workloads-in-nvidia-run-ai/using-inference/quick-starts/inference-quickstart.md). ## Troubleshooting Common Issues
Delayed responsiveness from the cmsh command If encountering slow response when running the `cmsh` command, try using the `cmsh-lazy-load` command (substituting it for cmsh wherever referenced in the above deployment steps). ```sh # Example: use of cmsh-lazy-load as substitute for cmsh cmsh-lazy-load -c "device list; quit" ```
Failed installation If encountering issues with installation failure (which should be evident immediately) ensure that the DGX node kernel parameters are not inadvertently forcing [Cgroup v1 vs Cgroup v2](https://kubernetes.io/blog/2024/08/14/kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode/): ```bash # the following kernel parameters should not be present systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller ```
Shared Storage (NFS) configuration If encountering issues indicating problems consistently accessing Persistent Volumes (PVs) ensure that NFSv3 for `/cm/shared` for both of the node categories that'll be used later in this guide. For example (please substitute the category name as appropriate for the DGX type): ```sh # Force NFSv3 for the worker node category cmsh -c "category use dgx-gb200-k8s; fsmounts; use /cm/shared; set mountoptions defaults,_netdev,vers=3; commit; quit" # Force NFSv3 on the CPU nodes cmsh -c "category use k8s-system-user; fsmounts; use /cm/shared; set mountoptions defaults,_netdev,vers=3; commit; quit" ```
MetalLB Load Balancer manual installation Since there's shared use of CPU nodes for the combined control plane elements in this architecture, BCM configures MetalLB and adjusts node labels to run. The following would be required as a manual step when deploying MetalLB in this manner: ```bash # Remove the exclusion preventing nodes from receiving load balancer traffic kubectl label nodes --all node.kubernetes.io/exclude-from-external-load-balancers- ``` {% hint style="info" %} **Note** The above is not required when using the BCM installation assistant. It's included here to assist with alternative deployment approaches on DGX SuperPOD / BasePOD. {% endhint %}
## Post-Installation Notes ### Adding new software images In air-gapped clusters, BCM cannot automatically provision new software images for worker nodes (this functionality requires internet access). When deploying new software images, administrators must manually install the packages into each new image by adjusting the configuration file to specify the appropriate target nodes and node categories, then running: ```bash cm-kubernetes-setup --airgap-install-system-packages -c /root/cm-kubernetes-setup.conf ``` ## Uninstallation Uninstalling Kubernetes also removes NVIDIA Run:ai. The simplest method is to run the `cm-kubernetes-setup` wizard and select **Uninstall**, or run: ```bash cm-kubernetes-setup --cluster --remove --yes-i-really-mean-it ``` To remove only NVIDIA Run:ai while leaving the Kubernetes cluster in place, use the `cm-runai-setup` wizard's **Uninstall** option, or run: ```bash cm-runai-setup --cluster --remove --yes-i-really-mean-it ``` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://run-ai-docs.nvidia.com/self-hosted/2.25/getting-started/installation/bcm-install/airgapped-deployment.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.