Ctrlk
Contact support
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Air-Gapped Deployment

This guide covers end-to-end deployment of NVIDIA Run:ai on a cluster with no internet access, including air-gapped-specific steps for preparing and transferring artifacts offline. From BCM 11.33.1, the installation is decoupled into two wizards: cm-kubernetes-setup provisions Kubernetes and the NVIDIA Run:ai dependencies, and cm-runai-setup then deploys NVIDIA Run:ai itself.

Complete the Preparations checklist before starting. All infrastructure requirements (IP addresses, DNS records, TLS certificates, credentials) are the same for both connected and air-gapped deployments.

Note

These instructions require BCM 11.33.1 or later.

Prerequisites and Requirements

  • BCM version: 11.33.1 or later on the air-gapped cluster

  • Kubernetes version: v1.34.7

  • Operating system: Ubuntu 22.04, Ubuntu 24.04, Rocky Linux 9u3, or RHEL 9u3

  • Architecture: All nodes in the cluster must share the same architecture (x86_64 or arm64). Mixed-architecture clusters require additional manual steps not covered by this guide.

The initial steps must be performed on a separate internet-connected host with:

  • The same OS and architecture as the air-gapped cluster

  • BCM package repositories configured

  • The airgap-scripts directory from BCM 11.33.1 or later

  • Internet access

Note

A BCM 11.33.1 virtual machine is the recommended internet-connected host for convenience, as it comes with the required package repositories and airgap-scripts already configured.

Prepare Air-Gapped Requirements

All steps in this section are performed on the internet-connected host.

  1. Add the airgap scripts to PATH:

    • If the cm-setup package is installed, the scripts are located at:

    • When the cm-setup module is loaded (enabled by default), this path is available via the K8S_AG_SCRIPTS environment variable:

    • If cm-setup is not available, copy the scripts to the host manually before proceeding.

  2. Install Helm to /usr/local/bin/helm:

    • Run the included helper script:

    • Verify the installation:

  3. Install Docker and skopeo. On a BCM head node:

    • Run apt update if needed, then install skopeo:

    • Run cm-docker-setup to configure Docker, then load the Docker module:

      Note

      During the cm-docker-setup installation, make sure to select the head node that will be running the Kubernetes wizard later.

    • Otherwise, follow the Docker Engine installation instructions for your OS.

    • Verify Docker is running:

    Note

    BCM uses skopeo for most image handling, but Docker is required to support multi-arch container images for Run:ai. To avoid Docker Hub pull rate limits (100 unauthenticated pulls per 6 hours), authenticate before downloading:

    Authenticated pulls are limited to 200 per 6 hours, which is sufficient for preparing the air-gapped tarball. These credentials are not included in the tarball or the air-gapped environment.

  4. Create a working directory and run the download scripts from within it:

    • Create the directory and navigate into it:

    • Run the OS-appropriate package download script, then download the container images, Helm charts, and CRDs:

    Note

    Run each script individually to verify it succeeds before proceeding to the next. All scripts write output to the current directory. Adjust --kube-version if deploying a different Kubernetes version.

    Multi-architecture clusters

    download_ubuntu_packages.sh, download_r9_packages.sh, and download_container_images.sh accept --arch to override the architecture of the host performing the download. Helm charts and CRDs are not architecture-specific. For example, to download container images for more than one architecture:

  5. Download the NVIDIA Run:ai air-gapped tarball to the same working directory. Air-gapped packages are published as an NGC Resource; browse the NGC Resource page to select the required version. This step requires an NGC Personal Key (beginning with nvapi-); see NGC Personal Keys for more information.

    • Before running the curl command, set the following variables. To avoid writing NGC_CLI_API_KEY to .bash_history, use read -rs -p "NGC API key: " NGC_CLI_API_KEY instead of a plain assignment:

    • Run the following command:

    • Verify the directory contents resemble the following:

    • Verify file integrity with sha256sum and confirm the checksum matches the value provided by NVIDIA Run:ai Support.

  6. From the parent directory, create a single archive and transfer it to the active head node of the air-gapped cluster (for example, via USB or a secure file transfer):

Generate the Deployment Configuration

All remaining steps are performed on the air-gapped cluster head node.

  1. Access the active BCM head node via ssh:

  2. Verify the BCM version:

  3. Create the configuration and validation files in /cm/shared/runai/ exactly as described in steps 3–4 of the Connected deployment guide (DGX platform configuration files, certificates, credentials, and validation test files).

  4. Start the Kubernetes wizard with the air-gap options so it pre-configures air-gapped deployment mode:

    Replace master.cm.cluster:5000 with your container registry address if different. master.cm.cluster:5000 is the default for the BCM container registry installed later in this guide.

  5. Proceed through the wizard screens exactly as described in Connected deployment (steps 6–38), with one difference: at the summary screen (step 39) select Save config & Exit to write the configuration to /root/cm-kubernetes-setup.conf.

    Note

    The setup does not execute automatically at this stage. The wizard displays the execution command, but the air-gapped package, registry, and image-push steps below must be completed before running the deployment.

Install Air-Gapped Package Requirements

  1. Extract the archive transferred from the internet-connected host:

  2. Navigate to the extracted directory:

    • On Ubuntu, move it to /tmp/airgapped first. This is required because apt runs as the _apt user, which lacks permissions to access files under /root:

    • On Rocky Linux or RHEL, change into the directory in place:

  3. Add the airgap scripts to PATH:

  4. Install the packages on the head node and into the required software images. The required software images are determined automatically from the configuration file:

    For BCM HA clusters, copy the files to the secondary head node and repeat this step, adding the --skip-headnode-check flag (a normal installation is not allowed from a secondary head node).

    Multi-architecture clusters

    The package source must include every CPU architecture referenced by software images in the Kubernetes cluster. Package installation automatically follows each software image's architecture.

Install Docker and Docker Registry

  1. Run the container registry wizard:

    • Select the active head node.

    • Ensure the container registry hostname is added, and use a custom domain name or additional Subject Alternative Name: master.cm.cluster.

    • Click Save & Deploy.

  2. On the head node, Helm must talk to this registry without TLS issues. Copy the registry CA and update the system trust store:

  3. Run the Docker wizard:

    • Select the active head node and click Save & Deploy.

  4. Verify Docker is running:

    Note

    docker login is not required in this step, as images will not be pulled from Docker Hub.

Push Container Images and Helm Charts

  1. Push all container images to the local registry:

  2. Push the Helm charts:

  3. Extract the downloaded CRD files:

    Multi-architecture clusters

    All images for the architectures you downloaded are pushed to the registry, irrespective of the cluster's own architecture.

Deploy Kubernetes

With the packages installed, the registry populated, and the CRDs extracted, deploy Kubernetes from the saved configuration. Use screen or tmux to prevent interruptions:

Note

During the deployment process all nodes that are members of the new Kubernetes cluster will be rebooted.

Deploy NVIDIA Run:ai Using the Wizard

Once cm-kubernetes-setup has completed, deploy NVIDIA Run:ai with the dedicated cm-runai-setup wizard. This wizard is provided by the cm-setup-runai-bcm11 package, which is pre-installed on the BCM head node(s).

  1. Run the wizard with --skip-packages (required for air-gapped deployments so the wizard does not attempt to auto-update over the internet):

  2. Select Deploy and click Ok to proceed:

Kubernetes operations

  1. Select the self-hosted NVIDIA Run:ai deployment mode and click Ok to proceed:

Kubernetes operations

  1. The wizard detects that it is deploying onto an air-gapped Kubernetes cluster and prompts for the NVIDIA Run:ai air-gapped tarball location. Enter the path to runai-airgapped-package-2.25.x.tar.gz and click Ok to proceed:

Kubernetes operations

  1. Provide the NVIDIA Run:ai configuration (registry credentials, control plane FQDN, and certificate/key paths) as described in step 4 of the Connected deployment cm-runai-setup section. Click Ok to proceed:

Kubernetes operations

Note

It's recommended to save all certificates, configuration files, and deployment artifacts into a persistent and accessible location in case of redeployment. The /cm/shared/runai/ directory referred to in this guide resides on a shared mount point and would be a suitable location. See the TLS certificates section for additional clarification.

  1. Select yes to deploy the NVIDIA Run:ai cluster. Click Ok to proceed:

Kubernetes operations

  1. Select the node category for the NVIDIA Run:ai control plane nodes. Select k8s-system-user, the same category used for the Kubernetes control plane nodes. Click Ok to proceed:

Kubernetes operations

  1. Select the required NVIDIA Run:ai version (v2.25.x). The x is replaced with the latest patch version available for the chosen minor version. Click Ok to proceed:

Kubernetes operations

  1. The wizard validates the control plane DNS configuration. Confirm the DNS check passes and click Ok to proceed:

Kubernetes operations

  1. Specify the load balancer IP address for NVIDIA Run:ai inference (Kourier), prepared as documented in the Pre-installation checklist section. Click Ok to proceed:

Kubernetes operations

  1. Select Save & Exit at the summary screen, then click Ok.

Kubernetes operations

Note

In BCM 11.33.1, always select Save & Exit here. A regression currently prevents going through the questions and continuing to the deployment in a single run.

  1. Run the deployment from the saved configuration (keep --skip-packages for air-gapped):

When the setup completes, NVIDIA Run:ai should be up and running:

Connect to NVIDIA Run:ai User Interface

Upon completion of cm-runai-setup, access NVIDIA Run:ai at the ingress IP or hostname specified earlier (e.g. runai.example.com). The NVIDIA Run:ai credentials required for login were entered down in step 5:

On first access, administrators are presented with an optional onboarding wizard that helps with initial setup tasks. The onboarding wizard can guide you through:

  • Configuring single sign-on (SSO)

  • Inviting the first research team

You can choose to complete or skip the onboarding wizard and perform these actions later.

After the BCM installation assistant completes, additional steps are required.

If multiple Kubernetes clusters are configured in this instance of BCM, load the correct Kubernetes module before running all post-wizard commands:

NVIDIA Dynamic Resource Allocation (DRA) Driver

The NVIDIA DRA Driver for GPUs extends how NVIDIA GPUs are consumed within Kubernetes. This is required to enable secure Internode Memory Exchange (IMEX) on Multi-Node NVLink (MNNVL) systems (e.g. GB200, GB300) for Kubernetes workloads and should be included with all NVIDIA GPU systems.

  1. Install using Helm:

NGC image pull secret (bandwidth tests only)

The bandwidth tests — dra-computedomain-test.yaml and ib-bandwidth-test.yaml — pull the entitled nvcr.io/nvidia/nv-mission-control/nvbandwidth image and reference an image pull secret named ngc-nvcr. Before applying either, create that secret in the default namespace (replace <NGC_API_KEY> with a valid NGC key):

The NCCL tests (ib-nccl-test.yaml, roce-nccl-test.yaml) use the public nvcr.io/nvidia/pytorch image and do not need this secret.

  1. Multi-Node NVLink (MNNVL) platforms (e.g. GB200, GB300) only - Create a dra-computedomain-test.yaml file in /cm/shared/runai from the Validation tests. The test co-locates its worker pods within a single NVL clique automatically via podAffinity with topologyKey: nvidia.com/gpu.clique, so there is no clique ID to edit for a single-rack NVL72 cluster. For multi-rack systems, adjust the podAffinity topologyKey to match your topology:

  2. Multi-Node NVLink (MNNVL) platforms (e.g. GB200, GB300) only - Validate the test successfully completed and inspect the logs of the launcher:

  3. Multi-Node NVLink (MNNVL) platforms (e.g. GB200, GB300) only - Cleanup test:

Enable DRA and Multi-Node NVLink

The default NVIDIA Run:ai configuration does not expose DRA features. After installing the DRA components, this can be enabled by modifying the runaiconfig in the cluster. See Advanced cluster configurations for more details:

Instructions for validating the change and reverting if necessary:

Configure the Network Operator

In version 11.33.1 of the BCM installation assistant, the Network Operator requires additional configuration on DGX B200 / GB200 & B300 / GB300 SuperPOD / BasePOD systems. While the operator is installed in a preceding step, it does not automatically initialize or configure SR-IOV and secondary network plugins.

The following CRD resources have to be created in the exact order as below:

  • SR-IOV Network Policies for each NVIDIA InfiniBand NIC

  • An nvIPAM IP address pool

  • SR-IOV InfiniBand networks

  1. Create SR-IOV network node policies using the nic-cluster-policy.yaml that was created in an earlier step:

  2. Create an IPAM IP Pool using the combined-ippools.yaml that was created in an earlier step:

  3. Create the SR-IOV IB networks using the combined-sriovbnet.yaml that was created in an earlier step:

  4. Create the SR-IOV Node Pool configuration using the sriov-node-pool-config.yaml appropriate for the DGX platform:

Note

This will typically reconfigure NICs and may result in a node reboot. The supplied YAML sets the maxUnavailable field to 20%. This value should be adjusted to align with your operational requirements. A value of 1 would have the effect of serializing the upgrade and would result in blocking upon a single node failure. It may be appropriate for a small lab deployment to set it to 100%. This would prevent any single machine failure from blocking the remaining nodes from upgrading. For larger clusters, setting the value to a lower percentage means that the upgrade process will be effectively split into batches.

  1. Validate by describing one of the DGX nodes and checking for SRIOV devices:

Note

It might take several minutes for these settings to take effect. If the sriovnetworkconfig daemon changes the NIC config, then a node reboot will occur.

  1. Create the test file matching your platform's fabric in /cm/shared/runai from the Validation tests page, then validate by running it. Apply only the test that matches the cluster's fabric (InfiniBand or Spectrum-X / RoCE), not both:

Spectrum-X (RoCE) deployments

The InfiniBand tests above target the InfiniBand fabric. DGX systems configured for NVIDIA Spectrum-X Ethernet (RoCE) — for example DGX B300 SuperPOD deployments using Spectrum-X — use a different fabric: run the roce-nccl-test.yaml test (the Spectrum-X bullet above) instead of the InfiniBand tests. Do not run the InfiniBand tests on a Spectrum-X / RoCE-configured cluster, and do not run the RoCE test on an InfiniBand cluster.

Note

The Network Operator will restart the DGX nodes if the number of Virtual Functions in the SR-IOV Network Policy file does not match the NVIDIA/Mellanox firmware configuration.

(Optional) Apply Security Policies

By default, BCM Kubernetes deployment has permissive security policies to ease in development environments. For production clusters or in secure environments, it's recommended to take additional steps to harden the cluster. This includes steps such as configuring permission manager, applying Kyverno policies, and applying Calico policies.

For deployments of NVIDIA Run:ai as a part of NVIDIA Mission Control, please reach out to your NVIDIA representative for the latest example configurations and suggested policies. The Mission Control software installation guide's Kubernetes Security Hardening documentation provides guidance for application and links for obtaining the latest policy manifests.

(Optional) Create Node Pools

See Node pools to create and manage groups of nodes (either by predefined node label or administrator-defined node labels). This optional configuration step can be used for advanced deployment scenarios to allocate different resources across teams or projects.

(Optional) Add Additional Users

See Users for steps on adding additional users beyond the initially created account or configuring SSO authentication.

(Optional) Install the NVIDIA Run:ai Command Line Tool

To obtain the command line binary, see the Install and configure CLI section.

Test the Command Line Tool Installation

Validate the installation by running the following command:

Note

If NVIDIA Run:ai had previously been installed via BCM, it may be necessary to update the command line version.

Set the Control Plane URL

The following step is required for Windows users only. Linux and Mac clients are configured via the installation script.

Run the following command (substituting the NVIDIA Run:ai control plane FQDN value specified in previous steps) to create the config.json file in the default path:

To validate the installation, please refer to the quick start guides for deploying single-GPU training jobs, multi-node training jobs, single-GPU inference jobs, and multi-GPU inference jobs. Certain NGC workloads may require adding an NGC API key and docker credentials into the cluster as an image pull secret; see the image pull secret instructions in Validation tests.

  1. Validate the ingress IP for NVIDIA Run:ai inference is configured. EXTERNAL-IP should have the value configured in the prior MetalLB steps:

  2. Validate distributed training workloads, see Run your first distributed training workload.

  3. Validate distributed inference workloads, see Run your first custom inference workload.

Troubleshooting Common Issues

Delayed responsiveness from the cmsh command

If encountering slow response when running the cmsh command, try using the cmsh-lazy-load command (substituting it for cmsh wherever referenced in the above deployment steps).

Failed installation

If encountering issues with installation failure (which should be evident immediately) ensure that the DGX node kernel parameters are not inadvertently forcing Cgroup v1 vs Cgroup v2:

Shared Storage (NFS) configuration

If encountering issues indicating problems consistently accessing Persistent Volumes (PVs) ensure that NFSv3 for /cm/shared for both of the node categories that'll be used later in this guide. For example (please substitute the category name as appropriate for the DGX type):

MetalLB Load Balancer manual installation

Since there's shared use of CPU nodes for the combined control plane elements in this architecture, BCM configures MetalLB and adjusts node labels to run. The following would be required as a manual step when deploying MetalLB in this manner:

Note

The above is not required when using the BCM installation assistant. It's included here to assist with alternative deployment approaches on DGX SuperPOD / BasePOD.

Post-Installation Notes

Adding new software images

In air-gapped clusters, BCM cannot automatically provision new software images for worker nodes (this functionality requires internet access). When deploying new software images, administrators must manually install the packages into each new image by adjusting the configuration file to specify the appropriate target nodes and node categories, then running:

Uninstallation

Uninstalling Kubernetes also removes NVIDIA Run:ai. The simplest method is to run the cm-kubernetes-setup wizard and select Uninstall, or run:

To remove only NVIDIA Run:ai while leaving the Kubernetes cluster in place, use the cm-runai-setup wizard's Uninstall option, or run:

PreviousConnected DeploymentNextAuthentication and Authorization

Last updated