Upgrade

Before Upgrade

Before proceeding with the upgrade, it's crucial to apply the specific prerequisites associated with your current version of NVIDIA Run:ai and every version in between up to the version you are upgrading to.

Helm

NVIDIA Run:ai requires Helm 3.14 or later. Before you continue, validate your installed helm client version. To install or upgrade Helm, see Installing Helm. If you are installing an air-gapped version of NVIDIA Run:ai, the NVIDIA Run:ai tar file contains the helm binary.

Software Files

Connected

Run the helm command below:

helm repo add runai-backend https://runai.jfrog.io/artifactory/cp-charts-prod
helm repo update

Air-gapped

• Ask for a tar file runai-air-gapped-<NEW-VERSION>.tar.gz from NVIDIA Run:ai customer support. The file contains the new version you want to upgrade to. <NEW-VERSION> is the updated version of the NVIDIA Run:ai control plane.

• Upload the images as described here.

Upgrade Control Plane

System and Network Requirements

Before upgrading the NVIDIA Run:ai control plane, validate that the latest system requirements and network requirements are met, as they can change from time to time.

Upgrade

Note

To upgrade to a specific version, modify the --version flag by specifying the desired <VERSION>. You can find all available versions by using the helm search repo runai-backend/control-plane --versions command.

Upgrading from Version 2.16

You must perform a two-step upgrade:

1. Upgrade to version 2.18:

Connected

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml
helm upgrade runai-backend -n runai-backend runai-backend/control-plane --version "2.18.0" -f runai_control_plane_values.yaml --reset-then-reuse-values

Air-gapped

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml
helm upgrade runai-backend control-plane-2.18.0.tgz -n runai-backend -f runai_control_plane_values.yaml --reset-then-reuse-values

1. Then upgrade to the required version:

Connected

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml
helm upgrade runai-backend -n runai-backend runai-backend/control-plane --version "<VERSION>" -f runai_control_plane_values.yaml --reset-then-reuse-values

Air-gapped

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml
helm upgrade runai-backend control-plane-<NEW-VERSION>.tgz -n runai-backend -f runai_control_plane_values.yaml --reset-then-reuse-values

Upgrading from Version 2.17 or Later

If your current version is 2.17 or higher, you can upgrade directly to the required version:

Connected

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml
helm upgrade runai-backend -n runai-backend runai-backend/control-plane --version "<VERSION>" -f runai_control_plane_values.yaml --reset-then-reuse-values

Air-gapped

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml
helm upgrade runai-backend control-plane-<NEW-VERSION>.tgz -n runai-backend -f runai_control_plane_values.yaml --reset-then-reuse-values

Upgrade Cluster

System and Network Requirements

Before upgrading the NVIDIA Run:ai cluster, validate that the latest system requirements and network requirements are met, as they can change from time to time.

Note

It is highly recommended to upgrade the Kubernetes version together with the NVIDIA Run:ai cluster version, to ensure compatibility with latest supported version of your Kubernetes distribution.

Getting Installation Instructions

Follow the setup and installation instructions below to get the installation instructions to upgrade the NVIDIA Run:ai cluster.

Note

To upgrade to a specific version, modify the --version flag by specifying the desired <VERSION>. You can find all available versions by using the helm search repo runai/runai-cluster --versions command.

Setup

1. In the NVIDIA Run:ai UI, go to Clusters

2. Select the cluster you want to upgrade

3. Click INSTALLATION INSTRUCTIONS

4. Optional: Select the NVIDIA Run:ai cluster version (latest, by default)

5. Click CONTINUE

Installation Instructions

1. Follow the installation instructions. Run the Helm commands provided on your Kubernetes cluster. See the below if installation fails.

2. Click DONE

3. Once installation is complete, validate the cluster is Connected and listed with the new cluster version (see the cluster troubleshooting scenarios). Once you have done this, the cluster is upgraded to the latest version.

Troubleshooting

If you encounter an issue with the cluster upgrade, use the troubleshooting scenarios below.

Installation Fails

If the NVIDIA Run:ai cluster upgrade fails, check the installation logs to identify the issue.

Run the following script to print the installation logs:

curl -fsSL https://raw.githubusercontent.com/run-ai/public/main/installation/get-installation-logs.sh

Cluster Status

If the NVIDIA Run:ai cluster upgrade completes, but the cluster status does not show as Connected, refer to the cluster troubleshooting scenarios.