This section includes release information for the self-hosted version of NVIDIA Run:ai:

• New Features and Enhancements – Highlights major updates introduced in each version, including new capabilities, UI improvements, and changes to system behavior..

• Hotfixes – Lists patches applied to released versions, including critical fixes and behavior corrections.

Feature Life Cycle

NVIDIA Run:ai uses life cycle labels to indicate the maturity and stability of features across releases:

• Experimental - This feature is in early development. It may not be stable and could be removed or changed significantly in future versions. Use with caution.

• Beta - This feature is still being developed for official release in a future version and may have some limitations. Use with caution.

• Legacy - This feature is scheduled to be removed in future versions. We recommend using alternatives if available. Use only if necessary.

Uninstall the Control Plane

To delete the control plane, run:

helm uninstall runai-backend -n runai-backend

Uninstall the Cluster

NVIDIA Run:ai Components

As part of the installation process, you will install:

• A managing cluster/s

• One or more

Both the control plane and clusters require Kubernetes. Typically, the control plane and first cluster are installed on the same Kubernetes cluster.

Installation Types

The self-hosted option is for organizations that cannot use a SaaS solution due to data leakage concerns. NVIDIA Run:ai self-hosting comes with two variants:

NVIDIA Run:ai authentication and authorization enables a streamlined experience for the user with precise controls covering the data each user can see and the actions each user can perform in the NVIDIA Run:ai platform.

Authentication verifies user identity during login, and authorization assigns the user with specific permissions according to the assigned .

Authenticated access is required to use all aspects of the NVIDIA Run:ai interfaces, including the NVIDIA Run:ai platform, the NVIDIA Run:ai Command Line Interface (CLI) and APIs.

Authentication

There are multiple methods to authenticate and access NVIDIA Run:ai.

Single Sign-On (SSO)

NVIDIA Run:ai supports three methods to set up SSO:

When using SSO, it is highly recommended to manage at least one local user, as a breakglass account (an emergency account), in case access to SSO is not possible.

Username and Password

Username and password access can be used when SSO integration is not possible.

Secret Key (for Application Programmatic Access)

Secret is the authentication method for . Applications use the NVIDIA Run:ai APIs to perform automated tasks including scripts and pipelines based on their assigned .

Authorization

The NVIDIA Run:ai platform uses Role Base Access Control (RBAC) to manage authorization. Once a user or an application is authenticated, they can perform actions according to their assigned access rules.

Role Based Access Control (RBAC) in NVIDIA Run:ai

While Kubernetes RBAC is limited to a single cluster, NVIDIA Run:ai expands the scope of Kubernetes RBAC, making it easy for administrators to manage access rules across multiple clusters.

RBAC at NVIDIA Run:ai is configured using access rules. An access rule is the assignment of a to a : <Subject> is a <Role> in a <Scope>.

• Subject

  • A user, a group, or an application assigned with the role

• Role

  • A set of permissions that can be assigned to subjects. Roles at NVIDIA Run:ai are system defined and cannot be created, edited or deleted.

  • A permission is a set of actions (view, edit, create and delete) over a NVIDIA Run:ai entity (e.g. projects, workloads, users). For example, a role might allow a user to create and read Projects, but not update or delete them

• Scope

  • A scope is part of an organization in which a set of permissions (roles) is effective. Scopes include Projects, Departments, Clusters, Account (all clusters).

Below is an example of an access rule: [email protected] is a Department admin in Department: A

This article explains the procedure to create your own user applications.

Applications are used for API integrations with NVIDIA Run:ai. An application contains a client ID and a client secret. With the client credentials, you can obtain a token as detailed in and use it within subsequent API calls.

Creating an Application

To create an application:

1. Click the user avatar at the top right corner, then select Settings

2. Click +APPLICATION

3. Enter the application’s name

4. Click CREATE

5. Copy the Client ID and Client secret and store securely

6. Click DONE

You can create up to 20 user applications.

Regenerating a Client Secret

To regenerate a client secret:

1. Locate the application you want to regenerate its client secret

2. Click Regenerate client secret

3. Click REGENERATE

4. Copy the New client secret and store it securely

5. Click DONE

Deleting an Application

1. Locate the application you want to delete

2. Click on the trash icon

3. On the dialog, click DELETE to confirm

Using API

Go to the API reference to view the available actions.

NVIDIA Run:ai supports service mesh implementations. When a service mesh is deployed with sidecar injection, specific configurations must be applied to ensure compatibility with NVIDIA Run:ai. This document outlines the required changes for the NVIDIA Run:ai control plane and cluster.

Control Plane Configuration

By default, NVIDIA Run:ai prevents Istio from injecting sidecar containers into system jobs in the control plane. For other service mesh solutions, users must manually add annotations during installation.

To disable sidecar injection in the NVIDIA Run:ai control plane, modify the Helm values file by adding the required pod labels to the following components. See for more details.

Example for :

Cluster Configuration

Installation Phase

Sidecar containers injected by some service mesh solutions can prevent NVIDIA Run:ai installation hooks from completing. To avoid this, modify the Helm installation command to include the required labels or annotations:

Example for :

Workloads

To prevent sidecar injection in workloads created at runtime (such as training workloads), update the runaiconfig resource. See for more details:

Deploying NVIDIA Run:ai in mission-critical environments requires proper monitoring and maintenance of resources to ensure workloads run and are deployed as expected.

Details on how to monitor different parts of the physical resources in your Kubernetes system, including and , can be found in the monitoring and maintenance section. Adjacent configuration and troubleshooting sections also cover high availability, and clusters, , and to meet compliance requirements.

In addition to monitoring NVIDIA Run:ai resources, it is also highly recommended to monitor NVIDIA Run:ai runs on Kubernetes, which manages containerized applications. In particular, focus on three main layers:

NVIDIA Run:ai Control Plane and Cluster Services

This is the highest layer and includes the parts of NVIDIA Run:ai pods, which run in containers managed by Kubernetes.

Kubernetes Cluster

This layer includes the main Kubernetes system that runs and manages NVIDIA Run:ai components. Important elements to monitor include:

• The health of the cluster and nodes (machines in the cluster).

• The status of key Kubernetes services, such as the API server. For detailed information on managing clusters, see the .

Host Infrastructure

This is the base layer, representing the actual machines (virtual or physical) that make up the cluster IT teams need to handle:

• Managing CPU, memory, and storage

• Keeping the operating system updated

• Setting up the network and balancing the load

NVIDIA Run:ai does not require any special configurations at this level.

The articles below explain how to monitor these layers, maintain system security and compliance, and ensure the reliable operation of NVIDIA Run:ai in critical environments.

Shared storage is a critical component in AI and machine learning workflows, particularly in scenarios involving distributed training and shared datasets. In AI and ML environments, data must be readily accessible across multiple nodes, especially when training large models or working with vast datasets. Shared storage enable seamless access to data, ensuring that all nodes in a distributed training setup can read and write to the same datasets simultaneously. This setup not only enhances efficiency but is also crucial for maintaining consistency and speed in high-performance computing environments.

While NVIDIA Run:ai Platform supports a variety of remote data sources, such as Git and S3, it is often more efficient to keep data close to the compute resources. This proximity is typically achieved through the use of shared storage, accessible to multiple nodes in your Kubernetes cluster.

Shared Storage

When implementing shared storage in Kubernetes, there are two primary approaches:

• Utilizing the of your storage provider (Recommended)

• Using a direct NFS (Network File System) mount

NVIDIA Run:ai support both direct NFS mount and Kubernetes Storage Classes.

Kubernetes Storage Classes

Storage classes in Kubernetes defines how storage is provisioned and managed. This allows you to select storage types optimized for AI workloads. For example, you can choose storage with high IOPS (Input/Output Operations Per Second) for rapid data access during intensive training sessions, or tiered storage options to balance cost and performance-based on your organization’s requirements. This approach supports dynamic provisioning, enabling storage to be allocated on-demand as required by your applications.

NVIDIA Run:ai data sources such as and leverage storage class to manage and allocate storage efficiently. This ensures that the most suitable storage option is always accessible, contributing to the efficiency and performance of AI workloads.

Direct NFS Mount

Direct NFS allows you to mount a shared file system directly across multiple nodes in your Kubernetes cluster. This method provides a straightforward way to share data among nodes and is often used for simple setups or when a dedicated NFS server is available.

However, using NFS can present challenges related to security and control. Direct NFS setups might lack the fine-grained control and security features available with storage class.

This section explains how to restore a NVIDIA Run:ai cluster on a different Kubernetes environment.

In the event of a critical Kubernetes failure or alternatively, if you want to migrate a NVIDIA Run:ai cluster to a new Kubernetes environment, simply reinstall the NVIDIA Run:ai cluster. Once you have reinstalled and reconnected the cluster - projects, workloads and other cluster data is synced automatically.

The restoration or back-up of NVIDIA Run:ai cluster and which are stored locally on the Kubernetes cluster is optional and they can be restored and backed-up separately.

Backup

As back-up of data is not required, the backup procedure is optional for advanced deployments, as explained above.

Backup Cluster Configurations

To backup NVIDIA Run:ai cluster configurations:

1. Run the following command in your terminal:

2. Once the runaiconfig_back.yaml back-up file is created, save the file externally, so that it can be retrieved later.

Restore

Follow the steps below to restore the NVIDIA Run:ai cluster on a new Kubernetes environment.

Prerequisites

Before restoring the NVIDIA Run:ai cluster, it is essential to validate that it is both disconnected and uninstalled.

1. If the Kubernetes cluster is still available, the NVIDIA Run:ai cluster - make sure not to remove the cluster from the Control Plane

2. Navigate to the Cluster page in the NVIDIA Run:ai platform

3. Search for the cluster, and make sure its status is Disconnected

Re-installing NVIDIA Run:ai Cluster

1. Follow the NVIDIA Run:ai cluster instructions and ensure all prerequisites are met

2. If you have a back-up of the cluster configurations, reload it once the installation is complete

3. Navigate to the Cluster page in the NVIDIA Run:ai platform

4. Search for the cluster, and make sure its status is Connected

Karpenter is an open-source, Kubernetes cluster autoscaler built for cloud deployments. Karpenter optimizes the cloud cost of a customer’s cluster by moving workloads between different node types, consolidating workloads into fewer nodes, using lower-cost nodes where possible, scaling up new nodes when needed, and shutting down unused nodes.

Karpenter’s main goal is cost optimization. Unlike Karpenter, NVIDIA Run:ai’s Scheduler optimizes for fairness and resource utilization. Therefore, there are a few potential friction points when using both on the same cluster.

Friction Points Using Karpenter with NVIDIA Run:ai

1. Karpenter looks for “unschedulable” pending workloads and may try to scale up new nodes to make those workloads schedulable. However, in some scenarios, these workloads may exceed their quota parameters, and the NVIDIA Run:ai Scheduler will put them into a pending state.

2. Karpenter is not aware of the NVIDIA Run:ai fractions mechanism and may try to interfere incorrectly.

3. Karpenter preempts any type of workload (i.e., high-priority, non-preemptible workloads will potentially be interrupted and moved to save cost).

4. Karpenter has no pod-group (i.e., workload) notion or gang scheduling awareness, meaning that Karpenter is unaware that a set of “arbitrary” pods is a single workload. This may cause Karpenter to schedule those pods into different node pools (in the case of multi-node-pool workloads) or scale up or down a mix of wrong nodes.

Mitigating the Friction Points

NVIDIA Run:ai Scheduler mitigates the friction points using the following techniques (each numbered bullet below corresponds to the related friction point listed above):

1. Karpenter uses a “nominated node” to recommend a node for the Scheduler. The NVIDIA Run:ai Scheduler treats this as a “preferred” recommendation, meaning it will try to use this node, but it’s not required and it may choose another node.

2. Fractions - Karpenter won’t consolidate nodes with one or more pods that cannot be moved. The NVIDIA Run:ai reservation pod is marked as ‘do not evict’ to allow the NVIDIA Run:ai Scheduler to control the scheduling of fractions.

3. Non-preemptible workloads - NVIDIA Run:ai marks non-preemptible workloads as ‘do not evict’ and Karpenter respects this annotation.

4. NVIDIA Run:ai node pools (single-node-pool workloads) - Karpenter respects the ‘node affinity’ that NVIDIA Run:ai sets on a pod, so Karpenter uses the node affinity for its recommended node. For the gang-scheduling/pod-group (workload) notion, NVIDIA Run:ai Scheduler considers Karpenter directives as preferred recommendations rather than mandatory instructions and overrides Karpenter instructions where appropriate.

Deployment Considerations

• Using multi-node-pool workloads

  • Workloads may include a list of optional node pools. Karpenter is not aware that only a single node pool should be selected out of that list for the workload. It may therefore recommend putting pods of the same workload into different node pools and may scale up nodes from different node pools to serve a “multi-node-pool” workload instead of nodes on the selected single node pool.

  • If this becomes an issue (i.e., if Karpenter scales up the wrong node types), users can set an inter-pod affinity using the node pool label or another common label as a ‘topology’ identifier. This will force Karpenter to choose nodes from a single-node pool per workload, selecting from any of the node pools listed as allowed by the workload.

  • An alternative approach is to use a single-node pool for each workload instead of multi-node pools.

• Consolidation

  • To make Karpenter more effective when using its consolidation function, users should consider separating preemptible and non-preemptible workloads, either by using node pools, node affinities, taint/tolerations, or inter-pod anti-affinity.

  • If users don’t separate preemptible and non-preemptible workloads (i.e., make them run on different nodes), Karpenter’s ability to consolidate (bin-pack) and shut down nodes will be reduced, but it is still effective.

• Conflicts between bin-packing and spread policies

  • If NVIDIA Run:ai is used with a scheduling spread policy, it will clash with Karpenter’s default bin-packs/consolidation policy, and the outcome may be a deployment that is not optimized for any of these policies.

  • Usually spread is used for Inference, which is non-preemptible and therefore not controlled by Karpenter (NVIDIA Run:ai Scheduler will mark those workloads as ‘do not evict’ for Karpenter), so this should not present a real deployment issue for customers.

This section details the security considerations for deploying NVIDIA Run:ai. It is intended to help administrators and security officers understand the specific permissions required by NVIDIA Run:ai.

Access to the Kubernetes Cluster

NVIDIA Run:ai integrates with Kubernetes clusters and requires specific permissions to successfully operate. These are permissions are controlled with configuration flags that dictate how NVIDIA Run:ai interacts with cluster resources. Prior to installation, security teams can review the permissions and ensure it aligns with their organization’s policies.

Permissions and their Related Use Case

NVIDIA Run:ai provides various security-related permissions that can be customized to fit specific organizational needs. Below are brief descriptions of the key use cases for these customizations:

Secure Installation

Many organizations enforce IT compliance rules for Kubernetes, with strict access control for installing and running workloads. OpenShift uses Security Context Constraints (SCC) for this purpose. NVIDIA Run:ai fully supports SCC, ensuring integration with OpenShift's security requirements.

Security Vulnerabilities

The platform is actively monitored for security vulnerabilities, with regular scans conducted to identify and address potential issues. Necessary fixes are applied to ensure that the software remains secure and resilient against emerging threats, providing a safe and reliable experience.

Operating NVIDIA Run:ai at scale ensures that the system can efficiently handle fluctuating workloads while maintaining optimal performance. As clusters grow—whether due to an increasing number of nodes or a surge in workload demand—NVIDIA Run:ai services must be appropriately tuned to support large-scale environments.

This guide outlines the best practices for optimizing NVIDIA Run:ai for high-performance deployments, including NVIDIA Run:ai system services configurations, vertical scaling (adjusting CPU and memory resources) and where applicable, horizontal scaling (replicas).

NVIDIA Run:ai Services

Vertical Scaling

Each of the NVIDIA Run:ai containers has default resource requirements that reflect an average customer load. With significantly larger cluster loads, certain NVIDIA Run:ai services will require more CPU and memory resources. NVIDIA Run:ai supports configuring these resources for each NVIDIA Run:ai service group separately. For instructions and more information, see NVIDIA Run:ai services resource management.

Scheduling Services

The scheduling services group should be scaled together with the number of nodes and the number of workloads handled by the Scheduler (running / pending). These resource recommendations are based on internal benchmarks performed on stressed environments:

Sync and Workload Services

The sync and workload service groups are less sensitive for scale. The recommendation for large or intensive environments is set to the following:

Horizontal Scaling

By default, NVIDIA Run:ai cluster services are deployed with a single replica. For large scale and intensive environments it is recommended to scale the NVIDIA Run:ai services horizontally by increasing the number of replicas. For more information, see NVIDIA Run:ai services replicas.

Metrics Collection

NVIDIA Run:ai relies on Prometheus to scrape cluster metrics and forward them to the NVIDIA Run:ai control plane. The volume of metrics generated is directly proportional to the number of nodes, workloads, and projects in the system. When operating at scale—reaching hundreds, and thousands of nodes and projects—the system generates a significant volume of metrics which can place a strain on the cluster and the network bandwidth.

To mitigate this impact, it is recommended to tune the Prometheus remote-write configurations. See remote write tuning to read more about the tuning parameters available via the remote write configuration and refer to this article for optimizing Prometheus remote write performance.

You can apply the remote-write configurations required as described in advanced cluster configurations.

The following example demonstrates the recommended approach in NVIDIA Run:ai for tuning Prometheus remote-write configurations:

remoteWrite:
  queueConfig:
    capacity: 5000
    maxSamplesPerSend: 1000
    maxShards: 100

NVIDIA Run:ai is a GPU orchestration and optimization platform that helps organizations maximize compute utilization for AI workloads. By optimizing the use of expensive compute resources, NVIDIA Run:ai accelerates AI development cycles, and drives faster time-to-market for AI-powered innovations.

Built on Kubernetes, NVIDIA Run:ai supports dynamic GPU allocation, workload submission, workload scheduling, and resource sharing, ensuring that AI teams get the compute power they need while IT teams maintain control over infrastructure efficiency.

How NVIDIA Run:ai Helps Your Organization

For Infrastructure Administrators

NVIDIA Run:ai centralizes cluster management and optimizes infrastructure control by offering:

• Centralized cluster management – Manage all clusters from a single platform, ensuring consistency and control across environments.

• Usage monitoring and capacity planning – Gain real-time and historical insights into GPU consumption across clusters to optimize resource allocation and plan future capacity needs efficiently.

• Policy enforcement – Define and enforce security and usage policies to align GPU consumption with business and compliance requirements.

• Enterprise-grade authentication – Integrate with your organization's identity provider for streamlined authentication (Single Sign On) and role-based access control (RBAC).

• Kubernetes-native application – Install as a Kubernetes-native application, seamlessly extending Kubernetes for native cloud experience and operational standards (install, upgrade, configure).

For Platform Administrators

NVIDIA Run:ai simplifies AI infrastructure management by providing a structured approach to managing AI initiatives, resources, and user access. It enables platform administrators maintain control, efficiency, and scalability across their infrastructure:

• AI Initiative structuring and management – Map and set up AI initiatives according to your organization's structure, ensuring clear resource allocation.

• Centralized GPU resource management – Enable seamless sharing and pooling of GPUs across multiple users, reducing idle time and optimizing utilization.

• User and access control – Assign users (AI practitioners, ML engineers) to specific projects and departments to manage access and enforce security policies, utilizing role-based access control (RBAC) to ensure permissions align with user roles.

• Workload scheduling – Use scheduling to prioritize and allocate GPUs based on workload needs.

• Monitoring and insights – Track real-time and historical data on GPU usage to help track resource consumption and optimize costs.

For AI Practitioners

NVIDIA Run:ai empowers data scientists and ML engineers by providing:

• Optimized workload scheduling – Ensure high-priority jobs get GPU resources. Workloads dynamically receive resources based on demand.

• Fractional GPU usage – Request and utilize only a fraction of a GPU's memory, ensuring efficient resource allocation and leaving room for other workloads.

• AI initiatives lifecycle support – Run your entire AI initiatives lifecycle – Jupyter Notebooks, training jobs, and inference workloads efficiently.

• Interactive session – Ensure an uninterrupted experience when working on Jupyter Notebooks without taking away GPUs.

• Scalability for training and inference – Support for distributed training across multiple GPUs and auto-scales inference workloads.

• Integrations – Integrate with popular ML frameworks - PyTorch, TensorFlow, XGBoost, Knative, Spark, Kubeflow Pipelines, Apache Airflow, Argo workloads, Ray and more.

• Flexible workload submission – Submit workloads using the NVIDIA Run:ai UI, API, CLI or run third-party workloads.

NVIDIA Run:ai System Components

NVIDIA Run:ai is made up of two components both installed over a Kubernetes cluster:

• NVIDIA Run:ai cluster – Provides scheduling and workload management, extending Kubernetes native capabilities.

• NVIDIA Run:ai control plane – Provides resource management, handles workload submission and provides cluster monitoring and analytics.

NVIDIA Run:ai Cluster

The NVIDIA Run:ai cluster is responsible for scheduling AI workloads and efficiently allocating GPU resources across users and projects:

• NVIDIA Run:ai Scheduler – Applies AI-aware rules to efficiently schedule workloads submitted by AI practitioners.

• Workload management – Handles workload management which includes the researcher code running as a Kubernetes container and the system resources required to run the code, such as storage, credentials, network endpoints to access the container and so on.

• Kubernetes operator-based deployment – Installed as a Kubernetes Operator to automate deployment, upgrades and configuration of NVIDIA Run:ai cluster services.

• Storage – Supports Kubernetes-native storage using Storage Classes, allowing organizations to bring their own storage solutions. Additionally, it also integrates with external storage solutions such as Git, S3, and NFS to support various data requirements.

• Secured communication – Uses an outbound-only, secured (SSL) connection to synchronize with the NVIDIA Run:ai control plane.

• Private – NVIDIA Run:ai only synchronizes metadata and operational metrics (e.g., workloads, nodes) with the control plane. No proprietary data, model artifacts, or user data sets are ever transmitted, ensuring full data privacy and security.

NVIDIA Run:ai Control Plane

The NVIDIA Run:ai control plane provides a centralized management interface for organizations to oversee their GPU infrastructure across multiple locations/subnets, accessible via Web UI, API and CLI. The control plane can be deployed on the cloud or on-premise for organizations that require local control over their infrastructure (self-hosted).

• Multi-cluster management – Manages multiple NVIDIA Run:ai clusters for a single tenant across different locations and subnets from a single unified interface.

• Resource and access management – Allows administrators to define Projects, Departments and user roles, enforcing policies for fair resource distribution.

• Workload submission and monitoring – Allows teams to submit workloads, track usage, and monitor GPU performance in real time.

Installation Types

There are two main installation options:

This section explains the available configurations for customizing the NVIDIA Run:ai control plane and cluster installation.

Control Plane Helm Chart Values

The NVIDIA Run:ai control plane installation can be customized to support your environment via Helm values files or Helm install flags. See Advanced control plane configurations.

Cluster Helm Chart Values

The NVIDIA Run:ai cluster installation can be customized to support your environment via Helm values files or Helm install flags.

These configurations are saved in the runaiconfig Kubernetes object and can be edited post-installation as needed. For more information, see Advanced cluster configurations.

The following table lists the available Helm chart values that can be configured to customize the NVIDIA Run:ai cluster installation.

NVIDIA Run:ai provides metrics and telemetry for both physical cluster entities such as clusters, nodes, and node pools and application organization entities such as departments and projects. Metrics represent over-time data while telemetry represents current analytics data. This data is essential for monitoring and analyzing the performance and health of your platform.

Consuming Metrics and Telemetry Data

Users can consume the data based on their permissions:

1. API - Access the data programmatically through the NVIDIA Run:ai API.

2. CLI - Use the NVIDIA Run:ai Command Line Interface to query and manage the data.

3. UI - Visualize the data through the NVIDIA Run:ai user interface.

API

• Metrics API - Access over-time detailed analytics data programmatically.

• Telemetry API - Access current analytics data programmatically.

Refer to metrics and telemetry to see the full list of supported metrics and telemetry APIs.

CLI

Use the list and describe commands to fetch and manage the data. See CLI reference for more details.

UI Views

Refer to metrics and telemetry to see the full list of supported metrics and telemetry.

• Overview dashboard - Provides a high-level summary of the cluster's health and performance, including key metrics such as GPU utilization, memory usage, and node status. Allows administrators to quickly identify any potential issues or areas for optimization. Offers advanced analytics capabilities for analyzing GPU usage patterns and identifying trends. Helps administrators optimize resource allocation and improve cluster efficiency.

• Quota management - Enables administrators to monitor and manage GPU quotas across the cluster. Includes features for setting and adjusting quotas, tracking usage, and receiving alerts when quotas are exceeded.

• Workload visualizations - Provides detailed insights into the resource usage and utilization of each GPU in the cluster. Includes metrics such as GPU memory utilization, core utilization, and power consumption. Allows administrators to identify GPUs that are under-utilized and overloaded.

• Node and node pool visualizations - Similar to workload visualizations, but focused on the resource usage and utilization of each GPU within a specific node or node pool. Helps administrators identify potential issues or bottlenecks at the node level.

• Advanced NVIDIA metrics - Provides access to a range of advanced NVIDIA metrics, such as GPU temperature, fan speed, and voltage. Enables administrators to monitor the health and performance of GPUs in greater detail. This data is available at the node and workload level. To enable these metrics, contact NVIDIA Run:ai customer support.

The following network requirements are for the NVIDIA Run:ai components installation and usage.

External Access

Set out below are the domains to whitelist and ports to open for installation, upgrade, and usage of the application and its management.

Inbound Rules

To allow your organization’s NVIDIA Run:ai users to interact with the cluster using the , or access specific UI features, certain inbound ports need to be open:

Outbound Rules

For the NVIDIA Run:ai cluster installation and usage, certain outbound ports must be open:

The NVIDIA Run:ai installation has that require additional components to be installed on the cluster. This article includes simple installation examples which can be used optionally and require the following cluster outbound ports to be open:

Internal Network

Ensure that all Kubernetes nodes can communicate with each other across all necessary ports. Kubernetes assumes full interconnectivity between nodes, so you must configure your network to allow this seamless communication. Specific port requirements may vary depending on your network setup.

System and Network Requirements

Before installing the NVIDIA Run:ai control plane, validate that the and are met. For air-gapped environments, make sure you have the prepared.

Permissions

As part of the installation, you will be required to install the NVIDIA Run:ai control plane . The Helm charts require Kubernetes administrator permissions. You can review the exact objects that are created by the charts using the --dry-run on both helm charts.

Installation

Kubernetes

OpenShift

Connect to NVIDIA Run:ai User Interface

1. Open your browser and go to:

1. Log in using the default credentials:

   • User: [email protected]

   • Password: Abcd!234

You will be prompted to change the password.

This section explains the procedure to manage users and their permissions.

Users can be managed locally, or via the identity provider (Idp), while assigned with to manage permissions. For example, user [email protected] is a department admin in department A.

Users Table

The Users table can be found under Access in the NVIDIA Run:ai platform.

The users table provides a list of all the users in the platform. You can manage users and user permissions (access rules) for both local and .

The Users table consists of the following columns:

Customizing the Table View

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values

• Search - Click SEARCH and type the value to search by

• Sort - Click each column header to sort by

• Column selection - Click COLUMNS and select the columns to display in the table

• Download table - Click MORE and then Click Download as CSV. Export to CSV is limited to 20,000 rows.

Creating a Local User

To create a local user:

1. Click +NEW LOCAL USER

2. Enter the user’s Email address

3. Click CREATE

4. Review and copy the user’s credentials:

   • User Email

   • Temporary password to be used on first sign-in

5. Click DONE

Adding an Access Rule to a User

To create an access rule:

1. Select the user you want to add an access rule for

2. Click ACCESS RULES

3. Click +ACCESS RULE

4. Select a role

5. Select a scope

6. Click SAVE RULE

7. Click CLOSE

Deleting a User’s Access Rule

To delete an access rule:

1. Select the user you want to remove an access rule from

2. Click ACCESS RULES

3. Find the access rule assigned to the user you would like to delete

4. Click on the trash icon

5. Click CLOSE

Resetting a User's Password

To reset a user’s password:

1. Select the user you want to reset it’s password

2. Click RESET PASSWORD

3. Click RESET

4. Review and copy the user’s credentials:

   • User Email

   • Temporary password to be used on next sign-in

5. Click DONE

Deleting a User

1. Select the user you want to delete

2. Click DELETE

3. In the dialog, click DELETE to confirm

Using API

Go to the , API reference to view the available actions.

This section explains the procedure to manage Access rules.

Access rules provide users, groups, or applications privileges to system entities. An access rule is the assignment of a to a : <Subject> is a <Role> in a <Scope>. For example, user [email protected] is a department admin in department A.

Access Rules Table

The Access rules table can be found under Access in the NVIDIA Run:ai platform.

The Access rules table provides a list of all the access rules defined in the platform and allows you to manage them.

The Access rules table consists of the following columns:

Customizing the Table View

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values

• Search - Click SEARCH and type the value to search by

• Sort - Click each column header to sort by

• Column selection - Click COLUMNS and select the columns to display in the table

• Download table - Click MORE and then Click Download as CSV. Export to CSV is limited to 20,000 rows.

Adding a New Access Rule

To add a new access rule:

1. Click +NEW ACCESS RULE

2. Select a subject User, SSO Group, or Application

3. Select or enter the subject identifier:

   • User Email for a local user created in NVIDIA Run:ai or for SSO user as recognized by the IDP

   • Group name as recognized by the IDP

   • Application name as created in NVIDIA Run:ai

4. Select a role

5. Select a scope

6. Click SAVE RULE

Editing an Access Rule

Access rules cannot be edited. To change an access rule, you must delete the rule, and then create a new rule to replace it.

Deleting an Access Rule

1. Select the access rule you want to delete

2. Click DELETE

3. On the dialog, click DELETE to confirm

Viewing Your User Access Rule

To view the assigned roles and scopes you have access to:

1. Click the user avatar at the top right corner, then select Settings

2. Click User details

The list of assigned roles and scopes will be displayed.

Using API

Go to the API reference to view the available actions.

This section provides details about NVIDIA Run:ai’s Audit log.

The NVIDIA Run:ai control plane provides the audit log API and event history table in the NVIDIA Run:ai UI. Both reflect the same information regarding changes to business objects: clusters, projects and assets etc.

Event History Table

The Event history table can be found under Event history in the NVIDIA Run:ai UI.

The Event history table consists of the following columns:

Customizing the Table View

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values

• Search - Click SEARCH and type the value to search by

• Sort - Click each column header to sort by

• Column selection - Click COLUMNS and select the columns to display in the table

• Download table - Click MORE and then Click Download as CSV or Download as JSON

Using the Event History Date Selector

The Event history table saves events for the last 90 days. However, the table itself presents up to the last 30 days of information due to the potentially very high number of operations that might be logged during this period.

To view older events, or to refine your search for more specific results or fewer results, use the time selector and change the period you search for. You can also refine your search by clicking and using ADD FILTER accordingly.

Using API

Go to the reference to view the available actions. Since the amount of data is not trivial, the API is based on paging. It retrieves a specified number of items for each API call. You can get more data by using subsequent calls.

Limitations

Submissions of workloads are not audited. As a result, the system does not track or log details of workload submissions, such as timestamps or user activity.

Before Upgrade

Before proceeding with the upgrade, it's crucial to apply the specific prerequisites associated with your current version of NVIDIA Run:ai and every version in between up to the version you are upgrading to.

Helm

NVIDIA Run:ai requires 3.14 or later. Before you continue, validate your installed helm client version. To install or upgrade Helm, see . If you are installing an air-gapped version of NVIDIA Run:ai, the NVIDIA Run:ai tar file contains the helm binary.

Software Files

Upgrade Control Plane

System and Network Requirements

Before upgrading the NVIDIA Run:ai control plane, validate that the latest and are met, as they can change from time to time.

Upgrade

To upgrade from 2.17 or later, run the following:

Upgrade Cluster

System and Network Requirements

Before upgrading the NVIDIA Run:ai cluster, validate that the latest and are met, as they can change from time to time.

Getting Installation Instructions

Follow the setup and installation instructions below to get the installation instructions to upgrade the NVIDIA Run:ai cluster.

Setup

1. In the NVIDIA Run:ai UI, go to Clusters

2. Select the cluster you want to upgrade

3. Click INSTALLATION INSTRUCTIONS

4. Optional: Select the NVIDIA Run:ai cluster version (latest, by default)

5. Click CONTINUE

Installation Instructions

1. Follow the installation instructions. Run the Helm commands provided on your Kubernetes cluster. See the below if .

2. Click DONE

3. Once installation is complete, validate the cluster is Connected and listed with the new cluster version (see the ). Once you have done this, the cluster is upgraded to the latest version.

Troubleshooting

If you encounter an issue with the cluster upgrade, use the troubleshooting scenarios below.

Installation Fails

If the NVIDIA Run:ai cluster upgrade fails, check the installation logs to identify the issue.

Run the following script to print the installation logs:

Cluster Status

If the NVIDIA Run:ai cluster upgrade completes, but the cluster status does not show as Connected, refer to .

NVIDIA Run:ai assets are preconfigured building blocks that simplify the workload submission effort and remove the complexities of Kubernetes and networks for AI practitioners.

Workload assets enable organizations to:

• Create and reuse preconfigured setup for code, data, storage and resources to be used by AI practitioners to simplify the process of submitting workloads

• Share the preconfigured setup with a wide audience of AI practitioners with similar needs

Workload Asset Types

There are four workload asset types used by the workload:

• The container image, tools and connections for the workload

• The type of data, its origin and the target storage location such as PVCs or cloud storage buckets where datasets are stored

• The compute specification, including GPU and CPU compute and memory

• The secrets to be used to access sensitive data, services, and applications such as docker registry or S3 buckets

Asset Scope

When a workload asset is created, a is required. The scope defines who in the organization can view and/or use the asset.

Who Can Create an Asset?

Any subject (user, application, or SSO group) with a that has permissions to Create an asset, can do so within their scope.

Who Can Use an Asset?

Assets are used when submitting workloads. Any subject (user, application or SSO group) with a that has permissions to Create workloads, can also use assets.

Who Can View an Asset?

Any subject (user, application, or SSO group) with a that has permission to View an asset, can do so within their scope.

This article explains how to designate specific node roles in a Kubernetes cluster to ensure optimal performance and reliability in production deployments.

For optimal performance in production clusters, it is essential to avoid extensive CPU usage on GPU nodes where possible. This can be done by ensuring the following:

• NVIDIA Run:ai system-level services run on dedicated CPU-only nodes.

• Workloads that do not request GPU resources (e.g. Machine Learning jobs) are executed on CPU-only nodes.

NVIDIA Run:ai services are scheduled on the defined node roles by applying Kubernetes Node Affinity using node labels .

Prerequisites

To perform these tasks, make sure to install the NVIDIA Run:ai Administrator CLI.

Configure Node Roles

The following node roles can be configured on the cluster:

• System node: Reserved for NVIDIA Run:ai system-level services.

• GPU Worker node: Dedicated for GPU-based workloads.

• CPU Worker node: Used for CPU-only workloads.

System Nodes

NVIDIA Run:ai system nodes run system-level services required to operate. This can be done via the Kubectl (recommended) or via NVIDIA Run:ai Administrator CLI.

By default, NVIDIA Run:ai applies a node affinity rule to prefer nodes that are labeled with node-role.kubernetes.io/runai-system for system services scheduling. You can modify the default node affinity rule by:

• Editing the spec.global.affinity configuration parameter as detailed in Advanced cluster configurations.

• Editing the global.affinity configuration as detailed in Advanced control plane configurations for self-hosted deployments.

Kubectl

To set a system role for a node in your Kubernetes cluster using Kubectl, follow these steps:

1. Use the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

2. Run one of the following commands to label the node with its role:

   Copy

   kubectl label nodes <node-name> node-role.kubernetes.io/runai-system=true
   kubectl label nodes <node-name> node-role.kubernetes.io/runai-system=false

NVIDIA Run:ai Administrator CLI

To set a system role for a node in your Kubernetes cluster, follow these steps:

1. Run the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

2. Run one of the following commands to set or remove a node’s role:

   Copy

   runai-adm set node-role --runai-system-worker <node-name>
   runai-adm remove node-role --runai-system-worker <node-name>

The set node-role command will label the node and set relevant cluster configurations.

Worker Nodes

NVIDIA Run:ai worker nodes run user-submitted workloads and system-level DeamonSets required to operate. This can be managed via the Kubectl (recommended) or via NVIDIA Run:ai Administrator CLI.

By default, GPU workloads are scheduled on GPU nodes baed on the nvidia.com/gpu.present label. When global.nodeAffinity.restrictScheduling is set to true via the Advanced cluster configurations:

• GPU Workloads are scheduled with node affinity rule to require nodes that are labeled with node-role.kubernetes.io/runai-gpu-worker

• CPU-only Workloads are scheduled with node affinity rule to require nodes that are labeled with node-role.kubernetes.io/runai-cpu-worker

Kubectl

To set a worker role for a node in your Kubernetes cluster using Kubectl, follow these steps:

1. Validate the global.nodeAffinity.restrictScheduling is set to true in the cluster’s Configurations.

2. Use the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

3. Run one of the following commands to label the node with its role. Replace the label and value (true/false) to enable or disable GPU/CPU roles as needed:

   Copy

   kubectl label nodes <node-name> node-role.kubernetes.io/runai-gpu-worker=true
   kubectl label nodes <node-name> node-role.kubernetes.io/runai-cpu-worker=false

NVIDIA Run:ai Administrator CLI

To set worker role for a node in your Kubernetes cluster via NVIDIA Run:ai Administrator CLI, follow these steps:

1. Use the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

2. Run one of the following commands to set or remove a node’s role. <node-role> must be either --gpu-worker or --cpu-worker :

   Copy

   runai-adm set node-role <node-role> <node-name>
   runai-adm remove node-role <node-role> <node-name>

The set node-role command will label the node and set cluster configuration global.nodeAffinity.restrictScheduling true.

NVIDIA’s Multi-Instance GPU (MIG) enables splitting a GPU into multiple logical GPU devices, each with its own memory and compute portion of the physical GPU.

NVIDIA provides two MIG strategies:

• Single - A GPU can be divided evenly. This means all MIG profiles are the same.

• Mixed - A GPU can be divided into different profiles.

The NVIDIA Run:ai platform supports running workloads using NVIDIA MIG. Administrators can set the Kubernetes nodes to their preferred MIG strategy and configure the appropriate MIG profiles for researchers and MLOPS engineers to use.

This guide explains how to configure MIG in each strategy to submit workloads. It also outlines the individual implications of each strategy and best practices for administrators.

Before You Start

To use MIG single and mixed strategy effectively, make sure to familiarize yourself with the following NVIDIA resources:

• NVIDIA Multi-Instance GPU

• MIG User Guide

• GPU Operator with MIG

Configuring Single MIG Strategy

When deploying MIG using single strategy, all GPUs within a node are configured with the same profile. For example, a node might have GPUs configured with 3 MIG slices of profile type 1g.20gb, or 7 MIG slices of profile 1g.10gb. With this strategy, MIG profiles are displayed as whole GPU devices by CUDA.

The NVIDIA Run:ai platform discovers these MIG profiles as whole GPU devices as well, ensuring MIG devices are transparent to the end-user (practitioner). For example, a node that consists of 8 physical GPUs split into MIG slices, 3×2g20gb slices each, is discovered by the NVIDIA Run:ai platform as a node with 24 GPU devices.

Users can submit workloads by requesting a specific number of GPU devices (X GPU) and NVIDIA Run:ai will allocate X MIG slices (logical devices). The NVIDIA Run:ai platform deducts X GPUs from the workload’s Project quota, regardless of whether this ‘logical GPU’ represents 1/3 of a physical GPU device or 1/7 of a physical GPU device.

Configuring Mixed MIG Strategy

When deploying MIG using mixed strategy, each GPU in a node can be configured with a different combination of MIG profiles such as 2×2g.20gb and 3×1g.10gb. For details on supported combinations per GPU type, refer to Supported MIG Profiles.

In mixed strategy, physical GPU devices continue to be displayed as physical GPU devices by CUDA, and each MIG profile is shown individually. The NVIDIA Run:ai platform identifies the physical GPU devices normally, however, MIG profiles are not visible in the UI or node APIs.

When submitting third-party workloads with this strategy, the user should explicitly specify the exact requested MIG profile (for example, nvidia.com/gpu.product: A100-SXM4-40GB-MIG-3g.20gb). The NVIDIA Run:ai Scheduler finds a node that can provide this specific profile and binds it to the workload.

A third-party workload submitted with a MIG profile of type Xg.Ygb (e.g. 3g.40gb or 2g.20gb) is considered as consuming X GPUs. These X GPUs will be deducted from the workload’s project quota of GPUs. For example, a 3g.40gb profile deducts 3 GPUs from the associated Project’s quota, while 2g.20gb deducts 2 GPUs from the associated Project’s quota. This is done to maintain a logical ratio according to the characteristics of the MIG profile.

Best Practices for Administrators

Single Strategy

• Configure proper and uniform sizes of MIG slices (profiles) across all GPUs within a node.

• Set the same MIG profiles on all nodes of a single node pool.

• Create separate node pools with different MIG profile configurations allowing users to select the pool that best matches their workloads’ needs.

• Ensure Project quotas are allocated according to the MIG profile sizes.

Mixed Strategy

• Use mixed strategy with workloads that require diverse resources. Make sure to evaluate the workload requirements and plan accordingly.

• Configure individual MIG profiles on each node by using a limited set of MIG profile combinations to minimize complexity. Make sure to evaluate your requirements and node configurations.

• Ensure Project quotas are allocated according to the MIG profile sizes.

This article explains the procedure to configure and manage scheduling rules.

Scheduling rules are restrictions applied to workloads. These restrictions apply to either the resources (nodes) on which workloads can run or the duration of the run time. Scheduling rules are set for Projects or Departments and apply to specific workload types. Once scheduling rules are set for a project or department, all matching workloads associated with the project have the restrictions applied to them, as defined, when the workload was submitted. New scheduling rules added to a project are not applied over previously created workloads associated with that project.

There are three types of scheduling rules:

Workload Duration (Time Limit)

This rule limits the duration of a workload run time. Workload run time is calculated as the total time in which the workload was in status Running. You can apply a single rule per workload type - Preemptive Workspaces, Non-preemptive Workspaces, and Training.

Idle GPU Time Limit

This rule limits the total GPU time of a workload. Workload idle time is counted from the first time the workload is in status Running and the GPU was idle. Idleness is calculated by employing the runai_gpu_idle_seconds_per_workload metric. This metric determines the total duration of zero GPU utilization within each 30-second interval. If the GPU remains idle throughout the 30-second window, 30 seconds are added to the idleness sum; otherwise, the idleness count is reset. You can apply a single rule per workload type - “Preemptible” Workspaces, “Non-preemptible” Workspaces, and Training.

Node Type (Affinity)

Node type is used to select a group of nodes, typically with specific characteristics such as a hardware feature, storage type, fast networking interconnection, etc. The Scheduler uses node type as an indication of which nodes should be used for your workloads, within this project.

Node type is a label in the form of run.ai/type and a value (e.g. run.ai/type = dgx200) that the administrator uses to tag a set of nodes. Adding the node type to the project’s scheduling rules mandates the user to submit workloads with a node type label/value pairs from this list, according to the workload type - Workspace or Training. The Scheduler then schedules workloads using a node selector, targeting nodes tagged with the NVIDIA Run:ai node type label/value pair. Node pools and a node type can be used in conjunction. For example, specifying a node pool and a smaller group of nodes from that node pool that includes a fast SSD memory or other unique characteristics.

Labelling Nodes for Node Types Grouping

The administrator should use a node label with the key of run.ai/type and any coupled value

To assign a label to nodes you want to group, set the ‘node type (affinity)’ on each relevant node:

1. Obtain the list of nodes and their current labels by copying the following to your terminal:

kubectl get nodes --show-labels

1. Annotate a specific node with a new label by copying the following to your terminal:

kubectl label node <node-name> run.ai/type=<value>

Adding a Scheduling Rule to a Project or Department

To add a scheduling rule:

1. Select the project/department for which you want to add a scheduling rule

2. Click EDIT

3. In the Scheduling rules section click +RULE

4. Select the rule type

5. Select the workload type and time limitation period

6. For Node type, choose one or more labels for the desired nodes

7. Click SAVE

Editing the Scheduling Rule

To edit a scheduling rule:

1. Select the project/department for which you want to edit its scheduling rule

2. Click EDIT

3. Find the scheduling rule you would like to edit

4. Edit the rule

5. Click SAVE

Deleting the Scheduling Rule

To delete a scheduling rule:

1. Select the project/department from which you want to delete a scheduling rule

2. Click EDIT

3. Find the scheduling rule you would like to delete

4. Click on the x icon

5. Click SAVE

Using API

Go to the Projects API reference to view the available actions

At NVIDIA Run:ai, Administrators can access a suite of tools designed to facilitate efficient account management. This article focuses on two key features: workload policies and workload scheduling rules. These features empower admins to establish default values and implement restrictions allowing enhanced control, assuring compatibility with organizational policies, and optimizing resource usage and utilization.

Workload Policies

A workload policy is an end-to-end solution for AI managers and administrators to control and simplify how workloads are submitted. This solution allows them to set best practices, enforce limitations, and standardize processes for the submission of workloads for AI projects within their organization. It acts as a key guideline for data scientists, researchers, ML & MLOps engineers by standardizing submission practices and simplifying the workload submission process.

Why Use a Workload Policy?

Implementing workload policies is essential when managing complex AI projects within an enterprise for several reasons:

1. Resource control and management - Defining or limiting the use of costly resources across the enterprise via a centralized management system to ensure efficient allocation and prevent overuse.

2. Setting best practices - Provide managers with the ability to establish guidelines and standards to follow, reducing errors amongst AI practitioners within the organization.

3. Security and compliance - Define and enforce permitted and restricted actions to uphold organizational security and meet compliance requirements.

4. Simplified setup - Conveniently allow setting defaults and streamline the workload submission process for AI practitioners.

5. Scalability and diversity

   1. Multi-purpose clusters with various workload types that may have different requirements and characteristics for resource usage.

   2. The organization has multiple hierarchies, each with distinct goals, objectives, and degrees of flexibility.

   3. Manage multiple users and projects with distinct requirements and methods, ensuring appropriate utilization of resources.

Understanding the Mechanism

The following sections provide details of how the workload policy mechanism works.

Cross-Interface Enforcement

The policy enforces the workloads regardless of whether they were submitted via UI, CLI, Rest APIs, or Kubernetes YAMLs.

Policy Types

NVIDIA Run:ai’s policies enforce NVIDIA Run:ai workloads. The policy type is per NVIDIA Run:ai workload type. This allows administrators to set different policies for each workload type.

Policy Structure - Rules, Defaults, and Imposed Assets

A policy consists of rules for limiting and controlling the values of fields of the workload. In addition to rules, some defaults allow the implementation of default values to different workload fields. These default values are not rules, as they simply suggest values that can be overridden during the workload submission.

Furthermore, policies allow the enforcement of workload assets. For example, as an admin, you can impose a data source of type PVC to be used by any workload submitted.

For more information, see rules, defaults and imposed assets.

Scope of Effectiveness

Numerous teams working on various projects require the use of different tools, requirements, and safeguards. One policy may not suit all teams and their requirements. Hence, administrators can select the scope to cover the effectiveness of the policy. When a scope is selected, all of its subordinate units are also affected. As a result, all workloads submitted within the selected scope are controlled by the policy.

For example, if a policy is set for Department A, all workloads submitted by any of the projects within this department are controlled.

A scope for a policy can be:

The different scoping of policies also allows the breakdown of the responsibility between different administrators. This allows delegation of ownership between different levels within the organization. The policies, containing rules and defaults, propagate* down the organizational tree, forming an “effective” policy that enforces any workload submitted by users within the project.

If a rule for a specific field is already occupied by a policy in the organization, another unit within the same branch cannot submit an additional rule on the same field. As a result, administrators of higher scopes must request lower-scope administrators to free up the specific rule from their policy. However, defaults of the same field can be submitted by different organizational policies, as they are “soft” rules that are not critical to override, and the smallest level of the default is the one that becomes the effective default (project default ‚”wins” vs department default, department default “wins” vs cluster default etc.).

Scheduling Rules

Scheduling rules limit a researcher's access to resources and provides a way for the admin to control resource allocation and prevent the waste of resources. Admins should use the rules to prevent GPU idleness, prevent GPU hogging and allocate specific types of resources to different types of workloads.

Admin can limit the duration of a workload, the duration of the idle time, or the type of nodes the workload can use. Rules are defined for and apply to all workloads in the project or department. In addition, rules can be applied to a specific type of workload in a project or department (workspace, standard training, or inference). When a workload reaches the limitation of the rule, it is stopped if the rule is time-limited. The rule type prevents the workload from being scheduled on nodes that violate the rule limitation.

This section explains the procedure to manage templates.

A template is a pre-set configuration that is used to quickly configure and submit workloads using existing assets. A template consists of all the assets a workload needs, allowing researchers to submit a workload in a single click, or make subtle adjustments to differentiate them from each other.

Workspace Templates Table

The Templates table can be found under Workload manager in the NVIDIA Run:ai User interface.

The Templates table provides a list of all the templates defined in the platform, and allows you to manage them.

The Templates table consists of the following columns:

Customizing the Table View

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values

• Search - Click SEARCH and type the value to search by

• Sort - Click each column header to sort by

• Column selection - Click COLUMNS and select the columns to display in the table

• Download table - Click MORE and then click Download as CSV. Export to CSV is limited to 20,000 rows.

• Refresh (optional) - Click REFRESH to update the table with the latest data

• Show/Hide details (optional) - Click to view additional information on the selected row

Adding a New Workspace Template

To add a new template:

1. Click +NEW TEMPLATE

2. Set the scope for the template

3. Enter a name for the template

4. Select the environment for your workload

5. Select the node resources needed to run your workload - or - Click +NEW COMPUTE RESOURCE

6. Set the volume needed for your workload

7. Create a new data source

8. Set auto-deletion, annotations and labels, as required

9. Click CREATE TEMPLATE

Copying a Template

To copy an existing template:

1. Select the template you want to copy

2. Click MAKE A COPY

3. Enter a name for the template. The name must be unique.

4. Update the template and click CREATE TEMPLATE

Renaming a Template

To rename an existing template:

1. Select the template you want to rename

2. Click Rename and edit the name/description

Deleting a Template

To delete a template:

1. Select the template you want to delete

2. Click DELETE

3. Confirm you want to delete the template

Using API

Go to the Workload template API reference to view the available actions

This section explains the procedure to manage your organization's applications.

Applications are used for API integrations with NVIDIA Run:ai. An application contains a client ID and a client secret. With the client credentials, you can obtain a token as detailed in and use it within subsequent API calls.

Applications are assigned with to manage permissions. For example, application ci-pipeline-prod is assigned with a Researcher role in Cluster: A.

Applications Table

The Applications table can be found under Access in the NVIDIA Run:ai platform.

The Applications table provides a list of all the applications defined in the platform, and allows you to manage them.

The Applications table consists of the following columns:

Customizing the Table View

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values

• Search - Click SEARCH and type the value to search by

• Sort - Click each column header to sort by

• Column selection - Click COLUMNS and select the columns to display in the table

• Download table - Click MORE and then Click Download as CSV. Export to CSV is limited to 20,000 rows.

Creating an Application

To create an application:

1. Click +NEW APPLICATION

2. Enter the application’s name

3. Click CREATE

4. Copy the Client ID and Client secret and store them securely

5. Click DONE

Adding an Access Rule to an Application

To create an access rule:

1. Select the application you want to add an access rule for

2. Click ACCESS RULES

3. Click +ACCESS RULE

4. Select a role

5. Select a scope

6. Click SAVE RULE

7. Click CLOSE

Deleting an Access Rule from an Application

To delete an access rule:

1. Select the application you want to remove an access rule from

2. Click ACCESS RULES

3. Find the access rule assigned to the user you would like to delete

4. Click on the trash icon

5. Click CLOSE

Regenerating a Client Secret

To regenerate a client secret:

1. Locate the application you want to regenerate its client secret

2. Click REGENERATE CLIENT SECRET

3. Click REGENERATE

4. Copy the New client secret and store it securely

5. Click DONE

Deleting an Application

1. Select the application you want to delete

2. Click DELETE

3. On the dialog, click DELETE to confirm

Using API

Go to the , API reference to view the available actions.

In the world of machine learning (ML), the journey from raw data to actionable insights is a complex process that spans multiple stages. Each stage of the AI lifecycle requires different tools, resources, and frameworks to ensure optimal performance. NVIDIA Run:ai simplifies this process by offering specialized workload types tailored to each phase, facilitating a smooth transition across various stages of the ML workflows.

The ML lifecycle usually begins with the experimental work on data and exploration of different modeling techniques to identify the best approach for accurate predictions. At this stage, resource consumption is usually moderate as experimentation is done on a smaller scale. As confidence grows in the model's potential and its accuracy, the demand for compute resources increases. This is especially true during the training phase, where vast amounts of data need to be processed, particularly with complex models such as large language models (LLMs), with their huge parameter sizes, that often require distributed training across multiple GPUs to handle the intensive computational load.

Finally, once the model is ready, it moves to the inference stage, where it is deployed to make predictions on new, unseen data. NVIDIA Run:ai's workload types are designed to correspond with the natural stages of this lifecycle. They are structured to align with the specific resource and framework requirements of each phase, ensuring that AI researchers and data scientists can focus on advancing their models without worrying about infrastructure management.

NVIDIA Run:ai offers three workload types that correspond to a specific phase of the researcher’s work:

• Workspaces – For experimentation with data and models.

• Training – For resource-intensive tasks such as model training and data preparation.

• Inference – For deploying and serving the trained model.

Workspaces: The Experimentation Phase

The Workspace is where data scientists conduct initial research, experiment with different data sets, and test various algorithms. This is the most flexible stage in the ML lifecycle, where models and data are explored, tuned, and refined. The value of workspaces lies in the flexibility they offer, allowing the researcher to iterate quickly without being constrained by rigid infrastructure.

• Framework flexibility

  Workspaces support a variety of machine learning frameworks, as researchers need to experiment with different tools and methods.

• Resource requirements

  Workspaces are often lighter on resources compared to the training phase, but they still require significant computational power for data processing, analysis, and model iteration.

  Hence, the default for the NVIDIA Run:ai workspaces considerations is to allow scheduling those workloads without the ability to preempt them once the resources were allocated. However, this non-preemptible state doesn’t allow utilizing more resources outside of the project’s deserved quota.

See to learn more about how to submit a workspace via the NVIDIA Run:ai platform. For quick starts, see .

Training: Scaling Resources for Model Development

As models mature and the need for more robust data processing and model training increases, NVIDIA Run:ai facilitates this shift through the Training workload. This phase is resource-intensive, often requiring distributed computing and high-performance clusters to process vast data sets and train models.

• Training architecture

  For training workloads NVIDIA Run:ai allows you to specify the architecture - standard or distributed. The distributed architecture is relevant for larger data sets and more complex models that require utilizing multiple nodes. For the distributed architecture, NVIDIA Run:ai allows you to specify different configurations for the master and workers and select which framework to use - PyTorch, XGBoost, MPI, TensorFlow and JAX. In addition, as part of the distributed configuration, NVIDIA Run:ai enables the researchers to schedule their distributed workloads on nodes within the same region, zone, placement group, or any other topology.

• Resource requirements

  Training tasks demand high memory, compute power, and storage. NVIDIA Run:ai ensures that the allocated resources match the scale of the task and allows those workloads to utilize more compute resources than the project’s deserved quota. Make sure that if you wish your training workload not to be preempted, specify the number of GPUs that are in your quota.

See and to learn more about how to submit a training workload via the NVIDIA Run:ai UI. For quick starts, see and .

Inference: Deploying and Serving models

Once a model is trained and validated, it moves to the Inference stage, where it is deployed to make predictions (usually in a production environment). This phase is all about efficiency and responsiveness, as the model needs to serve real-time or batch predictions to end-users or other systems.

• Inference-specific use cases

  Naturally, inference workloads are required to change and adapt to the ever-changing demands to meet SLA. For example, additional replicas may be deployed, manually or automatically, to increase compute resources as part of a horizontal scaling approach or a new version of the deployment may need to be rolled out without affecting the running services.

• Resource requirements

  Inference models differ in size and purpose, leading to varying computational requirements. For example, small OCR models can run efficiently on CPUs, whereas LLMs typically require significant GPU memory for deployment and serving. Inference workloads are considered production-critical and are given the highest priority to ensure compliance with SLAs. Additionally, NVIDIA Run:ai ensures that inference workloads cannot be preempted, maintaining consistent performance and reliability.

See to learn more about how to submit an inference workload via the NVIDIA Run:ai UI.

NVIDIA Run:ai supports simultaneous submission of multiple workloads to single or multi-GPUs when using . This is achieved by slicing the GPU memory between the different workloads according to the requested GPU fraction, and by using NVIDIA’s GPU time-slicing to share the GPU compute runtime. NVIDIA Run:ai ensures each workload receives the exact share of the GPU memory (= gpu_memory * requested), while the NVIDIA GPU time-slicing splits the GPU runtime evenly between the different workloads running on that GPU.

To provide customers with predictable and accurate GPU compute resource scheduling, NVIDIA Run:ai’s GPU time-slicing adds fractional compute capabilities on top of NVIDIA Run:ai GPU fraction capabilities.

How GPU Time-Slicing Works

While the default NVIDIA GPU time-slicing allows for sharing the GPU compute runtime evenly without splitting or limiting the runtime of each workload, NVIDIA Run:ai’s GPU time-slicing mechanism gives each workload exclusive access to the full GPU for a limited amount of time, lease time, in each scheduling cycle, plan time. This cycle repeats itself for the lifetime of the workload. Using the GPU runtime this way guarantees a workload is granted its requested GPU compute resources proportionally to its requested GPU fraction, but also allows splitting GPU unused compute time up to a requested Limit.

For example, when there are 2 workloads running on the same GPU, with NVIDIA’s default GPU time slicing, each workload gets 50% of the GPU compute runtime, even if one workload requests 25% of the GPU memory, and the other workload requests 75% of the GPU memory. With the NVIDIA Run:ai GPU time-slicing, the first workload will get 25% of the GPU compute time and the second will get 75%. If one of the workloads does not use its deserved GPU compute time, the others can split that time evenly between them. As shown in the example, if one of the workloads does not request the GPU for some time, the other will get the full GPU compute time.

GPU Time-Slicing Modes

NVIDIA Run:ai offers two GPU time-slicing modes:

• Strict - Each workload gets its precise GPU compute fraction, which equals to its requested GPU (memory) fraction. In terms of official Kubernetes resource specification, this means:

• Fair - Each workload is guaranteed at least its GPU compute fraction, but at the same time can also use additional GPU runtime compute slices that are not used by other idle workloads. Those excess time slices are divided equally between all workloads running on that GPU (after each got at least its requested GPU compute fraction). In terms of official Kubernetes resource specification, this means:

The figure below illustrates how Strict time-slicing mode uses the GPU from Lease (slice) and Plan (cycle) perspective:

The figure below illustrates how Fair time-slicing mode uses the GPU from Lease (slice) and Plan (cycle) perspective:

Time-Slicing Plan and Lease Times

Each GPU scheduling cycle is a plan. The plan is determined by the lease time and granularity (precision). By default, basic lease time is 250ms with 5% granularity (precision), which means the plan (cycle) time is: 250 / 0.05 = 5000ms (5 Sec). Using these values, a workload that requests gpu-fraction=0.5 gets 2.5s runtime out of the 5s cycle time.

Different workloads require different SLA and precision, so it also possible to tune the lease time and precision for customizing the time-slicing capabilities to your cluster.

Once timeSlicing is enabled in the runaiconfig, all submitted GPU fractions or GPU memory workloads will have their gpu-compute-request/limit set automatically by the system, depending on the annotation used on the time-slicing mode:

• Strict compute resources:

• Fair compute resources:

Enabling GPU Time-Slicing

NVIDIA Run:ai’s GPU time-slicing is a cluster flag which changes the default NVIDIA time-slicing used by GPU fractions. For more details, see .

Enable GPU time-slicing by setting the following cluster flag in the runaiconfig file:

If the timeSlicing flag is not set, the system continues to use the default NVIDIA GPU time-slicing to maintain backward compatibility.

The Node Level Scheduler optimizes the performance of your pods and maximizes the utilization of GPUs by making optimal local decisions on GPU allocation to your pods. While the chooses the specific node for a pod, it has no visibility to the node’s GPUs' internal state. The Node Level Scheduler is aware of the local GPUs' states and makes optimal local decisions such that it can optimize both the GPU utilization and pods’ performance running on the node’s GPUs.

This guide provides an overview of the best use cases for the Node Level Scheduler and instructions for configuring it to maximize GPU performance and pod efficiency.

Deployment Considerations

• While the Node Level Scheduler applies to all , it will best optimize the performance of burstable workloads. Burstable workloads are workloads that use , giving those more GPU memory than requested and up to the Limit specified.

• Burstable workloads are always susceptible to an OOM Kill signal if the owner of the excess memory requires it back. This means that using the Node Level Scheduler with inference or training workloads may cause pod preemption.

• Using interactive workloads with notebooks is the best use case for burstable workloads and Node Level Scheduler. These workloads behave differently since the OOM Kill signal will cause the notebooks' GPU process to exit but not the notebook itself. This keeps the interactive pod running and retrying to attach a GPU again.

Interactive Notebooks Use Case

This use case is one scenario that shows how Node Level Scheduler locally optimizes and maximizes GPU utilization and workspaces’ performance.

1. The below shows a node with 2 GPUs and 2 submitted workspaces:

1. The Scheduler instructs the node to put the 2 workspaces on a single GPU, a single GPU and leaving the other free for a workload that requires resources. This means GPU#2 is idle while the two workspaces can only use up to half a GPU, even if they temporarily need more:

1. With the Node Level Scheduler enabled, the local decision will be to spread those 2 workspaces on 2 GPUs and allow them to maximize both workspaces’ performance and GPUs’ utilization by bursting out up to the full GPU memory and GPU compute resources:

1. The NVIDIA Run:ai Scheduler still sees a node with one fully empty GPU and one fully occupied GPU. When a 3rd workload is scheduled, and it requires a full GPU (or more than 0.5 GPU), the Scheduler will schedule it to that node, and the Node Level Scheduler will move one of the workspaces to run with the other in GPU#1, as was the Scheduler’s initial plan. Moving the workspace from GPU#1 back to GPU#2 maintains the workspace running while the GPU process within the Jupyter notebook is killed and re-established on GPU#2, continuing to serve the workspace:

Using Node Level Scheduler

The Node Level Scheduler can be enabled per node pool. To use Node Level Scheduler, follow the below steps.

Enable on Your Cluster

1. Enable the Node Level Scheduler at the cluster level (per cluster) by:

   1. Editing the runaiconfig as follows. For more details, see :

   2. Or, using the following kubectl patch command:

Enable on a Node Pool

Enable Node Level Scheduler on any of the node pools:

1. Select Resources → Node pools

2. or

3. Under the Resource Utilization Optimization tab, change the number of workloads on each GPU to any value other than Not Enforced (i.e. 2, 3, 4, 5)

The Node Level Scheduler is now ready to be used on that node pool.

Submit a Workload

In order for a workload to be considered by the Node Level Scheduler for rerouting, it must be submitted with a GPU Request and Limit where the Limit is larger than the Request:

• Enable and set

• Then using dynamic GPU fractions

This section provides detailed instructions on how to manage both planned and unplanned node downtimes in a Kubernetes cluster running NVIDIA Run:ai. It covers all the steps to maintain service continuity and ensure the proper handling of workloads during these events.

Prerequisites

• Access to Kubernetes cluster - Administrative access to the Kubernetes cluster, including permissions to run kubectl commands

• Basic knowledge of Kubernetes - Familiarity with Kubernetes concepts such as nodes, taints, and workloads

• NVIDIA Run:ai installation - The NVIDIA Run:ai software installed and configured within your Kubernetes cluster

• Node naming conventions - Know the names of the nodes within your cluster, as these are required when executing the commands

Node Types

This section distinguishes between two types of nodes within a NVIDIA Run:ai installation:

• Worker nodes - Nodes on which AI practitioners can submit and run workloads

• NVIDIA Run:ai system nodes - Nodes on which the NVIDIA Run:ai software runs, managing the cluster's operations

Worker Nodes

Worker nodes are responsible for running workloads. When a worker node goes down, either due to planned maintenance or unexpected failure, workloads ideally migrate to other available nodes or wait in the queue to be executed when possible.

Training vs. Interactive Workloads

The following workload types can run on worker nodes:

• Training workloads - These are long-running processes that, in case of node downtime, can automatically move to another node.

• Interactive workloads - These are short-lived, interactive processes that require manual intervention to be relocated to another node.

Planned Maintenance

Before stopping a worker node for maintenance, perform the following steps:

1. Prevent new workloads on the node

   To stop the Kubernetes Scheduler from assigning new workloads to the node and to safely remove all existing workloads, copy the following command to your terminal:

   Copy

   kubectl taint nodes <node-name> runai=drain:NoExecute

   • <node-name> Replace this placeholder with the actual name of the node you want to drain

   • kubectl taint nodes This command is used to add a taint to the node, which prevents any new pods from being scheduled on it

   • runai=drain:NoExecute This specific taint ensures that all existing pods on the node are evicted and rescheduled on other available nodes, if possible

   Result: The node stops accepting new workloads, and existing workloads either migrate to other nodes or are placed in a queue for later execution.

2. Shut down and perform maintenance

   After draining the node, you can safely shut it down and perform the necessary maintenance tasks.

3. Restart the node

   Once maintenance is complete and the node is back online, remove the taint to allow the node to resume normal operations. Copy the following command to your terminal:

   Copy

   kubectl taint nodes <node-name> runai=drain:NoExecute-

   runai=drain:NoExecute- The - at the end of the command indicates the removal of the taint. This allows the node to start accepting new workloads again.

   Result: The node rejoins the cluster's pool of available resources, and workloads can be scheduled on it as usual.

Unplanned Downtime

In the event of unplanned downtime:

1. Automatic restart If a node fails but immediately restarts, all services and workloads automatically resume.

2. Extended downtime

   If the node remains down for an extended period, drain the node to migrate workloads to other nodes. Copy the following command to your terminal:

   Copy

   kubectl taint nodes <node-name> runai=drain:NoExecute

   The command works the same as in the planned maintenance section, ensuring that no workloads remain scheduled on the node while it is down.

3. Reintegrate the node

   Once the node is back online, remove the taint to allow it to rejoin the cluster's operations. Copy the following command to your terminal:

   Copy

   kubectl taint nodes <node-name> runai=drain:NoExecute- 

   Result: This action reintegrates the node into the cluster, allowing it to accept new workloads.

4. Permanent shutdown

   If the node is to be permanently decommissioned, remove it from Kubernetes with the following command:

   Copy

   kubectl delete node <node-name>

   • kubectl delete node This command completely removes the node from the cluster

   • <node-name> Replace this placeholder with the actual name of the node

   Result: The node is no longer part of the Kubernetes cluster. If you plan to bring the node back later, it must be rejoined to thel cluster using the steps outlined in the next section.

NVIDIA Run:ai System Nodes

In a production environment, the services responsible for scheduling, submitting and managing NVIDIA Run:ai workloads operate on one or more NVIDIA Run:ai system nodes. It is recommended to have more than one system node to ensure high availability. If one system node goes down, another can take over, maintaining continuity. If a second system node does not exist, you must designate another node in the cluster as a temporary NVIDIA Run:ai system node to maintain operations.

The protocols for handling planned maintenance and unplanned downtime are identical to those for worker nodes. Refer to the above section for detailed instructions.

Rejoining a Node into the Kubernetes Cluster

To rejoin a node to the Kubernetes cluster, follow these steps:

1. Generate a join command on the master node

   On the master node, copy the following command to your terminal:

   Copy

   kubeadm token create --print-join-command

   • kubeadm token create This command generates a token that can be used to join a node to the Kubernetes cluster.

   • --print-join-command This option outputs the full command that needs to be run on the worker node to rejoin it to the cluster.

   Result: The command outputs a kubeadm join command.

2. Run the join command on the worker node

   Copy the kubeadm join command generated from the previous step and run it on the worker node that needs to rejoin the cluster.

   Copy

   kubeadm join <master-ip>:<master-port> 
   --token <token> \ --discovery-token-ca-cert-hash sha256:<hash>

   The kubeadm join command re-enrolls the node into the cluster, allowing it to start participating in the cluster's workload scheduling.

3. Verify node rejoining

   Verify that the node has successfully rejoined the cluster by running:

   Copy

   kubectl get nodes

   • kubectl get nodes This command lists all nodes currently part of the Kubernetes cluster, along with their status

   Result: The rejoined node should appear in the list with a status of Ready

4. Re-label nodes

   Once the node is ready, ensure it is labeled according to its role within the cluster.

AI initiatives refer to advancing research, development, and implementation of AI technologies. These initiatives represent your business needs and involve collaboration between individuals, teams, and other stakeholders. AI initiatives require compute resources and a methodology to effectively and efficiently use those compute resources and split them among the different AI initiatives stakeholders. The building blocks of AI compute resources are GPUs, CPUs, and memory, which are built into nodes (servers) and can be further grouped into node pools. Nodes and node pools are part of a Kubernetes cluster.

To manage AI initiatives in NVIDIA Run:ai you should:

• Map your organization and initiatives to projects and optionally departments

• Map compute resources (node pools and quotas) to projects and optionally departments

• Assign users (e.g. AI practitioners, ML engineers, Admins) to projects and departments

Mapping Your Organization

The way you map your AI initiatives and organization into NVIDIA Run:ai projects and departments should reflect your organization’s structure and Project management practices. There are multiple options, and we provide you here with 3 examples of typical forms in which to map your organization, initiatives, and users into NVIDIA Run:ai, but of course, other ways that suit your requirements are also acceptable.

Based on Individuals

A typical use case would be students (individual practitioners) within a faculty (business unit) - an individual practitioner may be involved in one or more initiatives. In this example, the resources are accounted for by the student (project) and aggregated per faculty (department).

Department = business unit / Project = individual practitioner

Based on Business Units

A typical use case would be an AI service (business unit) split into AI capabilities (initiatives) - an individual practitioner may be involved in several initiatives. In this example, the resources are accounted for by Initiative (project) and aggregated per AI service (department).

Department = business unit / Project = initiative

Based on the Organizational Structure

A typical use case would be a business unit split into teams - an individual practitioner is involved in a single team (project) but the team may be involved in several AI initiatives. In this example, the resources are accounted for by team (project) and aggregated per business unit (department).

Department = business unit / Project = team

Mapping Your Resources

AI initiatives require compute resources such as GPUs and CPUs to run. Compute resources in any organization are limited, either due to the number of servers (nodes) owned by the organization is limited, the budget it can spend to lease resources in the cloud or spending for in-house servers is also limited. Every organization strives to optimize the usage of its resources by maximizing their utilization and providing all users with their needs. Therefore, the organization needs to split resources according to the organization's internal priorities and budget constraints. But even after splitting the resources, the orchestration layer should still provide fairness between the resourced consumers, and allow access to unused resources to minimize scenarios of idle resources.

Another aspect of resource management is how to group your resources effectively, especially in large environments, or environments that are made of heterogeneous types of hardware, where some users need to use specific hardware types, or where other users should avoid occupying critical hardware of some users or initiatives.

NVIDIA Run:ai assists you with all of these complex issues by allowing you to map your cluster resources to node pools, then map each Project and Department a quota allocation per node pool, and set access rights to unused resources (over quota) per node pool.

Grouping Your Resources

There are several reasons why you would group resources (nodes) into node pools:

• Control the GPU type to use in heterogeneous hardware environment - in many cases, AI models can be optimized per hardware type they will use, e.g. a training workload that is optimized for H100 does not necessarily run optimally on an A100, and vice versa. Therefore segmenting into node pools, each with a different hardware type gives the AI researcher and ML engineer better control of where to run.

• Quota control - splitting to node pools allows the admin to set specific quota per hardware type, e.g. give high priority project guaranteed access to advanced GPU hardware, while keeping lower priority project with a lower quota or even with no quota at all for that high-end GPU, but give it a “best-effort” access only (i.e. if the high priority guaranteed project is not using those resources).

• Multi-region or multi-availability-zone cloud environments - if some or all of your clusters run on the cloud (or even on-premise) but any of your clusters uses different physical locations or different topologies (e.g. racks), you probably want to segment your resources per region/zone/topology to be able to control where to run your workloads, how much quota to assign to specific environments (per project, per department), even if all those locations are all using the same hardware type. This methodology can help in optimizing the performance of your workloads because of the superior performance of local computing such as the locality of distributed workloads, local storage etc.

• Explainability and predictability - large environments are complex to understand, this becomes even more complex when an environment is loaded. To maintain users’ satisfaction and their understanding of the resources state, as well as to keep predictability of your workload chances to get scheduled, segmenting your cluster into smaller pools may significantly help.

• Scale - NVIDIA Run:ai implementation of node pools has many benefits, one of the main of them is scale. Each node pool has its own Scheduler instance, therefore allowing the cluster to handle more nodes and schedule workloads faster when segmented into node pools vs. one large cluster. To allow your workloads to use any resource within a cluster that is split to node pools, a second-level Scheduler is in charge of scheduling workloads between node pools according to your preferences and resource availability.

• Prevent mutual exclusion - Some AI workloads consume CPU-only resources, to prevent those workloads from consuming the CPU resources of GPU nodes and thus block GPU workloads from using those nodes, it is recommended to group CPU-only nodes into a dedicated node pool(s) and assign a quota for CPU projects to CPU node-pools only while keeping GPU node-pools with zero quota and optionally “best-effort” over quota access for CPU-only projects.

Grouping Examples

Set out below are illustrations of different grouping options.

Example: grouping nodes by topology

Example: grouping nodes by hardware type

Assigning Your Resources

After the initial grouping of resources, it is time to associate resources to AI initiatives, this is performed by assigning quotas to projects and optionally to departments. Assigning GPU quota to a project, on a node pool basis, means that the workloads submitted by that project are entitled to use those GPUs as guaranteed resources and can use them for all workload types.

However, what happens if the project requires more resources than its quota? This depends on the type of workloads that the user wants to submit. If the user requires more resources for non-preemptible workloads, then the quota must be increased, because non-preemptible workloads require guaranteed resources. On the other hand, if the type of workload is, for example, a model Training workload that is preemptible - in this case the project can exploit unused resources of other projects, as long as the other projects don’t need them. over quota is set per project on a node-pool basis and per department.

Administrators can use quota allocations to prioritize resources between users, teams, and AI initiatives. The administrator can completely prevent the use of certain node pools by a project or department by setting the node pool quota to 0 and disabling over quota for that node pool, or it can keep the quota to 0 and enable over quota to that node pool and allow access based on resource availability only (e.g. unused GPUs). However, when a project with a non-zero quota needs to use those resources, the Scheduler reclaims those resources back and preempts the preemptible workloads of over quota projects. As an administrator, you can also have an impact on the amount of over quota resources a project or department uses.

It is essential to make sure that the sum of all projects' quota does NOT surpass that of the Department, and that the sum of all departments does not surpass the number of physical resources, per node pool and for the entire cluster (we call such behavior - ‘over-subscription’). The reason over-subscription is not recommended is that it may produce unexpected scheduling decisions, especially those that might preempt ‘non-preemptive’ workloads or fail to schedule workloads within quota, either non-preemptible or preemptible, thus quota cannot be considered anymore as ‘guaranteed’. Admins can opt-in a system flag that helps to prevent over-subscription scenarios.

Example: assigning resources to projects

Assigning Users to Projects and Departments

NVIDIA Run:ai system is using ‘Role Based Access Control’ (RBAC) to manage users’ access rights to the different objects of the system, its resources, and the set of allowed actions. To allow AI researchers, ML engineers, Project Admins, or any other stakeholder of your AI initiatives to access projects and use AI compute resources with their AI initiatives, the administrator needs to assign users to projects. After a user is assigned to a project with the proper role, e.g. ‘L1 Researcher’, the user can submit and monitor its workloads under that project. Assigning users to departments is usually done to assign ‘Department Admin’ to manage a specific department. Other roles, such as ‘L1 Researcher’, can also be assigned to departments, this allows the researcher access to all projects within that department.

Scopes in an Organization

This is an example of an organization, as represented in the NVIDIA Run:ai platform:

The organizational tree is structured from top down under a single node headed by the account. The account is comprised of clusters, departments and projects.

After mapping and building your hierarchal structured organization as shown above, you can assign or associate various NVIDIA Run:ai components (e.g. workloads, roles, assets, policies, and more) to different parts of the organization - these organizational parts are the Scopes. The following organizational example consists of 5 optional scopes:

Next Steps

Now that resources are grouped into node pools, organizational units or business initiatives are mapped into projects and departments, projects’ quota parameters are set per node pool, and users are assigned to projects, you can finally submit workloads from a project and use compute resources to run your AI initiatives.

This section explains the procedure to manage workload policies.

Workload Policies Table

The Workload policies table can be found under Policies in the NVIDIA Run:ai platform.

The Workload policies table provides a list of all the policies defined in the platform, and allows you to manage them.

The Workload policies table consists of the following columns:

Customizing the Table View

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values

• Search - Click SEARCH and type the value to search by

• Sort - Click each column header to sort by

• Column selection - Click COLUMNS and select the columns to display in the table

• Refresh - Click REFRESH to update the table with the latest data

Adding a Policy

To create a new policy:

1. Click +NEW POLICY

2. Select a scope

3. Select the workload type

4. Click +POLICY YAML

5. In the YAML editor type or paste a YAML policy with defaults and rules. You can utilize the following references and examples:

   • Policy YAML reference

   • Policy YAML examples

6. Click SAVE POLICY

Editing a Policy

1. Select the policy you want to edit

2. Click EDIT

3. Update the policy and click APPLY

4. Click SAVE POLICY

Troubleshooting

Listed below are issues that might occur when creating or editing a policy via the YAML Editor:

Viewing a Policy

To view a policy:

1. Select the policy for which you want to view its policies.

2. Click VIEW POLICY

3. In the Policy form per workload section, view the workload rules and defaults:

   • Parameter The workload submission parameter that Rules and Defaults are applied to

   • Type (applicable for data sources only) The data source type (Git, S3, nfs, pvc etc.)

   • Default The default value of the Parameter

   • Rule Set up constraint on workload policy field

   • Source The origin of the applied policy (cluster, department or project)

Deleting a Policy

1. Select the policy you want to delete

2. Click DELETE

3. On the dialog, click DELETE to confirm the deletion

Using API

Go to the Policies API reference to view the available actions.

This section explains the procedure of managing reports in NVIDIA Run:ai.

Reports allow users to access and organize large amounts of data in a clear, CSV-formatted layout. They enable users to monitor resource consumption, analyze trends, and make data-driven decisions to optimize their AI workloads effectively.

Report Types

Currently, only “Consumption Reports” are available, which provides insights into the consumption of resources such as GPU, CPU, and CPU memory across organizational units.

Reports Table

The Reports table can be found under Analytics in the NVIDIA Run:ai platform.

The Reports table provides a list of all the reports defined in the platform and allows you to manage them.

Users are able to access the reports they have generated themselves. Users with project viewing permissions throughout the tenant can access all reports within the tenant.

The Reports table comprises the following columns:

Reports Status

The following table describes the reports' condition and whether they were created successfully:

Customizing the Table View

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values

• Search - Click SEARCH and type the value to search by

• Sort - Click each column header to sort by

• Column selection - Click COLUMNS and select the columns to display in the table

Creating a New Report

Before you start, make sure you have a project.

To create a new report:

1. Click +NEW REPORT

2. Enter a name for the report (if the name already exists, you will need to choose a different one)

3. Optional: Provide a description of the report

4. Set the report’s data collection period

   • Start date - The date at which the report data commenced

   • End date - The date at which the report data concluded

5. Set the report segmentation and filters

   • Filters - Filter by project or department name

   • Segment by - Data is collected and aggregated based on the segment

6. Click CREATE REPORT

Deleting a Report

1. Select the report you want to delete

2. Click DELETE

3. On the dialog, click DELETE to confirm

Downloading a report

1. Select the report you want to download

2. Click DOWNLOAD CSV

Enabling Reports for Self-Hosted Accounts

Reports must be saved in a storage solution compatible with S3. To activate this feature for self-hosted accounts, the storage needs to be linked to the account. The configuration should be incorporated into two ConfigMap objects within the Control Plane.

1. Edit the runai-backend-org-unit-service ConfigMap:

   Copy

   kubectl edit cm runai-backend-org-unit-service -n runai-backend

2. Add the following lines to the file:

   Copy

   S3_ENDPOINT: <S3_END_POINT_URL>
   S3_ACCESS_KEY_ID: <S3_ACCESS_KEY_ID>
   S3_ACCESS_KEY: <S3_ACCESS_KEY>
   S3_USE_SSL: "true"
   S3_BUCKET: <BUCKET_NAME>

3. Edit the runai-backend-metrics-service ConfigMap:

   Copy

   kubectl edit cm runai-backend-metrics-service -n runai-backend

4. Add the following lines to the file:

   Copy

   S3_ENDPOINT: <S3_END_POINT_URL>
   S3_ACCESS_KEY_ID: <S3_ACCESS_KEY_ID>
   S3_ACCESS_KEY: <S3_ACCESS_KEY>
   S3_USE_SSL: "true"

5. In addition on the same file, under config.yaml section, add the following right after log_level: \"Info\":

   Copy

   reports:\n s3_config:\n bucket: \"<BUCKET_NAME>\"\n

6. Restart the deployments:

   Copy

   kubectl rollout restart deployment runai-backend-metrics-service runai-backend-org-unit-service -n runai-backend

7. Refresh the page to see Reports under Analytics in the NVIDIA Run:ai platform.

Using API

To view the available actions, go to the Reports API reference.

Efficient resource allocation is critical for managing AI and compute-intensive workloads in Kubernetes clusters. The NVIDIA Run:ai Scheduler enhances Kubernetes' native capabilities by introducing advanced scheduling principles such as fairness, quota management, and dynamic resource balancing. It ensures that workloads, whether simple single-pod or complex distributed tasks, are allocated resources effectively while adhering to organizational policies and priorities.

This guide explores the NVIDIA Run:ai Scheduler’s allocation process, preemption mechanisms, and resource management. Through examples and detailed explanations, you'll gain insights into how the Scheduler dynamically balances workloads to optimize cluster utilization and maintain fairness across projects and departments.

Allocation Process

Pod Creation and Prouping

When a workload is submitted, the workload controller creates a pod or pods (for distributed training workloads or deployment based inference). When the Scheduler gets a submit request with the first pod, it creates a pod group and allocates all the relevant building blocks of that workload. The next pods of the same workload are attached to the same pod group.

Queue Management

A workload, with its associated pod group, is queued in the appropriate scheduling queue. In every scheduling cycle, the Scheduler ranks the order of queues by calculating their precedence for scheduling.

Resource Binding

The next step is for the Scheduler to find nodes for those pods, assign the pods to their nodes (bind operation), and bind other building blocks of the pods such as storage, ingress and so on. If the pod group has a gang scheduling rule attached to it, the Scheduler either allocates and binds all pods together, or puts all of them into pending state. It retries to schedule them all together in the next scheduling cycle. The Scheduler also updates the status of the pods and their associated pod group. Users are able to track the workload submission process both in the CLI or NVIDIA Run:ai UI. For more details on submitting and managing workloads, see Workloads.

Preemption

If the Scheduler cannot find resources for the submitted workloads (and all of its associated pods), and the workload deserves resources either because it is under its queue quota fairshare, the Scheduler tries to reclaim resources from other queues. If this does not solve the resource issue, the Scheduler tries to preempt lower priority preemptible workloads within the same queue (project).

Reclaim Preemption Between Projects and Departments

Reclaim is an inter-project and inter-department resource balancing action that takes back resources from one project or department that has used them as an over quota. It returns the resources back to a project (or department) that deserves those resources as part of its deserved quota, or to balance fairness between projects (or departments), so a project (or department) does not exceed its fairshare (portion of the unused resources).

This mode of operation means that a lower priority workload submitted in one project (e.g. training) can reclaim resources from a project that runs a higher priority workload (e.g. preemptive workspace) if fairness balancing is required.

Priority Preemption Within a Project

Higher priority workloads may preempt lower priority preemptible workloads within the same project/node pool queue. For example, in a project that runs a training workload that exceeds the project quota for a certain node pool, a newly submitted workspace within the same project/node pool may stop (preempt) the training workload if there are not enough over quota resources for the project within that node pool to run both workloads (e.g. workspace using in-quota resources and training using over quota resources).

Quota, Over Quota, and Fairshare

The NVIDIA Run:ai Scheduler strives to ensure fairness between projects and between departments. This means each department and project always strive to get their deserved quota, and unused resources are split between projects according to known rules (e.g. over quota weights).

If a project needs more resources even beyond its fairshare, and the Scheduler finds unused resources that no other project needs, this project can consume resources even beyond its fairshare.

Some scenarios can prevent the Scheduler from fully providing deserved quota and fairness:

• Fragmentation or other scheduling constraints such affinities, taints etc.

• Some requested resources, such as GPUs and CPU memory, can be allocated, while others, like CPU cores, are insufficient to meet the request. As a result, the Scheduler will place the workload in a pending state until the required resource becomes available.

Example of Splitting Quota

The example below illustrates a split of quota between different projects and departments using several node pools:

The example below illustrates how fairshare is calculated per project/node pool for the above example:

• For each Project:

  • The over quota (OQ) portion of each project (per node pool) is calculated as:

  [(OQ-Weight) / (Σ Projects OQ-Weights)] x (Unused Resource per node pool)

  • Fairshare is calculated as the sum of quota + over quota.

• In Project 2, we assume that out of the 36 available GPUs in node pool A, 20 GPUs are currently unused. This means either these GPUs are not part of any project’s quota, or they are part of a project’s quota but not used by any workloads of that project:

  • Project 2 over quota share:

    [(Project 2 OQ-Weight) / (Σ all Projects OQ-Weights)] x (Unused Resource within node pool A)

    [(3) / (2 + 3 + 1)] x (20) = (3/6) x 20 = 10 GPUs

  • Fairshare = deserved quota + over quota = 6 +10 = 16 GPUs. Similarly, fairshare is also calculated for CPU and CPU memory. The Scheduler can grant a project more resources than its fairshare if the Scheduler finds resources not required by other projects that may deserve those resources.

• In Project 3, fairshare = deserved quota + over quota = 0 +3 = 3 GPUs. Project 3 has no guaranteed quota, but it still has a share of the excess resources in node pool A. The NVIDIA Run:ai Scheduler ensures that Project 3 receives its part of the unused resources for over quota, even if this results in reclaiming resources from other projects and preempting preemptible workloads.

Fairshare Balancing

The Scheduler constantly re-calculates the fairshare of each project and department per node pool, represented in the scheduler as queues, resulting in the re-balancing of resources between projects and between departments. This means that a preemptible workload that was granted resources to run in one scheduling cycle, can find itself preempted and go back to pending state while waiting for resources in the next cycle.

A queue, representing a scheduler-managed object for each project or department per node pool, can be in one of 3 states:

• In-quota: The queue’s allocated resources ≤ queue deserved quota. The Scheduler’s first priority is to ensure each queue receives its deserved quota.

• Over quota but below fairshare: The queue’s deserved quota < queue’s allocated resources <= queue’s fairshare. The Scheduler tries to find and allocate more resources to queues that need resources beyond their deserved quota and up to their fairshare.

• Over-fairshare and over quota: The queue’s fairshare < queue’s allocated resources. The Scheduler tries to allocate resources to queues that need even more resources beyond their fairshare.

When re-balancing resources between queues of different projects and departments, the Scheduler goes in the opposite direction, i.e. first take resources from over-fairshare queues, then from over quota queues, and finally, in some scenarios, even from queues that are below their deserved quota.

Next Steps

Now that you have gained insights into how the Scheduler dynamically balances workloads to optimize cluster utilization and maintain fairness across projects and departments, you can submit workloads. Before submitting your workloads, it’s important to familiarize yourself with the following key topics:

• Introduction to workloads - Learn what workloads are and what is supported for both NVIDIA Run:ai and third-party workloads.

• NVIDIA Run:ai workload types - Explore the various NVIDIA Run:ai workload types available and understand their specific purposes to enable you to choose the most appropriate workload type for your needs.

The workload priority management feature allows you to change the priority of a workload within a project. The priority determines the workload's position in the project scheduling queue managed by the NVIDIA Run:ai Scheduler. By adjusting the priority, you can increase the likelihood that a workload will be scheduled and preferred over others within the same project, ensuring that critical tasks are given higher priority and resources are allocated efficiently.

You can change the priority of a workload by selecting one of the predefined values from the NVIDIA Run:ai priority dictionary. This can be done using the NVIDIA Run:ai UI, API or CLI, depending on the workload type.

Priority Dictionary

Workload priority is defined by selecting a string name from a predefined list in the NVIDIA Run:ai priority dictionary. Each string corresponds to a specific Kubernetes PriorityClass, which in turn determines scheduling behavior, such as whether the workload is preemptible or allowed to run over quota.

Preemptible vs Non-Preemptible Workloads

• Non-preemptible workloads must run within the project’s deserved quota, cannot use over-quota resources, and will not be interrupted once scheduled.

• Preemptible workloads can use opportunistic compute resources beyond the project’s quota but may be interrupted at any time.

Default Priority per Workload

Both NVIDIA Run:ai and third-party workloads are assigned a default priority. The below table shows the default priority per workload type:

Supported Priority Overrides per Workload

The below table shows the default priority listed in the previous section and the supported override options per workload:

How to Override Priority

You can override the default priority when submitting a workload through the UI, API, or CLI depending on the workload type.

Workspaces

To use the override options:

• UI: Enable "Allow the workload to exceed the project quota" when submitting a workspace

• API: Set PriorityClass in the Workspaces API

• CLI: Submit a workspace using the --priority flag

  Copy

  runai workspace submit --priority priority-class

Training Workloads

To use the override options:

• API: Set PriorityClass in the Trainings API

• CLI: Submit training using the --priority flag

  Copy

  runai training submit --priority priority-class

Single Sign-On (SSO) is an authentication scheme, allowing users to log-in with a single pair of credentials to multiple, independent software systems.

This article explains the procedure to to NVIDIA Run:ai using the OpenID Connect protocol in OpenShift V4.

Prerequisites

Before starting, make sure you have the following available from your OpenShift cluster:

• :

  • ClientID - The ID used to identify the client with the Authorization Server.

  • Client Secret - A secret password that only the Client and Authorization Server know.

• Base URL - The OpenShift API Server endpoint (for example, )

Setup

Adding the Identity Provider

1. Go to General settings

2. Open the Security section and click +IDENTITY PROVIDER

3. Select OpenShift V4

4. Enter the Base URL, Client ID, and Client Secret from your OpenShift OAuth client.

5. Copy the Redirect URL to be used in your OpenShift OAuth client

6. Optional: Enter the user attributes and their value in the identity provider as shown in the below table

7. Click SAVE

8. Optional: Enable Auto-Redirect to SSO to automatically redirect users to your configured identity provider’s login page when accessing the platform.

Testing the Setup

1. Open the NVIDIA Run:ai platform as an admin

2. Add to an SSO user defined in the IDP

3. Open the NVIDIA Run:ai platform in an incognito browser tab

4. On the sign-in page click CONTINUE WITH SSO You are redirected to the OpenShift IDP sign-in page

5. In the identity provider sign-in page, log in with the SSO user who you granted with access rules

6. If you are unsuccessful signing-in to the identity provider, follow the section below

Editing the Identity Provider

You can view the identity provider details and edit its configuration:

1. Go to General settings

2. Open the Security section

3. On the identity provider box, click Edit identity provider

4. You can edit either the Base URL, Client ID, Client Secret, or the User attributes

Removing the Identity Provider

You can remove the identity provider configuration:

1. Go to General settings

2. Open the Security section

3. On the identity provider card, click Remove identity provider

4. In the dialog, click REMOVE to confirm

Troubleshooting