This section includes release information for the self-hosted version of NVIDIA Run:ai:

• New Features and Enhancements - Highlights major updates introduced in each version, including new capabilities, UI improvements, and changes to system behavior..

• Hotfixes - Lists patches applied to released versions, including critical fixes and behavior corrections.

Feature Life Cycle

NVIDIA Run:ai uses life cycle labels to indicate the maturity and stability of features across releases:

• Experimental - This feature is in early development. It may not be stable and could be removed or changed significantly in future versions. Use with caution.

• Beta - This feature is still being developed for official release in a future version and may have some limitations. Use with caution.

• Legacy - This feature is scheduled to be removed in future versions. We recommend using alternatives if available. Use only if necessary.

NVIDIA Run:ai is a GPU orchestration and optimization platform that helps organizations maximize compute utilization for AI workloads. By optimizing the use of expensive compute resources, NVIDIA Run:ai accelerates AI development cycles, and drives faster time-to-market for AI-powered innovations.

Built on Kubernetes, NVIDIA Run:ai supports dynamic GPU allocation, workload submission, workload scheduling, and resource sharing, ensuring that AI teams get the compute power they need while IT teams maintain control over infrastructure efficiency.

NVIDIA Run:ai centralizes cluster management and optimizes infrastructure control by offering:

• - Manage all clusters from a single platform, ensuring consistency and control across environments.

As part of the installation process, you will install:

• A managing cluster/s

• One or more

Both the control plane and clusters require Kubernetes. Typically, the control plane and first cluster are installed on the same Kubernetes cluster.

The self-hosted option is for organizations that cannot use a SaaS solution due to data leakage concerns. NVIDIA Run:ai self-hosting comes with two variants:

Deploying NVIDIA Run:ai in mission-critical environments requires proper monitoring and maintenance of resources to ensure workloads run and are deployed as expected.

Details on how to monitor different parts of the physical resources in your Kubernetes system, including and , can be found in the monitoring and maintenance section. Adjacent configuration and troubleshooting sections also cover high availability, and clusters, , and to meet compliance requirements.

In addition to monitoring NVIDIA Run:ai resources, it is also highly recommended to monitor NVIDIA Run:ai runs on Kubernetes, which manages containerized applications. In particular, focus on three main layers:

This is the highest layer and includes the parts of NVIDIA Run:ai pods, which run in containers managed by Kubernetes.

This layer includes the main Kubernetes system that runs and manages NVIDIA Run:ai components. Important elements to monitor include:

Uninstall the Control Plane

To delete the control plane, run:

helm uninstall runai-backend -n runai-backend

Uninstall the Cluster

Uninstall of NVIDIA Run:ai cluster from the Kubernetes cluster does not delete existing projects, departments or workloads submitted by users.

To uninstall the NVIDIA Run:ai cluster, run the following helm command in your terminal:

To remove the NVIDIA Run:ai cluster from the NVIDIA Run:ai Platform, see Removing a cluster.

This article explains the procedure to create your own user applications.

Applications are used for API integrations with NVIDIA Run:ai. An application contains a client ID and a client secret. With the client credentials, you can obtain a token as detailed in and use it within subsequent API calls.

To create an application:

1. Click the user avatar at the top right corner, then select Settings

This section details the security considerations for deploying NVIDIA Run:ai. It is intended to help administrators and security officers understand the specific permissions required by NVIDIA Run:ai.

NVIDIA Run:ai integrates with Kubernetes clusters and requires specific permissions to successfully operate. These are permissions are controlled with configuration flags that dictate how NVIDIA Run:ai interacts with cluster resources. Prior to installation, security teams can review the permissions and ensure it aligns with their organization’s policies.

NVIDIA Run:ai provides various security-related permissions that can be customized to fit specific organizational needs. Below are brief descriptions of the key use cases for these customizations:

Shared storage is a critical component in AI and machine learning workflows, particularly in scenarios involving distributed training and shared datasets. In AI and ML environments, data must be readily accessible across multiple nodes, especially when training large models or working with vast datasets. Shared storage enable seamless access to data, ensuring that all nodes in a distributed training setup can read and write to the same datasets simultaneously. This setup not only enhances efficiency but is also crucial for maintaining consistency and speed in high-performance computing environments.

While NVIDIA Run:ai Platform supports a variety of remote data sources, such as Git and S3, it is often more efficient to keep data close to the compute resources. This proximity is typically achieved through the use of shared storage, accessible to multiple nodes in your Kubernetes cluster.

When implementing shared storage in Kubernetes, there are two primary approaches:

• Utilizing the of your storage provider (Recommended)

NVIDIA Run:ai assets are preconfigured building blocks that simplify the workload submission effort and remove the complexities of Kubernetes and networks for AI practitioners.

Workload assets enable organizations to:

• Create and reuse preconfigured setup for code, data, storage and resources to be used by AI practitioners to simplify the process of submitting workloads

• Share the preconfigured setup with a wide audience of AI practitioners with similar needs

NVIDIA Run:ai authentication and authorization enables a streamlined experience for the user with precise controls covering the data each user can see and the actions each user can perform in the NVIDIA Run:ai platform.

Authentication verifies user identity during login, and authorization assigns the user with specific permissions according to the assigned access rules.

Authenticated access is required to use all aspects of the NVIDIA Run:ai interfaces, including the NVIDIA Run:ai platform, the NVIDIA Run:ai Command Line Interface (CLI) and APIs.

Authentication

There are multiple methods to authenticate and access NVIDIA Run:ai.

Single Sign-On (SSO)

NVIDIA Run:ai supports three methods to set up SSO:

When using SSO, it is highly recommended to manage at least one local user, as a breakglass account (an emergency account), in case access to SSO is not possible.

Username and password access can be used when SSO integration is not possible.

Secret is the authentication method for . Applications use the NVIDIA Run:ai APIs to perform automated tasks including scripts and pipelines based on their assigned .

The NVIDIA Run:ai platform uses Role Base Access Control (RBAC) to manage authorization. Once a user or an application is authenticated, they can perform actions according to their assigned access rules.

While Kubernetes RBAC is limited to a single cluster, NVIDIA Run:ai expands the scope of Kubernetes RBAC, making it easy for administrators to manage access rules across multiple clusters.

RBAC at NVIDIA Run:ai is configured using access rules. An access rule is the assignment of a to a : <Subject> is a <Role> in a <Scope>.

• Subject

  • A user, a group, or an application assigned with the role

• Role

Below is an example of an access rule: username@company.com is a Department admin in Department: A

Karpenter is an open-source, Kubernetes cluster autoscaler built for cloud deployments. Karpenter optimizes the cloud cost of a customer’s cluster by moving workloads between different node types, consolidating workloads into fewer nodes, using lower-cost nodes where possible, scaling up new nodes when needed, and shutting down unused nodes.

Karpenter’s main goal is cost optimization. Unlike Karpenter, NVIDIA Run:ai’s Scheduler optimizes for fairness and resource utilization. Therefore, there are a few potential friction points when using both on the same cluster.

Friction Points Using Karpenter with NVIDIA Run:ai

1. Karpenter looks for “unschedulable” pending workloads and may try to scale up new nodes to make those workloads schedulable. However, in some scenarios, these workloads may exceed their quota parameters, and the NVIDIA Run:ai Scheduler will put them into a pending state.

2. Karpenter is not aware of the NVIDIA Run:ai fractions mechanism and may try to interfere incorrectly.

3. Karpenter preempts any type of workload (i.e., high-priority, non-preemptible workloads will potentially be interrupted and moved to save cost).

4. Karpenter has no pod-group (i.e., workload) notion or gang scheduling awareness, meaning that Karpenter is unaware that a set of “arbitrary” pods is a single workload. This may cause Karpenter to schedule those pods into different node pools (in the case of multi-node-pool workloads) or scale up or down a mix of wrong nodes.

NVIDIA Run:ai Scheduler mitigates the friction points using the following techniques (each numbered bullet below corresponds to the related friction point listed above):

1. Karpenter uses a “nominated node” to recommend a node for the Scheduler. The NVIDIA Run:ai Scheduler treats this as a “preferred” recommendation, meaning it will try to use this node, but it’s not required and it may choose another node.

2. Fractions - Karpenter won’t consolidate nodes with one or more pods that cannot be moved. The NVIDIA Run:ai reservation pod is marked as ‘do not evict’ to allow the NVIDIA Run:ai Scheduler to control the scheduling of fractions.

3. Non-preemptible workloads - NVIDIA Run:ai marks non-preemptible workloads as ‘do not evict’ and Karpenter respects this annotation.

• Using multi-node-pool workloads

  • Workloads may include a list of optional node pools. Karpenter is not aware that only a single node pool should be selected out of that list for the workload. It may therefore recommend putting pods of the same workload into different node pools and may scale up nodes from different node pools to serve a “multi-node-pool” workload instead of nodes on the selected single node pool.

  • If this becomes an issue (i.e., if Karpenter scales up the wrong node types), users can set an inter-pod affinity using the node pool label or another common label as a ‘topology’ identifier. This will force Karpenter to choose nodes from a single-node pool per workload, selecting from any of the node pools listed as allowed by the workload.

This guide outlines the best practices for configuring the NVIDIA Run:ai platform to ensure high availability and maintain service continuity during system failures or under heavy load. The goal is to reduce downtime and eliminate single points of failure by leveraging Kubernetes best practices alongside NVIDIA Run:ai specific configuration options. The NVIDIA Run:ai platform relies on two fundamental high availability strategies:

• Use of system nodes - Assigning multiple dedicated nodes for critical system services ensures control, resource isolation, and enables system-level scaling.

• Replication of core and third-party services - Configuring multiple replicas of essential services, including both platform and third-party components, distributes workloads and reduces single points of failure. If a component fails on one node, requests can seamlessly route to another instance.

System Nodes

The NVIDIA Run:ai platform allows you to dedicate specific nodes (system nodes) exclusively for core platform services. This approach provides improved operational isolation and easier resource management.

Ensure that at least three system nodes are configured to support high availability. If you use only a single node for core services, horizontally scaled components will not be distributed, resulting in a single point of failure. See for more details. This practice applies to both the NVIDIA Run:ai cluster and control plane (self-hosted).

The NVIDIA Run:ai control plane runs in the runai-backend namespace and consists of multiple Kubernetes and . To achieve high availability, it is recommended to configure multiple replicas during installation or upgrade using Helm flags.

In addition, the control plane supports autoscaling for certain services to handle variable load and improve system resiliency. Autoscaling can be enabled or configured during installation or upgrade using Helm flags.

Each of the NVIDIA Run:ai deployments can be set to scale up, by adding a helm settings on install/upgrade. For a full list of settings, contact NVIDIA Run:ai support.

To increase the replica count, use the following NVIDIA Run:ai control plane Helm flag:

NVIDIA Run:ai uses the following third-party components which are managed as Kubernetes StatefulSets. For more information, see :

• PostgreSQL - The internal PostgreSQL cannot be scaled horizontally. To connect NVIDIA Run:ai to an external PostgreSQL service which can be configured for high availability, see .

• Thanos - To enable Thanos autoscaling, use the following NVIDIA Run:ai control plane helm flags:

• Keycloak - By default, Keycloak sets a minimum of 3 pods and will scale to more on transaction load. To scale Keycloak, use the following NVIDIA Run:ai control plane helm flags:

By default, NVIDIA Run:ai cluster services are deployed with a single replica. To achieve high availability, it is recommended to configure multiple replicas for core NVIDIA Run:ai services. For more information, see .

NVIDIA Run:ai provides metrics and telemetry for both physical cluster entities such as clusters, nodes, and node pools and application organization entities such as departments and projects. Metrics represent over-time data while telemetry represents current analytics data. This data is essential for monitoring and analyzing the performance and health of your platform.

Consuming Metrics and Telemetry Data

Users can consume the data based on their permissions:

1. API - Access the data programmatically through the NVIDIA Run:ai API.

2. CLI - Use the NVIDIA Run:ai Command Line Interface to query and manage the data.

3. UI - Visualize the data through the NVIDIA Run:ai user interface.

• Metrics API - Access over-time detailed analytics data programmatically.

• Telemetry API - Access current analytics data programmatically.

Refer to to see the full list of supported metrics and telemetry APIs.

Use the list and describe commands to fetch and manage the data. See for more details.

Refer to to see the full list of supported metrics and telemetry.

• Overview dashboard - Provides a high-level summary of the cluster's health and performance, including key metrics such as GPU utilization, memory usage, and node status. Allows administrators to quickly identify any potential issues or areas for optimization. Offers advanced analytics capabilities for analyzing GPU usage patterns and identifying trends. Helps administrators optimize resource allocation and improve cluster efficiency.

• Quota management - Enables administrators to monitor and manage GPU quotas across the cluster. Includes features for setting and adjusting quotas, tracking usage, and receiving alerts when quotas are exceeded.

• Workload visualizations - Provides detailed insights into the resource usage and utilization of each GPU in the cluster. Includes metrics such as GPU memory utilization, core utilization, and power consumption. Allows administrators to identify GPUs that are under-utilized and overloaded.

Data volumes offer a powerful solution for storing, managing, and sharing AI training data within the NVIDIA Run:ai platform. They promote collaboration, simplify data access control, and streamline the AI development lifecycle.

Acting as a central repository for organizational data resources, data volumes can represent datasets or raw data, that is stored in Kubernetes Persistent Volume Claims (PVCs).

Why Use a Data Volume?

1. Sharing with multiple scopes Unlike other NVIDIA Run:ai data sources, data volumes can be shared across projects, departments, or clusters, encouraging data reuse and collaboration within the organization.

2. Storage saving A single copy of the data can be used across multiple

1. Sharing large data sets In large organizations, the data is often stored in a remote location, which can be a barrier for large model training. Even if the data is transferred into the cluster, sharing it easily with multiple users is still challenging. Data volumes can help share the data seamlessly, with maximum security and control.

2. Sharing data with colleagues When sharing training results, generated data sets, or other artifacts with team members is needed, data volumes can help make the data available easily.

To create a data volume, there must be a with a PVC in its namespace.

Working with data volumes is currently available using the API. To view the available actions, go to the API reference.

Data volume creation is limited to .

Data volume sharing (adding scopes) is limited to .

Once created, the data volume is available to its originating project (see the prerequisites above).

Data volumes can be shared with additional scopes in the organization.

Data volumes are used when . Any user, application or SSO group with a that has permissions to create workloads can also use data volumes.

Researchers can list available data volumes within their permitted scopes for easy selection.

Before proceeding with the upgrade, it's crucial to apply the specific prerequisites associated with your current version of NVIDIA Run:ai and every version in between up to the version you are upgrading to.

To ensure a smooth and supported upgrade process:

• Align control plane and cluster versions - For best results, upgrade the control plane and cluster components to the same NVIDIA Run:ai version during the same maintenance window. Keeping versions aligned helps avoid unexpected behavior caused by version mismatches and ensures full compatibility across platform components.

• Upgrade order - When performing an upgrade:

This section explains the available configurations for customizing the NVIDIA Run:ai control plane and cluster installation.

The NVIDIA Run:ai control plane installation can be customized to support your environment via Helm or flags. See .

The NVIDIA Run:ai cluster installation can be customized to support your environment via Helm or flags.

These configurations are saved in the runaiconfig Kubernetes object and can be edited post-installation as needed. For more information, see .

The following table lists the available Helm chart values that can be configured to customize the NVIDIA Run:ai cluster installation.

This section provides instructions for IT administrators on collecting NVIDIA Run:ai logs for support, including prerequisites, CLI commands, and log file retrieval. It also covers enabling verbose logging for Prometheus and the NVIDIA Run:ai Scheduler.

To collect NVIDIA Run:ai logs, follow these steps:

• Ensure that you have administrator-level access to the Kubernetes cluster where NVIDIA Run:ai is installed.

• The NVIDIA Run:ai (CLI) must be installed.

By default, Kubernetes uses its own native scheduler to determine pod placement. The NVIDIA Run:ai platform provides a custom scheduler, runai-scheduler, which is used by default for workloads submitted using the platform. This section outlines how to configure third-party workloads, such as those submitted directly to Kubernetes, to run with the , runai-scheduler, instead of the default Kubernetes scheduler.

To use the NVIDIA Run:ai Scheduler for third-party workloads, specify it in the workload’s YAML file. This instructs Kubernetes to schedule the workload using the NVIDIA Run:ai Scheduler instead of the default one.

For example:

If modifying the workload YAML is not possible, you can enforce the use of the NVIDIA Run:ai Scheduler for all workloads in a given namespace (i.e., NVIDIA Run:ai ) by applying an annotation. Once applied, all workloads submitted to the annotated namespace will automatically use the NVIDIA Run:ai Scheduler without requiring individual YAML modifications.

The following section provides the information needed to prepare for a NVIDIA Run:ai installation.

Software Artifacts

The following software artifacts should be used when installing the control plane and cluster.

Kubernetes

OpenShift

Private Docker Registry (Optional)

Set Up Your Environment

External Postgres Database (Optional)

If you have opted to use an external PostgreSQL database, you need to perform initial setup to ensure successful installation. Follow these steps:

1. Create a SQL script file, edit the parameters below, and save it locally:

   • Replace <DATABASE_NAME> with a dedicate database name for NVIDIA Run:ai in your PostgreSQL database.

   • Replace <ROLE_NAME> with a dedicated role name (user) for NVIDIA Run:ai database.

   • Replace <ROLE_PASSWORD> with a password for the new PostgreSQL role.

   • Replace <GRAFANA_PASSWORD> with the password to be set for Grafana integration.

2. Run the following command on a machine where PostgreSQL client (pgsql) is installed:

   • Replace <POSTGRESQL_HOST> with the PostgreSQL ip address or hostname.

   • Replace <POSTGRESQL_USER> with the PostgreSQL username.

This section explains the procedure to manage Access rules.

Access rules provide users, groups, or applications privileges to system entities. An access rule is the assignment of a role to a subject in a scope: <Subject> is a <Role> in a <Scope>. For example, user user@domain.com is a department admin in department A.

Access Rules Table

The Access rules table can be found under Access in the NVIDIA Run:ai platform.

The Access rules table provides a list of all the access rules defined in the platform and allows you to manage them.

The Access rules table consists of the following columns:

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values

• Search - Click SEARCH and type the value to search by

• Sort - Click each column header to sort by

• Column selection - Click COLUMNS and select the columns to display in the table

To add a new access rule:

1. Click +NEW ACCESS RULE

2. Select a subject User, SSO Group, or Application

3. Select or enter the subject identifier:

Access rules cannot be edited. To change an access rule, you must delete the rule, and then create a new rule to replace it.

1. Select the access rule you want to delete

2. Click DELETE

3. On the dialog, click DELETE to confirm

Go to the API reference to view the available actions.

This section explains the procedure to manage users and their permissions.

Users can be managed locally, or via the identity provider (Idp), while assigned with access rules to manage permissions. For example, user user@domain.com is a department admin in department A.

Users Table

The Users table can be found under Access in the NVIDIA Run:ai platform.

The users table provides a list of all the users in the platform. You can manage users and user permissions (access rules) for both local and SSO users.

The Users table consists of the following columns:

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values

• Search - Click SEARCH and type the value to search by

• Sort - Click each column header to sort by

• Column selection - Click COLUMNS and select the columns to display in the table

To create a local user:

1. Click +NEW LOCAL USER

2. Enter the user's Email address

3. Click CREATE

4. Review and copy the user's credentials:

To create an access rule:

1. Select the user you want to add an access rule for

2. Click ACCESS RULES

3. Click +ACCESS RULE

4. Select a role

To delete an access rule:

1. Select the user you want to remove an access rule from

2. Click ACCESS RULES

3. Find the access rule assigned to the user you would like to delete

4. Click on the trash icon

To reset a user's password:

1. Select the user you want to reset it's password

2. Click RESET PASSWORD

3. Click RESET

4. Review and copy the user's credentials:

1. Select the user you want to delete

2. Click DELETE

3. In the dialog, click DELETE to confirm

Go to the , API reference to view the available actions.

The identity of the user inside a container determines its access to various resources. For example, network file systems often rely on this identity to control access to mounted volumes. As a result, propagating the correct user identity into a container is crucial for both functionality and security.

By default, containers in both Docker and Kubernetes run as the root user. This means any process inside the container has full administrative privileges, capable of modifying system files, installing packages, or changing configurations.

While this level of access provides researchers with maximum flexibility, it conflicts with modern enterprise security practices. If the container’s root identity is propagated to external systems (e.g., network-attached storage), it can result in elevated permissions outside the container, increasing the risk of security breaches.

NVIDIA Run:ai Controls for User Identity and Privileges

NVIDIA Run:ai allows you to enhance security and enforce organizational policies by:

• Controlling root access and privilege escalation within containers

• Propagating the user identity to align with enterprise access policies

NVIDIA Run:ai supports security-related workload configurations to control user permissions and restrict privilege escalation. These options are available via the API and CLI during workload creation:

• runAsNonRoot / --run-as-user - Force the container to run as non-root user.

• allowPrivilegeEscalation / --allow-privilege-escalation - Allow the container to use setuid binaries to escalate privileges, even when running as a non-root user. This setting can increase security risk and should be disabled if elevated privileges are not required.

Administrators can enforce secure defaults across the environment using , ensuring consistent workload behavior aligned with organizational security practices.

A best practice is to store the User Identifier (UID) and Group Identifier (GID) in the organization's directory. NVIDIA Run:ai allows you to pass these values to the container and use them as the container identity. To perform this, you must set up and perform the steps for UID/GID integration.

It is possible to explicitly pass user identity when creating an or submitting a :

• From the image - Use the UID/GID defined in the container image.

• From the IdP token - Use identity attributes provided by the SSO identity provider (available only in SSO-enabled installations).

• Custom - Manually set the User ID (UID), Group ID (GID) and supplementary groups that can run commands in the container.

Administrators can enforce secure defaults across the environment using , ensuring consistent workload behavior aligned with organizational security practices.

In OpenShift, Security Context Constraints (SCCs) manage pod-level security, including root access. By default, containers are assigned a random non-root UID, and flags such as --run-as-user and --allow-privilege-escalation are disabled.

On non-OpenShift Kubernetes clusters, similar enforcement can be achieved using tools like , which applies system-level policies to restrict containers from running as root.

By default, OpenShift restricts setting specific user and group IDs (UIDs/GIDs) in workloads through its SCCs. To allow NVIDIA Run:ai workloads to run with explicitly defined UIDs and GIDs, a cluster administrator must modify the relevant SCCs.

To enable UID and GID assignment:

1. Edit the runai-user-job SCC:

2. Edit the runai-jupyter-notebook SCC (only required if using Jupyter environments):

3. In both SCC definitions, ensure the following sections are configured:

These settings allow NVIDIA Run:ai to pass specific UID and GID values into the container, enabling compatibility with identity-aware file systems and enterprise access controls.

When containers run as a specific user, the user must have a home directory defined within the image. Otherwise, starting a shell session will fail due to the absence of a home directory.

Since pre-creating a home directory for every possible user is impractical, NVIDIA Run:ai offers the createHomeDir / --create-home-dir option. When enabled, this flag creates a temporary home directory for the user inside the container at runtime. By default, the directory is created at /home/<username>.

Operating NVIDIA Run:ai at scale ensures that the system can efficiently handle fluctuating workloads while maintaining optimal performance. As clusters grow, whether due to an increasing number of nodes or a surge in workload demand, NVIDIA Run:ai services must be appropriately tuned to support large-scale environments.

This guide outlines the best practices for optimizing NVIDIA Run:ai for high-performance deployments, including NVIDIA Run:ai system services configurations, vertical scaling (adjusting CPU and memory resources) and where applicable, horizontal scaling (replicas).

NVIDIA Run:ai Services

Vertical Scaling

Each of the NVIDIA Run:ai containers has default resource requirements that reflect an average customer load. With significantly larger cluster loads, certain NVIDIA Run:ai services will require more CPU and memory resources. NVIDIA Run:ai supports configuring these resources for each NVIDIA Run:ai service group separately. For instructions and more information, see NVIDIA Run:ai services resource management.

Scheduling Services

The scheduling services group should be scaled together with the number of nodes and the number of workloads handled by the Scheduler (running / pending). These resource recommendations are based on internal benchmarks performed on stressed environments:

The sync and workload service groups are less sensitive for scale. The recommendation for large or intensive environments is set to the following:

By default, NVIDIA Run:ai cluster services are deployed with a single replica. For large scale and intensive environments it is recommended to scale the NVIDIA Run:ai services horizontally by increasing the number of replicas. For more information, see .

NVIDIA Run:ai relies on to scrape cluster metrics and forward them to the NVIDIA Run:ai control plane. The volume of metrics generated is directly proportional to the number of nodes, workloads, and projects in the system. When operating at scale—reaching hundreds, and thousands of nodes and projects—the system generates a significant volume of metrics which can place a strain on the cluster and the network bandwidth.

To mitigate this impact, it is recommended to tune the Prometheus configurations. See to read more about the tuning parameters available via the remote write configuration and refer to this for optimizing Prometheus remote write performance.

You can apply the remote-write configurations required as described in

The following example demonstrates the recommended approach in NVIDIA Run:ai for tuning Prometheus remote-write configurations:

For clusters with more than 32 nodes (SuperPod and larger), increase the replica count for key control plane services to 2.

To set the replica count, use the following NVIDIA Run:ai control plane Helm flag:

Replicas for following services should not be increased: postgres, keycloak, grafana, thanos, nats, redoc, cluster-migrator, identity provider reconciler, settings migrator.

For Grafana, enable autoscaling first and then set the number of minReplicas. Use the following NVIDIA Run:ai control plane Helm flags:

is the third-party used by NVIDIA Run:ai to store metrics under a significant user load. Use the following NVIDIA Run:ai control plane Helm flags to increase resources for the Thanos query function:

This article explains how to designate specific node roles in a Kubernetes cluster to ensure optimal performance and reliability in production deployments.

For optimal performance in production clusters, it is essential to avoid extensive CPU usage on GPU nodes where possible. This can be done by ensuring the following:

• NVIDIA Run:ai system-level services run on dedicated CPU-only nodes.

• Workloads that do not request GPU resources (e.g. Machine Learning jobs) are executed on CPU-only nodes.

NVIDIA Run:ai services are scheduled on the defined node roles by applying using node labels .

To perform these tasks, make sure to install the NVIDIA Run:ai .

The following node roles can be configured on the cluster:

• System node: Reserved for NVIDIA Run:ai system-level services.

• GPU Worker node: Dedicated for GPU-based workloads.

• CPU Worker node: Used for CPU-only workloads.

NVIDIA Run:ai system nodes run system-level services required to operate. This can be done via the (preferred method) or via NVIDIA Run:ai .

By default, NVIDIA Run:ai applies a node affinity rule to prefer nodes that are labeled with node-role.kubernetes.io/runai-system for system services scheduling. You can modify the default node affinity rule by:

• Editing the spec.global.affinity configuration parameter as detailed in .

• Editing the global.affinity configuration as detailed in for self-hosted deployments

To set a system role for a node in your Kubernetes cluster using Kubectl, follow these steps:

1. Use the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

2. Run one of the following commands to label the node with its role:

To set a system role for a node in your Kubernetes cluster, follow these steps:

1. Run the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

2. Run one of the following commands to set or remove a node’s role

The set node-role command will label the node and set relevant cluster configurations.

NVIDIA Run:ai worker nodes run user-submitted workloads and system-level required to operate. This can be managed via the (preferred method) or via NVIDIA Run:ai ,

By default, GPU workloads are scheduled on GPU nodes based on the nvidia.com/gpu.present label. When global.nodeAffinity.restrictScheduling is set to true via the :

• GPU Workloads are scheduled with node affinity rule to require nodes that are labeled with node-role.kubernetes.io/runai-gpu-worker

• CPU-only Workloads are scheduled with node affinity rule to require nodes that are labeled with node-role.kubernetes.io/runai-cpu-worker

To set a worker role for a node in your Kubernetes cluster using Kubectl, follow these steps:

1. Validate the global.nodeAffinity.restrictScheduling is set to true in the cluster’s .

2. Use the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

3. Run one of the following commands to label the node with its role. Replace the label and value (true

To set worker role for a node in your Kubernetes cluster via NVIDIA Run:ai , follow these steps:

1. Use the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

2. Run one of the following commands to set or remove a node’s role. <node-role> must be either --gpu-worker or --cpu-worker :

The set node-role command will label the node and set cluster configuration global.nodeAffinity.restrictScheduling true.

Researchers may need to access containers remotely during workload execution. Common use cases include:

• Running a Jupyter Notebook inside the container

• Connecting PyCharm for remote Python development

• Viewing machine learning visualizations using TensorBoard

To enable this access, you must expose the relevant container ports.

Accessing the containers remotely requires exposing container ports. In Docker, ports are exposed by them when launching the container. NVIDIA Run:ai provides similar functionality within a Kubernetes environment.

Since Kubernetes abstracts the container's physical location, exposing ports is more complex. Kubernetes supports multiple methods for exposing container ports. For more details, refer to the .

Many tools used by researchers, such as Jupyter, TensorBoard, or VSCode, require remote access to the running workload's container. In NVIDIA Run:ai, this access is provided through dynamically generated URLs.

By default, NVIDIA Run:ai uses the provided to dynamically create SSL-secured URLs in the following format:

While path-based routing works with applications such as Jupyter Notebooks, it may not be compatible with other applications. Some applications assume they are running at the root file system, so hardcoded file paths and settings within the container may become invalid when running at a path other than the root. For example, if an application expects to access /etc/config.json but is served at /project-name/workspace-name, the file will not be found. This can cause the container to fail or not function as intended.

NVIDIA Run:ai provides support for host-based routing. When enabled, URLs follow the format:

This allows all workloads to run at the root path, avoiding file path issues and ensuring proper application behavior.

To enable host-based routing, perform the following steps:

1. Create a second DNS entry (A record) for *.<CLUSTER_URL>, pointing to the same IP as the cluster's .

2. Obtain a wildcard SSL certificate for this second DNS entry.

3. Add the certificate as a secret:

1. Create the following ingress rule and replace <CLUSTER_URL>:

1. Run the following:

1. Edit to generate the URLs correctly:

Once these requirements have been met, all workloads will automatically be assigned a secured URL with a subdomain, ensuring full functionality for all researcher applications.

This article explains the procedure to configure and manage scheduling rules.

Scheduling rules are restrictions applied to workloads. These restrictions apply to either the resources (nodes) on which workloads can run or the duration of the run time. Scheduling rules are set for Projects or Departments and apply to specific workload types. Once scheduling rules are set for a project or department, all matching workloads associated with the project have the restrictions applied to them, as defined, when the workload was submitted. New scheduling rules added to a project are not applied over previously created workloads associated with that project.

There are three types of scheduling rules:

Workload Duration (Time Limit)

This rule limits the duration of a workload run time. Workload run time is calculated as the total time in which the workload was in status Running. You can apply a single rule per workload type - Preemptive Workspaces, Non-preemptive Workspaces, and Training.

Idle GPU Time Limit

This rule limits the total GPU time of a workload. Workload idle time is counted from the first time the workload is in status Running and the GPU was idle. Idleness is calculated by employing the runai_gpu_idle_seconds_per_workload metric. This metric determines the total duration of zero GPU utilization within each 30-second interval. If the GPU remains idle throughout the 30-second window, 30 seconds are added to the idleness sum; otherwise, the idleness count is reset. You can apply a single rule per workload type - "Preemptible" Workspaces, "Non-preemptible" Workspaces, and Training.

Node type is used to select a group of nodes, typically with specific characteristics such as a hardware feature, storage type, fast networking interconnection, etc. The uses node type as an indication of which nodes should be used for your workloads, within this project.

Node type is a label in the form of run.ai/type and a value (e.g. run.ai/type = dgx200) that the administrator uses to tag a set of nodes. Adding the node type to the project's scheduling rules mandates the user to submit workloads with a node type label/value pairs from this list, according to the workload type - Workspace or Training. The Scheduler then schedules workloads using a node selector, targeting nodes tagged with the NVIDIA Run:ai node type label/value pair. Node pools and a node type can be used in conjunction. For example, specifying a node pool and a smaller group of nodes from that node pool that includes a fast SSD memory or other unique characteristics.

The administrator should use a node label with the key of run.ai/type and any coupled value

To assign a label to nodes you want to group, set the 'node type (affinity)' on each relevant node:

1. Obtain the list of nodes and their current labels by copying the following to your terminal:

1. Annotate a specific node with a new label by copying the following to your terminal:

To add a scheduling rule:

1. Select the project/department for which you want to add a scheduling rule

2. Click EDIT

3. In the Scheduling rules section click +RULE

To edit a scheduling rule:

1. Select the project/department for which you want to edit its scheduling rule

2. Click EDIT

3. Find the scheduling rule you would like to edit

4. Edit the rule

To delete a scheduling rule:

1. Select the project/department from which you want to delete a scheduling rule

2. Click EDIT

3. Find the scheduling rule you would like to delete

4. Click on the x icon

Go to the API reference to view the available actions.

The Node Level Scheduler optimizes the performance of your pods and maximizes the utilization of GPUs by making optimal local decisions on GPU allocation to your pods. While the NVIDIA Run:ai Scheduler chooses the specific node for a pod, it has no visibility to the node’s GPUs internal state. The Node Level Scheduler is aware of the local GPUs states and makes optimal local decisions such that it can optimize both the GPU utilization and pods’ performance running on the node’s GPUs.

This guide provides an overview of the best use cases for the Node Level Scheduler and instructions for configuring it to maximize GPU performance and pod efficiency.

Deployment Considerations

• While the Node Level Scheduler applies to all workload types, it will best optimize the performance of burstable workloads. Burstable workloads are workloads that use dynamic GPU fractions, giving those more GPU memory than requested and up to the Limit specified.

• Burstable workloads are always susceptible to an OOM Kill signal if the owner of the excess memory requires it back. This means that using the Node Level Scheduler with inference or training workloads may cause pod preemption.

• Using interactive workloads with notebooks is the best use case for burstable workloads and Node Level Scheduler. These workloads behave differently since the OOM Kill signal will cause the notebooks' GPU process to exit but not the notebook itself. This keeps the interactive pod running and retrying to attach a GPU again.

This use case is one scenario that shows how Node Level Scheduler locally optimizes and maximizes GPU utilization and pods’ performance.

1. The below shows a node with 2 GPUs and 2 submitted workspaces:

1. The Scheduler instructs the node to put the 2 workspaces on a single GPU, a single GPU and leaving the other free for a workload that requires resources. This means GPU#2 is idle while the two workspaces can only use up to half a GPU, even if they temporarily need more:

1. With the Node Level Scheduler enabled, the local decision will be to spread those 2 workspaces on 2 GPUs and allow them to maximize both workspaces’ performance and GPUs’ utilization by bursting out up to the full GPU memory and GPU compute resources:

1. The NVIDIA Run:ai Scheduler still sees a node with one fully empty GPU and one fully occupied GPU. When a 3rd workload is scheduled, and it requires a full GPU (or more than 0.5 GPU), the Scheduler will schedule it to that node, and the Node Level Scheduler will move one of the workspaces to run with the other in GPU#1, as was the Scheduler’s initial plan. Moving the workspace from GPU#1 back to GPU#2 maintains the workspace running while the GPU process within the Jupyter notebook is killed and re-established on GPU#2, continuing to serve the workspace:

The Node Level Scheduler can be enabled per node pool. To use Node Level Scheduler, follow the below steps.

1. Enable the Node Level Scheduler at the cluster level (per cluster) by:

   1. Editing the runaiconfig as follows. For more details, see :

   2. Or, using the following kubectl patch command:

Enable Node Level Scheduler on any of the node pools:

1. Select Resources → Node pools

2. or

3. Under the Resource Utilization Optimization tab, change the number of workloads on each GPU to any value other than Not Enforced (i.e. 2, 3, 4, 5)

The Node Level Scheduler is now ready to be used on that node pool.

In order for a workload to be considered by the Node Level Scheduler for rerouting, it must be submitted with a GPU Request and Limit where the Limit is larger than the Request:

• Enable and set

• Then using dynamic GPU fractions

This document outlines how to back up and restore a NVIDIA Run:ai deployment, including both the NVIDIA Run:ai cluster and control plane.

Back Up the Cluster

The restoration or backup of NVIDIA Run:ai advanced cluster configurations and customized deployment configurations which are stored locally on the Kubernetes cluster is optional and can be restored and backed up separately. As backup of data is not required, the backup procedure is optional for advanced deployments.

Save Cluster Configurations

To back up the NVIDIA Run:ai cluster configurations:

1. Run the following command in your terminal:

   Copy

   kubectl get runaiconfig runai -n runai -o yaml -o=jsonpath='{.spec}' > runaiconfig_backup.yaml

2. Once the runaiconfig_back.yaml backup file is created, save the file externally, so that it can be retrieved later.

In the event of a critical Kubernetes failure or alternatively, if you want to migrate a NVIDIA Run:ai cluster to a new Kubernetes environment, simply reinstall the NVIDIA Run:ai cluster. Once you have reinstalled and reconnected the cluster, projects, workloads and other cluster data are synced automatically. Follow the steps below to restore the NVIDIA Run:ai cluster on a new Kubernetes environment.

Before restoring the NVIDIA Run:ai cluster, it is essential to validate that it is both disconnected and uninstalled:

1. If the Kubernetes cluster is still available, the NVIDIA Run:ai cluster. Make sure not to remove the cluster from the control plane.

2. Navigate to the Clusters grid in the NVIDIA Run:ai UI

3. Locate the cluster and verify its status is Disconnected

1. Follow the NVIDIA Run:ai cluster instructions and ensure all are met.

2. If you have a backup of the cluster configurations, reload it once the installation is complete:

3. Navigate to the Clusters grid in the NVIDIA Run:ai UI

If your cluster configuration disables automatic namespace creation for projects, you must manually:

• Re-create each project namespace

• Reapply the required role bindings for access control

For more information, see .

By default, NVIDIA Run:ai utilizes an internal PostgreSQL database to manage control plane data. This database resides on a Kubernetes Persistent Volume (PV). To safeguard against data loss, it's essential to implement a reliable backup strategy.

Consider the following methods to back up the PostgreSQL database:

• PostgreSQL logical backup - Use pg_dump to create a logical backup of the database. Replace <password> with the appropriate PostgreSQL password. For example:

• Persistent volume backup - Back up the entire PV that stores the PostgreSQL data.

• Third-Party backup solutions - Integrate with external backup tools that support Kubernetes and PostgreSQL to automate and manage backups effectively.

NVIDIA Run:ai stores metrics history using . Thanos is configured to write data to a persistent volume (PV). To protect against data loss, it is recommended to regularly back up this volume.

The NVIDIA Run:ai control plane installation can be using --set flags during Helm deployment. These configuration overrides are preserved during upgrades but are not retained if Kubernetes is uninstalled or damaged. To ensure recovery, it's recommended to back up the full set of applied Helm customizations. You can retrieve the current configuration using:

Follow the steps below to restore the control plane including previously backed-up data and configurations:

1. Recreate the Kubernetes environment - Begin by provisioning a new Kubernetes or OpenShift cluster that meets all NVIDIA Run:ai installation .

2. Restore Persistent Volumes - Recover the PVs and ensure these volumes are correctly reattached or restored from your backup solution:

   • PostgreSQL database - Stores control plane metadata

This section explains the procedure to manage your organization's applications.

Applications are used for API integrations with NVIDIA Run:ai. An application contains a client ID and a client secret. With the client credentials, you can obtain a token as detailed in API authentication and use it within subsequent API calls.

Applications are assigned with access rules to manage permissions. For example, application ci-pipeline-prod is assigned with a Researcher role in Cluster: A.

Applications Table

The Applications table can be found under Access in the NVIDIA Run:ai platform.

The Applications table provides a list of all the applications defined in the platform, and allows you to manage them.

The Applications table consists of the following columns:

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values

• Search - Click SEARCH and type the value to search by

• Sort - Click each column header to sort by

• Column selection - Click COLUMNS and select the columns to display in the table

To create an application:

1. Click +NEW APPLICATION

2. Enter the application's name

3. Click CREATE

4. Copy the

To create an access rule:

1. Select the application you want to add an access rule for

2. Click ACCESS RULES

3. Click +ACCESS RULE

4. Select a role

To delete an access rule:

1. Select the application you want to remove an access rule from

2. Click ACCESS RULES

3. Find the access rule assigned to the user you would like to delete

4. Click on the trash icon

To regenerate a client secret:

1. Locate the application you want to regenerate its client secret

2. Click REGENERATE CLIENT SECRET

3. Click REGENERATE

4. Copy the

1. Select the application you want to delete

2. Click DELETE

3. On the dialog, click DELETE to confirm

Go to the , API reference to view the available actions.

In the world of machine learning (ML), the journey from raw data to actionable insights is a complex process that spans multiple stages. Each stage of the AI lifecycle requires different tools, resources, and frameworks to ensure optimal performance. NVIDIA Run:ai simplifies this process by offering specialized workload types tailored to each phase, facilitating a smooth transition across various stages of the ML workflows.

The ML lifecycle usually begins with the experimental work on data and exploration of different modeling techniques to identify the best approach for accurate predictions. At this stage, resource consumption is usually moderate as experimentation is done on a smaller scale. As confidence grows in the model's potential and its accuracy, the demand for compute resources increases. This is especially true during the training phase, where vast amounts of data need to be processed, particularly with complex models such as large language models (LLMs), with their huge parameter sizes, that often require distributed training across multiple GPUs to handle the intensive computational load.

Finally, once the model is ready, it moves to the inference stage, where it is deployed to make predictions on new, unseen data. NVIDIA Run:ai's workload types are designed to correspond with the natural stages of this lifecycle. They are structured to align with the specific resource and framework requirements of each phase, ensuring that AI researchers and data scientists can focus on advancing their models without worrying about infrastructure management.

NVIDIA Run:ai offers three workload types that correspond to a specific phase of the researcher’s work:

• Workspaces – For experimentation with data and models.

• Training – For resource-intensive tasks such as model training and data preparation.

• Inference – For deploying and serving the trained model.

The Workspace is where data scientists conduct initial research, experiment with different data sets, and test various algorithms. This is the most flexible stage in the ML lifecycle, where models and data are explored, tuned, and refined. The value of workspaces lies in the flexibility they offer, allowing the researcher to iterate quickly without being constrained by rigid infrastructure.

• Framework flexibility

  Workspaces support a variety of machine learning frameworks, as researchers need to experiment with different tools and methods.

• Resource requirements

  Workspaces are often lighter on resources compared to the training phase, but they still require significant computational power for data processing, analysis, and model iteration.

  Hence, the default for the NVIDIA Run:ai workspaces considerations is to allow scheduling those workloads without the ability to preempt them once the resources were allocated. However, this non-preemptible state doesn’t allow utilizing more resources outside of the project’s deserved quota.

See to learn more about how to submit a workspace via the NVIDIA Run:ai platform. For quick starts, see .

As models mature and the need for more robust data processing and model training increases, NVIDIA Run:ai facilitates this shift through the Training workload. This phase is resource-intensive, often requiring distributed computing and high-performance clusters to process vast data sets and train models.

• Training architecture

  For training workloads NVIDIA Run:ai allows you to specify the architecture - standard or distributed. The distributed architecture is relevant for larger data sets and more complex models that require utilizing multiple nodes. For the distributed architecture, NVIDIA Run:ai allows you to specify different configurations for the master and workers and select which framework to use - PyTorch, XGBoost, MPI, and TensorFlow. In addition, as part of the distributed configuration, NVIDIA Run:ai enables the researchers to schedule their distributed workloads on nodes within the same region, zone, placement group, or any other topology.

• Resource requirements

  Training tasks demand high memory, compute power, and storage. NVIDIA Run:ai ensures that the allocated resources match the scale of the task and allows those workloads to utilize more compute resources than the project’s deserved quota. Make sure that if you wish your training workload not to be preempted, specify the number of GPUs that are in your quota.

See and to learn more about how to submit a training workload via the NVIDIA Run:ai UI. For quick starts, see and .

Once a model is trained and validated, it moves to the Inference stage, where it is deployed to make predictions (usually in a production environment). This phase is all about efficiency and responsiveness, as the model needs to serve real-time or batch predictions to end-users or other systems.

• Inference-specific use cases

  Naturally, inference workloads are required to change and adapt to the ever-changing demands to meet SLA. For example, additional replicas may be deployed, manually or automatically, to increase compute resources as part of a horizontal scaling approach or a new version of the deployment may need to be rolled out without affecting the running services.

• Resource requirements

  Inference models differ in size and purpose, leading to varying computational requirements. For example, small OCR models can run efficiently on CPUs, whereas LLMs typically require significant GPU memory for deployment and serving. Inference workloads are considered production-critical and are given the highest priority to ensure compliance with SLAs. Additionally, NVIDIA Run:ai ensures that inference workloads cannot be preempted, maintaining consistent performance and reliability.

See to learn more about how to submit an inference workload via the NVIDIA Run:ai UI. For a quick start, see .

Single Sign-On (SSO) is an authentication scheme, allowing users to log-in with a single pair of credentials to multiple, independent software systems.

This article explains the procedure to to NVIDIA Run:ai using the OpenID Connect protocol in OpenShift V4.

Before starting, make sure you have the following available from your OpenShift cluster:

• :

The following network requirements are for the NVIDIA Run:ai components installation and usage.

Set out below are the domains to whitelist and ports to open for installation, upgrade, and usage of the application and its management.

To allow your organization’s NVIDIA Run:ai users to interact with the cluster using the , or access specific UI features, certain inbound ports need to be open.

This section provides detailed instructions on how to manage both planned and unplanned node downtimes in a Kubernetes cluster running NVIDIA Run:ai. It covers all the steps to maintain service continuity and ensure the proper handling of workloads during these events.

• Access to Kubernetes cluster - Administrative access to the Kubernetes cluster, including permissions to run kubectl commands

• Basic knowledge of Kubernetes - Familiarity with Kubernetes concepts such as nodes, taints, and workloads

NVIDIA Run:ai supports simultaneous submission of multiple workloads to single or multi-GPUs when using . This is achieved by slicing the GPU memory between the different workloads according to the requested GPU fraction, and by using NVIDIA’s GPU time-slicing to share the GPU compute runtime. NVIDIA Run:ai ensures each workload receives the exact share of the GPU memory (= gpu_memory * requested), while the NVIDIA GPU time-slicing splits the GPU runtime evenly between the different workloads running on that GPU.

To provide customers with predictable and accurate GPU compute resource scheduling, NVIDIA Run:ai’s GPU time-slicing adds fractional compute capabilities on top of NVIDIA Run:ai GPU fraction capabilities.

While the default NVIDIA GPU time-slicing allows for sharing the GPU compute runtime evenly without splitting or limiting the runtime of each workload, NVIDIA Run:ai’s GPU time-slicing mechanism gives each workload exclusive access to the full GPU for a limited amount of time, lease time, in each scheduling cycle, plan time. This cycle repeats itself for the lifetime of the workload. Using the GPU runtime this way guarantees a workload is granted its requested GPU compute resources proportionally to its requested GPU fraction, but also allows splitting GPU unused compute time up to a requested Limit.

For example, when there are 2 workloads running on the same GPU, with NVIDIA’s default GPU time slicing, each workload gets 50% of the GPU compute runtime, even if one workload requests 25% of the GPU memory, and the other workload requests 75% of the GPU memory. With the NVIDIA Run:ai GPU time-slicing, the first workload will get 25% of the GPU compute time and the second will get 75%. If one of the workloads does not use its deserved GPU compute time, the others can split that time evenly between them. As shown in the example, if one of the workloads does not request the GPU for some time, the other will get the full GPU compute time.

Efficient resource allocation is critical for managing AI and compute-intensive workloads in Kubernetes clusters. The NVIDIA Run:ai Scheduler enhances Kubernetes' native capabilities by introducing advanced scheduling principles such as fairness, quota management, and dynamic resource balancing. It ensures that workloads, whether simple single-pod or complex distributed tasks, are allocated resources effectively while adhering to organizational policies and priorities.

This guide explores the NVIDIA Run:ai Scheduler’s allocation process, preemption mechanisms, and resource management. Through examples and detailed explanations, you'll gain insights into how the Scheduler dynamically balances workloads to optimize cluster utilization and maintain fairness across .

When a workload is submitted, the workload controller creates a pod or pods (for distributed training workloads or deployment based inference). When the Scheduler gets a submit request with the first pod, it creates a and allocates all the relevant building blocks of that workload. The next pods of the same workload are attached to the same pod group.

A workload, with its associated pod group, is queued in the appropriate . In every scheduling cycle, the Scheduler ranks the order of queues by calculating their precedence for scheduling.

At NVIDIA Run:ai, Administrators can access a suite of tools designed to facilitate efficient account management. This article focuses on two key features: workload policies and workload scheduling rules. These features empower admins to establish default values and implement restrictions allowing enhanced control, assuring compatibility with organizational policies, and optimizing resource usage and utilization.

A workload policy is an end-to-end solution for AI managers and administrators to control and simplify how workloads are submitted. This solution allows them to set best practices, enforce limitations, and standardize processes for the submission of workloads for AI projects within their organization. It acts as a key guideline for data scientists, researchers, ML & MLOps engineers by standardizing submission practices and simplifying the workload submission process.

Implementing workload policies is essential when managing complex AI projects within an enterprise for several reasons:

1. Resource control and management - Defining or limiting the use of costly resources across the enterprise via a centralized management system to ensure efficient allocation and prevent overuse.

This section explains the procedure of managing reports in NVIDIA Run:ai.

Reports allow users to access and organize large amounts of data in a clear, CSV-formatted layout. They enable users to monitor resource consumption, analyze trends, and make data-driven decisions to optimize their AI workloads effectively.

Currently, only "Consumption Reports" are available, which provides insights into the consumption of resources such as GPU, CPU, and CPU memory across organizational units.

The Reports table can be found under Analytics in the NVIDIA Run:ai platform.

The Reports table provides a list of all the reports defined in the platform and allows you to manage them.

Users are able to access the reports they have generated themselves. Users with project viewing permissions throughout the tenant can access all reports within the tenant.

NVIDIA’s Multi-Instance GPU (MIG) enables splitting a GPU into multiple logical GPU devices, each with its own memory and compute portion of the physical GPU.

NVIDIA provides two MIG strategies:

• Single - A GPU can be divided evenly. This means all MIG profiles are the same.

• Mixed - A GPU can be divided into different profiles.

The NVIDIA Run:ai platform supports running workloads using NVIDIA MIG. Administrators can set the Kubernetes nodes to their preferred MIG strategy and configure the appropriate MIG profiles for researchers and MLOPS engineers to use.

This section provides details about NVIDIA Run:ai's Audit log.

The NVIDIA Run:ai control plane provides the audit log API and event history table in the NVIDIA Run:ai UI. Both reflect the same information regarding changes to business objects: clusters, projects and assets etc.

The Event history table can be found under Event history in the NVIDIA Run:ai UI.

The Event history table consists of the following columns: