This article explains how to designate specific node roles in a Kubernetes cluster to ensure optimal performance and reliability in production deployments.

For optimal performance in production clusters, it is essential to avoid extensive CPU usage on GPU nodes where possible. This can be done by ensuring the following:

• NVIDIA Run:ai system-level services run on dedicated CPU-only nodes.

• Workloads that do not request GPU resources (e.g. Machine Learning jobs) are executed on CPU-only nodes.

NVIDIA Run:ai services are scheduled on the defined node roles by applying Kubernetes Node Affinity using node labels .

Prerequisites

To perform these tasks, make sure to install the NVIDIA Run:ai .

Configure Node Roles

The following node roles can be configured on the cluster:

• System node: Reserved for NVIDIA Run:ai system-level services.

• GPU Worker node: Dedicated for GPU-based workloads.

• CPU Worker node: Used for CPU-only workloads.

System Nodes

NVIDIA Run:ai system nodes run system-level services required to operate. This can be done via the (recommended) or via NVIDIA Run:ai .

By default, NVIDIA Run:ai applies a node affinity rule to prefer nodes that are labeled with node-role.kubernetes.io/runai-system for system services scheduling. You can modify the default node affinity rule by:

• Editing the spec.global.affinity configuration parameter as detailed in .

• Editing the global.affinity configuration as detailed in for self-hosted deployments.

Kubectl

To set a system role for a node in your Kubernetes cluster using Kubectl, follow these steps:

1. Use the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

2. Run one of the following commands to label the node with its role:

NVIDIA Run:ai Administrator CLI

To set a system role for a node in your Kubernetes cluster, follow these steps:

1. Run the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

2. Run one of the following commands to set or remove a node’s role:

The set node-role command will label the node and set relevant cluster configurations.

Worker Nodes

NVIDIA Run:ai worker nodes run user-submitted workloads and system-level required to operate. This can be managed via the (recommended) or via NVIDIA Run:ai .

By default, GPU workloads are scheduled on GPU nodes based on the nvidia.com/gpu.present label. When global.nodeAffinity.restrictScheduling is set to true via the :

• GPU Workloads are scheduled with node affinity rule to require nodes that are labeled with node-role.kubernetes.io/runai-gpu-worker

• CPU-only Workloads are scheduled with node affinity rule to require nodes that are labeled with node-role.kubernetes.io/runai-cpu-worker

Kubectl

To set a worker role for a node in your Kubernetes cluster using Kubectl, follow these steps:

1. Validate the global.nodeAffinity.restrictScheduling is set to true in the cluster’s .

2. Use the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

3. Run one of the following commands to label the node with its role. Replace the label and value (true/false) to enable or disable GPU/CPU roles as needed:

NVIDIA Run:ai Administrator CLI

To set worker role for a node in your Kubernetes cluster via NVIDIA Run:ai , follow these steps:

1. Use the kubectl get nodes command to list all the nodes in your cluster and identify the name of the node you want to modify.

2. Run one of the following commands to set or remove a node’s role. <node-role> must be either --gpu-worker or --cpu-worker :

The set node-role command will label the node and set cluster configuration global.nodeAffinity.restrictScheduling true.

NVIDIA Run:ai supports service mesh implementations. When a service mesh is deployed with sidecar injection, specific configurations must be applied to ensure compatibility with NVIDIA Run:ai. This document outlines the required changes for the NVIDIA Run:ai control plane and cluster.

Control Plane Configuration

By default, NVIDIA Run:ai prevents Istio from injecting sidecar containers into system jobs in the control plane. For other service mesh solutions, users must manually add annotations during installation.

To disable sidecar injection in the NVIDIA Run:ai control plane, modify the Helm values file by adding the required pod labels to the following components. See for more details.

Example for :

Cluster Configuration

Installation Phase

Sidecar containers injected by some service mesh solutions can prevent NVIDIA Run:ai installation hooks from completing. To avoid this, modify the Helm installation command to include the required labels or annotations:

Example for :

Workloads

To prevent sidecar injection in workloads created at runtime (such as training workloads), update the runaiconfig resource. See for more details:

Karpenter is an open-source, Kubernetes cluster autoscaler built for cloud deployments. Karpenter optimizes the cloud cost of a customer’s cluster by moving workloads between different node types, consolidating workloads into fewer nodes, using lower-cost nodes where possible, scaling up new nodes when needed, and shutting down unused nodes.

Karpenter’s main goal is cost optimization. Unlike Karpenter, NVIDIA Run:ai’s Scheduler optimizes for fairness and resource utilization. Therefore, there are a few potential friction points when using both on the same cluster.

Friction Points Using Karpenter with NVIDIA Run:ai

1. Karpenter looks for “unschedulable” pending workloads and may try to scale up new nodes to make those workloads schedulable. However, in some scenarios, these workloads may exceed their quota parameters, and the NVIDIA Run:ai Scheduler will put them into a pending state.

2. Karpenter is not aware of the NVIDIA Run:ai fractions mechanism and may try to interfere incorrectly.

3. Karpenter preempts any type of workload (i.e., high-priority, non-preemptible workloads will potentially be interrupted and moved to save cost).

4. Karpenter has no pod-group (i.e., workload) notion or gang scheduling awareness, meaning that Karpenter is unaware that a set of “arbitrary” pods is a single workload. This may cause Karpenter to schedule those pods into different node pools (in the case of multi-node-pool workloads) or scale up or down a mix of wrong nodes.

Mitigating the Friction Points

NVIDIA Run:ai Scheduler mitigates the friction points using the following techniques (each numbered bullet below corresponds to the related friction point listed above):

1. Karpenter uses a “nominated node” to recommend a node for the Scheduler. The NVIDIA Run:ai Scheduler treats this as a “preferred” recommendation, meaning it will try to use this node, but it’s not required and it may choose another node.

2. Fractions - Karpenter won’t consolidate nodes with one or more pods that cannot be moved. The NVIDIA Run:ai reservation pod is marked as ‘do not evict’ to allow the NVIDIA Run:ai Scheduler to control the scheduling of fractions.

3. Non-preemptible workloads - NVIDIA Run:ai marks non-preemptible workloads as ‘do not evict’ and Karpenter respects this annotation.

4. NVIDIA Run:ai node pools (single-node-pool workloads) - Karpenter respects the ‘node affinity’ that NVIDIA Run:ai sets on a pod, so Karpenter uses the node affinity for its recommended node. For the gang-scheduling/pod-group (workload) notion, NVIDIA Run:ai Scheduler considers Karpenter directives as preferred recommendations rather than mandatory instructions and overrides Karpenter instructions where appropriate.

Deployment Considerations

• Using multi-node-pool workloads

  • Workloads may include a list of optional node pools. Karpenter is not aware that only a single node pool should be selected out of that list for the workload. It may therefore recommend putting pods of the same workload into different node pools and may scale up nodes from different node pools to serve a “multi-node-pool” workload instead of nodes on the selected single node pool.

  • If this becomes an issue (i.e., if Karpenter scales up the wrong node types), users can set an inter-pod affinity using the node pool label or another common label as a ‘topology’ identifier. This will force Karpenter to choose nodes from a single-node pool per workload, selecting from any of the node pools listed as allowed by the workload.

Helm Chart Values

The NVIDIA Run:ai control plane installation can be customized to support your environment via Helm values files or Helm install flags. Make sure to restart the relevant NVIDIA Run:ai pods so they can fetch the new configurations.

Additional Third-Party Configurations

The NVIDIA Run:ai control plane chart includes multiple sub-charts of third-party components:

• Data store- (postgresql)

• Metrics Store - (thanos)

• Identity & Access Management - (keycloakx)

PostgreSQL

If you have opted to connect to an , refer to the additional configurations table below. Adjust the following parameters based on your connection details:

1. Disable PostgreSQL deployment - postgresql.enabled

2. NVIDIA Run:ai connection details - global.postgresql.auth

3. Grafana connection details - grafana.dbUser, grafana.dbPassword

Thanos

Keycloakx

The keycloakx.adminUser can only be set during the initial installation. The admin password can be changed later through the Keycloak UI, but you must also update the keycloakx.adminPassword value in the Helm chart using helm upgrade. See for more details.

Changing Keycloak Admin Password

You can change the Keycloak admin password after deployment by performing the following steps:

1. Open the Keycloak UI at: https://<runai-domain>/auth

2. Sign in with your existing admin credentials as configured in your Helm values

3. Go to Users and select admin (or your admin username)

4. Open Credentials →

Grafana

Researchers may need to access containers remotely during workload execution. Common use cases include:

• Running a Jupyter Notebook inside the container

• Connecting PyCharm for remote Python development

• Viewing machine learning visualizations using TensorBoard

To enable this access, you must expose the relevant container ports.

Integration Support

Support for third-party integrations varies. When noted below, the integration is supported out of the box with NVIDIA Run:ai. For other integrations, our Customer Success team has prior experience assisting customers with setup. In many cases, the NVIDIA Enterprise Support Portal may include additional reference documentation provided on an as-is basis.

Kubernetes Workloads Integration

Kubernetes has several built-in resources that encapsulate running Pods. These are called and should not be confused with .

Examples of such resources are a Deployment that manages a stateless application, or a Job that runs tasks to completion.

A NVIDIA Run:ai workload encapsulates all the resources needed to run and creates/deletes them together. Since NVIDIA Run:ai is an open platform, it allows the scheduling of any Kubernetes Workflow.

For more information, see .

This guide provides actionable best practices for administrators to securely configure, operate, and manage NVIDIA Run:ai environments. Each section highlights both platform-native features and mapped Kubernetes security practices to maintain robust protection for workloads and resources.

Access Control (RBAC)

NVIDIA Run:ai uses Role‑Based Access Control to define what each user, group, or application can do, and where. Roles are assigned within a scope, such as a project, department, or cluster, and permissions cover actions like viewing, creating, editing, or deleting entities. Unlike Kubernetes RBAC, NVIDIA Run:ai’s RBAC works across multiple clusters, giving you a single place to manage access rules. See for more details.

Best Practices

• Assign the minimum required permissions to users, groups and applications.

• Segment duties using organizational scopes to restrict roles to specific projects or departments.

• Regularly audit access rules and remove unnecessary privileges, especially admin-level roles.

Kubernetes Connection

NVIDIA Run:ai predefined roles are automatically mapped to Kubernetes cluster roles (also predefined by NVIDIA Run:ai). This means administrators do not need to manually configure role mappings.

These cluster roles define permissions for the entities NVIDIA Run:ai manages and displays (such as workloads) and also apply to users who access cluster data directly through Kubernetes tools (for example, kubectl).

Authentication and Session Management

NVIDIA Run:ai supports several authentication methods to control platform access. You can use single sign-on (SSO) for unified enterprise logins, traditional username/password accounts if SSO isn’t an option, and API secret keys for automated application access. Authentication is mandatory for all interfaces, including the UI, CLI, and APIs, ensuring only verified users or applications can interact with your environment.

Administrators can also configure session timeout. This refers to the period of inactivity before a user is automatically logged out. Once the timeout is reached, the session ends and re‑authentication is required, helping protect against risks from unattended or abandoned sessions. See for more details.

Best Practices

• Integrate corporate SSO for centralized identity management.

• Enforce strong password policies for local accounts.

• Set appropriate session timeout values to minimize idle session risk.

• Prefer SSO to eliminate password management within NVIDIA Run:ai.

Kubernetes Connection

Configure the Kubernetes API server to validate tokens via NVIDIA Run:ai’s identity service, ensuring unified authentication across the platform. For more information, see .

Workload Policies: Enforcing Security at Submission

Workload policies allow administrators to define and enforce how AI workloads are submitted and controlled across projects and teams. With these policies, you can set clear rules and defaults for workload parameters such as which resources can be requested, required security settings, and which defaults should apply. Policies are enforced whether workloads are submitted via the UI, CLI, API or Kubernetes YAML, and can be scoped to specific projects, departments, or clusters for fine-grained control. See for more details.

Best Practices

• Enforce containers to run as non-root by default. Define policies that set constraints and defaults for workload submissions, such as requiring non-root users or specifying minimum UID/GID. Example security fields in policies:

  • security.runAsNonRoot: true

  • security.runAsUid: 1000

Kubernetes Connection

Map these policies to PodSecurityContext settings in Kubernetes, and enforce them with Pod Security Admission or Kyverno for stricter compliance.

Managing Namespace and Resource Creation

NVIDIA Run:ai offers flexible controls for how namespaces and resources are created and managed within your clusters. When a new project is set up, you can choose whether Kubernetes namespaces are created automatically, and whether users are auto-assigned to those projects. There are also options to manage how secrets are propagated across namespaces and to enable or disable resource limit enforcement using Kubernetes LimitRange objects. See for more details.

Best Practices

• Require admin approval for namespace creation to avoid sprawl.

• Limit secret propagation to essential cases only.

• Use Kubernetes LimitRanges and ResourceQuotas alongside NVIDIA Run:ai policies for layered resource control.

Tools and Serving Endpoint Access Control

NVIDIA Run:ai provides flexible options to control access to tools and serving endpoints. Access can be defined during workload submission or updated later, ensuring that only the intended users or groups can interact with the resource.

When configuring an endpoint or tool, users can select from the following access levels:

• Public - Everyone within the network can access with no authentication (serving endpoints).

• All authenticated users - Access is granted to anyone in the organization who can log in (NVIDIA Run:ai or SSO).

• Specific groups - Access is restricted to members of designated identity provider groups.

• Specific users - Access is restricted to individual users by email or username.

By default, network exposure is restricted, and access must be explicitly granted. Model endpoints automatically inherit RBAC and workload policy controls, ensuring consistent enforcement of role- and scope-based permissions across the platform. Administrators can also limit who can deploy, view, or manage endpoints, and should open network access only when required.

Best Practices

• Define explicit roles for model management/use.

• Restrict endpoint access to authorized users, groups and applications.

• Monitor and audit endpoint access logs.

Kubernetes Connection

Use Kubernetes NetworkPolicies to limit inter-pod and external traffic to model-serving pods. Pair with NVIDIA Run:ai RBAC for end-to-end control.

Secure Installation and Maintenance

A secure deployment is the foundation on which all other controls rest, and NVIDIA Run:ai’s installation procedures are built to align with organizational policies such as OpenShift Security Context Constraints (SCC). See for more details.

• Deploy NVIDIA Run:ai cluster following secure installation guides (including IT compliance mandates such as SCC for OpenShift).

• Run regular security scans and patch/update NVIDIA Run:ai deployments promptly when vulnerabilities are reported.

• Regularly review and update all security policies, both at the NVIDIA Run:ai and Kubernetes levels, to adapt to evolving risks.

Compliance and Data Privacy

NVIDIA Run:ai supports SaaS and self-hosted modes to satisfy a range of data security needs. The self-hosted mode keeps all models, logs, and user data entirely within your infrastructure; SaaS requires careful review of what (minimal) data is transmitted for platform operations and analytics. See for more details.

• Use the self-hosted mode when full control over the environment is required - including deployment and day-2 operations such as upgrades, monitoring, backup, and metadata restore.

• Ensure transmission to the NVIDIA Run:ai cloud is scoped (in SaaS mode) and aligns with organization policy.

• Encrypt secrets and sensitive resources; control secret propagation.

• Document and audit data flows for regulatory alignment.

The identity of the user inside a container determines its access to various resources. For example, network file systems often rely on this identity to control access to mounted volumes. As a result, propagating the correct user identity into a container is crucial for both functionality and security.

By default, containers in both Docker and Kubernetes run as the root user. This means any process inside the container has full administrative privileges, capable of modifying system files, installing packages, or changing configurations.

While this level of access provides researchers with maximum flexibility, it conflicts with modern enterprise security practices. If the container’s root identity is propagated to external systems (e.g., network-attached storage), it can result in elevated permissions outside the container, increasing the risk of security breaches.

Advanced cluster configurations can be used to tailor your NVIDIA Run:ai cluster deployment to meet specific operational requirements and optimize resource management. By fine-tuning these settings, you can enhance functionality, ensure compatibility with organizational policies, and achieve better control over your cluster environment. This article provides guidance on implementing and managing these configurations to adapt the NVIDIA Run:ai cluster to your unique needs.

After the NVIDIA Run:ai cluster is installed, you can adjust various settings to better align with your organization's operational needs and security requirements.

Modify Cluster Configurations

Advanced cluster configurations in NVIDIA Run:ai are managed through the runaiconfig Kubernetes Custom Resource. To edit the cluster configurations, run:

To see the full runaiconfig object structure, use:

Configurations

The following configurations allow you to enable or disable features, control permissions, and customize the behavior of your NVIDIA Run:ai cluster:

NVIDIA Run:ai Services Resource Management

NVIDIA Run:ai cluster includes many different services. To simplify resource management, the configuration structure allows you to configure the containers CPU / memory resources for each service individually or group of services together.

Apply the following configuration in order to change resources request and limit for a group of services:

Or, apply the following configuration in order to change resources request and limit for each service individually:

For resource recommendations, see .

NVIDIA Run:ai Services Replicas

By default, all NVIDIA Run:ai containers are deployed with a single replica. Some services support multiple replicas for redundancy and performance.

To simplify configuring replicas, a global replicas configuration can be set and is applied to all supported services:

This can be overwritten for specific services (if supported). Services without the replicas configuration does not support replicas:

Prometheus

The Prometheus instance in NVIDIA Run:ai is used for metrics collection and alerting.

The configuration scheme follows the official and supports additional custom configurations. The PrometheusSpec schema is available using the spec.prometheus.spec configuration.

A common use case using the PrometheusSpec is for metrics retention. This prevents metrics loss during potential connectivity issues and can be achieved by configuring local temporary metrics retention. For more information, see :

In addition to the PrometheusSpec schema, some custom NVIDIA Run:ai configurations are also available:

• Additional labels – Set additional labels for NVIDIA Run:ai's sent by Prometheus.

• Log level configuration – Configure the logLevel setting for the Prometheus container.

NVIDIA Run:ai Managed Nodes

To include or exclude specific nodes from running workloads within a cluster managed by NVIDIA Run:ai, use the nodeSelectorTerms flag. For additional details, see .

Label the nodes using the below:

• key: Label key (e.g., zone, instance-type).

• operator: Operator defining the inclusion/exclusion condition (In, NotIn, Exists, DoesNotExist).

• values: List of values for the key when using In or NotIn.

The below example shows how to include NVIDIA GPUs only and exclude all other GPU types in a cluster with mixed nodes, based on product type GPU label:

S3 and Git Sidecar Images

For air-gapped environments, when working with a , it is required to replace the default sidecar images in order to use the Git and S3 data source integrations. Use the following configurations: