# Advanced Cluster Configurations

Advanced cluster configurations can be used to tailor your NVIDIA Run:ai cluster deployment to meet specific operational requirements and optimize resource management. By fine-tuning these settings, you can enhance functionality, ensure compatibility with organizational policies, and achieve better control over your cluster environment. This article provides guidance on implementing and managing these configurations to adapt the NVIDIA Run:ai cluster to your unique needs.

After the NVIDIA Run:ai cluster is installed, you can adjust various settings to better align with your organization's operational needs and security requirements.

## Modify Cluster Configurations

Advanced cluster configurations in NVIDIA Run:ai are managed through the `runaiconfig` [Kubernetes Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). To edit the cluster configurations, run:

```bash
kubectl edit runaiconfig runai -n runai
```

To see the full `runaiconfig` object structure, use:

```bash
kubectl get crds/runaiconfigs.run.ai -n runai -o yaml
```

## Configurations

The following configurations allow you to enable or disable features, control permissions, and customize the behavior of your NVIDIA Run:ai cluster:

| Key                                                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spec.global.affinity` *(object)*                                       | <p>Sets the system nodes where NVIDIA Run:ai system-level services are scheduled. Using global.affinity will overwrite the <a href="node-roles">node roles</a> set using the Administrator CLI (runai-adm).<br>Default: Prefer to schedule on nodes that are labeled with <code>node-role.kubernetes.io/runai-system</code></p>                                                                                                                                                                                                                                                    |
| `spec.global.nodeAffinity.restrictScheduling` *(boolean)*               | <p>Enables setting <a href="node-roles">node roles</a> and restricting workload scheduling to designated nodes<br>Default: <code>false</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `spec.global.tolerations` *(object)*                                    | Configure Kubernetes tolerations for NVIDIA Run:ai system-level services                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `spec.global.ingress.ingressClass`                                      | NVIDIA Run:ai uses NGINX as the default ingress controller. If your cluster has a different ingress controller, you can configure the ingress class to be created by NVIDIA Run:ai.                                                                                                                                                                                                                                                                                                                                                                                                |
| `spec.global.subdomainSupport` *(boolean)*                              | <p>Allows the creation of subdomains for ingress endpoints, enabling access to workloads via unique subdomains on the <a href="../../../getting-started/installation/install-using-helm/system-requirements#fully-qualified-domain-name-fqdn">Fully Qualified Domain Name (FQDN)</a>. For details, see <a href="container-access/external-access-to-containers">External Access to Containers</a>.<br>Default: <code>false</code></p>                                                                                                                                              |
| `spec.global.devicePluginBindings` *(boolean)*                          | <p>Instruct NVIDIA Run:ai fractions to use device plugin for host mount instead of NVIDIA Run:ai fractions using explicit host path mount configuration on the pod. See <a href="../../platform-management/runai-scheduler/resource-optimization/fractions">GPU fractions</a> and <a href="../../platform-management/runai-scheduler/resource-optimization/dynamic-fractions">dynamic GPU fractions</a>.<br>Default: <code>false</code></p>                                                                                                                                        |
| `spec.global.enableWorkloadOwnershipProtection` *(boolean)*             | <p>Prevents users within the same project from deleting workloads created by others. This enhances workload ownership security and ensures better collaboration by restricting unauthorized modifications or deletions.<br>Default: <code>false</code></p>                                                                                                                                                                                                                                                                                                                         |
| `spec.project-controller.createNamespaces` *(boolean)*                  | <p>Allows Kubernetes namespace creation for new projects<br>Default: <code>true</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `spec.project-controller.createRoleBindings` *(boolean)*                | <p>Specifies if role bindings should be created in the project's namespace<br>Default: <code>true</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `spec.project-controller.limitRange` *(boolean)*                        | <p>Specifies if limit ranges should be defined for projects<br>Default: <code>true</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `spec.project-controller.clusterWideSecret` *(boolean)*                 | <p>Allows Kubernetes Secrets creation at the cluster scope. See <a href="../../../workloads-in-nvidia-run-ai/assets/credentials#creating-secrets-in-advance">Credentials</a> for more details.<br>Default: <code>true</code></p>                                                                                                                                                                                                                                                                                                                                                   |
| `spec.workload-controller.additionalPodLabels` *(object)*               | Set workload's [Pod Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels) in a format of key/value pairs. These labels are applied to all pods.                                                                                                                                                                                                                                                                                                                                                                                                        |
| `spec.workload-controller.failureResourceCleanupPolicy`                 | <p>NVIDIA Run:ai cleans the workload's unnecessary resources:</p><ul><li><code>All</code> - Removes all resources of the failed workload</li><li><code>None</code> - Retains all resources</li><li><code>KeepFailing</code> - Removes all resources except for those that encountered issues (primarily for debugging purposes)</li></ul><p>Default: <code>All</code></p>                                                                                                                                                                                                          |
| `spec.workload-controller.GPUNetworkAccelerationEnabled`                | <p>Enables GPU network acceleration. See <a href="../../platform-management/aiinitiatives/resources/using-gb200">Using GB200 NVL72 and Multi-Node NVLink Domains</a> for more details.<br>Default: <code>false</code></p>                                                                                                                                                                                                                                                                                                                                                          |
| `spec.mps-server.enabled` *(boolean)*                                   | <p>Enabled when using <a href="https://docs.nvidia.com/deploy/mps/index.html">NVIDIA MPS</a><br>Default: <code>false</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `spec.daemonSetsTolerations` *(object)*                                 | Configure Kubernetes tolerations for NVIDIA Run:ai daemonSets / engine                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `spec.runai-container-toolkit.logLevel` *(boolean)*                     | <p>Specifies the NVIDIA Run:ai-container-toolkit logging level: either 'SPAM', 'DEBUG', 'INFO', 'NOTICE', 'WARN', or 'ERROR'<br>Default: <code>INFO</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `spec.runai-container-toolkit.enabled` *(boolean)*                      | <p>Enables workloads to use <a href="../../platform-management/runai-scheduler/resource-optimization/fractions">GPU fractions</a><br>Default: <code>true</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                |
| `node-scale-adjuster.args.gpuMemoryToFractionRatio` *(object)*          | <p>A scaling-pod requesting a single GPU device will be created for every 1 to 10 pods requesting fractional GPU memory (1/gpuMemoryToFractionRatio). This value represents the ratio (0.1-0.9) of fractional GPU memory (any size) to GPU fraction (portion) conversion.<br>Default: <code>0.1</code></p>                                                                                                                                                                                                                                                                         |
| `spec.global.core.dynamicFractions.enabled` *(boolean)*                 | <p>Enables <a href="../../platform-management/runai-scheduler/resource-optimization/dynamic-fractions">dynamic GPU fractions</a><br>Default: <code>true</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `spec.global.core.swap.enabled` *(boolean)*                             | <p>Enables <a href="../../platform-management/runai-scheduler/resource-optimization/memory-swap">memory swap</a> for GPU workloads<br>Default: <code>false</code></p>                                                                                                                                                                                                                                                                                                                                                                                                              |
| `spec.global.core.swap.biDirectional` *(string)*                        | <p>Sets the read/write memory mode of GPU memory swap to bi-directional (fully duplex). This produces higher performance (typically +80%) vs. uni-directional (simplex) read-write operations. For more details, see <a href="../../platform-management/runai-scheduler/resource-optimization/memory-swap">GPU memory swap</a>.<br>Default: <code>false</code></p>                                                                                                                                                                                                                 |
| `spec.global.core.swap.mode` *(string)*                                 | <p>Sets the GPU to CPU memory swap method to use UVA and optimized memory prefetch for optimized performance in some scenarios. For more details, see <a href="../../platform-management/runai-scheduler/resource-optimization/memory-swap">GPU memory swap</a>.<br>Default: None. The parameter is not set by default. To add this parameter set <code>mode=mapped</code> .</p>                                                                                                                                                                                                   |
| `spec.global.core.nodeScheduler.enabled` *(boolean)*                    | <p>Enables the <a href="../../platform-management/runai-scheduler/resource-optimization/node-level-scheduler">node-level scheduler</a><br>Default: <code>false</code></p>                                                                                                                                                                                                                                                                                                                                                                                                          |
| `spec.global.core.timeSlicing.mode` *(string)*                          | <p>Sets the <a href="../../platform-management/runai-scheduler/resource-optimization/time-slicing">GPU time-slicing mode</a>. Possible values:</p><ul><li><code>timesharing</code> - all pods on a GPU share the GPU compute time evenly.</li><li><code>strict</code> - each pod gets an exact time slice according to its memory fraction value.</li><li><code>fair</code> - each pod gets an exact time slice according to its memory fraction value and any unused GPU compute time is split evenly between the running pods.</li></ul><p>Default: <code>timesharing</code></p> |
| `spec.runai-scheduler.args.fullHierarchyFairness` *(boolean)*           | <p>Enables fairness between departments, on top of projects fairness<br>Default: <code>true</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `spec.runai-scheduler.args.defaultStalenessGracePeriod`                 | <p>Sets the timeout in seconds before the scheduler evicts a stale pod-group (gang) that went below its min-members in running state:</p><ul><li><code>0s</code> - Immediately (no timeout)</li><li><code>-1</code> - Never</li></ul><p>Default: <code>60s</code></p>                                                                                                                                                                                                                                                                                                              |
| `spec.pod-grouper.args.gangSchedulingKnative` *(boolean)*               | <p>Enables gang scheduling for inference workloads.For backward compatibility with versions earlier than v2.19, change the value to false<br>Default: <code>false</code></p>                                                                                                                                                                                                                                                                                                                                                                                                       |
| `spec.pod-grouper.args.gangScheduleArgoWorkflow` *(boolean)*            | <p>Groups all pods of a single ArgoWorkflow workload into a single Pod-Group for gang scheduling<br>Default: <code>true</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `spec.runai-scheduler.args.verbosity` *(int)*                           | <p>Configures the level of detail in the logs generated by the scheduler service<br>Default: <code>4</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `spec.limitRange.cpuDefaultRequestCpuLimitFactorNoGpu` *(string)*       | <p>Sets a default ratio between the CPU request and the limit for workloads without GPU requests<br>Default: <code>0.1</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `spec.limitRange.memoryDefaultRequestMemoryLimitFactorNoGpu` *(string)* | <p>Sets a default ratio between the memory request and the limit for workloads without GPU requests<br>Default: <code>0.1</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `spec.limitRange.cpuDefaultRequestGpuFactor` *(string)*                 | <p>Sets a default amount of CPU allocated per GPU when the CPU is not specified<br>Default: <code>100</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `spec.limitRange.cpuDefaultLimitGpuFactor` *(int)*                      | <p>Sets a default CPU limit based on the number of GPUs requested when no CPU limit is specified<br>Default: <code>NO DEFAULT</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `spec.limitRange.memoryDefaultRequestGpuFactor` *(string)*              | <p>Sets a default amount of memory allocated per GPU when the memory is not specified<br>Default: <code>100Mi</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `spec.limitRange.memoryDefaultLimitGpuFactor` *(string)*                | <p>Sets a default memory limit based on the number of GPUs requested when no memory limit is specified<br>Default: <code>NO DEFAULT</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                     |

### NVIDIA Run:ai Services Resource Management

NVIDIA Run:ai cluster includes many different services. To simplify resource management, the configuration structure allows you to configure the containers CPU / memory resources for each service individually or group of services together.

| Service Group      | Description                                                                                                      | NVIDIA Run:ai containers                                                        |
| ------------------ | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| SchedulingServices | Containers associated with the NVIDIA Run:ai Scheduler                                                           | Scheduler, StatusUpdater, MetricsExporter, PodGrouper, PodGroupAssigner, Binder |
| SyncServices       | Containers associated with syncing updates between the NVIDIA Run:ai cluster and the NVIDIA Run:ai control plane | Agent, ClusterSync, AssetsSync                                                  |
| WorkloadServices   | Containers associated with submitting NVIDIA Run:ai workloads                                                    | <p>WorkloadController,</p><p>JobController</p>                                  |

Apply the following configuration in order to change resources request and limit for a group of services:

```yaml
spec:
  global:
   <service-group-name>: # schedulingServices | syncServices | workloadServices
     resources:
       limits:
         cpu: 1000m
         memory: 1Gi
       requests:
         cpu: 100m
         memory: 512Mi
```

Or, apply the following configuration in order to change resources request and limit for each service individually:

```yaml
spec:
  <service-name>: # for example: pod-grouper
    resources:
      limits:
        cpu: 1000m
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 512Mi
```

For resource recommendations, see [Vertical scaling](https://run-ai-docs.nvidia.com/self-hosted/2.22/procedures/scaling#vertical-scaling).

### NVIDIA Run:ai Services Replicas

By default, all NVIDIA Run:ai containers are deployed with a single replica. Some services support multiple replicas for redundancy and performance.

To simplify configuring replicas, a global replicas configuration can be set and is applied to all supported services:

```yaml
spec:
  global: 
    replicaCount: 1 # default
```

This can be overwritten for specific services (if supported). Services without the `replicas` configuration does not support replicas:

<pre class="language-yaml"><code class="lang-yaml"><strong>spec:
</strong>  &#x3C;service-name>: # for example: pod-grouper
    replicas: 1 # default
</code></pre>

### Prometheus

The Prometheus instance in NVIDIA Run:ai is used for metrics collection and alerting.

The configuration scheme follows the official [PrometheusSpec](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1.PrometheusSpec) and supports additional custom configurations. The PrometheusSpec schema is available using the `spec.prometheus.spec` configuration.

A common use case using the PrometheusSpec is for metrics retention. This prevents metrics loss during potential connectivity issues and can be achieved by configuring local temporary metrics retention. For more information, see [Prometheus Storage](https://prometheus.io/docs/prometheus/latest/storage/#storage):

```yaml
spec:  
  prometheus:
    spec: # PrometheusSpec
      retention: 2h # default 
      retentionSize: 20GB
```

In addition to the PrometheusSpec schema, some custom NVIDIA Run:ai configurations are also available:

* Additional labels – Set additional labels for NVIDIA Run:ai's [built-in alerts](https://run-ai-docs.nvidia.com/self-hosted/2.22/procedures/system-monitoring#built-in-alerts) sent by Prometheus.
* Log level configuration – Configure the `logLevel` setting for the Prometheus container.
* Advanced metrics - Use `prometheus.spec.config.advancedMetricsEnabled` to activate GPU profiling metrics from NVIDIA DCGM. When enabled, Prometheus collects and aggregates advanced GPU performance data such as SM activity, memory bandwidth, and tensor core utilization. For setup instructions, see [Advanced metrics](https://run-ai-docs.nvidia.com/self-hosted/2.22/platform-management/monitor-performance/advanced-metrics).

```yaml
spec:  
  prometheus:
    logLevel: info # debug | info | warn | error
    additionalAlertLabels:
      - env: prod # example
```

### NVIDIA Run:ai Managed Nodes

To include or exclude specific nodes from running workloads within a cluster managed by NVIDIA Run:ai, use the `nodeSelectorTerms` flag. For additional details, see [Kubernetes nodeSelector](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector).

Label the nodes using the below:

* key: Label key (e.g., zone, instance-type).
* operator: Operator defining the inclusion/exclusion condition (In, NotIn, Exists, DoesNotExist).
* values: List of values for the key when using In or NotIn.

The below example shows how to include NVIDIA GPUs only and exclude all other GPU types in a cluster with mixed nodes, based on product type GPU label:

```yaml
spec:   
  global:
     managedNodes:
       inclusionCriteria:
          nodeSelectorTerms:
          - matchExpressions:
            - key: nvidia.com/gpu.product  
              operator: Exists
```

### S3 and Git Sidecar Images <a href="#s3-and-git-sidecar-images" id="s3-and-git-sidecar-images"></a>

For air-gapped environments, when working with a [Local Certificate Authority](https://run-ai-docs.nvidia.com/self-hosted/2.22/getting-started/installation/install-using-helm/system-requirements#local-certificate-authority), it is required to replace the default sidecar images in order to use the Git and S3 data source integrations. Use the following configurations:

```yaml
spec:
  workload-controller:    
    s3FileSystemImage:
      name: goofys       
      registry: runai.jfrog.io/op-containers-prod      
      tag: 3.12.24    
    gitSyncImage:      
      name: git-sync      
      registry: registry.k8s.io     
      tag: v4.4.0
```