# Workload Priority Control

The workload priority management feature allows you to change the priority of a workload within a project. The priority determines the workload's position in the project scheduling queue managed by the NVIDIA Run:ai [Scheduler](/self-hosted/2.23/platform-management/runai-scheduler/scheduling/how-the-scheduler-works.md). By adjusting the priority, you can increase the likelihood that a workload will be scheduled and preferred over others within the same project, ensuring that critical tasks are given higher priority and resources are allocated efficiently. The workload's priority also affects whether it can consume over-quota resources and whether it is subject to preemption by higher-priority workloads.

You can change the priority of a workload by selecting one of the predefined values from the NVIDIA Run:ai priority dictionary.

{% hint style="info" %}
**Note**

This applies only within a single project. It does not impact the scheduling queues or workloads of other projects.
{% endhint %}

## Priority Dictionary

Workload priority is defined by selecting a priority from a predefined list in the NVIDIA Run:ai priority dictionary. Each string corresponds to a specific Kubernetes [PriorityClass](/self-hosted/2.23/platform-management/runai-scheduler/scheduling/concepts-and-principles.md#priority-and-preemption), which in turn determines scheduling behavior, such as whether the workload is preemptible or allowed to run over quota.

<table><thead><tr><th width="160.5390625">Priority</th><th>Kubernetes Value</th><th>Preemption</th><th>Over Quota</th></tr></thead><tbody><tr><td><code>very-low</code></td><td>25</td><td>Preemptible</td><td>Available</td></tr><tr><td><code>low</code></td><td>40</td><td>Preemptible</td><td>Available</td></tr><tr><td><code>medium-low</code></td><td>65</td><td>Preemptible</td><td>Available</td></tr><tr><td><code>medium</code></td><td>80</td><td>Preemptible</td><td>Available</td></tr><tr><td><code>medium-high</code></td><td>90</td><td>Preemptible</td><td>Available</td></tr><tr><td><code>high</code></td><td>125</td><td>Non-preemptible</td><td>Not available</td></tr><tr><td><code>very-high</code></td><td>150</td><td>Non-preemptible</td><td>Not available</td></tr></tbody></table>

### Preemptible vs Non-Preemptible Workloads

* **Non-preemptible workloads** must run within the project’s deserved quota, cannot use over-quota resources, and will not be interrupted once scheduled.
* **Preemptible workloads** can use opportunistic compute resources beyond the project’s quota but may be interrupted at any time.

## Default Priority per Workload

NVIDIA Run:ai defines the following default mappings of workload types to priorities. To retrieve the default priority per workload type, refer to the [List workload types](https://run-ai-docs.nvidia.com/api/2.23/workloads/workload-properties#get-api-v1-workload-types) API.

{% hint style="info" %}
**Note**

* For more information on workload support, see [Introduction to workloads](/self-hosted/2.23/workloads-in-nvidia-run-ai/introduction-to-workloads.md).
* [Legacy priority values](/self-hosted/2.21/platform-management/runai-scheduler/scheduling/workload-priority-control.md) remain available for backward compatibility
* Changing the priority is not supported for NVCF workloads.
  {% endhint %}

<table><thead><tr><th width="197.58203125"></th><th width="308.23046875">Workload Types</th><th>Default Priority</th></tr></thead><tbody><tr><td><strong>NVIDIA Run:ai native workloads</strong></td><td>Workspaces, Standard training, Distributed training, Custom inference, Hugging Face inference, NVIDIA NIM inference</td><td>Workspaces = <code>high</code><br>Training = <code>low</code><br>Inference = <code>very-high</code></td></tr><tr><td><strong>NVIDIA</strong></td><td>NIM services, NVIDIA Cloud Functions (NVCF)</td><td><code>very-high</code></td></tr><tr><td><strong>Kubernetes</strong></td><td>Deployment, StatefulSet, ReplicaSet, Pod, Service, CronJob, Job, JobSet</td><td>Deployment, StatefulSet, ReplicaSet, Pod, Service, CronJob = <code>very-high</code><br>JobSet = <code>low</code><br>Job = <code>high</code></td></tr><tr><td><strong>Kubeflow</strong></td><td>TFJob, PyTorchJob, MPIJob, XGBoostJob, Notebook, ScheduledWorkflow</td><td>TFJob, PyTorchJob, MPIJob, XGBoostJob = <code>low</code><br>Notebook = <code>high</code><br>ScheduledWorkflow = <code>very-high</code></td></tr><tr><td><strong>Ray</strong></td><td>RayService, RayCluster, RayJob</td><td>RayService = <code>very-high</code><br>RayCluster, RayJob = <code>low</code></td></tr><tr><td><strong>Tekton</strong></td><td>PipelineRun, TaskRun</td><td>PipelineRun = <code>very-high</code><br>TaskRun = <code>high</code></td></tr><tr><td><strong>Additional Frameworks</strong></td><td>SeldonDeployment, AMLJob, DevWorkspace, VirtualMachineInstance, KServe, Workflow</td><td>SeldonDeployment, KServe, Workflow = <code>very-high</code><br>DevWorkspace = <code>high</code><br>AMLJob, VirtualMachineInstance = <code>low</code></td></tr></tbody></table>

## Setting Priority During Workload Submission

{% hint style="info" %}
**Note**

Changing a workload’s priority may impact its ability to be scheduled. For example, switching a workload from a `low` priority (which allows over-quota usage) to `high` priority (which requires in-quota resources) may reduce its chances of being scheduled in cases where the required quota is unavailable.
{% endhint %}

* Set the priority when submitting NVIDIA Run:ai workloads via the UI, CLI, or API:
  * **UI** - Set workload priority under **General** settings (flexible submission only)
  * **API** - Set using the `PriorityClass` field
  * **CLI** - Set using the `--priority` flag
* Set the workload's priority by adding the following label to your YAML under the `metadata.labels` section of your workload definition and use the following values, `very-low`, `medium-low`, `medium`, `medium-high`, `high`, `very-high` :

  ```yaml
  metadata:
    labels:
      priorityClassName: <priority>
  ```

## Updating the Default Priority Mapping

Administrators can change the default priority and preemptibility assigned to a workload type by updating the mapping using the [NVIDIA Run:ai API](https://run-ai-docs.nvidia.com/api/2.23/). To update the priority mapping:

1. Retrieve the list of workload types and their IDs using `GET /api/v1/workload-types`.
2. Identify the `workloadTypeId` of the workload type you want to modify.
3. Retrieve the list of available priorities and their IDs using `GET /api/v1/workload-priorities`.
4. Send a request to update the workload type with the new priority using\
   `PUT /api/v1/workload-types/{workloadTypeId}` and include the `priorityId` in the request body.

## Using API

Go to the [Workload priorities](https://run-ai-docs.nvidia.com/api/2.23/workloads/workload-properties#get-api-v1-workload-priorities) API reference to view the available actions.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://run-ai-docs.nvidia.com/self-hosted/2.23/platform-management/runai-scheduler/scheduling/workload-priority-control.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.