# Workload Priority Control

The workload priority management feature allows you to change the priority of a workload within a project. The priority determines the workload's position in the project scheduling queue managed by the NVIDIA Run:ai [Scheduler](https://run-ai-docs.nvidia.com/self-hosted/2.21/platform-management/runai-scheduler/scheduling/how-the-scheduler-works). By adjusting the priority, you can increase the likelihood that a workload will be scheduled and preferred over others within the same project, ensuring that critical tasks are given higher priority and resources are allocated efficiently.

You can change the priority of a workload by selecting one of the predefined values from the NVIDIA Run:ai priority dictionary. This can be done using the NVIDIA Run:ai UI, API or CLI, depending on the workload type.

{% hint style="info" %}
**Note**

This applies only within a single project. It does not impact the scheduling queues or workloads of other projects.
{% endhint %}

## Priority Dictionary

Workload priority is defined by selecting a **string name** from a predefined list in the NVIDIA Run:ai priority dictionary. Each string corresponds to a specific Kubernetes [PriorityClass](https://run-ai-docs.nvidia.com/self-hosted/2.21/platform-management/runai-scheduler/concepts-and-principles#priority-and-preemption), which in turn determines scheduling behavior, such as whether the workload is preemptible or allowed to run over quota.

{% hint style="info" %}
**Note**

The numeric priority levels (1 = highest, 4 = lowest) are descriptive only and are **not** part of the NVIDIA Run:ai priority dictionary.
{% endhint %}

<table><thead><tr><th width="92.68359375">Priority Level</th><th>Name (string)</th><th>Preemption</th><th>Over Quota</th></tr></thead><tbody><tr><td>1</td><td><code>inference</code></td><td>Non-preemptible</td><td>Not available</td></tr><tr><td>2</td><td><code>build</code></td><td>Non-preemptible</td><td>Not available</td></tr><tr><td>3</td><td><code>interactive-preemptible</code></td><td>Preemptible</td><td>Available</td></tr><tr><td>4</td><td><code>train</code></td><td>Preemptible</td><td>Available</td></tr></tbody></table>

### Preemptible vs Non-Preemptible Workloads

* **Non-preemptible workloads** must run within the project’s deserved quota, cannot use over-quota resources, and will not be interrupted once scheduled.
* **Preemptible workloads** can use opportunistic compute resources beyond the project’s quota but may be interrupted at any time.

## Default Priority per Workload

Both NVIDIA Run:ai and third-party workloads are assigned a default priority. The below table shows the default priority per workload type:

<table><thead><tr><th width="217.69140625">Workload Type</th><th width="238.765625">Default Priority</th></tr></thead><tbody><tr><td><a href="../../../workloads-in-nvidia-run-ai/using-workspaces/running-workspace">Workspaces</a></td><td><code>build</code></td></tr><tr><td><a href="../../../workloads-in-nvidia-run-ai/using-training">Training</a></td><td><code>train</code></td></tr><tr><td><a href="../../../workloads-in-nvidia-run-ai/using-inference">Inference</a></td><td><code>inference</code></td></tr><tr><td><a href="../../../workloads-in-nvidia-run-ai/introduction-to-workloads">Third-party workloads</a></td><td><code>train</code></td></tr><tr><td><a href="../../../workloads-in-nvidia-run-ai/using-inference/nvcf">NVIDIA Cloud Functions (NVCF)</a></td><td><code>inference</code></td></tr></tbody></table>

## Supported Priority Overrides per Workload

{% hint style="info" %}
**Note**

Changing a workload’s priority may impact its ability to be scheduled. For example, switching a workload from a `train` priority (which allows over-quota usage) to `build` priority (which requires in-quota resources) may reduce its chances of being scheduled in cases where the required quota is unavailable.
{% endhint %}

The below table shows the default priority listed in the previous section and the supported override options per workload:

<table><thead><tr><th width="137.11328125">Workload Type</th><th data-type="checkbox">interactive-preemptible</th><th data-type="checkbox">build</th><th data-type="checkbox">train</th><th data-type="checkbox">inference</th></tr></thead><tbody><tr><td><a href="../../../workloads-in-nvidia-run-ai/using-workspaces/running-workspace">Workspaces</a></td><td>true</td><td>true</td><td>false</td><td>false</td></tr><tr><td><a href="../../../workloads-in-nvidia-run-ai/using-training">Training</a></td><td>true</td><td>true</td><td>true</td><td>false</td></tr><tr><td><a href="../../../workloads-in-nvidia-run-ai/using-inference">Inference</a></td><td>false</td><td>false</td><td>false</td><td>true</td></tr><tr><td><a href="../../../workloads-in-nvidia-run-ai/introduction-to-workloads">Third-party workloads</a></td><td>true</td><td>true</td><td>true</td><td>false</td></tr><tr><td><a href="../../../workloads-in-nvidia-run-ai/using-inference/nvcf">NVIDIA Cloud Functions (NVCF)</a></td><td>false</td><td>false</td><td>false</td><td>true</td></tr></tbody></table>

## How to Override Priority

You can override the default priority when submitting a workload through the UI, API, or CLI depending on the workload type.

### Workspaces

To use the override options:

* **UI:** Enable **"Allow the workload to exceed the project quota"** when [submitting a workspace](https://run-ai-docs.nvidia.com/self-hosted/2.21/workloads-in-nvidia-run-ai/using-workspaces/running-workspace)
* **API:** Set `PriorityClass` in the [Workspaces](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workspaces) API
* **CLI:** [Submit a workspace](https://run-ai-docs.nvidia.com/self-hosted/2.21/reference/cli/runai/runai_workspace_submit) using the `--priority` flag

  ```sh
  runai workspace submit --priority priority-class
  ```

### Training Workloads

To use the override options:

* **API:** Set `PriorityClass` in the [Trainings](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/trainings) API
* **CLI:** [Submit training](https://run-ai-docs.nvidia.com/self-hosted/2.21/reference/cli/runai/runai_training_submit) using the `--priority` flag

  ```sh
  runai training submit --priority priority-class
  ```