# Metrics and Telemetry

Metrics are numeric measurements recorded **over time** that are emitted from the NVIDIA Run:ai cluster and telemetry is a numeric measurement recorded in real-time when emitted from the NVIDIA Run:ai cluster.

## Scopes

NVIDIA Run:ai provides control-plane API which supports and aggregates analytics at various levels.

<table><thead><tr><th width="298.04296875">Level</th><th>Description</th></tr></thead><tbody><tr><td>Cluster</td><td>A cluster is a set of nodes pools and nodes. With Cluster metrics, metrics are aggregated at the Cluster level. In the NVIDIA Run:ai user interface, metrics are available in the Overview dashboard.</td></tr><tr><td>Node</td><td>Data is aggregated at the node level.</td></tr><tr><td>Node pool</td><td>Data is aggregated at the node pool level.</td></tr><tr><td>Workload</td><td>Data is aggregated at the workload level. In some workloads, e.g. with distributed workloads, these metrics aggregate data from all worker pods.</td></tr><tr><td>Pod</td><td>The basic unit of execution.</td></tr><tr><td>Project</td><td>The basic organizational unit. Projects are the tool to implement resource allocation policies as well as the segregation between different initiatives.</td></tr><tr><td>Department</td><td>Departments are a grouping of projects.</td></tr></tbody></table>

## Supported Metrics

| Metric name in API               | Applicable API endpoint                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Metric name in UI per grid                                           | Applicable UI grid                                                                                                                                                                                                                                                |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ALLOCATED_GPU`                  | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li></ul>                                                                                                                                                                                                                                                                                                                     | <ul><li>GPU devices (allocated)</li><li>Allocated GPUs</li></ul>     | <ul><li><a href="../before-you-start#ui-views">Overview dashboard</a></li><li><a href="../../aiinitiatives/resources/node-pools#show-hide-details">Node pools</a></li></ul>                                                                                       |
| `AVG_WORKLOAD_WAIT_TIME`         | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li></ul>                                                                                                                                                                                                                                                                                                                     |                                                                      |                                                                                                                                                                                                                                                                   |
| `CPU_LIMIT_CORES`                | [Workloads](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | CPU limit                                                            | [Workloads](https://run-ai-docs.nvidia.com/self-hosted/workloads-in-nvidia-run-ai/workloads#metrics)                                                                                                                                                              |
| `CPU_MEMORY_LIMIT_BYTES`         | [Workloads](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | CPU memory limit                                                     | [Workloads](https://run-ai-docs.nvidia.com/self-hosted/workloads-in-nvidia-run-ai/workloads#metrics)                                                                                                                                                              |
| `CPU_MEMORY_REQUEST_BYTES`       | [Workloads](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | CPU memory request                                                   | [Workloads](https://run-ai-docs.nvidia.com/self-hosted/workloads-in-nvidia-run-ai/workloads#metrics)                                                                                                                                                              |
| `CPU_MEMORY_USAGE_BYTES`         | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li></ul>                                                                                                                                                                                                                                                                                                                                                  | CPU memory usage                                                     | [Workloads](https://run-ai-docs.nvidia.com/self-hosted/workloads-in-nvidia-run-ai/workloads#metrics)                                                                                                                                                              |
| `CPU_MEMORY_UTILIZATION`         | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul>                                                                                                                                                                                      | CPU memory utilization                                               | <ul><li><a href="../before-you-start#ui-views">Overview dashboard</a></li><li><a href="../../aiinitiatives/resources/node-pools#show-hide-details">Node pools</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul>     |
| `CPU_REQUEST_CORES`              | [Workloads](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | CPU request                                                          | [Workloads](https://run-ai-docs.nvidia.com/self-hosted/workloads-in-nvidia-run-ai/workloads)                                                                                                                                                                      |
| `CPU_USAGE_CORES`                | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li></ul>                                                                                                                                                                                                                   | CPU usage                                                            | [Workloads](https://run-ai-docs.nvidia.com/self-hosted/workloads-in-nvidia-run-ai/workloads#metrics)                                                                                                                                                              |
| `CPU_UTILIZATION`                | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul>                                                                                                                                                                                      | <ul><li>CPU compute utilization</li><li>CPU utilization</li></ul>    | <ul><li><a href="../before-you-start#ui-views">Overview dashboard</a> and <a href="../../aiinitiatives/resources/node-pools#show-hide-details">Node pools</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul>         |
| `GPU_ALLOCATION`                 | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-departmentid-metrics">Departments</a></li></ul>                                                                                                                                                                          | GPU devices (allocated)                                              | [Overview dashboard](https://run-ai-docs.nvidia.com/self-hosted/platform-management/before-you-start#ui-views)                                                                                                                                                    |
| `GPU_MEMORY_REQUEST_BYTES`       | [Workloads](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | GPU memory request                                                   | [Workloads](https://run-ai-docs.nvidia.com/self-hosted/workloads-in-nvidia-run-ai/workloads#metrics)                                                                                                                                                              |
| `GPU_MEMORY_USAGE_BYTES`         | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul>                                                                                                                                                                                                                   | GPU memory usage                                                     | [Workloads](https://run-ai-docs.nvidia.com/self-hosted/workloads-in-nvidia-run-ai/workloads#metrics)                                                                                                                                                              |
| `GPU_MEMORY_USAGE_BYTES_PER_GPU` | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li></ul>                                                                                                                                                                                                                                                                                                                                                              | GPU memory usage per GPU                                             | [Workloads per pod](https://run-ai-docs.nvidia.com/self-hosted/workloads-in-nvidia-run-ai/workloads#metrics)                                                                                                                                                      |
| `GPU_MEMORY_UTILIZATION`         | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li></ul>                                                                                                                                                                                                                                                                                                                     | GPU memory utilization                                               | <ul><li><a href="../before-you-start#ui-views">Overview dashboard</a></li><li><a href="../../aiinitiatives/resources/node-pools#show-hide-details">Node pools</a></li></ul>                                                                                       |
| `GPU_MEMORY_UTILIZATION_PER_GPU` | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | GPU memory utilization per GPU                                       | [Nodes](https://run-ai-docs.nvidia.com/self-hosted/aiinitiatives/resources/nodes#show-hide-details)                                                                                                                                                               |
| `GPU_QUOTA`                      | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-departmentid-metrics">Departments</a></li></ul> | Quota                                                                | [Quota management](https://run-ai-docs.nvidia.com/self-hosted/platform-management/before-you-start#ui-views)                                                                                                                                                      |
| `GPU_UTILIZATION`                | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li></ul>                              | GPU compute utilization                                              | <ul><li><a href="../before-you-start#ui-views">Overview dashboard</a></li><li><a href="../../aiinitiatives/resources/node-pools#show-hide-details">Node pools</a></li><li><a href="../../../workloads-in-nvidia-run-ai/workloads#metrics">Workloads</a></li></ul> |
| `GPU_UTILIZATION_PER_GPU`        | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li></ul>                                                                                                                                                                                                                                                                                                                                                              | GPU utilization per GPU                                              | [Nodes](https://run-ai-docs.nvidia.com/self-hosted/aiinitiatives/resources/nodes#show-hide-details)                                                                                                                                                               |
| `TOTAL_GPU`                      | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li></ul>                                                                                                                                                                                                                                                                                                                     | <ul><li>GPU devices total</li><li>Total GPUs</li></ul>               | <ul><li><a href="../before-you-start#ui-views">Overview dashboard</a></li><li><a href="../../aiinitiatives/resources/node-pools#show-hide-details">Node pools</a></li></ul>                                                                                       |
| `TOTAL_GPU_NODES`                | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li></ul>                                                                                                                                                                                                                                                                                                                     |                                                                      |                                                                                                                                                                                                                                                                   |
| `GPU_UTILIZATION_DISTRIBUTION`   | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li></ul>                                                                                                                                                                                                                                                                                                                     | GPU utilization distribution                                         | [Node pools](https://run-ai-docs.nvidia.com/self-hosted/aiinitiatives/resources/node-pools#show-hide-details)                                                                                                                                                     |
| `UNALLOCATED_GPU`                | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/clusters#get-api-v1-clusters-clusteruuid-metrics">Clusters</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodepools#get-api-v1-clusters-clusteruuid-nodepools-nodepoolname-metrics">Node pools</a></li></ul>                                                                                                                                                                                                                                                                                                                     | <ul><li>GPU devices (unallocated)</li><li>Unallocated GPUs</li></ul> | <ul><li><a href="../before-you-start#ui-views">Overview dashboard</a></li><li><a href="../../aiinitiatives/resources/node-pools#show-hide-details">Node pools</a></li></ul>                                                                                       |
| `CPU_QUOTA_MILLICORES`           | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-departmentid-metrics">Departments</a></li></ul>                                                                                                                                                                                                                                                                                                                     |                                                                      |                                                                                                                                                                                                                                                                   |
| `CPU_MEMORY_QUOTA_MB`            | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-departmentid-metrics">Departments</a></li></ul>                                                                                                                                                                                                                                                                                                                     |                                                                      |                                                                                                                                                                                                                                                                   |
| `CPU_ALLOCATION_MILLICORES`      | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-departmentid-metrics">Departments</a></li></ul>                                                                                                                                                                                                                                                                                                                     |                                                                      |                                                                                                                                                                                                                                                                   |
| `CPU_MEMORY_ALLOCATION_MB`       | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-departmentid-metrics">Departments</a></li></ul>                                                                                                                                                                                                                                                                                                                     |                                                                      |                                                                                                                                                                                                                                                                   |
| `POD_COUNT`                      | [Workloads](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |                                                                      |                                                                                                                                                                                                                                                                   |
| `RUNNING_POD_COUNT`              | [Workloads](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |                                                                      |                                                                                                                                                                                                                                                                   |
| `NVLINK_BANDWIDTH_TOTAL`         | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li></ul>                                                                                                                                                                                                                                                                                                                                                              |                                                                      | <ul><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li><li><a href="../../../workloads-in-nvidia-run-ai/workloads#metrics">Workloads per pod</a></li></ul>                                                                         |

### GPU Profiling

NVIDIA provides extended metrics as shown [here](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html#profiling-metrics).

{% hint style="info" %}
**Note**

GPU profiling metrics are disabled by default. If unavailable, your administrator must enable it under **General settings** → Analytics → GPU profiling metrics. Before enabling, the administrator must configure GPU profiling through the DCGM Exporter and NVIDIA Run:ai Prometheus integration. For configuration steps, see [GPU profiling metrics](https://run-ai-docs.nvidia.com/self-hosted/platform-management/monitor-performance/gpu-profiling-metrics).
{% endhint %}

<table><thead><tr><th width="177">Metric name in API</th><th width="177">Applicable API endpoint</th><th width="177">Metric name in UI</th><th>Applicable UI table</th></tr></thead><tbody><tr><td><code>GPU_FP16_ENGINE_ACTIVITY_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>GPU FP16 engine activity</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_FP32_ENGINE_ACTIVITY_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>GPU FP32 engine activity</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_FP64_ENGINE_ACTIVITY_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>GPU FP64 engine activity</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_GRAPHICS_ENGINE_ACTIVITY_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>Graphics engine activity</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_MEMORY_BANDWIDTH_UTILIZATION_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>Memory bandwidth utilization</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_NVLINK_RECEIVED_BANDWIDTH_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>NVLink received bandwidth</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_NVLINK_TRANSMITTED_BANDWIDTH_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>NVLink transmitted bandwidth</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_PCIE_RECEIVED_BANDWIDTH_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>PCIe received bandwidth</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_PCIE_TRANSMITTED_BANDWIDTH_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>PCIe transmitted bandwidth</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_SM_ACTIVITY_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>GPU SM activity</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_SM_OCCUPANCY_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>GPU SM occupancy</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_TENSOR_ACTIVITY_PER_GPU</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></li></ul></td><td>GPU tensor activity</td><td><ul><li><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></li><li><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></li></ul></td></tr><tr><td><code>GPU_OOMKILL_SWAP_OUT_OF_RAM_COUNT_PER_GPU</code></td><td><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></td><td>OOMKill swap out of RAM count</td><td><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></td></tr><tr><td><code>GPU_OOMKILL_BURST_COUNT_PER_GPU</code></td><td><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></td><td>OOMKill burst count</td><td><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></td></tr><tr><td><code>GPU_OOMKILL_IDLE_COUNT_PER_GPU</code></td><td><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-nodeid-metrics">Nodes</a></td><td>OOMKill idle count</td><td><a href="../../aiinitiatives/resources/nodes#show-hide-details">Nodes</a></td></tr><tr><td><code>GPU_SWAP_MEMORY_BYTES_PER_GPU</code></td><td><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></td><td>GPU swap memory</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr></tbody></table>

### NVIDIA NIM

NVIDIA NIM metrics provide workload-level observability, including key runtime and performance data such as request throughput, latency, and token usage for LLMs. See [NIM observability metrics via API](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/api-guides/nim-observability-metrics-via-api) for more details.

<table><thead><tr><th width="177.42578125">Metric name in API</th><th width="177">Applicable API endpoint</th><th width="176.8828125">Metric name in UI</th><th>Applicable UI table</th></tr></thead><tbody><tr><td><code>NIM_NUM_REQUESTS_RUNNING</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li></ul></td><td>Request concurrency by status</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr><tr><td><code>NIM_NUM_REQUESTS_WAITING</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li></ul></td><td>Request concurrency by status</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr><tr><td><code>NIM_NUM_REQUEST_MAX</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li></ul></td><td>Request concurrency by status</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr><tr><td><code>NIM_REQUEST_SUCCESS_TOTAL</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li></ul></td><td>Request count by status</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr><tr><td><code>NIM_REQUEST_FAILURE_TOTAL</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li></ul></td><td>Request count by status</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr><tr><td><code>NIM_GPU_CACHE_USAGE_PERC</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li></ul></td><td>GPU KV cache utilization</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr><tr><td><code>NIM_TIME_TO_FIRST_TOKEN_SECONDS</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li></ul></td><td>Time to first token (TTFT)</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr><tr><td><code>NIM_E2E_REQUEST_LATENCY_SECONDS</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li></ul></td><td>End to end request latency</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr><tr><td><code>NIM_TIME_TO_FIRST_TOKEN_SECONDS_PERCENTILES</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li></ul></td><td>Time to first token (TTFT) by percentiles</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr><tr><td><code>NIM_E2E_REQUEST_LATENCY_SECONDS_PERCENTILES</code></td><td><ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics">Pods</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-workloadid-metrics">Workloads</a></li></ul></td><td>End to end request latency by percentiles</td><td><a href="../../../workloads-in-nvidia-run-ai/workloads#show-hide-details">Workloads</a></td></tr></tbody></table>

## Supported Telemetry

| Metric                              | Applicable API endpoint                                                                                                                                                                                                                                                                                                                                                                                                                              | Metric name in UI              | Applicable UI table                                                                                                                                                                                                      |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `WORKLOADS_COUNT`                   | [Workloads](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-telemetry)                                                                                                                                                                                                                                                                                                                                       |                                |                                                                                                                                                                                                                          |
| `ALLOCATED_GPUS`                    | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | Allocated GPUs                 | [Nodes](https://run-ai-docs.nvidia.com/self-hosted/platform-management/aiinitiatives/resources/nodes)                                                                                                                    |
| `GPU_allocation`                    | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/workloads/workloads#get-api-v1-workloads-telemetry">Workloads</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-telemetry">Departments</a></li></ul> |                                |                                                                                                                                                                                                                          |
| `READY_GPU_NODES`                   | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | Ready / Total GPU nodes        | [Overview dashboard](https://run-ai-docs.nvidia.com/self-hosted/platform-management/before-you-start#ui-views)                                                                                                           |
| `READY_GPUS`                        | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | Ready / Total GPU devices      | [Overview dashboard](https://run-ai-docs.nvidia.com/self-hosted/platform-management/before-you-start#ui-views)                                                                                                           |
| `TOTAL_GPU_NODES`                   | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | Ready / Total GPU nodes        | [Overview dashboard](https://run-ai-docs.nvidia.com/self-hosted/platform-management/before-you-start#ui-views)                                                                                                           |
| `TOTAL_GPUS`                        | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | Ready / Total GPU devices      | [Overview dashboard](https://run-ai-docs.nvidia.com/self-hosted/platform-management/before-you-start#ui-views)                                                                                                           |
| `IDLE_ALLOCATED_GPUS`               | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | Idle allocated GPU devices     | [Overview dashboard](https://run-ai-docs.nvidia.com/self-hosted/platform-management/before-you-start#ui-views)                                                                                                           |
| `FREE_GPUS`                         | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | Free GPU devices               | [Nodes](https://run-ai-docs.nvidia.com/self-hosted/platform-management/aiinitiatives/resources/nodes)                                                                                                                    |
| `TOTAL_CPU_CORES`                   | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | CPU (Cores)                    | [Nodes](https://run-ai-docs.nvidia.com/self-hosted/platform-management/aiinitiatives/resources/nodes)                                                                                                                    |
| `USED_CPU_CORES`                    | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               |                                |                                                                                                                                                                                                                          |
| `ALLOCATED_CPU_CORES`               | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry">Nodes</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-telemetry">Departments</a></li></ul>         | <p>Allocated CPU cores<br></p> | [Nodes](https://run-ai-docs.nvidia.com/self-hosted/platform-management/aiinitiatives/resources/nodes)                                                                                                                    |
| `TOTAL_GPU_MEMORY_BYTES`            | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | GPU memory                     | [Nodes](https://run-ai-docs.nvidia.com/self-hosted/platform-management/aiinitiatives/resources/nodes)                                                                                                                    |
| `USED_GPU_MEMORY_BYTES`             | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | Used GPU memory                | [Nodes](https://run-ai-docs.nvidia.com/self-hosted/platform-management/aiinitiatives/resources/nodes)                                                                                                                    |
| `TOTAL_CPU_MEMORY_BYTES`            | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | CPU memory                     | [Nodes](https://run-ai-docs.nvidia.com/self-hosted/platform-management/aiinitiatives/resources/nodes)                                                                                                                    |
| `USED_CPU_MEMORY_BYTES`             | [Nodes](https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry)                                                                                                                                                                                                                                                                                                                                               | Used CPU memory                | [Nodes](https://run-ai-docs.nvidia.com/self-hosted/platform-management/aiinitiatives/resources/nodes)                                                                                                                    |
| `ALLOCATED_CPU_MEMORY_BYTES`        | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/nodes#get-api-v1-nodes-telemetry">Nodes</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-telemetry">Departments</a></li></ul>         | Allocated CPU memory           | <ul><li><a href="../aiinitiatives/resources/nodes">Nodes</a></li><li><a href="../aiinitiatives/organization/projects">Projects</a></li><li><a href="../aiinitiatives/organization/departments">Departments</a></li></ul> |
| `GPU_QUOTA`                         | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-telemetry">Departments</a></li></ul>                                                                                                                                   | GPU quota                      | <ul><li><a href="../aiinitiatives/organization/projects">Projects</a></li><li><a href="../aiinitiatives/organization/departments">Departments</a></li></ul>                                                              |
| `CPU_QUOTA`                         | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-telemetry">Departments</a></li></ul>                                                                                                                                   |                                |                                                                                                                                                                                                                          |
| `MEMORY_QUOTA`                      | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-telemetry">Departments</a></li></ul>                                                                                                                                   |                                |                                                                                                                                                                                                                          |
| `GPU_ALLOCATION_NON_PREEMPTIBLE`    | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-telemetry">Departments</a></li></ul>                                                                                                                                   |                                |                                                                                                                                                                                                                          |
| `CPU_ALLOCATION_NON_PREEMPTIBLE`    | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-telemetry">Departments</a></li></ul>                                                                                                                                   |                                |                                                                                                                                                                                                                          |
| `MEMORY_ALLOCATION_NON_PREEMPTIBLE` | <ul><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/projects#get-api-v1-org-unit-projects-projectid-metrics">Projects</a></li><li><a href="https://app.gitbook.com/s/b5QLzc5pV7wpXz3CDYyp/organizations/departments#get-api-v1-org-unit-departments-telemetry">Departments</a></li></ul>                                                                                                                                   |                                |                                                                                                                                                                                                                          |