# Metrics and Telemetry Metrics are numeric measurements recorded **over time** that are emitted from the NVIDIA Run:ai cluster and telemetry is a numeric measurement recorded in real-time when emitted from the NVIDIA Run:ai cluster. ## Scopes NVIDIA Run:ai provides control-plane API which supports and aggregates analytics at various levels. | Level | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Cluster | A cluster is a set of nodes pools and nodes. With Cluster metrics, metrics are aggregated at the Cluster level. In the NVIDIA Run:ai user interface, metrics are available in the Overview dashboard. | | Node | Data is aggregated at the node level. | | Node pool | Data is aggregated at the node pool level. | | Workload | Data is aggregated at the workload level. In some workloads, e.g. with distributed workloads, these metrics aggregate data from all worker pods. | | Pod | The basic unit of execution. | | Project | The basic organizational unit. Projects are the tool to implement resource allocation policies as well as the segregation between different initiatives. | | Department | Departments are a grouping of projects. | ## Supported Metrics | Metric name in API | Applicable API endpoint | Metric name in UI per grid | Applicable UI grid | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `ALLOCATED_GPU` | | | | | `AVG_WORKLOAD_WAIT_TIME` | | | | | `CPU_LIMIT_CORES` | [Workloads](https://run-ai-docs.nvidia.com/api/2.22/workloads/workloads#get-api-v1-workloads-workloadid-metrics) | CPU limit | [Workloads](/self-hosted/2.22/workloads-in-nvidia-run-ai/workloads.md#metrics) | | `CPU_MEMORY_LIMIT_BYTES` | [Workloads](https://run-ai-docs.nvidia.com/api/2.22/workloads/workloads#get-api-v1-workloads-workloadid-metrics) | CPU memory limit | [Workloads](/self-hosted/2.22/workloads-in-nvidia-run-ai/workloads.md#metrics) | | `CPU_MEMORY_REQUEST_BYTES` | [Workloads](https://run-ai-docs.nvidia.com/api/2.22/workloads/workloads#get-api-v1-workloads-workloadid-metrics) | CPU memory request | [Workloads](/self-hosted/2.22/workloads-in-nvidia-run-ai/workloads.md#metrics) | | `CPU_MEMORY_USAGE_BYTES` | | CPU memory usage | [Workloads](/self-hosted/2.22/workloads-in-nvidia-run-ai/workloads.md#metrics) | | `CPU_MEMORY_UTILIZATION` | | CPU memory utilization | | | `CPU_REQUEST_CORES` | [Workloads](https://run-ai-docs.nvidia.com/api/2.22/workloads/workloads#get-api-v1-workloads-workloadid-metrics) | CPU request | [Workloads](/self-hosted/2.22/workloads-in-nvidia-run-ai/workloads.md) | | `CPU_USAGE_CORES` | | CPU usage | [Workloads](/self-hosted/2.22/workloads-in-nvidia-run-ai/workloads.md#metrics) | | `CPU_UTILIZATION` | | | | | `GPU_ALLOCATION` | | GPU devices (allocated) | [Overview dashboard](/self-hosted/2.22/platform-management/monitor-performance/before-you-start.md#ui-views) | | `GPU_MEMORY_REQUEST_BYTES` | [Workloads](https://run-ai-docs.nvidia.com/api/2.22/workloads/workloads#get-api-v1-workloads-workloadid-metrics) | GPU memory request | [Workloads](/self-hosted/2.22/workloads-in-nvidia-run-ai/workloads.md#metrics) | | `GPU_MEMORY_USAGE_BYTES` | | GPU memory usage | [Workloads](/self-hosted/2.22/workloads-in-nvidia-run-ai/workloads.md#metrics) | | `GPU_MEMORY_USAGE_BYTES_PER_GPU` | | GPU memory usage per GPU | [Workloads per pod](/self-hosted/2.22/workloads-in-nvidia-run-ai/workloads.md#metrics) | | `GPU_MEMORY_UTILIZATION` | | GPU memory utilization | | | `GPU_MEMORY_UTILIZATION_PER_GPU` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-nodeid-metrics) | GPU memory utilization per GPU | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md#show-hide-details) | | `GPU_QUOTA` | | Quota | [Quota management](/self-hosted/2.22/platform-management/monitor-performance/before-you-start.md#ui-views) | | `GPU_UTILIZATION` | | GPU compute utilization | | | `GPU_UTILIZATION_PER_GPU` | | GPU utilization per GPU | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md#show-hide-details) | | `TOTAL_GPU` | | | | | `TOTAL_GPU_NODES` | | | | | `GPU_UTILIZATION_DISTRIBUTION` | | GPU utilization distribution | [Node pools](/self-hosted/2.22/platform-management/aiinitiatives/resources/node-pools.md#show-hide-details) | | `UNALLOCATED_GPU` | | | | | `CPU_QUOTA_MILLICORES` | | | | | `CPU_MEMORY_QUOTA_MB` | | | | | `CPU_ALLOCATION_MILLICORES` | | | | | `CPU_MEMORY_ALLOCATION_MB` | | | | | `POD_COUNT` | [Workloads](https://run-ai-docs.nvidia.com/api/2.22/workloads/workloads#get-api-v1-workloads-workloadid-metrics) | | | | `RUNNING_POD_COUNT` | [Workloads](https://run-ai-docs.nvidia.com/api/2.22/workloads/workloads#get-api-v1-workloads-workloadid-metrics) | | | ### Advanced Metrics NVIDIA provides extended metrics as shown [here](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html#profiling-metrics). {% hint style="info" %} **Note** Advanced metrics are disabled by default. If unavailable, your Administrator must enable it under **General Settings** → Analytics → Advanced metrics. Before enabling, the administrator must configure GPU profiling through the DCGM Exporter and NVIDIA Run:ai Prometheus integration. For configuration steps, see [Advanced metrics](/self-hosted/2.22/platform-management/monitor-performance/advanced-metrics.md). {% endhint %} | Metric name in API | Applicable API endpoint | Metric name in UI | Applicable UI table | | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GPU_FP16_ENGINE_ACTIVITY_PER_GPU` | | GPU FP16 engine activity | | | `GPU_FP32_ENGINE_ACTIVITY_PER_GPU` | | GPU FP32 engine activity | | | `GPU_FP64_ENGINE_ACTIVITY_PER_GPU` | | GPU FP64 engine activity | | | `GPU_GRAPHICS_ENGINE_ACTIVITY_PER_GPU` | | Graphics engine activity | | | `GPU_MEMORY_BANDWIDTH_UTILIZATION_PER_GPU` | | Memory bandwidth utilization | | | `GPU_NVLINK_RECEIVED_BANDWIDTH_PER_GPU` | | NVLink received bandwidth | | | `GPU_NVLINK_TRANSMITTED_BANDWIDTH_PER_GPU` | | NVLink transmitted bandwidth | | | `GPU_PCIE_RECEIVED_BANDWIDTH_PER_GPU` | | PCIe received bandwidth | | | `GPU_PCIE_TRANSMITTED_BANDWIDTH_PER_GPU` | | PCIe transmitted bandwidth | | | `GPU_SM_ACTIVITY_PER_GPU` | | GPU SM activity | | | `GPU_SM_OCCUPANCY_PER_GPU` | | GPU SM occupancy | | | `GPU_TENSOR_ACTIVITY_PER_GPU` | | GPU tensor activity | | | `GPU_OOMKILL_SWAP_OUT_OF_RAM_COUNT_PER_GPU` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-nodeid-metrics) | OOMKill swap out of RAM count | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md#show-hide-details) | | `GPU_OOMKILL_BURST_COUNT_PER_GPU` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-nodeid-metrics) | OOMKill burst count | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md#show-hide-details) | | `GPU_OOMKILL_IDLE_COUNT_PER_GPU` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-nodeid-metrics) | OOMKill idle count | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md#show-hide-details) | | `GPU_SWAP_MEMORY_BYTES_PER_GPU` | [Pods](https://run-ai-docs.nvidia.com/api/2.22/workloads/pods#get-api-v1-workloads-workloadid-pods-podid-metrics) | GPU swap memory | [Workloads](/self-hosted/2.22/workloads-in-nvidia-run-ai/workloads.md#show-hide-details) | ## Supported Telemetry | Metric | Applicable API endpoint | Metric name in UI | Applicable UI table | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `WORKLOADS_COUNT` | [Workloads](https://run-ai-docs.nvidia.com/api/2.22/workloads/workloads#get-api-v1-workloads-telemetry) | | | | `ALLOCATED_GPUS` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | Allocated GPUs | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md) | | `GPU_allocation` | | | | | `READY_GPU_NODES` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | Ready / Total GPU nodes | [Overview dashboard](/self-hosted/2.22/platform-management/monitor-performance/before-you-start.md#ui-views) | | `READY_GPUS` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | Ready / Total GPU devices | [Overview dashboard](/self-hosted/2.22/platform-management/monitor-performance/before-you-start.md#ui-views) | | `TOTAL_GPU_NODES` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | Ready / Total GPU nodes | [Overview dashboard](/self-hosted/2.22/platform-management/monitor-performance/before-you-start.md#ui-views) | | `TOTAL_GPUS` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | Ready / Total GPU devices | [Overview dashboard](/self-hosted/2.22/platform-management/monitor-performance/before-you-start.md#ui-views) | | `IDLE_ALLOCATED_GPUS` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | Idle allocated GPU devices | [Overview dashboard](/self-hosted/2.22/platform-management/monitor-performance/before-you-start.md#ui-views) | | `FREE_GPUS` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | Free GPU devices | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md) | | `TOTAL_CPU_CORES` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | CPU (Cores) | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md) | | `USED_CPU_CORES` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | | | | `ALLOCATED_CPU_CORES` | |

Allocated CPU cores

| [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md) | | `TOTAL_GPU_MEMORY_BYTES` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | GPU memory | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md) | | `USED_GPU_MEMORY_BYTES` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | Used GPU memory | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md) | | `TOTAL_CPU_MEMORY_BYTES` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | CPU memory | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md) | | `USED_CPU_MEMORY_BYTES` | [Nodes](https://run-ai-docs.nvidia.com/api/2.22/organizations/nodes#get-api-v1-nodes-telemetry) | Used CPU memory | [Nodes](/self-hosted/2.22/platform-management/aiinitiatives/resources/nodes.md) | | `ALLOCATED_CPU_MEMORY_BYTES` | | Allocated CPU memory | | | `GPU_QUOTA` | | GPU quota | | | `CPU_QUOTA` | | | | | `MEMORY_QUOTA` | | | | | `GPU_ALLOCATION_NON_PREEMPTIBLE` | | | | | `CPU_ALLOCATION_NON_PREEMPTIBLE` | | | | | `MEMORY_ALLOCATION_NON_PREEMPTIBLE` | | | | --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://run-ai-docs.nvidia.com/self-hosted/2.22/platform-management/monitor-performance/metrics.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.