Metrics and Telemetry

Metrics are numeric measurements recorded over time that are emitted from the NVIDIA Run:ai cluster and telemetry is a numeric measurement recorded in real-time when emitted from the NVIDIA Run:ai cluster.

Scopes

NVIDIA Run:ai provides control-plane API which supports and aggregates analytics at various levels.

Level	Description
Cluster	A cluster is a set of nodes pools and nodes. With Cluster metrics, metrics are aggregated at the Cluster level. In the NVIDIA Run:ai user interface, metrics are available in the Overview dashboard.
Node	Data is aggregated at the node level.
Node pool	Data is aggregated at the node pool level.
Workload	Data is aggregated at the workload level. In some workloads, e.g. with distributed workloads, these metrics aggregate data from all worker pods.
Pod	The basic unit of execution.
Project	The basic organizational unit. Projects are the tool to implement resource allocation policies as well as the segregation between different initiatives.
Department	Departments are a grouping of projects.

Supported Metrics

Metric name in API	Applicable API endpoint	Metric name in UI per grid	Applicable UI grid
ALLOCATED_GPU	Clusters Node pools	GPU devices (allocated) Allocated GPUs	Overview dashboard Node pools
AVG_WORKLOAD_WAIT_TIME	Clusters Node pools
CPU_LIMIT_CORES	Workloads	CPU limit	Workloads
CPU_MEMORY_LIMIT_BYTES	Workloads	CPU memory limit	Workloads
CPU_MEMORY_REQUEST_BYTES	Workloads	CPU memory request	Workloads
CPU_MEMORY_USAGE_BYTES	Workloads Pods	CPU memory usage	Workloads
CPU_MEMORY_UTILIZATION	Clusters Node pools Nodes	CPU memory utilization	Overview dashboard Node pools Nodes
CPU_REQUEST_CORES	Workloads	CPU request	Workloads
CPU_USAGE_CORES	Nodes Workloads Pods	CPU usage	Workloads
CPU_UTILIZATION	Clusters Node pools Nodes	CPU compute utilization CPU utilization	Overview dashboard and Node pools Nodes
GPU_ALLOCATION	Workloads Projects Departments	GPU devices (allocated)	Overview dashboard
GPU_MEMORY_REQUEST_BYTES	Workloads	GPU memory request	Workloads
GPU_MEMORY_USAGE_BYTES	Workloads Pods Nodes	GPU memory usage	Workloads
GPU_MEMORY_USAGE_BYTES_PER_GPU	Nodes Pods	GPU memory usage per GPU	Workloads per pod
GPU_MEMORY_UTILIZATION	Clusters Node pools	GPU memory utilization	Overview dashboard Node pools
GPU_MEMORY_UTILIZATION_PER_GPU	Nodes	GPU memory utilization per GPU	Nodes
GPU_QUOTA	Clusters Node pools Projects Departments	Quota	Quota management
GPU_UTILIZATION	Clusters Node pools Workloads Pods	GPU compute utilization	Overview dashboard Node pools Workloads
GPU_UTILIZATION_PER_GPU	Nodes Pods	GPU utilization per GPU	Nodes
TOTAL_GPU	Clusters Node pools	GPU devices total Total GPUs	Overview dashboard Node pools
TOTAL_GPU_NODES	Clusters Node pools
GPU_UTILIZATION_DISTRIBUTION	Clusters Node pools	GPU utilization distribution	Node pools
UNALLOCATED_GPU	Clusters Node pools	GPU devices (unallocated) Unallocated GPUs	Overview dashboard Node pools
CPU_QUOTA_MILLICORES	Projects Departments
CPU_MEMORY_QUOTA_MB	Projects Departments
CPU_ALLOCATION_MILLICORES	Projects Departments
CPU_MEMORY_ALLOCATION_MB	Projects Departments
POD_COUNT	Workloads
RUNNING_POD_COUNT	Workloads
NVLINK_BANDWIDTH_TOTAL	Nodes Pods		Nodes Workloads per pod

GPU Profiling

NVIDIA provides extended metrics as shown here. To enable these metrics, please contact NVIDIA Run:ai customer support.

Note

GPU profiling metrics are disabled by default. If unavailable, your administrator must enable it under General settings → Analytics → GPU profiling metrics.

Metric name in API	Applicable API endpoint	Metric name in UI	Applicable UI table
GPU_FP16_ENGINE_ACTIVITY_PER_GPU	Pods Nodes	GPU FP16 engine activity	Workloads Nodes
GPU_FP32_ENGINE_ACTIVITY_PER_GPU	Pods Nodes	GPU FP32 engine activity	Workloads Nodes
GPU_FP64_ENGINE_ACTIVITY_PER_GPU	Pods Nodes	GPU FP64 engine activity	Workloads Nodes
GPU_GRAPHICS_ENGINE_ACTIVITY_PER_GPU	Pods Nodes	Graphics engine activity	Workloads Nodes
GPU_MEMORY_BANDWIDTH_UTILIZATION_PER_GPU	Pods Nodes	Memory bandwidth utilization	Workloads Nodes
GPU_NVLINK_RECEIVED_BANDWIDTH_PER_GPU	Pods Nodes	NVLink received bandwidth	Workloads Nodes
GPU_NVLINK_TRANSMITTED_BANDWIDTH_PER_GPU	Pods Nodes	NVLink transmitted bandwidth	Workloads Nodes
GPU_PCIE_RECEIVED_BANDWIDTH_PER_GPU	Pods Nodes	PCIe received bandwidth	Workloads Nodes
GPU_PCIE_TRANSMITTED_BANDWIDTH_PER_GPU	Pods Nodes	PCIe transmitted bandwidth	Workloads Nodes
GPU_SM_ACTIVITY_PER_GPU	Pods Nodes	GPU SM activity	Workloads Nodes
GPU_SM_OCCUPANCY_PER_GPU	Pods Nodes	GPU SM occupancy	Workloads Nodes
GPU_TENSOR_ACTIVITY_PER_GPU	Pods Nodes	GPU tensor activity	Workloads Nodes
GPU_OOMKILL_SWAP_OUT_OF_RAM_COUNT_PER_GPU	Nodes	OOMKill swap out of RAM count	Nodes
GPU_OOMKILL_BURST_COUNT_PER_GPU	Nodes	OOMKill burst count	Nodes
GPU_OOMKILL_IDLE_COUNT_PER_GPU	Nodes	OOMKill idle count	Nodes
GPU_SWAP_MEMORY_BYTES_PER_GPU	Pods	GPU swap memory	Workloads

NVIDIA NIM

NVIDIA NIM metrics provide workload-level observability, including key runtime and performance data such as request throughput, latency, and token usage for LLMs. See NIM observability metrics via API for more details.

Metric name in API	Applicable API endpoint	Metric name in UI	Applicable UI table
NIM_NUM_REQUESTS_RUNNING	Pods Workloads	Request concurrency by status	Workloads
NIM_NUM_REQUESTS_WAITING	Pods Workloads	Request concurrency by status	Workloads
NIM_NUM_REQUEST_MAX	Pods Workloads	Request concurrency by status	Workloads
NIM_REQUEST_SUCCESS_TOTAL	Pods Workloads	Request count by status	Workloads
NIM_REQUEST_FAILURE_TOTAL	Pods Workloads	Request count by status	Workloads
NIM_GPU_CACHE_USAGE_PERC	Pods Workloads	GPU KV cache utilization	Workloads
NIM_TIME_TO_FIRST_TOKEN_SECONDS	Pods Workloads
NIM_E2E_REQUEST_LATENCY_SECONDS	Pods Workloads
NIM_TIME_TO_FIRST_TOKEN_SECONDS_PERCENTILES	Pods Workloads	Time to first token (TTFT) by percentiles	Workloads
NIM_E2E_REQUEST_LATENCY_SECONDS_PERCENTILES	Pods Workloads	End to end request latency by percentiles	Workloads

Supported Telemetry

Metric	Applicable API endpoint	Metric name in UI	Applicable UI table
WORKLOADS_COUNT	Workloads
ALLOCATED_GPUS	Nodes	Allocated GPUs	Nodes
GPU_allocation	Workloads Projects Departments
READY_GPU_NODES	Nodes	Ready / Total GPU nodes	Overview dashboard
READY_GPUS	Nodes	Ready / Total GPU devices	Overview dashboard
TOTAL_GPU_NODES	Nodes	Ready / Total GPU nodes	Overview dashboard
TOTAL_GPUS	Nodes	Ready / Total GPU devices	Overview dashboard
IDLE_ALLOCATED_GPUS	Nodes	Idle allocated GPU devices	Overview dashboard
FREE_GPUS	Nodes	Free GPU devices	Nodes
TOTAL_CPU_CORES	Nodes	CPU (Cores)	Nodes
USED_CPU_CORES	Nodes
ALLOCATED_CPU_CORES	Nodes Projects Departments	Allocated CPU cores	Nodes
TOTAL_GPU_MEMORY_BYTES	Nodes	GPU memory	Nodes
USED_GPU_MEMORY_BYTES	Nodes	Used GPU memory	Nodes
TOTAL_CPU_MEMORY_BYTES	Nodes	CPU memory	Nodes
USED_CPU_MEMORY_BYTES	Nodes	Used CPU memory	Nodes
ALLOCATED_CPU_MEMORY_BYTES	Nodes Projects Departments	Allocated CPU memory	Nodes Projects Departments
GPU_QUOTA	Projects Departments	GPU quota	Projects Departments
CPU_QUOTA	Projects Departments
MEMORY_QUOTA	Projects Departments
GPU_ALLOCATION_NON_PREEMPTIBLE	Projects Departments
CPU_ALLOCATION_NON_PREEMPTIBLE	Projects Departments
MEMORY_ALLOCATION_NON_PREEMPTIBLE	Projects Departments