GPU Monitoring Fleet Page

Docs > GPU Monitoring > GPU Monitoring Fleet Page

Overview

GPU Fleet Explorer provides a detailed breakdown across AI infrastructure (such as GPU devices, hosts, or Ray clusters) and AI workloads (such as Pods, Kube Containers, or Training Runs) for a specified time frame. This page helps you uncover provisioning inefficiencies and workload performance optimizations through resource telemetry, including GPU utilization, host-level metrics, and costs. It also surfaces Datadog’s real-time detection of issues impacting your fleet and workloads, with guidance on how to remediate them.

Detect issues with out-of-the-box monitors

Datadog provides several out-of-the-box (OOTB) monitor templates that detect common GPU issues in real time, including:

Temperature spikes
Power cap throttling
Unmet GPU requests
XID errors
ECC errors
Bursty workloads
Idle devices

You can customize any monitor’s thresholds to fit your organization’s needs.

To access these templates, click the Monitors dropdown in the top-right corner of the page.

Monitors dropdown in the top-right corner of the GPU Fleet page, showing OOTB monitor templates for Temperature, Power Cap Throttling, Unmet GPU Requests, Critical XID Errors, General XID Errors, ECC Errors, Bursty Workloads, and Idle Devices

Break down your fleet by any tag

GPU Fleet Explorer gives you visibility from your AI workloads down to the underlying AI infrastructure. You can switch between workload entities like pods and training runs, and infrastructure entities such as devices, hosts, and clusters.

Side navigation bar showing AI Infrastructure entities (Devices, Hosts, Kube Clusters, Ray Clusters) and AI Workloads entities (Pods, Kube Containers, Training Runs)

Note: Kube Clusters, Pods, and Kube Containers options in the side navigation are only available if you use Kubernetes.

Use the filter dropdowns at the top of the page to filter by a specific Provider, Device Type, Cluster, Region, Service, Data Center, Environment, or Team.

You can also Search or Group by other tags using the search and group-by fields. For example, you can group by Service to view a row in the table for each unique service. Click the > button next to any entry to see the devices for that service.

GPU Fleet table showing services with their device types, with the row expand button highlighted

Filter dropdowns and Group by selector at the top of the GPU Fleet page

Use case-driven views and recommendations

GPU Monitoring’s Fleet Explorer page provides two dedicated use case-driven views:

Provisioning: Allocate capacity and manage quotas.
Performance: Optimize workload efficiency and throughput.

Provisioning

The Provisioning tab detects any hardware health issues impacting the allocation of your devices to workloads and provides guidance on how to remediate them.

For each detected issue, Datadog recommends step-by-step remediation actions to help you resolve it.

Recommended remediation actions for a critical XID error

Provisioning summary graph

The summary graph provides out-of-the-box (OOTB) visualizations for key telemetry tied to your selected use case-driven view. For the Provisioning use case, this breaks down your Provisioned, Allocated, and Active devices so you can reclaim wasted idle spend and rediscover available devices that can be allocated to workloads.

Summary graph showing Provisioned Devices, Allocated Devices, and Active Devices breakdowns

Expand this section below to see a table of the available options and what they represent.

See full list of Provisioning summary graph options

Option	Definition
Provisioned Devices	Breakdown of provisioned devices and active devices.
Allocated Devices	Breakdown of available devices by allocated vs. unallocated.
Active Devices	Breakdown of allocated devices by active vs. idle.

Performance

The Performance tab detects any hardware health or workload issues that are stalling the workloads running on your GPU devices. This provides a single source of truth and guidance to platform engineers and AI/ML teams on how to remediate those issues.

For each detected issue, Datadog recommends step-by-step remediation actions to help you resolve it.

Recommended remediation actions for a GPU power cap throttling issue

Performance summary graph

The summary graph provides out-of-the-box (OOTB) visualizations for key telemetry tied to your selected use case-driven view. For the Performance use case, this breaks down key resource utilization metrics like GPU Utilization or GPU Saturation alongside network fabric metrics, power, temperature, and more.

Summary graph showing GPU Util, GPU Saturation, and GPU Memory breakdowns

Expand the section below to see a table of the available options and what they represent.

See full list of Performance summary graph options

Option	Definition
CPU Util	Percentage of time the CPU spent running user space processes.
Host Memory	Percentage of usable memory in use.
GPU Util	Average percentage of time each streaming multiprocessor was active (lower values indicate idle time).
GPU Saturation	Measures how fully the GPU’s parallel execution capacity is being used during the time frame (average ratio of active warps to the maximum warps supported per streaming multiprocessor across all SMs).
GPU Memory	Percentage of GPU memory used to total GPU memory limit.
PCIe RX	Bytes received through PCI from the GPU device per second.
PCIe TX	Bytes transmitted through PCI to the GPU device per second.
NVLink RX	Total RX of all NVLINK links.
NVLink TX	Total TX of all NVLINK links.
Graphics Activity	Fraction of time the GPU was performing any compute work during the interval. A coarse signal of whether the GPU is busy or idle.
Power	Power usage for the GPU device. On GA100 and previous architectures, this represents the instantaneous power at that moment. For newer architectures, it represents the average power draw (Watts) over one second.
Temperature	Temperature of a GPU device.
SM Clock	SM clock frequency in MHz.
NVLink Active Links	Number of active NVLINK links for the device.
ECC Errors	Total count of uncorrected ECC errors.
XID Errors	Count of NVIDIA XID errors, indicating hardware or driver-level issues.

Inventory of your GPU-powered infrastructure

This table breaks down your GPU-powered infrastructure by any tag of your choosing. If you haven’t specified an additional tag in the Group by field, results are grouped by your selected view: Cluster, Host, or Device.

You can click on the gear icon to customize which metrics are displayed within the table. Expand the section below to see a full list of the available metrics.

See the full list of available metrics

Metric	Definition	Metric Name	Provisioning Tab	Performance Tab
Idle Cost	(Only nonzero for time frames longer than 2 days) The cost of GPU resources that are reserved and allocated, but not used.	`gpu_monitoring.estimated_idle_cost`	✓	✓
Total Devices	GPU devices with Datadog’s GPU Monitoring correctly configured and reporting metrics.	`kubernetes_state.node.gpu_capacity`	✓	—
Kubernetes Available	Healthy GPU devices that are powered on and available for allocation, according to the Kubernetes orchestrator.	`kubernetes_state.node.gpu_allocatable`	✓	—
Allocated Devices	(Only available if using Kubernetes) Count of devices that have been allocated to a workload.	`gpu.device.total`	✓	—
Unallocated Devices	Count of devices not allocated and available for use during time frame.		✓	—
Active Devices	Count of devices that are actively used for a workload or are busy. If using Kubernetes: count of allocated devices that are actively used for a workload.	`gpu.gr_engine_active`	✓	—
Idle Devices	GPU devices allocated to workloads but not doing any work during the time frame. A device is considered idle if `gpu.gr_engine_active` equals 0.	`gpu.gr_engine_active`	✓	—
CPU Utilization	Percentage of time the CPU spent running user space processes.	`system.cpu.user`	—	✓
Host Memory	Percentage of usable memory in use.	`system.mem.pct_usable`	—	✓
GPU Utilization	Average percentage of time each streaming multiprocessor was active (lower values indicate idle time).	`gpu.sm_active`	—	✓
GPU Saturation	Measures how fully the GPU’s parallel execution capacity is being used during the time frame (average ratio of active warps to the maximum warps supported per streaming multiprocessor across all SMs).	`gpu.sm_occupancy`	—	✓
GPU Memory	Percentage of GPU memory used to total GPU memory limit.	`100 - (gpu.memory.free / gpu.memory.limit * 100)`	—	✓
PCIe RX Throughput	Bytes received through PCI from the GPU device per second.	`gpu.pci.throughput.rx`	—	✓
PCIe TX Throughput	Bytes transmitted through PCI to the GPU device per second.	`gpu.pci.throughput.tx`	—	✓
NVLink RX	Total RX of all NVLINK links.	`gpu.nvlink.throughput.raw.rx`	—	✓
NVLink TX	Total TX of all NVLINK links.	`gpu.nvlink.throughput.raw.tx`	—	✓
Power	Power usage for the GPU device. Note: On GA100 and previous architectures, this represents the instantaneous power at that moment. For newer architectures, it represents the average power draw (Watts) over one second.	`gpu.power.usage`	—	✓
Temperature	Temperature of a GPU device.	`gpu.temperature`	—	✓

Details side panel

Clicking any row in the Fleet table opens a side panel with more details for the selected cluster, host, or device.

Connected entities

Datadog’s GPU Monitoring doesn’t need to rely on NVIDIA’S DCGM Exporter. It uses the Datadog Agent to observe GPUs directly, providing insight into GPU usage and costs for pods and processes. Under the Connected Entities section in any detail view, you can see SM activity, GPU core utilization (only if System Probe is enabled), and the memory usage of pods, processes, and Slurm jobs. This helps you identify which workloads to cut or optimize to decrease total spend.

Note: The Pods tab is only available if you’re using Kubernetes.

Within this side panel, you have a cluster-specific funnel that identifies:

Number of Total, Allocated (Kubernetes users only) , Active, and Effective devices within that particular cluster
Estimated total and idle cost of that cluster
Connected entities of that cluster: pods, processes, and Slurm jobs
Four key metrics (customizable) for that cluster: Core Utilization (only if System probe is enabled), Memory Utilization, PCIe Throughput, and Graphics Activity
Table of hosts associated with that cluster

Within this side panel, you have a host-specific view that identifies:

Host-related metadata such as provider, instance type, CPU utilization, system memory used, system memory total, system I/O util, SM activity, and temperature
(only available for Kubernetes users) The specific GPU devices allocated to that host sorted by Graphics Engine Activity
Connected entities of that host: pods, processes, and Slurm jobs

Within this side panel, you have a device-specific view that identifies:

Recommendations (if any) for how to use this device more effectively
Device-related details: device type, SM activity, and temperature
Four key metrics tied to GPUs: SM Activity, Memory Utilization, Power, and Graphics Engine Activity
Connected entities of that cluster: pods and processes

Installation recommendations

Datadog actively surveys your infrastructure and detects installation gaps that may diminish the value you get out of GPU Monitoring. In this modal, you can find installation update recommendations to get the optimal value of GPU Monitoring. For example, making sure your hosts have the latest version of the Datadog Agent installed, installing the latest version of the NVIDIA driver, and checking for misconfigured hosts.

Do not use Datadog Agent v7.82.0, which can cause unexpected kernel panics.

To view advanced GPU Monitoring features such as attribution of GPU resources by related processes or SLURM jobs, you must enable Live Processes and the Slurm integration, respectively.