GPU utilisation is a timesheet, and AWS is rightsizing from it

NVIDIA defines the figure nvidia-smi prints under “GPU-Util” as the “Percent of time over the past sample period during which one or more kernels was executing on the GPU” [1]. That’s a timesheet. It records that the GPU turned up. Anyone who has managed people knows what a timesheet proves, and it isn’t output.

An H100 in SXM form has 132 streaming multiprocessors, or SMs [2]. A kernel that occupies one of them and never finishes keeps “one or more kernels executing” for 100% of every sample period. By definition, the timesheet reads 100% with 131 SMs idle. One employee in the building, lights on in every office, and the attendance register says full house.

NVIDIA publishes the metric that measures the thing you care about, in the DCGM documentation. SM activity is “The fraction of time at least one warp was active on a multiprocessor, averaged over all multiprocessors” [3]. The same page carries a worked example: “if a GPU has N SMs then a kernel using N blocks that runs over the entire time interval will correspond to an activity of 1 (100%). A kernel using N/5 blocks that runs over the entire time interval will correspond to an activity of 0.2 (20%)” [3]. Put that second kernel on the timesheet and it scores 100%. One workload, the same interval, 100% on one metric and 20% on the other, and only one of them knows the GPU is four-fifths empty.

NVIDIA also tells you how to read SM activity: “A value of 0.8 or greater is necessary, but not sufficient, for effective use of the GPU. A value less than 0.5 likely indicates ineffective GPU usage” [3]. The same page admits that “active” is a generous word, since “warps waiting on memory requests are considered active” [3]. So even the good metric counts a warp stood in a queue as working. Consider what that says about the bar the bad one clears.

There’s no reading guide for the timesheet. There’s nothing to read. A kernel ran.

What AWS reads

AWS Compute Optimizer produces the GPU rightsizing recommendations. It has covered GPU instances since September 2023, when AWS described the recommendations as identifying “instance types, number of GPUs, and provisioned IOPS settings” for accelerated instances [4], and since June 2025 its idle-GPU findings flow into Cost Optimization Hub as well [5]. This is the tool. When someone in a FinOps review says “the recommender hasn’t flagged the GPU fleet”, this is what they mean, and this is what they’re trusting.

Its documentation lists the GPU inputs in full: “Compute Optimizer analyzes the following NVIDIA GPU metrics: nvidia_smi_utilization_gpu, nvidia_smi_memory_used, nvidia_smi_encoder_stats_session_count, nvidia_smi_encoder_stats_average_fps, nvidia_smi_encoder_stats_average_latency, nvidia_smi_temperature_gpu” [6]. Three of those are video-encoder statistics, one is the temperature, and the two that remain are time with a kernel running and memory used. The “GPU over-provisioned” finding is “identified by analyzing the GPUUtilization and GPUMemoryUtilization metrics of the current instance during the look-back period” [7].

And nvidia_smi_utilization_gpu is the timesheet. The CloudWatch agent documentation, which defines what gets collected, says so in AWS’s own words: “The percentage of time over the past sample period during which one or more kernels on the GPU was running” [8].

The Compute Optimizer page describes the same input as “The percentage of allocated GPUs that are currently in use on the instance” [6]. That would be a lovely metric. Somebody should build it. What’s on the page is the CPU sentence from the same table, “The percentage of allocated EC2 compute units that are in use on the instance” [6], with the noun swapped, and it describes something the agent doesn’t collect. A rightsizing service whose own documentation misdescribes its input is a poor advert for the rightsizing.

Memory used is a real signal for memory-bound rightsizing. A model that fits in 24 GB doesn’t need an 80 GB card, and the recommender can see that. It says nothing about compute. A model loaded into memory occupies the same bytes whether the GPU is saturated or dozing.

So the recommender’s entire view of GPU compute is one number, and that number can’t tell a saturated GPU from one running a single warp. The industry spent the best part of a decade teaching engineers that a bare CPU percentage is a poor basis for rightsizing. Then the GPUs arrived, at the prices GPUs command, and the number that got adopted is worse than the CPU percentage, because the CPU figure is at least measured across the allocated compute units [6]. The GPU figure counts attendance.

The number gets thinner on the way in

The nvidia-smi sample period “may be between 1 second and 1/6 second depending on the product” [1]. The CloudWatch agent collects on an interval whose default is 60 seconds [9]. Compute Optimizer “uses the maximum utilization point within each five-minute time interval over the lookback period” [10].

At the defaults that’s five readings per five-minute window, each describing at most one second, and the recommender keeps the highest. On those three documents, the compute figure Compute Optimizer reasons from describes at most one second in every 300, selected for being the busiest. If I did my timesheets on that basis I’d bill every client for the full week on the strength of one good minute, and I’d be the most productive consultant in Scotland. The 14-day look-back, or 93 days with enhanced infrastructure metrics switched on [10], repeats the trick 4,032 times over a fortnight. Repeating a bad sample doesn’t improve it.

Silence is the finding

When Compute Optimizer says a GPU instance is over-provisioned, the timesheet and the memory figure were both low across the look-back: the GPU was clocked out. Take that finding seriously. When Compute Optimizer says nothing, nothing follows. A fleet where every GPU runs a kernel that uses a fifth of the chip reads 100% busy to the recommender for the entire look-back and never produces a finding.

Silence is easy to carry upward as “no GPU waste identified”. It gets a tick in the monthly review, nobody asks a follow-up question, and the invoice stays what it was. On this evidence it should go upward as “no GPU waste visible to the tool”, which is a different sentence, and a much cheaper one to be wrong about.

The idle detection AWS added in June 2025 for Auto Scaling groups on G and P instance types runs on the same agent data: “Once you enable the NVIDIA CloudWatch agent, Compute Optimizer analyzes utilization data and identifies groups that have completed jobs and remained idle” [5]. Idle, it can see. Half-used, it can’t. Idle gets noticed eventually, because somebody asks why a dark fleet is on the bill. Half-used doesn’t, because the dashboard says 100%, and nobody has ever been asked to explain a dashboard that says 100%.

AWS has company

NVIDIA’s own Prometheus exporter for its GPUs, dcgm-exporter, ships a default counter list with the same blind spot. DCGM_FI_DEV_GPU_UTIL, the timesheet, is switched on. DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_SM_OCCUPANCY are in the file, commented out [11], like a note left by a previous tenant. The tensor-pipe and graphics-engine activity counters are on, which is something, but a dashboard built on the defaults has only the timesheet to show under the name “GPU utilisation”.

AWS’s own Container Insights for EKS does better than its EC2 path. It carries a tensor core utilisation metric for every node, pod, and container alongside GPU utilisation [12]. Compute Optimizer’s documented GPU inputs don’t include it [6]. One part of AWS collects a metric that would help the other part, and the two don’t appear to have met.

Everyone reports GPU utilisation. Nobody defines it. I’ve given a whole talk on what happens to a word once everybody uses it and nobody defines it, and “utilisation” is going the way “value” went: a number that’s always available, always green, and never attached to a definition anyone can be held to.

What to do

1. Collect the hardware counters. In DCGM terms: SM activity (field 1002), SM occupancy (1003), tensor pipe activity (1004), and DRAM activity (1005) [3]. In dcgm-exporter, uncomment the two SM lines [11]. One naming trap: NVIDIA’s current documentation calls field 1002 DCGM_FI_PROF_SM_UTIL_RATIO [3][13]; the exporter’s counter file calls it DCGM_FI_PROF_SM_ACTIVE [11]. Same field.
2. Judge against NVIDIA’s bands. SM activity under 0.5 “likely indicates ineffective GPU usage”; 0.8 is “necessary, but not sufficient” [3].
3. Treat Compute Optimizer’s GPU findings as a floor. Over-provisioned means idle, and it’s probably right. No finding means no information, and “no information” is a line the monthly review can’t be allowed to translate into “no waste”.
4. Build the rightsizing case on the interval average of SM activity over a representative period, and print the timesheet figure beside it so the reader sees the gap for themselves. The gap is the business case.
5. For training fleets, ask for model FLOPs utilisation (MFU) as well. Trainy, an ML infrastructure vendor, describes a customer run that had reached “100% GPU utilization and significant power draw” while “only hitting ~20% MFUs” [14]. It’s a vendor’s anecdote about its own customer, with no hardware or model named, so weigh it accordingly. It’s also exactly the shape the definitions predict.
6. Ask whichever vendor sold you the GPU optimisation module which metric it reads. If the answer takes more than one sentence, you have your answer.
7. If you report GPU utilisation to a CFO, say which metric. If you don’t know which metric, that’s the finding.

What I haven’t done

I haven’t run both metrics side by side on a production fleet. The argument above rests on definitions, quoted from NVIDIA and AWS, and on NVIDIA’s own worked example. One independent reproduction exists: a 2023 post ran a one-thread kernel in an infinite loop (simple_kernel<<<1, 1>>>) and reported nvidia-smi showing 100% GPU-Util on a V100 while the DCGM counters showed SM occupancy under 20% [15]. If you have DCGM profiling data and Compute Optimizer findings for the same instances, I’d like to see them. I’ve brought my receipts. Bring yours.

Sources

All read on 5 November 2026.

1. NVIDIA, nvidia-smi documentation, “Utilization”: docs.nvidia.com/deploy/nvidia-smi
2. NVIDIA, “NVIDIA Hopper Architecture In-Depth”: developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth
3. NVIDIA, DCGM documentation, “Profiling metrics”: docs.nvidia.com/datacenter/dcgm/latest/learn/modules/profiling.html
4. AWS, “AWS Compute Optimizer supports rightsizing for G4dn and P3 instances”, 5 September 2023: aws.amazon.com/about-aws/whats-new/2023/09/aws-compute-optimizer-rightsizing-g4dn-p3-instances
5. AWS, “AWS Compute Optimizer now identifies idle EC2 Auto Scaling groups with GPU instances”, 16 June 2025: aws.amazon.com/about-aws/whats-new/2025/06/aws-compute-optimizer-idle-ec2-auto-scaling-groups
6. AWS, Compute Optimizer User Guide, “EC2 instance metrics”: docs.aws.amazon.com/compute-optimizer/latest/ug/ec2-metrics-analyzed.html
7. AWS, Compute Optimizer User Guide, “Viewing EC2 instance recommendations”, finding reasons: docs.aws.amazon.com/compute-optimizer/latest/ug/view-ec2-recommendations.html
8. AWS, CloudWatch User Guide, “Collect NVIDIA GPU metrics”: docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-NVIDIA-GPU.html
9. AWS, CloudWatch User Guide, “Manually create or edit the CloudWatch agent configuration file”, metrics_collection_interval: docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html
10. AWS, Compute Optimizer User Guide, “Metrics analyzed by AWS Compute Optimizer”: docs.aws.amazon.com/compute-optimizer/latest/ug/metrics.html
11. NVIDIA, dcgm-exporter, etc/default-counters.csv, main branch: github.com/NVIDIA/dcgm-exporter/blob/main/etc/default-counters.csv
12. AWS, CloudWatch User Guide, “Container Insights metrics for Amazon EKS”, NVIDIA GPU metrics: docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-enhanced-EKS.html
13. NVIDIA, DCGM API reference, “Field Identifiers”: docs.nvidia.com/datacenter/dcgm/latest/dcgm-api/dcgm-api-field-ids.html
14. Trainy, “GPU Utilization is a Misleading Metric”: trainy.ai/blog/gpu-utilization-misleading
15. Arthur Chiao, “Understanding NVIDIA GPU Performance: Utilization vs. Saturation”, 27 August 2023: arthurchiao.art/blog/understanding-gpu-performance