The short answer: our “GPU outliers” page is a triage view. It ranks the worst graphics cost seen for each tested feature in the selected releases. It is not a statistical anomaly detector, and it is definitely not one universal number that can be compared across every machine.

≤10GPU-ms per wall second
expected band
50investigate
a sensitive dashboard marker
>100poor band
in the harness report

The workload is a feature, not a shader demo

The cleanest reproduction is scroll-giant-thread. The harness opens a large seeded conversation, places the pointer over the message pane, and alternates 600-pixel wheel events every 50 milliseconds for five seconds. That exercises the thing we care about: T3 Code rendering a realistic, large timeline under sustained input. Playwright documents that wheel events are dispatched without waiting for scrolling to finish, so the test is intentionally continuous rather than scroll, settle, repeat.

The same measurement window wraps other expensive product paths—streaming assistant output, a 10,000-line terminal burst, and Electron picture-in-picture capture. The dashboard can therefore answer “which feature and release made the graphics path work hardest?” The scrolling test is simply the controlled place to reproduce a finding.

Two signals that must not be mixed

GPU busy time is attributed work in the Chromium GPU process, normalized to one wall second.

GPU-helper CPU is CPU time burned by that same helper. On software-rendered Linux, this is the honest rendering-cost signal.

Chromium deliberately puts raster and compositor work into a dedicated GPU process; its architecture notes say the graphics-driver calls happen there. The harness discovers that typed process through the Chrome DevTools Protocol, or through Electron’s documented app.getAppMetrics(). That process boundary is useful because it follows the app’s graphics work rather than whole-machine activity.

The normalization is deliberately plain: 250 milliseconds of attributed GPU work during a five-second action is 50 GPU-ms/s. Each scenario normally runs five measured repetitions; reused environments get an unrecorded warm-up first. Grafana shows their median; the harness report grades the p75, which is less forgiving of an intermittently expensive run.

Same hardware does not mean same sensor

The fleet has two human-readable device types: a MacBook Pro and three identical Intel i7-8700 Linux workers. That is the right level for ordinary performance comparisons. GPU measurement adds another dimension, gpu_backend, because the machines do not all expose the same counters.

  • MacBook Pro / AGX: the harness samples cumulative Apple Silicon GPU counters from the I/O Registry. Apple documents the I/O Registry, but not these particular AGX accounting fields as a stable public API. This is measured and useful, but version-sensitive.
  • Linux / DRM fdinfo: when the driver exposes it, the harness reads per-client drm-engine-* busy-time counters. The Linux kernel specification defines those fields and warns that driver support is optional.
  • Linux / no GPU attribution: a zero GPU-busy value means “unavailable,” not necessarily “free.” The dashboard excludes those zeros from GPU-busy comparisons and uses GPU-helper CPU instead.
  • NVIDIA fallback: where available, nvidia-smi pmon provides sampled per-process SM utilization. That is an estimate, not the same precision as a cumulative busy-time counter.

The practical rule: compare adjacent releases for the same scenario, surface, device type, and measurement backend. Do not average AGX, DRM, an NVIDIA estimate, and an unavailable counter into one confident-looking line.

Why “outlier” needs scare quotes

The ranking panel selects each release’s median result, averages identical workers within that release, keeps the worst value within the chosen time range, and sorts descending. The dashboard opens on the last seven days. Prometheus’s last_over_time recovers the point for each release series; the surrounding aggregation chooses the worst release. There is no z-score, interquartile range, or median absolute deviation.

That makes the panel useful for finding where to look, but a single tall bar is not a verdict. A sustained step across adjacent releases is stronger evidence. So is a matching increase in dropped frames or wall time. The next move is to rerun the exact combination and inspect its individual samples before assigning a cause.

The thresholds are heuristics

The harness explicitly calls its graphics bands derived rather than vendor standards. At or below 10 GPU-ms/s is the expected band for this mostly-text interface. The dashboard’s 50 ms/s line is an early “investigate this” marker. The report’s poor band begins above 100 ms/s. Neither threshold is an Apple, Chromium, Electron, Linux, or NVIDIA conformance limit.

That distinction matters. The goal is not to manufacture a red alert. It is to make a release regression obvious early enough that someone can reproduce it, find the render loop or repaint that changed, and fix it before a coding tool quietly taxes its user all day.

How I read the dashboard now

  1. Choose one device type and confirm the GPU measurement backend.
  2. Start with “worst cost seen in range” to find a feature worth opening.
  3. Use the build timeline to find the first release where the step appears.
  4. Check GPU-helper CPU, dropped frames, and wall time before calling it user-visible.
  5. Rerun the suspicious release and scenario; treat one isolated point as a lead, not proof.

Sources and implementation notes