terraform apply reported nine checks created, no errors. Which established exactly one thing: nine check definitions now exist in a vendor’s database.
The scripts inside them run somewhere else entirely — on Grafana’s probe machines, on their schedule, with no path back to your terminal. A script that aborts on its first line produces an apply that looks identical to one that works. That gap, between configured and measuring, is where this system tried to fail three separate times.
TL;DR: A clean apply proves the config was accepted, nothing more. Verify with probe_success and count(probe_browser_web_vital_lcp) in Explore plus the error log in Loki, because a broken check script fails on the vendor’s probes and nowhere you’d look. Then: every vitals panel went red because k6 reports milliseconds and the panels were declared as seconds — with CLS the sole green panel, which is the tell, since CLS is the only unitless one. And the failure this setup can’t see on its own is a page silently dropping out of a script, so the dashboard carries a tile that counts reporting URLs against a number Terraform bakes in from the inventory.
How Do You Know a Check Is Actually Running?
You ask the data, because nothing else will tell you.
The first armed run failed. Not visibly — the checks existed, the dashboard was drawn, terraform plan was clean. The failure was in the logs on the vendor’s side:
scenario 'ui' doesn't have a specified executor type
That’s the incomplete options block — a documentation snippet showing the browser fragment, copied as though it were a whole options object. The point here isn’t the fix. It’s that Terraform had done its job perfectly and the system was measuring nothing, and there is no arrangement of plan, apply, or state inspection that would have said so.
So a template change gets three checks afterward, in Explore:
probe_success # 1 per check and probe
count(probe_browser_web_vital_lcp) # 0 means the script never reached a page
and the errors themselves in Loki:
{check_name="browser"} |= "level=error"
The middle one is the load-bearing query. probe_success can be 1 for a check whose script threw after navigating — the run completed, the vendor is satisfied, and no vital was recorded. A count of zero on the vitals is the difference between "green" and "working."
Worth building the same gap into the rollout rather than only into the checking. The first apply creates the checks without running them:
# Armed on 2026-08-28 after the checks had been reviewed in the UI.
variable "enabled" {
description = "Arm the checks. false creates them without running them."
default = true
}
Create, inspect, then arm. Nothing is billed in between, and the review happens against real objects rather than a plan diff.
Why Was Every Panel Red Except CLS?
Because k6 reports web vitals in milliseconds, and the panels had been declared in seconds. Grafana rendered 613 ms as "613 s", compared it against thresholds meant for seconds, and painted the entire dashboard red while the sites were, in fact, fast.
The diagnosis is the exception. Cumulative Layout Shift is the one Core Web Vital with no unit — it’s a ratio. It was the only panel still green.
Everything red except the unitless metric is a unit bug, not an outage. That is a genuinely reusable tell, and it generalizes past this dashboard: when a fault hits every member of a set except the one that differs in exactly one property, that property is the fault. A real performance collapse has no reason to spare CLS; a seconds-versus-milliseconds error cannot touch it.
What the numbers looked like once the units were right:
| Location | per page | LCP p75 | TTFB p75 |
|---|---|---|---|
| Frankfurt | 0.55–0.98 s | 220 ms | 17 ms |
| Paris | ~1.8 s | 612 ms | 72 ms |
| London | ~2.0 s | 698 ms | 81 ms |
Comfortably inside the "good" band, LCP included, against a 2500 ms budget. The dashboard had been claiming a site-wide emergency.
What Failure Can This Setup Not See?
A page quietly leaving a script.
Consider the shape of it: several sites share one check, so a check covers many URLs. If one URL stops being visited — an inventory edit, a template change, a goto that throws where the loop swallows it — then probe_success stays 1, every panel stays green, the alert rules have nothing to fire on, and that page is simply no longer measured. There is no red anywhere. The monitoring reports health for a page it isn’t looking at.
Nothing in the metrics catches that, because the absence of a series is not a series. So the count has to be compared against an expectation that lives outside the metrics — and Terraform already holds it, in the inventory:
title = "URLs reporting, all checks (of ${length(local.entries)} in the inventory)"
The expected number is baked in at generation time. Fewer URLs reporting than configured turns the tile red.
With one deliberate exception that took a round to get right: this tile ignores the dashboard filters. Every other panel follows them. Filter the view down to two news checks and 15 URLs report — correct, healthy, and displayed as a failure, because the tile was comparing a filtered count against the full-estate threshold. The fix isn’t to make the threshold follow the filter; it’s to make the tile refuse the filter, because it answers a question about the whole estate rather than about the current view.
That’s the general rule worth taking: a panel comparing against a global expectation must not inherit a local filter. Mixing the two produces a monitor that goes red when someone narrows the view, which trains people to ignore it — the worst outcome available for an indicator whose entire job is to be believed.
What Else Reads Wrong on a Dashboard?
Three things that aren’t bugs but will be reported as bugs.
Check runtime next to page metrics. A check walking twelve pages takes about 25 seconds. Placed beside LCP and TTFB, that number reads as though a page took 25 seconds to load. It describes the check — roughly 2 seconds per page including a browser start each — so it belongs in a collapsed section at the bottom, with the other mechanics. The dashboard is otherwise ordered by the sequence in which questions actually get asked: is everything green, what stands out, then each page individually, then how the check itself behaved.
Ad blockers break classic dashboards entirely. Not the custom one specifically — every classic dashboard, including the vendor’s own. The per-check Synthetics pages keep working, because those are app pages rather than dashboards. The result is a failure that looks extremely specific to your configuration when it’s precisely the opposite, and it will consume an afternoon before someone tries a different browser.
Mobile is emulation, not a slow phone. The mobile check sets a 412×915 viewport at scale 2.625, touch, and an Android user agent. It changes viewport, user agent, and touch. It throttles neither CPU nor network. Those numbers describe the mobile layout over a datacenter connection, which is a useful thing to track and not the thing most people assume they’re reading. Label it on the panel, or someone will quote it in a meeting as what users experience.
And one absence that’s a design decision rather than a gap: there is no INP. Interaction to Next Paint requires a real interaction, and a test browser doesn’t click. It isn’t collectible synthetically at all — it belongs to real-user monitoring, and its absence from a lab dashboard is the boundary working as intended. An earlier post covers what INP does catch, in the place it can be caught.
What’s the Generalizable Lesson Here?
- A successful
applyproves the vendor accepted your configuration. It proves nothing about execution, because the execution happens on their machines. Any infrastructure-as-code that ships a script to be run elsewhere needs a verification step that queries the result, not the state file. probe_successand "a measurement arrived" are different questions. Check for the presence of the metric you actually care about; a count of zero is the honest signal, and green is not.- When a fault spares exactly one member of a set, the property that member doesn’t share is the fault. Every vital red except the unitless one is a units bug — no outage has that shape.
- The absence of a series is not a series, so it can’t alert. Any expectation about how many things should be reporting has to come from outside the metrics — from the same source of truth that generated the config, ideally baked in at generation time.
- A panel that compares against a global expectation must ignore local filters. A monitor that goes red because someone narrowed the view is a monitor people learn to dismiss.
- Put mechanics next to mechanics. A check-level duration sitting beside page-level metrics will be read as a page-level number, and the person misreading it will be right to.
- Say on the panel what the measurement actually is. Emulated mobile is a layout measurement over a fast link, not a slow-phone measurement, and the gap between those two is where a dashboard starts producing confident wrong decisions.