Every profiling tool in this series so far watches your own request/response cycle — a query, an interaction, a paint event. This one doesn’t. It starts from a claim made entirely outside your infrastructure: Search Console’s Crawl Stats report, which is Google’s own measurement of how long its crawler waited on your server, aggregated in a dashboard you can’t instrument.
The claim, on a WordPress VIP-style multisite network of regional news sites: Site B’s average crawl response time was running at roughly double Site A’s. Two sibling sites, same platform, same shared infrastructure, same page-cache layer. If one is really twice as slow to Googlebot, that’s worth knowing.
TL;DR: Search Console claimed one site’s average crawl response time was double a sibling’s, but the access logs disagreed — until split by bot type: Google’s ads-quality crawler made up 4x the share of the slower site’s traffic, and a fat p90 tail (not the median) turned out to come from tracking-parameter URLs generating guaranteed cache misses. Fix: strip known tracking parameters from the cache key at the edge.
What Did the First Pass at the Logs Show?
The obvious move is to pull the platform’s access-log export — most managed WordPress hosts that operate at this scale expose one, often through a BigQuery-backed dataset — filter to user_agent LIKE '%googlebot%', and compare averages directly.
They didn’t match the premise. Across every slice tried over a two-week window, Site B was not consistently slower — if anything it trended slightly faster:
| Slice | Site A avg | Site B avg |
|---|---|---|
| All Googlebot requests, 2 days | 0.251s | 0.223s |
cache_status = MISS only (real origin work), 2 days |
0.702s | 0.934s |
status = 200 only, 7 days |
0.411s | 0.270s |
| Daily trend, 14 days | ~0.25–0.35s | ~0.21–0.32s |
The MISS-only row is the one exception, and it’s a real signal, just not yet the full story: Site B ran a lower overall cache-hit ratio (roughly 53% HIT+STALE against Site A’s 72%), which meant a larger share of its Googlebot traffic was hitting the origin at all — and origin hits, unsurprisingly, ran somewhat slower once isolated. That’s a real difference. It’s not a 2x difference, and it doesn’t match what Search Console was reporting.
At this point the honest move is to say so and ask where the "roughly double" number actually came from, rather than keep slicing the same dataset hoping a 2x gap appears. It didn’t come from the same place the logs did — it came from Search Console’s own Crawl Stats export, which measures something the access logs structurally can’t see: Googlebot’s own round-trip, including DNS and TLS negotiation, not just server-side processing time.
What Number Actually Explained the Gap?
Search Console’s Crawl Stats report breaks its sample down by crawler type — Smartphone, Desktop, "Page resource," and an AdsBot bucket for Google’s ad-quality crawler. That breakdown was the whole answer:
| Bot type | Site A share | Site B share |
|---|---|---|
| AdsBot | 12.4% | 42.0% |
| Smartphone | 50.1% | 31.2% |
| Desktop | 14.7% | 8.4% |
| Page resource | 11.6% | 4.6% |
Site B was getting crawled disproportionately by Google’s ads-quality crawler — a heavier, JS-capable fetch that’s inherently slower per request than a plain HTML fetch — and that crawler made up more than four times the share of Site B’s total crawl volume compared to Site A. An aggregate average with that much weight shifted toward one slow bucket will report as "generally slower" even when the ordinary crawl traffic underneath it is fine, which is exactly what the first-pass log query, blind to bot type, had missed.
Cross-referencing the raw logs by the specific user-agent string (Mediapartners-Google, the actual UA behind the AdsBot bucket) confirmed it directly: 2.516s average on Site B against 1.058s on Site A for that crawler specifically — a real, large, bot-specific gap the aggregate query had been diluting.
Why Follow the Tail Instead of the Average?
An average that large usually isn’t uniform, and it wasn’t. Splitting Mediapartners-Google MISS traffic by percentile showed the median was actually fine — 0.58s — while the 90th percentile ran to 16.7 seconds, against 2.2 seconds on Site A at the same percentile. A fat tail, not a general slowdown. Something specific was happening to a subset of requests.
The subset turned out to be requests carrying tracking parameters — fbclid, utm_source, utm_medium, and friends, left over from social and newsletter referral links. Mediapartners-Google re-fetches the exact URL a real visitor loaded, query string and all. A page cache keyed on the full URL treats every distinct parameter value as a distinct cache entry, which means every one of these requests is a guaranteed MISS — a full WordPress render, every time, for a URL variant nobody will ever request again.
| Share of MISS traffic with tracking params | Avg response, tracking param present | Avg response, clean URL | |
|---|---|---|---|
| Site A | 63% | 2.05s | 1.11s |
| Site B | 65% | 4.51s | 0.56s |
Both sites take roughly the same share of this traffic. Site B’s origin simply degrades further under it — individual requests spiked to 12–29 seconds, concentrated on a handful of viral regional stories getting re-fetched repeatedly with a different fbclid value each time, almost certainly a request pile-up against a smaller share of backend capacity for that particular site.
What’s the Fix (and Why Isn’t It a Code Change)?
Strip the known tracking parameters from the cache key at the CDN/edge layer, before the request reaches origin — so a request carrying ?fbclid=abc123 hits the same cache entry as the clean URL instead of manufacturing a fresh one. This is a configuration change at the caching layer, not application code, and it applies cluster-wide rather than per-site: Site A shows the identical pattern, just with enough backend headroom that it hasn’t produced a visible Search Console anomaly yet. Fixing it only where the symptom currently shows would leave the same defect live everywhere else, waiting for its own traffic spike.
What’s the Generalizable Lesson Here?
- Search Console’s Crawl Stats numbers and your own access logs are not measuring the same thing. One includes DNS/TLS/network round-trip from Google’s vantage point; the other only sees server-side processing. A gap between what they report isn’t necessarily a lie in either direction — check what each one is actually measuring before trying to reconcile them.
- An aggregate average across bot types can look "generally slower" purely because one crawler type — inherently heavier, unrelated to your code — makes up a larger share of the mix. Split by bot/UA before drawing a conclusion from a single averaged number.
- Once you have a real gap, check percentiles, not just the mean. A fat tail (fine median, terrible p90) points at a specific triggering condition; a uniform shift points at something structural.
- Any content-negotiation or cache-key logic that doesn’t account for tracking parameters will treat every referral-tagged URL as a fresh cache miss. If your cache key includes the full query string, decide deliberately which parameters actually belong in it.
- A fix found on one symptomatic instance of a shared config almost always applies to its siblings too, even where nothing has visibly broken yet. Ship it cluster-wide, not site-by-site.