Benjamin Scarbrough

First-party monitoring existed, but the resource type had no support for third-party monitoring, and nothing covered the AFD/CDN layer sitting in front of 50+ services the way ours was set up. Not a gap in what had been configured, but a gap in what the platform made possible. Nobody was addressing it.

Assemble the monitoring plane out of Azure primitives rather than wait for a product: a Log Analytics workspace, and 36 Azure Monitor alert rules tied directly to the resources, covering latency, error conditions, and absence of traffic. That last one matters most. A CDN's worst failure is silence, and anything alerting only on the errors it receives never sees it.

It has already caught real drops that nothing else would have seen. Modest by design: the point was never volume of alerting, it was that a whole layer of the estate went from unobservable to observable.

Conditions cleared on their own, and every ticket still had to be adjusted by hand. Auto-mitigation sat at 0% across the portfolio, not because nothing recovered, but because the incident monitoring service had no authorized way to say so.

Auto-mitigation rules on every service monitor, switched on rather than left staged, and the certificate chain that let the incident monitoring service authenticate to the incident service and mitigate a ticket once the underlying condition was healthy again. The rules were the easy half; without that trust chain they were inert.

A peak of ~95% over 9 months, with auto-resolution rising into a 39-65% band. The band is wide on purpose: full auto-resolution only ever went to failure modes with a known-safe recovery, and the rest still stop at a human. A machine that resolves an incident it does not understand is not reliability, it is a slower outage.

Audit cycles usually start reactive. The request lands, and then a scramble begins to reconstruct evidence nobody was maintaining, because getting ahead of it is not anybody in particular's job.

Know what the cycle will ask for and have it standing before it is asked, unprompted. It was inside my scope and simply was not being addressed. Writing the process down while doing it mattered as much as the preparation. It is the team's documentation now, so the next cycle starts from a procedure rather than from one person's memory.

Full collection came down to ~2 days across multiple annual cycles. Audit readiness is what makes a service sellable into compliance-restricted public sector markets, so the difference between weeks and days is the difference between a market being open or closed.