--- title: "Taming a 3am Pager: SLOs and Error Budgets That Stuck" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/post/slo-error-budget-rollout-case-study --- ![Blog post image for Taming a 3am Pager: SLOs and Error Budgets That Stuck - How a team traded dozens of noisy, cause-based alerts for a handful of SLO burn-rate pages, and used error budgets to actually change how it decided what to ship.](/_astro/hero.an3xFV8m_2roe2M.webp) [Home](/)›[Case studies](/case-studies)›[All Categories](/case-studies/categories)›[Reliability](/case-studies/categories/reliability) Case studies [Prev in ReliabilityMulti-Region Active-Active for a Payments API](/case-studies/post/multi-region-active-active-payments) [Reliability](/case-studies/categories/reliability)[DevOps](/case-studies/categories/devops)[Management](/case-studies/categories/management) # Taming a 3am Pager: SLOs and Error Budgets That Stuck [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 29 Aug 2026Updated: 30 Aug 202607 Mins read09 Mins listen [Markdown for AI(opens in a new tab)](/post/slo-error-budget-rollout-case-study/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR How a team traded dozens of noisy, cause-based alerts for a handful of SLO burn-rate pages, and used error budgets to actually change how it decided what to ship. Series [Reliability & Resilience Engineering](/series/reliability--resilience-engineering)1/1 All posts in this series (1) Case Studies1 1. [Taming a 3am Pager: SLOs and Error Budgets That StuckYou are here](/case-studies/post/slo-error-budget-rollout-case-study) ### Taming a 3am Pager: SLOs and Error Budgets That Stuck Contents [Impact](#impact)[The problem](#the-problem)[Constraints](#constraints)[Architecture](#architecture)[Implementation](#implementation)[Results](#results)[Lessons](#lessons)[Frequently Asked Questions](#frequently-asked-questions)[References](#references) A platform team was losing people to burnout, and the cause was the pager. On-call meant a phone that went off day and night with alerts about CPU, memory, and pod restarts, the overwhelming majority of which resolved themselves before anyone finished reading them. The alerts described causes, not symptoms, and they fired whether or not a single customer was affected. This is the story of replacing that noise with SLO-based paging, and of the harder change that made it stick: an error budget policy the whole team actually agreed to follow. ## [Impact](#impact) ~0/wkpages before ~0/wkpages after 0%fewer 3am pages 0SLOs that mattered The headline number the team cared about was pages per week, which fell from roughly forty to around six over the first two months (measured from PagerDuty). More importantly, the pages that remained almost always corresponded to something a customer could feel. On-call stopped being a punishment, and the error budget gave planning meetings a shared, unemotional way to decide between shipping features and paying down reliability. ## [The problem](#the-problem) The team had good intentions and bad alerts. Over the years, every incident spawned a new alert rule (“we should have known the disk was filling up”), and none were ever retired. The result was a wall of cause-based alerts: CPU over 80%, memory over 90%, a pod that restarted, a queue that grew for five minutes. Each one paged. The trouble is that none of those conditions reliably means a customer is having a bad time. A pod restarting is often Kubernetes doing its job. CPU at 85% is fine if latency is fine. So on-call learned to glance at a page, see it was “the usual,” and go back to sleep, which is exactly the habit that lets a real incident slip through. The noise was worse than tiring. It trained people to ignore the pager. ## [Constraints](#constraints) The rollout had to work within some hard limits. The team ran on Kubernetes with Prometheus and Alertmanager already in place, so the solution had to build on that stack rather than introduce a new monitoring vendor. It could not require re-instrumenting every service at once, so it had to start from metrics the services already emitted (request counts, statuses, and latencies at the ingress). And it had to be a gradual change, because you cannot ask a team to trust a new paging model overnight; the old alerts had to run alongside the new ones until the new ones earned trust. There was also a human constraint. An error budget policy is worthless if leadership overrides it the first time a deadline is at stake, so the policy needed buy-in from engineering management before a single alert was rewritten. ## [Architecture](#architecture) The core idea is to alert on symptoms, not causes. Instead of paging when a resource crosses a threshold, page when the service is burning through its error budget fast enough to matter. Before, dozens of resource-threshold alerts paged regardless of customer impact. After, a single SLI (good requests over total) drives burn-rate alerts, so a page means users are actually being hurt. The plumbing sits entirely on the existing Prometheus stack. Recording rules compute the SLI (the ratio of good requests to total) so it is cheap to query. Multi-window burn-rate rules then decide severity, and Alertmanager routes a fast burn to the pager while a slow burn goes to Slack or a ticket. Grafana shows the SLO and remaining budget so anyone can see the state at a glance. Prometheus scrapes request metrics, recording rules define the SLI, and multi-window burn-rate rules feed Alertmanager, which pages only on a fast burn and files slow burns as tickets. Grafana renders the budget dashboard. ## [Implementation](#implementation) The first step was picking SLIs that reflect the customer experience. The team settled on two per service, availability (the fraction of requests that did not return a 5xx) and latency (the fraction served under a target like 300ms), and set a 99.9% SLO over a rolling 30-day window. That target implies an error budget of 0.1%, which is about 43 minutes of “bad” time per month. The SLI was captured as a Prometheus recording rule so the burn-rate queries stayed fast: slo-rules.yaml ``` 1groups:2 - name: slo-sli3 rules:4 - record: job:http_request_error_ratio:rate5m5 expr: |6 sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)7 /8 sum(rate(http_requests_total[5m])) by (job) ``` The paging decision uses the multi-window, multi-burn-rate approach from the Google SRE workbook: page only when a short window and a longer window both show a high burn rate, which catches fast, serious problems while ignoring brief blips. burn-rate-alerts.yaml ``` 1groups:2 - name: slo-burn-rate3 rules:4 # Fast burn: 2% of the monthly budget in 1 hour -> page.5 - alert: ErrorBudgetFastBurn6 expr: |7 job:http_request_error_ratio:rate5m > (14.4 * 0.001)8 and9 job:http_request_error_ratio:rate1h > (14.4 * 0.001)10 for: 2m11 labels: {severity: page}12 # Slow burn: 10% of the budget in 3 days -> ticket, not a 3am call.13 - alert: ErrorBudgetSlowBurn14 expr: |15 job:http_request_error_ratio:rate30m > (3 * 0.001)16 and17 job:http_request_error_ratio:rate6h > (3 * 0.001)18 for: 15m19 labels: {severity: ticket} ``` Alertmanager then routed by the `severity` label: `page` went to PagerDuty, `ticket` went to a Slack channel and opened a low-priority issue. The rollout itself was deliberately gradual, running old and new alerts in parallel until the SLO alerts had proven themselves. 1. Instrument or confirm request metrics at the ingress for each service. 2. Add the SLI recording rule and a Grafana SLO dashboard, and watch it for a week with no paging. 3. Turn on the slow-burn (ticket) alert and compare it against the old alerts for false positives. 4. Turn on the fast-burn (page) alert, still alongside the legacy alerts. 5. Once the SLO pages match real incidents for two weeks, delete the old cause-based alerts. The last piece was the part that actually changed behavior: a written error budget policy, agreed with management, that sprint planning consults every cycle. Each planning cycle the team checks the remaining budget on the dashboard and consults the policy: a healthy budget means ship features, a spent budget means reliability work takes priority until it recovers. ## [Results](#results) Two months in, paging volume had dropped from roughly forty pages a week to about six, and the pages that remained were almost all tied to genuine customer impact. The on-call rotation stopped being the shift everyone dreaded, and a couple of engineers who had been quietly heading for the door stayed. The subtler result was cultural. Because the error budget was visible and agreed, the recurring argument about whether to ship a risky feature before a deadline turned into a quick check: the budget was healthy, so the feature shipped, with everyone comfortable that there was room to absorb a mistake. The one time the budget was nearly spent, the team paused feature work for a few days to fix a flaky dependency, and nobody had to have a feelings-based fight about it. ## [Lessons](#lessons) The biggest lesson is that the technology was the easy 20%. Recording rules and burn-rate alerts are well-documented and took a couple of weeks. The hard 80% was social: getting agreement that a page must mean customer impact, and that the error budget policy would be honored even when a deadline loomed. Without that agreement, SLOs are just another dashboard. The second lesson is to roll out in parallel and delete old alerts only once the new ones have earned trust. Deleting the legacy alerts on day one would have felt reckless and invited a “see, we missed something” reaction at the first hiccup. Running both, then retiring the old ones with evidence, made the change uncontroversial. ## [Frequently Asked Questions](#frequently-asked-questions) A cause (high CPU, a pod restart) may or may not affect users, so cause-based alerts page constantly without correlating to real pain. A symptom (elevated errors or latency) is by definition something users feel. Paging on symptoms means every page is worth a human’s attention, which is what makes the pager trustworthy again. Burn rate is how fast you are spending your error budget relative to the SLO window. A burn rate of 1 spends the whole budget exactly over the window; higher means faster. Requiring both a short and a long window to show a high burn rate before paging filters out brief spikes (short window only) and avoids waiting too long on a real outage (long window only). It was chosen from what users actually needed and what the system already delivered, not pulled from the air. The team looked at historical availability, confirmed 99.9% was both achievable and good enough for the product, and left room to tighten it later. Chasing an extra nine (99.99%) would have cost far more engineering effort than the product needed. A slow burn means the budget is being spent gradually, which is a real issue but not an emergency, so it belongs in the next working day rather than at 3am. If a slow burn accelerates, the fast-burn rule catches it and pages. The two rules together cover both the creeping degradation and the sudden outage. The policy says risky feature releases pause and the team prioritizes reliability work until the budget recovers. The key is that this is agreed in advance and in writing, so it is a rule the team follows rather than a negotiation held under pressure during every incident. A few, but downgraded. Things like a disk approaching full are genuinely predictive and cheap to keep as tickets, so they stayed as low-priority notifications rather than pages. The point was not to delete all infrastructure signals, only to stop paging on ones that do not correlate with customer impact. ## [References](#references) - [Google SRE Workbook: Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/) - [Google SRE Book: Service Level Objectives](https://sre.google/sre-book/service-level-objectives/) - [Prometheus recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) - [Prometheus alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) - [Alertmanager routing](https://prometheus.io/docs/alerting/latest/configuration/) Was this useful? ## Tags [#SLO](/case-studies/tags/slo)[#Error Budget](/case-studies/tags/error-budget)[#SRE](/case-studies/tags/sre)[#Observability](/case-studies/tags/observability)[#Alerting](/case-studies/tags/alerting)[#On Call](/case-studies/tags/on-call)[#Prometheus](/case-studies/tags/prometheus) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fslo-error-budget-rollout-case-study "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=Taming%20a%203am%20Pager%3A%20SLOs%20and%20Error%20Budgets%20That%20Stuck&url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fslo-error-budget-rollout-case-study "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fslo-error-budget-rollout-case-study&title=Taming%20a%203am%20Pager%3A%20SLOs%20and%20Error%20Budgets%20That%20Stuck&summary=How%20a%20team%20traded%20dozens%20of%20noisy%2C%20cause-based%20alerts%20for%20a%20handful%20of%20SLO%20burn-rate%20pages%2C%20and%20used%20error%20budgets%20to%20actually%20change%20how%20it%20decided%20what%20to%20ship.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=Taming%20a%203am%20Pager%3A%20SLOs%20and%20Error%20Budgets%20That%20Stuck%20https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fslo-error-budget-rollout-case-study "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fslo-error-budget-rollout-case-study&text=Taming%20a%203am%20Pager%3A%20SLOs%20and%20Error%20Budgets%20That%20Stuck "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fslo-error-budget-rollout-case-study&title=Taming%20a%203am%20Pager%3A%20SLOs%20and%20Error%20Budgets%20That%20Stuck "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fslo-error-budget-rollout-case-study&t=Taming%20a%203am%20Pager%3A%20SLOs%20and%20Error%20Budgets%20That%20Stuck "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fslo-error-budget-rollout-case-study&media=&description=How%20a%20team%20traded%20dozens%20of%20noisy%2C%20cause-based%20alerts%20for%20a%20handful%20of%20SLO%20burn-rate%20pages%2C%20and%20used%20error%20budgets%20to%20actually%20change%20how%20it%20decided%20what%20to%20ship. "Share on Pinterest")[Email]() ## Comments ## You might also enjoy More posts on similar topics [![Migrating a Monolith to Kubernetes Without a Big-Bang Cutover](/_astro/hero.CAKh7bXG_Z1PnEBt.webp)](/case-studies/post/monolith-to-kubernetes-strangler-migration) ## [Migrating a Monolith to Kubernetes Without a Big-Bang Cutover](/case-studies/post/monolith-to-kubernetes-strangler-migration) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps](/case-studies/categories/devops) - [Cloud Native](/case-studies/categories/cloud-native) - [Architecture](/case-studies/categories/architecture) Almost every failed "let's move off the monolith" project shares one detail: the plan was a big-bang cutover. Rewrite in parallel, pick a weekend, flip the switch, and pray. This is the opposite of th [#Kubernetes](/case-studies/tags/kubernetes)[#EKS](/case-studies/tags/eks)[#Migration](/case-studies/tags/migration)+4 tags [read more](/case-studies/post/monolith-to-kubernetes-strangler-migration) [![QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog](/_astro/hero.BfjMKoMg_ZI2zvl.webp)](/case-studies/post/quenchworks-zero-cve-catalog) ## [QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog](/case-studies/post/quenchworks-zero-cve-catalog) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Security](/case-studies/categories/security) - [DevOps](/case-studies/categories/devops) - [Cloud Native](/case-studies/categories/cloud-native) When Bitnami moved its long-trusted catalog behind a paid tier, thousands of teams woke up to a supply-chain problem they didn't choose. The free images they had pinned in production would stop gettin [#Containers](/case-studies/tags/containers)[#Wolfi](/case-studies/tags/wolfi)[#Helm](/case-studies/tags/helm)+5 tags [read more](/case-studies/post/quenchworks-zero-cve-catalog) [![Multi-Region Active-Active for a Payments API](/_astro/hero.RPbRRCdE_flG0g.webp)](/case-studies/post/multi-region-active-active-payments) ## [Multi-Region Active-Active for a Payments API](/case-studies/post/multi-region-active-active-payments) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Architecture](/case-studies/categories/architecture) - [Cloud Computing](/case-studies/categories/cloud-computing) - [Reliability](/case-studies/categories/reliability) A payments API that moves real money had been running comfortably in a single AWS region for years. It was reliable until the day it was not: a regional control-plane incident took the whole service o [#Multi Region](/case-studies/tags/multi-region)[#Active Active](/case-studies/tags/active-active)[#Payments](/case-studies/tags/payments)+5 tags [read more](/case-studies/post/multi-region-active-active-payments) [![Building an Internal Developer Platform on Backstage and GitOps](/_astro/hero.Dq3xrist_1gdSPN.webp)](/case-studies/post/internal-developer-platform-backstage-gitops) ## [Building an Internal Developer Platform on Backstage and GitOps](/case-studies/post/internal-developer-platform-backstage-gitops) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps](/case-studies/categories/devops) - [Platform Engineering](/case-studies/categories/platform-engineering) - [Cloud Native](/case-studies/categories/cloud-native) Product teams were spending more time waiting on the platform team than building features. Spinning up a new service meant opening a ticket and waiting for someone to provision a repo, wire up CI, wri [#Backstage](/case-studies/tags/backstage)[#GitOps](/case-studies/tags/gitops)[#Argo CD](/case-studies/tags/argo-cd)+4 tags [read more](/case-studies/post/internal-developer-platform-backstage-gitops) [![Cutting a SaaS AWS Bill 41% Without Slowing Delivery](/_astro/hero.DJTB593d_Z1iASgs.webp)](/case-studies/post/aws-cost-optimization-saas-case-study) ## [Cutting a SaaS AWS Bill 41% Without Slowing Delivery](/case-studies/post/aws-cost-optimization-saas-case-study) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Cloud Computing](/case-studies/categories/cloud-computing) - [DevOps](/case-studies/categories/devops) - [Cloud Native](/case-studies/categories/cloud-native) A growing SaaS ran on EKS with a full GitOps pipeline, and it was over its AWS budget nearly every month. The reflex from leadership was the usual one: freeze features until the bill comes down. That [#AWS](/case-studies/tags/aws)[#EKS](/case-studies/tags/eks)[#Kubernetes](/case-studies/tags/kubernetes)+7 tags [read more](/case-studies/post/aws-cost-optimization-saas-case-study) [![Zero-Downtime PostgreSQL Major-Version Upgrade at Scale](/_astro/hero.C03RcOLI_141vsK.webp)](/case-studies/post/zero-downtime-postgres-upgrade) ## [Zero-Downtime PostgreSQL Major-Version Upgrade at Scale](/case-studies/post/zero-downtime-postgres-upgrade) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps](/case-studies/categories/devops) - [Databases](/case-studies/categories/databases) - [Cloud Computing](/case-studies/categories/cloud-computing) A multi-terabyte PostgreSQL 12 database was reaching end of life, and the business ran around the clock, so the usual answer of "schedule a maintenance window" was off the table. We upgraded it to Pos [#PostgreSQL](/case-studies/tags/postgresql)[#Logical Replication](/case-studies/tags/logical-replication)[#Zero Downtime](/case-studies/tags/zero-downtime)+3 tags [read more](/case-studies/post/zero-downtime-postgres-upgrade) 6 related posts