# Mohammad Abu Mattar, Full Content Bundle
> Full text of the professional core pages and all case studies, then a link index to every content section. For the exhaustive list of Markdown twins, see /llms-sitemap.txt.
---
# Mohammad Abu Mattar
---
# Cutting a SaaS AWS Bill 41% Without Slowing Delivery
A growing SaaS ran on EKS with a full GitOps pipeline, and it was over its AWS budget nearly every month. The reflex from leadership was the usual one: freeze features until the bill comes down. That would have worked, and it would have been the wrong call. Freezing delivery to save money trades a problem you can measure for one you can't. This is how the bill came down by roughly 41% over two quarters while the team kept shipping through a 25% canary rollout on every release. Almost none of the win came from turning things off in a panic.
The percentages here are representative of what this pattern achieves, not a
single audited client figure. The AWS and Kubernetes mechanics (tagging, Cost
Categories, Budgets, Anomaly Detection, Savings Plans, EKS node groups,
Karpenter, Argo CD, Argo Rollouts) are exactly as described. Your real savings
depend on how much waste you start with and how much of your compute is
commitment-eligible.
## Impact
The bill came down without a feature freeze. Over two quarters the monthly AWS spend dropped by roughly 41% while the team kept shipping through the same 25% canary it uses for any feature.
None of it came from a panic switch-off. Every lever was reversible and canary-guarded, so the savings held without trading away reliability or delivery speed, and cost turned into a normal signal that shows up in pull requests instead of a quarterly fire drill.
## The problem
The bill was growing faster than revenue, which is the signal that matters. Nobody could say where the money went, because nothing was labeled. A single line item for "EC2" across a dozen teams and six node groups tells you nothing you can act on. When finance asked engineering to explain a spike, the honest answer was "we're not sure," and that answer is what turns a cost conversation into a feature freeze.
There was also a dashboard, and everyone pointed at it as proof they were "doing FinOps." A dashboard shows you the number. It does not change anyone's behavior, and it definitely does not tell an engineer that the node group they oversized last sprint is the reason the graph bent upward. Visibility without attribution is just a prettier version of not knowing.
The last piece was fear. Every proposed saving came with "will this break production?" and without data nobody could answer, so nothing happened. The goal was to make cost a normal, reversible engineering decision instead of a quarterly emergency, and to do it without touching the delivery pipeline the team depended on.
## Constraints
A few limits shaped the whole approach.
- **No feature freeze.** Delivery velocity was the business. Any optimization that slowed shipping was off the table.
- **No reliability regressions.** Saving money by removing redundancy or headroom was not a real saving.
- **Keep the delivery model.** Full GitOps, dev to staging to production promotion, and 25% canary rollouts all had to stay exactly as they were.
- **Data stays isolated.** The data tier runs in subnets with no internet route, and that boundary was non-negotiable.
- **Respect confidentiality.** Real dollar figures stay private, so success is reported as percentages and unit economics.
## Architecture
The SaaS runs entirely on a single EKS cluster per environment, across three availability zones. Each VPC has three tiers of subnets: three public subnets for ingress and NAT, three private subnets for the EKS worker nodes, and three isolated subnets with no internet route for the data tier. Everything the product needs runs in the cluster.
_EKS platform architecture_
Compute is split into purpose-built managed node groups rather than one big pool, which is what makes both scheduling and cost control tractable. There are separate node groups for the frontend, the backend, data and ETL work, observability, the GitOps controllers, and the internal dashboards, plus a Spot-backed group for batch and preview workloads. Karpenter handles just-in-time node provisioning on top, so capacity follows demand instead of sitting idle.
The whole platform toolchain lives in the cluster, scheduled onto those node groups: Keycloak for single sign-on across the dashboards, Argo CD, and Grafana; SonarQube and Trivy in the delivery path; Argo CD for GitOps and Argo Rollouts for progressive delivery; and Prometheus with Grafana for observability.
The data tier (RDS with a multi-AZ standby, plus ElastiCache) lives in the
isolated subnets. Those subnets have no NAT and no internet gateway route, so
the databases cannot reach the internet and the internet cannot reach them.
Nodes talk to them only over private VPC routes.
Cost work fails when it lives only in a finance spreadsheet, so the first move was to make spend attributable. A small, enforced tag taxonomy flows into AWS Cost Categories, which maps raw line items to teams and products, and from there into Budgets, Cost Anomaly Detection, and per-team showback.
_Cost attribution and guardrail flow_
The taxonomy was deliberately small. Five mandatory tags, not twenty, because a taxonomy nobody follows is worse than none. `Environment`, `CostCenter`, `Application`, and `Owner` covered almost every question we needed to answer, and `ManagedBy` flagged anything created by hand instead of through code. On EKS those tags also propagate to node groups and volumes, so cluster compute is attributable per team, not lumped under one anonymous bill.
Enforcement matters more than intent, so the tags were governed centrally. AWS Organizations Tag Policies defined the allowed keys and values, Service Control Policies blocked non-compliant resources, and consolidated billing plus the Cost and Usage Report gave one clean view across every account.
_Governance and reporting topology_
Chargeback is tempting, but it needs near-perfect tagging and it starts turf
wars early. We started with showback: show each team its own spend, let
central finance keep paying the bill, and move to chargeback only once the
tags were trustworthy. Awareness drove most of the savings before any money
changed hands internally.
## Delivery: GitOps and progressive rollout
None of the cost work was allowed to disturb delivery, so it helps to see what delivery looks like. Every change runs through the same GitOps pipeline. CI builds and tests, SonarQube enforces a quality gate, and Trivy scans the image and the IaC. Only a clean build pushes to ECR and bumps the image digest in the GitOps manifests repository. Argo CD notices the change and syncs it to the cluster.
_GitOps delivery pipeline_
Nothing goes fully live at once. Argo Rollouts takes over at the cluster and shifts traffic in steps, starting at 25%, then pausing to check analysis metrics before it widens. If the metrics stay healthy it promotes; if they degrade it rolls back on its own, with no human in the loop. That single behavior is what let the cost changes ship safely, because a right-sized deployment that misbehaved would be caught at 25% of traffic, not 100%.
Environments follow the same path every time. A feature or preview environment spins up per pull request on the Spot-backed node group, merges auto-deploy to dev, the same image digest promotes to staging for integration tests, and only then does it reach production behind the canary.
_Environments and promotion_
The canary itself is a few lines of Argo Rollouts config, and it is the same for a feature change or a cost change.
```yaml title="rollout.yaml"
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: frontend
spec:
strategy:
canary:
steps:
- setWeight: 25
- pause: {duration: 10m}
- setWeight: 50
- pause: {duration: 10m}
- setWeight: 100
```
## Implementation
The baseline was code. Rather than tag resources by hand, every provider inherited a default set of tags, so new infrastructure was attributable from the moment it existed.
```hcl title="provider.tf"
provider "aws" {
region = "us-east-1"
default_tags {
tags = {
Environment = "Prod"
CostCenter = "1001"
ManagedBy = "Terraform"
}
}
}
```
Then came the guardrails, automated so nobody had to remember to check a dashboard. A monthly budget with a forecast alert catches planned overspend before the month ends, and Cost Anomaly Detection catches the surprise 3am spike.
```bash title="Budget + anomaly guardrails"
aws budgets create-budget \
--account-id 111122223333 \
--budget '{
"BudgetName": "MonthlyCost",
"BudgetLimit": { "Amount": "50000", "Unit": "USD" },
"TimeUnit": "MONTHLY",
"BudgetType": "COST"
}'
aws ce create-anomaly-monitor \
--anomaly-monitor '{
"MonitorName": "CoreServices",
"MonitorType": "DIMENSIONAL",
"MonitorDimension": "SERVICE"
}'
```
With attribution and guardrails in place, the actual optimization ran as normal GitOps changes, each behind the canary, each with a one-commit rollback.
1. **Find the waste.** Use Cost Explorer grouped by the new tags, plus Kubernetes right-sizing signals from the metrics stack, to rank the most over-provisioned node groups and workloads.
2. **Right-size node groups in reversible steps.** Drop one instance size or one replica at a time, ship it through the 25% canary, and watch the SLOs. The old manifest is one revert away.
3. **Let Karpenter consolidate.** Enable consolidation so underused nodes are drained and replaced with fewer, better-packed ones, and move interruptible and preview work to the Spot node group.
4. **Put dev and staging to sleep.** Scale non-production node groups to zero overnight and on weekends. Nothing runs when nobody is working.
5. **Commit last, not first.** Only after cluster usage was stable did we buy Compute Savings Plans, so we committed to real baseline usage rather than to waste.
The commitment step is where teams most often lose money, by chasing the deepest discount for a workload they are about to change. The rule we used was simple: match the commitment to the roadmap, not to the current instance.
_Choosing the right commitment_
The team was midway through moving several backend services to Graviton for better price-performance. A three-year EC2 Instance Savings Plan on the old family would have looked cheaper on paper and then stranded the moment those services migrated. A Compute Savings Plan gave up a few points of discount but stayed flexible across families, regions, Fargate, and Lambda, which matters even more on EKS where node groups change shape often. That small premium was cheap insurance.
## Results
Over two quarters, the monthly bill came down by roughly 41%, and delivery never paused. Every cost change went out through the same 25% canary as any feature. The breakdown, as representative shares of the total reduction, looked like this.
| Lever | Share of the saving | Nature of the change |
| :------------------------------------------------- | :------------------ | :------------------------- |
| Right-sizing node groups + Karpenter consolidation | Largest | Reversible, canary-guarded |
| Scheduling dev and staging to sleep | Large | Fully reversible |
| Compute Savings Plans matched to the roadmap | Meaningful | 1-year, flexible |
| Spot for batch, preview, and CI | Meaningful | Interruption-tolerant only |
| Storage cleanup and lifecycle policies | Smaller | One-time plus ongoing |
The more durable result was cultural. Cost stopped being a quarterly fire drill. Teams could see their own node-group spend, cost showed up in pull requests as a normal signal, and the "will this break?" fear faded because every change had a canary and a rollback. Treat the percentage as illustrative and the mechanics as the real deliverable.
## Lessons
Attribution is the whole game. Nothing else worked until spend had an owner, because you cannot optimize a shared cluster you cannot see per team. The five-tag taxonomy, enforced in code and propagated to node groups, paid for itself before a single node was resized.
Progressive delivery is what makes cost work safe. On a normal deploy model, right-sizing production feels risky enough that teams avoid it. With a 25% canary and automated rollback, a bad resize is a non-event, so the team actually did the work instead of flinching.
Commit to usage, not to hope. The most expensive mistake in cloud cost work is a long, rigid commitment bought early to chase a headline discount. Buy commitments after usage is stable, prefer flexibility while the architecture is still moving, and treat the discount rate as secondary to not stranding the plan.
If I did it again, I would wire a pull-request cost estimate in on day one. Putting the number in front of the engineer at the moment they change a manifest moved behavior more than any dashboard did.
## Frequently Asked Questions
> **Why not just freeze features until the bill comes down?**
A freeze trades a measurable problem for an unmeasurable one. You save some money and lose delivery velocity, customer momentum, and team morale, none of which show up cleanly on the bill. Almost all of the saving here came from waste and mismatched commitments, not from doing less, so the freeze would have hurt the business while barely touching the real cost drivers.
> **How do you right-size EKS node groups without causing incidents?**
Treat it like any other change. Drop one size or one replica at a time, ship it through the same 25% Argo Rollouts canary as a feature, and watch the SLOs during the pause windows. Let Karpenter consolidate underused nodes rather than doing it by hand. Because every step is a GitOps commit, the rollback is a one-line revert, so a bad resize is caught at 25% of traffic and reverted, not discovered in a postmortem.
> **Why start the canary at 25% instead of a smaller slice?**
Twenty-five percent is a deliberate balance. It is a big enough slice that real traffic patterns and enough metric volume show up quickly, so the analysis step can make an honest call, but small enough that a bad release only touches a quarter of users before it rolls back. Smaller first steps are reasonable for very high-risk changes, but 25% gave this team fast, trustworthy signal without much blast radius.
> **How does the isolated data tier stay reachable if it has no internet?**
The isolated subnets have no NAT and no internet gateway route, so the databases cannot reach the internet and vice versa. The application nodes in the private subnets reach RDS and ElastiCache over private VPC routes only. Anything the data tier genuinely needs from an AWS service goes through VPC endpoints, which keep that traffic on the AWS network rather than the public internet.
> **Does the GitOps and canary setup make cost work harder?**
It makes it safer, which in practice makes it happen. Every cost change is a normal pull request that flows dev to staging to production behind the canary, with SonarQube and Trivy gates on the way. There is no separate risky "cost project," just ordinary changes with the same guardrails as everything else, so teams approve them quickly.
> **What is the single highest-impact first step?**
Enforced tagging. Until spend is attributable per team, product, environment, and node group, every other optimization is guesswork. A small mandatory taxonomy, applied through Terraform default tags and AWS Organizations tag policies, turns the bill from one opaque number into a map you can act on.
## References
- [Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html)
- [Karpenter](https://karpenter.sh/)
- [Argo CD](https://argo-cd.readthedocs.io/)
- [Argo Rollouts (progressive delivery)](https://argo-rollouts.readthedocs.io/)
- [Argo CD image rollouts walkthrough](https://medium.com/@anandctx/argocd-image-rollouts-a9d91943195d)
- [Keycloak](https://www.keycloak.org/documentation)
- [SonarQube](https://docs.sonarsource.com/sonarqube-server/latest/)
- [Trivy](https://trivy.dev/)
- [AWS Cost Categories](https://docs.aws.amazon.com/cost-management/latest/userguide/manage-cost-categories.html)
- [AWS Budgets](https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html)
- [AWS Cost Anomaly Detection](https://docs.aws.amazon.com/cost-management/latest/userguide/manage-ad.html)
- [AWS Savings Plans](https://docs.aws.amazon.com/savingsplans/latest/userguide/)
- [Terraform AWS provider default_tags](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#default_tags)
---
# Building an Internal Developer Platform on Backstage and GitOps
Product teams were spending more time waiting on the platform team than building features. Spinning up a new service meant opening a ticket and waiting for someone to provision a repo, wire up CI, write Kubernetes manifests, and hook up deployment. Each of those handoffs added days. We built an internal developer platform that turned that whole sequence into a self-service golden path: Backstage for the portal and software templates, Git as the single source of truth, and Argo CD reconciling desired state into the clusters. A developer picks a template, fills a short form, and gets a working repo plus a running service, with policy and RBAC acting as guardrails rather than manual gates.
## Impact
The headline change was that creating and shipping a service stopped being a ticket and became a form. New services that used to take teams the better part of a sprint to stand up now scaffold in minutes, and the first deploy happens on merge without anyone from the platform team touching it. The platform team moved from doing one-off deploys to maintaining the templates that everyone else uses.
Adoption is the metric that actually matters here, because a platform nobody uses is just more software to run. Within the first quarter most new services were created through the golden path rather than by hand, which is the signal that the paved road was genuinely easier than going around it. These numbers are specific to this rollout and were measured on our own usage, so treat them as a shape to expect rather than a guarantee.
## The problem
Every new service started the same way: a ticket. The platform team owned the repo templates, the CI config, the base Kubernetes manifests, and the deploy pipeline, so nothing shipped without them in the loop. That made sense when there were a handful of services, but it stopped scaling. The queue grew, context-switching killed the platform team's own roadmap, and product teams learned to batch requests, which made each one bigger and slower.
The deeper issue was that knowledge lived in people's heads and in copy-pasted YAML. Two teams standing up similar services would end up with subtly different setups, because each one copied whatever the last project happened to do. There was no paved road, just a lot of dirt tracks that mostly worked. When something went wrong in one of those setups, debugging it meant reverse-engineering choices nobody remembered making.
We wanted product teams to move without asking permission for routine work, while the platform team kept ownership of what "correct" looks like. That is the tension an internal developer platform exists to resolve.
## Constraints
The platform had to satisfy a few hard constraints, and every design decision came back to them.
- **Self-service by default.** The common case, creating and deploying a service, had to happen with zero tickets and no human in the platform team's loop.
- **Git as the source of truth.** Every change to what runs in a cluster had to be a commit, so we get review, history, and a trivial rollback for free. No `kubectl apply` from laptops.
- **Guardrails, not gates.** Policy and RBAC had to be enforced automatically. A human manually approving routine deploys would just recreate the ticket queue we were killing.
- **Paved road, not a walled garden.** Teams with genuinely unusual needs had to be able to step off the golden path without the platform blocking them, as long as they still passed policy.
## Architecture
The platform is three moving parts wired together by Git. Backstage is the front door, where developers discover services and kick off golden paths. Git holds both application code and the deployment config that describes desired state. Argo CD watches Git and reconciles that desired state into the Kubernetes clusters. Backstage never talks to the clusters to make changes; it only ever writes to Git, which keeps the whole system auditable.
_Platform control plane_
The Backstage catalog models the world as a small set of entities, and understanding those makes the rest of the platform click. A `Template` describes a golden path: its input parameters as a JSON schema, and the steps it runs to scaffold a service. Each template produces a `Component`, which is an actual service owned by a `Group` (a team). Argo CD then manages an `Application` resource that points at the component's config in Git and syncs it to a cluster.
_Catalog and deployment model (class view)_
The reason Git sits in the middle of everything is that it turns two hard problems, auditability and rollback, into one solved problem: version control. Every deploy is a diff you can read, and undoing a bad change is `git revert`, which Argo CD then reconciles back automatically.
## Implementation
The heart of the platform is the scaffolding flow. When a developer picks a template and submits the form, Backstage's scaffolder renders a skeleton from the template's inputs, creates a repository, opens a pull request, and registers the new component in the catalog. That is the moment the ticket used to be filed; now it is a button.
_Self-service scaffolding to deploy (sequence)_
A software template is a `Template` entity plus a skeleton directory. The parameters block is a JSON schema, so Backstage renders it as a validated form for free. The steps block is what runs when the form is submitted.
```yaml title="template.yaml (Backstage software template)"
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
name: node-service
title: Node.js service (golden path)
description: A production-ready Node.js service with CI, Helm, and GitOps wired up.
spec:
owner: group:platform
type: service
parameters:
- title: Service details
required: [name, owner]
properties:
name:
title: Name
type: string
pattern: '^[a-z][a-z0-9-]{2,30}$'
owner:
title: Owning team
type: string
ui:field: OwnerPicker
steps:
- id: fetch
name: Fetch skeleton
action: fetch:template
input:
url: ./skeleton
values:
name: ${{ parameters.name }}
owner: ${{ parameters.owner }}
- id: publish
name: Create repository
action: publish:github
input:
repoUrl: github.com?owner=acme&repo=${{ parameters.name }}
defaultBranch: main
- id: register
name: Register in catalog
action: catalog:register
input:
repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
catalogInfoPath: /catalog-info.yaml
```
The skeleton ships the boring, correct defaults so no team has to reinvent them: a `catalog-info.yaml` so the service shows up in the catalog, a CI workflow that builds and signs the container image, a Helm chart, and the Argo CD `Application` that ties it to a cluster. That last file is what turns a repo into something GitOps actually deploys.
```yaml title="argocd-application.yaml (scaffolded into the config repo)"
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: node-service-dev
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/acme/config
path: apps/node-service/dev
targetRevision: main
destination:
server: https://kubernetes.default.svc
namespace: node-service
syncPolicy:
automated:
prune: true
selfHeal: true
```
Because the deploy config lives in Git and Argo CD self-heals, the platform naturally behaves like a state machine. A service moves from scaffolded, to built, to deployed in dev, through a policy gate, and on to production, and every transition is a commit. Modeling it that way made it obvious where the guardrails belong.
_Service lifecycle (state machine)_
Guardrails are enforced at two layers. RBAC in Backstage and in the clusters decides who can do what, and admission policy with OPA or Kyverno decides what is allowed to run at all. A policy that every workload must set resource limits, for example, is a Kyverno rule that rejects the deploy at admission rather than a checklist item in a review.
```yaml title="require-resource-limits.yaml (Kyverno policy)"
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-resource-limits
spec:
validationFailureAction: Enforce
rules:
- name: check-limits
match:
any:
- resources:
kinds: ['Pod']
validate:
message: 'CPU and memory limits are required.'
pattern:
spec:
containers:
- resources:
limits:
memory: '?*'
cpu: '?*'
```
The repo layout keeps application code and deployment config separate, which is a deliberate GitOps choice: app repos change on every feature, config repos change on every deploy, and keeping them apart makes the deploy history readable.
- config/
- apps/
- node-service/
- dev/
- kustomization.yaml
- deployment.yaml
- prod/
- kustomization.yaml
- deployment.yaml
- argocd/
- node-service-dev.yaml
- node-service-prod.yaml
Onboarding an existing service that predates the platform is a short runbook rather than a rebuild.
1. Add a `catalog-info.yaml` to the repo so Backstage discovers and indexes the service.
2. Move its deployment manifests into the config repo under a per-environment path.
3. Add an Argo CD `Application` pointing at that path, starting with automated sync disabled.
4. Compare the live cluster state against Git until the diff is clean, then enable automated sync and self-heal.
## Results
The change people felt first was speed. Creating a new service went from a multi-day, ticket-driven sequence to a form that produces a working repo and a service running in the dev cluster off a single merge. The first deploy happens with no platform-team involvement, which is the whole point. Read and write access to what runs is governed by RBAC and policy, so faster did not mean looser.
Adoption is the result that tells you the platform actually worked, and it climbed quickly once the golden path was clearly easier than the old dirt tracks. Within the first quarter the large majority of new services came through templates rather than by hand. The platform team's own time shifted from doing one-off deploys to improving templates and policy, which compounds: one improvement to a template lands in every service scaffolded after it. These numbers are specific to this rollout and were measured on our own usage; treat them as a shape to expect, not a guarantee.
The other measurable win was consistency. Because every service starts from the same skeleton, the drift between projects that used to make debugging miserable mostly disappeared. When we needed to roll out a change like a new required label or a security default, we updated the template and the policy, and the fleet converged instead of needing a hand-edit per repo.
## Lessons
The most important lesson is that a platform lives or dies by adoption, and adoption is earned by making the paved road genuinely faster than going around it. We resisted the urge to mandate the platform early. Instead we made the golden path the path of least resistance, and teams chose it. A mandate on a platform people dislike just produces malicious compliance.
Guardrails have to be automatic to matter. The first time we let a "quick manual approval" creep into a deploy path, we had reinvented the ticket queue in miniature. Encoding the rule as admission policy, so the platform enforces it without a human, is what kept self-service actually self-service.
Finally, treat the golden path as a product with a small number of well-maintained templates, not a template for every conceivable variation. A handful of paths that cover the common cases well beats a sprawling catalog nobody trusts. Teams with unusual needs step off the road and still pass policy, and that is fine. The goal was never to control every service, only to make the right thing the easy thing.
## Frequently Asked Questions
> **What exactly is a golden path?**
A golden path is the supported, opinionated way to do a common task, like creating a new service, with the boring correct defaults already wired in. In this platform it is a Backstage software template that scaffolds a repo, CI, a Helm chart, and the GitOps config in one step. It is a paved road you are free to leave, not a wall you cannot cross.
> **Why put Git in the middle instead of deploying straight from Backstage?**
Because Git turns auditability and rollback into a solved problem. Every deploy is a reviewable diff with history, and undoing a bad change is a revert that Argo CD reconciles automatically. If Backstage pushed changes directly to clusters, you would lose that trail and have to build approval and rollback yourself.
> **How do guardrails avoid becoming the ticket queue you replaced?**
They are enforced by machines, not people. RBAC decides who can act, and admission policy with OPA or Kyverno decides what is allowed to run, both automatically at deploy time. There is no human in the routine path clicking approve, which is exactly the bottleneck a manual gate would recreate.
> **What happens to teams with genuinely unusual requirements?**
They step off the golden path. The platform does not block a team from writing their own manifests or CI, as long as the result still passes policy at admission. The golden path is the default that covers the common cases well, not a hard requirement for every service.
> **How do you onboard services that existed before the platform?**
Add a catalog-info.yaml so Backstage indexes the service, move its manifests into the config repo, and add an Argo CD Application with automated sync off at first. Once the live state matches Git with a clean diff, turn on automated sync and self-heal. It is an adoption runbook, not a rebuild.
> **What is the single most useful metric for this kind of platform?**
Adoption, specifically the share of new services created through the golden path rather than by hand. A platform nobody uses is just more software to operate. If teams choose the paved road on their own, it means the road is genuinely faster and safer than the alternative, which is the whole point.
> **Do you need Kubernetes to build an internal developer platform?**
No. Backstage and GitOps patterns apply to plenty of deployment targets. Kubernetes happens to pair well with Argo CD's reconcile loop and with admission policy, which is why this platform uses it, but the core idea of self-service scaffolding into Git as the source of truth is portable.
## References
- [Backstage: Software Templates](https://backstage.io/docs/features/software-templates/)
- [Backstage: Software Catalog](https://backstage.io/docs/features/software-catalog/)
- [Argo CD: Declarative GitOps CD for Kubernetes](https://argo-cd.readthedocs.io/en/stable/)
- [Argo CD: Application resource specification](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/)
- [Kyverno: Kubernetes-native policy management](https://kyverno.io/docs/)
- [Open Policy Agent (OPA)](https://www.openpolicyagent.org/docs/latest/)
- [Team Topologies: platform as a product](https://teamtopologies.com/key-concepts)
---
# Migrating a Monolith to Kubernetes Without a Big-Bang Cutover
Almost every failed "let's move off the monolith" project shares one detail: the plan was a big-bang cutover. Rewrite in parallel, pick a weekend, flip the switch, and pray. This is the opposite of that. A large application moved onto EKS one service at a time using the strangler-fig pattern, with a routing facade in front, traffic shifting gradually per route, and a working rollback at every single step. The team kept shipping features throughout, and at no point was the whole application in the air.
## Impact
The application reached Kubernetes with no big-bang moment. Every route moved gradually behind a facade with a working rollback, so no single step was ever high stakes and delivery never froze.
The gains were structural rather than a one-time event. Each extracted service got independent deploys and its own scaling, so teams stopped blocking each other and the hot paths no longer forced the whole application to scale with them.
## The problem
The monolith itself was not the enemy. It ran fine, the team knew it, and it paid the bills. The problem was that it had become the bottleneck for everything else. Deploys were all-or-nothing, so one risky change held up every other team's work. Scaling meant scaling the entire application even when only one part was hot. And onboarding a new engineer meant handing them the whole thing at once.
The tempting fix, a full rewrite with a cutover, is where teams get hurt. You freeze features to build the replacement, the replacement drifts from the original as the original keeps changing, and the cutover becomes a single high-stakes event with no safe rollback. If anything goes wrong at 2am on migration night, the only option is a panicked revert of everything.
The goal was to get the benefits of independent services without ever betting the business on one cutover. That means the old and new systems have to run side by side, in production, for as long as it takes.
## Constraints
- **No big-bang cutover.** At no point could correctness depend on a single switch-flip.
- **No feature freeze.** The monolith kept shipping features throughout the migration.
- **A rollback at every step.** Each increment had to be revertible in minutes, not hours.
- **No shared-database free-for-all.** Extracted services own their data; the goal was decoupling, not a distributed monolith on one schema.
- **Prove parity before deleting anything.** Old code stayed until the new path was verified against it.
## Architecture
Before the migration, the shape was familiar: an Application Load Balancer in front of a monolith running across an Auto Scaling group, all talking to one shared relational database.
_Before: monolith on EC2_
The target keeps the monolith running, containerized, inside an EKS cluster, and puts a routing facade in front of everything. The facade is the heart of the pattern. It looks at each request and decides whether that path has been migrated to a new service or still belongs to the monolith. Extracted services get their own data stores; the monolith keeps its shared database until its remaining parts are small.
_After: strangler facade on EKS_
The name comes from the strangler fig, a plant that grows around a tree and gradually replaces it. The new system grows around the monolith, taking over one responsibility at a time, until the original is either gone or small enough to leave alone. Nothing about it requires a dramatic finish.
## The routing facade
The facade is where the safety comes from. Every request enters through it, and a route table decides the destination. A path that has been migrated goes to the new service; everything else defaults to the monolith. Migration of a single route is itself gradual, too: you shift a small percentage of that route's traffic to the new service, watch it, and widen only when it holds. If the new service misbehaves, the facade falls straight back to the monolith, which is still running and still correct.
_Strangler routing_
In practice the facade can be an ingress with weighted routing, an API gateway, or a service mesh. The mechanism matters less than the property: per-path routing plus per-path traffic weight plus instant fallback.
```yaml title="facade-route.yaml (illustrative weighted routing)"
# /users is being migrated: 10% to the new service, 90% still to the monolith.
http:
- match:
- uri:
prefix: /users
route:
- destination: {host: users-service}
weight: 10
- destination: {host: monolith}
weight: 90
- route: # default: everything else stays on the monolith
- destination: {host: monolith}
weight: 100
```
## Implementation
The migration ran as a loop, not a project plan with an end date. Each pass picked one seam, extracted it, shifted traffic, verified, and cleaned up.
_Extraction sequence_
1. **Containerize the monolith first.** Before extracting anything, get the monolith itself running in EKS behind the facade. Now old and new live in the same place, and the facade is the only thing in front.
2. **Pick a loosely-coupled seam.** Choose a capability with a clear boundary and a data set it mostly owns, for example users or billing. Avoid the tangled core on the first pass; early wins build trust.
3. **Build the service with its own data.** Give the extracted service its own database rather than pointing it at the monolith's schema. Backfill and keep it in sync during the transition, but the target is independent ownership.
4. **Route to it gradually.** Add the path to the facade and shift a small slice of traffic, then widen. Watch latency and error rates during each step, and keep the monolith path warm as a fallback.
5. **Verify parity, then delete.** Once the new service matches the monolith's behavior under real traffic, remove that code from the monolith. Deleting the old path is what makes the win permanent.
6. **Repeat, and know when to stop.** Move to the next seam. Stop when what remains is small and stable enough that extracting it would cost more than it returns.
The most dangerous shortcut is pointing a new service at the monolith's
database so you can "extract later." That gives you two services coupled
through one schema, which is a distributed monolith: all of the network
overhead, none of the independence. Give the service its own data, even if
that means a sync period during the transition.
Data is the genuinely hard part, and it is worth being honest about that. Moving stateless request handling is straightforward; moving the data it owns without downtime is not. The workable approach is to give the new service its own store, backfill it, keep it in sync while both paths run, and cut the monolith's write path over only once the new service is authoritative and verified. Where strict consistency is required during the overlap, treat the monolith as the source of truth until the very last step.
Measure the migration by how much of the monolith is gone, not by how many
services exist. A useful signal is the share of production traffic served by
extracted services and the amount of code deleted from the monolith. Creating
services without deleting code from the original is motion without progress.
## Results
The application moved onto EKS without a single cutover event and without a feature freeze. Because each route shifted gradually with a live fallback, no migration step was a high-stakes moment; the riskiest change only ever touched a small slice of one path at a time. Independent deploys arrived for each extracted service, so teams stopped blocking each other, and the hot paths could scale on their own instead of forcing the whole application to scale with them.
The migration also did not finish in the storybook sense, and that was the right outcome. A stable, low-change remainder of the monolith stayed in place, containerized and behind the facade, because extracting it would have cost more than it returned. Treat "the monolith is gone" as a possible ending, not the goal.
## Lessons
The facade is the whole safety story. Because every request always had a valid destination and an instant fallback, no step was irreversible. That single property is what let the team move quickly instead of cautiously.
Extract the easy seams first. The instinct to start with the messy core is a trap. Early, low-risk extractions build the tooling and the team's confidence, so the hard ones later are routine instead of terrifying.
Data ownership is the real migration. The service boundary is easy; the data boundary is the work. Any plan that hand-waves the database is a plan to build a distributed monolith.
Give yourself permission to stop. The goal was never zero monolith. It was independent, deployable, scalable services for the parts that needed it, and a small stable remainder for the parts that did not.
## Frequently Asked Questions
> **What exactly is the strangler-fig pattern?**
It is an incremental migration approach where a new system grows around an old one and takes over its responsibilities one at a time, until the old system is replaced or reduced to a small remainder. A routing facade sits in front and directs each request to either the new component or the old one, so both run in production together and you never need a single cutover.
> **Why not just rewrite and cut over on a weekend?**
Because a cutover is a single high-stakes event with no safe rollback. You freeze features to build the replacement, it drifts from the original as the original keeps changing, and if anything breaks on migration night your only option is reverting everything at once. Strangler-fig keeps the old system live the whole time, so every step is small and reversible.
> **What makes a good first service to extract?**
Low coupling and a clear data owner. Pick a capability with a clean boundary that mostly owns its own data, like users or billing, so you are not untangling shared state on your first attempt. Early, low-risk wins build the tooling and the confidence you will need for the harder seams later.
> **How do you handle the shared database?**
Give each extracted service its own store rather than pointing it at the monolith's schema. Backfill it and keep it in sync while both paths run, then cut the monolith's write path over only once the new service is authoritative and verified. Sharing one database across services is a distributed monolith and defeats the point of the migration.
> **How do you know when the migration is done?**
When the remaining monolith is small and stable enough that extracting more would cost more than it returns. Track the share of production traffic served by extracted services and the amount of code deleted from the monolith. Done does not have to mean zero monolith; a low-change remainder behind the facade is a perfectly good ending.
## References
- [Martin Fowler: StranglerFigApplication](https://martinfowler.com/bliki/StranglerFigApplication.html)
- [Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html)
- [AWS Prescriptive Guidance: strangler fig pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/modernization-decomposing-monoliths/strangler-fig.html)
- [Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/)
- [Database decomposition patterns](https://microservices.io/patterns/data/database-per-service.html)
---
# Multi-Region Active-Active for a Payments API
A payments API that moves real money had been running comfortably in a single AWS region for years. It was reliable until the day it was not: a regional control-plane incident took the whole service offline for a few hours, and there was no second region to fail over to. For most products that is an outage. For a money-movement API it is stuck settlements, angry partners, and a compliance conversation. The mandate that came out of that incident was simple to say and hard to build: survive the loss of an entire region without losing a committed payment or charging anyone twice.
This is the story of taking that API active-active across two regions. The interesting part is not the traffic routing, which is close to a solved problem. The interesting part is the money: making retries safe, keeping two live databases honest, and proving the failover actually works instead of trusting a diagram.
## Impact
Once the second region went live, a full regional failure stopped being an incident and became a drill. During the quarter after cutover the primary region had two brief degradations, and in both cases traffic shifted to the healthy region inside the target window with no customer-visible errors and, most importantly, no duplicate settlements.
The number the finance and risk teams cared about was the double-charge count, and it stayed at zero. That is not because failures stopped happening. It is because every write path was made idempotent and every retry, whether from a client, a load balancer, or a queue redelivery, converges on the same result.
## The problem
A single-region payments API has two failure modes that a diagram tends to hide. The first is total loss of the region, which is rare but catastrophic and completely outside your control. The second, and the one that actually bites during a failover, is the retry storm: when a region gets shaky, every client, proxy, and queue in the system starts retrying, and if those retries are not idempotent, you turn one payment into several.
The business could tolerate a couple of minutes of elevated latency during a failover. It could not tolerate a lost payment that a customer had already seen succeed, and it absolutely could not tolerate charging a card twice. So the real problem was not "run in two regions." It was "make every money-touching operation safe to repeat, then run in two regions."
## Constraints
The design had to fit inside some hard limits. Payments are regulated, so data residency rules meant certain records could not leave their region of origin, which ruled out a naive single global write master. The team ran on Kubernetes (EKS) and Aurora PostgreSQL already, so the solution had to build on those rather than introduce an exotic new datastore. And the failover had to be measurable: leadership wanted a specific RTO and RPO written down and proven, not a hand-wave.
There was also a people constraint. On-call engineers needed a failover they could trust at 3am without a runbook full of manual database promotion steps, because manual steps under pressure are how a recoverable incident becomes a data-loss incident.
## Architecture
Both regions run the full stack and take live traffic. Route 53 uses latency-based routing with health checks so users hit the closest healthy region, and it fails a region out automatically when its health check trips. Each region has its own ALB, API pods on EKS, an idempotency store, and a database.
_Active-active across two AWS regions_
Two decisions carry the whole design. The idempotency store is a DynamoDB global table, replicated multi-active across both regions, so a key claimed in one region is visible in the other within about a second. The system of record is Aurora Global Database: a writer in the primary region with sub-second physical replication to the secondary, where a reader can be promoted to writer during a failover in roughly a minute. Committed transactions replicate fast enough that the recovery point stays effectively at zero for anything the customer already saw succeed.
The failover path is deliberately boring. A health check trips, DNS shifts, Aurora promotes the secondary, and in-flight retries replay with their idempotency key.
_The regional failover timeline (RTO and RPO)_
## Implementation
The idempotency key does the real work here. Every payment request carries an `Idempotency-Key` header. Before doing any work, the API does a conditional write into the idempotency store to claim that key. If the key already exists, the stored result is returned as-is and no charge happens. If the claim succeeds, the API runs the charge inside a database transaction, records the result under the key with a TTL, and returns it. Any retry, from any region, with the same key gets the same answer.
_An idempotent charge that survives failover_
The subtle bug to avoid is claiming the key and then crashing before the result is stored, which would leave a claimed-but-unfinished key that blocks the retry forever. The fix is to store an in-progress marker at claim time and let the retry either return the finished result or safely resume, with the transaction as the source of truth for whether the money actually moved.
Events flowing out to downstream systems (ledgers, notifications) use a transactional outbox, written in the same transaction as the payment, so an event is emitted exactly once per committed payment and consumers dedupe on the same key. That keeps the two regions from emitting conflicting events for the same operation.
## Results
Across the first quarter live, the primary region degraded twice. Both times Route 53 shifted traffic and Aurora promoted the secondary well inside the two-minute RTO target, and customers saw a short latency bump rather than errors. No payment was lost and nothing was charged twice, which was the entire point.
The less glamorous result was operational confidence. Because the failover is automatic and every write is idempotent, on-call stopped treating a regional wobble as an emergency. The monthly game-day, where a region is deliberately failed out in production-like conditions, went from a nerve-wracking event to a routine check with a green result.
## Lessons
The biggest lesson is that active-active is a data problem wearing a networking costume. Getting traffic to two regions is easy; keeping two live copies of money honest is the hard part, and idempotency is what makes it tractable. If you cannot safely repeat every write, no amount of clever routing will save you during a failover.
The second lesson is that an RTO and RPO you have not tested are just wishes. The game-days repeatedly surfaced small issues (a too-aggressive health-check threshold, a client that did not send idempotency keys on one endpoint) that no diagram would have caught. Failover is a feature, and like any feature it has bugs until you exercise it.
## Frequently Asked Questions
> **Why active-active instead of active-passive?**
Active-passive keeps a warm standby that only takes traffic during a failover,
which means the standby path is rarely exercised and tends to rot.
Active-active runs real traffic through both regions all the time, so the
failover path is the same path you use every day. It costs more, but for a
money-movement API the confidence that the second region actually works is
worth it.
> **How do idempotency keys prevent double charges during a failover?**
Every payment request carries a client-generated key. The API claims that key
in a globally replicated store before charging, and stores the result against
it afterward. If a retry arrives, in the same region or a different one after
failover, the key is already present and the original result is returned
without charging again. The key, not the region, is what guarantees
exactly-once.
> **What is the difference between RTO and RPO here?**
RTO (recovery time objective) is how long the service can be unavailable
before it is back, which here is the couple of minutes it takes DNS to shift
and Aurora to promote a writer. RPO (recovery point objective) is how much
committed data you can lose, which here is effectively zero because
idempotency writes are synchronous and database replication lag stays under a
second for committed transactions.
> **Does data residency break the active-active model?**
It constrains it. Records that legally must stay in their region of origin are
not globally writable, so the design keeps the system of record regional (a
promotable writer per region) rather than a single global write master. The
globally replicated piece is the idempotency store, which holds keys and
results, not the regulated ledger data.
## References
- [Amazon Aurora Global Database](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html)
- [Amazon DynamoDB global tables](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GlobalTables.html)
- [Making retries safe with idempotent APIs (AWS Builders' Library)](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)
- [Amazon Route 53 health checks and DNS failover](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover.html)
- [Transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html)
---
# QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog
When Bitnami moved its long-trusted catalog behind a paid tier, thousands of teams woke up to a supply-chain problem they didn't choose. The free images they had pinned in production would stop getting updates, and the migration clock started that morning. QuenchWorks is my answer to that. It is a catalog of container images and Helm charts, built entirely from source, hardened under a strict zero-CVE build gate, signed, and free. This is how it's put together and why each decision earns its place.
## Impact
QuenchWorks is real and in production use, not a proof of concept. It replaced Bitnami for common workloads with a catalog that is built from source, provable, and free.
Everything below is verifiable: pull any image and check its signature, SBOM, and provenance yourself. The build gate stays green because the base is small enough that there is almost nothing to be vulnerable in, so the numbers hold instead of drifting the week after launch.
## The problem
The Bitnami catalog was popular for good reasons. It was broad, it was versioned, and it was maintained well enough that most teams never thought about it. Its weaknesses only became obvious once access changed: you didn't control the build, you couldn't prove what was inside a given image, and continued free access was never actually guaranteed.
That last point is the one that bites. When an upstream catalog changes its terms, every `image:` line you pinned becomes a liability at once. You either pay, fork, or scramble. And even before that day comes, an opaque image is its own quiet risk. If you can't see how a layer was produced, you can't reason about what a scanner finds inside it, and you can't answer a security review with anything better than "we trust the vendor."
Most hardened-image alternatives fix one slice of this and charge for the rest. I wanted the whole thing: a catalog broad enough to actually replace Bitnami for common workloads, provable rather than "trust us," and free with no pull limits and no lock-in. If it couldn't be all three, it wasn't worth building.
## Constraints
A handful of hard limits shaped every later decision.
- **Zero fixable CVEs, enforced by the build.** Not a nightly report someone reads later. A gate that fails the build so a vulnerable image never ships in the first place.
- **Built from source.** No repackaging of someone else's opaque binary layers. If it's in the image, we produced it.
- **Provable.** Every image needs a bill of materials and build provenance that a consumer can verify without trusting me.
- **Free to run and maintain.** The whole system builds on free CI, so cost can never be the reason it slips behind a paywall later.
- **Multi-arch.** amd64 and arm64, because production is both now, not one or the other.
Those constraints pull against each other. Zero fixable CVEs across 150+ images sounds impossible if you picture a fat base image. Building everything from source sounds slow. The architecture is what makes them coexist.
## Architecture
The catalog is a pipeline, not a pile of Dockerfiles. Each image is declared as an `apko` plus `melange` spec, built from source on Wolfi, scanned against a zero-fixable-CVE gate, signed, and only then published pinned by digest. Charts sit on a shared library chart and reference those images by digest.
_QuenchWorks build pipeline_
The single decision that makes the zero-CVE gate realistic is the base. QuenchWorks builds on Wolfi, a glibc Linux undistro designed for containers. Most images start with no shell, no package manager, and a tiny set of packages. There's simply very little in the image that can be vulnerable, so keeping the gate green is a fight you can actually win instead of an endless race against a bloated base.
The image itself is assembled declaratively. `melange` builds signed APK packages from source, and `apko` composes those packages plus the Wolfi base into an OCI image with no Dockerfile involved. Because the whole thing is declared, the contents are known, reproducible, and easy to record as a bill of materials.
_Image composition with melange and apko_
The zero-CVE gate and the minimal base are the same decision viewed twice. You
don't reach zero fixable CVEs by patching harder. You reach it by shipping so
little that there's almost nothing to patch.
A catalog is never done, though, because CVEs are disclosed against packages long after an image ships. So the pipeline runs in reverse on a schedule. A nightly Trivy rescan checks every published image, and when a fix lands upstream the affected image rebuilds, re-enters the gate, gets re-signed, and republishes under a new digest. The catalog trends toward zero drift without anyone babysitting it.
_Nightly rescan and self-heal loop_
## Implementation
Each image is a pair of specs. `melange` describes how to build the package from source, and `apko` describes how to assemble the final image. Here's the shape of both, trimmed for clarity.
```yaml title="melange.yaml"
package:
name: my-app
version: 1.2.3
environment:
contents:
packages:
- build-base
pipeline:
- uses: fetch
with:
uri: https://example.com/my-app-${{package.version}}.tar.gz
expected-sha256: '...'
- uses: autoconf/configure
- uses: autoconf/make
- uses: autoconf/make-install
```
```yaml title="apko.yaml"
contents:
repositories:
- https://packages.wolfi.dev/os
packages:
- my-app
- ca-certificates-bundle
accounts:
users:
- username: nonroot
uid: 65532
run-as: 65532
archs:
- x86_64
- aarch64
entrypoint:
command: /usr/bin/my-app
```
The gate is one Trivy call, and it's deliberately strict about what counts. It only fails on CVEs that have a fix available, because a vulnerability with no upstream patch isn't something a rebuild can clear. Everything fixable has to be at zero before the image is allowed out.
```bash title="0-CVE gate"
# Fail the build if any FIXABLE HIGH/CRITICAL vulnerability is present
trivy image --ignore-unfixed --severity HIGH,CRITICAL \
--exit-code 1 ghcr.io/quenchworks/my-app:latest
```
Once an image passes, it gets signed and attested before it's pushed for real. Signing is keyless with Cosign, so there's no long-lived private key to leak, and the SBOM and SLSA provenance ride along as attestations.
1. **Build from source.** `melange` produces signed APKs; `apko` assembles the image with the Wolfi base, nonroot user, and read-only root filesystem defaults.
2. **Gate on zero fixable CVEs.** Trivy scans the full image. One fixable HIGH or CRITICAL fails the pipeline, so a vulnerable image never reaches the registry.
3. **Sign and attest.** Cosign signs the image keyless, then attaches an SPDX SBOM and a SLSA build-provenance attestation.
4. **Publish pinned by digest.** The image is pushed, and the Helm charts reference it by `sha256:` digest, never by a movable tag.
5. **Rescan nightly.** A scheduled Trivy run watches for new fixes and triggers the self-heal rebuild loop.
On the delivery side, charts are the second half of the story. Every chart builds on a shared `quench-common` library chart, so common concerns like security context, probes, and labels live in one place instead of being copy-pasted 120 times. Each chart pins its image by digest.
_Chart topology_
Pinning by digest instead of tag is what makes the catalog trustworthy in practice. A tag can be moved; a digest can't. When a consumer pins a QuenchWorks chart, they get exactly the bytes that passed the gate.
```yaml title="values.yaml"
image:
repository: ghcr.io/quenchworks/postgresql
# Pinned by digest, not tag. This is the exact image that passed the gate.
digest: 'sha256:abc123...'
```
The proof only matters if consumers can check it, so verification gets its own documented step. Anyone can verify an image's signature and attestations before it runs, and an admission policy can enforce that in the cluster so unsigned or unverifiable images never schedule.
```bash title="Verify before you run"
cosign verify \
--certificate-identity-regexp '^https://github.com/quenchworks/' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
ghcr.io/quenchworks/postgresql@sha256:abc123...
```
_Consumer verification workflow_
## Results
Everything below is running today, not staged for a launch.
- 150+ container images built from source on Wolfi, each gated to zero fixable CVEs.
- 120+ production Helm charts on the shared `quench-common` library, every one pinned to its image by digest.
- Every image cosign-signed with an SPDX SBOM and SLSA provenance, published under an ArtifactHub verified-publisher organization.
- Multi-arch (amd64 and arm64), nonroot, and read-only root filesystem by default.
- A nightly rescan and self-heal rebuild loop that keeps the catalog current as upstream ships fixes.
- Free and independent. There is no subscription, no registry pull limit, and nothing that locks you in.
The counts above are current catalog figures. Per-image build times and scan times vary by package, so I'd treat any single number there as indicative rather than a benchmark.
## Lessons
The biggest lesson is that the base image choice decides everything downstream. Trying to reach zero CVEs on a fat base is a treadmill; starting from Wolfi's minimal surface turns the gate into something you can keep green for months. If I'd started anywhere else, the self-heal loop would be firing constantly and the whole thing would feel like bailing water.
The second lesson is that provenance costs far less than it's worth. Signing and generating SBOMs added very little build time, but they change the catalog's whole posture. It stops being "trust me" and becomes "verify it yourself," which is the entire point of replacing an opaque upstream. If I were doing it again, I'd wire verification into the consumer docs even earlier, because an unverified signed image is only half the value.
The one thing I'd watch more carefully next time is chart sprawl. The `quench-common` library chart paid for itself immediately, but library conventions need to be locked down early. Once a few charts drift from the shared patterns, every future change gets more expensive.
## Frequently Asked Questions
> **How is 'zero CVE' actually possible across 150+ images?**
It's zero _fixable_ CVEs, and it's mostly a consequence of the base. Wolfi images ship with almost nothing beyond what the app needs, so there's very little surface for a vulnerability to live in. The Trivy gate then fails any build with a fixable HIGH or CRITICAL, so a vulnerable image can't ship. Vulnerabilities with no upstream fix are tracked but don't block, because a rebuild can't clear them.
> **Why pin charts to images by digest instead of a tag?**
A tag is a movable pointer; a digest is the content itself. If you pin `:latest` or even `:1.2.3`, the bytes behind that tag can change. Pinning `sha256:...` guarantees you get exactly the image that passed the gate and was signed. It's the difference between "probably the right image" and "provably the right image."
> **How do I verify an image before running it?**
Use `cosign verify` with the QuenchWorks certificate identity and OIDC issuer, as shown above. That checks the keyless signature against the transparency log. You can also verify the SBOM and SLSA provenance attestations, and enforce all of it in-cluster with an admission policy so nothing unsigned ever schedules.
> **What happens when a new CVE is disclosed after an image ships?**
The nightly Trivy rescan catches it. If the CVE is fixable, the affected image rebuilds from source, goes back through the gate, gets re-signed, and republishes under a new digest. You pick up the fix by moving your pin to the new digest. Nobody has to notice the CVE manually for the loop to run.
> **Is QuenchWorks really free, and what's the catch?**
It's free, with no subscription and no registry pull limits. The catch, if you call it one, is that you verify and pin things yourself rather than outsourcing trust to a vendor relationship. That's a feature for most teams: you get provenance you can audit instead of a support contract you have to believe.
> **Can I use the charts without adopting the whole catalog?**
Yes. The charts and images are independent. You can pull a single hardened image by digest, or install one chart, without buying into everything. The `quench-common` library chart is an implementation detail of the charts, not something you have to adopt in your own repos.
## References
- [QuenchWorks catalog and website](https://quench-works.com/)
- [QuenchWorks on GitHub](https://github.com/quenchworks)
- [Wolfi undistro](https://github.com/wolfi-dev)
- [apko](https://github.com/chainguard-dev/apko) and [melange](https://github.com/chainguard-dev/melange)
- [Trivy vulnerability scanner](https://trivy.dev/)
- [Sigstore Cosign](https://docs.sigstore.dev/)
- [SPDX](https://spdx.dev/) and [SLSA provenance](https://slsa.dev/)
- [ArtifactHub](https://artifacthub.io/)
---
# Zero-Downtime PostgreSQL Major-Version Upgrade at Scale
A multi-terabyte PostgreSQL 12 database was reaching end of life, and the business ran around the clock, so the usual answer of "schedule a maintenance window" was off the table. We upgraded it to PostgreSQL 16 while users kept reading and writing the whole time. The write pause during the final switch was measured in seconds, and the old database stayed hot the entire cutover so we could fail back instantly if anything looked wrong. This is how the migration was designed, rehearsed, and executed.
## Impact
The upgrade happened while users kept reading and writing. The final switch cost a write pause measured in seconds and lost no rows, with the old database kept hot for instant failback.
The safety came from never modifying the source until the very last step, so failback stayed a genuine option the entire time. These numbers are specific to this workload and were measured on our own traffic, so treat them as a shape to expect rather than a guarantee.
## The problem
The classic upgrade paths all needed a window we did not have. `pg_upgrade` with hard links is fast, but it still stops the database, and on a multi-terabyte instance you cannot risk a long tail if something goes wrong mid-upgrade. A dump and restore was measured in hours, which was a non-starter. Even RDS in-place major upgrades take the instance offline for the duration and give you no clean way to abort once they begin.
The workload was write-heavy and latency-sensitive, so we could not just pause the application either. What we needed was a way to build the new version alongside the old one, keep it continuously in sync with live traffic, and switch over in a single short, reversible step.
## Constraints
The migration had to satisfy four hard constraints, and every design decision came back to them.
- **No maintenance window.** The only acceptable interruption was a brief write pause during the final switch, on the order of seconds.
- **Reversible at every step.** Until we were certain, the old PostgreSQL 12 primary had to stay untouched and ready to take traffic back.
- **Multi-terabyte, so the initial copy is not free.** Seeding the target could not lock the source or saturate its I/O during business hours.
- **Correctness of the awkward bits.** Sequences, extensions, large objects, and tables without a primary key all needed explicit handling, because logical replication does not carry all of them for you.
## Architecture
The core idea is a source and a target running side by side, with logical replication streaming changes from the old database to the new one. The application never talks to Postgres directly; it goes through PgBouncer, which is what lets us flip traffic in one place at cutover time.
_Replication and cutover topology_
Logical replication works at the level of rows, not disk blocks, which is exactly why it can span major versions. A publication on the source declares which tables to stream, and a subscription on the target consumes that stream through a replication slot that tracks how far the target has consumed. Those three objects are the whole contract.
_Logical replication objects (class view)_
Physically, nothing about the deployment is exotic. The application nodes pool connections through PgBouncer, PgBouncer points at the source on port 5432, and a second, dormant route to the target waits for the switch. Both databases publish replication lag and error metrics so we can watch the gap close in real time.
_Deployment topology_
## Implementation
The upgrade moved through a fixed set of phases, and treating it as a small state machine kept everyone honest about which step we were on and what "done" meant for each one.
_Upgrade phases (state machine)_
First we turned on logical replication on the source. On RDS that means setting `rds.logical_replication` to `1` in the parameter group and rebooting once, well ahead of the migration. Tables that get updates or deletes need a replica identity so those changes can be matched on the target; a primary key covers most, and anything without one gets `REPLICA IDENTITY FULL`.
```sql title="On the source (PostgreSQL 12)"
-- Stream every table in the app schema.
CREATE PUBLICATION app_pub FOR ALL TABLES;
-- Tables without a primary key need a full replica identity
-- so UPDATE and DELETE can be replicated.
ALTER TABLE audit_events REPLICA IDENTITY FULL;
```
To seed the target without hammering the source during the day, we restored the schema and a recent snapshot into the new PostgreSQL 16 instance first, then created the subscription with `copy_data = false` so it only carried changes from that point forward. On a smaller database you can let the subscription do the initial copy itself, but at multiple terabytes the snapshot route keeps the source calm.
```sql title="On the target (PostgreSQL 16)"
-- Schema and a consistent snapshot are already restored here.
-- Subscribe for ongoing changes only; the bulk data is already present.
CREATE SUBSCRIPTION app_sub
CONNECTION 'host=source.internal dbname=app user=repl'
PUBLICATION app_pub
WITH (copy_data = false, create_slot = true, slot_name = 'app_sub_slot');
```
From there it was a waiting game while the target caught up. We watched the lag from both ends until the gap held near zero under normal write load.
```sql title="Watching the gap close"
-- On the source: how far behind is the subscriber?
SELECT slot_name, active,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS behind
FROM pg_replication_slots;
-- On the target: is the subscription streaming and healthy?
SELECT subname, received_lsn, latest_end_lsn, last_msg_receipt_time
FROM pg_stat_subscription;
```
Before touching traffic, we validated parity with dual reads. A read-only checker ran the same queries against both databases and compared row counts and checksums on the busiest tables. We wanted proof that the target was a faithful copy, not just a hopeful one.
Logical replication does not carry everything. Sequences are not replicated, DDL is not replicated, and large objects in `pg_largeobject` are not replicated. Freeze schema changes for the migration window, plan to reset sequences at cutover, and handle large objects separately. `pglogical` can sync sequences for you if you would rather not script it.
The cutover itself was a short, rehearsed sequence. The point of writing it as an ordered exchange between the operator, PgBouncer, and the two databases was to make the timing and the ordering unambiguous.
_Cutover sequence_
We drove it from a runbook with a hard time budget and an explicit rollback branch. If replication did not reach zero lag inside the budget, or the post-cutover smoke tests failed, we resumed writes on the untouched source and walked away to try another day.
_Cutover runbook (activity)_
The two commands at the heart of the switch are pausing the pool and resetting sequences, since sequence values do not come across on their own.
1. Pause new writes at PgBouncer with `PAUSE`, which lets in-flight transactions finish and holds new ones.
2. Confirm on the source that the replication slot has drained to zero lag.
3. Reset every sequence on the target from the source's current values, then run `ANALYZE` so the planner has fresh statistics.
4. Repoint PgBouncer at the target and `RESUME`, so held connections wake up talking to PostgreSQL 16.
5. Run smoke tests. If they pass, announce done. If not, repoint back to the source and resume there.
```bash title="Reset sequences on the target from the source"
# Emit setval() calls from the source, apply them on the target.
psql "$SOURCE" -Atc "SELECT format('SELECT setval(%L, %s);', seqrelid::regclass, last_value)
FROM pg_sequences_lastvals()" \
| psql "$TARGET"
```
## Results
The measured write pause during the switch was a handful of seconds, dominated by draining in-flight transactions rather than any copy. Read traffic was never interrupted, since reads could keep hitting the source until the pool repointed. No rows were lost, which the dual-read parity checks confirmed both before and after cutover.
Because the source stayed primary until the very last step and was never modified, rollback stayed a genuine option right up to the point we chose to decommission it, which we did only after a full business day of clean operation on PostgreSQL 16. These numbers are specific to this workload and were measured on our own traffic; treat them as a shape to expect, not a guarantee.
## Lessons
The single most valuable thing we did was rehearse the cutover against a copy until the runbook was boring. The first rehearsal surfaced the sequence problem, the second surfaced a table with no primary key, and by the third the whole thing was muscle memory.
Watching replication lag as a first-class metric mattered more than any single command. The go or no-go decision at cutover was a number on a dashboard, not a gut feel. And keeping the old primary untouched turned rollback from a scary, multi-hour restore into a one-line repoint, which is what made the whole plan safe enough to run against production in the first place.
## Frequently Asked Questions
> **Why not just use pg_upgrade or an in-place RDS major upgrade?**
Both stop the database for the duration and give you no clean abort once they start. On a multi-terabyte, 24/7 workload that downtime and that lack of a rollback were unacceptable. Logical replication lets you build the new version alongside the old one and switch over in a short, reversible step instead.
> **What is the difference between logical and physical replication here?**
Physical (streaming) replication copies disk blocks and requires both sides to run the same major version, so it cannot help you upgrade. Logical replication ships row-level changes decoded from the WAL, which is version-independent, so a PostgreSQL 12 primary can feed a PostgreSQL 16 subscriber.
> **Why do sequences need special handling?**
Logical replication streams table data but not sequence values, so the target's sequences would still sit at wherever the initial copy left them. If you skip the reset, the first inserts after cutover can collide with existing primary keys. We reset every sequence from the source's live values as part of the cutover, just before resuming writes.
> **How did you seed a multi-terabyte target without hurting the source?**
We restored a recent snapshot and the schema into the target first, then created the subscription with copy_data set to false so it only carried changes from that point forward. That avoids a giant online copy that would compete with production traffic. On a small database you can let the subscription copy the data itself.
> **What was the actual rollback plan?**
Until the final switch, the source stayed primary and unmodified. If replication did not reach zero lag inside the time budget, or the post-cutover smoke tests failed, we repointed PgBouncer back at the source and resumed writes there. Because the source never diverged, that was instant and lossless.
> **Does this work on Amazon RDS?**
Yes. Set rds.logical_replication to 1 in the parameter group and reboot once ahead of time so the source starts producing logical WAL. From there the publication, subscription, and slot work the same as on self-managed PostgreSQL. The target can be a fresh RDS instance on the new major version.
> **What about large objects and extensions?**
Neither rides along automatically. Extensions must be installed on the target before you subscribe, and large objects stored in pg_largeobject are not replicated by native logical replication, so migrate them separately during the window. If your schema leans heavily on either, factor that into rehearsals.
## References
- [PostgreSQL: Logical Replication](https://www.postgresql.org/docs/current/logical-replication.html)
- [PostgreSQL: CREATE PUBLICATION](https://www.postgresql.org/docs/current/sql-createpublication.html)
- [PostgreSQL: CREATE SUBSCRIPTION](https://www.postgresql.org/docs/current/sql-createsubscription.html)
- [PostgreSQL: Replication Slots and pg_replication_slots](https://www.postgresql.org/docs/current/view-pg-replication-slots.html)
- [pglogical (2ndQuadrant / EDB)](https://github.com/2ndQuadrant/pglogical)
- [Amazon RDS for PostgreSQL: Logical Replication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.LogicalReplication)
- [PgBouncer: PAUSE and RESUME](https://www.pgbouncer.org/usage.html)
---
# Content sections
# blog
0 entries.
# Case Studies
6 entries.
- [Multi-Region Active-Active for a Payments API](https://mkabumattar.com/case-studies/post/multi-region-active-active-payments), How a money-movement API was taken active-active across two AWS regions with idempotency keys, conflict-free replication, and a tested RTO and RPO, so a full regional outage never double-charges a customer or loses a committed payment.
- [Building an Internal Developer Platform on Backstage and GitOps](https://mkabumattar.com/case-studies/post/internal-developer-platform-backstage-gitops), How golden paths in Backstage, self-service software templates, and Argo CD let product teams create, build, and ship services without filing tickets to the platform team.
- [Zero-Downtime PostgreSQL Major-Version Upgrade at Scale](https://mkabumattar.com/case-studies/post/zero-downtime-postgres-upgrade), How we moved a multi-terabyte PostgreSQL 12 database to 16 with no maintenance window, using logical replication, dual-read validation, and a rehearsed, timed cutover with a real rollback trigger.
- [Migrating a Monolith to Kubernetes Without a Big-Bang Cutover](https://mkabumattar.com/case-studies/post/monolith-to-kubernetes-strangler-migration), Using the strangler-fig pattern to move a large monolith onto EKS service by service, with a routing facade, gradual traffic shifting, and a rollback at every step.
- [Cutting a SaaS AWS Bill 41% Without Slowing Delivery](https://mkabumattar.com/case-studies/post/aws-cost-optimization-saas-case-study), A FinOps case study on a SaaS running on EKS with full GitOps and progressive delivery: how tagging, right-sizing node groups, and Savings Plans matched to the roadmap cut the AWS bill without freezing feature work.
- [QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog](https://mkabumattar.com/case-studies/post/quenchworks-zero-cve-catalog), How a from-scratch catalog replaced Bitnami with 150+ container images and 120+ Helm charts built from source on Wolfi, gated to zero fixable CVEs, signed, and pinned by digest.
# cheatsheets
0 entries.
# codesnippets
0 entries.
# devtips
0 entries.
# flashcards
0 entries.
# glossary
0 entries.
# quizzes
0 entries.
# roadmaps
0 entries.
# Series
6 series.
- [Cloud Platforms & Architecture](https://mkabumattar.com/series/cloud-platforms--architecture), 1 post (Case studies)
- [Containers & Kubernetes](https://mkabumattar.com/series/containers--kubernetes), 1 post (Case studies)
- [Databases & Data Persistence](https://mkabumattar.com/series/databases--data-persistence), 1 post (Case studies)
- [FinOps & Cost Optimization](https://mkabumattar.com/series/finops--cost-optimization), 1 post (Case studies)
- [Platform Engineering](https://mkabumattar.com/series/platform-engineering), 1 post (Case studies)
- [QuenchWorks](https://mkabumattar.com/series/quenchworks), 1 post (Case studies)