Skip to content
masterclass BuddyTech Buzz

We Deleted a Custom Resource and the Operator Quietly Snapshot-Then-Nuked Our Volumes

Learn how to run databases on Kubernetes with operators without losing data to a reconciliation loop or a careless delete.

MCB Editorial

10 min read ·

New story

Listen to this story

2 min preview

Learning Objectives & Key Takeaways

0 / 4 Completed
Architecture Ready
100%
What you will learn
  1. StatefulSets give stable identity and storage but never replicate or back up your data automatically.
  2. Operators extend StatefulSets by reconciling a custom resource into PVCs, StatefulSets, Services, and Jobs.
  3. PVCs from volumeClaimTemplates are retained by default on delete and on scale-down unless you set persistentVolumeClaimRetentionPolicy.
  4. Editing operator-managed child resources directly gets silently overwritten on the next reconcile.
  5. Always check an operator's deletion and retention behavior before you delete its custom resource or any PVC.

The Friday Afternoon Delete That Took the Weekend With It

A platform team at a mid-size fintech in Pune was decommissioning a staging Postgres cluster managed by an operator before a long weekend. They ran the equivalent of deleting the operator's custom resource, assuming it behaved like deleting a plain StatefulSet, where PVCs are retained by default. The operator, however, had its own deletion workflow: it took a snapshot, removed the StatefulSet, and then removed the underlying volumes as part of its documented cleanup.

The snapshot existed, but nobody had verified the operator's restore procedure in staging before, and the on-call engineer spent Saturday morning reading unfamiliar CRD status fields instead of enjoying a holiday. Production was never touched, but the incident retro surfaced an uncomfortable truth: the team understood StatefulSets well but had never read the operator's own documentation on deletion semantics.

In the rest of this story
  1. Why StatefulSet Alone Is Not a Database
  2. The Official Sequence: From Bare Cluster to Running Database
  3. A Runnable Example: StatefulSet With Explicit Retention Policy
  4. Operators in Practice: One CRD, Many Children, One Deletion Policy of Its Own
  5. Diagnosing the Pending Pod: A Field Guide
1 code example · 10 min read in full
Member-only story

Checking your membership…

Members read every Tech Buzz story in full, with code examples, audio and the discussion, plus every course, quiz and interview set in the Learning Dashboard.

0

Written by

MCB Editorial

Tech Buzz Daily Desk. Contributing deep-dive engineering architectures, cloud paradigms, and system design insights to Master Class Buddy.