“Keep the last fourteen” is a sentence about time. A disk only understands bytes. Here is the morning that difference cost us, and the small change that fixed it for good.
- The result: the shared drive went from 385 MB free to 6.3 GB free, and stays bounded after every nightly run.
- Who: a single-developer machine backing up ten projects’ databases to one small shared drive.
- Timeframe: a 2026-09-27 “disk full” incident and the retention change that followed.
Table of Contents
Why count-based backup rotation fills the disk
A retention rule like “keep the newest 14 dumps per database, delete the rest” looks complete. It isn’t. It measures recency – how far back your restores can reach – and is blind to capacity – how much space those restores occupy. The two agree only when every dump is the same size, and dumps never stay the same size.
The arithmetic the rule can’t do:
765 MB per dump × 14 kept = 10.7 GB — for ONE databaseThat database had grown for months while the retention count sat frozen at 14. The policy never changed; the bytes behind each slot tripled underneath it. Count-based rotation cannot notice, because fourteen is fourteen whether a dump is 40 MB or 800 MB. On a disk shared with nine other projects, one database quietly claimed about 12 of the 13 GB of backups – and then the drive ran out.
A backup system that fills its own disk stops taking backups at the moment the failure is least visible – overnight, while rotation still deletes the oldest file.
Read the bytes before you touch anything
The first move in any “disk full” incident is to stop guessing and group the offenders by size. A size-by-prefix pass over the backup directory told the whole story on one screen: one database’s dumps dwarfed everything else, and – the surprise – there were two naming schemes for it.
A retired backup script had written files named backup_<db>_<timestamp>.dump. The current one wrote <db>_<reason>_<timestamp>.dump. The current rotation deleted old files by matching <db>_*.dump – a pattern that does not match backup_<db>_*. So several gigabytes of dumps from the dead scheme were immortal: no rotation on the machine would ever match them again.
Rotation is coupled to naming: retire one backup script for another, and every file under the old name becomes invisible to the new cleanup.
That orphaned pile had sat growing since the day the old script was retired, counted by nobody. Reaping it by hand recovered the first few gigabytes. But reaping is a one-time repair; the structural problem was still the count-based rule.
The fix: give retention two axes
The repair is not “lower keep to six.” That just moves the cliff closer and still can’t see bytes. The missing primitive is a capacity axis sitting next to the recency one:
- Recency (
keep = N) – per database, keep the newest N dumps. Keep this; it is the right tool for “how far back can I restore.” - Capacity (a per-destination byte budget) – a hard ceiling on total dump bytes per drive. The small shared disk gets a budget; the roomy drive and the off-site copy stay unbounded and keep long histories.
After the normal keep-rotation runs, a second pass enforces the budget on any drive that has one. While a drive is over its ceiling, it deletes the oldest dump whose database still has more than a floor number of copies there, and never a database’s single newest dump. Three properties make it safe to run unattended every night:
- Age-ordered, so retention is predictable – the oldest file always goes first.
- Floor-protected, so capacity pressure can never erase a database’s last few safety copies, however tight the budget.
- Newest-preserved, so the most recent backup of every database is untouchable.
Two smaller decisions matter more than they look.
A budget of zero must mean “no limit,” not “delete everything” – the opposite default turns one fat-fingered digit into an unrecoverable mistake.
A budget of 0, or an unset one, has to mean unbounded. And the pass must only ever delete files the backup system itself created; an unrecognized file in the directory is left strictly alone. A cleanup that deletes “anything not matching a known pattern” is exactly how you lose the one file you forgot to name correctly.
What it looked like after
Two changes landed together: a one-time reap of the orphaned dumps from the retired scheme – verified redundant first, because you never delete a backup you have not confirmed is superseded – and the standing byte budget, which trimmed the big database’s history on the small disk down to fit, oldest first, leaving the newest handful plus the floor.
The shared drive went from 385 MB free to 6.3 GB free. More important than the number: it stays bounded. The budget runs after every nightly backup, so the database can keep growing and the small disk cannot be driven to the wall again. The cliff is gone, not postponed.
This sits one layer below the database itself – the same “the default is not the real limit” lesson that bites with your Postgres connection limit: the number you configured is not the number the system is actually bounded by.
The short version
- Count-based rotation is a recency policy wearing a capacity costume. Keep it, but never make it your only defense on a disk you share.
- Add a per-destination byte budget: age-ordered, floor-protected, newest-preserved. It’s the primitive that actually understands “this disk is only so big.”
- When you retire a backup scheme, reap its files by hand. New rotation patterns do not inherit old names; orphans are silent and immortal.
- Make destructive defaults safe: a zero or missing budget means unbounded, and only ever delete files the system itself created.
- Treat a self-filling backup disk as an incident in waiting. The failure is invisible until the night it isn’t.
Frequently Asked Questions
Why did keeping only 14 backups still fill the disk?
Because “keep 14” counts files, not bytes. When one database’s dump grew to 765 MB, fourteen of them became 10.7 GB – on a disk shared with nine other projects. The retention rule never changed; the size behind each of its fourteen slots did. A count-based rule cannot see that, which is why it kept deleting the oldest file while the total marched toward the disk’s limit.
What is a per-destination byte budget?
A ceiling on the total size of backups allowed on a specific drive, enforced separately from the per-database keep count. After the normal “keep the newest N” rotation runs, a budget pass checks each drive that has a ceiling and, while it is over, deletes the oldest dumps until it fits. It is a capacity rule, where keep is a recency rule; you want both.
Won’t a byte budget delete my only backup?
Not if it is built correctly. The budget pass must honor a floor – a minimum number of dumps per database it will never go below – and must never delete any database’s single newest copy, no matter how tight the ceiling. If a drive still can’t fit under budget without breaking those guarantees, the right behavior is to stop and report, not to keep deleting.
Should I just lower the keep count instead?
That moves the cliff closer rather than removing it, and it is still blind to size: a lower count of a still-growing dump eventually overruns the disk again. Lowering keep also shortens your restore window for every database equally, including small ones that cost almost nothing to retain. A byte budget targets the actual constraint – the disk – without shortening recency for databases that aren’t the problem.
Why were there backups that nothing ever deleted?
Because a retired backup script had written files under a different name than the current one, and the current rotation only matched its own naming pattern. Rotation is coupled to naming: the moment you change how backups are named or swap one script for another, files under the old name fall outside the new cleanup and accumulate forever. Reap the old scheme by hand when you retire it.
How often should the budget run?
After every backup run. Enforcing capacity immediately after new dumps are written means the drive is brought back under its ceiling the moment it is exceeded, so a growing database can never accumulate across several nights into another disk-full event. A budget that only runs occasionally re-introduces the window you were trying to close.
