From 1a59edec182e65859cedfd678a6ff722a6c96819 Mon Sep 17 00:00:00 2001 From: Thom Lamb Date: Wed, 26 Aug 2026 10:40:39 -0500 Subject: [PATCH] docs(ops): prove the backups during the cutover rather than waiting for a schedule (#192) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 4b created the directories and step 7 checked the containers were Up. Neither established that a restorable file actually gets written, and those are not the same claim — the database backup does not run until 03:00 and the uploads archive not until Sunday 04:00, so a stack that looks correct at the end of a cutover can be four days from its first evidence. Both tools take a manual trigger, so the wait is unnecessary. The runbook now forces one run of each, checks the sizes are plausible, and greps the dump for `COPY` lines on the real tables — a dump of an empty database succeeds and looks fine, which is the one way this check could otherwise lie. It also confirms the healthchecks agree with where the files landed. A check whose `find` path disagrees with where the tool actually writes reports unhealthy forever while the backups are working perfectly, and that is a thing to discover on the day the stack is built rather than a year later. `starting` corrected to `unhealthy` in the surrounding text: during `start_period` Docker reports `starting`, which is what an operator actually sees and what the previous wording got wrong. Proven against production during the cutover on 2026-08-26 — a 41K dump and a 15M archive, both landing where the healthchecks look. --- docs/ops/production-stack-cutover.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/ops/production-stack-cutover.md b/docs/ops/production-stack-cutover.md index ee0b0cc..75b0339 100644 --- a/docs/ops/production-stack-cutover.md +++ b/docs/ops/production-stack-cutover.md @@ -135,7 +135,33 @@ Verify after the stack is up, as part of step 7: sudo docker ps --filter name=backup --format '{{.Names}}\t{{.Status}}' ``` -Both must be **Up**. Both will also report *unhealthy* at first and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads. Read those healthchecks after the first run, not before. +Both must be **Up**. Both will also report *starting* rather than healthy, and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads. + +**Do not wait for those runs to find out whether this works.** Up and scheduled is not the same as producing a restorable file, and the difference is one command each: + +```bash +sudo docker exec redefined-designs-db-backup-syn /backup.sh +ls -lht /volume1/configs/redefined-designs/backups/postgres/daily | head + +sudo docker exec redefined-designs-uploads-backup-syn backup +ls -lht /volume1/configs/redefined-designs/backups/uploads | head +``` + +Check the sizes are plausible. A dump of an empty database succeeds and tells you nothing, so confirm the dump carries rows rather than only a schema: + +```bash +zcat /volume1/configs/redefined-designs/backups/postgres/daily/redefined-latest.sql.gz \ + | grep -E "^COPY public\.(items|customers|orders)" +``` + +Then confirm the healthchecks agree with where the files landed — a check whose `find` path disagrees with where the tool actually writes reports unhealthy forever, and is worth catching now rather than in a year: + +```bash +sudo docker inspect --format '{{.Name}} {{.State.Health.Status}}' \ + redefined-designs-db-backup-syn redefined-designs-uploads-backup-syn +``` + +Both reach `healthy` within their check intervals once a file exists — an hour for the database, six for uploads. ## 5. Remove the old stack