docs(ops): prove the backups during the cutover rather than waiting for a schedule (#192)
Linting / lint (pull_request) Successful in 2m45s
SonarQube Analysis / sonarqube (pull_request) Failing after 30m13s

Step 4b created the directories and step 7 checked the containers were Up. Neither established that a restorable file actually gets written, and those are not the same claim — the database backup does not run until 03:00 and the uploads archive not until Sunday 04:00, so a stack that looks correct at the end of a cutover can be four days from its first evidence.

Both tools take a manual trigger, so the wait is unnecessary. The runbook now forces one run of each, checks the sizes are plausible, and greps the dump for `COPY` lines on the real tables — a dump of an empty database succeeds and looks fine, which is the one way this check could otherwise lie.

It also confirms the healthchecks agree with where the files landed. A check whose `find` path disagrees with where the tool actually writes reports unhealthy forever while the backups are working perfectly, and that is a thing to discover on the day the stack is built rather than a year later.

`starting` corrected to `unhealthy` in the surrounding text: during `start_period` Docker reports `starting`, which is what an operator actually sees and what the previous wording got wrong.

Proven against production during the cutover on 2026-08-26 — a 41K dump and a 15M archive, both landing where the healthchecks look.
This commit is contained in:
2026-08-26 10:40:39 -05:00
parent 724e9ce19d
commit 1a59edec18
+27 -1
View File
@@ -135,7 +135,33 @@ Verify after the stack is up, as part of step 7:
sudo docker ps --filter name=backup --format '{{.Names}}\t{{.Status}}'
```
Both must be **Up**. Both will also report *unhealthy* at first and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads. Read those healthchecks after the first run, not before.
Both must be **Up**. Both will also report *starting* rather than healthy, and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads.
**Do not wait for those runs to find out whether this works.** Up and scheduled is not the same as producing a restorable file, and the difference is one command each:
```bash
sudo docker exec redefined-designs-db-backup-syn /backup.sh
ls -lht /volume1/configs/redefined-designs/backups/postgres/daily | head
sudo docker exec redefined-designs-uploads-backup-syn backup
ls -lht /volume1/configs/redefined-designs/backups/uploads | head
```
Check the sizes are plausible. A dump of an empty database succeeds and tells you nothing, so confirm the dump carries rows rather than only a schema:
```bash
zcat /volume1/configs/redefined-designs/backups/postgres/daily/redefined-latest.sql.gz \
| grep -E "^COPY public\.(items|customers|orders)"
```
Then confirm the healthchecks agree with where the files landed — a check whose `find` path disagrees with where the tool actually writes reports unhealthy forever, and is worth catching now rather than in a year:
```bash
sudo docker inspect --format '{{.Name}} {{.State.Health.Status}}' \
redefined-designs-db-backup-syn redefined-designs-uploads-backup-syn
```
Both reach `healthy` within their check intervals once a file exists — an hour for the database, six for uploads.
## 5. Remove the old stack