docs(ops): prove the backups during the cutover rather than waiting for a schedule (#192)
Step 4b created the directories and step 7 checked the containers were Up. Neither established that a restorable file actually gets written, and those are not the same claim — the database backup does not run until 03:00 and the uploads archive not until Sunday 04:00, so a stack that looks correct at the end of a cutover can be four days from its first evidence. Both tools take a manual trigger, so the wait is unnecessary. The runbook now forces one run of each, checks the sizes are plausible, and greps the dump for `COPY` lines on the real tables — a dump of an empty database succeeds and looks fine, which is the one way this check could otherwise lie. It also confirms the healthchecks agree with where the files landed. A check whose `find` path disagrees with where the tool actually writes reports unhealthy forever while the backups are working perfectly, and that is a thing to discover on the day the stack is built rather than a year later. `starting` corrected to `unhealthy` in the surrounding text: during `start_period` Docker reports `starting`, which is what an operator actually sees and what the previous wording got wrong. Proven against production during the cutover on 2026-08-26 — a 41K dump and a 15M archive, both landing where the healthchecks look.
This commit is contained in:
@@ -135,7 +135,33 @@ Verify after the stack is up, as part of step 7:
|
||||
sudo docker ps --filter name=backup --format '{{.Names}}\t{{.Status}}'
|
||||
```
|
||||
|
||||
Both must be **Up**. Both will also report *unhealthy* at first and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads. Read those healthchecks after the first run, not before.
|
||||
Both must be **Up**. Both will also report *starting* rather than healthy, and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads.
|
||||
|
||||
**Do not wait for those runs to find out whether this works.** Up and scheduled is not the same as producing a restorable file, and the difference is one command each:
|
||||
|
||||
```bash
|
||||
sudo docker exec redefined-designs-db-backup-syn /backup.sh
|
||||
ls -lht /volume1/configs/redefined-designs/backups/postgres/daily | head
|
||||
|
||||
sudo docker exec redefined-designs-uploads-backup-syn backup
|
||||
ls -lht /volume1/configs/redefined-designs/backups/uploads | head
|
||||
```
|
||||
|
||||
Check the sizes are plausible. A dump of an empty database succeeds and tells you nothing, so confirm the dump carries rows rather than only a schema:
|
||||
|
||||
```bash
|
||||
zcat /volume1/configs/redefined-designs/backups/postgres/daily/redefined-latest.sql.gz \
|
||||
| grep -E "^COPY public\.(items|customers|orders)"
|
||||
```
|
||||
|
||||
Then confirm the healthchecks agree with where the files landed — a check whose `find` path disagrees with where the tool actually writes reports unhealthy forever, and is worth catching now rather than in a year:
|
||||
|
||||
```bash
|
||||
sudo docker inspect --format '{{.Name}} {{.State.Health.Status}}' \
|
||||
redefined-designs-db-backup-syn redefined-designs-uploads-backup-syn
|
||||
```
|
||||
|
||||
Both reach `healthy` within their check intervals once a file exists — an hour for the database, six for uploads.
|
||||
|
||||
## 5. Remove the old stack
|
||||
|
||||
|
||||
Reference in New Issue
Block a user