From 724e9ce19d6b9db3f545b3c53a24a001f78e4f4e Mon Sep 17 00:00:00 2001 From: Thom Lamb Date: Wed, 26 Aug 2026 10:14:10 -0500 Subject: [PATCH 1/2] docs(ops): say that the backup directories have to be created (#192) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The stack gained two backup services in #147 and nothing has ever told anyone to create the directories they mount. `backup-and-restore.md` reads from both paths and the compose file mounts both, but no document creates them — while the README does exactly that for QA's data directories, ownership notes and all. Production's cutover runbook said nothing. Hit for real during the cutover: both backup containers sat in Created, never started, and `docker logs` on them reported only that nothing matched the filter, because a container that never ran has no output. Portainer showed them beside the healthy ones and the stack looked deployed. That silence is the reason this is worth a step of its own rather than a footnote. A backup regime that never started is indistinguishable from a working one until someone needs a restore, which is the failure mode the healthchecks in #147 exist to catch — and those healthchecks cannot fire on a container that is not running. The cutover runbook gains the directory creation before the stack is created, and step 7 now counts containers rather than only checking the app: four, all Up, with Created called out as the thing to look for. Counted from the compose file rather than from memory — the first draft said five. `backup-and-restore.md` gains the same note where it describes the destinations, since anyone reading that page is already thinking about paths. No `chown`, deliberately stated: both backup images run as root, unlike the Postgres image whose data directory needs uid 999, and an unnecessary chown instruction is how people learn to run them without thinking. --- docs/ops/backup-and-restore.md | 7 +++++++ docs/ops/production-stack-cutover.md | 27 +++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/docs/ops/backup-and-restore.md b/docs/ops/backup-and-restore.md index e16f266..1dcbe21 100644 --- a/docs/ops/backup-and-restore.md +++ b/docs/ops/backup-and-restore.md @@ -11,6 +11,13 @@ Two services in `docker-compose.prod.yml`, alongside the app and the database. | `redefined-designs-db-backup-syn` | `pg_dump` of `redefined`, gzipped | Daily, 03:00 | 7 daily, 4 weekly, 6 monthly | `/volume1/configs/redefined-designs/backups/postgres` | | `redefined-designs-uploads-backup-syn` | `tar.gz` of the uploads directory | Weekly, Sunday 04:00 | 56 days | `/volume1/configs/redefined-designs/backups/uploads` | +Both destinations are bind mounts that have to exist before the stack starts. Nothing creates them, and nothing complains when they are missing — the container sits in **Created**, never runs, and so has no logs to explain itself. Creating them is a step in the cutover runbook; on any other deploy, check for them first: + +```bash +sudo mkdir -p /volume1/configs/redefined-designs/backups/{postgres,uploads} +sudo docker ps --filter name=backup --format '{{.Names}} {{.Status}}' +``` + Both prune automatically. Both report **unhealthy** when their newest artifact is older than their interval plus grace, so a regime that has quietly stopped shows up in Portainer's stack view beside the app rather than being discovered during a restore. The database dumper is pinned to `prodrigestivill/postgres-backup-local:16`, matching the server's major version. `pg_dump` refuses to dump a server newer than itself, so a floating tag would be a backup that stops working the day Postgres is upgraded — silently, because nothing reads the dumps until they are needed. diff --git a/docs/ops/production-stack-cutover.md b/docs/ops/production-stack-cutover.md index ed452ac..ee0b0cc 100644 --- a/docs/ops/production-stack-cutover.md +++ b/docs/ops/production-stack-cutover.md @@ -115,6 +115,28 @@ sudo docker inspect --format '{{.Config.Cmd}}' redefined-designs:latest That `Cmd` check is worth doing. It is what makes the container migrate before serving, so deployed code can never be ahead of the schema. +## 4b. Create the backup directories + +The stack gained two backup services in #147, and they bind-mount directories that no earlier deploy of this application ever needed: + +```bash +sudo mkdir -p /volume1/configs/redefined-designs/backups/postgres +sudo mkdir -p /volume1/configs/redefined-designs/backups/uploads +ls -la /volume1/configs/redefined-designs/backups/ +``` + +No `chown`. Both backup images run as root, unlike the Postgres image, whose data directory has to be owned by uid 999. + +This is easy to skip because nothing fails loudly. A backup container whose mount path is missing does not crash — it sits in **Created**, having never started, and `docker logs` on it says only that nothing matched, because there is no output from a container that never ran. Portainer shows it beside the healthy ones and the stack looks deployed. The failure surfaces the day someone needs a restore. + +Verify after the stack is up, as part of step 7: + +```bash +sudo docker ps --filter name=backup --format '{{.Names}}\t{{.Status}}' +``` + +Both must be **Up**. Both will also report *unhealthy* at first and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads. Read those healthchecks after the first run, not before. + ## 5. Remove the old stack In Portainer: **Stacks → the stack from step 2 → Delete**. @@ -204,8 +226,13 @@ The counts must match what you recorded in step 2. If they are zero, the contain ```bash # Are the images still served? sudo docker exec redefined-designs-syn ls /app/uploads | head + +# Did every container in the stack actually start? +sudo docker ps -a --filter name=redefined-designs --format '{{.Names}}\t{{.Status}}' ``` +Four containers, all **Up** — `redefined-designs-syn`, `redefined-designs-db-syn`, and the two backup services from step 4b. A container reading **Created** never started — that is a missing bind-mount directory or a deploy that aborted part way, and it is silent, because a container that never ran has no logs to read. + Then load the storefront in a **private window** — aggressive bundle caching on this project has produced false "still broken" reports after correct deploys. Check that product images render, sign in, and open `/admin` to confirm authentik and the admin gate still agree. ## 8. If it goes wrong From 1a59edec182e65859cedfd678a6ff722a6c96819 Mon Sep 17 00:00:00 2001 From: Thom Lamb Date: Wed, 26 Aug 2026 10:40:39 -0500 Subject: [PATCH 2/2] docs(ops): prove the backups during the cutover rather than waiting for a schedule (#192) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 4b created the directories and step 7 checked the containers were Up. Neither established that a restorable file actually gets written, and those are not the same claim — the database backup does not run until 03:00 and the uploads archive not until Sunday 04:00, so a stack that looks correct at the end of a cutover can be four days from its first evidence. Both tools take a manual trigger, so the wait is unnecessary. The runbook now forces one run of each, checks the sizes are plausible, and greps the dump for `COPY` lines on the real tables — a dump of an empty database succeeds and looks fine, which is the one way this check could otherwise lie. It also confirms the healthchecks agree with where the files landed. A check whose `find` path disagrees with where the tool actually writes reports unhealthy forever while the backups are working perfectly, and that is a thing to discover on the day the stack is built rather than a year later. `starting` corrected to `unhealthy` in the surrounding text: during `start_period` Docker reports `starting`, which is what an operator actually sees and what the previous wording got wrong. Proven against production during the cutover on 2026-08-26 — a 41K dump and a 15M archive, both landing where the healthchecks look. --- docs/ops/production-stack-cutover.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/ops/production-stack-cutover.md b/docs/ops/production-stack-cutover.md index ee0b0cc..75b0339 100644 --- a/docs/ops/production-stack-cutover.md +++ b/docs/ops/production-stack-cutover.md @@ -135,7 +135,33 @@ Verify after the stack is up, as part of step 7: sudo docker ps --filter name=backup --format '{{.Names}}\t{{.Status}}' ``` -Both must be **Up**. Both will also report *unhealthy* at first and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads. Read those healthchecks after the first run, not before. +Both must be **Up**. Both will also report *starting* rather than healthy, and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads. + +**Do not wait for those runs to find out whether this works.** Up and scheduled is not the same as producing a restorable file, and the difference is one command each: + +```bash +sudo docker exec redefined-designs-db-backup-syn /backup.sh +ls -lht /volume1/configs/redefined-designs/backups/postgres/daily | head + +sudo docker exec redefined-designs-uploads-backup-syn backup +ls -lht /volume1/configs/redefined-designs/backups/uploads | head +``` + +Check the sizes are plausible. A dump of an empty database succeeds and tells you nothing, so confirm the dump carries rows rather than only a schema: + +```bash +zcat /volume1/configs/redefined-designs/backups/postgres/daily/redefined-latest.sql.gz \ + | grep -E "^COPY public\.(items|customers|orders)" +``` + +Then confirm the healthchecks agree with where the files landed — a check whose `find` path disagrees with where the tool actually writes reports unhealthy forever, and is worth catching now rather than in a year: + +```bash +sudo docker inspect --format '{{.Name}} {{.State.Health.Status}}' \ + redefined-designs-db-backup-syn redefined-designs-uploads-backup-syn +``` + +Both reach `healthy` within their check intervals once a file exists — an hour for the database, six for uploads. ## 5. Remove the old stack