diff --git a/docs/ops/backup-and-restore.md b/docs/ops/backup-and-restore.md index e16f266..1dcbe21 100644 --- a/docs/ops/backup-and-restore.md +++ b/docs/ops/backup-and-restore.md @@ -11,6 +11,13 @@ Two services in `docker-compose.prod.yml`, alongside the app and the database. | `redefined-designs-db-backup-syn` | `pg_dump` of `redefined`, gzipped | Daily, 03:00 | 7 daily, 4 weekly, 6 monthly | `/volume1/configs/redefined-designs/backups/postgres` | | `redefined-designs-uploads-backup-syn` | `tar.gz` of the uploads directory | Weekly, Sunday 04:00 | 56 days | `/volume1/configs/redefined-designs/backups/uploads` | +Both destinations are bind mounts that have to exist before the stack starts. Nothing creates them, and nothing complains when they are missing — the container sits in **Created**, never runs, and so has no logs to explain itself. Creating them is a step in the cutover runbook; on any other deploy, check for them first: + +```bash +sudo mkdir -p /volume1/configs/redefined-designs/backups/{postgres,uploads} +sudo docker ps --filter name=backup --format '{{.Names}} {{.Status}}' +``` + Both prune automatically. Both report **unhealthy** when their newest artifact is older than their interval plus grace, so a regime that has quietly stopped shows up in Portainer's stack view beside the app rather than being discovered during a restore. The database dumper is pinned to `prodrigestivill/postgres-backup-local:16`, matching the server's major version. `pg_dump` refuses to dump a server newer than itself, so a floating tag would be a backup that stops working the day Postgres is upgraded — silently, because nothing reads the dumps until they are needed. diff --git a/docs/ops/production-stack-cutover.md b/docs/ops/production-stack-cutover.md index ed452ac..ee0b0cc 100644 --- a/docs/ops/production-stack-cutover.md +++ b/docs/ops/production-stack-cutover.md @@ -115,6 +115,28 @@ sudo docker inspect --format '{{.Config.Cmd}}' redefined-designs:latest That `Cmd` check is worth doing. It is what makes the container migrate before serving, so deployed code can never be ahead of the schema. +## 4b. Create the backup directories + +The stack gained two backup services in #147, and they bind-mount directories that no earlier deploy of this application ever needed: + +```bash +sudo mkdir -p /volume1/configs/redefined-designs/backups/postgres +sudo mkdir -p /volume1/configs/redefined-designs/backups/uploads +ls -la /volume1/configs/redefined-designs/backups/ +``` + +No `chown`. Both backup images run as root, unlike the Postgres image, whose data directory has to be owned by uid 999. + +This is easy to skip because nothing fails loudly. A backup container whose mount path is missing does not crash — it sits in **Created**, having never started, and `docker logs` on it says only that nothing matched, because there is no output from a container that never ran. Portainer shows it beside the healthy ones and the stack looks deployed. The failure surfaces the day someone needs a restore. + +Verify after the stack is up, as part of step 7: + +```bash +sudo docker ps --filter name=backup --format '{{.Names}}\t{{.Status}}' +``` + +Both must be **Up**. Both will also report *unhealthy* at first and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads. Read those healthchecks after the first run, not before. + ## 5. Remove the old stack In Portainer: **Stacks → the stack from step 2 → Delete**. @@ -204,8 +226,13 @@ The counts must match what you recorded in step 2. If they are zero, the contain ```bash # Are the images still served? sudo docker exec redefined-designs-syn ls /app/uploads | head + +# Did every container in the stack actually start? +sudo docker ps -a --filter name=redefined-designs --format '{{.Names}}\t{{.Status}}' ``` +Four containers, all **Up** — `redefined-designs-syn`, `redefined-designs-db-syn`, and the two backup services from step 4b. A container reading **Created** never started — that is a missing bind-mount directory or a deploy that aborted part way, and it is silent, because a container that never ran has no logs to read. + Then load the storefront in a **private window** — aggressive bundle caching on this project has produced false "still broken" reports after correct deploys. Check that product images render, sign in, and open `/admin` to confirm authentik and the admin gate still agree. ## 8. If it goes wrong