bermudalamb
724e9ce19d
The stack gained two backup services in #147 and nothing has ever told anyone to create the directories they mount. `backup-and-restore.md` reads from both paths and the compose file mounts both, but no document creates them — while the README does exactly that for QA's data directories, ownership notes and all. Production's cutover runbook said nothing. Hit for real during the cutover: both backup containers sat in Created, never started, and `docker logs` on them reported only that nothing matched the filter, because a container that never ran has no output. Portainer showed them beside the healthy ones and the stack looked deployed. That silence is the reason this is worth a step of its own rather than a footnote. A backup regime that never started is indistinguishable from a working one until someone needs a restore, which is the failure mode the healthchecks in #147 exist to catch — and those healthchecks cannot fire on a container that is not running. The cutover runbook gains the directory creation before the stack is created, and step 7 now counts containers rather than only checking the app: four, all Up, with Created called out as the thing to look for. Counted from the compose file rather than from memory — the first draft said five. `backup-and-restore.md` gains the same note where it describes the destinations, since anyone reading that page is already thinking about paths. No `chown`, deliberately stated: both backup images run as root, unlike the Postgres image whose data directory needs uid 999, and an unnecessary chown instruction is how people learn to run them without thinking.
7.6 KiB
Backup and restore
What protects production's data, what each part does and does not cover, and how to get it back. See #147.
What runs
Two services in docker-compose.prod.yml, alongside the app and the database.
| Service | What | When | Kept | Lands in |
|---|---|---|---|---|
redefined-designs-db-backup-syn |
pg_dump of redefined, gzipped |
Daily, 03:00 | 7 daily, 4 weekly, 6 monthly | /volume1/configs/redefined-designs/backups/postgres |
redefined-designs-uploads-backup-syn |
tar.gz of the uploads directory |
Weekly, Sunday 04:00 | 56 days | /volume1/configs/redefined-designs/backups/uploads |
Both destinations are bind mounts that have to exist before the stack starts. Nothing creates them, and nothing complains when they are missing — the container sits in Created, never runs, and so has no logs to explain itself. Creating them is a step in the cutover runbook; on any other deploy, check for them first:
sudo mkdir -p /volume1/configs/redefined-designs/backups/{postgres,uploads}
sudo docker ps --filter name=backup --format '{{.Names}} {{.Status}}'
Both prune automatically. Both report unhealthy when their newest artifact is older than their interval plus grace, so a regime that has quietly stopped shows up in Portainer's stack view beside the app rather than being discovered during a restore.
The database dumper is pinned to prodrigestivill/postgres-backup-local:16, matching the server's major version. pg_dump refuses to dump a server newer than itself, so a floating tag would be a backup that stops working the day Postgres is upgraded — silently, because nothing reads the dumps until they are needed.
What this does not protect against
Three gaps, stated plainly because a backup you are wrong about is worse than one you know the limits of.
The disk. Both artifact directories are on /volume1, the same volume as the live database and uploads. This protects against a bad migration, a dropped table, a bad deploy and a stack deletion. It does not protect against the volume failing. Getting a copy off /volume1 is a Synology-side job — Hyper Backup to another volume, an external disk, or offsite — and until that exists this is a convenience rather than a guarantee.
The stack's own teardown. These services run while the stack runs. Deleting the Portainer stack deletes them along with everything else, so they cannot cover the moment they would be most wanted. The manual pg_dump in README's deploy steps stays for exactly that reason: a routine regime and a snapshot before a risky operation are different jobs, and neither replaces the other.
The gap between the two. The database is dumped daily and uploads weekly, so a restore pairs a database from one moment with images from another. Two mismatches are possible. An image with no row is harmless — an orphaned file nobody references. A row with no image is a broken thumbnail on one recent item, and the original is usually recoverable, because an admin who added an item that recently still has the photograph. Neither is data loss, and paying for a synchronised snapshot to avoid a recoverable broken thumbnail is not worth the complexity.
Checking it is working
# Both should say (healthy). unhealthy means nothing has been written inside
# the window — the regime has stopped, and this is the whole point of the check.
sudo docker ps --filter "name=backup" --format "{{.Names}}\t{{.Status}}"
# What is actually on disk, newest last.
ls -lht /volume1/configs/redefined-designs/backups/postgres/daily | head
ls -lht /volume1/configs/redefined-designs/backups/uploads | head
A dump whose size is wildly different from its predecessors is worth opening. A dump that is a few hundred bytes is a failed dump that exited successfully.
Restoring the database
Read this before running it. --clean --if-exists is baked into the dumps, so restoring drops and recreates every table it touches. Against production that discards whatever is currently there. Take a fresh dump first, whatever state you think the database is in.
1. Practise on a throwaway database first
This is not optional ceremony. An untested backup is a file of unknown validity, and the failure modes — a truncated dump, a dump taken mid-migration — look exactly like a good one until the day it matters.
BACKUP=/volume1/configs/redefined-designs/backups/postgres/daily/redefined-<date>.sql.gz
sudo docker exec -i redefined-designs-db-syn \
psql -U redefined -d postgres -c "CREATE DATABASE restore_test"
gunzip -c "$BACKUP" | sudo docker exec -i redefined-designs-db-syn \
psql -U redefined -d restore_test
Then verify it is a real database rather than an empty one:
# Row counts against the tables that matter. Compare them with production.
sudo docker exec -i redefined-designs-db-syn psql -U redefined -d restore_test -c "
SELECT 'items' AS t, count(*) FROM items
UNION ALL SELECT 'customers', count(*) FROM customers
UNION ALL SELECT 'orders', count(*) FROM orders
UNION ALL SELECT 'checkouts', count(*) FROM checkouts;"
# And that the schema is at the same migration as the code expects.
sudo docker exec -i redefined-designs-db-syn psql -U redefined -d restore_test \
-c "SELECT name FROM pgmigrations ORDER BY id DESC LIMIT 1"
sudo docker exec -i redefined-designs-db-syn \
psql -U redefined -d postgres -c "DROP DATABASE restore_test"
The migration check is the one most easily skipped and the most likely to bite: a dump older than the current code restores a schema the app will fail against on boot. The container runs migrate.js up at start, so it may repair itself — but only forwards, and only if the migrations are still compatible.
2. Restore for real
# Stop the app so nothing writes while the schema is being replaced.
sudo docker stop redefined-designs-syn
gunzip -c "$BACKUP" | sudo docker exec -i redefined-designs-db-syn \
psql -U redefined -d redefined
sudo docker start redefined-designs-syn
# Migrations run at container start, so watch them land before serving.
sudo docker logs -f redefined-designs-syn | head -30
Restoring uploads
The archives are plain tar.gz unless BACKUP_PASSPHRASE is set in the stack.
ARCHIVE=/volume1/configs/redefined-designs/backups/uploads/uploads-<date>.tar.gz
# Look before extracting. The archive contains a `backup/uploads/` prefix.
tar -tzf "$ARCHIVE" | head
# Extract somewhere harmless first, then move what is needed into place.
mkdir -p /volume1/configs/redefined-designs/restore-tmp
tar -xzf "$ARCHIVE" -C /volume1/configs/redefined-designs/restore-tmp
Restore into the live directory by copying rather than replacing it wholesale, unless the intent really is to discard everything added since the archive was taken. Image paths are stored in the database as /uploads/<file>, so a file missing from disk is a broken image on one item rather than an error, which makes a partial restore safe and a wrong wholesale one quietly destructive.
On BACKUP_PASSPHRASE
Leave it unset unless there is a reason. If it is set, the archives are GPG-encrypted, and an archive nobody can decrypt is not a backup. A passphrase kept only in Portainer's stack variables is lost with the NAS — precisely the disaster the backups exist for. Set it only if the passphrase itself lives somewhere that survives the NAS.
Open
- An off-
/volume1copy, by whatever Synology-side mechanism is chosen, recorded here once it exists - A restore performed at least once against
restore_testand the result verified, per the section above
Neither is done. Until the second one is, this document describes an untested procedure.