feat(ops): schedule database and uploads backups, and document the restore (#147)
The only copy of every customer, order and one-of-a-kind item was the live Postgres data directory, plus whatever the deploy checklist's manual pg_dump happened to have caught. That dump is good and stays, but it only runs when someone deploys: a quiet week meant the newest copy of real customer data was a week old, and nothing bounded the gap. Two services rather than one, because they are different jobs. The database is small, changes constantly and wants a logical dump — daily, gzipped, 7/4/6 daily-weekly-monthly retention. Uploads are large and append-mostly and want an archive — weekly, 56 days, mounted read-only so a backup process cannot damage the thing it is backing up. Forcing both through one tool would serve one of them badly. The dumper is pinned to postgres-backup-local:16 to match the server. pg_dump refuses to dump a server newer than itself, so a floating tag is a backup that stops working the day Postgres is upgraded — silently, because nothing reads a dump until it is needed. It depends_on the database's existing pg_isready healthcheck, which is the constraint that shaped this: a dumper is a client, and without that the first run after a NAS reboot races Postgres coming up. Both carry a staleness healthcheck rather than trusting the schedule. A regime that stopped a month ago is indistinguishable from a working one until a restore is attempted, and `find -mmin` is the cheapest thing that tells them apart. It surfaces in Portainer beside the app rather than somewhere separate to remember to look. Windows are the interval plus grace — 26 hours daily, 9 days weekly — so a late run is not a failure, and start_period covers the first cycle when nothing has been written yet. Verified rather than assumed. Both images were pulled and checked to have a shell and `find`, since a CMD-SHELL healthcheck against an image without one reports unhealthy forever. postgres-backup-local:16 ships pg_dump 16.10 against the postgres:16 server. The healthcheck expression was exercised three ways in the image itself — empty directory, fresh artifact, and one aged three days — and returns unhealthy, healthy, unhealthy. The compose file parses and the #118 drift guard still passes over it. Three things these deliberately do not cover, written into the compose file and the doc rather than left to be discovered: They run while the stack runs, so they cannot protect the stack's own teardown. Deleting the Portainer stack deletes them too. That is why the deploy checklist's manual dump stays, and README now says so where the checklist is. They write to the same volume as the data they protect. That survives a bad migration, a dropped table, a bad deploy and a stack deletion, and not the disk. Getting a copy off /volume1 is a Synology-side job and is what turns this from a convenience into a guarantee. Daily database against weekly uploads leaves a window where a restore pairs the two from different moments. An orphaned image is harmless; a row without its image is a broken thumbnail on one recent item, usually still on the admin's machine. Neither is data loss, and a synchronised snapshot is not worth the complexity to avoid it. The restore procedure leads with practising on a throwaway database, because an untested backup is a file of unknown validity and a truncated dump looks exactly like a good one until it matters. It checks row counts and the pgmigrations head — the migration check being the one most easily skipped and most likely to bite, since a dump older than the code restores a schema the app will fail against. Both open items are listed as unticked in the doc: no off-volume copy exists yet, and no restore has been performed. Until the second is done this documents an untested procedure, and it says so. Refs #147
This commit is contained in:
@@ -246,6 +246,8 @@ The middleware is attached to each admin **router** rather than to a path prefix
|
||||
|
||||
Only after the change has been reviewed in QA.
|
||||
|
||||
The scheduled backups in `docker-compose.prod.yml` do **not** replace step 1 below. They run while the stack runs, so they cannot cover a deploy that recreates it — and a nightly dump is up to a day old, where this one is seconds old. See [docs/ops/backup-and-restore.md](docs/ops/backup-and-restore.md) for what each covers and how to restore either.
|
||||
|
||||
```bash
|
||||
# 1. Back up first — the container migrates the schema on its own.
|
||||
sudo docker exec -t redefined-designs-db-syn pg_dump -U redefined -d redefined \
|
||||
|
||||
@@ -70,6 +70,10 @@
|
||||
# PAYPAL_CLIENT_ID Live PayPal credentials. Required because DEMO_MODE is
|
||||
# PAYPAL_CLIENT_SECRET false below; the app refuses to start without them.
|
||||
# PAYPAL_WEBHOOK_ID
|
||||
# BACKUP_PASSPHRASE Optional. Set it and the uploads archives are
|
||||
# encrypted at rest; leave it empty and they are not.
|
||||
# See docs/ops/backup-and-restore.md before setting it —
|
||||
# an archive nobody can decrypt is not a backup.
|
||||
# USPS_CLIENT_ID Optional. Leave unset to run without address
|
||||
# USPS_CLIENT_SECRET validation; the app degrades gracefully rather than
|
||||
# failing, so an empty value is a working configuration.
|
||||
@@ -209,3 +213,119 @@ services:
|
||||
options:
|
||||
max-size: 10m
|
||||
max-file: "3"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Backups (#147)
|
||||
#
|
||||
# Two services rather than one, because they are different jobs on different
|
||||
# cadences. The database is small, changes constantly, and wants a logical
|
||||
# dump. Uploads are large, append-mostly, and want an archive. Forcing both
|
||||
# through one tool serves one of them badly.
|
||||
#
|
||||
# WHAT THESE DO NOT COVER, and it matters:
|
||||
#
|
||||
# They run while the stack runs, so they cannot protect the stack's own
|
||||
# teardown. Deleting the Portainer stack deletes these containers along with
|
||||
# everything else. The manual pg_dump in README's deploy steps therefore
|
||||
# stays exactly where it is — a routine regime and a snapshot taken before a
|
||||
# risky operation are different jobs, and neither replaces the other.
|
||||
#
|
||||
# And they write to the same volume as the data they protect. That survives a
|
||||
# bad migration, a dropped table, a bad deploy and a stack deletion. It does
|
||||
# not survive the disk. Getting a copy off /volume1 is a Synology-side job —
|
||||
# Hyper Backup to another volume, an external disk, or offsite — and it is
|
||||
# what turns this from a convenience into a guarantee. See
|
||||
# docs/ops/backup-and-restore.md.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
redefined-designs-db-backup-syn:
|
||||
# Pinned to 16 to match the server. pg_dump refuses to dump a server newer
|
||||
# than itself, so a floating tag here is a backup that stops working on the
|
||||
# day Postgres is upgraded — silently, since nothing reads the dumps until
|
||||
# they are needed.
|
||||
image: prodrigestivill/postgres-backup-local:16
|
||||
container_name: redefined-designs-db-backup-syn
|
||||
environment:
|
||||
- TZ=America/Chicago
|
||||
- POSTGRES_HOST=redefined-designs-db-syn
|
||||
- POSTGRES_PORT=5432
|
||||
- POSTGRES_DB=redefined
|
||||
- POSTGRES_USER=redefined
|
||||
- POSTGRES_PASSWORD=${DB_PASSWORD}
|
||||
# Daily at 03:00. Late enough that a deploy is unlikely to be in flight,
|
||||
# and pg_dump takes a consistent snapshot anyway, so a dump running while
|
||||
# customers are shopping is fine.
|
||||
- SCHEDULE=@daily
|
||||
- BACKUP_KEEP_DAYS=7
|
||||
- BACKUP_KEEP_WEEKS=4
|
||||
- BACKUP_KEEP_MONTHS=6
|
||||
# --clean --if-exists so the dump can be restored over an existing
|
||||
# database without hand-dropping it first, which is the state a real
|
||||
# restore happens in.
|
||||
- POSTGRES_EXTRA_OPTS=--clean --if-exists
|
||||
volumes:
|
||||
- /volume1/configs/redefined-designs/backups/postgres:/backups
|
||||
depends_on:
|
||||
redefined-designs-db-syn:
|
||||
# The dumper is a client and needs a server accepting connections. This
|
||||
# is the constraint that shaped the design: without it the first run
|
||||
# after a NAS reboot races Postgres coming up.
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
# Unhealthy when nothing has been written inside the window. A backup
|
||||
# regime that stopped a month ago is indistinguishable from a working one
|
||||
# until a restore is attempted, and this is the cheapest thing that tells
|
||||
# them apart. It shows in Portainer beside the app rather than somewhere
|
||||
# separate to remember to look.
|
||||
#
|
||||
# 1560 minutes is 26 hours: the daily interval plus two hours of grace, so
|
||||
# a dump that runs a little late is not reported as a failure.
|
||||
test: ["CMD-SHELL", "find /backups -name '*.sql.gz' -mmin -1560 | grep -q ."]
|
||||
interval: 1h
|
||||
timeout: 30s
|
||||
retries: 3
|
||||
# Nothing exists until the first scheduled run, so without this the
|
||||
# container reports unhealthy for its first day on every fresh deploy.
|
||||
start_period: 25h
|
||||
restart: unless-stopped
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: 10m
|
||||
max-file: "3"
|
||||
|
||||
redefined-designs-uploads-backup-syn:
|
||||
image: offen/docker-volume-backup:v2
|
||||
container_name: redefined-designs-uploads-backup-syn
|
||||
environment:
|
||||
- TZ=America/Chicago
|
||||
# Weekly, not daily. Uploads are append-mostly and much larger than the
|
||||
# database, so a daily full archive would mostly be copies of itself.
|
||||
- BACKUP_CRON_EXPRESSION=0 4 * * 0
|
||||
- BACKUP_FILENAME=uploads-%Y-%m-%dT%H-%M-%S.tar.gz
|
||||
- BACKUP_ARCHIVE=/archive
|
||||
# Eight weeks. Shorter than the database's tail because each archive is
|
||||
# far bigger, and an image that was deleted two months ago is not
|
||||
# something anyone is restoring.
|
||||
- BACKUP_RETENTION_DAYS=56
|
||||
- BACKUP_PRUNING_PREFIX=uploads-
|
||||
# Optional. An empty value means no encryption, which is the default.
|
||||
- GPG_PASSPHRASE=${BACKUP_PASSPHRASE}
|
||||
volumes:
|
||||
# Read-only. A backup process with write access to the thing it is backing
|
||||
# up is a way to lose both at once.
|
||||
- /volume1/configs/redefined-designs/uploads:/backup/uploads:ro
|
||||
- /volume1/configs/redefined-designs/backups/uploads:/archive
|
||||
healthcheck:
|
||||
# Nine days: the weekly interval plus two days of grace.
|
||||
test: ["CMD-SHELL", "find /archive -name 'uploads-*' -mmin -12960 | grep -q ."]
|
||||
interval: 6h
|
||||
timeout: 30s
|
||||
retries: 3
|
||||
start_period: 8d
|
||||
restart: unless-stopped
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: 10m
|
||||
max-file: "3"
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
# Backup and restore
|
||||
|
||||
What protects production's data, what each part does and does not cover, and how to get it back. See #147.
|
||||
|
||||
## What runs
|
||||
|
||||
Two services in `docker-compose.prod.yml`, alongside the app and the database.
|
||||
|
||||
| Service | What | When | Kept | Lands in |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `redefined-designs-db-backup-syn` | `pg_dump` of `redefined`, gzipped | Daily, 03:00 | 7 daily, 4 weekly, 6 monthly | `/volume1/configs/redefined-designs/backups/postgres` |
|
||||
| `redefined-designs-uploads-backup-syn` | `tar.gz` of the uploads directory | Weekly, Sunday 04:00 | 56 days | `/volume1/configs/redefined-designs/backups/uploads` |
|
||||
|
||||
Both prune automatically. Both report **unhealthy** when their newest artifact is older than their interval plus grace, so a regime that has quietly stopped shows up in Portainer's stack view beside the app rather than being discovered during a restore.
|
||||
|
||||
The database dumper is pinned to `prodrigestivill/postgres-backup-local:16`, matching the server's major version. `pg_dump` refuses to dump a server newer than itself, so a floating tag would be a backup that stops working the day Postgres is upgraded — silently, because nothing reads the dumps until they are needed.
|
||||
|
||||
## What this does not protect against
|
||||
|
||||
Three gaps, stated plainly because a backup you are wrong about is worse than one you know the limits of.
|
||||
|
||||
**The disk.** Both artifact directories are on `/volume1`, the same volume as the live database and uploads. This protects against a bad migration, a dropped table, a bad deploy and a stack deletion. It does not protect against the volume failing. **Getting a copy off `/volume1` is a Synology-side job** — Hyper Backup to another volume, an external disk, or offsite — and until that exists this is a convenience rather than a guarantee.
|
||||
|
||||
**The stack's own teardown.** These services run while the stack runs. Deleting the Portainer stack deletes them along with everything else, so they cannot cover the moment they would be most wanted. The manual `pg_dump` in README's deploy steps stays for exactly that reason: a routine regime and a snapshot before a risky operation are different jobs, and neither replaces the other.
|
||||
|
||||
**The gap between the two.** The database is dumped daily and uploads weekly, so a restore pairs a database from one moment with images from another. Two mismatches are possible. An image with no row is harmless — an orphaned file nobody references. A row with no image is a broken thumbnail on one recent item, and the original is usually recoverable, because an admin who added an item that recently still has the photograph. Neither is data loss, and paying for a synchronised snapshot to avoid a recoverable broken thumbnail is not worth the complexity.
|
||||
|
||||
## Checking it is working
|
||||
|
||||
```bash
|
||||
# Both should say (healthy). unhealthy means nothing has been written inside
|
||||
# the window — the regime has stopped, and this is the whole point of the check.
|
||||
sudo docker ps --filter "name=backup" --format "{{.Names}}\t{{.Status}}"
|
||||
|
||||
# What is actually on disk, newest last.
|
||||
ls -lht /volume1/configs/redefined-designs/backups/postgres/daily | head
|
||||
ls -lht /volume1/configs/redefined-designs/backups/uploads | head
|
||||
```
|
||||
|
||||
A dump whose size is wildly different from its predecessors is worth opening. A dump that is a few hundred bytes is a failed dump that exited successfully.
|
||||
|
||||
## Restoring the database
|
||||
|
||||
**Read this before running it.** `--clean --if-exists` is baked into the dumps, so restoring **drops and recreates every table it touches**. Against production that discards whatever is currently there. Take a fresh dump first, whatever state you think the database is in.
|
||||
|
||||
### 1. Practise on a throwaway database first
|
||||
|
||||
This is not optional ceremony. An untested backup is a file of unknown validity, and the failure modes — a truncated dump, a dump taken mid-migration — look exactly like a good one until the day it matters.
|
||||
|
||||
```bash
|
||||
BACKUP=/volume1/configs/redefined-designs/backups/postgres/daily/redefined-<date>.sql.gz
|
||||
|
||||
sudo docker exec -i redefined-designs-db-syn \
|
||||
psql -U redefined -d postgres -c "CREATE DATABASE restore_test"
|
||||
|
||||
gunzip -c "$BACKUP" | sudo docker exec -i redefined-designs-db-syn \
|
||||
psql -U redefined -d restore_test
|
||||
```
|
||||
|
||||
Then verify it is a real database rather than an empty one:
|
||||
|
||||
```bash
|
||||
# Row counts against the tables that matter. Compare them with production.
|
||||
sudo docker exec -i redefined-designs-db-syn psql -U redefined -d restore_test -c "
|
||||
SELECT 'items' AS t, count(*) FROM items
|
||||
UNION ALL SELECT 'customers', count(*) FROM customers
|
||||
UNION ALL SELECT 'orders', count(*) FROM orders
|
||||
UNION ALL SELECT 'checkouts', count(*) FROM checkouts;"
|
||||
|
||||
# And that the schema is at the same migration as the code expects.
|
||||
sudo docker exec -i redefined-designs-db-syn psql -U redefined -d restore_test \
|
||||
-c "SELECT name FROM pgmigrations ORDER BY id DESC LIMIT 1"
|
||||
|
||||
sudo docker exec -i redefined-designs-db-syn \
|
||||
psql -U redefined -d postgres -c "DROP DATABASE restore_test"
|
||||
```
|
||||
|
||||
The migration check is the one most easily skipped and the most likely to bite: a dump older than the current code restores a schema the app will fail against on boot. The container runs `migrate.js up` at start, so it may repair itself — but only forwards, and only if the migrations are still compatible.
|
||||
|
||||
### 2. Restore for real
|
||||
|
||||
```bash
|
||||
# Stop the app so nothing writes while the schema is being replaced.
|
||||
sudo docker stop redefined-designs-syn
|
||||
|
||||
gunzip -c "$BACKUP" | sudo docker exec -i redefined-designs-db-syn \
|
||||
psql -U redefined -d redefined
|
||||
|
||||
sudo docker start redefined-designs-syn
|
||||
|
||||
# Migrations run at container start, so watch them land before serving.
|
||||
sudo docker logs -f redefined-designs-syn | head -30
|
||||
```
|
||||
|
||||
## Restoring uploads
|
||||
|
||||
The archives are plain `tar.gz` unless `BACKUP_PASSPHRASE` is set in the stack.
|
||||
|
||||
```bash
|
||||
ARCHIVE=/volume1/configs/redefined-designs/backups/uploads/uploads-<date>.tar.gz
|
||||
|
||||
# Look before extracting. The archive contains a `backup/uploads/` prefix.
|
||||
tar -tzf "$ARCHIVE" | head
|
||||
|
||||
# Extract somewhere harmless first, then move what is needed into place.
|
||||
mkdir -p /volume1/configs/redefined-designs/restore-tmp
|
||||
tar -xzf "$ARCHIVE" -C /volume1/configs/redefined-designs/restore-tmp
|
||||
```
|
||||
|
||||
Restore into the live directory by copying rather than replacing it wholesale, unless the intent really is to discard everything added since the archive was taken. Image paths are stored in the database as `/uploads/<file>`, so a file missing from disk is a broken image on one item rather than an error, which makes a partial restore safe and a wrong wholesale one quietly destructive.
|
||||
|
||||
### On `BACKUP_PASSPHRASE`
|
||||
|
||||
Leave it unset unless there is a reason. If it is set, the archives are GPG-encrypted, and **an archive nobody can decrypt is not a backup**. A passphrase kept only in Portainer's stack variables is lost with the NAS — precisely the disaster the backups exist for. Set it only if the passphrase itself lives somewhere that survives the NAS.
|
||||
|
||||
## Open
|
||||
|
||||
- [ ] An off-`/volume1` copy, by whatever Synology-side mechanism is chosen, recorded here once it exists
|
||||
- [ ] A restore performed at least once against `restore_test` and the result verified, per the section above
|
||||
|
||||
Neither is done. Until the second one is, this document describes an untested procedure.
|
||||
Reference in New Issue
Block a user