main
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bermudalamb
|
724e9ce19d |
docs(ops): say that the backup directories have to be created (#192)
The stack gained two backup services in #147 and nothing has ever told anyone to create the directories they mount. `backup-and-restore.md` reads from both paths and the compose file mounts both, but no document creates them — while the README does exactly that for QA's data directories, ownership notes and all. Production's cutover runbook said nothing. Hit for real during the cutover: both backup containers sat in Created, never started, and `docker logs` on them reported only that nothing matched the filter, because a container that never ran has no output. Portainer showed them beside the healthy ones and the stack looked deployed. That silence is the reason this is worth a step of its own rather than a footnote. A backup regime that never started is indistinguishable from a working one until someone needs a restore, which is the failure mode the healthchecks in #147 exist to catch — and those healthchecks cannot fire on a container that is not running. The cutover runbook gains the directory creation before the stack is created, and step 7 now counts containers rather than only checking the app: four, all Up, with Created called out as the thing to look for. Counted from the compose file rather than from memory — the first draft said five. `backup-and-restore.md` gains the same note where it describes the destinations, since anyone reading that page is already thinking about paths. No `chown`, deliberately stated: both backup images run as root, unlike the Postgres image whose data directory needs uid 999, and an unnecessary chown instruction is how people learn to run them without thinking. |
||
|
bermudalamb
|
ebefcbb76b |
feat(ops): schedule database and uploads backups, and document the restore (#147)
The only copy of every customer, order and one-of-a-kind item was the live Postgres data directory, plus whatever the deploy checklist's manual pg_dump happened to have caught. That dump is good and stays, but it only runs when someone deploys: a quiet week meant the newest copy of real customer data was a week old, and nothing bounded the gap. Two services rather than one, because they are different jobs. The database is small, changes constantly and wants a logical dump — daily, gzipped, 7/4/6 daily-weekly-monthly retention. Uploads are large and append-mostly and want an archive — weekly, 56 days, mounted read-only so a backup process cannot damage the thing it is backing up. Forcing both through one tool would serve one of them badly. The dumper is pinned to postgres-backup-local:16 to match the server. pg_dump refuses to dump a server newer than itself, so a floating tag is a backup that stops working the day Postgres is upgraded — silently, because nothing reads a dump until it is needed. It depends_on the database's existing pg_isready healthcheck, which is the constraint that shaped this: a dumper is a client, and without that the first run after a NAS reboot races Postgres coming up. Both carry a staleness healthcheck rather than trusting the schedule. A regime that stopped a month ago is indistinguishable from a working one until a restore is attempted, and `find -mmin` is the cheapest thing that tells them apart. It surfaces in Portainer beside the app rather than somewhere separate to remember to look. Windows are the interval plus grace — 26 hours daily, 9 days weekly — so a late run is not a failure, and start_period covers the first cycle when nothing has been written yet. Verified rather than assumed. Both images were pulled and checked to have a shell and `find`, since a CMD-SHELL healthcheck against an image without one reports unhealthy forever. postgres-backup-local:16 ships pg_dump 16.10 against the postgres:16 server. The healthcheck expression was exercised three ways in the image itself — empty directory, fresh artifact, and one aged three days — and returns unhealthy, healthy, unhealthy. The compose file parses and the #118 drift guard still passes over it. Three things these deliberately do not cover, written into the compose file and the doc rather than left to be discovered: They run while the stack runs, so they cannot protect the stack's own teardown. Deleting the Portainer stack deletes them too. That is why the deploy checklist's manual dump stays, and README now says so where the checklist is. They write to the same volume as the data they protect. That survives a bad migration, a dropped table, a bad deploy and a stack deletion, and not the disk. Getting a copy off /volume1 is a Synology-side job and is what turns this from a convenience into a guarantee. Daily database against weekly uploads leaves a window where a restore pairs the two from different moments. An orphaned image is harmless; a row without its image is a broken thumbnail on one recent item, usually still on the admin's machine. Neither is data loss, and a synchronised snapshot is not worth the complexity to avoid it. The restore procedure leads with practising on a throwaway database, because an untested backup is a file of unknown validity and a truncated dump looks exactly like a good one until it matters. It checks row counts and the pgmigrations head — the migration check being the one most easily skipped and most likely to bite, since a dump older than the code restores a schema the app will fail against. Both open items are listed as unticked in the doc: no off-volume copy exists yet, and no restore has been performed. Until the second is done this documents an untested procedure, and it says so. Refs #147 |