c704c07b89d8deca42fc47a780032b911a97ded5
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  bermudalambandClaude Opus 5
|
70d04186b1 |
docs(ops): stop the compose header contradicting itself, and name the four variables step 2 dropped (#204)
Three corrections a review of #196 found, all in text #196 itself rewrote. **The compose header asserted something it falsified three lines later.** "All must be set in Portainer for this stack. All are secrets except DEMO_MODE" — but three PayPal entries are unused while `DEMO_MODE` is `true`, three more are marked Optional, and `SMTP_FROM` is not a secret. The runbook sends the operator to that exact block as authoritative, so an operator cutting over during the demo interim reads "all must be set", has no live PayPal credentials — which is the whole reason the interim exists — and either stops or invents a `BACKUP_PASSPHRASE`, which is how you get archives nobody can decrypt. The blanket claim is gone; each entry already says whether it is required and when. **Step 2 enumerated nine of the thirteen interpolated names.** `USPS_CLIENT_ID`, `USPS_CLIENT_SECRET`, `UPLOADS_BASE_URL` and `BACKUP_PASSPHRASE` were missing. The USPS pair is the one that matters, and it now gets a sentence of its own: losing it is the only silent failure in this step. Address validation is skipped when those are empty rather than failing, so checkout keeps working and quietly stops validating addresses, with no crash loop and nothing in step 7 that would notice. The list also now says to take everything the stack holds rather than working from the list, because an enumeration reads as a checklist however it is introduced. **The `DB_PASSWORD` failure was described wrongly, and its error names a variable the operator never typed.** It does not fail to authenticate against its data directory — the app never reaches a connection attempt, `checkAlwaysRequired` refuses at boot, and the message says `PGPASSWORD` because the compose file injects it as `PGPASSWORD=${DB_PASSWORD}`. An operator grepping for `DB_PASSWORD` finds nothing. That is now stated. The crash-loop examples were reordered to match the case the section claims to be about. It headlines "the likeliest outcome of a missed step 2", but the only block shown was the PayPal triple, which cannot occur during the demo interim, and #196 replaced it with a `DEMO_MODE`-only block that is not what a missed step 2 produces either. A wholesale miss is two problems led by `PGPASSWORD`; that is now first, with the `DEMO_MODE`-only and mistyped-value forms after it. Every quoted line was verified by running the real validator against production's entry set rather than by reading the source — `2 problem(s)` and `1 problem(s)` counts included — and all four now match character for character. The mistyped-value message is quoted in full rather than truncated, which was the point of the complaint that produced it. `DEMO_MODE` is no longer called "the only one that is not a secret", which was false of `SMTP_FROM` and `UPLOADS_BASE_URL`. It is the only one that is a setting rather than a credential, which is true and a better hook. Verified: 278 backend unit tests pass, including the compose guard that parses this file. Closes #204 Refs #196 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
bermudalambandClaude Opus 5
|
2fe3aa055f |
docs(ops): make the cutover runbook agree with the compose file about DEMO_MODE (#196)
#190 turned `DEMO_MODE` from a hardcoded compose value into a Portainer stack variable with no default, and updated the compose header. The runbook that actually creates the stack was not updated with it, so the document and the file it deploys have been disagreeing since — in the three places most likely to be read under pressure. Step 2's list of stack variables to record did not include `DEMO_MODE`, and step 6 says to add the variables from steps 2 and 3. An operator following this literally creates a stack that cannot boot. It is now in the list, with a paragraph of its own: it is the only one of the nine that is not a secret, which is exactly why it is the easy one to skip past. The troubleshooting section said `DEMO_MODE` was hardcoded and therefore could not be missing, so its absence from the error list proved nothing. That was the most dangerous sentence in the file — an unset `DEMO_MODE` is now the *first* thing to check rather than something to rule out. The line now says it moved and why. The crash-loop example was the PayPal triple, which cannot occur while `DEMO_MODE` is `true`. The message an operator will actually see during the demo interim — `DEMO_MODE is required and must be exactly 'true' or 'false'` — appeared nowhere in the runbook. Both forms are shown now, in the order they are likely to be hit. Added what neither document said: Compose only *warns* about an unset variable and deploys anyway. In Portainer's stack UI that warning is easy to miss, and the container then crash-loops under `restart: unless-stopped` — loud in the log, invisible in a glance at the stack list. The container log is the signal, not the deploy output. The compose header carried two claims that #190 falsified and did not correct: that the PayPal secrets are required "because DEMO_MODE is false below", and that only secrets are interpolated. Both now describe the file as it is. This is the drift `composeEnvironment.test.ts` exists to prevent, surfacing in the one place no test can reach — the guard keeps the compose file honest about its own intent and cannot see the runbook beside it. Verified: 278 backend unit tests pass, including the compose guard that parses this file, and a sweep for the stale claims finds none left. Closes #196 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
bermudalamb
|
1a59edec18 |
docs(ops): prove the backups during the cutover rather than waiting for a schedule (#192)
Step 4b created the directories and step 7 checked the containers were Up. Neither established that a restorable file actually gets written, and those are not the same claim — the database backup does not run until 03:00 and the uploads archive not until Sunday 04:00, so a stack that looks correct at the end of a cutover can be four days from its first evidence. Both tools take a manual trigger, so the wait is unnecessary. The runbook now forces one run of each, checks the sizes are plausible, and greps the dump for `COPY` lines on the real tables — a dump of an empty database succeeds and looks fine, which is the one way this check could otherwise lie. It also confirms the healthchecks agree with where the files landed. A check whose `find` path disagrees with where the tool actually writes reports unhealthy forever while the backups are working perfectly, and that is a thing to discover on the day the stack is built rather than a year later. `starting` corrected to `unhealthy` in the surrounding text: during `start_period` Docker reports `starting`, which is what an operator actually sees and what the previous wording got wrong. Proven against production during the cutover on 2026-08-26 — a 41K dump and a 15M archive, both landing where the healthchecks look. |
||
|
bermudalamb
|
724e9ce19d |
docs(ops): say that the backup directories have to be created (#192)
The stack gained two backup services in #147 and nothing has ever told anyone to create the directories they mount. `backup-and-restore.md` reads from both paths and the compose file mounts both, but no document creates them — while the README does exactly that for QA's data directories, ownership notes and all. Production's cutover runbook said nothing. Hit for real during the cutover: both backup containers sat in Created, never started, and `docker logs` on them reported only that nothing matched the filter, because a container that never ran has no output. Portainer showed them beside the healthy ones and the stack looked deployed. That silence is the reason this is worth a step of its own rather than a footnote. A backup regime that never started is indistinguishable from a working one until someone needs a restore, which is the failure mode the healthchecks in #147 exist to catch — and those healthchecks cannot fire on a container that is not running. The cutover runbook gains the directory creation before the stack is created, and step 7 now counts containers rather than only checking the app: four, all Up, with Created called out as the thing to look for. Counted from the compose file rather than from memory — the first draft said five. `backup-and-restore.md` gains the same note where it describes the destinations, since anyone reading that page is already thinking about paths. No `chown`, deliberately stated: both backup images run as root, unlike the Postgres image whose data directory needs uid 999, and an unnecessary chown instruction is how people learn to run them without thinking. |
||
|
bermudalamb
|
0cec9b3c1f |
docs(ops): name the crash loop the cutover runbook was most likely to cause (#175)
Step 7 listed a missing `ADMIN_GATE_SECRET` — a warning the container starts through — and said nothing about the three PayPal secrets, which are a hard error that crash-loops it. That is backwards: the secrets are the likeliest thing to be missing after a stack replacement, because Portainer stack variables belong to the stack and are discarded with it, and `DEMO_MODE` is false in production so the app refuses to start without them. Hit for real following the runbook. The verify section now shows the actual log block, says what it means, and gives the way to tell a missing variable from a misnamed one: a hardcoded value cannot be missing, so its absence from the error list proves nothing, while an interpolated variable that stays quiet while others complain proves substitution works and those others are simply unset. That is the reading that turns the log into a diagnosis instead of a list. It also says the loop is harmless while the values are fetched — the container refuses before it serves anything or touches data — and where the values live if the old stack is already gone, since the webhook id in particular is readable rather than only recreatable. Step 2 now names every interpolated variable rather than describing them in general, and states the consequence of each class going missing. A general instruction to record the environment is easy to read as already done. |
||
|
bermudalamb
|
cf9003bc2d |
docs(ops): write down how to cut production over to the committed compose file (#175)
#118 put `docker-compose.prod.yml` in the repository and argued at length for why it belongs there. It never produced the procedure for getting production from where it is — a stack living in Portainer's web editor — to where that file expects it to be. What existed was scattered and none of it was a procedure. The compose header described the destination: name the stack this, deploy it as a git repository stack with these settings, rename any variables whose names differ. The README's deployment section is a good runbook for the routine redeploy, and assumes the stack is already in the right form. So the riskiest deployment operation this project has was the one with no steps written down. `docs/ops/production-stack-cutover.md` is those steps, in the order that matters, with the two facts that decide whether the operation is safe stated up front rather than left to be inferred from the volumes block. The first is that all persistent data is bind-mounted from the NAS filesystem rather than held in Docker-managed volumes, so deleting the stack cannot lose the database or the product images. That is what makes the cutover recoverable at all, and step 2 verifies it rather than trusting it — the `docker inspect` there discriminates `bind` from `volume` and says to stop on `volume`, because this runbook does not cover that case. The second is that both services set an explicit `container_name`, so the old containers must be gone before the new stack starts or the deploy fails on a collision that reads like a Portainer bug rather than a sequencing mistake. It also captures what is simply lost if not recorded first. Portainer stack variables belong to the stack and are discarded with it; they are all secrets, and a stack brought back up with a different `DB_PASSWORD` than the data directory was initialised with cannot authenticate against its own database. Recording them is step 2 and it is what makes the rollback credible. The verification section separates the boot warnings that are correct in production — no mail allowlist, no uploads origin yet — from the one that means a variable never arrived. That distinction is the whole failure mode #118 was about: a stack variable whose name matches nothing in the file is substituted nowhere and never reaches the container, and the failure reads as "I set it and it says it is not set". Every diagnostic command in it was run rather than written from memory. The mount inspection was checked against a container that genuinely uses a named volume, to confirm it reports the difference the step depends on. The compose header now points at the runbook for the procedure and keeps the rationale that belongs at the point of use — why there is no `build:`, why most values are hardcoded, why `NODE_ENV` is absent. The README says plainly that its section is the routine deploy and links to the other one. Closes #175 |
||
|
bermudalamb
|
ebefcbb76b |
feat(ops): schedule database and uploads backups, and document the restore (#147)
The only copy of every customer, order and one-of-a-kind item was the live Postgres data directory, plus whatever the deploy checklist's manual pg_dump happened to have caught. That dump is good and stays, but it only runs when someone deploys: a quiet week meant the newest copy of real customer data was a week old, and nothing bounded the gap. Two services rather than one, because they are different jobs. The database is small, changes constantly and wants a logical dump — daily, gzipped, 7/4/6 daily-weekly-monthly retention. Uploads are large and append-mostly and want an archive — weekly, 56 days, mounted read-only so a backup process cannot damage the thing it is backing up. Forcing both through one tool would serve one of them badly. The dumper is pinned to postgres-backup-local:16 to match the server. pg_dump refuses to dump a server newer than itself, so a floating tag is a backup that stops working the day Postgres is upgraded — silently, because nothing reads a dump until it is needed. It depends_on the database's existing pg_isready healthcheck, which is the constraint that shaped this: a dumper is a client, and without that the first run after a NAS reboot races Postgres coming up. Both carry a staleness healthcheck rather than trusting the schedule. A regime that stopped a month ago is indistinguishable from a working one until a restore is attempted, and `find -mmin` is the cheapest thing that tells them apart. It surfaces in Portainer beside the app rather than somewhere separate to remember to look. Windows are the interval plus grace — 26 hours daily, 9 days weekly — so a late run is not a failure, and start_period covers the first cycle when nothing has been written yet. Verified rather than assumed. Both images were pulled and checked to have a shell and `find`, since a CMD-SHELL healthcheck against an image without one reports unhealthy forever. postgres-backup-local:16 ships pg_dump 16.10 against the postgres:16 server. The healthcheck expression was exercised three ways in the image itself — empty directory, fresh artifact, and one aged three days — and returns unhealthy, healthy, unhealthy. The compose file parses and the #118 drift guard still passes over it. Three things these deliberately do not cover, written into the compose file and the doc rather than left to be discovered: They run while the stack runs, so they cannot protect the stack's own teardown. Deleting the Portainer stack deletes them too. That is why the deploy checklist's manual dump stays, and README now says so where the checklist is. They write to the same volume as the data they protect. That survives a bad migration, a dropped table, a bad deploy and a stack deletion, and not the disk. Getting a copy off /volume1 is a Synology-side job and is what turns this from a convenience into a guarantee. Daily database against weekly uploads leaves a window where a restore pairs the two from different moments. An orphaned image is harmless; a row without its image is a broken thumbnail on one recent item, usually still on the admin's machine. Neither is data loss, and a synchronised snapshot is not worth the complexity to avoid it. The restore procedure leads with practising on a throwaway database, because an untested backup is a file of unknown validity and a truncated dump looks exactly like a good one until it matters. It checks row counts and the pgmigrations head — the migration check being the one most easily skipped and most likely to bite, since a dump older than the code restores a schema the app will fail against. Both open items are listed as unticked in the doc: no off-volume copy exists yet, and no restore has been performed. Until the second is done this documents an untested procedure, and it says so. Refs #147 |