ebefcbb76bc9e04dfd8d47cb65efeaef3f97d126
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bermudalamb
|
ebefcbb76b |
feat(ops): schedule database and uploads backups, and document the restore (#147)
The only copy of every customer, order and one-of-a-kind item was the live Postgres data directory, plus whatever the deploy checklist's manual pg_dump happened to have caught. That dump is good and stays, but it only runs when someone deploys: a quiet week meant the newest copy of real customer data was a week old, and nothing bounded the gap. Two services rather than one, because they are different jobs. The database is small, changes constantly and wants a logical dump — daily, gzipped, 7/4/6 daily-weekly-monthly retention. Uploads are large and append-mostly and want an archive — weekly, 56 days, mounted read-only so a backup process cannot damage the thing it is backing up. Forcing both through one tool would serve one of them badly. The dumper is pinned to postgres-backup-local:16 to match the server. pg_dump refuses to dump a server newer than itself, so a floating tag is a backup that stops working the day Postgres is upgraded — silently, because nothing reads a dump until it is needed. It depends_on the database's existing pg_isready healthcheck, which is the constraint that shaped this: a dumper is a client, and without that the first run after a NAS reboot races Postgres coming up. Both carry a staleness healthcheck rather than trusting the schedule. A regime that stopped a month ago is indistinguishable from a working one until a restore is attempted, and `find -mmin` is the cheapest thing that tells them apart. It surfaces in Portainer beside the app rather than somewhere separate to remember to look. Windows are the interval plus grace — 26 hours daily, 9 days weekly — so a late run is not a failure, and start_period covers the first cycle when nothing has been written yet. Verified rather than assumed. Both images were pulled and checked to have a shell and `find`, since a CMD-SHELL healthcheck against an image without one reports unhealthy forever. postgres-backup-local:16 ships pg_dump 16.10 against the postgres:16 server. The healthcheck expression was exercised three ways in the image itself — empty directory, fresh artifact, and one aged three days — and returns unhealthy, healthy, unhealthy. The compose file parses and the #118 drift guard still passes over it. Three things these deliberately do not cover, written into the compose file and the doc rather than left to be discovered: They run while the stack runs, so they cannot protect the stack's own teardown. Deleting the Portainer stack deletes them too. That is why the deploy checklist's manual dump stays, and README now says so where the checklist is. They write to the same volume as the data they protect. That survives a bad migration, a dropped table, a bad deploy and a stack deletion, and not the disk. Getting a copy off /volume1 is a Synology-side job and is what turns this from a convenience into a guarantee. Daily database against weekly uploads leaves a window where a restore pairs the two from different moments. An orphaned image is harmless; a row without its image is a broken thumbnail on one recent item, usually still on the admin's machine. Neither is data loss, and a synchronised snapshot is not worth the complexity to avoid it. The restore procedure leads with practising on a throwaway database, because an untested backup is a file of unknown validity and a truncated dump looks exactly like a good one until it matters. It checks row counts and the pgmigrations head — the migration check being the one most easily skipped and most likely to bite, since a dump older than the code restores a schema the app will fail against. Both open items are listed as unticked in the doc: no off-volume copy exists yet, and no restore has been performed. Until the second is done this documents an untested procedure, and it says so. Refs #147 |
||
|
bermudalamb
|
e48c7f585b |
fix(deploy): run the image QA reviewed rather than rebuilding production (#146)
The production compose committed in #145 carried `build:` and `pull_policy: build`, copied from the QA stack without thinking about what they mean there. QA builds from this repository because QA is where a change is first assembled and reviewed. Production is not that; production is where the reviewed thing runs. Worse, it contradicted the deploy this repository already documents. README's production steps promote the exact image QA reviewed — `docker tag redefined-designs:qa redefined-designs:latest` — with the stated reason that what ships is what was tested. A compose file that rebuilds instead quietly overrode that, and the two would have disagreed at the moment it mattered. Rebuilding would be a defensible shortcut if a rebuild of the same commit produced the same image. It does not. The Dockerfile copies package.json without package-lock.json and installs with `npm install`, so both lockfiles in this repository are ignored in every build stage and every dependency range is resolved afresh. Two builds of one commit, minutes apart, can differ in any transitive dependency that published in between. "Same git ref" is therefore not "same image", and the reviewed bytes are the only thing that is. So the service now declares `image: redefined-designs:latest` and nothing else. The image has to exist before the stack starts; a first deploy or a pruned NAS fails with "image not found" rather than silently building something new. That is the intended behaviour and is written into the file rather than left to be discovered. The header records what removing `pull_policy: build` costs, because it is not free. That option exists in QA to stop a redeploy reusing a stale tag and appearing to succeed while running old code. Production reintroduces the same risk by a different route — a redeploy that reuses the previous `latest` because nobody re-tagged — so the promotion is load-bearing rather than a convenience, and the deploy has to say which image it is promoting. Also corrects the README's claim that production runs from a stack outside this repository and so cannot be checked. That was true when it was written and stopped being true in #118; leaving it would have taught the next reader that the environment which just failed to boot is the one nothing watches. The remaining half of #146 — copying the lockfiles and installing with `npm ci` so any rebuild means something — is not in this commit. It changes what CI and QA build as well as production, and belongs with its own verification that an unchanged commit produces an unchanged dependency tree. Refs #146 |
||
|
bermudalamb
|
9ac2fa3ba1 |
feat(scripts): switch Node automatically, and add a test runner (#140)
start-local.ps1 knew exactly what was wrong when Node was too old and then made you fix it by hand. Assert-NodeVersion read node --version, found a major below 20, and threw a message telling you to run `nvm use 24.13.1` and start again in a new shell. A good error for a problem the script could simply solve — and since nvm's default here is 18.16.1, it was hit on every fresh shell. It now runs `nvm use latest` itself and -Stop puts the machine back to 18.16.1. The revert also runs when a start fails partway: without that, a run dying in migrations leaves the machine switched with nothing started, and the -Stop that would restore it is never reached. nvm rewrites a machine-global symlink rather than changing one shell, so this changes the Node version for every terminal on the machine while a script runs. That is the intent — the point is to work in whatever shell is already open — but it is announced every time rather than done quietly. The switch is verified rather than trusted. nvm-windows exits 0 for switches that did not happen: a version it cannot find, a symlink it cannot rewrite without elevation, and — observed here — a rewrite issued immediately after another one, where the directory symlink is briefly still the old target. That last case turned up while testing this change: `nvm use latest` reported success and left Node on 18.16.1. So the result is read back and retried once, and nvm's own output is captured rather than discarded, because suppressing it hid the only message that explained the failure. run-tests.ps1 runs the suites: -Suite unit|integration|e2e|all. One script with a parameter rather than three, because the version switch, the database bring-up and the TEST_PGPORT handling are shared and three copies would drift. The integration suite gets its own throwaway Postgres started and stopped around it, in a finally so a failing suite still tidies up. The e2e suite checks the backend is answering first and says what to start, rather than leaving twenty-five specs to fail on a refused connection that names nothing. `all` runs cheapest and most isolated first, so a break several suites would show is reported by the one that localises it best. The version switching lives in scripts/NodeVersion.ps1, dot-sourced by both, since two copies of it would drift and the half that drifts is the half nobody runs. Closes #140 |
||
|
bermudalamb
|
a2500a9901 |
test(backend): fail the build when the compose file lacks a required variable (#107)
The one-line compose fix in the previous commit unblocks QA. This is the part that stops it happening again, and it is the more useful half. The failure was not really a missing variable. It was that nothing connected two files: envValidation.ts gained a required variable, docker-compose.qa.yml did not set it, and nothing noticed until a container refused to boot on deploy. CI passed the whole time, because CI supplies its own environment and never reads the compose file — which is exactly why "CI is green" was the wrong evidence to have offered. So a unit test now reads the compose file and asserts it sets everything the validator demands. It imports ALWAYS_REQUIRED rather than restating it, which is the only version of this test worth having: a copied list would pass forever while the next variable added to the validator went unguarded in precisely the same way. Two further assertions earn their place. UPLOADS_DIR is hardcoded rather than taken from a stack variable on the grounds that it must agree with the volume mapping, so the test checks it against the mount rather than leaving that a claim in a comment. And ADMIN_GATE_SECRET must be present as an interpolation rather than a literal, since a secret in the repository would defeat the point of having one. There is also a test guarding the test: a regex that matched nothing would make every other assertion in the file vacuously true, so one case asserts that parsing found entries at all. Fired deliberately rather than assumed. Removing the UPLOADS_DIR line reproduces the original failure as two failing tests; restoring it returns to ten passing. A guard that has only ever been observed passing is not known to guard anything. What this cannot do is check production, which runs from a Portainer stack outside this repository. That gap is now written into the README beside the validation rules, along with the reason a variable set only in Portainer's stack UI never reaches the container: stack variables are interpolated into the compose file, not handed to the service. 172 unit tests pass, lint unchanged at 4 warnings. Refs #107 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
bermudalamb
|
9c9e9c3ded |
feat(backend): check the environment at boot instead of discovering it later (#64)
The backend reads environment variables in a couple of dozen places and validated none of them. A missing or misspelled one was undefined until the first line of code that happened to need it, which could be a long time after the container reported healthy — and several of those failures are silent and customer-visible. DEMO_MODE is the one that mattered most. It was read as "demo unless the value is exactly the string false", so DEMO_MODE=False, DEMO_MODE=0, or any typo meant demo mode stayed on and the shop quietly stopped charging anyone. It is now required and strict: exactly 'true' or 'false', and anything else refuses to start while quoting the value it was given, so the typo is visible in the message rather than inferred. Two requirements are conditional, and that is what makes them expressible at all. PayPal credentials are demanded only when DEMO_MODE=false, because QA runs with none of them on purpose and an unconditional rule would be simply wrong there. PUBLIC_URL is demanded only when SMTP is configured, because its only job is building links in email — an environment that cannot send mail does not need it, and requiring it everywhere would break every existing local setup to prevent nothing. UPLOADS_DIR gets no such reprieve: its fallback is correct inside the container and wrong everywhere else, so inheriting it writes uploads somewhere nobody is looking. Every problem is reported at once rather than one per restart, and the process then exits — the same shape as the container refusing to start on a failed migration rather than serving against a schema it does not match. Warnings are printed but do not stop anything: SMTP absent, the admin gate inactive, or an allowlist missing while mail can be sent. That last one is new and earns its place, since SMTP with no allowlist means the environment can reach real customers, which is what #87 exists to prevent. The admin-gate warning moved here from server.ts, so one place says what this container is and is not configured to do. validateEnv is a pure function of the environment handed to it rather than a reader of process.env, so it is tested exhaustively without booting anything or mutating global state. It is called from server.ts and deliberately not from app.ts: the integration suite imports app directly and would otherwise become a configuration exercise. Its rules are one small function each at module level, because cognitive complexity counts everything declared inside a function and the first version scored 24 against a limit of 15. Verified as a real process, not only in tests. A missing DEMO_MODE, a DEMO_MODE of 'False', real payments with no PayPal credentials, and half-configured SMTP each exit 1 with the problems listed; a valid environment starts and serves. Note the exit codes were checked without a pipe, because $? after `| head` reports head rather than node and had first suggested a clean exit. 141 unit, 169 integration and 94 end-to-end passing, lint unchanged at 0 errors and 8 warnings. Both CI workflows already set all six always-required variables plus DEMO_MODE, so the pipeline is unaffected. Refs #64 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
bermudalamb
|
89fc7c5c1b |
feat(backend): add an application-layer gate to the admin API (#63)
Authorization for the admin panel and the admin API has lived entirely in one auth_request regex in an Nginx Proxy Manager config outside this repository. That control is real and it works — nothing is publicly exposed today — but it is invisible from the code, untested here, and not reviewed when this code changes. Three things follow from that, and the first is the one worth the change. An admin route added at a path the regex does not match is unprotected the moment it is written, and nothing in Express indicates that. Anything reaching the published container port directly bypasses authentik entirely. And locally there is no gate at all, so no developer ever sees the boundary being enforced. requireAdminGate is attached to each admin router rather than to a path prefix, which is what makes it useful rather than merely redundant with the proxy. An admin router added later at some other path inherits the gate; because the proxy only injects the header on paths its regex matches, that router refuses on its first request instead of being quietly public. A 403 in that situation is the boundary reporting that it has drifted. The gate is optional, and unset means exactly today's behaviour. That keeps local development and all 113 existing admin test call sites working untouched, and means shipping the image before configuring the proxy cannot take the admin panel down. What it does not do is stay silent about it: the server warns at boot when the gate is inactive, naming what is unprotected. This project has been bitten repeatedly by controls that report success while doing nothing, and an unconfigured gate should be a visible choice rather than an invisible one. An empty value is treated as unset rather than as a secret, because enforcing an empty secret would admit any caller sending an empty header. Comparison is timing-safe over SHA-256 digests of both sides: timingSafeEqual throws on buffers of unequal length, so comparing raw values would turn a short header into a 500 rather than a 403, and a length check first would leak the secret's length. Turning it on requires the secret in two places at once — the stack environment and a proxy_set_header line on the gated location in NPM. Setting only one gives 403s until the other catches up. That coupling, and the three consequences above, are now written into the README beside the deployment section, since none of it is visible from the code. Verified over real HTTP as well as in tests. Booting without the secret logs the warning and serves admin normally; booting with it returns 403 for a missing header, 403 for a wrong one, 200 for the right one, and leaves the public storefront at 200 throughout, with each refusal logged distinguishably and without echoing the value it was sent. 8 new unit tests, 9 new integration tests covering every admin router separately — a correct middleware nobody mounted would pass the unit tests and protect nothing. 106 unit and 153 integration passing, lint 0 errors and 8 warnings unchanged. Refs #63 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
 bermudalambandClaude Opus 5
|
c546bd5141 |
docs: make QA review a standing step in the change checklist
Records the two-environment layout and a numbered checklist every change follows, with QA review as a required stop rather than a judgement call. Production is not where a bad deploy should be found, which is what happened with the categories/tags release. Production promotion now retags the image QA reviewed rather than rebuilding, so what ships is exactly what was tested, and the README carries the full command sequence with a verification gate at each step. Also records the two gates that have already failed here: confirming a pushed commit is actually on the branch, and that a schema/code ordering problem cannot be caught by any local suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
bermudalambandClaude Opus 5
|
c01e434747 |
fix(qa): build the QA image before deploying the stack (#25)
Deploying the stack first failed with "pull access denied for redefined-designs, repository does not exist or may require docker login". The README had the Portainer stack creation ahead of the build, so no local redefined-designs:qa image existed and Compose fell back to pulling from Docker Hub, where the repository does not exist. The build step now comes first in the one-time setup, and the review workflow says to rebuild before restarting the stack — the stack builds nothing itself and would otherwise run whatever was last tagged :qa. pull_policy: never makes the failure legible: a missing local image now reports itself as missing rather than as a registry authentication problem. Noted as safe to remove if the NAS's Compose ever rejects the key, since it only affects the error message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
bermudalambandClaude Opus 5
|
47f00a3485 |
feat(qa): add the QA environment stack definition (#25)
A disposable stack for reviewing merged-but-undeployed changes online, running alongside production on the same NAS. Every value that could collide with production is changed: container names, host port (32751), volume paths, database name, and image tag. The Postgres data directory is the critical one — sharing production's would mean QA writing into production's database files. QA deliberately differs from production in three ways. DEMO_MODE=true with no PayPal credentials, so the full cart and checkout path is exercisable with no route to live PayPal. No SMTP configuration, so a QA run cannot email anyone; the mailer already degrades gracefully when unset. And restart: "no", so a NAS reboot does not quietly bring QA back up and leave it running. The stack must be named redefined-designs-qa in Portainer: the stack name becomes the compose project name, and reusing production's would make compose reconcile the two against each other and remove the production containers. README covers the one-time NAS directory setup, the review workflow, and how to reset QA data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
bermudalambandClaude Opus 5
|
c77fdad2b9 |
fix: run migrations on boot and stop failures rendering as empty (#23)
The storefront showed no inventory after deploying the categories/tags release. No data was lost: the code queried categories/item_tags/ items.category_id against a database where the migration had not been run, and that failure was invisible at every layer. Three changes, each addressing one layer: Migrations now run at container start, so deployed code cannot be ahead of the schema and the easily-forgotten manual `docker exec migrate.js up` step disappears. migrate.js waits for Postgres to accept connections first, since the NAS brings the DB container up slower than the app, and still exits non-zero so a bad migration stops the container rather than serving a half-migrated schema. Express 4 does not forward a rejected async handler, and no error middleware was mounted, so a failing query never responded at all. Async routes are now wrapped and an error middleware guarantees a 500. A hung request is indistinguishable from an empty result in the UI, which is how a schema mismatch came to read as "the store has no items". The storefront now separates "request failed" from "no items" and offers a retry. fetchItems/fetchFilterOptions throw on a non-OK response rather than returning the parsed error body, which would have been set as the item list and crashed the grid on .map. Also restores the project-context update from 7fb5764, which was left out of PR #24 and ended up dangling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
bermudalamb
|
7f4479605a | chore: replace manual SQL migrations with node-pg-migrate | ||
|
bermudalamb
|
da240621db | docs: fix local-dev commands for PowerShell, document branching/commit conventions | ||
|
bermudalamb
|
921022c658 |
Add backend unit/integration tests, Playwright e2e tests, README, cookie Secure fix
SonarQube Analysis / sonarqube (push) Successful in 4m20s
|