bermudalamb
cf9003bc2d
#118 put `docker-compose.prod.yml` in the repository and argued at length for why it belongs there. It never produced the procedure for getting production from where it is — a stack living in Portainer's web editor — to where that file expects it to be. What existed was scattered and none of it was a procedure. The compose header described the destination: name the stack this, deploy it as a git repository stack with these settings, rename any variables whose names differ. The README's deployment section is a good runbook for the routine redeploy, and assumes the stack is already in the right form. So the riskiest deployment operation this project has was the one with no steps written down. `docs/ops/production-stack-cutover.md` is those steps, in the order that matters, with the two facts that decide whether the operation is safe stated up front rather than left to be inferred from the volumes block. The first is that all persistent data is bind-mounted from the NAS filesystem rather than held in Docker-managed volumes, so deleting the stack cannot lose the database or the product images. That is what makes the cutover recoverable at all, and step 2 verifies it rather than trusting it — the `docker inspect` there discriminates `bind` from `volume` and says to stop on `volume`, because this runbook does not cover that case. The second is that both services set an explicit `container_name`, so the old containers must be gone before the new stack starts or the deploy fails on a collision that reads like a Portainer bug rather than a sequencing mistake. It also captures what is simply lost if not recorded first. Portainer stack variables belong to the stack and are discarded with it; they are all secrets, and a stack brought back up with a different `DB_PASSWORD` than the data directory was initialised with cannot authenticate against its own database. Recording them is step 2 and it is what makes the rollback credible. The verification section separates the boot warnings that are correct in production — no mail allowlist, no uploads origin yet — from the one that means a variable never arrived. That distinction is the whole failure mode #118 was about: a stack variable whose name matches nothing in the file is substituted nowhere and never reaches the container, and the failure reads as "I set it and it says it is not set". Every diagnostic command in it was run rather than written from memory. The mount inspection was checked against a container that genuinely uses a named volume, to confirm it reports the difference the step depends on. The compose header now points at the runbook for the procedure and keeps the rationale that belongs at the point of use — why there is no `build:`, why most values are hardcoded, why `NODE_ENV` is absent. The README says plainly that its section is the routine deploy and links to the other one. Closes #175
11 KiB
Cutting production over to the committed compose file
A one-time migration: production stops running from a stack definition that exists only in Portainer's web editor and starts running from docker-compose.prod.yml in this repository, deployed as a git repository stack.
This is not the routine deploy. For promoting a reviewed change to a stack that is already in this form, use Promoting a reviewed change to production instead. Do that one often; do this one once.
What this actually changes
Nothing about how production runs. Same image, same containers, same data, same port, same proxy in front of it. What changes is where the definition lives, and therefore whether anything can check it — backend/tests/unit/composeEnvironment.test.ts reads every compose file in this repository and hands its environment to the real validator. A stack in the web editor is invisible to that. On 2026-08-23 production refused to boot because UPLOADS_DIR had no line in it, and no test could have caught that. See #118.
After the cutover, changing production means changing this file and merging it.
The two facts that decide whether this is safe
Your data is not in Docker. Both services bind-mount from the NAS filesystem:
/volume1/configs/redefined-designs/postgres → the database
/volume1/configs/redefined-designs/uploads → product images
Those are paths on the NAS, not named Docker volumes. Deleting the stack removes containers; it does not touch either directory. This is why the cutover is recoverable at all — but step 2 verifies it rather than trusting it, because the whole procedure rests on it being true of the stack you have now.
Container names are fixed. redefined-designs-syn and redefined-designs-db-syn are set explicitly in the compose file, so two stacks cannot hold them at once. The old stack must be gone before the new one comes up, or the deploy fails with a name collision that reads like a Portainer bug rather than a sequencing mistake.
Before you start
Production will be down for the length of this, which is a few minutes if nothing surprises you. Pick a time when that is fine.
You need SSH to the NAS and Portainer access.
Read backup-and-restore.md first if you have not. The scheduled backups in the compose file do not cover this operation — they run while the stack runs, so they cannot cover a deploy that recreates it.
1. Back up
Not optional, and not the nightly dump. This is the one deploy that recreates everything, so take a backup that is seconds old rather than up to a day old.
# Database
sudo docker exec -t redefined-designs-db-syn pg_dump -U redefined -d redefined \
> /volume1/configs/redefined-designs/cutover-$(date +%Y%m%d-%H%M).sql
ls -lh /volume1/configs/redefined-designs/cutover-*.sql
The size must be plausible — a dump of a few kilobytes means it dumped an empty database and you should stop and find out why before going further.
# Uploads
sudo tar -czf /volume1/configs/redefined-designs/cutover-uploads-$(date +%Y%m%d-%H%M).tar.gz \
-C /volume1/configs/redefined-designs uploads
Copy both somewhere that is not /volume1. A backup on the same disk as the thing it protects is a convenience, not a backup.
2. Record what exists today
Everything here is lost when the stack is deleted, and the rollback in step 8 is only credible if you have it. Write it down somewhere outside Portainer.
The stack name, exactly as Portainer shows it. If it is not redefined-designs, note that — the new stack must be created with that name, because the stack name becomes the compose project name and reusing QA's would make Compose reconcile the two against each other.
Every stack environment variable, name and value. These are all secrets. Portainer shows them under the stack's editor. The database password is among them, and a new stack brought up with a different DB_PASSWORD than the data directory was initialised with will fail to authenticate against its own database.
The image the app container is running:
sudo docker inspect --format '{{.Config.Image}}' redefined-designs-syn
sudo docker images | grep redefined-designs
The mounts — this is the verification that the whole procedure rests on:
sudo docker inspect --format '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}}{{"\n"}}{{end}}' \
redefined-designs-db-syn redefined-designs-syn
Every line must say bind, and the sources must be under /volume1/configs/redefined-designs. If any line says volume, stop. Your data is in a Docker-managed volume that stack deletion may remove, and this runbook does not cover that case — you would need to migrate that volume onto the bind-mount path first.
A record of what is in the database, so step 7 can prove nothing was lost:
sudo docker exec -it redefined-designs-db-syn psql -U redefined -d redefined \
-c "SELECT count(*) FROM items;" \
-c "SELECT count(*) FROM customers;" \
-c "SELECT name, run_on FROM pgmigrations ORDER BY run_on DESC LIMIT 3;"
3. Reconcile the variable names
docker-compose.prod.yml reads a specific set of names. The authoritative list is in the header comment of that file, under "Required stack environment variables". If the stack you recorded in step 2 uses different names for any of them, the new stack must use the names the file reads.
This matters more than it looks. Portainer's stack variables are substituted into the compose file as ${VAR}; they are not handed to the container. A variable whose name does not match anything in the file is silently substituted nowhere, and the failure reads as "I set it and the app says it is not set".
Write out the mapping now, while the old stack still exists to check against.
4. Make sure the image exists
The new stack has no build:. It runs redefined-designs:latest, which must already be on the NAS, or the deploy fails with "image not found" rather than quietly building one (#146).
sudo docker images | grep 'redefined-designs.*latest'
If it is missing, or if you want production to run the image QA reviewed rather than whatever latest currently points at:
sudo docker tag redefined-designs:qa redefined-designs:latest
sudo docker inspect --format '{{.Config.Cmd}}' redefined-designs:latest
# must print: [sh -c node migrate.js up && node dist/server.js]
That Cmd check is worth doing. It is what makes the container migrate before serving, so deployed code can never be ahead of the schema.
5. Remove the old stack
In Portainer: Stacks → the stack from step 2 → Delete.
This removes the containers. It does not remove /volume1/configs/redefined-designs/postgres or .../uploads, which you verified as bind mounts in step 2.
Confirm the names are actually free, because the next step fails on a collision:
sudo docker ps -a | grep redefined-designs
Anything still listed for production must be removed before continuing:
sudo docker rm -f redefined-designs-syn redefined-designs-db-syn
Leave QA's containers alone — they are redefined-designs-qa-* and are a different stack.
6. Create the git repository stack
Portainer: Stacks → Add stack → Repository.
| Field | Value |
|---|---|
| Name | redefined-designs — not redefined-designs-qa |
| Build method | Repository |
| Repository URL | https://gitea.bermudalamb.synology.me/bermudalamb/redefined-designs |
| Reference | refs/heads/main |
| Compose path | docker-compose.prod.yml |
Add the environment variables from steps 2 and 3, under the names the compose file reads.
Leave any "pull latest image" or re-pull option OFF. There is no registry to pull this image from; it exists only on the NAS. Turning it on makes a present image report itself as a registry authentication failure.
Deploy the stack.
7. Verify
In this order. Each step answers a different question, and the later ones are meaningless if an earlier one failed.
# Did it start, and start once?
sudo docker logs redefined-designs-syn | head -40
Migration output must appear before listening on 3000, and listening on 3000 must appear exactly once. Repeats mean a crash loop.
Expected warnings. These are correct and must not be silenced:
MAIL_ALLOWLIST is not set— production is the one environment that has to reach real customers.UPLOADS_BASE_URL is not set— the uploads origin is not configured yet (#103). Expected until an NPM host exists for it.
Warnings that mean something is wrong:
ADMIN_GATE_SECRET is not set— the variable did not reach the container. Almost certainly a name mismatch from step 3. The admin API is protected only by the proxy until this is fixed.- Any
[config]error, which stops the container rather than warning.
# Is the data the same data?
sudo docker exec -it redefined-designs-db-syn psql -U redefined -d redefined \
-c "SELECT count(*) FROM items;" \
-c "SELECT count(*) FROM customers;" \
-c "SELECT name, run_on FROM pgmigrations ORDER BY run_on DESC LIMIT 3;"
The counts must match what you recorded in step 2. If they are zero, the container came up against a fresh data directory rather than the existing one — stop, and check the mount paths before doing anything else.
# Are the images still served?
sudo docker exec redefined-designs-syn ls /app/uploads | head
Then load the storefront in a private window — aggressive bundle caching on this project has produced false "still broken" reports after correct deploys. Check that product images render, sign in, and open /admin to confirm authentik and the admin gate still agree.
8. If it goes wrong
Nothing has been destroyed. The data directories were never touched, and you recorded the old definition in step 2.
Delete the new stack, recreate the old one from that record with its original variables, and deploy it. Production comes back on the same data.
Restoring the step 1 dump is a last resort, not a first move — reach for it only if the database itself is damaged, and follow backup-and-restore.md.
After
The compose file in this repository is now what production runs. Changing it means changing it here and merging, and composeEnvironment.test.ts checks it on every push.
Two things worth doing while you are in there:
- Confirm
ADMIN_GATE_SECRETis genuinely set, rather than assumed. It is the application-layer half of the admin boundary (#63), and the boot log tells you plainly. - Portainer will now show the stack as out of date when
mainmoves. That is the point, but note that redeploying from git does not repull or rebuild the image — promoting a new image is still the separate, deliberate step the README documents.