redefined-designs/docs/ops/production-stack-cutover.md

# Cutting production over to the committed compose file

A one-time migration: production stops running from a stack definition that exists only in Portainer's web editor and starts running from `docker-compose.prod.yml` in this repository, deployed as a git repository stack.

This is not the routine deploy. For promoting a reviewed change to a stack that is already in this form, use [Promoting a reviewed change to production](../../README.md#promoting-a-reviewed-change-to-production) instead. Do that one often; do this one once.

## What this actually changes

Nothing about how production runs. Same image, same containers, same data, same port, same proxy in front of it. What changes is where the definition lives, and therefore whether anything can check it — `backend/tests/unit/composeEnvironment.test.ts` reads every compose file in this repository and hands its environment to the real validator. A stack in the web editor is invisible to that. On 2026-08-23 production refused to boot because `UPLOADS_DIR` had no line in it, and no test could have caught that. See #118.

After the cutover, changing production means changing this file and merging it.

## The two facts that decide whether this is safe

**Your data is not in Docker.** Both services bind-mount from the NAS filesystem:

```
/volume1/configs/redefined-designs/postgres   →  the database
/volume1/configs/redefined-designs/uploads    →  product images
```

Those are paths on the NAS, not named Docker volumes. Deleting the stack removes containers; it does not touch either directory. This is why the cutover is recoverable at all — but step 2 verifies it rather than trusting it, because the whole procedure rests on it being true of the stack you have *now*.

**Container names are fixed.** `redefined-designs-syn` and `redefined-designs-db-syn` are set explicitly in the compose file, so two stacks cannot hold them at once. The old stack must be gone before the new one comes up, or the deploy fails with a name collision that reads like a Portainer bug rather than a sequencing mistake.

## Before you start

Production will be down for the length of this, which is a few minutes if nothing surprises you. Pick a time when that is fine.

You need SSH to the NAS and Portainer access.

Read [backup-and-restore.md](backup-and-restore.md) first if you have not. The scheduled backups in the compose file do **not** cover this operation — they run while the stack runs, so they cannot cover a deploy that recreates it.

---

## 1. Back up

Not optional, and not the nightly dump. This is the one deploy that recreates everything, so take a backup that is seconds old rather than up to a day old.

```bash
# Database
sudo docker exec -t redefined-designs-db-syn pg_dump -U redefined -d redefined \
  > /volume1/configs/redefined-designs/cutover-$(date +%Y%m%d-%H%M).sql
ls -lh /volume1/configs/redefined-designs/cutover-*.sql
```

The size must be plausible — a dump of a few kilobytes means it dumped an empty database and you should stop and find out why before going further.

```bash
# Uploads
sudo tar -czf /volume1/configs/redefined-designs/cutover-uploads-$(date +%Y%m%d-%H%M).tar.gz \
  -C /volume1/configs/redefined-designs uploads
```

Copy both somewhere that is not `/volume1`. A backup on the same disk as the thing it protects is a convenience, not a backup.

## 2. Record what exists today

Everything here is lost when the stack is deleted, and the rollback in step 8 is only credible if you have it. Write it down somewhere outside Portainer.

**The stack name**, exactly as Portainer shows it. If it is not `redefined-designs`, note that — the new stack must be created with that name, because the stack name becomes the compose project name and reusing QA's would make Compose reconcile the two against each other.

**Every stack environment variable, name and value.** They belong to the stack, and deleting it discards them. This is the step whose omission is felt hardest. The compose file interpolates thirteen names — `DEMO_MODE`, `DB_PASSWORD`, `SMTP_USER`, `SMTP_PASSWORD`, `SMTP_FROM`, `ADMIN_GATE_SECRET`, `PAYPAL_CLIENT_ID`, `PAYPAL_CLIENT_SECRET`, `PAYPAL_WEBHOOK_ID`, `USPS_CLIENT_ID`, `USPS_CLIENT_SECRET`, `UPLOADS_BASE_URL` and `BACKUP_PASSPHRASE` — and an unset one substitutes to an empty string rather than failing. None of it is recoverable from anything in this repository. Take everything the stack holds rather than working from this list; it is here to say how much there is, and it is checked against the file rather than from memory.

`USPS_CLIENT_ID` and `USPS_CLIENT_SECRET` deserve naming because losing them is the one failure here that is completely silent. Address validation is skipped when they are empty rather than failing, so checkout keeps working and quietly stops validating addresses. Nothing in step 7 catches it, and there is no crash loop to notice.

A missing `DB_PASSWORD` does not fail the way you would expect either: the app never reaches a connection attempt. It refuses at boot, and the message names `PGPASSWORD` rather than the variable you set, because the compose file injects it as `PGPASSWORD=${DB_PASSWORD}`. Grep the log for the name in the error, not the name in Portainer.

`DEMO_MODE` is the odd one out and the easiest to miss, because it is the only one that is a setting rather than a credential. It must be exactly `true` or `false`. There is no default in the compose file, deliberately (#190) — a default would decide whether the shop takes money on the operator's behalf, silently, whichever way it pointed — so an unset `DEMO_MODE` refuses to boot rather than guessing. Production is `true` for now, the interim from #191; setting it to `false` is what restores real payments, and doing that without all three PayPal secrets present crash-loops the container.

Copy them somewhere before you delete anything.

**The image the app container is running:**

```bash
sudo docker inspect --format '{{.Config.Image}}' redefined-designs-syn
sudo docker images | grep redefined-designs
```

**The mounts** — this is the verification that the whole procedure rests on:

```bash
sudo docker inspect --format '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}}{{"\n"}}{{end}}' \
  redefined-designs-db-syn redefined-designs-syn
```

Every line must say `bind`, and the sources must be under `/volume1/configs/redefined-designs`. If any line says `volume`, **stop**. Your data is in a Docker-managed volume that stack deletion may remove, and this runbook does not cover that case — you would need to migrate that volume onto the bind-mount path first.

**A record of what is in the database**, so step 7 can prove nothing was lost:

```bash
sudo docker exec -it redefined-designs-db-syn psql -U redefined -d redefined \
  -c "SELECT count(*) FROM items;" \
  -c "SELECT count(*) FROM customers;" \
  -c "SELECT name, run_on FROM pgmigrations ORDER BY run_on DESC LIMIT 3;"
```

## 3. Reconcile the variable names

`docker-compose.prod.yml` reads a specific set of names. The authoritative list is in the header comment of that file, under "Required stack environment variables". If the stack you recorded in step 2 uses different names for any of them, the new stack must use the names the file reads.

This matters more than it looks. Portainer's stack variables are substituted into the compose file as `${VAR}`; they are not handed to the container. A variable whose name does not match anything in the file is silently substituted nowhere, and the failure reads as "I set it and the app says it is not set".

Write out the mapping now, while the old stack still exists to check against.

## 4. Make sure the image exists

The new stack has no `build:`. It runs `redefined-designs:latest`, which must already be on the NAS, or the deploy fails with "image not found" rather than quietly building one (#146).

```bash
sudo docker images | grep 'redefined-designs.*latest'
```

If it is missing, or if you want production to run the image QA reviewed rather than whatever `latest` currently points at:

```bash
sudo docker tag redefined-designs:qa redefined-designs:latest
sudo docker inspect --format '{{.Config.Cmd}}' redefined-designs:latest
# must print: [sh -c node migrate.js up && node dist/server.js]
```

That `Cmd` check is worth doing. It is what makes the container migrate before serving, so deployed code can never be ahead of the schema.

## 4b. Create the backup directories

The stack gained two backup services in #147, and they bind-mount directories that no earlier deploy of this application ever needed:

```bash
sudo mkdir -p /volume1/configs/redefined-designs/backups/postgres
sudo mkdir -p /volume1/configs/redefined-designs/backups/uploads
ls -la /volume1/configs/redefined-designs/backups/
```

No `chown`. Both backup images run as root, unlike the Postgres image, whose data directory has to be owned by uid 999.

This is easy to skip because nothing fails loudly. A backup container whose mount path is missing does not crash — it sits in **Created**, having never started, and `docker logs` on it says only that nothing matched, because there is no output from a container that never ran. Portainer shows it beside the healthy ones and the stack looks deployed. The failure surfaces the day someone needs a restore.

Verify after the stack is up, as part of step 7:

```bash
sudo docker ps --filter name=backup --format '{{.Names}}\t{{.Status}}'
```

Both must be **Up**. Both will also report *starting* rather than healthy, and that is correct — their healthchecks assert a recent backup exists, and nothing exists until the first scheduled run. `start_period` allows 25 hours for the database and 8 days for uploads.

**Do not wait for those runs to find out whether this works.** Up and scheduled is not the same as producing a restorable file, and the difference is one command each:

```bash
sudo docker exec redefined-designs-db-backup-syn /backup.sh
ls -lht /volume1/configs/redefined-designs/backups/postgres/daily | head

sudo docker exec redefined-designs-uploads-backup-syn backup
ls -lht /volume1/configs/redefined-designs/backups/uploads | head
```

Check the sizes are plausible. A dump of an empty database succeeds and tells you nothing, so confirm the dump carries rows rather than only a schema:

```bash
zcat /volume1/configs/redefined-designs/backups/postgres/daily/redefined-latest.sql.gz \
  | grep -E "^COPY public\.(items|customers|orders)"
```

Then confirm the healthchecks agree with where the files landed — a check whose `find` path disagrees with where the tool actually writes reports unhealthy forever, and is worth catching now rather than in a year:

```bash
sudo docker inspect --format '{{.Name}} {{.State.Health.Status}}' \
  redefined-designs-db-backup-syn redefined-designs-uploads-backup-syn
```

Both reach `healthy` within their check intervals once a file exists — an hour for the database, six for uploads.

## 5. Remove the old stack

In Portainer: **Stacks → the stack from step 2 → Delete**.

This removes the containers. It does not remove `/volume1/configs/redefined-designs/postgres` or `.../uploads`, which you verified as bind mounts in step 2.

Confirm the names are actually free, because the next step fails on a collision:

```bash
sudo docker ps -a | grep redefined-designs
```

Anything still listed for production must be removed before continuing:

```bash
sudo docker rm -f redefined-designs-syn redefined-designs-db-syn
```

Leave QA's containers alone — they are `redefined-designs-qa-*` and are a different stack.

## 6. Create the git repository stack

Portainer: **Stacks → Add stack → Repository**.

| Field | Value |
| --- | --- |
| Name | `redefined-designs` — not `redefined-designs-qa` |
| Build method | Repository |
| Repository URL | `https://gitea.bermudalamb.synology.me/bermudalamb/redefined-designs` |
| Reference | `refs/heads/main` |
| Compose path | `docker-compose.prod.yml` |

Add the environment variables from steps 2 and 3, under the names the compose file reads.

**Leave any "pull latest image" or re-pull option OFF.** There is no registry to pull this image from; it exists only on the NAS. Turning it on makes a present image report itself as a registry authentication failure.

Deploy the stack.

## 7. Verify

In this order. Each step answers a different question, and the later ones are meaningless if an earlier one failed.

```bash
# Did it start, and start once?
sudo docker logs redefined-designs-syn | head -40
```

Migration output must appear *before* `listening on 3000`, and `listening on 3000` must appear exactly once. Repeats mean a crash loop.

**Expected warnings.** These are correct and must not be silenced:

- `MAIL_ALLOWLIST is not set` — production is the one environment that has to reach real customers.
- `UPLOADS_BASE_URL is not set` — the uploads origin is not configured yet (#103). Expected until an NPM host exists for it.

**What a crash loop looks like, and it is the likeliest outcome of a missed step 2.** A `[config] refusing to start` block, then the whole boot sequence again, repeating.

Miss step 2 wholesale and it is two problems, led by a name you never typed:

```
[config] refusing to start — 2 problem(s) with the environment:
[config]   - PGPASSWORD is required and is not set.
[config]   - DEMO_MODE is required and must be exactly 'true' or 'false'. It decides whether real payments are taken, so it has to be stated rather than inherited.
```

Carry `DB_PASSWORD` across and only `DEMO_MODE` is left, which is the single likeliest form during the demo interim:

```
[config] refusing to start — 1 problem(s) with the environment:
[config]   - DEMO_MODE is required and must be exactly 'true' or 'false'. It decides whether real payments are taken, so it has to be stated rather than inherited.
```

A value that was set but mistyped reads differently, and the quotes are the only thing that distinguishes `true ` with a trailing space from `true`:

```
[config]   - DEMO_MODE must be exactly 'true' or 'false', but is 'True'. Anything else used to be read as demo mode, which meant a typo here quietly stopped the shop charging anyone.
```

The PayPal form appears once `DEMO_MODE` is `false` and real payments are on:

```
[config] refusing to start — 3 problem(s) with the environment:
[config]   - PAYPAL_CLIENT_ID is required when DEMO_MODE=false, because real payments are enabled.
[config]   - PAYPAL_CLIENT_SECRET is required when DEMO_MODE=false, because real payments are enabled.
[config]   - PAYPAL_WEBHOOK_ID is required when DEMO_MODE=false, because real payments are enabled.
```

Either way it is a stack variable that did not reach the container, and it is what happens when the ones recorded in step 2 were not carried across — they belong to the stack and were discarded with the old one. An unset stack variable substitutes to an empty string rather than failing, so the container receives `PAYPAL_CLIENT_ID=` and refuses it.

Compose itself only warns about this — `The "DEMO_MODE" variable is not set. Defaulting to a blank string.` — and deploys anyway. That warning is easy to miss in Portainer's stack UI, and the container then crash-loops under `restart: unless-stopped`: loud in the log, invisible in a glance at the stack list. The container log is the reliable signal, not the deploy output.

Read the list against the compose file to tell a missing variable from a misnamed one. Anything hardcoded there — `PAYPAL_ENV`, `UPLOADS_DIR` — cannot be missing, so its absence from the error list proves nothing. `DEMO_MODE` was in that list and is not any more: since #190 it is interpolated like the secrets, which makes it the first thing to check rather than something to rule out. What is diagnostic is an *interpolated* variable that is NOT in the list: if `ADMIN_GATE_SECRET` is quiet while the PayPal three complain, substitution is working and those three specifically are unset, rather than something being wrong with the stack.

Fix it in the stack's environment, under exactly the names the compose file reads, and redeploy. If the values are gone with the old stack, the client id and secret are in the PayPal developer dashboard under the live app, and the webhook id is on the webhook entry for `PUBLIC_URL/webhooks/paypal` — readable rather than only recreatable.

The loop is harmless while you fetch them. The container refuses before it serves anything and before it touches data. The site is down, which is the cost, and nothing is being damaged.

**Warnings that mean something is wrong:**

- `ADMIN_GATE_SECRET is not set` — a warning rather than an error, so the container starts. The admin API is then protected only by the proxy. Same cause as above: a variable that did not reach the container.
- Any other `[config]` error, which stops the container rather than warning.

```bash
# Is the data the same data?
sudo docker exec -it redefined-designs-db-syn psql -U redefined -d redefined \
  -c "SELECT count(*) FROM items;" \
  -c "SELECT count(*) FROM customers;" \
  -c "SELECT name, run_on FROM pgmigrations ORDER BY run_on DESC LIMIT 3;"
```

The counts must match what you recorded in step 2. If they are zero, the container came up against a fresh data directory rather than the existing one — stop, and check the mount paths before doing anything else.

```bash
# Are the images still served?
sudo docker exec redefined-designs-syn ls /app/uploads | head

# Did every container in the stack actually start?
sudo docker ps -a --filter name=redefined-designs --format '{{.Names}}\t{{.Status}}'
```

Four containers, all **Up** — `redefined-designs-syn`, `redefined-designs-db-syn`, and the two backup services from step 4b. A container reading **Created** never started — that is a missing bind-mount directory or a deploy that aborted part way, and it is silent, because a container that never ran has no logs to read.

Then load the storefront in a **private window** — aggressive bundle caching on this project has produced false "still broken" reports after correct deploys. Check that product images render, sign in, and open `/admin` to confirm authentik and the admin gate still agree.

## 8. If it goes wrong

Nothing has been destroyed. The data directories were never touched, and you recorded the old definition in step 2.

Delete the new stack, recreate the old one from that record with its original variables, and deploy it. Production comes back on the same data.

Restoring the step 1 dump is a last resort, not a first move — reach for it only if the database itself is damaged, and follow [backup-and-restore.md](backup-and-restore.md).

## After

The compose file in this repository is now what production runs. Changing it means changing it here and merging, and `composeEnvironment.test.ts` checks it on every push.

Two things worth doing while you are in there:

- Confirm `ADMIN_GATE_SECRET` is genuinely set, rather than assumed. It is the application-layer half of the admin boundary (#63), and the boot log tells you plainly.
- Portainer will now show the stack as out of date when `main` moves. That is the point, but note that redeploying from git does **not** repull or rebuild the image — promoting a new image is still the separate, deliberate step the README documents.