docs(ops): name the crash loop the cutover runbook was most likely to cause (#175)
Linting / lint (pull_request) Successful in 3m7s
SonarQube Analysis / sonarqube (pull_request) Failing after 27m56s

Step 7 listed a missing `ADMIN_GATE_SECRET` — a warning the container starts through — and said nothing about the three PayPal secrets, which are a hard error that crash-loops it. That is backwards: the secrets are the likeliest thing to be missing after a stack replacement, because Portainer stack variables belong to the stack and are discarded with it, and `DEMO_MODE` is false in production so the app refuses to start without them. Hit for real following the runbook.

The verify section now shows the actual log block, says what it means, and gives the way to tell a missing variable from a misnamed one: a hardcoded value cannot be missing, so its absence from the error list proves nothing, while an interpolated variable that stays quiet while others complain proves substitution works and those others are simply unset. That is the reading that turns the log into a diagnosis instead of a list.

It also says the loop is harmless while the values are fetched — the container refuses before it serves anything or touches data — and where the values live if the old stack is already gone, since the webhook id in particular is readable rather than only recreatable.

Step 2 now names every interpolated variable rather than describing them in general, and states the consequence of each class going missing. A general instruction to record the environment is easy to read as already done.
This commit is contained in:
2026-08-26 08:35:01 -05:00
parent f5e3ec3e99
commit 0cec9b3c1f
+22 -3
View File
@@ -60,7 +60,9 @@ Everything here is lost when the stack is deleted, and the rollback in step 8 is
**The stack name**, exactly as Portainer shows it. If it is not `redefined-designs`, note that — the new stack must be created with that name, because the stack name becomes the compose project name and reusing QA's would make Compose reconcile the two against each other.
**Every stack environment variable, name and value.** These are all secrets. Portainer shows them under the stack's editor. The database password is among them, and a new stack brought up with a different `DB_PASSWORD` than the data directory was initialised with will fail to authenticate against its own database.
**Every stack environment variable, name and value.** These are all secrets, they belong to the stack, and deleting it discards them. This is the step whose omission is felt hardest: the compose file interpolates `DB_PASSWORD`, `SMTP_USER`, `SMTP_PASSWORD`, `SMTP_FROM`, `ADMIN_GATE_SECRET`, `PAYPAL_CLIENT_ID`, `PAYPAL_CLIENT_SECRET` and `PAYPAL_WEBHOOK_ID`, and an unset one substitutes to an empty string rather than failing. A missing `DB_PASSWORD` cannot authenticate against its own data directory; a missing PayPal secret crash-loops the container, because `DEMO_MODE` is false here and the app refuses to pretend it can take payments. Neither is recoverable from anything in this repository.
Copy them somewhere before you delete anything.
**The image the app container is running:**
@@ -167,10 +169,27 @@ Migration output must appear *before* `listening on 3000`, and `listening on 300
- `MAIL_ALLOWLIST is not set` — production is the one environment that has to reach real customers.
- `UPLOADS_BASE_URL is not set` — the uploads origin is not configured yet (#103). Expected until an NPM host exists for it.
**What a crash loop looks like, and it is the likeliest outcome of a missed step 2.** A `[config] refusing to start` block, then the whole boot sequence again, repeating:
```
[config] refusing to start — 3 problem(s) with the environment:
[config] - PAYPAL_CLIENT_ID is required when DEMO_MODE=false, because real payments are enabled.
[config] - PAYPAL_CLIENT_SECRET is required when DEMO_MODE=false, because real payments are enabled.
[config] - PAYPAL_WEBHOOK_ID is required when DEMO_MODE=false, because real payments are enabled.
```
That is a secret that did not reach the container, and it is what happens when the stack variables recorded in step 2 were not carried across — they belong to the stack and were discarded with the old one. An unset stack variable substitutes to an empty string rather than failing, so the container receives `PAYPAL_CLIENT_ID=` and refuses it.
Read the list against the compose file to tell a missing variable from a misnamed one. Anything hardcoded there — `PAYPAL_ENV`, `DEMO_MODE`, `UPLOADS_DIR` — cannot be missing, so its absence from the error list proves nothing. What is diagnostic is an *interpolated* variable that is NOT in the list: if `ADMIN_GATE_SECRET` is quiet while the PayPal three complain, substitution is working and those three specifically are unset, rather than something being wrong with the stack.
Fix it in the stack's environment, under exactly the names the compose file reads, and redeploy. If the values are gone with the old stack, the client id and secret are in the PayPal developer dashboard under the live app, and the webhook id is on the webhook entry for `PUBLIC_URL/webhooks/paypal` — readable rather than only recreatable.
The loop is harmless while you fetch them. The container refuses before it serves anything and before it touches data. The site is down, which is the cost, and nothing is being damaged.
**Warnings that mean something is wrong:**
- `ADMIN_GATE_SECRET is not set`the variable did not reach the container. Almost certainly a name mismatch from step 3. The admin API is protected only by the proxy until this is fixed.
- Any `[config]` error, which stops the container rather than warning.
- `ADMIN_GATE_SECRET is not set`a warning rather than an error, so the container starts. The admin API is then protected only by the proxy. Same cause as above: a variable that did not reach the container.
- Any other `[config]` error, which stops the container rather than warning.
```bash
# Is the data the same data?