Production rebuilds its own image rather than running the one QA tested, and the builds are not reproducible anyway #146

New Issue

2026-08-23T16:38:36-05:00

bermudalamb commented

2026-08-23 16:38:36 -05:00

docker-compose.prod.yml as committed in #145 carries build: and pull_policy: build, copied from the QA stack. That makes Portainer build production's image from the repository at deploy time.

It looked like a simplification — it removes the manual "retag redefined-designs:qa as redefined-designs:latest" step from the deploy. What it actually removes is the property that step exists to provide: that production runs the exact image QA was reviewed against.

That would be a defensible trade if a rebuild of the same commit produced the same image. It does not.

The builds ignore the lockfiles entirely

Dockerfile copies only package.json and installs with npm install:

COPY frontend/package.json ./
RUN npm install
...
COPY backend/package.json ./
RUN npm install
...
COPY --from=backend-build /app/backend/package.json ./
RUN npm install --omit=dev

backend/package-lock.json and frontend/package-lock.json both exist in the repository and are never copied into any build stage. npm install without a lockfile resolves every dependency range afresh, so two builds of the same commit, minutes apart, can differ in any transitive dependency that published in between.

So "build production from the same git ref QA was built from" does not mean "run what QA tested". It means "run something built from the same source, with whatever the registry was serving at the time".

This is the quiet kind of drift: nothing fails, the versions are all legal per the ranges, and the difference only shows up as behaviour nobody can reproduce.

Two fixes, and they are not alternatives

Deploy the artifact that was tested. Production's compose should drop build: and pull_policy: build and simply declare image: redefined-designs:latest, with promotion staying an explicit step. There is no registry on the NAS, so promotion is a local docker tag — crude, but it does move the reviewed bytes rather than a re-derivation of them.

Make builds reproducible regardless. Copy the lockfiles into each stage and use npm ci:

COPY backend/package.json backend/package-lock.json ./
RUN npm ci

npm ci installs exactly the lockfile and fails if package.json and the lockfile disagree, which is the check that is missing today. This matters for CI as much as for deploys — the test suites are currently run against dependency trees that nothing pins either.

Worth doing both. The first makes the deploy honest about what it ships; the second makes any rebuild — CI, QA, a local reproduction of a production bug — mean something.

Caveat worth checking before switching

Dropping build: means the image must already exist on the NAS when the stack starts. A first deploy, or a NAS that has pruned images, then fails with "image not found" rather than building one. The promotion step becomes load-bearing rather than a convenience, and that should be written into the deploy notes rather than discovered.

pull_policy: build also carries a comment in both compose files warning that without it a redeploy can appear to succeed while running old code. Removing it reintroduces exactly that risk in a different shape: a redeploy that reuses redefined-designs:latest because nobody re-tagged. The mitigation is the same either way — the deploy has to state which image it is promoting.

Acceptance criteria

Production's compose runs image: redefined-designs:latest with no build: or pull_policy: build
The promotion step is documented where the deploy is documented, including that it is now required rather than optional
The Dockerfile copies both lockfiles and installs with npm ci in every stage
A rebuild of an unchanged commit produces the same dependency tree, verified rather than assumed

`docker-compose.prod.yml` as committed in #145 carries `build:` and `pull_policy: build`, copied from the QA stack. That makes Portainer build production's image from the repository at deploy time. It looked like a simplification — it removes the manual "retag `redefined-designs:qa` as `redefined-designs:latest`" step from the deploy. What it actually removes is the property that step exists to provide: that production runs **the exact image QA was reviewed against**. That would be a defensible trade if a rebuild of the same commit produced the same image. It does not. ## The builds ignore the lockfiles entirely `Dockerfile` copies only `package.json` and installs with `npm install`: ```dockerfile COPY frontend/package.json ./ RUN npm install ... COPY backend/package.json ./ RUN npm install ... COPY --from=backend-build /app/backend/package.json ./ RUN npm install --omit=dev ``` `backend/package-lock.json` and `frontend/package-lock.json` both exist in the repository and are never copied into any build stage. `npm install` without a lockfile resolves every dependency range afresh, so two builds of the same commit, minutes apart, can differ in any transitive dependency that published in between. So "build production from the same git ref QA was built from" does not mean "run what QA tested". It means "run something built from the same source, with whatever the registry was serving at the time". This is the quiet kind of drift: nothing fails, the versions are all legal per the ranges, and the difference only shows up as behaviour nobody can reproduce. ## Two fixes, and they are not alternatives **Deploy the artifact that was tested.** Production's compose should drop `build:` and `pull_policy: build` and simply declare `image: redefined-designs:latest`, with promotion staying an explicit step. There is no registry on the NAS, so promotion is a local `docker tag` — crude, but it does move the reviewed bytes rather than a re-derivation of them. **Make builds reproducible regardless.** Copy the lockfiles into each stage and use `npm ci`: ```dockerfile COPY backend/package.json backend/package-lock.json ./ RUN npm ci ``` `npm ci` installs exactly the lockfile and fails if `package.json` and the lockfile disagree, which is the check that is missing today. This matters for CI as much as for deploys — the test suites are currently run against dependency trees that nothing pins either. Worth doing both. The first makes the deploy honest about what it ships; the second makes any rebuild — CI, QA, a local reproduction of a production bug — mean something. ## Caveat worth checking before switching Dropping `build:` means the image must already exist on the NAS when the stack starts. A first deploy, or a NAS that has pruned images, then fails with "image not found" rather than building one. The promotion step becomes load-bearing rather than a convenience, and that should be written into the deploy notes rather than discovered. `pull_policy: build` also carries a comment in both compose files warning that without it a redeploy can appear to succeed while running old code. Removing it reintroduces exactly that risk in a different shape: a redeploy that reuses `redefined-designs:latest` because nobody re-tagged. The mitigation is the same either way — the deploy has to state which image it is promoting. ## Acceptance criteria - Production's compose runs `image: redefined-designs:latest` with no `build:` or `pull_policy: build` - The promotion step is documented where the deploy is documented, including that it is now required rather than optional - The Dockerfile copies both lockfiles and installs with `npm ci` in every stage - A rebuild of an unchanged commit produces the same dependency tree, verified rather than assumed

bermudalamb referenced this issue

2026-08-23 16:38:59 -05:00

Nothing backs up the production database #147

bermudalamb referenced this issue from a commit

2026-08-23 16:50:28 -05:00

fix(deploy): run the image QA reviewed rather than rebuilding production (#146)

bermudalamb referenced this issue

2026-08-23 16:51:08 -05:00

fix(deploy): run the image QA reviewed rather than rebuilding production (#146) #148

bermudalamb self-assigned this 2026-08-23 16:51:37 -05:00

bermudalamb added this to the Code Quality and Hardening 2 project 2026-08-23 16:51:45 -05:00

bermudalamb added reference feature/146-promote-tested-image