docs(intake): design and implementation plans for the intake pipeline (#220) #221

Merged
bermudalamb merged 5 commits from feature/220-intake-pipeline-design into main 2026-08-29 10:32:49 -05:00
Owner

Documents only — no implementation. Opened so the design and plans can be read and reviewed in Gitea.

Contents

File What it is
docs/superpowers/specs/2026-08-29-intake-pipeline-design.md The design for #220
docs/superpowers/plans/2026-08-29-intake-upload-links.md Implementation plan for #222 — slice 1, nine tasks
docs/superpowers/plans/2026-08-29-strip-exif-and-reencode.md Implementation plan for #226 — five tasks

The #226 plan is here for convenience rather than because it belongs to this epic — #226 stands on its own and is first in the approved sequence. Say the word and it moves to its own PR.

What is being designed

A named, revocable link lets someone without an account send in photos of one item plus a free-text note. A background worker asks Claude to draft a name, a marketing description, a category and tags from the existing taxonomy, and a suggested price. The admin is emailed, and the item sits unpublished until the admin has read the copy, checked the price and published it deliberately.

Most of this already exists

The design adds a pipeline in front of the existing lifecycle rather than a parallel one beside it. pending has been the unpublished state since #90 and is already excluded from every public and storefront query; the upload path already validates magic bytes against a three-type allowlist; mail already has admin-editable templates, placeholder validation and the MAIL_ALLOWLIST guard; and node-cron is already the background-work pattern.

Genuinely new: a way in for someone with no admin account, the first LLM integration in this codebase, and somewhere to review a draft.

Pricing — revised in the second commit

The first commit made price_cents nullable and refused to publish an unpriced item. That was reversed on review: the column stays NOT NULL and gains a default of 80.00, and where the model suggests a price the worker writes it onto the item.

This makes the feature considerably cheaper to build. The column's type is unchanged, so the fifteen files that read price_cents — the cart and the checkout among them — keep working untouched.

It also gives up a guarantee, deliberately. The storefront can now be reached by a price the admin never chose. The protection moves out of the schema and into the review queue, where it is weaker:

  • 80.00 is a plausible price, not an obvious sentinel. 0 would render "$0.00" and read as a bug; 80.00 reads as a decision, so a default left unnoticed sells the item rather than announcing itself.
  • The residual protection is that publishing is still a deliberate act from the queue — the email carries no publish button, so the price is always on screen before anything ships.
  • price_source (ai / default / admin) marks a price nobody has confirmed. Publishing unconfirmed is allowed; it just says so at the point of publishing.

Things the plans changed about the design

Reading the codebase while planning turned up three corrections, all carried in the later commits:

  • The image caps in the design were invented. admin.ts already defines MAX_IMAGES_PER_REQUEST = 6 and MAX_IMAGE_BYTES = 8_000_000 for this exact pipeline. Two sets of caps on one path is a defect waiting to happen, so the plan uses the existing ones.
  • The feature flag protects nothing in slice 1 — nothing is reachable until a link exists. It moves to #223, where the paid API call appears.
  • A gap in the design, not a simplification of it: as specced, a stranger with a bad token would still cause bytes to be written to the uploads volume and then deleted. That is disk churn an unauthenticated caller controls, leaning on an unlink a crash could skip. The link is now resolved before multer runs, and the ordering is asserted by a test.

Still worth pushing back on

The placeholder item name. The item row is created before the AI has named it, so it lands with the submission timestamp as its name. The alternative — asking the uploader for a working title — was considered and declined, so the ugliness is absorbed server-side.

Format preservation in the #226 plan. Not converting to WebP costs compression on PNGs, and buys a backfill that touches no database rows. Explained in that plan's decisions table.

Closes nothing on merge. Sub-issues: #222, #223, #224, #225, #227, and #226 separately.

Ref #220

Documents only — no implementation. Opened so the design and plans can be read and reviewed in Gitea. ## Contents | File | What it is | | --- | --- | | `docs/superpowers/specs/2026-08-29-intake-pipeline-design.md` | The design for #220 | | `docs/superpowers/plans/2026-08-29-intake-upload-links.md` | Implementation plan for #222 — slice 1, nine tasks | | `docs/superpowers/plans/2026-08-29-strip-exif-and-reencode.md` | Implementation plan for #226 — five tasks | The #226 plan is here for convenience rather than because it belongs to this epic — #226 stands on its own and is first in the approved sequence. Say the word and it moves to its own PR. ## What is being designed A named, revocable link lets someone without an account send in photos of one item plus a free-text note. A background worker asks Claude to draft a name, a marketing description, a category and tags from the existing taxonomy, and a suggested price. The admin is emailed, and the item sits unpublished until the admin has read the copy, checked the price and published it deliberately. ## Most of this already exists The design adds a pipeline in front of the existing lifecycle rather than a parallel one beside it. `pending` has been the unpublished state since #90 and is already excluded from every public and storefront query; the upload path already validates magic bytes against a three-type allowlist; mail already has admin-editable templates, placeholder validation and the `MAIL_ALLOWLIST` guard; and `node-cron` is already the background-work pattern. Genuinely new: a way in for someone with no admin account, the first LLM integration in this codebase, and somewhere to review a draft. ## Pricing — revised in the second commit The first commit made `price_cents` nullable and refused to publish an unpriced item. That was reversed on review: the column stays `NOT NULL` and gains a **default of 80.00**, and where the model suggests a price the worker writes it onto the item. **This makes the feature considerably cheaper to build.** The column's type is unchanged, so the fifteen files that read `price_cents` — the cart and the checkout among them — keep working untouched. **It also gives up a guarantee, deliberately.** The storefront can now be reached by a price the admin never chose. The protection moves out of the schema and into the review queue, where it is weaker: - 80.00 is a *plausible* price, not an obvious sentinel. `0` would render "$0.00" and read as a bug; 80.00 reads as a decision, so a default left unnoticed sells the item rather than announcing itself. - The residual protection is that publishing is still a deliberate act **from the queue** — the email carries no publish button, so the price is always on screen before anything ships. - `price_source` (`ai` / `default` / `admin`) marks a price nobody has confirmed. Publishing unconfirmed is allowed; it just says so at the point of publishing. ## Things the plans changed about the design Reading the codebase while planning turned up three corrections, all carried in the later commits: - **The image caps in the design were invented.** `admin.ts` already defines `MAX_IMAGES_PER_REQUEST = 6` and `MAX_IMAGE_BYTES = 8_000_000` for this exact pipeline. Two sets of caps on one path is a defect waiting to happen, so the plan uses the existing ones. - **The feature flag protects nothing in slice 1** — nothing is reachable until a link exists. It moves to #223, where the paid API call appears. - **A gap in the design, not a simplification of it:** as specced, a stranger with a bad token would still cause bytes to be written to the uploads volume and then deleted. That is disk churn an unauthenticated caller controls, leaning on an unlink a crash could skip. The link is now resolved *before* multer runs, and the ordering is asserted by a test. ## Still worth pushing back on **The placeholder item name.** The item row is created before the AI has named it, so it lands with the submission timestamp as its name. The alternative — asking the uploader for a working title — was considered and declined, so the ugliness is absorbed server-side. **Format preservation in the #226 plan.** Not converting to WebP costs compression on PNGs, and buys a backfill that touches no database rows. Explained in that plan's decisions table. Closes nothing on merge. Sub-issues: #222, #223, #224, #225, #227, and #226 separately. Ref #220
bermudalamb added 1 commit 2026-08-29 07:44:07 -05:00
docs(intake): design the upload-link, AI-draft and review-queue pipeline (#220)
SonarQube Analysis / sonarqube (pull_request) Failing after 16m43s
Linting / lint (pull_request) Successful in 1m52s
dbe6a0cf8f
A named, revocable link lets someone without an account send in photos of one item plus a note; a background worker drafts the listing; the admin is emailed and publishes it deliberately from a review queue.

Most of the lifecycle already exists and is reused rather than rebuilt: `pending` has been the unpublished state since #90 and is already excluded from every public query, the upload path already validates magic bytes against a three-type allowlist, mail already has editable templates and an allowlist guard, and node-cron is already the background-work pattern. What is new is a way in for someone with no admin account, the first LLM integration in this codebase, and somewhere to review a draft.

The design turns on one invariant: nothing reaches the storefront at a price a model guessed. The suggested price lives on the draft and never on the item, the email carries no publish button, and the publish path refuses an item with no price. That is also why `price_cents` becomes nullable rather than defaulting to zero — a sentinel that formats as "$0.00" is the same class of quiet failure as `DEMO_MODE` once being "demo unless the value is exactly false", and nullability makes the compiler enumerate all fifteen call sites instead.

Ref #220
bermudalamb added 1 commit 2026-08-29 07:52:13 -05:00
docs(intake): price arriving items rather than leaving them unpriced (#220)
SonarQube Analysis / sonarqube (pull_request) Failing after 16m51s
Linting / lint (pull_request) Successful in 1m59s
722bade383
`price_cents` stays NOT NULL and gains a default of 80.00. Where the model suggests a price the worker writes it onto the item; where it does not, the default stands.

This is cheaper to build than the nullable design it replaces — the column's type is unchanged, so the fifteen files that read `price_cents`, the cart and the checkout among them, keep working untouched, and the migration adds a default and nothing else.

It also gives up a guarantee. The storefront can now be reached by a price the admin never chose, so the protection moves out of the schema and into the review queue, where it is weaker. 80.00 is a plausible number rather than an obvious sentinel, so a default left unnoticed sells the item instead of announcing itself the way "$0.00" would. `price_source` is added to make that legible: the queue labels a price as coming from the model, the default, or the admin, and marks anything unconfirmed as such at the point of publishing. Publishing unconfirmed remains allowed — that is the decision taken — but it is stated rather than silent.

Publishing still happens only from the queue, and the notification email still carries no publish button.

Ref #220
bermudalamb added 3 commits 2026-08-29 10:28:38 -05:00
The first of four slices from the intake design, and the only one that is worth planning in detail yet — the later slices' shape depends on what this one actually produces.

Seven tasks: the schema, extracting the validated image-upload pipeline out of routes/admin.ts so the public endpoint reuses it rather than growing a near-copy of it, token generation and hashing, the admin API for issuing and revoking links, the public submission endpoint, the submission page, and the admin screen.

Three things the plan settles that the design left open or got wrong. The image caps become the constants already in the codebase rather than the 10-photo and 10 MB figures the design invented, because two different caps on one pipeline is a defect waiting to happen. The feature flag is dropped from this slice: nothing is reachable until a link exists, and the flag earns its keep in slice 2 where a paid API call appears. And the link is resolved before multer runs, so a stranger holding a bad token cannot cause a byte to be written to the uploads volume — cleanup afterwards would leave an unauthenticated caller in control of disk churn, and leans on an unlink that a crash between write and delete would skip. That ordering is asserted by a test, so a later reordering fails loudly rather than silently.

Ref #220
Two tasks added to the slice-1 plan, closing the half of the upload gap the middleware ordering does not.

The ordering fix stops a caller with a bad token writing anything. A caller with a working one can still send six eight-megabyte files per request against a limiter that allows twenty requests a window, and nothing checks whether the volume can take it. That volume is shared with the admin upload path, so intake filling it is a shop outage rather than an intake outage.

Task 8 refuses an upload when less than a gigabyte remains, on the admin item routes as well as intake, failing closed because a volume that cannot be measured is not one to assume is empty. Task 9 makes an absent cap mean the bounded default of twenty-five rather than unlimited: the router as written treated omission as "no limit", so the ordinary act of creating a link produced an unbounded one, and a cap that has to be remembered is not a control.

Two larger findings are filed rather than folded in. Re-encoding uploads to strip EXIF and cut stored bytes (#226) touches the shared pipeline and adds a native dependency; a global ceiling with an abuse alert (#227) needs its own state and an email. The EXIF one is worth stating plainly: nothing strips metadata today, so an uploaded phone photo publishes the coordinates it was taken at, at a public URL. That is already true of the admin path and is not introduced here, but this slice widens who can put such a file there.

Ref #220
docs(uploads): plan the EXIF stripping and re-encode (#226)
SonarQube Analysis / sonarqube (pull_request) Failing after 16m30s
Linting / lint (pull_request) Successful in 1m59s
65f9d00785
Five tasks: prove sharp installs where it actually runs, the re-encode policy as a pure module, wiring it into the single middleware every upload path already passes through, the backfill over already-stored photos, and the deployment sequence.

Re-encoding rather than deleting tags. Deleting requires knowing every tag that could carry something sensitive, across formats and camera makers, indefinitely; rebuilding the file from decoded pixels leaves nothing that could have been missed. Same reasoning that makes uploadTypes.ts an allowlist.

Format is preserved rather than normalised to WebP. Converting would compress better but changes every stored extension, and therefore item_images.image_path, turning the backfill into a rename with a window where rows point at files that no longer exist. A privacy fix does not need that risk, and the backfill consequently touches no database rows at all.

The backfill is lossy and irreversible, so it reports by default and needs --apply, writes to a temporary file and renames so an interruption cannot leave a half-written image being served, and is idempotent by construction: a file already stripped and already within bounds is skipped rather than put through a second lossy pass. That property is a pure function with its own unit test, because being wrong about it degrades every image a little more on every run.

Two traps the plan handles that the issue only named. An animated WebP read without the animated flag decodes to a single frame and is silently written back as a still, so the flag is set for WebP and only WebP — it changes how resize reads height, which would be wrong for the other types. And sharp before 0.33 has no withExif, which the tests need to build their fixture; on an older version they fail as though the stripping were broken.

Ref #226
bermudalamb changed title from docs(intake): design the upload-link, AI-draft and review-queue pipeline (#220) to docs(intake): design and implementation plans for the intake pipeline (#220) 2026-08-29 10:29:21 -05:00
bermudalamb merged commit 5a3c0db0bc into main 2026-08-29 10:32:49 -05:00
bermudalamb deleted branch feature/220-intake-pipeline-design 2026-08-29 10:32:49 -05:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: bermudalamb/redefined-designs#221