docs(intake): design the upload-link, AI-draft and review-queue pipeline (#220)

A named, revocable link lets someone without an account send in photos of one item plus a note; a background worker drafts the listing; the admin is emailed and publishes it deliberately from a review queue. Most of the lifecycle already exists and is reused rather than rebuilt: `pending` has been the unpublished state since #90 and is already excluded from every public query, the upload path already validates magic bytes against a three-type allowlist, mail already has editable templates and an allowlist guard, and node-cron is already the background-work pattern. What is new is a way in for someone with no admin account, the first LLM integration in this codebase, and somewhere to review a draft. The design turns on one invariant: nothing reaches the storefront at a price a model guessed. The suggested price lives on the draft and never on the item, the email carries no publish button, and the publish path refuses an item with no price. That is also why `price_cents` becomes nullable rather than defaulting to zero — a sentinel that formats as "$0.00" is the same class of quiet failure as `DEMO_MODE` once being "demo unless the value is exactly false", and nullability makes the compiler enumerate all fifteen call sites instead. Ref #220
2026-08-29 07:43:47 -05:00
parent e933cd19f9
commit dbe6a0cf8f
1 changed files with 178 additions and 0 deletions
@@ -0,0 +1,178 @@
 # Intake Pipeline — Design
 **Issue:** [#220 — Shared upload links, AI-drafted listings, and an admin review queue](https://gitea.bermudalamb.synology.me/bermudalamb/redefined-designs/issues/220)
 **Date:** 2026-08-29
 **Status:** Draft
 ## Goal
 Someone who is not the admin can send in photos of one item through a link they were given. A draft listing is written for it automatically. The admin is told by email, and the item sits unpublished until the admin has read the copy, typed a price and published it deliberately.
 ## Where this starts from
 Most of the lifecycle this feature needs is already built, which is why the design below adds a pipeline in front of it rather than a parallel one beside it.
 - **`pending` is already the unpublished state.** `items.status` defaults to `'pending'` since #90, `NON_PUBLIC_STATUSES` keeps it out of every public and storefront query, and `POST /admin/items/:id/mark-available` and `/unpublish` already move an item across that line. Nothing here needs a new visibility concept, and inventing one would mean a second thing to keep correct.
 - **Uploads are already hardened.** Multer writes to `UPLOADS_DIR`, `uploadTypes.ts` checks the declared type against the file's leading bytes, the allowlist is three image types, and `uploads.ts` refuses to serve anything it does not recognise. A new upload path that did not go through those is a new hole.
 - **Mail already exists,** including admin-editable markdown bodies with placeholder validation, the `MAIL_ALLOWLIST` guard that stops non-production environments mailing real people, and `PUBLIC_URL` for building links.
 - **Background work has a pattern.** `cron.schedule('0 9 * * *', ...)` in `server.ts` drives the cart reminders. This is a single-instance deployment, which is what makes in-process scheduling and the in-memory rate-limit store tenable.
 Three things do not exist: a way in for someone without an admin account, any LLM integration at all, and anywhere to review a draft.
 ## Decisions
 Settled in conversation before this was written, and recorded on the issue.
 | Question | Decision |
 | --- | --- |
 | What the "shared folder" is | A web upload page, not a watched NAS share or a mailbox |
 | Who can upload | Anyone holding a named, revocable link — no account |
 | What the uploader supplies | Photos of one item, plus one free-text note |
 | What the AI drafts | Name, marketing description, category and tags from the existing taxonomy, and a suggested price |
 | Where a submission lives pre-approval | An `items` row at `status='pending'`, not a separate submissions table |
 | How the admin acts on it | Signed one-click links in the email, plus a review queue in the admin |
 | Whether one click can publish | No. Publishing always requires a typed price |
 | Model | Sonnet 5, in an env var |
 ## The invariant
 **Nothing reaches the storefront at a price a model guessed.** Everything below follows from that: the suggested price lives on the draft and never on the item, the email has no publish button, and the publish path refuses an item with no price.
 ## Architecture
 ### Schema
 Three migrations.
 ```sql
 CREATE TABLE upload_links (
  id SERIAL PRIMARY KEY,
  label TEXT NOT NULL,
  -- The token is never stored. A leaked database is not also a leaked set of
  -- working upload links, and the admin screen can show a token exactly once,
  -- at creation, for the same reason a password reset link is not re-readable.
  token_hash TEXT NOT NULL UNIQUE,
  revoked_at TIMESTAMPTZ,
  submission_count INTEGER NOT NULL DEFAULT 0,
  -- Null means no cap. A link handed to a regular contributor is open-ended;
  -- one handed out for a single box of stock is not.
  max_submissions INTEGER,
  last_used_at TIMESTAMPTZ,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
 );
 CREATE TABLE item_drafts (
  id SERIAL PRIMARY KEY,
  item_id INTEGER NOT NULL UNIQUE REFERENCES items(id) ON DELETE CASCADE,
  -- SET NULL rather than CASCADE: deleting a link must not delete the items
  -- that came in through it. Provenance is lost; the goods are not.
  upload_link_id INTEGER REFERENCES upload_links(id) ON DELETE SET NULL,
  submitter_note TEXT,
  state TEXT NOT NULL DEFAULT 'queued',   -- queued | drafting | ready | failed | discarded
  attempts INTEGER NOT NULL DEFAULT 0,
  model TEXT,
  ai_name TEXT,
  ai_description TEXT,
  ai_category_id INTEGER REFERENCES categories(id) ON DELETE SET NULL,
  ai_tag_names TEXT[],
  -- Deliberately not on items. See the invariant.
  ai_suggested_price_cents INTEGER,
  ai_error TEXT,
  input_tokens INTEGER,
  output_tokens INTEGER,
  cost_micros INTEGER,
  drafted_at TIMESTAMPTZ,
  reviewed_at TIMESTAMPTZ,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
 );
 ALTER TABLE items ALTER COLUMN price_cents DROP NOT NULL;
 ```
 **On making `price_cents` nullable.** This is the most invasive line in the design: `price_cents` is referenced across fifteen files including the cart and the checkout. The alternative is to insert a submission at `0` and guard publishing on `price_cents > 0`, which changes no types and no other file.
 It is still the wrong choice. Zero is a real number that formats as a real price, and a sentinel that renders as "$0.00" in a shop is precisely the failure this codebase already has a scar from — `DEMO_MODE` meaning "demo unless the value is exactly false" quietly stopped the shop charging anyone. Null cannot be formatted by accident. And the blast radius is not hidden work but the opposite: making the column nullable makes the TypeScript type `number | null`, and the compiler then names every one of those fifteen sites and refuses to build until each has said what it does with an unpriced item. A sentinel gets the same reach with none of the enumeration.
 The publish guard belongs in `mark-available` alongside the existing status checks, and is worth an integration test of its own.
 ### Ingestion
 `POST /api/intake/:token` — public, unauthenticated, `uploadImages` for the files.
 1. Hash the token, look up a matching `upload_links` row that is not revoked. No match, revoked, or over its cap: `404`. Not `403` — whether a link exists is not something a stranger needs to be able to distinguish, which is the reasoning `uploads.ts` already applies.
 2. Validate the photos through the existing `uploadTypes` checks. Caps: at most 10 photos, at most 10 MB each.
 3. In one transaction: insert the `items` row (`status='pending'`, `price_cents=NULL`, a placeholder name — the submission timestamp — replaced by the draft or the admin), insert `item_images`, insert `item_drafts` at `state='queued'`, bump the link's counter.
 4. Respond immediately. The uploader is told it arrived and that a person will look at it. **The AI is not called here** — a slow or failing API call must not turn into a failed upload for someone who did nothing wrong.
 Rate limited by IP through a new limiter in `rateLimit.ts`. `keyByCallerAndEmail` does not fit — there is no email — so this needs its own key function over `ipKeyGenerator` alone, and the existing comment about a bare `ip:` bucket being a shared allowance applies and is accepted here: the link itself is the per-caller identity, and its counter is the per-caller cap.
 ### The drafting worker
 A new `backend/src/intakeDrafting.ts`, driven both by a call at the end of a successful submission and by a `node-cron` sweeper that picks up anything left `queued` or stuck in `drafting`. The sweeper is what makes a restart mid-draft recoverable rather than a permanently stalled row.
 The request, via `@anthropic-ai/sdk` and `client.messages.parse()` with `zodOutputFormat`, so the shape is validated rather than parsed out of prose:
 - Every photo as a base64 `image` block.
 - The submitter's note verbatim, clearly framed as **the only trustworthy factual claims available**. The system prompt says plainly: describe what is visible, use the note for anything not visible, and never state a material, age, maker or provenance that is in neither. On a one-of-a-kind item, an invented "1930s hand-thrown stoneware" is not a cosmetic error — it is a false claim on a storefront.
 - The existing categories and tags as the closed set to choose from, so the draft lands inside the taxonomy the filters already work on.
 - The suggested price framed as a rough starting point.
 The note is untrusted input from an unauthenticated stranger. It is passed as data, and nothing the model returns is executed, interpolated into SQL, or rendered as HTML — the drafted description goes through the same markdown-with-`html: false` treatment as every other stored body, which is what makes "a model wrote this" and "a person wrote this" equally safe to render.
 Result: `state='ready'`, the fields populated, token counts and computed cost recorded. Failure: `attempts` incremented, `ai_error` stored, and after three attempts `state='failed'` — which still leaves a perfectly good pending item with photos in the review queue, just with no draft copy. A failed AI call must never lose someone's submission.
 **Spend ceiling.** A monthly cap (`INTAKE_MONTHLY_BUDGET_USD`) summed from `cost_micros`. Past it, submissions still save and still notify; only the API call is skipped, with `state='failed'` and an error saying so. The ceiling protects the bill, and it must not be the thing that loses inventory.
 ### Notification and signed links
 Reuses `emailTemplates.ts` with a new `intakeDraft` key, so the copy is editable from the settings screen like every other template, with `reviewUrl` required.
 Action links are HMAC-signed with a new `INTAKE_ACTION_SECRET` over `(draftId, action, expiry)`, verified with a timing-safe comparison, good for 30 days.
 - **Regenerate** — back to `state='queued'`.
 - **Discard** — `state='discarded'` and the item unpublished; recoverable from the queue, because a one-click destructive action reachable from an inbox should not be final.
 - **Review & publish** — an ordinary deep link into the admin queue, behind authentik like the rest of `/admin`. It carries no signature and grants nothing.
 The two signed actions are deliberately the ones whose worst case is a wasted API call or a recoverable hide. Nothing a single click can do puts an item on the storefront.
 ### The review queue
 A new admin screen listing drafts by state: photos, the submitter's note, which link it came from, the drafted copy in editable fields, the suggested price shown as a suggestion beside an empty price input, and Publish / Regenerate / Discard. Publish writes the edited values onto the item and calls the existing `mark-available`.
 Reuses the existing admin item components where they fit rather than growing a second item editor.
 ### Configuration
 New variables, added to `envValidation.ts` and — per #107, enforced by `composeEnvironment.test.ts` — to both compose files if they go on the required list.
 | Variable | Requirement |
 | --- | --- |
 | `ANTHROPIC_API_KEY` | Required when intake is enabled |
 | `INTAKE_MODEL` | Optional, defaults to `claude-sonnet-5` |
 | `INTAKE_MONTHLY_BUDGET_USD` | Optional, defaults to `20` |
 | `INTAKE_ACTION_SECRET` | Required when intake is enabled |
 Intake is feature-flagged off by default, so none of these become required for an environment that does not run it. `PUBLIC_URL` is already required alongside SMTP and is what the review links are built from.
 ## Error handling
 The through-line: **a submission is the only irreplaceable thing here.** Photos of a one-of-a-kind object may not be retakeable — the item may not be in the sender's hands any more. Every failure mode below is arranged so the photos survive it.
 | Failure | Behaviour |
 | --- | --- |
 | API call fails or times out | Retry to three attempts, then `state='failed'`. Item and photos intact, reviewable with no draft |
 | Budget exhausted | Submission saved, draft skipped, admin still notified |
 | Malformed model output | Rejected by the schema, counts as a failed attempt |
 | SMTP down | Draft still `ready` and visible in the queue; the queue, not the email, is the source of truth |
 | Link revoked mid-upload | `404`. Already-submitted items are unaffected |
 | Restart mid-draft | The cron sweeper re-queues anything stuck in `drafting` |
 ## Testing
 - **Unit** — token hashing and constant-time verification, HMAC signing and expiry, the budget calculation, the prompt builder's handling of an empty note, and the publish guard's price rule. All pure, in the style of `keyByCallerAndEmail` and `isAllowedRecipient` being exported specifically to be tested directly.
 - **Integration** — submission through a valid link creates a pending item with images and a queued draft; a revoked link gets `404`; publishing without a price is refused; a signed action link works once and an expired or tampered one does not. The Anthropic client is stubbed; no test spends money.
 - **E2E** — the submission page, and the queue round trip from draft to published. Assertions scoped to the item under test rather than the whole grid, since the dev database never truncates.
 ## Out of scope
 Contributor accounts, a watched NAS folder, email-in submission, multi-item submissions, image editing or cropping, and regenerating with a steer ("try again, warmer"). All are additive later; none change the schema above.