redefined-designs/docs/superpowers/specs/2026-08-29-intake-pipeline-design.md

# Intake Pipeline — Design

**Issue:** [#220 — Shared upload links, AI-drafted listings, and an admin review queue](https://gitea.bermudalamb.synology.me/bermudalamb/redefined-designs/issues/220)
**Date:** 2026-08-29
**Status:** Draft

## Goal

Someone who is not the admin can send in photos of one item through a link they were given. A draft listing is written for it automatically. The admin is told by email, and the item sits unpublished until the admin has read the copy, checked the price and published it deliberately.

## Where this starts from

Most of the lifecycle this feature needs is already built, which is why the design below adds a pipeline in front of it rather than a parallel one beside it.

- **`pending` is already the unpublished state.** `items.status` defaults to `'pending'` since #90, `NON_PUBLIC_STATUSES` keeps it out of every public and storefront query, and `POST /admin/items/:id/mark-available` and `/unpublish` already move an item across that line. Nothing here needs a new visibility concept, and inventing one would mean a second thing to keep correct.
- **Uploads are already hardened.** Multer writes to `UPLOADS_DIR`, `uploadTypes.ts` checks the declared type against the file's leading bytes, the allowlist is three image types, and `uploads.ts` refuses to serve anything it does not recognise. A new upload path that did not go through those is a new hole.
- **Mail already exists,** including admin-editable markdown bodies with placeholder validation, the `MAIL_ALLOWLIST` guard that stops non-production environments mailing real people, and `PUBLIC_URL` for building links.
- **Background work has a pattern.** `cron.schedule('0 9 * * *', ...)` in `server.ts` drives the cart reminders. This is a single-instance deployment, which is what makes in-process scheduling and the in-memory rate-limit store tenable.

Three things do not exist: a way in for someone without an admin account, any LLM integration at all, and anywhere to review a draft.

## Decisions

Settled in conversation before this was written, and recorded on the issue.

| Question | Decision |
| --- | --- |
| What the "shared folder" is | A web upload page, not a watched NAS share or a mailbox |
| Who can upload | Anyone holding a named, revocable link — no account |
| What the uploader supplies | Photos of one item, plus one free-text note |
| What the AI drafts | Name, marketing description, category and tags from the existing taxonomy, and a suggested price |
| Where a submission lives pre-approval | An `items` row at `status='pending'`, not a separate submissions table |
| How the admin acts on it | Signed one-click links in the email, plus a review queue in the admin |
| Whether one click can publish | No. Publishing always happens from the review queue |
| What an item is priced at on arrival | The AI's suggestion, or 80.00 when it has none |
| Model | Sonnet 5, in an env var |

## The invariant

**Nothing reaches the storefront without the admin publishing it from the queue.** The item arrives `pending`, which is already invisible to every public query, and only `mark-available` moves it. The email carries no publish button, so the admin has necessarily seen the item and its price before anything ships.

The price is the deliberate exception, and it is worth being precise about what was traded. An arriving item is priced immediately — the model's suggestion if it made one, otherwise 80.00 — so the review queue's price field is pre-filled rather than blank. The admin can change it, but is not forced to, and an unchanged field publishes at a number the admin did not choose.

That is a real risk and it is accepted knowingly. It is bounded by the queue: publishing is a deliberate act on a screen showing the price, not something that can happen from an inbox or by a timeout. What is given up is the stronger guarantee that a machine-guessed price could never reach the storefront at all.

Two consequences follow, and both are load-bearing:

- **80.00 is a plausible price, not an obvious sentinel.** `0` would render as "$0.00" and read as a bug to anyone who saw it; 80.00 renders as a decision. A default that went unnoticed therefore sells the item rather than announcing itself. This is the argument for surfacing it loudly in the queue rather than for choosing a different number.
- **The queue must make the price's provenance visible.** A price field is not enough. The queue shows whether the number came from the model, from the 80.00 default, or from the admin, so "nobody has looked at this price" is legible at a glance instead of being indistinguishable from a considered one.

## Architecture

### Schema

Three migrations.

```sql
CREATE TABLE upload_links (
  id SERIAL PRIMARY KEY,
  label TEXT NOT NULL,
  -- The token is never stored. A leaked database is not also a leaked set of
  -- working upload links, and the admin screen can show a token exactly once,
  -- at creation, for the same reason a password reset link is not re-readable.
  token_hash TEXT NOT NULL UNIQUE,
  revoked_at TIMESTAMPTZ,
  submission_count INTEGER NOT NULL DEFAULT 0,
  -- Null means no cap. A link handed to a regular contributor is open-ended;
  -- one handed out for a single box of stock is not.
  max_submissions INTEGER,
  last_used_at TIMESTAMPTZ,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE item_drafts (
  id SERIAL PRIMARY KEY,
  item_id INTEGER NOT NULL UNIQUE REFERENCES items(id) ON DELETE CASCADE,
  -- SET NULL rather than CASCADE: deleting a link must not delete the items
  -- that came in through it. Provenance is lost; the goods are not.
  upload_link_id INTEGER REFERENCES upload_links(id) ON DELETE SET NULL,
  submitter_note TEXT,
  state TEXT NOT NULL DEFAULT 'queued',   -- queued | drafting | ready | failed | discarded
  attempts INTEGER NOT NULL DEFAULT 0,
  model TEXT,
  ai_name TEXT,
  ai_description TEXT,
  ai_category_id INTEGER REFERENCES categories(id) ON DELETE SET NULL,
  ai_tag_names TEXT[],
  -- Kept even though the suggestion is also copied onto the item, so what the
  -- model proposed stays readable after the admin has edited the item's price.
  -- Without it there is no way to ask later whether the model's numbers were
  -- any good.
  ai_suggested_price_cents INTEGER,
  -- ai | default | admin. What the item's current price actually came from.
  -- Derivable by comparing three numbers, but only fragilely: a model that
  -- happens to suggest exactly 8000, or an admin who deliberately types the
  -- model's number, both collapse the comparison. Recorded rather than
  -- inferred, because the queue uses it to say "nobody has chosen this price".
  price_source TEXT NOT NULL DEFAULT 'default',
  ai_error TEXT,
  input_tokens INTEGER,
  output_tokens INTEGER,
  cost_micros INTEGER,
  drafted_at TIMESTAMPTZ,
  reviewed_at TIMESTAMPTZ,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- An arriving item is always priced, so the column stays NOT NULL and only
-- gains a fallback. 80.00 applies when the model declined to suggest anything;
-- a suggestion, when there is one, is written over it by the worker.
ALTER TABLE items ALTER COLUMN price_cents SET DEFAULT 8000;
```

**On `price_cents` staying NOT NULL.** The alternative considered was making it nullable so that "no price yet" was expressible, with the publish path refusing an unpriced item. That was rejected in favour of always having a price.

The practical effect is that this design costs far less to build. `price_cents` is referenced across fifteen files including the cart and the checkout, and every one of them keeps working untouched — the column's type does not change, so nothing downstream has to learn about an item that has no price. The migration adds a default and nothing else.

What it costs is covered under the invariant above: the storefront can now be reached by a price the admin never chose, so the protection moves from the schema into the queue's presentation, where it is weaker. `price_source` exists to make that presentation possible.

The number lives in the migration rather than in configuration. Changing a default price is a rare, deliberate act with a record attached, which is the right amount of friction; an env var would let it drift silently between environments, and a wrong default is not visible anywhere until something has already sold at it.

### Ingestion

`POST /api/intake/:token` — public, unauthenticated, `uploadImages` for the files.

1. Hash the token, look up a matching `upload_links` row that is not revoked. No match, revoked, or over its cap: `404`. Not `403` — whether a link exists is not something a stranger needs to be able to distinguish, which is the reasoning `uploads.ts` already applies.
2. Validate the photos through the existing `uploadTypes` checks. Caps: at most 10 photos, at most 10 MB each.
3. In one transaction: insert the `items` row (`status='pending'`, no `price_cents` given so the 8000 default applies, a placeholder name — the submission timestamp — replaced by the draft or the admin), insert `item_images`, insert `item_drafts` at `state='queued'` and `price_source='default'`, bump the link's counter.
4. Respond immediately. The uploader is told it arrived and that a person will look at it. **The AI is not called here** — a slow or failing API call must not turn into a failed upload for someone who did nothing wrong.

Rate limited by IP through a new limiter in `rateLimit.ts`. `keyByCallerAndEmail` does not fit — there is no email — so this needs its own key function over `ipKeyGenerator` alone, and the existing comment about a bare `ip:` bucket being a shared allowance applies and is accepted here: the link itself is the per-caller identity, and its counter is the per-caller cap.

### The drafting worker

A new `backend/src/intakeDrafting.ts`, driven both by a call at the end of a successful submission and by a `node-cron` sweeper that picks up anything left `queued` or stuck in `drafting`. The sweeper is what makes a restart mid-draft recoverable rather than a permanently stalled row.

The request, via `@anthropic-ai/sdk` and `client.messages.parse()` with `zodOutputFormat`, so the shape is validated rather than parsed out of prose:

- Every photo as a base64 `image` block.
- The submitter's note verbatim, clearly framed as **the only trustworthy factual claims available**. The system prompt says plainly: describe what is visible, use the note for anything not visible, and never state a material, age, maker or provenance that is in neither. On a one-of-a-kind item, an invented "1930s hand-thrown stoneware" is not a cosmetic error — it is a false claim on a storefront.
- The existing categories and tags as the closed set to choose from, so the draft lands inside the taxonomy the filters already work on.
- The suggested price framed as a rough starting point.

The note is untrusted input from an unauthenticated stranger. It is passed as data, and nothing the model returns is executed, interpolated into SQL, or rendered as HTML — the drafted description goes through the same markdown-with-`html: false` treatment as every other stored body, which is what makes "a model wrote this" and "a person wrote this" equally safe to render.

Result: `state='ready'`, the fields populated, token counts and computed cost recorded. Where the model returned a price, it is written onto the item and `price_source='ai'`; where it did not, the item keeps the 8000 default and `price_source` stays `'default'`. The suggested price is recorded on the draft either way.

Failure: `attempts` incremented, `ai_error` stored, and after three attempts `state='failed'` — which still leaves a perfectly good pending item with photos in the review queue, just with no draft copy and priced at the default. A failed AI call must never lose someone's submission.

**Spend ceiling.** A monthly cap (`INTAKE_MONTHLY_BUDGET_USD`) summed from `cost_micros`. Past it, submissions still save and still notify; only the API call is skipped, with `state='failed'` and an error saying so. The ceiling protects the bill, and it must not be the thing that loses inventory.

### Notification and signed links

Reuses `emailTemplates.ts` with a new `intakeDraft` key, so the copy is editable from the settings screen like every other template, with `reviewUrl` required.

Action links are HMAC-signed with a new `INTAKE_ACTION_SECRET` over `(draftId, action, expiry)`, verified with a timing-safe comparison, good for 30 days.

- **Regenerate** — back to `state='queued'`.
- **Discard** — `state='discarded'` and the item unpublished; recoverable from the queue, because a one-click destructive action reachable from an inbox should not be final.
- **Review & publish** — an ordinary deep link into the admin queue, behind authentik like the rest of `/admin`. It carries no signature and grants nothing.

The two signed actions are deliberately the ones whose worst case is a wasted API call or a recoverable hide. Nothing a single click can do puts an item on the storefront.

### The review queue

A new admin screen listing drafts by state: photos, the submitter's note, which link it came from, the drafted copy in editable fields, the price, and Publish / Regenerate / Discard. Publish writes the edited values onto the item and calls the existing `mark-available`.

The price field carries the weight the schema no longer does, so it is not an ordinary input. It is pre-filled, and it is labelled with where the number came from — the model, or the 80.00 default, or the admin — with anything that is not `price_source='admin'` marked visibly as unconfirmed. Editing it sets `price_source='admin'`. Publishing something still marked unconfirmed is allowed, because that is the decision taken, but it says so plainly at the point of publishing rather than after.

Reuses the existing admin item components where they fit rather than growing a second item editor.

### Configuration

New variables, added to `envValidation.ts` and — per #107, enforced by `composeEnvironment.test.ts` — to both compose files if they go on the required list.

| Variable | Requirement |
| --- | --- |
| `ANTHROPIC_API_KEY` | Required when intake is enabled |
| `INTAKE_MODEL` | Optional, defaults to `claude-sonnet-5` |
| `INTAKE_MONTHLY_BUDGET_USD` | Optional, defaults to `20` |
| `INTAKE_ACTION_SECRET` | Required when intake is enabled |

Intake is feature-flagged off by default, so none of these become required for an environment that does not run it. `PUBLIC_URL` is already required alongside SMTP and is what the review links are built from.

## Error handling

The through-line: **a submission is the only irreplaceable thing here.** Photos of a one-of-a-kind object may not be retakeable — the item may not be in the sender's hands any more. Every failure mode below is arranged so the photos survive it.

| Failure | Behaviour |
| --- | --- |
| API call fails or times out | Retry to three attempts, then `state='failed'`. Item and photos intact, reviewable with no draft, priced at the default |
| Budget exhausted | Submission saved, draft skipped, admin still notified, item priced at the default |
| Malformed model output | Rejected by the schema, counts as a failed attempt |
| SMTP down | Draft still `ready` and visible in the queue; the queue, not the email, is the source of truth |
| Link revoked mid-upload | `404`. Already-submitted items are unaffected |
| Restart mid-draft | The cron sweeper re-queues anything stuck in `drafting` |

## Testing

- **Unit** — token hashing and constant-time verification, HMAC signing and expiry, the budget calculation, the prompt builder's handling of an empty note, and the `price_source` transitions. All pure, in the style of `keyByCallerAndEmail` and `isAllowedRecipient` being exported specifically to be tested directly.
- **Integration** — submission through a valid link creates a pending item with images, a queued draft and the 8000 default; a model suggestion overwrites that price and sets `price_source='ai'`; a failed or budget-skipped draft leaves the default in place; a revoked link gets `404`; a signed action link works once and an expired or tampered one does not. The Anthropic client is stubbed; no test spends money.
- **E2E** — the submission page, and the queue round trip from draft to published. Assertions scoped to the item under test rather than the whole grid, since the dev database never truncates.

## Out of scope

Contributor accounts, a watched NAS folder, email-in submission, multi-item submissions, image editing or cropping, and regenerating with a steer ("try again, warmer"). All are additive later; none change the schema above.