# Background removal: the rembg sidecar

Setup notes for the Python image-processing stack behind the review queue's background-removal option.

**Status: adopted (#281).** The engine choice is settled — see "The one thing that must not be got wrong" below. Everything here was run and measured on 2026-09-02 against `danielgatis/rembg:latest`, on a Windows dev box under Docker Desktop. **The NAS is a different machine and will be slower**; treat these as an upper bound on capability, not a promise.

## Why a sidecar and not the host's Python

The application runs in a container. Python installed on the NAS host is not reachable from inside it, and putting Python and ONNX into the application image would add roughly 300 MB to an image that is already 1.06 GB. A sidecar keeps that weight out of the deployable and lets the two be restarted independently.

## The one thing that must not be got wrong

**Always send `model=u2net` explicitly. Never rely on the default.**

The default model this image downloads is `bria-rmbg`. BRIA's RMBG models are licensed for **non-commercial use**; this is a shop. That is the same trap that ruled out `@imgly/background-removal-node`, and it is easy to walk into because it happens silently on the first request.

Specifying `u2net` is not only the licensing answer — it is better on every measured axis:

| | `bria-rmbg` (default) | `u2net` (Apache-2.0) |
|---|---|---|
| Licence | **Non-commercial** | **Apache-2.0** |
| Warm request, 2000×1500 | 14–20 s | **1.1–2.3 s** |
| Model on disk | 977 MB | 168 MB |

A ten-times speed difference decides the interaction design on its own: at 1–2 s a click can wait, at 15–20 s it cannot.

## Running it

```bash
docker volume create rembg-models

docker run -d --name rembg \
  -p 7000:7000 \
  -v rembg-models:/root/.rembg \
  danielgatis/rembg:latest s --no-ui -p 7000
```

- `--no-ui` disables the bundled Gradio interface. It is not needed for API use and the image's own help notes it reduces idle CPU — which matters on a box also running Gitea, QA and production.
- The volume is on `/root/.rembg`, **not** `/root/.u2net`. The older path appears in much of the documentation online and is wrong for this image: models land in `/root/.rembg/models/<name>/`. Mounting the wrong path silently re-downloads 168 MB on every container start.
- Startup takes about **40 seconds** before the port answers. Anything health-checking it needs to allow for that.

Add to a Portainer stack as a second service on the same network, and have the application address it by service name rather than by host port.

## The API

```
POST /api/remove
  file=@photo.jpg      (multipart)
  model=u2net          (REQUIRED — see above)
```

Returns a PNG. Verified on a 2000×1500 input:

- `format: png`, `channels: 4`, `hasAlpha: true`
- dimensions preserved exactly
- alpha spans `0`–`255`, so it is a genuine cut-out rather than a flattened image with an alpha channel bolted on
- output was **1.7×** the input JPEG's bytes

The first call for a given model downloads it (about 17 s for `u2net`), so the first request after a fresh volume is slow. Pre-warm it rather than letting a person meet that delay:

```bash
curl -s -F "file=@any.jpg" -F "model=u2net" -o /dev/null http://localhost:7000/api/remove
```

`rembg d` downloads models without serving, if a build step is preferred.

## Costs

| | |
|---|---|
| Image | **4.24 GB** |
| `u2net` model in the volume | 168 MB |
| Application image, unchanged | 1.06 GB |

4.24 GB is the honest number and the main argument against this route. It buys no per-image cost, no external service, no credential, and no data leaving the NAS.

## What has not been established

- **Quality on a real photograph.** Everything above used a generated rectangle on a flat ground, which is trivially easy. The timings are sound because they depend on pixel count, but nothing here says how `u2net` handles a chipped vase on a patterned rug, and that is what consignors will send.
- **Behaviour on the NAS.** Slower, by an unknown factor.
- **Concurrency.** `--threads` exists and was not exercised. One request at a time is the safe assumption.

## Alternatives ruled out

**`@imgly/background-removal-node`** — AGPL, with commercial terms on request. Also pins `sharp ~0.32` against the project's `^0.35`, and a second sharp in the tree has caused a silent breakage here before.

**`rembg-node`** (the npm package, not this image) — MIT, and pulls no sharp, which looked ideal. But it downloads its model from a **Google Drive link**, which failed outright when tested, and its `modelPath` is readonly so a pre-baked model cannot be supplied through the API.

**A hosted API** (remove.bg, Photoroom) — clear terms, no disk cost, best quality. Roughly $0.20 an image, another credential, another external dependency in the pipeline, and would want the budget ceiling that #227 gave submissions.

## How the application uses it

`backend/src/intake/rembgClient.ts` is the only thing that talks to the sidecar. It posts to `/api/remove` and **always sends `model=u2net`**; a unit test asserts that parameter is present, because nothing about the returned image would reveal its absence.

`REMBG_URL` points at it — `http://rembg-syn:7000` on the NAS, where the container publishes `32700:7000`. The variable is optional in both compose files: unset means the submission page shows no checkbox, the review queue shows no control, and the drafting worker skips the step. An environment without a sidecar is a working environment.

Two entry points, one module (`backend/src/intake/backgroundRemoval.ts`):

- **The drafting worker**, honouring the checkbox on `/submit/:token`, which is ticked by default. The submitter's tick is recorded and acted on later, so nobody waits on a model run and an unreachable sidecar cannot fail an upload.
- **The review queue**, per photo, synchronously — 1.1–2.3 s warm is a wait an admin who just clicked a button can absorb.

Because removal follows drafting in the worker, an environment with no `ANTHROPIC_API_KEY` drafts nothing and so cuts out nothing. The per-photo control in the review queue is the way to do it by hand there.

The original file is never destroyed. `item_images.original_image_path` records where it went, and **Restore original** swaps it back. Nothing in the feature deletes a file or a row.