chore: replace manual SQL migrations with node-pg-migrate #7

Merged
bermudalamb merged 1 commits from feat/node-pg-migrate into main 2026-08-14 10:06:03 -05:00
8 changed files with 284 additions and 75 deletions
Showing only changes of commit 7f4479605a - Show all commits
+64
View File
@@ -0,0 +1,64 @@
# Redefined Designs — Project Context
## What this is
A storefront for one-of-a-kind items (each item has quantity 1 — once sold, it's gone). Built solo, deployed to a personal Synology NAS homelab via Docker/Portainer, behind Nginx Proxy Manager + authentik SSO for the admin panel.
**Repo**: `https://gitea.bermudalamb.synology.me/bermudalamb/redefined-designs`
**Production URL**: `https://redefined-designs.bermudalamb.synology.me`
**Local dev**: see `README.md`
## Tech stack
- **Backend**: Express + TypeScript, Postgres (`pg`), single Docker image
- **Frontend**: React + TypeScript + antd (Vite build), served as static files by the backend in production
- **Auth**: two separate systems —
- **Customers**: email/password, bcrypt, session cookie (`rd_session`), custom-built (not authentik)
- **Admin**: authentik SSO via NPM forward-auth, gated only on `/admin` and `/api/admin/*` — the storefront itself is public
- **Payments**: PayPal Orders v2 API (multi-item cart checkout, single transaction)
- **Shipping validation**: USPS Addresses API (OAuth2/REST, not the deprecated Web Tools XML API) — optional, degrades gracefully if unconfigured
- **Email**: nodemailer via Gmail SMTP — optional, degrades gracefully if unconfigured (logs a warning, skips sending)
- **Testing**: Jest (unit + integration against a disposable tmpfs Postgres), Playwright (e2e)
- **CI**: Gitea Actions — two workflows, `sonarqube.yml` (static analysis + TS build check) and `tests.yml` (unit/integration/e2e with job summaries)
## Data model (key tables)
- `items` — one-of-a-kind inventory. `status`: `available | reserved | sold`
- `item_images` — multiple images per item (front/back/etc.), ordered
- `customers` — accounts, with `marketing_consent` (explicit opt-in, GDPR-style — unchecked by default, timestamped, unsubscribe token)
- `carts` / `cart_items` — one cart per customer, items reserved with an admin-configurable expiry (`admin_settings.cart_expiry_hours`, default 24h)
- `shipping_addresses` — per-customer, USPS-validated when configured
- `checkouts` / `checkout_items` — one row per multi-item PayPal/demo transaction, snapshotting item+price at purchase time
- `orders` — per-item purchase records, linked to `checkout_id` and `customer_id`
## Key architectural decisions
1. **Cart replaced the old single-item "Buy Now" flow entirely.** Old `routes/paypal.ts` / `routes/demo.ts` (single-item) exist on disk but are unmounted — superseded by `routes/cartCheckout.ts` which does real multi-item PayPal orders (one `purchase_unit` with an itemized `items[]` breakdown, single transaction, single capture).
2. **One-of-a-kind + double-sell prevention**: adding to cart immediately flips `item.status` to `reserved` inside a DB transaction with `FOR UPDATE` locking — no two customers can hold the same item. Expired cart holds are swept every 5 minutes back to `available`.
3. **Demo mode** (`DEMO_MODE=true` env var): lets the whole site — including checkout — work with zero PayPal/USPS credentials configured, useful for local dev and just generally having a working fallback. This "degrade gracefully when a third-party integration isn't configured" pattern is used consistently: PayPal, USPS, SMTP all no-op safely if unset rather than crashing.
4. **GDPR/consent handling**: marketing consent is never pre-checked, is timestamped with the exact consent text shown, and customers have working self-service data export + account deletion endpoints (`/api/customers/me/export`, `DELETE /api/customers/me`).
5. **Admin panel security boundary**: NPM's Advanced nginx config only wraps `auth_request` around `location ~ ^/(admin|api/admin)` — everything else (storefront, cart, checkout, PayPal webhook) bypasses authentik entirely. This is a deliberate split, not an oversight — don't accidentally widen or narrow that regex without checking both directions.
6. **Cookie `secure` flag** is gated on `NODE_ENV === 'production'`, not hardcoded `true` — otherwise integration tests (plain HTTP, no TLS) silently fail to persist sessions. Learned this the hard way once already.
## Conventions (apply to all future work on this repo)
- **Every commit message follows [Conventional Commits](https://www.conventionalcommits.org/)**: `feat:`, `fix:`, `chore:`, `docs:`, `test:`, `ci:`, `refactor:`, etc.
- **Never commit directly to `main`.** Always branch: `feature/<short-description>` or `fix/<short-description>`. Open a PR, merge, branch auto-deletes (repo setting is on).
- File edits are made directly in VS Code (project is cloned locally) and pushed via PowerShell `git`, not the old NAS-SSH-with-Docker-wrapped-git workflow from earlier in this project's history (that was only used before the repo existed locally in VS Code).
- Production deploys are manual: `git pull` on the NAS copy → `docker build --no-cache` → stop/rm the container → redeploy the Portainer stack. No CD pipeline exists yet.
## Known gaps / natural next steps
- **Tests don't cover the cart/checkout/shipping feature yet** — only the pre-cart backend surface has unit/integration tests. Worth adding before trusting this in front of real customers.
- **No CD** — the Gitea Actions workflows run tests/analysis but don't deploy. Manual NAS rebuild is still required after every merge.
- **SonarQube CI still uses the `admin` token**, not a dedicated `gitea-ci` user (that was flagged as a to-do on the original .NET project this pattern was copied from, never circled back to for this repo).
- **USPS/PayPal credentials**: check whether these are actually populated in the Portainer stack env vars before assuming checkout works end-to-end in production — `DEMO_MODE` may still be the only thing actually exercised.
- **Double opt-in for marketing email** was discussed and deliberately deferred (single opt-in + easy unsubscribe was judged sufficient for now) — revisit if EU customer volume grows.
- **Daily cart reminder emails** fire via `node-cron` inside the app process at 9am container-local time — if the container restarts frequently or memory pressure causes crashes (a known NAS constraint per the broader homelab), reminders could silently stop firing with no alerting on that failure mode.
## Where to look first for common tasks
- Add/change a DB table → `backend/init.sql` (fresh deploys) **and** a one-off migration run manually via `docker exec ... psql` against the live DB (no migration framework — see `backend/migrations/` for the pattern used so far)
- Change cart/checkout behavior → `backend/src/routes/cartCheckout.ts` (the whole reserve → checkout → complete lifecycle lives here)
- Change admin-configurable settings → `admin_settings` table + `backend/src/routes/adminSettings.ts` + `frontend/src/admin/Settings.tsx`
- NPM/authentik/DSM reverse-proxy config for this app → not in this repo; documented in the homelab's broader Claude Project knowledge base, not here
+14 -9
View File
@@ -35,18 +35,23 @@ npm run db:test:up
This starts Postgres on `localhost:55432`, database `redefined_test`.
### 2. Load the schema
### 2. Run migrations
**PowerShell:**
```powershell
Get-Content init.sql | docker exec -i redefined-designs-test-db psql -U redefined_test -d redefined_test
```
**PowerShell / bash (same command, cross-platform via Node):**
**bash:**
```bash
docker exec -i redefined-designs-test-db psql -U redefined_test -d redefined_test < init.sql
```
cd backend
$env:PGHOST="localhost"; $env:PGPORT="55432"; $env:PGUSER="redefined_test"; $env:PGPASSWORD="redefined_test"; $env:PGDATABASE="redefined_test"
npm install
npm run migrate:up
This applies every migration in `backend/migrations/` in order, tracked in a `pgmigrations` table so re-running is always safe (already-applied migrations are skipped).
To add a new migration:
npm run migrate:create -- descriptive-name
This generates a timestamped file in `backend/migrations/` with `exports.up`/`exports.down` stubs — fill in `pgm.sql(...)` for both directions.
### 3. Configure environment variables
**PowerShell:**
+25
View File
@@ -0,0 +1,25 @@
FROM node:20-bookworm-slim AS frontend-build
WORKDIR /app/frontend
COPY frontend/package.json ./
RUN npm install
COPY frontend/ ./
RUN npm run build
FROM node:20-bookworm-slim AS backend-build
WORKDIR /app/backend
COPY backend/package.json ./
RUN npm install
COPY backend/ ./
RUN npm run build
FROM node:20-bookworm-slim
WORKDIR /app
COPY --from=backend-build /app/backend/package.json ./
RUN npm install --omit=dev
COPY --from=backend-build /app/backend/dist ./dist
COPY --from=backend-build /app/backend/migrate.js ./migrate.js
COPY --from=backend-build /app/backend/migrations ./migrations
COPY --from=frontend-build /app/frontend/dist ./public
ENV NODE_ENV=production
EXPOSE 3000
CMD ["node", "dist/server.js"]
-59
View File
@@ -1,59 +0,0 @@
CREATE TABLE IF NOT EXISTS items (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
description TEXT,
price_cents INTEGER NOT NULL,
status TEXT NOT NULL DEFAULT 'available',
reserved_until TIMESTAMPTZ,
sold_at TIMESTAMPTZ,
paypal_order_id TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS item_images (
id SERIAL PRIMARY KEY,
item_id INTEGER NOT NULL REFERENCES items(id) ON DELETE CASCADE,
image_path TEXT NOT NULL,
sort_order INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS customers (
id SERIAL PRIMARY KEY,
email TEXT NOT NULL UNIQUE,
password_hash TEXT NOT NULL,
name TEXT,
email_verified BOOLEAN NOT NULL DEFAULT false,
marketing_consent BOOLEAN NOT NULL DEFAULT false,
marketing_consent_at TIMESTAMPTZ,
marketing_consent_text TEXT,
unsubscribe_token TEXT NOT NULL UNIQUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS customer_sessions (
token TEXT PRIMARY KEY,
customer_id INTEGER NOT NULL REFERENCES customers(id) ON DELETE CASCADE,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS customer_tokens (
token TEXT PRIMARY KEY,
customer_id INTEGER NOT NULL REFERENCES customers(id) ON DELETE CASCADE,
kind TEXT NOT NULL,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS orders (
id SERIAL PRIMARY KEY,
item_id INTEGER REFERENCES items(id),
customer_id INTEGER REFERENCES customers(id) ON DELETE SET NULL,
processor TEXT NOT NULL,
processor_order_id TEXT,
amount_cents INTEGER,
status TEXT,
raw_event JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
+27
View File
@@ -0,0 +1,27 @@
const { runner } = require('node-pg-migrate');
const path = require('path');
const direction = process.argv[2] || 'up';
runner({
databaseUrl: {
host: process.env.PGHOST || 'localhost',
port: parseInt(process.env.PGPORT || '5432', 10),
user: process.env.PGUSER,
password: process.env.PGPASSWORD,
database: process.env.PGDATABASE
},
dir: path.resolve(__dirname, 'migrations'),
direction,
migrationsTable: 'pgmigrations',
count: direction === 'down' ? 1 : Infinity,
log: (msg) => console.log(msg)
})
.then((applied) => {
console.log(`Migration complete — ${applied.length} migration(s) ${direction === 'down' ? 'reverted' : 'applied'}.`);
process.exit(0);
})
.catch((err) => {
console.error('Migration failed:', err.message);
process.exit(1);
});
@@ -0,0 +1,129 @@
exports.up = (pgm) => {
pgm.sql(`
CREATE TABLE IF NOT EXISTS items (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
description TEXT,
price_cents INTEGER NOT NULL,
status TEXT NOT NULL DEFAULT 'available',
reserved_until TIMESTAMPTZ,
sold_at TIMESTAMPTZ,
paypal_order_id TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS item_images (
id SERIAL PRIMARY KEY,
item_id INTEGER NOT NULL REFERENCES items(id) ON DELETE CASCADE,
image_path TEXT NOT NULL,
sort_order INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS customers (
id SERIAL PRIMARY KEY,
email TEXT NOT NULL UNIQUE,
password_hash TEXT NOT NULL,
name TEXT,
email_verified BOOLEAN NOT NULL DEFAULT false,
marketing_consent BOOLEAN NOT NULL DEFAULT false,
marketing_consent_at TIMESTAMPTZ,
marketing_consent_text TEXT,
unsubscribe_token TEXT NOT NULL UNIQUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS customer_sessions (
token TEXT PRIMARY KEY,
customer_id INTEGER NOT NULL REFERENCES customers(id) ON DELETE CASCADE,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS customer_tokens (
token TEXT PRIMARY KEY,
customer_id INTEGER NOT NULL REFERENCES customers(id) ON DELETE CASCADE,
kind TEXT NOT NULL,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS admin_settings (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
INSERT INTO admin_settings (key, value) VALUES ('cart_expiry_hours', '24') ON CONFLICT (key) DO NOTHING;
CREATE TABLE IF NOT EXISTS carts (
id SERIAL PRIMARY KEY,
customer_id INTEGER NOT NULL UNIQUE REFERENCES customers(id) ON DELETE CASCADE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS cart_items (
id SERIAL PRIMARY KEY,
cart_id INTEGER NOT NULL REFERENCES carts(id) ON DELETE CASCADE,
item_id INTEGER NOT NULL UNIQUE REFERENCES items(id) ON DELETE CASCADE,
added_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL,
last_reminder_sent_at TIMESTAMPTZ
);
CREATE TABLE IF NOT EXISTS shipping_addresses (
id SERIAL PRIMARY KEY,
customer_id INTEGER NOT NULL REFERENCES customers(id) ON DELETE CASCADE,
full_name TEXT NOT NULL,
address_line1 TEXT NOT NULL,
address_line2 TEXT,
city TEXT NOT NULL,
state TEXT NOT NULL,
postal_code TEXT NOT NULL,
country TEXT NOT NULL DEFAULT 'US',
is_default BOOLEAN NOT NULL DEFAULT false,
usps_validated BOOLEAN NOT NULL DEFAULT false,
usps_standardized JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS checkouts (
id SERIAL PRIMARY KEY,
customer_id INTEGER REFERENCES customers(id) ON DELETE SET NULL,
shipping_address_id INTEGER REFERENCES shipping_addresses(id) ON DELETE SET NULL,
processor TEXT NOT NULL,
processor_order_id TEXT,
amount_cents INTEGER,
status TEXT NOT NULL DEFAULT 'pending',
raw_event JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS checkout_items (
checkout_id INTEGER NOT NULL REFERENCES checkouts(id) ON DELETE CASCADE,
item_id INTEGER NOT NULL REFERENCES items(id) ON DELETE CASCADE,
price_cents INTEGER NOT NULL,
PRIMARY KEY (checkout_id, item_id)
);
CREATE TABLE IF NOT EXISTS orders (
id SERIAL PRIMARY KEY,
item_id INTEGER REFERENCES items(id),
customer_id INTEGER REFERENCES customers(id) ON DELETE SET NULL,
checkout_id INTEGER REFERENCES checkouts(id) ON DELETE SET NULL,
processor TEXT NOT NULL,
processor_order_id TEXT,
amount_cents INTEGER,
status TEXT,
raw_event JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
`);
};
// Deliberately no destructive down migration for the baseline — reverting this
// would drop every table and all data. If you ever need to roll back past this
// point, do it manually and deliberately, not via `migrate:down`.
exports.down = (pgm) => {
pgm.sql(`SELECT 1;`);
};
+8 -2
View File
@@ -13,7 +13,10 @@
"test:integration": "jest -c jest.integration.config.js --runInBand",
"test:integration:json": "jest -c jest.integration.config.js --runInBand --json --outputFile=integration-results.json",
"db:test:up": "docker compose -f docker-compose.test.yml up -d",
"db:test:down": "docker compose -f docker-compose.test.yml down -v"
"db:test:down": "docker compose -f docker-compose.test.yml down -v",
"migrate:up": "node migrate.js up",
"migrate:down": "node migrate.js down",
"migrate:create": "node-pg-migrate create --migration-file-language js"
},
"dependencies": {
"express": "^4.19.2",
@@ -21,7 +24,9 @@
"multer": "^1.4.5-lts.1",
"bcryptjs": "^2.4.3",
"cookie-parser": "^1.4.6",
"nodemailer": "^6.9.14"
"nodemailer": "^6.9.14",
"node-cron": "^3.0.3",
"node-pg-migrate": "^7.6.1"
},
"devDependencies": {
"typescript": "^5.5.4",
@@ -33,6 +38,7 @@
"@types/bcryptjs": "^2.4.6",
"@types/cookie-parser": "^1.4.7",
"@types/nodemailer": "^6.4.15",
"@types/node-cron": "^3.0.11",
"jest": "^29.7.0",
"ts-jest": "^29.2.4",
"@types/jest": "^29.5.12",
+17 -5
View File
@@ -1,5 +1,5 @@
import { Pool } from 'pg';
import fs from 'fs';
import { runner } from 'node-pg-migrate';
import path from 'path';
export const testPool = new Pool({
@@ -11,17 +11,29 @@ export const testPool = new Pool({
});
export async function migrate(): Promise<void> {
const sql = fs.readFileSync(path.join(__dirname, '..', '..', '..', 'init.sql'), 'utf-8');
await testPool.query(sql);
await runner({
databaseUrl: {
host: process.env.TEST_PGHOST || 'localhost',
port: parseInt(process.env.TEST_PGPORT || '55432', 10),
user: process.env.TEST_PGUSER || 'redefined_test',
password: process.env.TEST_PGPASSWORD || 'redefined_test',
database: process.env.TEST_PGDATABASE || 'redefined_test'
},
dir: path.resolve(__dirname, '..', '..', '..', 'migrations'),
direction: 'up',
migrationsTable: 'pgmigrations',
count: Infinity
});
}
export async function resetDb(): Promise<void> {
await testPool.query(`
TRUNCATE TABLE orders, customer_tokens, customer_sessions, customers, item_images, items
TRUNCATE TABLE orders, checkout_items, checkouts, shipping_addresses, cart_items, carts,
customer_tokens, customer_sessions, customers, item_images, items
RESTART IDENTITY CASCADE
`);
}
export async function closeDb(): Promise<void> {
await testPool.end();
}
}