# Deployment — Coolify This document covers deploying the Polaris Task Force to **Coolify** (self-hosted PaaS). The app is built as a single Docker image (multi-stage: Bun 1.3.11 for build-time only, Node 22 at runtime — the production host CPU lacks AVX/AVX2, which Bun requires) and deployed once per environment, each backed by its own Coolify-managed PostgreSQL service. Local full-stack dev with the same image shape is described at the end (`docker compose up`). --- ## 1. Architecture at a glance ``` Coolify cluster ┌───────────────────────────────────────────────────────────┐ │ │ dev env │ ┌────────────┐ ┌─────────────┐ │ (polaris-dev) │ │ App svc │ ← │ Postgres │ (Coolify-managed, │ │ │ (Docker- │ │ svc │ per-env) │ │ │ file) │ │ + volume │ │ │ │ + media │ └─────────────┘ │ │ │ volume │ │ │ └─────┬──────┘ │ │ │ scheduled jobs (docker exec → node payload bin) │ └────────┼──────────────────────────────────────────────────┘ │ \________ builds pushed on git push (webhook) ``` - **One image**, parameterized by env vars per environment. No image bake per env required. - **PostgreSQL** is provisioned by Coolify as a *separate service* in each environment (`dev`, `stg`, `prd`). The app talks to it via `DATABASE_URI` injected by Coolify. - **Payload uploads** (`media/` collection) are written to a Coolify **Persistent Storage Volume** mounted at `/app/media` in the app container, so uploads survive container restarts, rollbacks, and rebuilds. - **Scheduled one-shot jobs** (game-tick, market-tick) run via `docker exec` Coolify Scheduled Tasks against the running app container — same image, same env, no second service required. - **The Discord bot ships inside the image** (`src/bot/`) and activates per-environment when `DISCORD_TOKEN` is set — prod only. It shares the same Postgres DB via the container env and inherits the single-instance constraint: **2 replicas = two bot instances** (duplicate command registration, double embeds/DMs). Notifications generated while the container is down are **not backfilled** (the Discord notification bridge polls with an in-memory cursor). > ⚠️ **Single-instance only.** The realtime SSE pipeline (`gameTick → /api/game-tick/notify → in-process bus → SSE /api/realtime → client router.refresh()`) uses an **in-memory subscriber bus** (`src/lib/realtime/bus.ts`) that is not shared across processes. Coolify *must* run **exactly 1 replica** of the app container per environment for SSE to work. If you scale the app horizontally, SSE stops working (live shipment/event log updates silently fail). Everything else continues to work; you'll need a distributed bus before you can scale. See `src/lib/realtime/bus.ts` and `AGENTS.md` for context. --- ## 2. Prerequisites in Coolify 1. A working **Coolify** instance (you said yours is at `https://coolify.onyxsimple.com`). 2. A **destination server** registered with Coolify (Docker engine reachable). 3. This repo reachable by Coolify (Git source — GitHub/GitLab/etc.) so it can build on push. A manual image-build flow is also fine if you prefer. 4. A Postgres-compatible **SMTP relay** if you want password-reset email (the app already supports Brevo; otherwise leave the `EMAIL_*` env vars blank). --- ## 3. Provisioning a per-environment PostgreSQL service For each environment (`dev`, `stg`, `prd`): 1. In Coolify, create (or open) the **Project** that represents the environment (e.g. `Polaris → dev`). 2. **New Resource → Database → PostgreSQL**. 3. Pick a sensible DB name, user, password. Coolify creates the database, exposes the connection string, and injects `POSTGRES_*` env vars into the same project automatically. 4. Note the **Connection String** Coolify generates (e.g. `postgres://ptf_app:@:5432/ptf-app-dev`). You'll wire this into the app service next. > Backups, version upgrades, and per-env resource limits are all handled by Coolify on this service — that's the whole point of hosting the DB individually. --- ## 4. Creating the app service (per environment) 1. Inside the same Coolify Project, **New Resource → Service / Application → From a Dockerfile** (or "From a Git repository" and point at the repo root — Coolify will detect `Dockerfile`). 2. Configure the build: - **Port**: `3000` (already `EXPOSE`'d). - **Health Check Path**: `/api/health` — Coolify will hit this to decide "healthy". The Docker `HEALTHCHECK` directive does the same thing inside the container. - **Build Pack**: Dockerfile. - **Don't** use the `docker-compose.yml` for the production app service — that file is for local dev (it bundles its own Postgres). Coolify manages the Postgres service for you. 3. Set the environment variables (Section 5). 4. Attach a **Persistent Storage** entry mapping `/app/media` (read/write). This is where Payload stores uploaded media. The directory is created and chowned inside the image; just mount the volume on top. 5. Deploy. First build will take a few minutes (Bun install + Next.js standalone build, `--max-old-space-size=8000` — make sure the build host has ≥ 4 GB free RAM, ideally 8). ### Build args you may want to override The Dockerfile pins Bun 1.3.11 (**build-time only**) and Node 22 alpine (runtime image). The production host does **not** need AVX/AVX2 — the runtime image is Node-only (see the Dockerfile header for why: Bun ≥ 1.2 hard-requires AVX2). To bump either, edit the Dockerfile: the `bun-v` in the base stage's install command, and `FROM node:-alpine` in the runner stage. --- ## 5. Environment variables (per environment) | Var | Required | Example / Notes | |---|---|---| | `DATABASE_URI` | ✅ | Coolify auto-injects this from the per-env Postgres service if linked. Format: `postgres://:@:5432/` | | `PAYLOAD_SECRET` | ✅ | Random 32+ chars. `openssl rand -hex 16`. **Unique per environment.** | | `APP_URL` | ✅ | Public base URL for this environment, e.g. `https://dev.ptf.example.com` | | `GAME_TICK_NOTIFY_SECRET` | ✅ | Shared secret for `/api/game-tick/notify`. **Must match** the value used by your scheduled job (Sec. 6). | | `EMAIL_FROM_ADDRESS` | — | Outbound email "from" address (e.g. `arma@onyxsimple.com`) | | `EMAIL_FROM_NAME` | — | Display name (e.g. `Arma`) | | `EMAIL_HOST` | — | SMTP host (e.g. `smtp-relay.brevo.com`) | | `EMAIL_PORT` | — | `587` | | `EMAIL_USERNAME` | — | SMTP user | | `EMAIL_PASSWORD` | — | SMTP password / API key | | `DISCORD_TOKEN` | iff bot enabled in this env | Set in **prod only** — presence activates the bot (activation gate below). | | `DISCORD_GUILD_ID` | iff `DISCORD_TOKEN` set | Config throws at import if the token is set and this is missing. | | `DISCORD_OPS_CHANNEL_ID` | — | Ops channel for attendance RSVP embeds. | | `DISCORD_ANNOUNCE_CHANNEL_ID` | — | Channel for `/announce`. | | `DISCORD_STAFF_ROLE_IDS` | — | Comma-separated staff role IDs (staff-only `/announce`). | | `DISCORD_ATTENDANCE_POLL_MS` | — | Attendance reconcile poll interval; default `60000`. | | `DISCORD_NOTIFICATION_POLL_MS` | — | Notification bridge poll interval; default `20000`. | > Never copy the dev `PAYLOAD_SECRET` into prod. Never copy `GAME_TICK_NOTIFY_SECRET`. Generate fresh per environment. ### Discord bot activation gate The bot ships inside the same app image and starts **only when `DISCORD_TOKEN` is present**: 1. **Presence gate** — the container entrypoint checks for `DISCORD_TOKEN`; if it's unset the bot never starts and the web app is unaffected. 2. **Fail-fast on missing guild id** — if `DISCORD_TOKEN` is set but `DISCORD_GUILD_ID` is missing, `src/bot/config.ts` throws at import time and the container fails fast. Set both together. 3. **Prod preflight checklist** — configure `DISCORD_OPS_CHANNEL_ID` and `DISCORD_STAFF_ROLE_IDS` **before** adding the token: on startup the bot immediately posts attendance embeds and starts DMing users per their notification preferences. 4. **Same-token-across-envs rationale** — the token is ONE Discord application credential, not a per-env secret. The value is identical across environments only because the presence gate restricts activation to prod; this does **not** conflict with the "never copy secrets between envs" guidance above, which governs per-env secrets like `PAYLOAD_SECRET` / `GAME_TICK_NOTIFY_SECRET`. 5. **Session-flapping warning** — never run the bot with the prod token from two places at once (e.g. local dev + prod): Discord force-disconnects one of the sessions. 6. **Env-clone warning** — a Coolify "clone environment" copy would carry the token along. Remove it from non-prod copies, or dev/stage would silently activate the bot. 7. **Recovery** — the entrypoint does **not** supervise/restart the bot. If the bot crashes, the web app stays healthy (announced subordinate-uptime default); recovery = Coolify Restart or redeploy. Troubleshooting: "token set but no bot login" → check the container logs for the bot's fail-fast line and verify the token/guild are valid. > ⚠️ The bot shares the single-instance constraint (Sec. 1): **running 2 replicas = two bot instances** — duplicate command registration, double embeds/DMs. Notifications generated while the container is down are **not backfilled** (in-memory cursor). --- ## 6. Scheduled jobs (game-tick, market-tick) The `game-tick` and `market-tick` Payload bins are registered on `payload.config.ts` and expected to run on a regular cadence (cron). In Coolify: ### Option A — Coolify Scheduled Tasks (recommended) Coolify has a **Scheduled Tasks** feature per service (` → Scheduled Tasks → Add`). Each scheduled task runs a command inside the *running service container* — equivalent to `docker exec`. Add: | Job | Schedule (cron) | Command | |---|---|---| | Process shipments | `*/5 * * * *` (every 5 min, tune to match `gameTickIntervalMinutes` in your Game Rules global) | `node node_modules/payload/bin.js game-tick` | | Refresh market | `*/10 * * * *` (every 10 min — adjust as needed) | `node node_modules/payload/bin.js market-tick` | The commands run inside the app container with the app's env (incl. `DATABASE_URI`, `APP_URL`, `GAME_TICK_NOTIFY_SECRET`), so the bins talk to the same DB and post the notify callback correctly. ### Option B — external cron calling `docker exec` over SSH If you prefer existing infra: ```cron */5 * * * * ssh -i /path/to/key root@coolify-host \ docker exec polaris-task-force-app- node node_modules/payload/bin.js game-tick */10 * * * * ssh -i /path/to/key root@coolify-host \ docker exec polaris-task-force-app- node node_modules/payload/bin.js market-tick ``` The container name comes from Coolify's per-service container name (visible on the service dashboard). ### Confirming a tick ran Each tick: - Updates `Shipments` rows in DB (status, fuel, arrival). - `POST`s to `/api/game-tick/notify` (internal SSE bus push → clients `router.refresh()`). If no connected clients, this just 204s. - Emits `GameEventLog` entries you'll see in the Payload admin → Game Event Logs. --- ## 7. Running migrations after a schema change If you add a new collection / field and create a new Payload migration under `src/migrations/`, run it once after deploying the new image: ``` docker exec node node_modules/payload/bin.js migrate ``` Coolify Scheduled Tasks support ad-hoc runs of the same command — fire it manually, watch the logs, then commit. **Never** run drizzle-kit (`bun run db migrate`) in prod; `bun run db push` fails on this DB because of the PostGIS `spatial_ref_sys` ownership issue (see `AGENTS.md` Gotchas). Use Payload's migration runner exclusively. For dev (where `push: true` in `payload.config.ts`), starting the dev server auto-applies schema changes — that's intentional and fine for dev databases. --- ## 8. Persistent media volume — important for rollbacks Payload's `Media` collection (`src/collections/Media.ts`) uses `upload: true` with no storage adapter, so files land on local disk under `/media` (i.e. `/app/media` inside the container, since `WORKDIR /app` and `process.cwd() = /app`). In Coolify: 1. On the app service → **Persistent Storage → Add Path**. 2. Host path or named volume → mapped to `/app/media` in the container. 3. Coolify preserves this volume across deploys, **including rollbacks**. Without this, every redeploy wipes user uploads. > Note on the local-disk default: this works fine for a self-hosted single-instance deployment, but it does **not** scale beyond one replica (same caveat as the SSE bus). If you outgrow the single-instance architecture (more replicas, multi-host), switch to an S3-compatible storage adapter — add it to `payload.config.ts` under `plugins` (the project already has a `// storage-adapter-placeholder` hook for this) and drop the `/app/media` volume. --- ## 9. Health checks - **Container** `HEALTHCHECK` (inside the Dockerfile): every 30 s, `wget http://127.0.0.1:3000/api/health`. 3 consecutive failures → unhealthy. - **Coolify** itself should also probe `GET /api/health` (configure the service Health Check Path to `/api/health`). Coolify will route traffic only to "healthy" containers, so it doubles as your readiness gate after deployment. - The health endpoint is deliberately DB-free — a transient database outage never marks the container unhealthy, so Coolify won't force a useless restart cycle while the Postgres service is bumping. Response shape: ```json { "ok": true, "uptime": 1243, "version": "0.1.6", "commit": "a1b2c3d", "buildTime": "2026-08-12T20:30:00.000Z" } ``` --- ## 10. Deploy flow (git push → live) Recommended flow: 1. Bump version locally: `bun pm version patch` (or your own flow). Edit `package.json#version` and push the commit. 2. Push to your environment's branch (e.g. `main` → prod, `staging` → stg, `develop` → dev), or whatever branch Coolify watches. 3. Coolify's webhook fires, builds the image, and rolls the container. 4. Coolify waits for `/api/health` to come back 200, then routes traffic. 5. If a new migration shipped with that deploy, run it via `docker exec` or a Scheduled Task (Sec. 7). > **Version metadata in builds.** The image build embeds git info into `src/generated/versionInfo.json` (shown in the admin VersionOverlay and on `/api/version`). Coolify **deletes `.git`** from the build context, so the build reads the commit SHA from the `SOURCE_COMMIT` build arg instead of `git` (see the Dockerfile's `ARG SOURCE_COMMIT` block). For that to work, enable **"Include Source Commit in Build"** under the application's *Advanced* settings (and leave "Inject Build Args to Dockerfile" on, which is the default) — otherwise the version overlay falls back to `version` only. Tags, commit date, and commit subject have no Coolify equivalent and stay null on Coolify builds; local builds (`docker build`, `bun run dev`) still use real `git`. No SSH, no `rsync`, no `systemctl`. The legacy `build/deploy.sh` (gitignored, the old SSH+systemd flow to `jmf-usrv-2404`) is **no longer used** — exclude it from future deploys. `bun run deploy` (the npm script) will still try to run it (it calls `postversion → bash ./build/deploy.sh`); don't run it from your local machine anymore, or remove that script entry from `package.json` once you've moved onto Coolify for good. --- ## 11. Rollbacks Coolify supports two rollout modes: 1. **Automatic rollback on failed health**: if the new container never becomes healthy (3 failed `/api/health` probes within the `start_period`), Coolify keeps the old container running and doesn't route traffic to the new one. Brings you back to the previous working version with no action. 2. **Manual rollback**: in the service dashboard,/files history → pick a previous image tag → "Rollback". This redeploys that exact image with the existing `/app/media` volume intact (because the volume is mounted, not tied to the image). Rollbacks do **not** roll the database schema. If a deploy includes a non-reversible migration that has already run, rolling back the *image* but keeping the DB can break the older image's assumptions. Mitigations: - Only ship additive migrations (don't drop columns in the same deploy that removes their consumer code — drop them in a follow-up deploy instead). - For destructive changes, take a Coolify DB snapshot first. --- ## 12. Local full-stack dev with `docker compose` For developers who want the production image shape locally (no Coolify required): ```bash cp .env.example .env # Edit at minimum PAYLOAD_SECRET, GAME_TICK_NOTIFY_SECRET. docker compose up --build ``` This boots: - `postgres` — Postgres 17 alpine, persisted to the `pgdata` named volume. - `app` — the production Dockerfile image, talking to the bundled Postgres via `DATABASE_URI` rewritten by compose from `POSTGRES_USER/_PASSWORD/_DB`. App on `http://localhost:3000`, Postgres on `localhost:5432`. Useful commands: ```bash docker compose up --build # rebuild + boot docker compose down # stop docker compose down -v # stop + wipe the Postgres volume docker compose exec app node node_modules/payload/bin.js game-tick # fire a tick manually docker compose exec app node node_modules/payload/bin.js migrate # apply migrations docker compose exec app sh # shell inside the app container ``` > `src/app/(frontend)/layout.tsx` gates on auth — first deploy is a clean DB; you'll need to seed an admin user. The existing seed scripts under `src/tools/seed/` run via bun locally; inside the container (no bun) use tsx: `docker compose exec app node --import tsx src/tools/seed/.ts` (refer to those files for the intended entry point). --- ## 13. Quick troubleshooting | Symptom | Likely cause | Fix | |---|---|---| | Container restart loops, never healthy | DB connection unreachable | Verify `DATABASE_URI` resolves from inside the container (`docker compose exec app wget -qO- $DATABASE_URI:5432` or `nc -zv ...`). Coolify: confirm the Postgres and app services are in the same Project so Coolify's network reaches it. | | 401 loop after deploy | `PAYLOAD_SECRET` changed without rotating user sessions | Keep `PAYLOAD_SECRET` constant across deploys of the same env. If you must rotate, users will need to re-log in (cookies are JWT-signed against this secret). | | Realtime updates stopped working | More than 1 replica running, OR scheduled tick stopped | Limit replicas to 1. Check the scheduled tasks in Coolify are firing and `GameEventLogs` show fresh `gameTick` events. | | First build OOMs | Host has < 8 GB RAM during Next build | The `bun run build` script sets `--max-old-space-size=8000`. Give the Coolify build host at least 8 GB, or lower this in `package.json#scripts.build` (memory will become the limiter earlier). | | `bun run db push` fails PostGIS error | This is expected (`must be owner of table spatial_ref_sys`) | Use `bun run payload migrate` only. See `AGENTS.md` Gotchas. | | Token set but bot never logs in / silent bot death | Entrypoint gate, config throw, or revoked/invalid token | Check the container logs for the bot's fail-fast line. Confirm `DISCORD_GUILD_ID` is set (the config throws at import if missing) and the token is valid/not revoked. The web app is unaffected — the bot is an in-process subordinate. | | Scheduled task / bot dies with `Illegal instruction (core dumped)` | Running Bun on the host CPU (no AVX/AVX2) | The runtime image is Node-only — never `docker exec ... bun ...`. Bins: `node node_modules/payload/bin.js `; bot: `node --import tsx src/bot/index.ts`. | --- Last updated: 2026-08-12. Verify against the live repo if anything looks stale.