polaris-task-force/DEPLOYMENT.md

Deployment — Coolify

This document covers deploying the Polaris Task Force to Coolify (self-hosted PaaS). The app is built as a single Docker image (multi-stage: Bun 1.3.11 for build-time only, Node 22 at runtime — the production host CPU lacks AVX/AVX2, which Bun requires) and deployed once per environment, each backed by its own Coolify-managed PostgreSQL service.

Local full-stack dev with the same image shape is described at the end (docker compose up).

---

1. Architecture at a glance

                  Coolify cluster
                  ┌───────────────────────────────────────────────────────────┐
                  │                                                           │
   dev env        │  ┌────────────┐   ┌─────────────┐                         │
  (polaris-dev)   │  │ App svc    │ ← │ Postgres    │ (Coolify-managed,       │
                  │  │ (Docker-   │   │ svc         │  per-env)               │
                  │  │  file)     │   │ + volume    │                         │
                  │  │  + media   │   └─────────────┘                         │
                  │  │   volume   │                                           │ 
                  │  └─────┬──────┘                                           │
                  │        │ scheduled jobs (docker exec → node payload bin)   │
                  └────────┼──────────────────────────────────────────────────┘
                           │
                           \________ builds pushed on git push (webhook)

• One image, parameterized by env vars per environment. No image bake per env required.
• PostgreSQL is provisioned by Coolify as a separate service in each environment (dev, stg, prd). The app talks to it via DATABASE_URI injected by Coolify.
• Payload uploads (media/ collection) are written to a Coolify Persistent Storage Volume mounted at /app/media in the app container, so uploads survive container restarts, rollbacks, and rebuilds.
• Scheduled one-shot jobs (game-tick, market-tick) run via docker exec Coolify Scheduled Tasks against the running app container — same image, same env, no second service required.
• The Discord bot ships inside the image (src/bot/) and activates per-environment when DISCORD_TOKEN is set — prod only. It shares the same Postgres DB via the container env and inherits the single-instance constraint: 2 replicas = two bot instances (duplicate command registration, double embeds/DMs). Notifications generated while the container is down are not backfilled (the Discord notification bridge polls with an in-memory cursor).

⚠️ Single-instance only. The realtime SSE pipeline (gameTick → /api/game-tick/notify → in-process bus → SSE /api/realtime → client router.refresh()) uses an in-memory subscriber bus (src/lib/realtime/bus.ts) that is not shared across processes. Coolify must run exactly 1 replica of the app container per environment for SSE to work. If you scale the app horizontally, SSE stops working (live shipment/event log updates silently fail). Everything else continues to work; you'll need a distributed bus before you can scale. See src/lib/realtime/bus.ts and AGENTS.md for context.

---

2. Prerequisites in Coolify

1. A working Coolify instance (you said yours is at https://coolify.onyxsimple.com).
2. A destination server registered with Coolify (Docker engine reachable).
3. This repo reachable by Coolify (Git source — GitHub/GitLab/etc.) so it can build on push. A manual image-build flow is also fine if you prefer.
4. A Postgres-compatible SMTP relay if you want password-reset email (the app already supports Brevo; otherwise leave the EMAIL_* env vars blank).

---

3. Provisioning a per-environment PostgreSQL service

For each environment (dev, stg, prd):

1. In Coolify, create (or open) the Project that represents the environment (e.g. Polaris → dev).
2. New Resource → Database → PostgreSQL.
3. Pick a sensible DB name, user, password. Coolify creates the database, exposes the connection string, and injects POSTGRES_* env vars into the same project automatically.
4. Note the Connection String Coolify generates (e.g. postgres://ptf_app:<password>@<coolify-internal-host>:5432/ptf-app-dev). You'll wire this into the app service next.

Backups, version upgrades, and per-env resource limits are all handled by Coolify on this service — that's the whole point of hosting the DB individually.

---

4. Creating the app service (per environment)

1. Inside the same Coolify Project, New Resource → Service / Application → From a Dockerfile (or "From a Git repository" and point at the repo root — Coolify will detect Dockerfile).
2. Configure the build:
   • Port: 3000 (already EXPOSE'd).
   • Health Check Path: /api/health — Coolify will hit this to decide "healthy". The Docker HEALTHCHECK directive does the same thing inside the container.
   • Build Pack: Dockerfile.
   • Don't use the docker-compose.yml for the production app service — that file is for local dev (it bundles its own Postgres). Coolify manages the Postgres service for you.
3. Set the environment variables (Section 5).
4. Attach a Persistent Storage entry mapping /app/media (read/write). This is where Payload stores uploaded media. The directory is created and chowned inside the image; just mount the volume on top.
5. Deploy. First build will take a few minutes (Bun install + Next.js standalone build, --max-old-space-size=8000 — make sure the build host has ≥ 4 GB free RAM, ideally 8).

Build args you may want to override

The Dockerfile pins Bun 1.3.11 (build-time only) and Node 22 alpine (runtime image). The production host does not need AVX/AVX2 — the runtime image is Node-only (see the Dockerfile header for why: Bun ≥ 1.2 hard-requires AVX2). To bump either, edit the Dockerfile: the bun-v<version> in the base stage's install command, and FROM node:<tag>-alpine in the runner stage.

---

5. Environment variables (per environment)

Var	Required	Example / Notes
DATABASE_URI	✅	Coolify auto-injects this from the per-env Postgres service if linked. Format: postgres://<user>:<pwd>@<host>:5432/<db>
PAYLOAD_SECRET	✅	Random 32+ chars. openssl rand -hex 16. Unique per environment.
APP_URL	✅	Public base URL for this environment, e.g. https://dev.ptf.example.com
GAME_TICK_NOTIFY_SECRET	✅	Shared secret for /api/game-tick/notify. Must match the value used by your scheduled job (Sec. 6).
EMAIL_FROM_ADDRESS	—	Outbound email "from" address (e.g. arma@onyxsimple.com)
EMAIL_FROM_NAME	—	Display name (e.g. Arma)
EMAIL_HOST	—	SMTP host (e.g. smtp-relay.brevo.com)
EMAIL_PORT	—	587
EMAIL_USERNAME	—	SMTP user
EMAIL_PASSWORD	—	SMTP password / API key
DISCORD_TOKEN	iff bot enabled in this env	Set in prod only — presence activates the bot (activation gate below).
DISCORD_GUILD_ID	iff DISCORD_TOKEN set	Config throws at import if the token is set and this is missing.
DISCORD_OPS_CHANNEL_ID	—	Ops channel for attendance RSVP embeds.
DISCORD_ANNOUNCE_CHANNEL_ID	—	Channel for /announce.
DISCORD_STAFF_ROLE_IDS	—	Comma-separated staff role IDs (staff-only /announce).
DISCORD_ATTENDANCE_POLL_MS	—	Attendance reconcile poll interval; default 60000.
DISCORD_NOTIFICATION_POLL_MS	—	Notification bridge poll interval; default 20000.

Never copy the dev PAYLOAD_SECRET into prod. Never copy GAME_TICK_NOTIFY_SECRET. Generate fresh per environment.

Discord bot activation gate

The bot ships inside the same app image and starts only when DISCORD_TOKEN is present:

1. Presence gate — the container entrypoint checks for DISCORD_TOKEN; if it's unset the bot never starts and the web app is unaffected.
2. Fail-fast on missing guild id — if DISCORD_TOKEN is set but DISCORD_GUILD_ID is missing, src/bot/config.ts throws at import time and the container fails fast. Set both together.
3. Prod preflight checklist — configure DISCORD_OPS_CHANNEL_ID and DISCORD_STAFF_ROLE_IDS before adding the token: on startup the bot immediately posts attendance embeds and starts DMing users per their notification preferences.
4. Same-token-across-envs rationale — the token is ONE Discord application credential, not a per-env secret. The value is identical across environments only because the presence gate restricts activation to prod; this does not conflict with the "never copy secrets between envs" guidance above, which governs per-env secrets like PAYLOAD_SECRET / GAME_TICK_NOTIFY_SECRET.
5. Session-flapping warning — never run the bot with the prod token from two places at once (e.g. local dev + prod): Discord force-disconnects one of the sessions.
6. Env-clone warning — a Coolify "clone environment" copy would carry the token along. Remove it from non-prod copies, or dev/stage would silently activate the bot.
7. Recovery — the entrypoint does not supervise/restart the bot. If the bot crashes, the web app stays healthy (announced subordinate-uptime default); recovery = Coolify Restart or redeploy. Troubleshooting: "token set but no bot login" → check the container logs for the bot's fail-fast line and verify the token/guild are valid.

⚠️ The bot shares the single-instance constraint (Sec. 1): running 2 replicas = two bot instances — duplicate command registration, double embeds/DMs. Notifications generated while the container is down are not backfilled (in-memory cursor).

---

6. Scheduled jobs (game-tick, market-tick, server-tick)

The game-tick and market-tick Payload bins are registered on payload.config.ts and expected to run on a regular cadence (cron). In Coolify:

Option A — Coolify Scheduled Tasks (recommended)

Coolify has a Scheduled Tasks feature per service (<service> → Scheduled Tasks → Add). Each scheduled task runs a command inside the running service container — equivalent to docker exec. Add:

Job	Schedule (cron)	Command
Process shipments	*/5 * * * * (every 5 min, tune to match gameTickIntervalMinutes in your Game Rules global)	node node_modules/payload/bin.js game-tick
Refresh market	*/10 * * * * (every 10 min — adjust as needed)	node node_modules/payload/bin.js market-tick
Server presence	* * * * * (every minute)	node node_modules/payload/bin.js server-tick

The commands run inside the app container with the app's env (incl. DATABASE_URI, APP_URL, GAME_TICK_NOTIFY_SECRET), so the bins talk to the same DB and post the notify callback correctly.

Option B — external cron calling docker exec over SSH

If you prefer existing infra:

*/5 * * * *   ssh -i /path/to/key root@coolify-host \
                docker exec polaris-task-force-app-<env> node node_modules/payload/bin.js game-tick
*/10 * * * *  ssh -i /path/to/key root@coolify-host \
                docker exec polaris-task-force-app-<env> node node_modules/payload/bin.js market-tick
* * * * *     ssh -i /path/to/key root@coolify-host \
                docker exec polaris-task-force-app-<env> node node_modules/payload/bin.js server-tick

The container name comes from Coolify's per-service container name (visible on the service dashboard).

Confirming a tick ran

Each tick:

• Updates Shipments rows in DB (status, fuel, arrival).
• POSTs to /api/game-tick/notify (internal SSE bus push → clients router.refresh()). If no connected clients, this just 204s.
• Emits GameEventLog entries you'll see in the Payload admin → Game Event Logs.

---

7. Running migrations after a schema change

If you add a new collection / field and create a new Payload migration under src/migrations/, run it once after deploying the new image:

docker exec <container-name> node node_modules/payload/bin.js migrate

Coolify Scheduled Tasks support ad-hoc runs of the same command — fire it manually, watch the logs, then commit. Never run drizzle-kit (bun run db migrate) in prod; bun run db push fails on this DB because of the PostGIS spatial_ref_sys ownership issue (see AGENTS.md Gotchas). Use Payload's migration runner exclusively.

For dev (where push: false in payload.config.ts), run bun run payload migrate after schema changes to apply them locally.

---

8. Persistent media volume — important for rollbacks

Payload's Media collection (src/collections/Media.ts) uses upload: true with no storage adapter, so files land on local disk under <projectRoot>/media (i.e. /app/media inside the container, since WORKDIR /app and process.cwd() = /app).

In Coolify:

1. On the app service → Persistent Storage → Add Path.
2. Host path or named volume → mapped to /app/media in the container.
3. Coolify preserves this volume across deploys, including rollbacks.

Without this, every redeploy wipes user uploads.

Note on the local-disk default: this works fine for a self-hosted single-instance deployment, but it does not scale beyond one replica (same caveat as the SSE bus). If you outgrow the single-instance architecture (more replicas, multi-host), switch to an S3-compatible storage adapter — add it to payload.config.ts under plugins (the project already has a // storage-adapter-placeholder hook for this) and drop the /app/media volume.

---

9. Health checks

• Container HEALTHCHECK (inside the Dockerfile): every 30 s, wget http://127.0.0.1:3000/api/health. 3 consecutive failures → unhealthy.
• Coolify itself should also probe GET /api/health (configure the service Health Check Path to /api/health). Coolify will route traffic only to "healthy" containers, so it doubles as your readiness gate after deployment.
• The health endpoint is deliberately DB-free — a transient database outage never marks the container unhealthy, so Coolify won't force a useless restart cycle while the Postgres service is bumping.

Response shape:

{ "ok": true, "uptime": 1243, "version": "0.1.6", "commit": "a1b2c3d", "buildTime": "2026-08-12T20:30:00.000Z" }

---

10. Deploy flow (git push → live)

Recommended flow:

1. Bump version locally: bun pm version patch (or your own flow). Edit package.json#version and push the commit.
2. Push to your environment's branch (e.g. main → prod, staging → stg, develop → dev), or whatever branch Coolify watches.
3. Coolify's webhook fires, builds the image, and rolls the container.
4. Coolify waits for /api/health to come back 200, then routes traffic.
5. If a new migration shipped with that deploy, run it via docker exec or a Scheduled Task (Sec. 7).

Version metadata in builds. The image build embeds git info into src/generated/versionInfo.json (shown in the admin VersionOverlay and on /api/version). Coolify deletes .git from the build context, so the build reads the commit SHA from the SOURCE_COMMIT build arg instead of git (see the Dockerfile's ARG SOURCE_COMMIT block). For that to work, enable "Include Source Commit in Build" under the application's Advanced settings (and leave "Inject Build Args to Dockerfile" on, which is the default) — otherwise the version overlay falls back to version only.

Commit metadata availability. On Coolify builds, SOURCE_COMMIT is the commit SHA, but tags, dates, and subjects are not injected automatically. If the build environment provides GIT_COMMIT_DATE and GIT_COMMIT_SUBJECT, the generator records them as lastUpdated and commitMessage; otherwise lastUpdated falls back to the image build time and the commit message remains unavailable. The existing canSeeCommit permission still gates commit details in the overlay.

Local builds (docker build, bun run dev) use real git and can populate all fields.

Version bump ordering. The postversion hook runs the production build after package.json has been bumped. This prevents the build from embedding the previous package version.

No SSH, no rsync, no systemctl. The legacy build/deploy.sh (gitignored, the old SSH+systemd flow to jmf-usrv-2404) is no longer used — exclude it from future deploys. bun run deploy bumps the package version and runs the production build; push the resulting version commit to the branch watched by Coolify to deploy it.

---

11. Rollbacks

Coolify supports two rollout modes:

1. Automatic rollback on failed health: if the new container never becomes healthy (3 failed /api/health probes within the start_period), Coolify keeps the old container running and doesn't route traffic to the new one. Brings you back to the previous working version with no action.
2. Manual rollback: in the service dashboard,/files history → pick a previous image tag → "Rollback". This redeploys that exact image with the existing /app/media volume intact (because the volume is mounted, not tied to the image).

Rollbacks do not roll the database schema. If a deploy includes a non-reversible migration that has already run, rolling back the image but keeping the DB can break the older image's assumptions. Mitigations:

• Only ship additive migrations (don't drop columns in the same deploy that removes their consumer code — drop them in a follow-up deploy instead).
• For destructive changes, take a Coolify DB snapshot first.

---

12. Local full-stack dev with docker compose

For developers who want the production image shape locally (no Coolify required):

cp .env.example .env
# Edit at minimum PAYLOAD_SECRET, GAME_TICK_NOTIFY_SECRET.
docker compose up --build

This boots:

• postgres — Postgres 17 alpine, persisted to the pgdata named volume.
• app — the production Dockerfile image, talking to the bundled Postgres via DATABASE_URI rewritten by compose from POSTGRES_USER/_PASSWORD/_DB.

App on http://localhost:3000, Postgres on localhost:5432.

Useful commands:

docker compose up --build              # rebuild + boot
docker compose down                    # stop
docker compose down -v                 # stop + wipe the Postgres volume
docker compose exec app node node_modules/payload/bin.js game-tick    # fire a tick manually
docker compose exec app node node_modules/payload/bin.js migrate      # apply migrations
docker compose exec app sh             # shell inside the app container

src/app/(frontend)/layout.tsx gates on auth — first deploy is a clean DB; you'll need to seed an admin user. The existing seed scripts under src/tools/seed/ run via bun locally; inside the container (no bun) use tsx: docker compose exec app node --import tsx src/tools/seed/<file>.ts (refer to those files for the intended entry point).

---

13. Quick troubleshooting

Symptom	Likely cause	Fix
Container restart loops, never healthy	DB connection unreachable	Verify DATABASE_URI resolves from inside the container (docker compose exec app wget -qO- $DATABASE_URI:5432 or nc -zv ...). Coolify: confirm the Postgres and app services are in the same Project so Coolify's network reaches it.
401 loop after deploy	PAYLOAD_SECRET changed without rotating user sessions	Keep PAYLOAD_SECRET constant across deploys of the same env. If you must rotate, users will need to re-log in (cookies are JWT-signed against this secret).
Realtime updates stopped working	More than 1 replica running, OR scheduled tick stopped	Limit replicas to 1. Check the scheduled tasks in Coolify are firing and GameEventLogs show fresh gameTick events.
First build OOMs	Host has < 8 GB RAM during Next build	The bun run build script sets --max-old-space-size=8000. Give the Coolify build host at least 8 GB, or lower this in package.json#scripts.build (memory will become the limiter earlier).
bun run db push fails PostGIS error	This is expected (must be owner of table spatial_ref_sys)	Use bun run payload migrate only. See AGENTS.md Gotchas.
Token set but bot never logs in / silent bot death	Entrypoint gate, config throw, or revoked/invalid token	Check the container logs for the bot's fail-fast line. Confirm DISCORD_GUILD_ID is set (the config throws at import if missing) and the token is valid/not revoked. The web app is unaffected — the bot is an in-process subordinate.
Scheduled task / bot dies with Illegal instruction (core dumped)	Running Bun on the host CPU (no AVX/AVX2)	The runtime image is Node-only — never docker exec ... bun .... Bins: node node_modules/payload/bin.js <cmd>; bot: node --import tsx src/bot/index.ts.

---

Last updated: 2026-08-12. Verify against the live repo if anything looks stale.