Z8MB1E
ddca600f04
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
282 lines
21 KiB
Markdown
282 lines
21 KiB
Markdown
# Deployment — Coolify
|
|
|
|
This document covers deploying the Polaris Task Force to **Coolify** (self-hosted PaaS). The app is built as a single Docker image (multi-stage: Bun 1.3.11 for build-time only, Node 22 at runtime — the production host CPU lacks AVX/AVX2, which Bun requires) and deployed once per environment, each backed by its own Coolify-managed PostgreSQL service.
|
|
|
|
Local full-stack dev with the same image shape is described at the end (`docker compose up`).
|
|
|
|
---
|
|
|
|
## 1. Architecture at a glance
|
|
|
|
```
|
|
Coolify cluster
|
|
┌───────────────────────────────────────────────────────────┐
|
|
│ │
|
|
dev env │ ┌────────────┐ ┌─────────────┐ │
|
|
(polaris-dev) │ │ App svc │ ← │ Postgres │ (Coolify-managed, │
|
|
│ │ (Docker- │ │ svc │ per-env) │
|
|
│ │ file) │ │ + volume │ │
|
|
│ │ + media │ └─────────────┘ │
|
|
│ │ volume │ │
|
|
│ └─────┬──────┘ │
|
|
│ │ scheduled jobs (docker exec → node payload bin) │
|
|
└────────┼──────────────────────────────────────────────────┘
|
|
│
|
|
\________ builds pushed on git push (webhook)
|
|
```
|
|
|
|
- **One image**, parameterized by env vars per environment. No image bake per env required.
|
|
- **PostgreSQL** is provisioned by Coolify as a *separate service* in each environment (`dev`, `stg`, `prd`). The app talks to it via `DATABASE_URI` injected by Coolify.
|
|
- **Payload uploads** (`media/` collection) are written to a Coolify **Persistent Storage Volume** mounted at `/app/media` in the app container, so uploads survive container restarts, rollbacks, and rebuilds.
|
|
- **Scheduled one-shot jobs** (game-tick, market-tick) run via `docker exec` Coolify Scheduled Tasks against the running app container — same image, same env, no second service required.
|
|
- **The Discord bot ships inside the image** (`src/bot/`) and activates per-environment when `DISCORD_TOKEN` is set — prod only. It shares the same Postgres DB via the container env and inherits the single-instance constraint: **2 replicas = two bot instances** (duplicate command registration, double embeds/DMs). Notifications generated while the container is down are **not backfilled** (the Discord notification bridge polls with an in-memory cursor).
|
|
|
|
> ⚠️ **Single-instance only.** The realtime SSE pipeline (`gameTick → /api/game-tick/notify → in-process bus → SSE /api/realtime → client router.refresh()`) uses an **in-memory subscriber bus** (`src/lib/realtime/bus.ts`) that is not shared across processes. Coolify *must* run **exactly 1 replica** of the app container per environment for SSE to work. If you scale the app horizontally, SSE stops working (live shipment/event log updates silently fail). Everything else continues to work; you'll need a distributed bus before you can scale. See `src/lib/realtime/bus.ts` and `AGENTS.md` for context.
|
|
|
|
---
|
|
|
|
## 2. Prerequisites in Coolify
|
|
|
|
1. A working **Coolify** instance (you said yours is at `https://coolify.onyxsimple.com`).
|
|
2. A **destination server** registered with Coolify (Docker engine reachable).
|
|
3. This repo reachable by Coolify (Git source — GitHub/GitLab/etc.) so it can build on push. A manual image-build flow is also fine if you prefer.
|
|
4. A Postgres-compatible **SMTP relay** if you want password-reset email (the app already supports Brevo; otherwise leave the `EMAIL_*` env vars blank).
|
|
|
|
---
|
|
|
|
## 3. Provisioning a per-environment PostgreSQL service
|
|
|
|
For each environment (`dev`, `stg`, `prd`):
|
|
|
|
1. In Coolify, create (or open) the **Project** that represents the environment (e.g. `Polaris → dev`).
|
|
2. **New Resource → Database → PostgreSQL**.
|
|
3. Pick a sensible DB name, user, password. Coolify creates the database, exposes the connection string, and injects `POSTGRES_*` env vars into the same project automatically.
|
|
4. Note the **Connection String** Coolify generates (e.g. `postgres://ptf_app:<password>@<coolify-internal-host>:5432/ptf-app-dev`). You'll wire this into the app service next.
|
|
|
|
> Backups, version upgrades, and per-env resource limits are all handled by Coolify on this service — that's the whole point of hosting the DB individually.
|
|
|
|
---
|
|
|
|
## 4. Creating the app service (per environment)
|
|
|
|
1. Inside the same Coolify Project, **New Resource → Service / Application → From a Dockerfile** (or "From a Git repository" and point at the repo root — Coolify will detect `Dockerfile`).
|
|
2. Configure the build:
|
|
- **Port**: `3000` (already `EXPOSE`'d).
|
|
- **Health Check Path**: `/api/health` — Coolify will hit this to decide "healthy". The Docker `HEALTHCHECK` directive does the same thing inside the container.
|
|
- **Build Pack**: Dockerfile.
|
|
- **Don't** use the `docker-compose.yml` for the production app service — that file is for local dev (it bundles its own Postgres). Coolify manages the Postgres service for you.
|
|
3. Set the environment variables (Section 5).
|
|
4. Attach a **Persistent Storage** entry mapping `/app/media` (read/write). This is where Payload stores uploaded media. The directory is created and chowned inside the image; just mount the volume on top.
|
|
5. Deploy. First build will take a few minutes (Bun install + Next.js standalone build, `--max-old-space-size=8000` — make sure the build host has ≥ 4 GB free RAM, ideally 8).
|
|
|
|
### Build args you may want to override
|
|
|
|
The Dockerfile pins Bun 1.3.11 (**build-time only**) and Node 22 alpine (runtime image). The production host does **not** need AVX/AVX2 — the runtime image is Node-only (see the Dockerfile header for why: Bun ≥ 1.2 hard-requires AVX2). To bump either, edit the Dockerfile: the `bun-v<version>` in the base stage's install command, and `FROM node:<tag>-alpine` in the runner stage.
|
|
|
|
---
|
|
|
|
## 5. Environment variables (per environment)
|
|
|
|
| Var | Required | Example / Notes |
|
|
|---|---|---|
|
|
| `DATABASE_URI` | ✅ | Coolify auto-injects this from the per-env Postgres service if linked. Format: `postgres://<user>:<pwd>@<host>:5432/<db>` |
|
|
| `PAYLOAD_SECRET` | ✅ | Random 32+ chars. `openssl rand -hex 16`. **Unique per environment.** |
|
|
| `APP_URL` | ✅ | Public base URL for this environment, e.g. `https://dev.ptf.example.com` |
|
|
| `GAME_TICK_NOTIFY_SECRET` | ✅ | Shared secret for `/api/game-tick/notify`. **Must match** the value used by your scheduled job (Sec. 6). |
|
|
| `EMAIL_FROM_ADDRESS` | — | Outbound email "from" address (e.g. `arma@onyxsimple.com`) |
|
|
| `EMAIL_FROM_NAME` | — | Display name (e.g. `Arma`) |
|
|
| `EMAIL_HOST` | — | SMTP host (e.g. `smtp-relay.brevo.com`) |
|
|
| `EMAIL_PORT` | — | `587` |
|
|
| `EMAIL_USERNAME` | — | SMTP user |
|
|
| `EMAIL_PASSWORD` | — | SMTP password / API key |
|
|
| `DISCORD_TOKEN` | iff bot enabled in this env | Set in **prod only** — presence activates the bot (activation gate below). |
|
|
| `DISCORD_GUILD_ID` | iff `DISCORD_TOKEN` set | Config throws at import if the token is set and this is missing. |
|
|
| `DISCORD_OPS_CHANNEL_ID` | — | Ops channel for attendance RSVP embeds. |
|
|
| `DISCORD_ANNOUNCE_CHANNEL_ID` | — | Channel for `/announce`. |
|
|
| `DISCORD_STAFF_ROLE_IDS` | — | Comma-separated staff role IDs (staff-only `/announce`). |
|
|
| `DISCORD_ATTENDANCE_POLL_MS` | — | Attendance reconcile poll interval; default `60000`. |
|
|
| `DISCORD_NOTIFICATION_POLL_MS` | — | Notification bridge poll interval; default `20000`. |
|
|
|
|
> Never copy the dev `PAYLOAD_SECRET` into prod. Never copy `GAME_TICK_NOTIFY_SECRET`. Generate fresh per environment.
|
|
|
|
### Discord bot activation gate
|
|
|
|
The bot ships inside the same app image and starts **only when `DISCORD_TOKEN` is present**:
|
|
|
|
1. **Presence gate** — the container entrypoint checks for `DISCORD_TOKEN`; if it's unset the bot never starts and the web app is unaffected.
|
|
2. **Fail-fast on missing guild id** — if `DISCORD_TOKEN` is set but `DISCORD_GUILD_ID` is missing, `src/bot/config.ts` throws at import time and the container fails fast. Set both together.
|
|
3. **Prod preflight checklist** — configure `DISCORD_OPS_CHANNEL_ID` and `DISCORD_STAFF_ROLE_IDS` **before** adding the token: on startup the bot immediately posts attendance embeds and starts DMing users per their notification preferences.
|
|
4. **Same-token-across-envs rationale** — the token is ONE Discord application credential, not a per-env secret. The value is identical across environments only because the presence gate restricts activation to prod; this does **not** conflict with the "never copy secrets between envs" guidance above, which governs per-env secrets like `PAYLOAD_SECRET` / `GAME_TICK_NOTIFY_SECRET`.
|
|
5. **Session-flapping warning** — never run the bot with the prod token from two places at once (e.g. local dev + prod): Discord force-disconnects one of the sessions.
|
|
6. **Env-clone warning** — a Coolify "clone environment" copy would carry the token along. Remove it from non-prod copies, or dev/stage would silently activate the bot.
|
|
7. **Recovery** — the entrypoint does **not** supervise/restart the bot. If the bot crashes, the web app stays healthy (announced subordinate-uptime default); recovery = Coolify Restart or redeploy. Troubleshooting: "token set but no bot login" → check the container logs for the bot's fail-fast line and verify the token/guild are valid.
|
|
|
|
> ⚠️ The bot shares the single-instance constraint (Sec. 1): **running 2 replicas = two bot instances** — duplicate command registration, double embeds/DMs. Notifications generated while the container is down are **not backfilled** (in-memory cursor).
|
|
|
|
---
|
|
|
|
## 6. Scheduled jobs (game-tick, market-tick, server-tick)
|
|
|
|
The `game-tick` and `market-tick` Payload bins are registered on `payload.config.ts` and expected to run on a regular cadence (cron). In Coolify:
|
|
|
|
### Option A — Coolify Scheduled Tasks (recommended)
|
|
|
|
Coolify has a **Scheduled Tasks** feature per service (`<service> → Scheduled Tasks → Add`). Each scheduled task runs a command inside the *running service container* — equivalent to `docker exec`. Add:
|
|
|
|
| Job | Schedule (cron) | Command |
|
|
|---|---|---|
|
|
| Process shipments | `*/5 * * * *` (every 5 min, tune to match `gameTickIntervalMinutes` in your Game Rules global) | `node node_modules/payload/bin.js game-tick` |
|
|
| Refresh market | `*/10 * * * *` (every 10 min — adjust as needed) | `node node_modules/payload/bin.js market-tick` |
|
|
| Server presence | `* * * * *` (every minute) | `node node_modules/payload/bin.js server-tick` |
|
|
|
|
The commands run inside the app container with the app's env (incl. `DATABASE_URI`, `APP_URL`, `GAME_TICK_NOTIFY_SECRET`), so the bins talk to the same DB and post the notify callback correctly.
|
|
|
|
### Option B — external cron calling `docker exec` over SSH
|
|
|
|
If you prefer existing infra:
|
|
|
|
```cron
|
|
*/5 * * * * ssh -i /path/to/key root@coolify-host \
|
|
docker exec polaris-task-force-app-<env> node node_modules/payload/bin.js game-tick
|
|
*/10 * * * * ssh -i /path/to/key root@coolify-host \
|
|
docker exec polaris-task-force-app-<env> node node_modules/payload/bin.js market-tick
|
|
* * * * * ssh -i /path/to/key root@coolify-host \
|
|
docker exec polaris-task-force-app-<env> node node_modules/payload/bin.js server-tick
|
|
```
|
|
|
|
The container name comes from Coolify's per-service container name (visible on the service dashboard).
|
|
|
|
### Confirming a tick ran
|
|
|
|
Each tick:
|
|
- Updates `Shipments` rows in DB (status, fuel, arrival).
|
|
- `POST`s to `/api/game-tick/notify` (internal SSE bus push → clients `router.refresh()`). If no connected clients, this just 204s.
|
|
- Emits `GameEventLog` entries you'll see in the Payload admin → Game Event Logs.
|
|
|
|
---
|
|
|
|
## 7. Running migrations after a schema change
|
|
|
|
If you add a new collection / field and create a new Payload migration under `src/migrations/`, run it once after deploying the new image:
|
|
|
|
```
|
|
docker exec <container-name> node node_modules/payload/bin.js migrate
|
|
```
|
|
|
|
Coolify Scheduled Tasks support ad-hoc runs of the same command — fire it manually, watch the logs, then commit. **Never** run drizzle-kit (`bun run db migrate`) in prod; `bun run db push` fails on this DB because of the PostGIS `spatial_ref_sys` ownership issue (see `AGENTS.md` Gotchas). Use Payload's migration runner exclusively.
|
|
|
|
For dev (where `push: false` in `payload.config.ts`), run `bun run payload migrate` after schema changes to apply them locally.
|
|
|
|
---
|
|
|
|
## 8. Persistent media volume — important for rollbacks
|
|
|
|
Payload's `Media` collection (`src/collections/Media.ts`) uses `upload: true` with no storage adapter, so files land on local disk under `<projectRoot>/media` (i.e. `/app/media` inside the container, since `WORKDIR /app` and `process.cwd() = /app`).
|
|
|
|
In Coolify:
|
|
1. On the app service → **Persistent Storage → Add Path**.
|
|
2. Host path or named volume → mapped to `/app/media` in the container.
|
|
3. Coolify preserves this volume across deploys, **including rollbacks**.
|
|
|
|
Without this, every redeploy wipes user uploads.
|
|
|
|
> Note on the local-disk default: this works fine for a self-hosted single-instance deployment, but it does **not** scale beyond one replica (same caveat as the SSE bus). If you outgrow the single-instance architecture (more replicas, multi-host), switch to an S3-compatible storage adapter — add it to `payload.config.ts` under `plugins` (the project already has a `// storage-adapter-placeholder` hook for this) and drop the `/app/media` volume.
|
|
|
|
---
|
|
|
|
## 9. Health checks
|
|
|
|
- **Container** `HEALTHCHECK` (inside the Dockerfile): every 30 s, `wget http://127.0.0.1:3000/api/health`. 3 consecutive failures → unhealthy.
|
|
- **Coolify** itself should also probe `GET /api/health` (configure the service Health Check Path to `/api/health`). Coolify will route traffic only to "healthy" containers, so it doubles as your readiness gate after deployment.
|
|
- The health endpoint is deliberately DB-free — a transient database outage never marks the container unhealthy, so Coolify won't force a useless restart cycle while the Postgres service is bumping.
|
|
|
|
Response shape:
|
|
|
|
```json
|
|
{ "ok": true, "uptime": 1243, "version": "0.1.6", "commit": "a1b2c3d", "buildTime": "2026-08-12T20:30:00.000Z" }
|
|
```
|
|
|
|
---
|
|
|
|
## 10. Deploy flow (git push → live)
|
|
|
|
Recommended flow:
|
|
|
|
1. Bump version locally: `bun pm version patch` (or your own flow). Edit `package.json#version` and push the commit.
|
|
2. Push to your environment's branch (e.g. `main` → prod, `staging` → stg, `develop` → dev), or whatever branch Coolify watches.
|
|
3. Coolify's webhook fires, builds the image, and rolls the container.
|
|
4. Coolify waits for `/api/health` to come back 200, then routes traffic.
|
|
5. If a new migration shipped with that deploy, run it via `docker exec` or a Scheduled Task (Sec. 7).
|
|
|
|
> **Version metadata in builds.** The image build embeds git info into `src/generated/versionInfo.json` (shown in the admin VersionOverlay and on `/api/version`). Coolify **deletes `.git`** from the build context, so the build reads the commit SHA from the `SOURCE_COMMIT` build arg instead of `git` (see the Dockerfile's `ARG SOURCE_COMMIT` block). For that to work, enable **"Include Source Commit in Build"** under the application's *Advanced* settings (and leave "Inject Build Args to Dockerfile" on, which is the default) — otherwise the version overlay falls back to `version` only.
|
|
>
|
|
> **Commit metadata availability.** On Coolify builds, `SOURCE_COMMIT` is the commit SHA, but tags, dates, and subjects are not injected automatically. If the build environment provides `GIT_COMMIT_DATE` and `GIT_COMMIT_SUBJECT`, the generator records them as `lastUpdated` and `commitMessage`; otherwise `lastUpdated` falls back to the image build time and the commit message remains unavailable. The existing `canSeeCommit` permission still gates commit details in the overlay.
|
|
>
|
|
> Local builds (`docker build`, `bun run dev`) use real `git` and can populate all fields.
|
|
|
|
> **Version bump ordering.** The `postversion` hook runs the production build after `package.json` has been bumped. This prevents the build from embedding the previous package version.
|
|
|
|
No SSH, no `rsync`, no `systemctl`. The legacy `build/deploy.sh` (gitignored, the old SSH+systemd flow to `jmf-usrv-2404`) is **no longer used** — exclude it from future deploys. `bun run deploy` bumps the package version and runs the production build; push the resulting version commit to the branch watched by Coolify to deploy it.
|
|
|
|
---
|
|
|
|
## 11. Rollbacks
|
|
|
|
Coolify supports two rollout modes:
|
|
|
|
1. **Automatic rollback on failed health**: if the new container never becomes healthy (3 failed `/api/health` probes within the `start_period`), Coolify keeps the old container running and doesn't route traffic to the new one. Brings you back to the previous working version with no action.
|
|
2. **Manual rollback**: in the service dashboard,/files history → pick a previous image tag → "Rollback". This redeploys that exact image with the existing `/app/media` volume intact (because the volume is mounted, not tied to the image).
|
|
|
|
Rollbacks do **not** roll the database schema. If a deploy includes a non-reversible migration that has already run, rolling back the *image* but keeping the DB can break the older image's assumptions. Mitigations:
|
|
- Only ship additive migrations (don't drop columns in the same deploy that removes their consumer code — drop them in a follow-up deploy instead).
|
|
- For destructive changes, take a Coolify DB snapshot first.
|
|
|
|
---
|
|
|
|
## 12. Local full-stack dev with `docker compose`
|
|
|
|
For developers who want the production image shape locally (no Coolify required):
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
# Edit at minimum PAYLOAD_SECRET, GAME_TICK_NOTIFY_SECRET.
|
|
docker compose up --build
|
|
```
|
|
|
|
This boots:
|
|
|
|
- `postgres` — Postgres 17 alpine, persisted to the `pgdata` named volume.
|
|
- `app` — the production Dockerfile image, talking to the bundled Postgres via `DATABASE_URI` rewritten by compose from `POSTGRES_USER/_PASSWORD/_DB`.
|
|
|
|
App on `http://localhost:3000`, Postgres on `localhost:5432`.
|
|
|
|
Useful commands:
|
|
|
|
```bash
|
|
docker compose up --build # rebuild + boot
|
|
docker compose down # stop
|
|
docker compose down -v # stop + wipe the Postgres volume
|
|
docker compose exec app node node_modules/payload/bin.js game-tick # fire a tick manually
|
|
docker compose exec app node node_modules/payload/bin.js migrate # apply migrations
|
|
docker compose exec app sh # shell inside the app container
|
|
```
|
|
|
|
> `src/app/(frontend)/layout.tsx` gates on auth — first deploy is a clean DB; you'll need to seed an admin user. The existing seed scripts under `src/tools/seed/` run via bun locally; inside the container (no bun) use tsx: `docker compose exec app node --import tsx src/tools/seed/<file>.ts` (refer to those files for the intended entry point).
|
|
|
|
---
|
|
|
|
## 13. Quick troubleshooting
|
|
|
|
| Symptom | Likely cause | Fix |
|
|
|---|---|---|
|
|
| Container restart loops, never healthy | DB connection unreachable | Verify `DATABASE_URI` resolves from inside the container (`docker compose exec app wget -qO- $DATABASE_URI:5432` or `nc -zv ...`). Coolify: confirm the Postgres and app services are in the same Project so Coolify's network reaches it. |
|
|
| 401 loop after deploy | `PAYLOAD_SECRET` changed without rotating user sessions | Keep `PAYLOAD_SECRET` constant across deploys of the same env. If you must rotate, users will need to re-log in (cookies are JWT-signed against this secret). |
|
|
| Realtime updates stopped working | More than 1 replica running, OR scheduled tick stopped | Limit replicas to 1. Check the scheduled tasks in Coolify are firing and `GameEventLogs` show fresh `gameTick` events. |
|
|
| First build OOMs | Host has < 8 GB RAM during Next build | The `bun run build` script sets `--max-old-space-size=8000`. Give the Coolify build host at least 8 GB, or lower this in `package.json#scripts.build` (memory will become the limiter earlier). |
|
|
| `bun run db push` fails PostGIS error | This is expected (`must be owner of table spatial_ref_sys`) | Use `bun run payload migrate` only. See `AGENTS.md` Gotchas. |
|
|
| Token set but bot never logs in / silent bot death | Entrypoint gate, config throw, or revoked/invalid token | Check the container logs for the bot's fail-fast line. Confirm `DISCORD_GUILD_ID` is set (the config throws at import if missing) and the token is valid/not revoked. The web app is unaffected — the bot is an in-process subordinate. |
|
|
| Scheduled task / bot dies with `Illegal instruction (core dumped)` | Running Bun on the host CPU (no AVX/AVX2) | The runtime image is Node-only — never `docker exec ... bun ...`. Bins: `node node_modules/payload/bin.js <cmd>`; bot: `node --import tsx src/bot/index.ts`. |
|
|
|
|
---
|
|
|
|
Last updated: 2026-08-12. Verify against the live repo if anything looks stale.
|