Z8MB1E
37d28893de
Env kill switch with raw-evidence retention, idempotent backfill, in-place dead-letter replay with reconciliation notes, command lease recovery on pull, ack terminal-state guards, and an operations runbook. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
75 lines
4 KiB
Markdown
75 lines
4 KiB
Markdown
# Operations runbook
|
|
|
|
Operational procedures for the operation ledger, its rollout controls, and the
|
|
Arma command bridge. Everything here assumes the roadmap in
|
|
`.omo/plans/ptf-4x-tarkov-milsim-roadmap.md` has shipped.
|
|
|
|
## Migration
|
|
|
|
- All operation schema lives in named Payload migrations under `src/migrations/`:
|
|
`20260918_180851_add_operation_ledger`, `20260918_194842_add_operation_allocation`,
|
|
`20260918_222549_add_operation_readiness`, `20260920_044743_add_zone_pressure`.
|
|
- Apply with `bun run payload migrate` (dev uses `push: false`; never `bun run db push`,
|
|
which fails on `spatial_ref_sys` ownership).
|
|
- Verify status with `bun run payload migrate --status` (or `bun run migrate --status`).
|
|
- Regenerated artifacts after schema or collection changes: `bun run generate:types`
|
|
and `bun run generate:importmap`. Never hand-edit `src/payload-types.ts`,
|
|
`src/payload-generated-schema.ts`, or `src/app/(payload)/admin/importMap.js`.
|
|
|
|
## Rollout and rollback-by-disable
|
|
|
|
- Kill switch: set `OPERATION_LEDGER_ENABLED=false` in the environment. Raw
|
|
`arma-sync-events` evidence continues to be stored (that collection's writes
|
|
never depend on the flag); only the derived ledger (operation events and
|
|
effects) is skipped. This is the staged-activation control.
|
|
- Re-enable by removing the flag or setting it to anything other than `"false"`.
|
|
- Catch-up after a disabled window or a deploy gap: run the backfill action
|
|
(`reconcileBackfill` in `src/app/(frontend)/operations/actions.ts`, requires
|
|
`operation-events:update`). It replays raw events with no ledger row;
|
|
`processOperationEvent` is idempotent by `sourceEvent`, so it is safe to run
|
|
repeatedly.
|
|
|
|
## Duplicate and rejected event recovery
|
|
|
|
- Inbound dedup is by `eventId` (`<serverId>:<event.id>`, unique). A duplicate
|
|
batch is a no-op.
|
|
- Rejected (`invalid-payload`, `unsupported-version`) and dead-lettered
|
|
(`unsupported-type`) rows are retained as evidence and derive zero effects.
|
|
- Replay a single rejected/dead-letter row from the AAR page (Reconciliation
|
|
panel) or the `reconcileOperationEvent` action. Replay re-validates the
|
|
immutable raw event and updates the row in place with a reconciliation note;
|
|
if still invalid it stays rejected and no effects are created.
|
|
- Raw sync events are never deleted or rewritten, by code or by hand.
|
|
|
|
## Command lease expiry and recovery
|
|
|
|
- `arma-commands` lifecycle: `queued` -> `delivered` (lease starts, `deliveredAt`
|
|
set) -> `succeeded`/`failed` (terminal, `completedAt` set).
|
|
- Lease timeout: 5 minutes (`COMMAND_LEASE_TIMEOUT_MS` in
|
|
`src/lib/arma-bridge/commands.ts`). Every pull request first re-queues
|
|
delivered commands whose lease expired without an ack, so a lost command is
|
|
re-delivered on the next pull.
|
|
- Acks are guarded: a duplicate ack is an accepted no-op (first result wins);
|
|
acking a command that was never delivered is rejected with HTTP 409.
|
|
|
|
## Inventory and economy repair
|
|
|
|
- Extraction deposits and locker credits are preflighted before any write; a
|
|
full HQ or locker produces a `failed` effect (never a silent loss, never a
|
|
partial deposit). Repair a failed effect by freeing capacity, then replaying
|
|
the failed effect through reconciliation after fixing the root cause.
|
|
- Economy deltas are bounded (faucet/drain caps of 1000) and provenance-marked
|
|
(`ECONOMY_NOTE_PREFIX`). `resetEconomyState` rebuilds market-state docs and
|
|
preserves provenance; operation provenance lives in the operation ledger and
|
|
is never consumed by the reset.
|
|
- Payments never precede the locker preflight in `buyListing`; a credit failure
|
|
after payment is compensated by an adjustment refund. Investigate any
|
|
`adjustment` transaction referencing an operation effect as evidence of that
|
|
compensation path.
|
|
|
|
## Metrics and logging
|
|
|
|
- All ledger activity logs under the `[Operations]` prefix via `payload.logger`.
|
|
- Bridge lease recovery logs under `[ArmaBridge]`.
|
|
- Ledger rows carry `reconciliationNotes`; every manual replay appends a note
|
|
with the acting user id and outcome.
|