polaris-task-force/docs/operations/runbook.md

Operations runbook

Operational procedures for the operation ledger, its rollout controls, and the Arma command bridge. Everything here assumes the roadmap in .omo/plans/ptf-4x-tarkov-milsim-roadmap.md has shipped.

Migration

• All operation schema lives in named Payload migrations under src/migrations/: 20260918_180851_add_operation_ledger, 20260918_194842_add_operation_allocation, 20260918_222549_add_operation_readiness, 20260920_044743_add_zone_pressure.
• Apply with bun run payload migrate (dev uses push: false; never bun run db push, which fails on spatial_ref_sys ownership).
• Verify status with bun run payload migrate --status (or bun run migrate --status).
• Regenerated artifacts after schema or collection changes: bun run generate:types and bun run generate:importmap. Never hand-edit src/payload-types.ts, src/payload-generated-schema.ts, or src/app/(payload)/admin/importMap.js.

Rollout and rollback-by-disable

• Kill switch: set OPERATION_LEDGER_ENABLED=false in the environment. Raw arma-sync-events evidence continues to be stored (that collection's writes never depend on the flag); only the derived ledger (operation events and effects) is skipped. This is the staged-activation control.
• Re-enable by removing the flag or setting it to anything other than "false".
• Catch-up after a disabled window or a deploy gap: run the backfill action (reconcileBackfill in src/app/(frontend)/operations/actions.ts, requires operation-events:update). It replays raw events with no ledger row; processOperationEvent is idempotent by sourceEvent, so it is safe to run repeatedly.

Duplicate and rejected event recovery

• Inbound dedup is by eventId (<serverId>:<event.id>, unique). A duplicate batch is a no-op.
• Rejected (invalid-payload, unsupported-version) and dead-lettered (unsupported-type) rows are retained as evidence and derive zero effects.
• Replay a single rejected/dead-letter row from the AAR page (Reconciliation panel) or the reconcileOperationEvent action. Replay re-validates the immutable raw event and updates the row in place with a reconciliation note; if still invalid it stays rejected and no effects are created.
• Raw sync events are never deleted or rewritten, by code or by hand.

Command lease expiry and recovery

• arma-commands lifecycle: queued -> delivered (lease starts, deliveredAt set) -> succeeded/failed (terminal, completedAt set).
• Lease timeout: 5 minutes (COMMAND_LEASE_TIMEOUT_MS in src/lib/arma-bridge/commands.ts). Every pull request first re-queues delivered commands whose lease expired without an ack, so a lost command is re-delivered on the next pull.
• Acks are guarded: a duplicate ack is an accepted no-op (first result wins); acking a command that was never delivered is rejected with HTTP 409.

Inventory and economy repair

• Extraction deposits and locker credits are preflighted before any write; a full HQ or locker produces a failed effect (never a silent loss, never a partial deposit). Repair a failed effect by freeing capacity, then replaying the failed effect through reconciliation after fixing the root cause.
• Economy deltas are bounded (faucet/drain caps of 1000) and provenance-marked (ECONOMY_NOTE_PREFIX). resetEconomyState rebuilds market-state docs and preserves provenance; operation provenance lives in the operation ledger and is never consumed by the reset.
• Payments never precede the locker preflight in buyListing; a credit failure after payment is compensated by an adjustment refund. Investigate any adjustment transaction referencing an operation effect as evidence of that compensation path.

Metrics and logging

• All ledger activity logs under the [Operations] prefix via payload.logger.
• Bridge lease recovery logs under [ArmaBridge].
• Ledger rows carry reconciliationNotes; every manual replay appends a note with the acting user id and outcome.