# Operations runbook

Operational procedures for the operation ledger, its rollout controls, and the
Arma command bridge. Everything here assumes the roadmap in
`.omo/plans/ptf-4x-tarkov-milsim-roadmap.md` has shipped.

## Migration

- All operation schema lives in named Payload migrations under `src/migrations/`:
  `20260918_180851_add_operation_ledger`, `20260918_194842_add_operation_allocation`,
  `20260918_222549_add_operation_readiness`, `20260920_044743_add_zone_pressure`.
- Apply with `bun run payload migrate` (dev uses `push: false`; never `bun run db push`,
  which fails on `spatial_ref_sys` ownership).
- Verify status with `bun run payload migrate --status` (or `bun run migrate --status`).
- Regenerated artifacts after schema or collection changes: `bun run generate:types`
  and `bun run generate:importmap`. Never hand-edit `src/payload-types.ts`,
  `src/payload-generated-schema.ts`, or `src/app/(payload)/admin/importMap.js`.

## Rollout and rollback-by-disable

- Kill switch: set `OPERATION_LEDGER_ENABLED=false` in the environment. Raw
  `arma-sync-events` evidence continues to be stored (that collection's writes
  never depend on the flag); only the derived ledger (operation events and
  effects) is skipped. This is the staged-activation control.
- Re-enable by removing the flag or setting it to anything other than `"false"`.
- Catch-up after a disabled window or a deploy gap: run the backfill action
  (`reconcileBackfill` in `src/app/(frontend)/operations/actions.ts`, requires
  `operation-events:update`). It replays raw events with no ledger row;
  `processOperationEvent` is idempotent by `sourceEvent`, so it is safe to run
  repeatedly.

## Duplicate and rejected event recovery

- Inbound dedup is by `eventId` (`<serverId>:<event.id>`, unique). A duplicate
  batch is a no-op.
- Rejected (`invalid-payload`, `unsupported-version`) and dead-lettered
  (`unsupported-type`) rows are retained as evidence and derive zero effects.
- Replay a single rejected/dead-letter row from the AAR page (Reconciliation
  panel) or the `reconcileOperationEvent` action. Replay re-validates the
  immutable raw event and updates the row in place with a reconciliation note;
  if still invalid it stays rejected and no effects are created.
- Raw sync events are never deleted or rewritten, by code or by hand.

## Command lease expiry and recovery

- `arma-commands` lifecycle: `queued` -> `delivered` (lease starts, `deliveredAt`
  set) -> `succeeded`/`failed` (terminal, `completedAt` set).
- Lease timeout: 5 minutes (`COMMAND_LEASE_TIMEOUT_MS` in
  `src/lib/arma-bridge/commands.ts`). Every pull request first re-queues
  delivered commands whose lease expired without an ack, so a lost command is
  re-delivered on the next pull.
- Acks are guarded: a duplicate ack is an accepted no-op (first result wins);
  acking a command that was never delivered is rejected with HTTP 409.

## Inventory and economy repair

- Extraction deposits and locker credits are preflighted before any write; a
  full HQ or locker produces a `failed` effect (never a silent loss, never a
  partial deposit). Repair a failed effect by freeing capacity, then replaying
  the failed effect through reconciliation after fixing the root cause.
- Economy deltas are bounded (faucet/drain caps of 1000) and provenance-marked
  (`ECONOMY_NOTE_PREFIX`). `resetEconomyState` rebuilds market-state docs and
  preserves provenance; operation provenance lives in the operation ledger and
  is never consumed by the reset.
- Payments never precede the locker preflight in `buyListing`; a credit failure
  after payment is compensated by an adjustment refund. Investigate any
  `adjustment` transaction referencing an operation effect as evidence of that
  compensation path.

## Metrics and logging

- All ledger activity logs under the `[Operations]` prefix via `payload.logger`.
- Bridge lease recovery logs under `[ArmaBridge]`.
- Ledger rows carry `reconciliationNotes`; every manual replay appends a note
  with the acting user id and outcome.