Veyronn Documentation

Monitoring and Incidents

Health checks, alerts, fail-closed responses, and recovery actions.

Routine health checks

| Signal | Expected | | --- | --- | | GET /api/health | HTTP 200 with BFF and database healthy | | GET /api/indexer/health | Advancing cursor and bounded lag | | Latest crypto price | Fresh accepted timestamp and advancing round | | Keeper wallet | Enough testnet ETH for automation transactions | | Conditionals | Actionable queue drains and receipts confirm | | Delegated intents | Queue state advances or records a specific failure | | Candles | Latest open bucket advances during source-market hours |

Alerts

Alert on:

  • stale prices across active markets
  • repeated Hermes or Data Streams reconnect failures
  • indexer cursor stall or excessive lag
  • database connection or migration failure
  • keeper balance below policy
  • repeated simulation or receipt failures
  • unexpected reorg depth
  • conditionals or delegated intents stuck beyond expiry policy
  • BFF or Vercel health failure

Oracle incident

  1. Confirm whether the source, Railway stream, signer, RPC, or contract verification is failing.
  2. Set affected markets reduce-only or inactive when freshness cannot be restored quickly.
  3. Do not increase report-age limits merely to clear failures.
  4. Preserve logs, source timestamps, reports, hashes, and affected actions.
  5. Restore service, verify multiple fresh rounds, then re-enable only after review.

Keeper incident

Direct wallet trading can remain available while automation is degraded. Stop the keeper replica, inspect deduplication and pending receipts, reconcile on-chain state, restore one executor, and replay only actions that remain valid and unexecuted.

Indexer incident

Compare the stored checkpoint hash with chain state. Roll back to the canonical ancestor or restore Postgres, then replay from a verified block. Chain state remains authoritative during the rebuild.

Frontend incident

Railway non-API routes should continue redirecting to Vercel after recovery. Verify canonical URL, API rewrite, wallet chain, price hydration, charts, and transaction simulation before closing the incident.