Trading bot state persistence: restart without trading blind

Persist evidence, not every temporary variable

Market caches, rolling indicators, and UI state can usually be reconstructed from source data. Money-moving state cannot. If a record answers “what did the bot intend, what did the venue accept, and what exposure exists?”, it belongs on durable storage before the process acknowledges success.

Use the venue's stable identifiers and enforce uniqueness locally. A fill ID should not post twice; a client order ID should map to one intent; an order transition should be append-only or auditable. Keep raw venue payloads where practical so normalization bugs can be investigated later through trading-bot logs.

The venue and local database know different truths

The exchange or broker is authoritative for what it accepted and filled. The local system is authoritative for why it acted, which strategy owned an order, and which event it has processed. Neither view is sufficient alone:

• An order can exist at the venue even if the acknowledgement never reached the database.
• A durable intent can exist locally even if the network request never reached the venue.
• A private-stream fill can be missed during disconnection while venue position state still changes.
• A manual trade can alter the account without a local intent.

Reconciliation joins the two views. It matches client IDs and venue IDs, imports previously unseen fills, compares open-order sets, calculates exposure from authoritative positions or balances, and flags anything it cannot explain. The order state machine in partial fills and reconciliation handles normal live transitions; persistence makes those transitions survive a restart.

Record intent before sending the order

The dangerous boundary is between a local write and a remote side effect. A practical order path is:

1. create a deterministic or unique client order ID;
2. validate the signal through risk and precision gates;
3. write the intent and a pending-submit record in one local transaction;
4. send the order with that client ID;
5. store the acknowledgement and venue order ID;
6. consume fills idempotently and update exposure;
7. mark the intent complete only when its lifecycle is resolved.

minimal durable order record{
  "intent_id": "meanrev-BTCUSD-20260718T101500Z",
  "client_order_id": "mr-7f3c...",
  "status": "SUBMITTING",
  "symbol": "BTC/USD",
  "side": "buy",
  "quantity": "0.0025",
  "strategy_version": "meanrev-12",
  "created_at": "2026-07-18T10:15:00.412Z"
}

If the process dies after step three, recovery can decide whether to submit. If it dies after step four, the stable client ID lets recovery query before acting. That is the core of duplicate-order prevention.

A small single-bot deployment can use an embedded transactional database with backups and integrity checks. A plain JSON file rewritten in place is fragile: a crash during write can truncate it, concurrent tasks can overwrite one another, and there is no transaction joining order intent to risk state. Technology matters less than atomic durability and testable semantics.

A safe startup recovery sequence

1. Start read-only. Disable new entries before connecting any strategy loop.
2. Load and validate local state. Check database integrity, schema version, strategy version, and unfinished intents.
3. Fetch venue snapshots. Obtain current open orders, recent orders and fills, balances, and positions where the API supports them.
4. Match identifiers. Resolve pending submissions by client ID before considering any retry.
5. Replay unseen events. Insert fills with uniqueness constraints and recalculate exposure.
6. Compare invariants. Position quantity, open-order coverage, reserved balance, and risk counters must agree.
7. Quarantine exceptions. Unknown orders, unexplained positions, and missing protection require an alert and an explicit policy.
8. Restore protection. Verify server-side stops or other approved risk-reducing orders without blindly duplicating them.
9. Resume deliberately. Enable strategy entries only after reconciliation records a healthy checkpoint.

This sequence is part of uptime and reliability. Automatic restart without recovery merely repeats a failure faster. If reconciliation cannot establish a known state, connect the condition to the kill-switch policy and require operator review.

Crash at every awkward moment

Happy-path unit tests do not prove persistence. Use a fake venue with deterministic responses, then terminate the bot immediately before and after every durable write and network call. Include these cases:

• intent saved, order never sent;
• order accepted, acknowledgement lost;
• partial fill received, local transaction interrupted;
• fill event delivered twice;
• private stream disconnected while an order fills;
• manual venue order appears during downtime;
• symbol rules or strategy version changed before restart;
• database is unavailable, locked, or fails an integrity check.

The invariant is stronger than “the process came back”: recovery must not double an order, discard a fill, reset a risk halt, or claim healthy status while local and venue exposure disagree. Monitor time to reconcile, unknown-order count, unmatched-fill count, checkpoint age, and backup restore tests.

Frequently asked questions