Trading bot clock synchronization: fix timestamp errors

Why signed trading requests need accurate time

Many private exchange endpoints require a timestamp or nonce inside the signed request. The venue checks both the signature and freshness. That prevents an intercepted instruction from being replayed much later. If the client's clock is too far ahead, too far behind, or the request waited too long between signing and arrival, the freshness check fails.

Clock-related failures have several sources:

• host drift — the server clock gradually diverges because synchronization is disabled or unhealthy;
• clock step — virtualization, suspend/resume, or a time service moves wall time abruptly;
• queue age — a request is signed, then sits behind other work before it is sent;
• network delay — routing or venue latency consumes the freshness window;
• unit errors — seconds are sent where milliseconds are expected, or a value is rounded incorrectly;
• stale offset — the client measured venue time once and kept that correction indefinitely.

These are authentication failures, not trading signals. Retrying the same already-signed payload only makes it older. Generate a fresh timestamp and signature for each attempt, subject to the safe retry rules in trading bot error handling.

Measure server offset and network uncertainty

If a venue exposes public server time, take a local timestamp immediately before the request and immediately after the response. The midpoint of those local readings is a better estimate than the receive time alone:

offset estimateround_trip = local_after - local_before
local_midpoint = (local_before + local_after) / 2
estimated_offset = venue_time - local_midpoint

uncertainty is at least about half the round trip

Take several samples and prefer a low-latency median rather than trusting one slow response. Store offset and round-trip time as separate metrics. A small estimated offset with huge round-trip uncertainty is not a clean health signal.

Design rules that prevent drift failures

1. Synchronize every host. Use the operating system's supported time service with multiple reliable peers, then monitor whether it is actually synchronized.
2. Sign immediately before send. Do not build signed requests in a queue. Queue the intent, then timestamp and sign at the network boundary.
3. Use wall time only where required. Signed calendar timestamps need wall time; elapsed durations and cooldowns should use a monotonic clock that cannot move backward.
4. Measure venue offset. Refresh it periodically and after reconnect, resume, migration, or a timestamp rejection.
5. Set a drift threshold. Warn well before the venue rejects requests; pause new exposure if reliable authenticated execution is in doubt.
6. Keep freshness windows intentional. A larger receive window can tolerate latency, but it should not be used to conceal a broken clock or overloaded queue.
7. Keep clocks out of strategy bars. Use exchange event timestamps consistently for candles and fills, while preserving receive time for latency analysis.

Never call wall time to measure a timeout. If the system clock steps backward, a wall-time timeout may last far too long; if it steps forward, it may fire immediately. Monotonic timers are appropriate for request deadlines, heartbeat age, cooldowns, and retry backoff.

Timestamp-error incident runbook

Repeated timestamp rejections can mean the bot cannot reliably manage exposure. Use a defined response:

1. pause new entries at the order gateway;
2. record the venue error, local wall time, monotonic latency, queue age, and last known offset;
3. discard stale signed payloads and stop blind retry loops;
4. verify host time-service synchronization without changing unrelated system settings;
5. sample the venue time endpoint and estimate offset plus round-trip uncertainty;
6. check timestamp units and confirm signing happens immediately before send;
7. send a safe authenticated read, then reconcile orders and positions;
8. resume only after several healthy samples and successful account reads.

If risk-reducing orders also require authenticated timestamps, the issue belongs in the highest operational alert tier. A separate broker interface or server-side protective order may provide defense in depth, but never assume a local stop can act while authentication is broken. Connect repeated clock failures to the trading bot kill switch.

Monitor the complete time path

A single “system clock okay” metric is not enough. Track:

• host synchronization state and estimated local offset;
• venue server-time offset and round-trip time;
• age from intent creation to signing and from signing to network send;
• timestamp rejection count by venue and endpoint;
• request queue depth and oldest item age;
• wall-clock step or host resume events;
• last successful authenticated read, order action, and reconciliation.

In a test adapter, inject an ahead clock, a behind clock, a sudden time jump, long queue delays, seconds-versus-milliseconds mistakes, and slow venue responses. Verify that monotonic deadlines still work, payloads are re-signed, the bot does not duplicate orders, and new entries remain paused until state is known. Also combine the test with simulated API throttling; rate-limit queues are a common source of stale signed requests.

Frequently asked questions