Distributed Trace Clock-Skew Reconciler

Company: Cisco

Difficulty: hard

Problem Statement

Distributed Trace Clock-Skew Reconciler A payments platform has just survived a multi-hour outage and the forensics team is reconstructing what happened. The evidence is a pile of logs pulled from M microservice hosts. Each host stamps its log lines with its own local clock, and every one of those clocks runs at a fixed, unknown offset from the others. Until the offsets are pinned down, the logs cannot be merged into a single timeline. The distributed tracing system emits calibration records . When a traced request crosses from host a to host b , the tracer measures the hop and reports the exact difference between the two hosts' clock readings for that instant. Each such record asserts a hard fact of the form "host b 's clock reads exactly d microseconds ahead of host a 's clock", where d may be negative. Chaining these facts propagates knowledge: once a and b are related and b and c are related, the offset between a and c is fully determined too. The evidence is not clean. A handful o