Probalytics Probalytics

Architecture

How the data is collected.

Every claim on this site about completeness depends on what happens upstream of the warehouse. This is that.

Collectors

Four redundant collectors per venue, on separate hosts with separate network paths. Redundancy is at the host and path level — one process dying, one uplink flapping or one subscription stalling does not take the others with it.

Polymarket and Kalshi run as separate collector fleets. Every collector subscribes to the full market set on its venue rather than a shard of it, so any single collector is sufficient to reconstruct a continuous stream.

Deduplication and reconciliation

Streams are merged on a hash of the resulting order book, scoped to market. The venue's sequence number can't serve as the merge key — different collectors receive different sequence numbers for the same change — so the book state itself is what identifies an event. Identical states from different collectors collapse to one row; the collector that observed it first is recorded, so ingest latency stays measurable per host.

Gap detection

Detection differs by venue, because each feed gives us something different to check against. Both mechanisms run continuously on the merged stream, and both end in the same place: an incident and a snapshot request.

Kalshi
Kalshi sends a sequence number per market. A missing sequence is detected on the merged stream, which raises an incident and triggers an automatic re-snapshot of that market.
Polymarket
Polymarket publishes a hash of the book. We reconstruct the book from the events we received and compare the two; a mismatch means we missed something, which raises an incident and requests a snapshot.

Reconnect behaviour

On re-subscription a single collector sees a brief gap before its stream resumes. Because the other collectors on that venue stay subscribed, the merged stream normally has no gap at all.

On reconnect a collector receives a fresh snapshot on its subscription and continues from there. Redeployments and updates are staged deliberately: collectors are rolled gradually, so cover is never removed from a venue and a deploy produces no gap.

Initial state for a newly observed market comes from the same snapshot, delivered at first subscription.

Provenance by table

VenueTableSource
PolymarketfillsPolygon (indexed on-chain)
PolymarketmarketsVenue API
PolymarketorderbookVenue API
KalshifillsVenue API
KalshimarketsVenue API
KalshiorderbookVenue API

Polymarket fills are indexed from Polygon, so any trade we report can be checked against the public ledger. Everything else comes from each venue's API and is stored on our side.

Storage

Book events land in ClickHouse and are written out to object storage as daily Parquet partitions, which are the durable copy. Retention is indefinite: rows are never aged out, because the archive's value is that a venue can't retract what we already hold.

Back to home