Networks retry. Browsers refire. Mobile SDKs flush the same queue twice. Every event pipeline is at-least-once whether you designed it that way or not, and without explicit deduplication those duplicates land directly in your conversion counts and lift numbers. This post covers dedupe key design and what duplicate events do to experiment analysis.
01Where duplicates come from
A duplicate event isn't a bug in any one component: it's what a distributed pipeline produces by default. A browser fires a beacon on page unload, the request is in flight when the tab closes, the SDK can't confirm delivery, and it queues the event again on the next load. A mobile client goes offline mid-flush, reconnects, and replays its local queue without knowing the first attempt actually landed. A collector accepts a write, commits it, and then the acknowledgment is lost on the way back to the sender, so the sender retries a request that already succeeded. None of this is misconfiguration; it's what "the network is unreliable" looks like in practice.
The infrastructure underneath doesn't close the gap as much as vendors imply. Kafka's idempotent producer only dedupes retries at the broker using producer IDs and sequence numbers; it says nothing about a duplicate minted upstream of the producer. Google Pub/Sub's exactly-once guarantee applies to pull subscribers in a single region and explicitly still allows publish-side duplicates. Dataflow's default exactly-once processing covers pipeline results, not external side effects: a `DoFn` can run more than once, and Dataflow deduplicates its own output after the fact, which doesn't help if the side effect already fired. Flink separates exactly-once state from exactly-once record delivery and only gets end-to-end guarantees when both the source and the sink participate in checkpointing. Every one of these systems is telling you the same thing: the guarantee is scoped to a boundary, and your conversion pipeline almost never lives entirely inside that boundary.
This is the same argument Saltzer, Reed, and Clark made about end-to-end correctness decades ago: some properties can only be fully guaranteed at the endpoints that actually understand the semantics, because every layer below can help but none of them can close the problem on its own. Applied here, only the application knows that "purchase 123 completed" fired by a retry, a backfill, and a redelivered webhook is the same semantic event. The transport can reduce duplicate volume; it cannot eliminate the need to decide, at the point where you record the event, whether you've already seen it.
02Why exactly-once is a myth you buy with idempotency
The workable contract is at-least-once delivery plus idempotent processing, not exactly-once delivery. That reframing matters because it puts the correctness obligation on the receiving side, where it can actually be discharged, instead of on the transport, where it can't be guaranteed end to end. The IETF's idempotency-key draft states the requirement plainly: the key must uniquely identify the semantic operation, must never be reused for a different payload, and should be paired with a fingerprint of the request so a server can tell a legitimate retry from an accidental key collision. Stripe and PayPal both implement this pattern for API mutations: a client-generated key, typically a UUID, that the server compares against the original request on every retry.
Analytics and experimentation events want a slightly different key shape than a payment API call, because the "operation" is usually a business fact with its own natural identity: an order ID, a subscription renewal, a specific exposure. A deterministic business key derived from that fact is often cleaner than a random retry token, because it makes replays and backfills idempotent for free: reprocess the same order twice, get the same key, get one row. The catch is namespace hygiene. A bare order ID is not a safe dedupe key the moment more than one environment, tenant, or event type can produce it: `prod:checkout_completed:merchant_42:order_12345` is a materially different (and safer) key than `12345`. Segment's own `messageId` dedupe is explicitly not scoped to a single source or workspace, which is exactly the failure mode a low-entropy key walks into.
TTL is the other place teams underestimate the problem. The instinct is to set the dedupe window to something a bit longer than your API timeout. The real requirement is to set it longer than your worst realistic replay horizon: mobile offline queues, delayed browser resends, a cautious backfill run days later. Segment keeps `messageId`s for roughly 24 hours at ingestion but extends Data Lake and Warehouse dedupe to a 7-day window on the same field; Amplitude dedupes `insert_id` over 7 days; Snowplow's cross-batch DynamoDB state carries its own TTL. A 24-hour window, which sounds generous, is routinely too short once backfills enter the picture, and for random keys, use enough entropy that collisions aren't a second source of false duplicates: 128-bit random identifiers are effectively collision-free at any scale an experimentation pipeline will hit, where 64-bit keys start accumulating real collision probability once you're issuing billions of them.
03Duplicates and experiment bias
Whether a duplicate hurts your experiment depends entirely on how the metric is defined, not on whether duplicates exist. A binary "did this user convert at least once in the window" metric, implemented as `COUNT(DISTINCT user_id)` or an `EXISTS` semi-join, absorbs duplicate rows for free: the same user converting twice is still one converter, so the point estimate is unchanged as long as identity and window stay stable across the retries. That safe case is narrower than it sounds, though. The moment a metric counts events instead of distinct users, or sums revenue per order, or defines its window on arrival time instead of business-event time, duplicates stop being harmless.
The reason this is a real threat to experiment validity (not just a data-quality nuisance) is that duplicate rates are rarely balanced across arms. A heavier variant page, a slower checkout flow, a variant that triggers more retry-prone client code, all of these correlate duplicate volume with treatment assignment. Work through the algebra and the result is uncomfortably sensitive: if treatment and control experience duplicate inflation rates δ_T and δ_C respectively, observed relative lift is (1+L)·(1+δ_T)/(1+δ_C) − 1, where L is the true lift. Balanced duplicates cancel out of that ratio entirely, but for small imbalances the bias is approximately (1+L)·(δ_T − δ_C), which means roughly one percentage point of relative-lift bias for every extra percentage point of duplicate-rate imbalance between arms. Under a true null effect, a 1-point gap between a 1% control duplicate rate and a 2% treatment duplicate rate alone produces an observed lift near 1%: a false positive built entirely out of retry noise, no real effect required.
Value-weighted metrics make this worse, not better. If duplicates are more likely on expensive events (a flaky payment provider retries more on larger charges, say), then revenue-per-user bias exceeds what the simple count-based model predicts, because the extra duplicate mass is concentrated on your highest-value rows. And window definition compounds the problem in a subtle way: if a metric window is measured from arrival time rather than the canonical event timestamp, a delayed retry duplicate is more likely to land inside a long weekly window than a short six-hour one, so identical underlying retry behavior produces different observed bias depending purely on window length. None of this is a reason to distrust experimentation: it's a reason to know exactly which of your metrics are unique-converter metrics (safe) and which are event- or value-count metrics (not safe without explicit dedupe), and to treat that as a design decision rather than an afterthought.
04Two layers of dedupe, kept separate
TraqLyte runs two dedupe mechanisms that solve different problems and deliberately don't share a code path. `Outcome` rows dedupe on `(assignment_id, dedupe_key)`: a domain-level guarantee that the same conversion event, identified by the caller's own business key, lands exactly once no matter how many times an at-least-once pipeline redelivers it. This is scoped to a single table, and it requires the caller to know which event is which; it says nothing about any other endpoint. Because `POST /outcome` already has this baked in, a duplicate POST with the same `dedupe_key` is simply a no-op that returns the same result: no separate transport-level protection is needed there, and none is applied.
Separately, `app/idempotency.py` implements transport-level idempotency for arbitrary mutating JSON routes, keyed on the `X-TL-Idempotency-Key` header and deduped on `(key, checksum)` per the security baseline. It exists to make an HTTP mutation survive a network retry by replaying the original response byte-for-byte, not to interpret business meaning. The reservation is inserted and committed before the handler runs, which is what lets the row's uniqueness constraint serialize concurrent duplicates across worker processes; an in-process lock wouldn't reach a second uvicorn worker. A key reused with a different request body is a 409, never a replay, because answering a different question with a cached answer is worse than refusing outright. And a key that's still in flight, versus one whose owning process died mid-request, is disambiguated purely by a reservation-age cutoff rather than any liveness check, since nothing else is available.
The reason these stay two mechanisms rather than being merged into one is what each one actually knows. `Outcome.dedupe_key` is caller-supplied and business-scoped: it can only ever protect the one table it's defined on, and it requires domain knowledge the transport layer doesn't have. The idempotency store is header-driven and route-agnostic: it protects any mutation, but has no idea whether two different-looking requests represent the same underlying business fact. The concrete case that motivated adding the transport layer was `POST /campaigns/{id}/publish`, which is not naturally idempotent: a duplicate request, absent this protection, creates a genuinely new `CampaignVersion` rather than a no-op. That's a transport problem, not a domain-dedupe problem, and solving it by overloading `Outcome`'s dedupe_key mechanism onto every route would have conflated two different questions ("did the network deliver this twice?" and "did the business event happen twice?") that need independently correct answers.
Sources
- Apache Kafka documentation: idempotent producer semantics, transactional exactly-once between Kafka topics, and delivery-guarantee definitions
- Google Cloud Pub/Sub documentation: scope and limits of exactly-once delivery for pull subscriptions, including publish-side duplicate caveats
- Google Cloud Dataflow documentation: exactly-once processing of pipeline results versus non-guaranteed external side effects
- Apache Flink documentation: exactly-once state versus exactly-once end-to-end record delivery
- Saltzer, Reed & Clark, "End-to-End Arguments in System Design": the foundational argument for why correctness belongs at the application endpoint
- IETF idempotency-key draft: key uniqueness, payload fingerprinting, and expiry-policy recommendations for HTTP mutations
- Stripe and PayPal API documentation: production idempotency-key implementations for payment mutations
- Segment documentation: `messageId`-based dedupe at API ingress, Warehouse, and Data Lake, and its unscoped-namespace limitation
- Snowplow documentation: at-least-once delivery model, `event_id`/`event_fingerprint`, and in-batch/cross-batch dedupe modes in the Spark transformer
- Amplitude documentation: `insert_id`-based dedupe windows for ingestion and backfills