Event dedup by timestamp watermark loses events
Polling an event API with an overlapping cursor ("read from the last position minus ten minutes, events may land late") is the right design — but only if the consumer can tell an already-counted event from a new one inside that overlap. A single "last counted timestamp" cannot: it drops the second of two events recorded in the same second (Mailchimp, for example, counts a click as an open when the tracking image did not load, and both can carry the same timestamp) and every late event whose timestamp is older than one already counted. Let the watermark decide only what is older than the overlap; inside it, dedupe by event identity — action + timestamp, with an occurrence count — and keep that set only for the overlap window.
Čtení událostí z cizího API s překryvem kurzoru („od poslední pozice minus deset minut, události chodí se zpožděním“) je správný návrh – ale jen když příjemce v tom překryvu pozná už započtenou událost od nové. Jediný „čas poslední započtené události“ to neumí: zahodí druhou ze dvou událostí zapsaných ve stejné sekundě (Mailchimp třeba počítá proklik i jako otevření, když se sledovací obrázek nenačetl, a obě můžou mít stejný čas) a každou opožděnou událost s časem starším než už započtená. Vodoznak ať rozhoduje jen o tom, co je starší než překryv; uvnitř něj deduplikuj podle identity události – akce + čas, i s počtem výskytů – a tu sadu drž jen pro okno překryvu.
Symptom
The email platform's campaign report shows 1 open and 1 click for a recipient. Your app shows "opened 1×" and no click — and the click never appears, no matter how many times you sync again, even though the cursor re-reads the last ten minutes on every run.
Why it happens
The per-recipient summary stored only the timestamp of the last counted event (lastAt) and skipped
anything not newer:
events.sort(byTimestamp).forEach(e => {
if (lastAt && e.at <= lastAt) return; // "already counted"
count(e);
lastAt = e.at;
});Two ordinary situations break it:
- Same-second events. Mailchimp counts a click as an open when the tracking image did not
load (About Open and Click Rates) —
common in mail clients that block images. In the case behind this article, the open and the
click came back from the email-activity endpoint with the same one-second timestamp. The first
one moves
lastAt; the second is<= lastAtand is dropped — on this run and on every later re-read. - Late events. The overlap exists because an event can become readable later than its timestamp suggests. If the per-recipient watermark has already moved past that timestamp, the overlap re-reads the event and the watermark throws it away. The watermark defeats the overlap it relies on.
Fix
const OVERLAP_MS = 10 * 60 * 1000;
// seen: { "o2026-09-28T10:46:03+00:00": 1, "k2026-09-28T10:46:03+00:00": 1 } – kept per recipient
const cutoff = lastAt ? Date.parse(lastAt) - OVERLAP_MS : NaN;
const batch: Record<string, number> = {};
events.sort(byTimestamp).forEach(e => {
if (!isNaN(cutoff) && !(Date.parse(e.at) > cutoff)) return; // older than the overlap
const key = e.action[0] + e.at;
batch[key] = (batch[key] || 0) + 1;
if (batch[key] <= (seen[key] || 0)) return; // this occurrence already counted
count(e);
lastAt = lastAt && lastAt > e.at ? lastAt : e.at; // never move backwards
});
// store max(seen, batch) per key, pruned to keys newer than lastAt - OVERLAP_MS- Count occurrences, not just presence: two clicks on different links in the same second are two.
- Prune the set to the overlap window so it stays a handful of entries per recipient.
- Records written before the fix have no set. Derive it from what the summary knows (last open or
last click equal to
lastAtcounts as seen once), so the next run picks up what was dropped.
Tests worth having
- Two different actions with the same timestamp in one batch → both counted; re-read → no change.
- A late event older than the last counted one, inside the overlap → counted once; re-read → no change.
- Two identical actions in the same second → counted twice; re-read → still two.
"Re-reading the same events changes nothing" alone passes with the buggy watermark — it is the simultaneous and the late case that expose it.