# Deduplication



Every webhook delivery system worth using is **at-least-once**. The producer retries on failure; the network drops packets; the receiver acks but its acknowledgment doesn't make it back. Plan for the same message arriving twice. Plan for it arriving twice within the same second from two competing workers.

The fix is well-known: index each receipt on a stable identifier (the `webhook-id` header), and ignore the second one. The hard part is the **atomically** — across concurrent workers, across crashes, with no race where two workers both think they're the first.

Postel ships a dedup helper that gets this right. The contract is simple: hand it an ID and a TTL, get back `{ duplicate: boolean }`. Concurrent calls with the same ID see exactly one `duplicate: false` and one or more `duplicate: true`.

## The dedup contract [#the-dedup-contract]

Configure a dedup adapter on the inbound source, then call `postel.inbound.<source>.dedup(messageId, options?)` from the verified-event branch:

```ts
import { Postel, Secret, InMemoryDedup } from "@postel/core";
import { config } from "./config.js";

const postel = Postel({
  inbound: {
    vendor: {
      verify: Secret(config.webhookSecret),
      dedup: InMemoryDedup(),
      dedupTtl: "24h",  // org-wide default; overridable per call
    },
  },
});

// In your route handler — the dedup key is the webhook-id header:
const { event } = await postel.inbound.vendor.verify(body, headers);
const { duplicate } = await postel.inbound.vendor.dedup(headers["webhook-id"]);

if (duplicate) {
  return new Response("duplicate", { status: 200 });
}

// First receipt. Do the work.
await processOrder(event.data);
```

A few things to note:

* **TTL is your choice.** Pick a window longer than the producer's retry budget — 24h to 7d is typical. The downside of a too-short TTL is accepting a duplicate after expiry; a too-long TTL costs more storage. Set `dedupTtl` once on the source or pass `{ ttl: "1h" }` per call.
* **The first-receipt branch must succeed.** If your work fails after `dedup` returns `{ duplicate: false }`, the next retry will be marked duplicate and skipped. Either move dedup after a successful commit, or use the transactional pattern below.
* **Concurrent calls race exactly once.** Two parallel `postel.inbound.vendor.dedup('msg_X')` calls return `{ duplicate: false }` from exactly one of them. The other(s) return `{ duplicate: true }`. The compliance suite tests this.
* **`postel.inbound.<source>.dedup` only exists when an adapter is configured.** Omit `dedup` from the source config and the method is not on the instance type — TypeScript catches the mistake at compile time.

## Storage adapters [#storage-adapters]

`dedup` is wired through a `DedupAdapter` — one method, `record(id, ttlSeconds, options?) → { duplicate }`. Each database storage adapter ships a matching dedup helper alongside its outbound storage; configure it on the inbound source, reusing the same connection. The per-adapter details (options, table schema, SQL strategy) live on each adapter's page:

| Adapter                                                      | Helper        | Strategy                           |
| ------------------------------------------------------------ | ------------- | ---------------------------------- |
| [`@postel/pg`](/docs/storage/pg#receiver-side-dedup)         | `PgDedup`     | `INSERT … ON CONFLICT`             |
| [`@postel/sqlite`](/docs/storage/sqlite#receiver-side-dedup) | `SqliteDedup` | `INSERT OR IGNORE`                 |
| [`@postel/mysql`](/docs/storage/mysql#receiver-side-dedup)   | `MysqlDedup`  | `INSERT … ON DUPLICATE KEY UPDATE` |

`InMemoryDedup()` (shown above) is built into `@postel/core` — per-instance and not shared, so it's fine for tests and single-process development but **not** for production with more than one process, where concurrent receivers can both accept the same message. For anything multi-process, use a database-backed adapter.

Implement `DedupAdapter` yourself to plug in another backend — e.g. Redis, for hosts that already run it. **Postel does not require Redis as a runtime dependency** — that's a deliberate design choice ([ADR 0001](https://github.com/postel-sh/postel/blob/main/decisions/0001-library-shape.md)). If you have Redis already, use it; if you don't, don't stand one up just for dedup.

## Delivery semantics of gate-level dedup [#delivery-semantics-of-gate-level-dedup]

The framework gates (`verifyWebhook` / `withWebhook` / `handleInbound`, via each adapter's `.post(...)`) support the same idea as sugar: pass `dedup: { ttl }` and the gate records the `webhook-id`, responding `2xx` + `X-Postel-Dedup-Result: duplicate` (skipping your handler) on a repeat within the TTL.

```ts
app.post(
  "/webhooks/vendor",
  ...withWebhook(postel.inbound.vendor, handleOrder, { dedup: { ttl: "24h" } }),
);
```

The ordering: the gate records the id, THEN runs your handler. If your handler throws, the gate releases the record it just wrote before the error propagates, so the producer's retry (issued because it saw your 5xx) is treated as unseen and reaches the handler again — a throwing handler never permanently burns the id. It works out of the box with `InMemoryDedup`, `PgDedup`, `SqliteDedup`, and `MysqlDedup`. A custom `DedupAdapter` that doesn't implement `release` simply keeps the id recorded on handler failure.

The release is automatic on every path where the gate wraps your handler — `handleInbound({ onVerified })`, `withWebhook`, every `XxxWebAdapter` facade route, and Hono's `verifyWebhook` middleware. Two gate forms run *before* your handler and can't observe its failure by themselves, so they need one extra line:

* **Fastify's bare `verifyWebhook` preHandler** — attach the exported `releaseDedupOnError` hook as the route's `onError`. (Routes registered through `FastifyWebAdapter` install it for you.)
* **NestJS's `WebhookGuard`** — pair it with `@UseInterceptors(WebhookReleaseInterceptor)` on dedup-gated routes.

One gate form can't release at all: **Express's bare `verifyWebhook` middleware** — Express routes a downstream handler's error to error middleware without re-entering earlier middleware, so a throw there keeps the id recorded for its TTL. Use the `ExpressWebAdapter` facade (or `withWebhook`) with dedup instead.

Two things this does **not** give you:

* **A custom `DedupAdapter` without `release` still burns the id on handler failure.** `release` is optional on `DedupAdapter`; the gate calls it only when the configured adapter provides it.
* **True concurrent duplicates can still both reach the handler.** If two deliveries of the same id arrive close enough together that the first is still inside your handler, the record-before-handler check still guarantees only one of them proceeds — but if *that one* subsequently fails and gets released, the id was briefly "burned" for the other, already-acknowledged delivery. Recovery still happens through the failing delivery's own retry, not through the concurrent duplicate's. If your handler isn't idempotent and you need to rule this out entirely, use the [manual dedup + transaction pattern](#where-to-put-the-dedup-call) below instead of the gate's `dedup` option.

## Where to put the dedup call [#where-to-put-the-dedup-call]

The naive placement is "before the work":

```ts
const { duplicate } = await postel.inbound.vendor.dedup(headers["webhook-id"]);
if (duplicate) return new Response("dup", { status: 200 });
await processOrder(event.data);
```

This is correct as long as `processOrder` is itself idempotent on its own writes. If it isn't — if you're charging a card, sending an email, calling an external API — you want **the dedup row and the side effect to commit together**.

The cleanest pattern is the **transactional outbox** in reverse: the dedup insert participates in the same DB transaction as the work. Every dedup call accepts an optional `tx` parameter that the adapter threads through to its storage.

```ts
await db.transaction(async (tx) => {
  const { duplicate } = await postel.inbound.vendor.dedup(headers["webhook-id"], { tx });
  if (duplicate) return; // tx commits with no work; safe.

  await processOrder(tx, event.data);  // joins the same tx
});
```

If `processOrder` throws, both the dedup row and the work roll back. The next retry sees the ID as unseen and tries again. If `processOrder` succeeds, both commit; the next retry is correctly skipped.

This is exactly the pattern the [outbound](/docs/outbound) sender uses for outbox inserts — the receiver's dedup row is the mirror image of the sender's outbox row.

## What's next [#whats-next]

* [Raw bytes](/docs/concepts/raw-bytes) — the silent failure mode that breaks signature verification.
* [Key rotation](/docs/inbound/key-rotation) — rotating without dropped messages.