Webhook ingestion for health data: making at-least-once delivery safe

A provider POSTs to your endpoint: user 7481's activity summary for 2026-07-26 changed. Your handler resolves the user, calls the provider's API for that day, adds 612 kcal to a running total, and returns 200. It took nine seconds. The provider timed out at five and had already queued a retry. The retry lands, does the same work, and now your user burned 1,224 kcal on a day they took one walk.

Nothing threw. No alert fired. The number is simply wrong on the home screen, and it stays wrong until someone opens a support ticket months later and you spend an afternoon proving it was your ingest path rather than their watch. That silence is the whole problem. In an e-commerce backend a duplicated event bills someone twice and you hear about it the same day. In a health backend a duplicated event lands in an aggregate, and a doubled calorie total just looks like a good Tuesday.

So the design decision is not "add idempotency". It is: the delivery and the effect need separate idempotency layers, and the effect must be a replace, never an increment. Everything below follows from that, plus one constraint that drives the whole shape — a slow handler is what manufactures the duplicates you are trying to defend against.

If you are here because a specific provider's pings never arrive at all, that is a different page: see why a Strava webhook stops firing. This one assumes delivery works and asks what happens when it works twice.

Most fitness webhooks do not carry the data

This is the fact that makes the health version of the pattern different, and it is easy to miss because the generic webhook literature is written around payloads that are the event. In fitness, the notification is usually a pointer: something about this user, in this window, for this metric, has changed. Go look.

Two things fall out of that table. First, the on-device platforms have independently arrived at the same two-step shape — notify that something changed, then go read it — which is decent evidence it is the right shape rather than a workaround. Second, and this is the useful part: because the payload is a pointer, the effect of processing it can be made idempotent by construction. Re-fetching a window and replacing what you hold for it is naturally repeatable. Parsing a delta out of a ping and applying it is not, and never will be.

The Health Connect row also has a commercial consequence worth stating early. Nobody can sell you Android health webhooks, because there is no push to resell. What a vendor can sell you is your own app's sync, surfaced to your server as a webhook. That is genuinely useful, but it has your app's reliability characteristics, not a cloud provider's.

Two idempotency layers, not one

The generic advice — dedupe on the event ID — is necessary and not sufficient here. Health providers legitimately re-notify the same day many times as that day fills in. Steps for 2026-07-26 will change at 09:00, at 13:00, and again when the watch syncs at 22:40, and each of those is a real update carrying a distinct event ID. Meanwhile a retried POST carries the same event ID and must do nothing.

So: dedupe the delivery on event identity, and make the effect a versioned replace on a natural key.

The dedupe primitive at the delivery layer is standard and worth citing rather than inventing. CloudEvents requires source plus id to be unique per distinct event and states that if a duplicate event is re-sent, for example due to a network error, it may carry the same id, and consumers may assume that events with identical source and id are duplicates. The Standard Webhooks spec says the same thing from the producer side: the unique id "is often used as an idempotency key, which lets a consumer ensure that they only process a specific event once, even if sent multiple times maliciously, in error, or due to networking issues."

The effect layer is where health-specific judgement enters, and there is one mistake we see constantly. Having correctly decided not to double-insert, people reach for INSERT ... ON CONFLICT DO NOTHING on the daily key. That is exactly as wrong as double-inserting, in the opposite direction: it pins the day to the first value it ever saw, which is the partial morning figure. The user's steps freeze at 2,100 and never move again. It has to be DO UPDATE, gated on a monotonic version.

Both mobile platforms model this as a versioned upsert, which we read as a strong signal it is the right shape. Health Connect documents that clientRecordVersion "determines conflict resolution outcome when there are multiple insertions of the same clientRecordId. Data with the highest clientRecordVersion takes precedence." Apple documents that when you save an object with HKMetadataKeySyncIdentifier, the system compares HKMetadataKeySyncVersion values and "if the new object has a greater sync version, the system replaces the old object with the new one." Neither platform ships a "last one in wins" rule and neither ships an increment.

Note also what Google documents about the gate: upserts do not automatically increment the version, so a stale version is silently ignored. That is excellent for replay safety and a genuine hazard for corrections — a real fix shipped with a stale version disappears without an error. Server-side, make the version monotonic and assign it yourself.

Acknowledge in milliseconds, then process

Here is the loop that catches people. A handler that does the provider fetch inline is slow. Slow handlers time out. Timeouts trigger provider-side retries. Retries are the duplicates. Your handler's latency is a direct input to your duplicate rate, which means the fix for duplicates is partly a fix for latency, not only a fix for keys.

The health-specific sharpening: the fetch you would be tempted to do inline is a rate-limited call against a per-user read quota, and it can 429. That couples your webhook endpoint's response time to the provider's availability, so your endpoint becomes slowest exactly when the provider is having a bad day — which is exactly when it retries hardest. That is a feedback loop, and it is why "acknowledge fast, process async" is a correctness requirement here rather than a performance nicety.

What the endpoint should do, and nothing else:

1. Read the raw request bytes before any middleware touches them.
2. Verify the signature over those exact bytes.
3. Reject if the signed timestamp is outside your replay tolerance.
4. INSERT ... ON CONFLICT DO NOTHING into the delivery table.
5. Return 2xx.

Step 4 conflicting is not an error. A duplicate delivery is a success from the provider's point of view — it delivered, you acknowledged, it stops retrying. Returning a non-2xx on a duplicate is how you convert one duplicate into an escalating retry storm.

There is one documented consequence of failing to acknowledge that is worth taking seriously, because it is the only endpoint-disable policy in this whole area we could actually verify. Apple documents that if you do not call the observer query's completion handler, HealthKit retries with a backoff algorithm, and "if your app fails to respond three times, HealthKit assumes your app can't receive data and stops sending background updates." An Apple DTS engineer has confirmed on the developer forums that this shutdown is as-designed behaviour. So the completion handler must be called on every path including the error branch — including the case where the device was locked and the read failed with errorDatabaseInaccessible, which Apple documents as a real outcome of a background wake and which you should treat as retryable rather than as "no data".

For cloud providers, read the retry schedule and any endpoint-disable policy off the provider's own docs rather than from memory, and do not assume there is no disable policy just because you have not found one.

Ordering is not a guarantee you have

Out-of-order arrival is normal here, not exceptional. A phone that was offline flushes a backlog on reconnect while live events keep arriving, so a two-day-old event and a two-minute-old event contend in the same queue. Some providers are reported to deliver backfilled history through the same webhook callback rather than in the HTTP response of the backfill request; we could not reach the primary documentation to confirm that, so verify it for your provider — but in our judgement you should design as though it were true, because the cost of being wrong is an ingest path that quietly assumes recency and a burst of two-year-old data that violates the assumption. (If you are still moving off polling, the polling-to-webhooks transition playbook covers the switchover; this page is about the read model you land in.)

The rule that follows: never resolve conflicts by arrival time. "Latest wins" applied to received_at will overwrite a fresh sample with a stale one the first time a backlog flushes. Compare on the provider's own version or modified-time field. If the provider gives you neither — and several do not — our fallback is to compare on (sample_end_time, ingested_at) and to record in the row that the ordering was inferred rather than given, so that when a number is disputed you can tell which rows were guesses.

Your delivery table is a copy of the user's health data

Signature verification is the one part of this pipeline that is not health-specific, so treat it as a checklist and copy the spec rather than reasoning about it. The Standard Webhooks spec signs the concatenation msg_id.timestamp.payload and states that including the attempt timestamp "is an important security measure meant to prevent replay attacks". Three lines and no more:

• Verify against the raw request bytes, because the spec warns that consumers "often accidentally parse the body as json, and then serialize it again, which can cause for failed verification due to minor changes in serialization."
• Write the verifier as a loop over a signature list, because the header is space-delimited precisely so a producer can sign with the old and the new key during rotation.
• Reject a signed timestamp outside a fixed tolerance, and keep the delivery-dedupe table for at least the provider's maximum retry horizon. Both numbers come off your provider's docs; we could not verify either.

Fitbit's X-Fitbit-Signature is reported to be HMAC-SHA1 over the request entity keyed by the client secret followed by an ampersand. That came to us from a community thread rather than Fitbit's own documentation, so treat it as a lead and verify before implementing.

Now the part that is specific to this domain, and the reason the section is named after it. Your delivery table is keyed by user and contains raw provider payloads, so it is a copy of the user's health data and it belongs on your erasure inventory. An idempotency table that survives a user-deletion purge will happily accept a late delivery for that user and re-materialize them. Purge the delivery rows and revoke the provider subscription in the same job, or you will meet the "deleted user came back" bug.

A schema sketch

Illustrative, not production-ready — the point is the shape of the keys.

-- Layer 1: the delivery. One row per POST the provider makes.
create table webhook_delivery (
  provider      text        not null,
  delivery_id   text        not null,  -- the provider's event/message id
  signed_at     timestamptz not null,  -- from the signed payload, not your clock
  received_at   timestamptz not null default now(),
  user_id       uuid,                  -- resolved from the provider's subject id
  metric        text,
  window_start  timestamptz,
  window_end    timestamptz,
  status        text        not null,  -- received | fetching | applied | dead
  attempts      int         not null default 0,
  raw_body      bytea       not null,  -- exactly the bytes you verified
  primary key (provider, delivery_id)
);

-- Layer 2: the effect. One row per (user, provider, metric, civil day, source).
create table daily_rollup (
  user_id     uuid        not null,
  provider    text        not null,
  metric      text        not null,
  local_date  date        not null,  -- stored civil date, computed at ingest
  source_id   text        not null,  -- device or writing app, never collapsed away
  value       numeric     not null,
  version     bigint      not null,  -- provider modified-time, or your own counter
  inferred    boolean     not null default false,  -- true if version was guessed
  computed_at timestamptz not null default now(),
  primary key (user_id, provider, metric, local_date, source_id)
);

And the apply, which is the sentence that actually prevents the doubled calorie total:

insert into daily_rollup
  (user_id, provider, metric, local_date, source_id, value, version)
values
  ($1, $2, $3, $4, $5, $6, $7)
on conflict (user_id, provider, metric, local_date, source_id)
do update set
  value       = excluded.value,      -- replace. never daily_rollup.value + excluded.value
  version     = excluded.version,
  computed_at = now()
where excluded.version > daily_rollup.version;

Three details in there earn their keep. local_date is a stored civil date computed at ingest, not a UTC day derived on read — a time_bucket over instants gives a user in UTC+9 a daily total that is nine hours wrong, every day; whose midnight defines the day is the long version. source_id stays in the key because two devices reporting the same hour are two honest measurements, not duplicates, and collapsing them is a different problem with a different fix — see resolving overlapping sources. And the where clause on the do update is what makes an out-of-order arrival a no-op instead of a regression.

The dead-letter queue, and what a human actually does with it

The health-specific DLQ design rule is that the payload alone is not enough. Store user_id, provider, metric and the affected time range alongside the raw body. Replay then means "re-pull this window from the provider", which is idempotent by the same argument as everything else on this page, rather than "re-apply this stored event", which may not be if your adapter has changed since the event was written.

That distinction matters because DLQ entries sort into three buckets and only one of them wants the old payload:

• Transient. Provider 5xx, your deploy, a queue hiccup. Bulk-replay the windows. No thinking required, and this should be a one-click operation, because if it is not, nobody will do it.
• Schema drift. The provider changed something — a unit, a nullable field, an enum value. Replaying the payload re-runs the broken parse. Re-pulling the window against your fixed adapter is the correct action, which is the whole argument for storing the window.
• Unresolvable subject. A notification for a provider user id you no longer have a link for. This one must not be replayed. If the user disconnected or was deleted and the provider subscription is still live, replaying resurrects data you were obliged to erase. The right action is to revoke the subscription, not to fix the row. Whether a subject should resolve to one user or to several is a schema question rather than a queue question, and it is settled in the identity model these callbacks are addressed to.

And the thing that changes your triage SLA: a DLQ entry in a health pipeline is not a lost log line, it is a specific person looking at a specific wrong number. Ours is a per-user, per-metric, per-day queue for exactly that reason — "how many users currently have a wrong day" is a question you can answer from it, and "how many messages are in the DLQ" is not.

What we would actually build

Concretely, for a backend ingesting two or more cloud providers:

1. A thin endpoint. Raw bytes, signature loop, timestamp tolerance, ON CONFLICT DO NOTHING insert, 2xx. No provider calls, no user lookups that can miss a cache, no writes to anything but the delivery table.
2. A resolver worker that turns a delivery into a fetch job keyed on (user_id, provider, metric, window). This is the highest-leverage piece and it is usually missing. Because providers re-notify the same day repeatedly, keying the job on the window rather than on the delivery collapses eleven pings for Tuesday into one API call. Given per-user read quotas, that debounce is often the difference between a backfill that finishes and one that spends its day getting 429d.
3. A fetch worker that re-reads the window from the provider and writes the versioned replace above. It never increments, and it is safe to run twice by construction.
4. A DLQ carrying (user_id, provider, metric, window), triaged into the three buckets above.
5. A reconciliation sweep that re-pulls a rolling recent window for every connected user on a schedule, regardless of whether any webhook arrived. In our judgement this is not optional. At-least-once delivery is a promise about events that get sent; it says nothing about events that were never sent because your endpoint was 502ing during a deploy, or because the provider disabled you under a policy you could not find documented. Apple's documented three-strikes shutdown is proof that at least one platform will stop talking to you permanently and silently. A sweep converts "we lost a day" into "we were a few hours late".

Note what is not on that list: an ordering guarantee, a global sequence number, or any attempt to process events in the order they were produced. You do not need one. If every effect is a versioned replace of a bounded window, order stops being load-bearing, which is a much cheaper property to maintain than ordered delivery.

The honest limits

Retry behaviour is undocumented to us. No specific provider's retry schedule, retry count, replay tolerance, or endpoint-disable policy is verified anywhere on this page, because the relevant vendor docs were unreachable during our research. The Apple three-strikes rule is the exception and it applies to on-device background delivery, not to a cloud webhook. Everything else is yours to look up.

Versioned replace assumes an honest version. If a provider's modified-time is wrong, or absent, or bumped on writes that changed nothing, you do not have ordering — you have a guess with a schema column. Record which rows were ordered on inference so the guess is visible later.

This design cannot recover a deletion it never heard about. Apple documents that deleted objects are temporary and "the system may remove them from the HealthKit store at any time to free up space", and Google documents that a DeletionChange carries only the record id and not the record type, for privacy — meaning you cannot act on an Android deletion unless you stored the Health Connect metadata.id at insert time. That is a schema decision made months before the first deletion arrives, and no amount of webhook hygiene fixes it retroactively.

Buying this is often the right call, and it is right for an unusually clear reason here. The pattern on this page is a day's work. The value is in the long tail underneath it: each provider's signature scheme, subject-id format, retry semantics, backfill delivery channel and re-notification cadence, several of which are not publicly documented at all. That tail is exactly what an aggregator absorbs — one signed endpoint, one event shape, one dedupe contract across providers. If you are integrating more than two or three sources, we would buy: health data aggregator APIs covers who sells that tail, and what a health data aggregator is covers the category if it is new to you.

What would make us build instead: a single provider, where the tail is short and the abstraction earns nothing; a requirement to hold raw provider payloads yourself for audit or compliance reasons; or a latency requirement an intermediary cannot meet. And one caveat in the other direction — an aggregator is another hop with its own at-least-once queue, so it adds a duplicate and reorder surface rather than removing one. You still need the delivery table and the versioned replace. You just need one of them instead of six.

In short

Dedupe the delivery on (provider, delivery_id). Make the effect a versioned replace on (user_id, provider, metric, local_date, source_id). Acknowledge in milliseconds so you stop generating your own duplicates. Order on the provider's version, never on arrival. Keep a sweep running because delivery guarantees are promises about messages that were sent.

Do that and a retried POST is a no-op instead of a support ticket with a real person's calorie total on it.