Monitoring a Health Data Pipeline for Silent Failures

A health ingestion pipeline that breaks loudly is a good day. The failure that actually costs you is the other one: a webhook subscription lapses, or a provider starts sending sleep duration in milliseconds instead of seconds, and nothing throws. Queue depth is fine. Error rate is zero. Your dashboards are green. Six weeks later a user emails to say their sleep chart went blank in June, and you discover you have six weeks of missing nights you cannot backfill because the provider's history window has closed behind you.

That is the design problem. A health pipeline's dominant failure mode is silence, and silence is not an exception you can catch. You have to go looking for it, on a schedule, per provider. The decision this page argues for: treat ingestion as a monitored system with its own SLOs and its own expected-shape assertions, defined per provider — not as an appendix to your application observability. Alert on the absence and the shape of data, not on exceptions.

Why "per provider" is the whole game

Suppose you serve users across six providers and Oura goes dark. Your total sample count drops by whatever share Oura holds — call it eight percent. Health data has strong weekly cycles, because weekends genuinely differ from weekdays, so an eight percent dip on a Saturday sits comfortably inside the normal band. Nothing fires. Meanwhile every Oura user has a broken product, and they are the ones who paid for a ring.

The deeper reason a global check cannot work is that the providers have structurally different sync cadences, and this is a health-specific fact rather than a general one. Apple's data reaches your server when the user opens your app or when a background wake happens — and Apple documents only an upper bound on background delivery ("at most once per time period"), never a minimum rate, plus a documented shutdown after three unacknowledged deliveries — designing around a wake you may never get is where those two facts become a client architecture. Health Connect has no push mechanism at all; Google documents flatly that "your app can't get notified of new data", so the client checks when it is in the foreground. A ring syncs when it is charged. A watch syncs when it is near a phone. A single global "no data in six hours" alarm over that mixture is either always firing or never firing, and both are the same amount of useless.

Freshness SLOs, one per provider

The most copyable model here is dbt's source freshness: a freshness block with a warn_after and an error_after (each a count plus a period), pointed at a loaded_at_field, with thresholds inherited from the source and overridable per table. Two thresholds, declared as configuration, checked on a schedule. dbt's docs call this "a critical component of defining Service Level Agreements (SLAs)", and the per-table override structure is exactly the shape a health pipeline needs — the threshold belongs to the provider, not to the pipeline.

The health adaptation that matters more, though, is what you measure. The naive metric is "when was the last row inserted for this provider", and it is close to worthless, because one very active user keeps it green while the other ninety-nine percent of the cohort has gone dark. The metric that works is the fraction of active users for this provider whose newest sample is older than N hours. A provider outage shows up as a slow drift in that fraction long before it shows up as zero rows, because a few users always sync.

You then have to define "active", and this is a judgement call with no right answer. A user who connected Fitbit in March and stopped wearing it is churn, not a pipeline incident, and leaving them in the denominator means your freshness metric slowly degrades forever. A reasonable default is "users with at least one sample from this provider in the trailing 14 days", accepting that a genuine two-week outage will start eating its own denominator.

# Illustrative only. Every number here is invented.
providers:
  apple_health:      # app-open and background-wake driven
    stale_after:      { warn: 36h, error: 96h }
    cohort_breach:    { warn: 0.15, error: 0.30 }
  health_connect:    # no push; client polls in foreground
    stale_after:      { warn: 36h, error: 96h }
    cohort_breach:    { warn: 0.15, error: 0.30 }
  oura:              # syncs when the ring is charged
    stale_after:      { warn: 30h, error: 72h }
    cohort_breach:    { warn: 0.10, error: 0.25 }

Do not copy those numbers. Derive yours from your own trailing data: measure the observed inter-sync gap per provider over a normal month, take the p95, and set warn somewhere above it. That is the only defensible way to pick a threshold, and you will retune it after every provider change.

Volume anomalies, weekday-matched and per metric

Once freshness is in place, volume anomaly detection catches the partial failures freshness misses. Two rules make it usable rather than noisy.

First, compare against the same weekday in prior weeks, per provider. Never day-over-day. The weekly cycle in health data is large enough that a naive "today is below eighty percent of yesterday" rule fires every Monday morning and every Saturday, and you will mute it within a fortnight.

Second, segment by metric, not just by provider, because the common real failure is partial. A provider keeps delivering steps and quietly stops delivering sleep — a scope was dropped in a re-consent flow, or a permission was revoked for one type. A provider-level volume check passes with flying colours; a provider-by-metric check catches it the next morning.

There is a third series worth putting on the same chart: new connections per provider. A broken OAuth redirect after a deploy stops all new links for one provider while every existing user keeps flowing normally. Total ingest volume is unchanged, so nothing you have looked at so far will notice. The only signal is that a provider that normally acquires connections every day has acquired none since Tuesday.

Implausible values, and why the 100,000-step day is not one bug

The classic detector is a range check, and the primitive is solved — Great Expectations ships ExpectColumnValuesToBeBetween with min_value, max_value and strictness flags, evaluated per row and then reduced to a percentage of rows that passed. That percentage semantics is the important part, and we will come back to it.

The interesting question is what a 100,000-step day actually means, because it is usually not a sensor error. In our experience it is one of three things: a manual entry with a fat-fingered extra zero, a unit confusion, or double-counting from two sources that both covered the same hours. The remedy differs completely across those three, which means the detector is useless unless it carries provenance.

Both platforms hand you that provenance — HKMetadataKeyWasUserEntered on iOS, recordingMethod on Health Connect — and both hand it over with caveats that make it three states rather than a boolean, which the page that branches on that flag sets out in full. For monitoring, the only thing that has to be true is that the flag survived every hop and reached this layer. Without it, every check below is ambiguous about which of the three causes it just found.

With provenance attached, the branch is clean. A manual entry at an implausible magnitude gets a much wider, differently shaped envelope and surfaces to the user as "confirm this entry?" — it is their typo, not your incident. A device-recorded sample at the same magnitude is your bug, and it goes to you.

The checks we would actually write, in descending order of how often they catch something real:

• Structural invariants, which need no physiology at all: two sleep sessions from the same source that intersect; a workout whose duration exceeds the gap between its own start and end; a daily total that does not equal the sum of its own intraday buckets. These fire almost exclusively on genuine ingestion bugs. Start here.
• Duration bounds: a sleep session longer than the night it belongs to. Compute the duration from instants, never from local times — a spring-forward night is 23 hours and an autumn night is 25, so a local-time duration check has two guaranteed false-positive days per user per year.
• Rate-of-change bounds: body weight moving several kilograms in a day is far more likely to be a unit swap or a second person on the scale than a physiological event.
• Physiological ranges for heart rate and similar: useful, and the most dangerous to over-tune. A device with poor optical contact legitimately reports a spurious zero. A real athlete has a resting heart rate that looks like an error. A real person does occasionally run an ultramarathon.

Which brings back the percentage semantics. Physiological outliers are real, so a zero-tolerance bound fires constantly and teaches everyone to ignore it. Alert on the rate of out-of-range values per provider per day, split by recording method, and never on an individual row. These bounds are engineering judgement, not clinical reference ranges, and anything you intend to act on medically should be reviewed by someone with a physiology background before it ships.

The nastiest failure: a silent semantic change

Type validation catches none of the changes that actually hurt, because the changes that hurt keep the type and change the meaning. Kilocalories become kilojoules. Metres become feet. Seconds become milliseconds. An integer bpm becomes a float. A nullable field that was always populated quietly starts being null. An enum gains a value your switch statement does not handle. Every one of those passes schema validation, deserialises cleanly, and produces wrong numbers on a user's screen with no error anywhere in your system.

What catches them is distribution monitoring: compute per-provider, per-metric statistics daily — median, p99, null rate, and for enums the distinct-value set — and alert on step changes. A kilojoule-for-kilocalorie swap is roughly a 4.18x shift in the median that appears within a single day across every user of one provider. Trivially visible as a distribution shift; completely invisible to schema validation.

Add one cheaper trick alongside it: hash the sorted set of key paths in each provider's response, per provider per day, and diff it. An unexpected new key is a free early warning that something changed upstream, usually days before the change reaches a field you read. Store the shape, not the payload — a monitoring store that accumulates raw health values is a compliance surface you did not mean to create.

For a concrete, documented example of exactly this class of change on a platform you already ship on: Google documents that from the June 2026 update, steps counted natively by Health Connect are attributed to a per-device, per-app Synthetic Package Name rather than the package name android, and that historical data recorded before that keeps android. Both values coexist permanently. If you group or dedupe by dataOrigin.packageName, one physical source splits into two across that boundary — no error, no type change, just a step count that starts behaving oddly. A monitor on the distinct-source set per user would have flagged it on day one.

A detection here is also a versioning decision, not just an alert: once you know the meaning changed, you have to decide whether history is recomputed under the new interpretation or partitioned at the cutover. That is the subject of metric versioning and recompute.

Reconciliation: compare your total against the platform's own

Where a platform publishes its own aggregate, compare yours against it. This is the check that catches dedupe bugs, and essentially nothing else does.

On Apple, HKStatisticsQuery and HKStatisticsCollectionQuery merge data across sources before calculating; a plain HKSampleQuery performs no merging at all. Apple's own framework engineers have said on the developer forums that an app reading through HKSampleQuery has to arbitrate between overlapping samples itself and is unlikely to reproduce HealthKit's merge correctly. Note also that statistics queries are quantity-only — there is no equivalent aggregate to reconcile workouts against.

On Android, Health Connect deduplicates only Activity and Sleep, only through the Aggregate API, and the ordering it uses comes from a priority list Google documents as user-owned: "Only end users can alter these priority lists." Your backend cannot read it, and the user can reorder it with no event emitted to you.

So be honest about what this check is. It is a drift detector, not a correctness target. You will not match, the gap is expected, and chasing it to zero is a waste of a quarter. The useful form is to track the distribution of the delta per provider per day and alert when the distribution moves. A steady three percent spread that becomes a forty percent spread overnight is a dedupe regression, or a newly connected source you are summing where you should be merging — which is precisely the bug that shows up as a doubled step count on someone's home screen. The mechanics of the merge itself belong to deduplicating health data across sources; here it is only a signal.

Practically, the cheapest implementation runs on the client: over a recent window, run the platform's aggregate query and your own sum, and ship both figures up as one row per user per day. Apple documents that statistical calculations "can take a considerable amount of time" over large numbers of samples, so scope it to a recent window rather than the full history.

Signal, cause, alert

What we would actually build

Build it in this order and stop when you run out of appetite. The first two catch most of what hurts.

1. Per-provider freshness on the cohort fraction. One scheduled query, one row per provider per run, one chart. It is the cheapest thing on the list and it catches the failure that costs the most.
2. Structural invariants. No physiology, no thresholds to argue about, and they fire only on real bugs.
3. Daily per-provider distribution statistics. Cheap, and the only thing on the list that catches a semantic change.
4. Weekday-matched volume bands per provider per metric, plus the new-connection series.
5. Reconciliation against the platform aggregate, treated as drift.

Mechanically this is one nightly job writing one row per (provider, metric, local_date) carrying sample count, distinct users, median, p99, null rate, out-of-range count split by recording method, and invariant-violation counts. That table is the monitoring substrate and every check above is a query over it, including the weekday comparison, which is just a self-join four weeks back. Retain it far longer than raw samples: it holds shape, not health values, so retention is a cheap decision rather than a compliance conversation.

One design rule is worth stating explicitly, because it is easy to get backwards. Monitor the rollups your product actually serves, not the raw sample table. A monitor that queries raw samples while your app reads from a rollup will sit green through a broken rollup refresh, and the rollup is what the user sees.

The honest limits

Every threshold on this page is a guess, including ours. There is no published freshness SLO for wearable providers, no standard step-count bound, and no reference distribution to compare yours against. Derive thresholds from your own trailing data and expect to retune them after every provider change. Anyone handing you a table of universal thresholds is selling something.

Alert fatigue is the real failure mode of this design. If you alert on every anomaly you will train the team to close alerts unread, and the one that mattered will be closed unread with the rest. Three disciplines help: page only on things that are both actionable and affecting users right now (a provider dark for most of its cohort pages; a key-path hash change becomes a ticket); make every alert name the provider and the affected cohort size, because "anomaly detected in ingestion" is a chore, not an alert; and delete or retune any check that fires more than once a week without producing an action. An unactioned check is worse than no check, because it consumes the attention the real ones need.

What this design still gets wrong:

• It cannot distinguish a broken pipeline from a user who stopped wearing the device. Freshness fires identically for both, and the classification is genuinely hard. You will live with false positives here — see missing data and gaps for how to represent the difference downstream once you have guessed at it.
• It cannot see deletions you never received. Apple documents that deleted objects are temporary and the system may remove them at any time, so a client that was offline long enough may never learn a sample was deleted, and no server-side diff recovers it — the sample is not in HealthKit any more to compare against. Every check above will report a fresh, plausible, healthy pipeline holding data the user believes they erased.
• It fires on upstream improvements as loudly as on bugs. If a vendor ships a better sleep-staging model, your median deep-sleep minutes shift and the distribution monitor is correct to fire, and there is nothing to fix. Separating these from real defects costs real engineer time and there is no clever trick for it.
• It says nothing about whether the numbers were right to begin with. Every check here detects change from a baseline. Ship with a bad unit conversion and the baseline is wrong, the pipeline is stable, and everything stays green forever.

When to buy instead of build. This is a lot of machinery for a team whose product is not a data platform, and an aggregator legitimately removes much of it: one normalized schema, one delivery mechanism, and a vendor whose own monitoring covers provider outages because their whole business depends on noticing them. With a small team and several providers, buying is very likely the right call and building this yourself is a distraction from your actual product. We would build in-house when per-source semantics are the differentiator (an aggregator's normalization decisions silently become yours, including the ones you disagree with), when you need provenance the aggregator flattens away, or when you are large enough that an unnoticed outage costs more than the platform team does. And even then you still need the freshness check — pointed at the aggregator. Putting a vendor between you and the provider does not remove silent failure; it relocates it and adds one more party who can go quiet. Start at health data aggregator APIs and what a health data aggregator is.

Close

Monitoring a health pipeline is mostly the discipline of writing down what "normal" looks like for each provider separately, then noticing when it stops being that. The instrumentation is the easy half. The honest half is accepting that your thresholds are guesses, retuning them when a provider changes, and deleting the checks nobody has ever acted on.

If you are chasing a specific symptom rather than designing the system that prevents it, wearable data arriving late is the page for that.