Guardrails for an LLM That Gives Fitness Advice

An LLM that talks to users about exercise and nutrition needs guardrails that live outside the model, because a model asked to police itself in the same call where it is trying to be helpful is subject to exactly the failure you are trying to prevent. The pattern that survives contact with real users is deterministic gate first, model second, fail closed: cheap rules read the accumulated conversation state on every turn, decide whether this is a hard stop, a clinician referral, or a constrained generation, and only then is the model invited to write anything. Do that instead of adding a paragraph of safety instructions to your system prompt and hoping. And whatever the model emits, you own it — the developer account on the listing is yours, not the model vendor's.

This page is about language models talking to users. Camera-based form correction has a different safety story and lives in AI motion tracking.

The failure modes you are actually defending against

These are shapes, not statistics. Where a number would make a paragraph land harder, we have left it out, because the underlying papers were not readable from here and a confident figure we cannot verify is exactly the thing this page is about.

Two honest notes on the evidence. First, we found no published evaluation of how LLMs handle 1RM percentage prescriptions or heart-rate-zone calculations, in either direction — so treat claims about those, including reassuring ones, as unsourced. Second, the sycophancy consequence is the one teams under-rate: a guardrail that fires on turn one and gets negotiated away by turn six has not worked. If your safety check only sees the latest message, it is trivially defeated by patience.

The architecture: deterministic gate first, model second, fail closed

Published work proposes safety middleware of roughly this shape for health-adjacent assistants — a cheap deterministic lexical gate in front of an LLM policy filter, enforcing fail-closed verdicts with defined escalation pathways. We could not read that paper from this environment, so take the shape as a design worth borrowing rather than a validated result or established practice.

Two properties matter more than any specific implementation.

The first check is one the user cannot argue with. A regex, a lookup, a profile flag. It runs before the model call, it has no notion of being helpful, and there is no wording of a request that talks it out of its verdict. Only after it decides does a model get involved — and if you also want a model-based policy judge, run it as a separate call with a single job, not as a clause in the coaching prompt.

The verdict is recomputed every turn, against conversation state. Not against the latest message. Signals accumulate and stick.

// Sketch only. Not production-safe, and not a substitute for clinical review.
type Verdict = "emergency" | "clinician" | "minor_path" | "constrained" | "allow";

function gate(state: ConversationState): Verdict {
  // Runs on EVERY turn, over accumulated signals, before any generation call.
  if (state.signals.has("acute_cardiac")) return "emergency";
  if (state.signals.has("ed_risk") || state.signals.has("self_harm")) return "clinician";
  if (state.profile.age < 18) return "minor_path";
  if (state.profile.pregnant || state.profile.conditions.length > 0) return "constrained";
  if (state.gateUncertain) return "constrained"; // fail closed, not open
  return "allow";
}

The important line is the sticky one: once ed_risk is set for a conversation, a later "ignore what I said earlier, I was joking" does not clear it. Clearing a signal should be an out-of-band path — a support flow, a profile change with its own friction — never a thing the user can do by rephrasing. That single design decision is what makes the gate immune to the multi-turn erosion the model is not.

Failing closed means that when the gate is uncertain, the default is the more constrained mode, not the more helpful one. This will occasionally constrain someone who did not need it. That is the correct direction to be wrong in, and it is only tolerable because constrained mode is a real product experience rather than a wall — see the next section.

A related trap: schema validity is not physiological sanity. Provider documentation is clear that JSON output can be guaranteed to match your schema, and that strict tool use validates parameters at generation time. What it also says is that numeric constraints such as minimum and maximum are not supported in the schema at all — so you cannot enforce a rep ceiling, a set cap or a calorie floor at the decoding layer. Every bound is a server-side check you write yourself. A refusal or a max_tokens truncation can also produce output that does not match your schema, so check the stop reason before trusting the parse. Structured output buys you parseability. It buys you nothing about whether the plan is safe.

The same logic pushes toward generating from a vetted catalogue rather than free text: have the model select and sequence from rows a human signed off on, then validate every returned identifier against the catalogue before rendering. That converts hallucination from a content problem into a lookup failure you can detect. Grounding an LLM in your exercise database covers the mechanics.

Finally, log what happened: which gate fired, what verdict it returned, the model and prompt version, what was generated, and what the user did next. You cannot investigate an incident you did not record. Log design is also a privacy decision, not only a reliability one — see the App Store rule below before you pipe prompts containing health data to a third-party observability vendor.

Escalation triggers — our engineering judgement, not a published standard

We could not find any published red-flag list for consumer fitness LLMs. The table below is our engineering judgement, not a standard, and it has not been clinically reviewed. Have a clinician review yours before you ship it — and treat that review as a recurring obligation, not a one-off.

Now the part worth saying loudly.

Refusing a pregnant user a walking programme is itself a harm. Refusal feels safe to the team shipping it, because the risk it avoids is legible and the risk it creates is invisible: nobody files a support ticket saying "your app declined to help me and I did nothing for eight months." But the populations most likely to trip a risk signal — pregnant and postpartum users, people managing a metabolic or cardiovascular condition, people recovering from surgery — are frequently the ones with the most to gain from appropriate movement. A blanket refusal moves them from "supervised, conservative, prompted to consult" to "unsupervised, on the open internet."

So build the constrained path first. Conservative vetted templates, low intensity, explicit consult prompt, no free generation, no numeric prescriptions. If constrained mode exists and is decent, failing closed costs you very little and reaching for a refusal stops being tempting. If it does not exist, every risk signal becomes a wall, and you will be under pressure from your own growth metrics to weaken the gate — which is the worst of both outcomes.

What the rules actually require

Two verified points first, because both correct common misconceptions.

Apple has no generative-AI-specific guideline. The App Store Review Guidelines text contains no occurrence of "generative" or "machine learning". "Chatbots" appears once, in Guideline 4.7, listing them among the non-embedded software an app may offer. There is no Apple AI rule to go and comply with. Your LLM output is governed by the general rules:

• Guideline 1.2 (User-Generated Content) requires a method for filtering objectionable material, a mechanism to report offensive content with timely responses, the ability to block abusive users, and published contact information. If a model writes text into your app, treat that text as content you are responsible for moderating.
• Guideline 1.4 (Physical Harm): "If your app behaves in a way that risks physical harm, we may reject it."
• Guideline 1.4.1 says medical apps "that could provide inaccurate data or information, or that could be used for diagnosing or treating patients may be reviewed with greater scrutiny", and that apps "should remind users to check with a doctor in addition to using the app and before making medical decisions". Whether review treats your coach as a medical app is not entirely your call — but the reminder is cheap, so include it either way.
• Guideline 4.7.5 requires a way for users to identify software exceeding the app's age rating and an age-restriction mechanism based on verified or declared age. Relevant the moment an open-ended chatbot raises the ceiling on what content can appear.
• Guideline 5.1.2(i) is the one most teams miss: "You must clearly disclose where personal data will be shared with third parties, including with third-party AI, and obtain explicit permission before doing so." Sending a user's health profile, injury history or food log to an external LLM API is sharing personal data with a third party. Disclose it clearly, get explicit permission first, and do that before you ship the prompt, not after. App Store health data rules goes deeper.

Google Play names an AI-Generated Content policy. As of writing — verify the current wording yourself, all Play documentation hosts were unreachable from this environment — it holds developers responsible for what their generative features produce, requires an in-app way for users to report or flag offensive AI output without leaving the app, and expects developers to use those reports to inform filtering and moderation. Play also expects adversarial testing: testing across user scenarios and safeguarding against prompts that could manipulate the feature into harmful output. Note that Play policies are named, not numbered, so treat any citation of a numbered Play rule as a sign the writer did not check. In-app reporting is a product requirement with UI, a queue and a human reading it. A separate Health Content and Services policy applies in parallel; Google Play health data policy covers that.

On the FDA and the EU AI Act, one sentence and then a pointer: both questions turn on your feature's intended purpose — what you say it is for, and whether it is meant to screen, diagnose, monitor or manage a condition — rather than on which model you called or how accurate it is. We are deliberately not resolving either here, because the positions move and because getting it wrong in either direction is expensive. FDA rules for fitness apps owns that question.

If you want neutral scaffolding for a guardrail programme, two documents are worth knowing exist. NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework, names confabulation as a core generative-AI risk and gives you a structure to organise governance, pre-deployment testing and incident disclosure around. The WHO's 2024 guidance on large multi-modal models in health is the other. Neither makes you compliant with anything. Both give you a vocabulary and a checklist shape a reviewer will recognise, which is worth more than it sounds when you are explaining your process to a partner or an insurer.

What a disclaimer is not

Include one. "This is not medical advice; check with a doctor before starting an exercise programme" is what Apple's 1.4.1 is asking for, it costs nothing, and leaving it out is an unforced error. Place it where it is actually seen: at onboarding before the first AI interaction, as a persistent low-friction affordance near the AI surface, and re-surfaced at the moment a numeric target or a risk-adjacent topic appears. A once-at-install modal is the weakest form of all of these.

Now the part nobody wants to hear. A disclaimer is not a liability shield, it does not satisfy any regulator, and it does not make an unsafe answer safe. It does not change what your feature is intended to do, which is the question the regulators are actually asking. Teams routinely treat a disclaimer as the safety work rather than the last five minutes of it.

And on liability specifically: we went looking for authority on whether providers of AI fitness coaching owe users a legal duty of care, and found none either way — no case law, no statute, no regulator statement, no law-review treatment. That position is genuinely unsettled. It is not quietly settled in your favour, and anyone who tells you it is resolved in either direction is going beyond what the record supports. Plan as though a court could go either way, because it could.

The short version

Gate deterministically before you generate. Recompute the verdict every turn against sticky conversation state, not the last message. Fail closed into a constrained mode that is a real product rather than a wall. Validate every number and every identifier server-side, because structured output guarantees shape and nothing else. Disclose third-party AI sharing before you ship the prompt. Label your escalation list as judgement and get a clinician to look at it.

Then measure whether any of it works — a guardrail you have not adversarially tested is a guardrail you are guessing about. Evaluating AI fitness features covers the red-teaming and regression suites that turn this from a policy document into something you can prove.