AI Food Logging: Text and Photo Nutrition Entry That Actually Works

AI food logging works when the model resolves input to a row in a vetted food database, and fails when the model generates nutrition numbers. "Two eggs and a slice of rye" or a photo of a plate goes in; a food identity plus a quantity comes out; your server multiplies the catalogue's per-100 g macros by that quantity and stores the result. The model never emits a calorie figure you persist. That one rule removes an entire class of arithmetic error, gives every entry a stable food_id your trends and charts can hang off, and makes each log auditable and editable rather than a black box the user can only delete.

The two input modes are not equally hard. Text logging is mostly an entity-resolution problem and is largely a solved shape. Photo logging is genuinely difficult, for a reason that is documented rather than speculative. This page covers both, plus the parts teams skip: portion uncertainty, correction UX, and mining user edits as an eval set. For the underlying nutrition data layer, see nutrition APIs; for how calorie numbers get produced in the first place, how fitness apps estimate calories.

Resolve, don't generate

Here is the same feature built four ways.

The best-documented shipping example follows row two. MyFitnessPal's Meal Scan and Voice Log are publicly announced camera and natural-language food entry that, as described by the company, match recognised foods to entries in its existing verified food database rather than generating nutrition values. Cal AI is the visible example of the other category — pure photo estimation as the product itself. Both categories ship; only one of them can show a user why an entry says what it says.

This is the same grounding argument that applies to workout generation from an exercise catalogue — the model's job is selection, your code's job is arithmetic — with one difference that shapes everything below: a food corpus is millions of messy user-generated rows rather than a few thousand clean ones, so the retriever is a genuinely harder piece of engineering. Grounding an LLM in a catalogue sets out why corpus shape decides that.

Text logging: an entity-resolution problem wearing an LLM costume

"Two eggs and a slice of rye" needs to become two structured items with a quantity, a unit, and something searchable. That is extraction, not reasoning, and it is a small, mechanical, high-volume call — the right job for a small fast model like claude-haiku-4-5 rather than your most expensive tier.

Constrain the response to a JSON schema so you get schema-valid data instead of prose you have to parse. Note what is not in this schema:

{
  "type": "object",
  "properties": {
    "items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "query":       { "type": "string" },
          "quantity":    { "type": "number" },
          "unit":        { "type": "string",
                           "enum": ["g", "ml", "item", "slice", "cup", "tbsp", "tsp"] },
          "preparation": { "type": "string",
                           "enum": ["raw", "boiled", "fried", "grilled", "baked", "unknown"] },
          "confidence":  { "type": "string", "enum": ["high", "medium", "low"] }
        },
        "required": ["query", "quantity", "unit", "preparation", "confidence"],
        "additionalProperties": false
      }
    }
  },
  "required": ["items"],
  "additionalProperties": false
}

There is no calories field, no protein field, no food_id field. The model cannot hallucinate a nutrition number it was never asked for, and it cannot hallucinate a catalogue ID because it never sees the ID space. Two implementation notes worth knowing before you ship: the schema layer supports enum but not numeric bounds like minimum and maximum, so "quantity must be positive and under some sane ceiling" is a server-side check, not a decoding-time guarantee; and a refusal or a max_tokens truncation can still return something that does not match your schema, so check the stop reason before you trust the parse.

Resolution then happens in your code:

for (const item of parsed.items) {
  // Food catalogues are large, messy and user-generated — this is a
  // legitimate hybrid lexical + vector search, unlike a small exercise table.
  const candidates = await searchFoods(item.query, { limit: 5 });
  if (candidates.length === 0) { draft.push(unresolved(item)); continue; }

  const row   = candidates[0];
  const grams = toGrams(item.quantity, item.unit, row); // uses the row's own serving weights

  draft.push({
    foodId: row.id,
    grams,
    kcal:   row.kcalPer100g * grams / 100,  // computed here, never copied from the model
    alternatives: candidates.slice(1),      // this is what powers one-tap correction
    source: "text",
    edited: false,
  });
}

A few things that decide whether this feels good or bad in practice:

• Retrieval is the real product. Users type brand names, restaurant dishes, and regional names that do not lexically match any row. A curated alias table beats embeddings where you can enumerate the aliases and it stays auditable; hybrid lexical-plus-vector search earns its place for the long tail. Corpus shape decides the retriever.
• Personalise the ranking, not the numbers. The user's own recent logs, favourites, and the meal's time of day are excellent, free re-ranking signals. This is a WHERE clause and an ORDER BY, not a model call.
• Let the model pick from a shortlist when it must pick at all. If you need disambiguation ("milk" — which one?), retrieve candidates first and have the model select from an enumerated list. Selection from a supplied set is a fundamentally more constrainable task than open generation.
• Skip the model entirely where you can. A barcode scan, "same as yesterday", or an exact-match search need no LLM call. The cheapest token is the one you never send.

Photo logging: much harder, for documented reasons

Photo logging is a vision-language model reading a still image, not movement analysis — camera work for exercise form and rep counting is a separate discipline covered under motion tracking. The failure modes here are specific to reading a plate.

The provider's own vision documentation is unusually candid, and it is worth reading it against the food use case rather than trusting a demo. It states that the model can make mistakes on low-quality, rotated, or very small images; that spatial reasoning and localisation are approximate; that counting is approximate and "might not always be precisely accurate, especially with large numbers of small objects"; that image compression artifacts degrade accuracy; and that outputs are not a substitute for professional medical advice or diagnosis.

Read that third point back into food logging and the product problem is obvious. How many almonds, dumplings, meatballs, gnocchi, prawns, or chicken nuggets are on this plate is precisely the small-object-counting case the docs flag as approximate — and for discrete foods, count is the dominant driver of portion size, and portion size is the dominant driver of calories. The error does not stay in the counting layer. It multiplies straight through to the number your user sees.

Everything else stacks on top of that: the rice under the curry is occluded, the oil the vegetables were cooked in is invisible, the dressing is inside the salad, and the sauce could be 40 calories or 400. Human dietitians estimating from photographs report the same obstacles. Published evaluations of photo-based calorie estimation exist and report meaningful error, including a tendency toward under-estimation — we could not verify those figures first-hand, so this page does not quote any of them. Treat systematic under-estimation as a bias to measure in your own data, not as a settled number. It matters commercially as well as clinically: a tracker that quietly flatters the user is a tracker that stops working.

Re-compressing the image on the client is a silent accuracy tax

This is the most common self-inflicted wound in photo logging, and it never shows up in a crash report.

A phone camera already hands you a lossy JPEG. Then the client resizes and re-encodes it for upload, maybe a server generates a thumbnail, maybe an image CDN re-encodes it again. Each lossy pass adds artifacts, and provider documentation is explicit that compression artifacts are detrimental to model performance, especially across multiple compression passes. You have not changed a line of prompt or model code, and your accuracy has quietly dropped.

The fix is to be deliberate about it:

• Downsample once, from the original. Resize the highest-quality frame you have straight to your target dimensions in a single step. Do not chain resize-then-recompress-then-thumbnail.
• Send the model the clean asset, not the display thumbnail your UI happens to already have in memory.
• Prefer resolution reduction over quality reduction. Fewer pixels costs you detail predictably; a low JPEG quality setting costs you detail unpredictably, in exactly the fine texture that distinguishes brown rice from quinoa.
• You control the token cost anyway. Images are processed as 28×28-pixel patches, so an image costs roughly ceil(width / 28) × ceil(height / 28) visual tokens — a 200×200 image is 64 tokens, 1000×1000 is 1296. Uploading a full-resolution camera image when a ~1000 px long edge suffices is pure waste, and the docs recommend downsampling client-side when the extra fidelity is not needed. Choose a target size on purpose and hold it constant so you can actually A/B it.

Pass cheap context alongside the image, in a deliberate order. Start with meal time and the user's own recent logs: those two are free, already in your database, and they resolve the most common ambiguity — which of five "milk" rows, which of three ways this person makes their oats. Add a restaurant geofence and cuisine history only if your correction-rate metric says the first two were not enough, because both cost integration work and neither helps a home-cooked plate. This ordering is engineering judgement rather than a published result, but it is close to free to test.

Portion estimation: ask for a band, not a number

A single confident portion figure is a false-precision problem, not a modelling problem. The honest output shape has a range in it:

"portion": {
  "type": "object",
  "properties": {
    "unit":   { "type": "string", "enum": ["g", "ml", "item"] },
    "low":    { "type": "number" },
    "likely": { "type": "number" },
    "high":   { "type": "number" },
    "anchor": { "type": "string" }
  }
}

anchor is the fiducial the estimate was made against — "relative to the 26 cm plate", "compared to the fork in frame", "hand for scale". Asking for it does two useful things: it gives the user something to argue with, and it gives you a diagnostic when estimates go wrong at scale. Note again that you cannot enforce low <= likely <= high at the schema layer, since numeric constraints are not supported there. Validate it server-side and repair deterministically rather than re-prompting, so a bad generation cannot loop into cost.

Then use the band in the UI instead of hiding it:

• Show it. "About 180 g (140–240 g)" is more truthful than "183 g", and users trust a system that admits a range more than one that is precisely wrong.
• Let width drive behaviour. A wide band is a signal to ask one cheap clarifying question ("bowl or plate?") rather than silently commit a number.
• Never display more precision downstream than the estimate supports. A calorie total rendered to the single kilocalorie, derived from a portion guess with a 100 g spread, is theatre.

Correction is the primary interaction, not an error path

If the realistic accuracy ceiling means the model produces a fast first draft, then editing that draft is the main event and should be designed like one.

• Land in a draft state. The entry appears editable and uncommitted, not saved-then-fixable.
• One tap to swap the food row. You already retrieved the runner-up candidates — show them. Making the user delete the entry and start a fresh search is how you lose the habit.
• One gesture to change portion. A slider bounded by the estimate's own band, plus common household units for that food row, beats a numeric keypad.
• Show provenance on every entry. Which catalogue row, which input mode (photo, text, barcode, manual), whether a human edited it. This is what makes an AI log defensible to the person keeping it.
• Never let the model editorialise on the log. "Great job staying under your calories" turns a logging feature into a coaching feature, with an entirely different risk profile and a different set of things it can get badly wrong.

What to instrument on every logged entry

Designing correction as the primary interaction has a useful side effect: every edit is a label, produced by the one person who actually knows what was on the plate. Evaluating AI fitness features covers how to mine that stream into a golden set. What is specific to food logging is what you record, and it is a short list you have to get right before the first entry is written.

Log, per entry: the proposed food_id and portion, the committed food_id and portion, whether the user edited either, and which candidate rank they selected if they swapped. That last field is the one teams omit and cannot backfill. From it you get:

• Top-1 and top-5 resolution accuracy, graded deterministically against the catalogue — no LLM judge required. If the right row was at rank 3, that is a retrieval bug rather than a model bug, and you fix it in ranking. Without the rank you cannot tell those two apart.
• Signed portion error, not just absolute error. A grader reporting mean absolute error hides a directional bias completely; only the signed mean tells you whether your product is systematically under-counting its users' intake.
• Edit rate segmented by food class — discrete countable items versus amorphous ones, home-cooked versus packaged, photo versus text. This is where the small-object counting weakness shows up in your own numbers rather than in someone else's paper, and it is the segmentation that tells you whether to fix the prompt or restrict the feature.

One caveat: meal photos and food logs are personal data, and retaining them for evaluation is a separate consent question from processing them to answer the user's request. Decide that deliberately.

The honest limits

Nobody estimates calories accurately from a photograph — not a model, not a dietitian. The comparison that matters is not "as good as a professional"; it is "consistent and convenient enough to beat the user logging nothing at all". Say that in your own product copy too. An app that claims photographic precision is setting up the review that eventually arrives.

A food logger is a disordered-eating surface, and that is a stronger claim than it sounds: restriction framing, compensatory-exercise framing and "how few calories can I eat" arrive through the logging flow itself, typed into a search box rather than a chat window. A feature you built as data entry is a risk surface. Screen it with the same deterministic gate you would put in front of a coach, route to a constrained path, and give under-18 users a policy path of their own — LLM safety for fitness advice has the triggers and the tiering.

Nothing here is medical advice, and the model vendor does not own what your app says — you do. A "not medical advice" line is good practice and an app-store expectation, but it is not a liability shield, and it does not make an unsafe answer safe.

Meal photos are health data leaving your servers. Sending them to a third-party model triggers Apple's Guideline 5.1.2(i) disclosure-and-explicit-permission requirement — worth settling before you build the upload path, since it constrains where the image can go; see app store health data rules.

Cost is arithmetic you control, and for this feature it is dominated by two things nothing else on the page touches: image resolution, which you set on the client, and call volume, which is three meals a day rather than one plan a week. Work it from your own measured token counts; what an AI fitness feature costs to run has the image-token formula and the levers in order.

Where to start

Ship text logging first. It is cheaper, it is the easier problem, it exercises the whole resolve-to-a-row pipeline end to end, and it gives you a correction dataset before you take on the harder input mode. Then add photo logging with an explicit uncertainty band, a correction-first UI, and a signed-error metric from day one.

The line that should survive every design review: the model chooses the row and estimates the amount; your database supplies the numbers.