How to Add AI Workout Tracking to a Flutter App

AI workout tracking means watching a user exercise through the camera and turning their body movement into signals your app can act on: reps, joint angles, and form cues. In Flutter you have two paths. The build path wires the camera plugin to Google's google_mlkit_pose_detection (a free, on-device pose model) and adds your own rep logic. The buy path drops in a commercial fitness SDK that bundles camera, pose, and rep-counting for you. This guide walks the build path end to end, then points at the buy path.

What you'll need

• A Flutter app targeting a real device (pose detection needs a live camera; simulators won't cut it).
• The camera plugin (Flutter team) for frame capture.
• google_mlkit_pose_detection for on-device pose estimation. As of 2026 the latest is 0.15.0 — check the current package on pub.dev before pinning.
• Platform minimums for ML Kit pose (as of 2026, verify against the package README): iOS deployment target 15.5 with Xcode 15.3 or newer, Android minSdk 21.
• Camera permission strings in Info.plist (iOS) and AndroidManifest.xml.

The pipeline is: capture frames, run the pose model, read keypoints, compute a joint angle, count a rep, show it. Six steps.

Step 1: Add the packages and request camera permission

Add both plugins to pubspec.yaml, then add the platform permission entries. On iOS add an NSCameraUsageDescription string to Info.plist; on Android add the android.permission.CAMERA permission to your manifest. Package versions move fast, so treat the versions below as a snapshot and confirm the current ones.

# pubspec.yaml (versions as of 2026 — check pub.dev for current)
dependencies:
  camera: ^0.11.0            # verify current version
  google_mlkit_pose_detection: ^0.15.0

Step 2: Stream camera frames

Create a CameraController, initialize it, then start an image stream. startImageStream hands you a CameraImage for every frame — that is your inference feed. Keep the work per frame light so the stream doesn't back up.

import 'package:camera/camera.dart';

final cameras = await availableCameras();
final controller = CameraController(
  cameras.first,
  ResolutionPreset.medium,   // medium keeps latency down; the model downsizes anyway
  enableAudio: false,
);
await controller.initialize();

await controller.startImageStream((CameraImage image) {
  // one callback per frame — convert and detect here (Steps 3-4)
});

Step 3: Run the pose detector

Build a PoseDetector once (not per frame), then call processImage on an InputImage built from each CameraImage. processImage returns a List<Pose> — usually one entry, since ML Kit pose tracks a single person. Close the detector when you tear the screen down.

The CameraImage-to-InputImage conversion (assembling planes, rotation, and format) is platform-specific boilerplate — copy it from the package's example app rather than hand-rolling it.

import 'package:google_mlkit_pose_detection/google_mlkit_pose_detection.dart';

// create ONCE, reuse across frames
final poseDetector = PoseDetector(options: PoseDetectorOptions());

Future<List<Pose>> detect(CameraImage image) async {
  final inputImage = _inputImageFromCameraImage(image); // from the example app
  return poseDetector.processImage(inputImage);
}

// on dispose:
poseDetector.close();

Step 4: Read the 33 landmarks

Each Pose exposes a landmarks map keyed by PoseLandmarkType. ML Kit's BlazePose-based model emits 33 body landmarks. Every PoseLandmark carries a type, an x and y position, and a likelihood (confidence) you'll gate on later.

for (final pose in poses) {
  // one specific landmark:
  final PoseLandmark? nose = pose.landmarks[PoseLandmarkType.nose];

  // or walk them all:
  pose.landmarks.forEach((PoseLandmarkType type, PoseLandmark landmark) {
    final double x = landmark.x;
    final double y = landmark.y;
    final double conf = landmark.likelihood;
  });
}

The three landmarks you need for a squat are the hip, knee, and ankle. The enum members for those (for example PoseLandmarkType.leftHip, PoseLandmarkType.leftKnee, PoseLandmarkType.leftAnkle) follow ML Kit's standard naming — verify the exact member names against the installed package's PoseLandmarkType enum.

Step 5: Compute a joint angle from three landmarks

A rep is driven by one scalar: the angle at a joint. For the knee angle, the vertex is the knee (B) and the two arms go to the hip (A) and ankle (C). Use the atan2 of the cross and dot products — it is numerically robust and stays valid at 0 and 180 degrees, where the plain acos form can throw or return NaN.

import 'dart:math' as math;

/// Interior angle at vertex B (0..180 degrees) for points A-B-C.
double jointAngle(PoseLandmark a, PoseLandmark b, PoseLandmark c) {
  final bax = a.x - b.x, bay = a.y - b.y; // vector B -> A
  final bcx = c.x - b.x, bcy = c.y - b.y; // vector B -> C
  final dot   = bax * bcx + bay * bcy;
  final cross = bax * bcy - bay * bcx;    // z of the 2D cross product
  final rad   = math.atan2(cross.abs(), dot);
  return rad * 180 / math.pi;             // 0..180
}

// Squat: standing knee angle is about 170-180 degrees; parallel is about 90.

The angle is scale-invariant, so pixel or normalized coordinates both work as long as x and y are in the same units. It is viewpoint-dependent, though: a 2D angle only matches the real joint angle when the limb is roughly parallel to the camera plane, so film squats and curls from the side.

Step 6: Count a rep with a hysteresis state machine

Rep counting is not machine learning — it is a two-state toggle over the angle with two thresholds (a dead-band), so noise near a single line can't produce phantom counts. Gate out low-confidence frames, smooth the angle with a light EMA, then count only on the full down-to-up round trip.

const double enterDown = 100;     // knee angle must drop below this = "down"
const double exitUp    = 160;     // then rise above this = "up"
const double minConf   = 0.6;     // per-landmark confidence gate
const double alpha      = 0.3;    // EMA factor (0.2-0.4: smooth but don't lag)

String state = 'up';
int reps = 0;
double? angleSmoothed;

void onPose(Pose pose) {
  final hip   = pose.landmarks[PoseLandmarkType.leftHip];
  final knee  = pose.landmarks[PoseLandmarkType.leftKnee];
  final ankle = pose.landmarks[PoseLandmarkType.leftAnkle];
  if (hip == null || knee == null || ankle == null) return;

  // 1) confidence gate — never let a garbage landmark drive the count
  final conf = [hip.likelihood, knee.likelihood, ankle.likelihood].reduce(math.min);
  if (conf < minConf) return;

  // 2) EMA smoothing on the scalar angle
  final raw = jointAngle(hip, knee, ankle);
  angleSmoothed = angleSmoothed == null ? raw : alpha * raw + (1 - alpha) * angleSmoothed!;
  final a = angleSmoothed!;

  // 3) hysteresis state machine — count on the down -> up completion only
  if (state == 'up' && a < enterDown) {
    state = 'down';                 // reached the bottom; don't count yet
  } else if (state == 'down' && a > exitUp) {
    state = 'up';
    reps += 1;                      // full rep completed — update your UI here
  }
}

Surface reps however your UI works — a setState on a counter widget, a ValueNotifier, or a stream feeding your rep display.

Make it reliable

• Two thresholds, not one. A single threshold makes a noisy angle chatter across the line and fire many counts per rep. The gap between enterDown and exitUp is the hysteresis band that debounces it.
• Gate on confidence. Occluded or low-likelihood landmarks jump to garbage positions and can trip a threshold. Skip those frames and hold state rather than counting them.
• Smooth, but not too much. Keep the EMA alpha around 0.2 to 0.4 so you kill jitter without lagging so far you miss fast reps.
• Count one edge only. Counting on both the down and up transitions double-counts every rep.

The buy path

If you'd rather not maintain the camera wiring, the pose model, and the rep and form logic yourself, commercial fitness SDKs (for example KinesteX, Kemtai, QuickPose) package the whole camera-to-rep-counting-to-form-feedback pipeline as a drop-in. They handle the camera lifecycle, pose inference and smoothing, exercise and rep classification, and often a prebuilt overlay UI. Their exact class names and method signatures vary by vendor — refer to each SDK's own Flutter or native documentation for the API.

One note for the end user

Because a person exercises in front of the camera, the input quality caps everything downstream. Ask users to keep their whole body in frame head to ankles, stand where there's even, front-facing light, and clear enough space to move. For squats and curls, a side view gives the truest joint angles.