Camera Pose Tracking for Fitness Apps: A Practical Guide

Camera pose tracking turns a plain video feed into a stream of body joint coordinates, and this guide builds that pipeline end to end on the web: load a pose model, pull frames from the camera, run detection per frame, and read the landmarks you need. You have two paths — build on a free, on-device pose primitive (MediaPipe or MoveNet) or buy a fitness SDK that wraps the whole thing — and this guide takes the build path with MediaPipe Pose Landmarker as the primary example.

What pose estimation actually gives you

Pose estimation is a model that looks at an image of a person and returns the pixel positions of their body joints — shoulders, elbows, hips, knees, and so on — as a set of numbered points called landmarks or keypoints. It does not count reps, judge form, or know what exercise is happening. It gives you coordinates, one set per video frame, and everything else is logic you build on top.

The mental model is a three-stage pipeline:

• Capture — get frames from the camera into a video element (or a native camera surface).
• Estimate — feed each frame to the pose model and get back landmarks.
• Interpret — read specific landmarks and turn them into something useful (a joint angle, a rep count, a form cue).

This guide covers capture and estimate, and shows you how to read the landmarks so the interpret stage is yours to build. It uses the web (JavaScript) stack because it is the fastest way to see keypoints on screen; the same concepts and class names carry to the Android and iOS builds of the same libraries.

Choosing a model: MediaPipe Pose Landmarker vs MoveNet

Two free, permissively licensed (Apache-2.0), on-device models dominate this space.

The practical difference is the 33 vs 17 landmark count. MoveNet's 17 COCO points cover the major joints you need for most exercises — shoulders, elbows, wrists, hips, knees, ankles — and nothing else. MediaPipe's 33 points are a superset: the same major joints plus face detail, hands (pinky, index, thumb), and feet (heel, foot index), plus a per-landmark visibility score and a z depth estimate. For most rep-counting and joint-angle work the shared major joints are all you touch, so either model works. Reach for MediaPipe's 33 points when you need feet or hands (balance work, some yoga), depth, or the metric worldLandmarks; reach for MoveNet when raw speed and a tiny footprint matter most.

The rest of this guide uses MediaPipe Pose Landmarker as the primary example, with the MoveNet equivalent shown as an alternative in Step 2.

Which landmark indices matter for exercises

Landmarks come back as a flat array; you address the joints you care about by index. These are the exercise-relevant indices for each model.

MediaPipe (33-point BlazePose topology):

MoveNet (17-point COCO topology):

Note the indices differ between the two models, so do not hardcode one set for the other. Also note both use a "left/right" convention from the subject's anatomical perspective, which is mirrored relative to what you see on screen. With MoveNet you can also address points by the name field on each keypoint (for example 'left_knee'), which sidesteps index confusion.

What you'll need

• A modern browser with a webcam and camera permission.
• A bundler-based web project (or any setup that can import from npm).
• As of 2026, the npm packages below — check the current package versions before you ship:
  • MediaPipe path: @mediapipe/tasks-vision
  • MoveNet path: @tensorflow-models/pose-detection, @tensorflow/tfjs-core, and a backend such as @tensorflow/tfjs-backend-webgl

Step 1: Install and load the pose model

Install the package and create the landmarker. Loading has two parts: a WASM fileset that runs the model, and the .task model file itself. MediaPipe ships lite, full, and heavy variants — start with lite for real-time web, and move up if you need more accuracy and the device can sustain the frame rate.

// npm i @mediapipe/tasks-vision   (as of 2026 — check the current version)
import { PoseLandmarker, FilesetResolver, DrawingUtils } from "@mediapipe/tasks-vision";

// Load the WASM fileset (served from the npm package or a CDN mirror).
const vision = await FilesetResolver.forVisionTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);

// Create the landmarker. VIDEO mode is the common webcam pattern.
const poseLandmarker = await PoseLandmarker.createFromOptions(vision, {
  baseOptions: {
    modelAssetPath:
      "https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/latest/pose_landmarker_lite.task",
    delegate: "GPU"        // or "CPU"
  },
  runningMode: "VIDEO",     // "IMAGE" | "VIDEO" | "LIVE_STREAM"
  numPoses: 1
});

The model runs entirely on-device: once the WASM fileset and .task file are loaded and cached, detection needs no network round-trip and no image data leaves the browser.

Step 2: Get the camera feed

Use the browser's getUserMedia to stream the webcam into a hidden or visible video element. The pose model reads frames directly from that element.

const video = document.getElementById("webcam"); // a <video> element

const stream = await navigator.mediaDevices.getUserMedia({
  video: { width: 640, height: 480 },
  audio: false
});
video.srcObject = stream;
await video.play();

If you prefer MoveNet, the capture step is identical — only the model setup changes. The MoveNet equivalent of Step 1 looks like this:

// npm i @tensorflow-models/pose-detection @tensorflow/tfjs-core @tensorflow/tfjs-backend-webgl
import * as poseDetection from "@tensorflow-models/pose-detection";
import * as tf from "@tensorflow/tfjs-core";
import "@tensorflow/tfjs-backend-webgl";

const detector = await poseDetection.createDetector(
  poseDetection.SupportedModels.MoveNet,
  { modelType: poseDetection.movenet.modelType.SINGLEPOSE_LIGHTNING }
);

Step 3: Run detection on each frame

Drive a render loop with requestAnimationFrame and call detectForVideo on every frame, passing a monotonically increasing timestamp in milliseconds. MediaPipe uses that timestamp to track motion across frames.

function renderLoop() {
  const startTimeMs = performance.now();
  const result = poseLandmarker.detectForVideo(video, startTimeMs);

  handleResult(result);            // read landmarks — see Step 4

  window.requestAnimationFrame(renderLoop);
}
renderLoop();

The MoveNet equivalent is const poses = await detector.estimatePoses(video); inside the same kind of loop — it accepts the video element directly and returns an array of poses.

Step 4: Read the landmarks

detectForVideo returns a PoseLandmarkerResult. Its landmarks field is an array with one entry per detected pose; each entry is the 33-point array. Each landmark has x and y normalized to the range 0.0 to 1.0 (fractions of image width and height), a z depth, and a visibility score from 0.0 to 1.0. Address the joints you need by index.

function handleResult(result) {
  for (const landmarks of result.landmarks) {   // one array per detected pose
    const leftShoulder = landmarks[11];          // {x, y, z, visibility}
    const leftElbow    = landmarks[13];
    const leftWrist    = landmarks[15];
    const leftHip      = landmarks[23];
    const leftKnee     = landmarks[25];
    const leftAnkle    = landmarks[27];

    // Multiply x by video width and y by height to get pixel coordinates.
    // result.worldLandmarks holds the same points in metric (meters) coords.
  }
}

With MoveNet you read poses[0].keypoints, where each keypoint is { x, y, score, name } already in pixel space (for example poses[0].keypoints[13] or the point whose name is 'left_knee').

From here the interpret stage is yours: feed three landmarks (for example shoulder, elbow, wrist) into a joint-angle formula, then run a threshold state machine over that angle to count reps. That logic is a separate topic; this pipeline is what produces the coordinates it consumes.

Step 5 (optional): Draw an overlay

To see that it works, draw the skeleton on a canvas layered over the video. MediaPipe ships DrawingUtils with helpers for landmarks and connectors.

const canvas = document.getElementById("overlay");
const ctx = canvas.getContext("2d");
const drawingUtils = new DrawingUtils(ctx);

function drawPose(result) {
  ctx.clearRect(0, 0, canvas.width, canvas.height);
  for (const landmarks of result.landmarks) {
    drawingUtils.drawLandmarks(landmarks);
    drawingUtils.drawConnectors(landmarks, PoseLandmarker.POSE_CONNECTIONS);
  }
}

Make it reliable

A few things separate a demo from something that holds up in real use:

• Gate on confidence. Occluded or low-confidence landmarks jump to garbage positions. Before you use a joint, check its visibility (MediaPipe) or score (MoveNet) and skip or hold the last good value when it falls below roughly 0.5 to 0.7. Tune per camera.
• Smooth before you measure. MediaPipe already applies a One-Euro filter to landmark coordinates internally, but any downstream scalar you compute (like a joint angle) still benefits from a light exponential moving average to kill jitter.
• Downscale, do not oversize. The model resizes frames to a small square internally (MoveNet Lightning ingests 192x192), so feeding a 4K frame wastes compute. A 640x480 capture is plenty. Enable the GPU delegate for the biggest speedup.
• Keep the detector warm. Both models track the person frame to frame; do not recreate the detector each frame.

If your end user is exercising in front of the camera, give them one setup instruction: stand back far enough that the whole body is in frame (head to ankles), keep the space clear, and use even, front-facing lighting. Cropped joints and backlit silhouettes are the two biggest causes of bad landmarks. A side view works best for squats, curls, and lunges; a front view is better for left/right symmetry checks.

Build vs buy, one more time

Getting keypoints on screen is the easy, commoditized part — it is a free, solved problem with the models above. The recurring engineering cost is everything after: reliable rep counting, per-exercise form rules, and accuracy across body types, camera angles, lighting, and low-end devices. If that interpret layer is your core product, build it on these primitives. If it is not, a fitness SDK wraps this whole pipeline plus the rep and form logic behind its own API — see its docs for the exact calls.