How to Add AI Workout Tracking to a Web App (JavaScript)

This guide walks you through adding camera-based workout tracking to a web app in plain JavaScript: capture the webcam with getUserMedia, run Google's MediaPipe Pose Landmarker on each frame to get 33 body landmarks, compute a joint angle to count a rep, and draw a skeleton overlay on a canvas. There are two paths to camera workout tracking: build on a free, on-device pose primitive (what this guide does with MediaPipe, with MoveNet via TensorFlow.js as the alternative), or buy a commercial coaching SDK that wraps rep counting and form rules for you. We take the build path so you can see exactly what happens end to end.

All the moving parts here run on-device in the browser. Once the model files are cached, no frames leave the user's machine, which is good for privacy and latency.

What you'll need

• A modern browser with WebGL/WebGPU and camera access, served over HTTPS or localhost (browsers block getUserMedia on insecure origins).
• A bundler or plain ES modules. As of 2026 the package is @mediapipe/tasks-vision; install it with npm i @mediapipe/tasks-vision, or import it from a CDN. Pin a specific version in production rather than @latest and check the current release before shipping.
• A .task pose model file (MediaPipe hosts lite, full, and heavy variants). Start with lite for real-time web.
• Basic HTML: one <video> element for the camera feed and one <canvas> stacked over it for the overlay.

The four building blocks are: get camera frames, run the pose model, read keypoints, compute the metric (an angle to drive a rep counter), then surface it in the UI. MediaPipe is the primary approach below; the last step shows the MoveNet alternative.

Step 1: Get the camera into a video element

Use the getUserMedia API to request the webcam and pipe the stream into a <video> element. Ask for a modest resolution — the model downscales internally, so feeding 4K just wastes CPU.

<video id="cam" playsinline></video>
<canvas id="overlay"></canvas>

const video = document.getElementById("cam");

async function startCamera() {
  const stream = await navigator.mediaDevices.getUserMedia({
    video: { width: 640, height: 480, facingMode: "user" },
    audio: false,
  });
  video.srcObject = stream;
  await video.play();
  return video;
}

Serve the page over HTTPS or localhost, otherwise the permission prompt never appears and the call rejects.

Step 2: Create the Pose Landmarker

Load the WASM fileset with FilesetResolver.forVisionTasks, then build the landmarker with PoseLandmarker.createFromOptions. Set runningMode to "VIDEO" for a webcam loop and pick the GPU delegate for speed.

import {
  PoseLandmarker,
  FilesetResolver,
  DrawingUtils,
} from "@mediapipe/tasks-vision";

async function createLandmarker() {
  // As of 2026, pin the version instead of @latest in production.
  const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
  );

  return PoseLandmarker.createFromOptions(vision, {
    baseOptions: {
      modelAssetPath:
        "https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/latest/pose_landmarker_lite.task",
      delegate: "GPU", // or "CPU"
    },
    runningMode: "VIDEO",
    numPoses: 1,
  });
}

This is async and can take a moment on first load while the WASM and .task files download; they're cached afterward.

Step 3: Run detection on every frame

Drive a render loop with requestAnimationFrame. In "VIDEO" mode you call detectForVideo(video, timestampMs) with a monotonically increasing timestamp — performance.now() is exactly that.

const canvas = document.getElementById("overlay");
const ctx = canvas.getContext("2d");

let poseLandmarker;

function renderLoop() {
  const startTimeMs = performance.now();
  const result = poseLandmarker.detectForVideo(video, startTimeMs);

  // result.landmarks is an array — one entry per detected pose.
  if (result.landmarks.length > 0) {
    const landmarks = result.landmarks[0]; // 33 normalized landmarks
    processPose(landmarks); // Steps 4-5
  }

  requestAnimationFrame(renderLoop);
}

// Kick things off:
(async () => {
  await startCamera();
  poseLandmarker = await createLandmarker();
  canvas.width = video.videoWidth;
  canvas.height = video.videoHeight;
  requestAnimationFrame(renderLoop);
})();

Each frame returns a PoseLandmarkerResult. Its landmarks array holds one set of 33 landmarks per detected person; with numPoses: 1 you read index 0.

Step 4: Read keypoints and compute a joint angle

Each of the 33 landmarks has x and y normalized to the range 0 to 1 (fractions of frame width and height), plus z and visibility. The BlazePose indices you care about for most exercises are shoulders 11 and 12, elbows 13 and 14, wrists 15 and 16, hips 23 and 24, knees 25 and 26, and ankles 27 and 28.

To count a rep you need a scalar signal. A joint angle is the classic choice. Compute the interior angle at a vertex B between points A and C using the atan2 of the cross and dot products — it is numerically robust and returns 0 to 180 degrees, unlike bare acos, which can return NaN near the extremes.

// Interior angle at vertex B, for keypoints A-B-C. Returns 0..180 degrees.
function angle(A, B, C) {
  const bax = A.x - B.x, bay = A.y - B.y; // vector B->A
  const bcx = C.x - B.x, bcy = C.y - B.y; // vector B->C
  const dot = bax * bcx + bay * bcy;
  const cross = bax * bcy - bay * bcx;
  return (Math.atan2(Math.abs(cross), dot) * 180) / Math.PI;
}

// Squat example: knee angle from hip (23) -> knee (25) -> ankle (27).
function processPose(lm) {
  const kneeAngle = angle(lm[23], lm[25], lm[27]);
  updateReps(kneeAngle, lm);
  drawOverlay(lm);
}

The angle is scale-invariant, so working in normalized coordinates is fine — just keep x and y in the same units. For a squat, standing is roughly 170 to 180 degrees and a parallel squat is about 90 degrees. A bicep curl uses the elbow: angle(shoulder, elbow, wrist), extended near 160 to 180 and fully flexed near 30 to 45.

Step 5: Count reps with a hysteresis state machine

Rep counting is not machine learning — it is a threshold state machine over that angle signal. Use two thresholds, not one, so noise around a single line cannot fire phantom counts. The angle must travel all the way down past an ENTER threshold and back up past an EXIT threshold before you count once, on the completed round trip. This dead-band is called hysteresis, and it is exactly what Google's own fitness samples do.

const ENTER_DOWN = 100; // must drop below this to reach "down"
const EXIT_UP = 160;    // must rise above this to reach "up"
const MIN_VIS = 0.6;    // per-landmark confidence gate
const ALPHA = 0.3;      // EMA smoothing factor

let state = "up";
let reps = 0;
let angleSmoothed = null;

function updateReps(rawAngle, lm) {
  // 1) Confidence gate: skip frames where the driving joints are unreliable.
  const vis = Math.min(
    lm[23].visibility, lm[25].visibility, lm[27].visibility
  );
  if (vis < MIN_VIS) return;

  // 2) Smooth the angle with an EMA to kill single-frame jitter.
  angleSmoothed =
    angleSmoothed == null
      ? rawAngle
      : ALPHA * rawAngle + (1 - ALPHA) * angleSmoothed;

  // 3) Hysteresis: count on the down -> up completion only.
  if (state === "up" && angleSmoothed < ENTER_DOWN) {
    state = "down";
  } else if (state === "down" && angleSmoothed > EXIT_UP) {
    state = "up";
    reps += 1;
    document.getElementById("repCount").textContent = reps;
  }
}

Three things make this reliable: the confidence gate (occluded landmarks jump to garbage positions and can trip a threshold, so skip those frames and hold state), the EMA smoothing on the scalar angle (keep alpha modest, roughly 0.2 to 0.4, so you damp jitter without lagging fast reps), and the two-threshold hysteresis itself. Count on one edge only — counting on both double-counts.

Step 6: Draw the skeleton overlay

MediaPipe ships a DrawingUtils helper that draws landmarks and connectors straight onto a canvas 2D context. Clear the canvas each frame first.

const drawingUtils = new DrawingUtils(ctx);

function drawOverlay(lm) {
  ctx.clearRect(0, 0, canvas.width, canvas.height);
  drawingUtils.drawConnectors(lm, PoseLandmarker.POSE_CONNECTIONS, {
    color: "#00e5ff",
    lineWidth: 3,
  });
  drawingUtils.drawLandmarks(lm, { color: "#ffffff", radius: 3 });
}

Position the <canvas> directly over the <video> with CSS (same size, absolute positioning) so the skeleton lines up with the person. DrawingUtils maps the normalized 0-to-1 coordinates back to canvas pixels for you.

The MoveNet alternative (TensorFlow.js)

If you would rather use TensorFlow.js, MoveNet through @tensorflow-models/pose-detection is the common alternative. It returns 17 COCO keypoints (fewer than MediaPipe's 33, no face/hand/foot detail) as {x, y, score, name} objects, already in pixel space. The rep-counting math from steps 4 and 5 is identical — you just read different indices.

import * as poseDetection from "@tensorflow-models/pose-detection";
import "@tensorflow/tfjs-backend-webgl";

const detector = await poseDetection.createDetector(
  poseDetection.SupportedModels.MoveNet,
  { modelType: poseDetection.movenet.modelType.SINGLEPOSE_LIGHTNING }
);

const poses = await detector.estimatePoses(video);
for (const kp of poses[0].keypoints) {
  const { x, y, score, name } = kp; // e.g. name === "left_knee"
}

MoveNet has two single-person variants: SINGLEPOSE_LIGHTNING (fastest, default) and SINGLEPOSE_THUNDER (more accurate, slower). Both are Apache-2.0 and vendor-stated to run in real time, above 30 FPS, on modern hardware — treat that as a documented vendor claim, not an independent benchmark. Package versions move; check the current release before you pin one.

Make it reliable

Camera and framing matter more than model choice for accuracy. A few practical rules:

• 2D angles are viewpoint-dependent. A 2D joint angle only matches the true angle when the limb is roughly parallel to the image plane. Use a side view for sagittal exercises like squats, curls, and lunges; use a front view for symmetry or knee-cave checks.
• Gate on confidence, average both sides. Reject landmarks below roughly 0.5 to 0.7 visibility before measuring, and average the left and right joint when both are visible to cut noise.
• Pick the model tier for the job. MediaPipe lite and MoveNet Lightning are latency-first; heavy/Thunder are accuracy-first for form grading. Enable the GPU delegate and downscale frames — do not feed full-resolution video.

If your end user is exercising in front of the camera, tell them to clear some space, keep their whole body in frame from head to ankles, and use even, front-facing lighting. Backlighting and cropped joints are the two fastest ways to wreck landmark quality.

MediaPipe Pose Landmarker and MoveNet are both Apache-2.0 and free. Getting keypoints is the solved, commoditized part; the rep logic, form rules, and per-exercise tuning on top are the real, recurring work. If you would rather not maintain that yourself, a commercial fitness SDK wraps these same steps behind its own API — see its docs for the exact calls.