How to Add AI Workout Tracking to a React Native App

If you want to count reps and check form from the phone camera inside a React Native app, the free-primitive path is react-native-vision-camera for the camera feed, a per-frame pose model running in a frame processor (a worklet on the camera thread), and a small amount of your own math to turn keypoints into a joint angle and a rep. The buy path is a commercial fitness SDK that wraps all of that as a drop-in. This guide walks the build path end to end and points you to the SDK path at the end.

One heads-up before you start: VisionCamera's frame-processor API has changed across major versions (v3, v4, v5), including the hook name and the Camera prop you pass the processor to. The camera and permission hooks below are stable, but for the frame-processor step you must confirm the exact API against the docs for the version you actually installed. This guide teaches the concept and shows a representative shape rather than pinning names that may not match your version.

What you'll build

A camera screen that:

1. Streams frames from the back camera.
2. Runs a pose model on each frame and returns body keypoints.
3. Computes a joint angle (elbow for a bicep curl, knee for a squat).
4. Counts a rep with a two-threshold state machine.
5. Shows the live count.

What you'll need

• A React Native app (bare or a config-plugin-friendly setup; frame processors need native modules, so Expo Go alone will not work — use a development build).
• react-native-vision-camera for the camera and frame processors. As of 2026 this is on v5; check the current package and match your code to your installed major version.
• A per-frame pose model. Two real options, covered in Step 2:
  • react-native-fast-tflite running a MoveNet or BlazePose .tflite model inside the worklet, or
  • a native ML Kit frame-processor plugin (for example the community react-native-vision-camera-v3-pose-detection) that runs Google ML Kit pose detection natively.
• react-native-worklets-core (the worklet runtime that frame processors build on).

Step 1: Set up the camera and permissions

Start with the parts of VisionCamera that have stayed stable across versions: useCameraPermission and useCameraDevice. Request permission, grab the back device, and render the Camera.

import { Camera, useCameraDevice, useCameraPermission } from 'react-native-vision-camera'
import { useEffect } from 'react'
import { StyleSheet } from 'react-native'

function WorkoutCamera() {
  const { hasPermission, requestPermission } = useCameraPermission()
  const device = useCameraDevice('back')

  useEffect(() => {
    if (!hasPermission) requestPermission()
  }, [hasPermission])

  if (device == null) return null
  return (
    <Camera
      style={StyleSheet.absoluteFill}
      device={device}
      isActive={true}
    />
  )
}

Use the rear camera when the user can prop the phone up — it has better optics and gives cleaner landmarks than the front camera. Add the camera usage-description entries your platform requires before shipping.

Step 2: Run a pose model per frame with a frame processor

VisionCamera lets you run a JavaScript worklet synchronously on the camera thread for every frame. Inside that worklet you call a pose model and get keypoints back. This is the concept to hold onto; the exact API is version-specific.

Important: the hook and the Camera prop have different names across versions. In v3/v4 the pattern was a useFrameProcessor hook whose returned worklet you passed to a frameProcessor prop; newer versions renamed the hook and switched to an outputs-style prop. Do not copy a hook name from a random tutorial — open the frame-processor page in the docs for your installed version and use the names it shows. What is constant across versions:

• The callback body must start with the 'worklet' directive.
• It runs on a parallel camera thread, once per frame.
• You must release each frame when done (for example frame.dispose()) or the pipeline stalls.

A representative shape (adjust the hook name and the Camera prop to your version):

// PSEUDO-CANONICAL — confirm the hook name and Camera prop against
// the frame-processor docs for YOUR installed VisionCamera version.
import { Camera, useCameraDevice } from 'react-native-vision-camera'
import { StyleSheet } from 'react-native'

function WorkoutCamera() {
  const device = useCameraDevice('back')

  // In v3/v4 this was useFrameProcessor(...) -> passed to frameProcessor={...}.
  // Newer versions renamed the hook and use an outputs-style prop.
  const frameProcessor = /* useFrameProcessorHookForYourVersion */ ((frame) => {
    'worklet'
    const keypoints = detectPose(frame) // your TFLite call or native plugin
    // hand keypoints back to JS (see Step 3)
  })

  if (device == null) return null
  return (
    <Camera
      style={StyleSheet.absoluteFill}
      device={device}
      isActive={true}
      pixelFormat="rgb"           // TF/TFLite models expect RGB; frames are YUV by default
      // Attach the processor using the prop your version documents
      // (frameProcessor={frameProcessor} in v3/v4, outputs-based in newer versions).
    />
  )
}

You have two real ways to fill in detectPose(frame):

• TFLite in the worklet. Load a MoveNet or BlazePose .tflite model with react-native-fast-tflite and run inference inside the worklet. Set pixelFormat="rgb" on the Camera because TensorFlow models expect RGB and VisionCamera delivers YUV by default. MoveNet Lightning takes a 192x192 input and returns 17 keypoints; Thunder takes 256x256 and is more accurate but slower.
• A native ML Kit frame-processor plugin. A community plugin such as react-native-vision-camera-v3-pose-detection wraps Google ML Kit's pose detector natively and exposes it as a frame-processor function you call from the worklet. ML Kit returns 33 BlazePose landmarks. This avoids shipping and wiring a raw .tflite yourself.

Pick one. TFLite gives you control over the exact model; the native plugin is less setup but ties you to that plugin's maintenance and version support.

Step 3: Read the keypoints

Whichever model you use, you get a list of keypoints, each with an x, an y, and a confidence score. MoveNet gives 17 (COCO ordering: nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles); ML Kit and BlazePose give 33. Map the ones you need into named points so the angle math reads cleanly.

type KP = { x: number; y: number; score: number }

// MoveNet COCO indices (17 keypoints)
function readMoveNet(keypoints: KP[]) {
  return {
    leftShoulder: keypoints[5],
    leftElbow: keypoints[7],
    leftWrist: keypoints[9],
    leftHip: keypoints[11],
    leftKnee: keypoints[13],
    leftAnkle: keypoints[15],
  }
}

Keypoints computed inside a worklet have to be handed back to the JS thread before you use them in React state — use your worklet runtime's shared-value or runOnJS mechanism for that hop. Keep the per-frame heavy work (inference, angle math) on the camera thread and send only the small result across.

Step 4: Compute the joint angle

A rep is driven by one joint angle: the elbow for a curl, the knee for a squat. The angle at vertex B for three points A, B, C is the interior angle between the vectors B-to-A and B-to-C. Use the atan2 of the cross and dot products — it is numerically robust and stays valid at the extremes, unlike a bare acos, which can return NaN near 0 and 180 degrees from floating-point rounding.

function angle(A, B, C) {
  const bax = A.x - B.x, bay = A.y - B.y   // vector B -> A
  const bcx = C.x - B.x, bcy = C.y - B.y   // vector B -> C
  const dot   = bax * bcx + bay * bcy
  const cross = bax * bcy - bay * bcx
  const rad   = Math.atan2(Math.abs(cross), dot)
  return rad * 180 / Math.PI               // 0..180 degrees
}

// Bicep curl: elbow at the vertex.
const elbow = angle(kp.leftShoulder, kp.leftElbow, kp.leftWrist)
// Extended arm is roughly 160-180 deg; fully flexed is roughly 30-45 deg.

// Squat: knee at the vertex.
const knee = angle(kp.leftHip, kp.leftKnee, kp.leftAnkle)
// Standing is roughly 170-180 deg; parallel is roughly 90 deg.

The angle is scale-invariant, so it does not matter whether x and y are in pixels or normalized units — just keep both axes in the same units. It is viewpoint-dependent, though: a 2D angle only matches the true joint angle when the limb is roughly parallel to the image plane, so film sagittal-plane moves (squat, curl, lunge) from the side.

Step 5: Count reps with a two-threshold state machine

Do not count on a single threshold — sensor noise makes the angle chatter across one line and fires many phantom counts. Use two thresholds (hysteresis): the signal has to travel all the way to one extreme and back before a rep counts. Gate on confidence first, then smooth, then run the state machine.

const ENTER_FLEX = 50    // must go BELOW this to reach the "flexed" bottom (curl)
const EXIT_EXT    = 150   // must go ABOVE this to return to "extended"
const MIN_SCORE   = 0.6   // per-keypoint confidence gate
const ALPHA       = 0.3   // EMA smoothing factor (0.2-0.4)

let state = 'extended'
let reps = 0
let angleS = null

function onFrame(kp) {
  // 1) Confidence gate: skip frames where the driving joints are unreliable.
  if (Math.min(kp.leftShoulder.score, kp.leftElbow.score, kp.leftWrist.score) < MIN_SCORE) return

  // 2) Smooth the scalar angle with an EMA to kill jitter.
  const raw = angle(kp.leftShoulder, kp.leftElbow, kp.leftWrist)
  angleS = angleS == null ? raw : ALPHA * raw + (1 - ALPHA) * angleS

  // 3) Hysteresis state machine — count on the full round trip only.
  if (state === 'extended' && angleS < ENTER_FLEX) {
    state = 'flexed'                 // reached the top of the curl; do NOT count yet
  } else if (state === 'flexed' && angleS > EXIT_EXT) {
    state = 'extended'
    reps += 1                        // count on the flexed -> extended completion
  }
}

Count on one edge only (here the return to extended), or you double-count. If a driving joint drops below the confidence gate, skip the frame and hold state rather than feeding a garbage angle into the machine.

Step 6: Surface the count in the UI

Push the rep count and current phase into React state and render it over the camera. Keep the update coming from the JS thread (the worklet hands the result across; do not call setState from inside the worklet).

import { useState } from 'react'
import { View, Text, StyleSheet } from 'react-native'

function RepOverlay({ reps, phase }: { reps: number; phase: string }) {
  return (
    <View style={styles.overlay} pointerEvents="none">
      <Text style={styles.count}>{reps}</Text>
      <Text style={styles.phase}>{phase}</Text>
    </View>
  )
}

const styles = StyleSheet.create({
  overlay: { position: 'absolute', top: 48, left: 0, right: 0, alignItems: 'center' },
  count: { fontSize: 72, fontWeight: '700', color: 'white' },
  phase: { fontSize: 18, color: 'white' },
})

Make it reliable

A few things separate a demo from something that counts correctly in a real living room:

• Keep smoothing and hysteresis. The confidence gate, the EMA on the angle, and the two-threshold dead-band are what stop double-counts. Removing any one of them brings the phantom reps back.
• Tune thresholds per exercise and camera. The numbers above are starting points. A gap of at least a few tens of degrees between enter and exit is what gives you the debounce; widen it if you still see chatter.
• Average or pick the better side. When both left and right joints are visible, average the two angles, or use whichever side has the higher confidence, to cut noise.
• Pick the model tier for the job. MoveNet Lightning (or a lite BlazePose) for latency; Thunder (or heavy) when form grading needs the accuracy and the device can hold the frame rate. Enable the GPU delegate where available.
• Confirm the frame-processor API for your version. Because the hook and prop names moved across v3, v4, and v5, a mismatch here is the most likely reason your build fails to compile. The docs page for your installed major version is the source of truth.

If the end user is exercising in front of the camera, give them one setup line: clear a few feet of space, get the whole body head-to-ankle in frame with even, front-facing lighting, and prop the phone at a side angle for squats and curls. Fitted clothing tracks better than baggy layers.

The buy path: a commercial fitness SDK

If motion analysis is not your core product, a commercial fitness SDK collapses every step above into a drop-in. Vendors such as KinesteX, Kemtai, and QuickPose bundle the camera lifecycle, the pose model, smoothing, rep counting, exercise or form classification, and a prebuilt overlay UI, and they ship React Native integrations. You install the SDK, initialize it, mount its view, and read rep and form events from a callback instead of wiring VisionCamera, running the model, and writing the state machine yourself.

The exact initializer signatures, view components, and callback shapes vary by vendor, so follow the SDK's own documentation for the precise API rather than any names reproduced here. Treat vendor-published accuracy and frames-per-second numbers as vendor-stated claims to verify in a trial. The trade-off is the usual one: less code and faster time-to-feature, against less control over the exact form logic and a recurring license cost.