How to Add AI Workout Tracking to an iOS App (Swift)

You can add camera-based workout tracking to an iOS app entirely on-device with Apple's Vision framework: stream frames from the camera, run VNDetectHumanBodyPoseRequest on each one, read the body joints, then compute a joint angle and count reps with a small state machine. No server, no network, and nothing leaves the phone.

There are two paths. Build on a free primitive: Apple Vision (this guide) or MediaPipe Tasks for iOS give you raw body joints and you write the rep and form logic yourself. Buy an SDK: a commercial fitness SDK wraps capture, pose, rep counting, and form feedback behind a few calls, at the cost of a license and less control. This guide takes the build path with Vision, because it ships in the OS and needs no dependencies.

What you'll need

• Xcode and a physical iPhone (the camera and Vision run poorly or not at all in the Simulator).
• iOS 14.0 or later for 2D body pose (VNDetectHumanBodyPoseRequest). iOS 17.0 or later if you later want the 3D request.
• An NSCameraUsageDescription string in your Info.plist.
• Frameworks: AVFoundation (camera) and Vision (pose). Both are part of the iOS SDK.

Vision returns 19 normalized 2D joints per person (nose, eyes, ears, neck, shoulders, elbows, wrists, root/mid-hip, hips, knees, ankles), each with a confidence score.

Step 1: Set up the camera capture session

Create an AVCaptureSession, add the front camera as input and an AVCaptureVideoDataOutput that delivers frames to a delegate on a background queue. This is the pipeline that will feed Vision.

import AVFoundation
import Vision

final class PoseCameraController: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {

    private let session = AVCaptureSession()
    private let videoOutput = AVCaptureVideoDataOutput()
    private let request = VNDetectHumanBodyPoseRequest()

    func configure() throws {
        session.beginConfiguration()

        guard let camera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .front),
              let input = try? AVCaptureDeviceInput(device: camera),
              session.canAddInput(input) else { return }
        session.addInput(input)

        videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "vision.queue"))
        if session.canAddOutput(videoOutput) { session.addOutput(videoOutput) }

        session.commitConfiguration()
        session.startRunning()
    }
}

Step 2: Feed each frame to Vision

Implement the sample-buffer delegate method. For every frame, wrap the buffer in a VNImageRequestHandler and perform your VNDetectHumanBodyPoseRequest. Reusing the one request object across frames keeps allocations down.

func captureOutput(_ output: AVCaptureOutput,
                   didOutput sampleBuffer: CMSampleBuffer,
                   from connection: AVCaptureConnection) {

    let handler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer,
                                        orientation: .up,
                                        options: [:])
    try? handler.perform([request])

    guard let observation = request.results?.first as? VNHumanBodyPoseObservation else { return }
    process(observation)
}

Set orientation to match how the device is held; .up is a starting point you should adjust for your capture orientation.

Step 3: Read the joints (and mind the coordinate origin)

Call recognizedPoints(.all) on the observation to get every joint as a VNRecognizedPoint. Each point has a .location (a CGPoint) and a .confidence. Skip low-confidence joints before you use them.

One gotcha that trips everyone up: Vision returns normalized coordinates in the range 0 to 1 with the origin at the BOTTOM-LEFT, y increasing upward. UIKit and Core Graphics use a top-left origin, so overlays land upside-down unless you convert. Use Apple's VNImagePointForNormalizedPoint helper, and note you still typically flip y when drawing into a top-left UIKit context.

func process(_ observation: VNHumanBodyPoseObservation) {
    guard let points = try? observation.recognizedPoints(.all) else { return }

    for (jointName, point) in points {
        guard point.confidence > 0 else { continue }   // drop unreliable joints
        let normalized = point.location                 // CGPoint in [0,1], BOTTOM-LEFT origin

        // Map into image pixel space when you need to draw an overlay:
        let imagePoint = VNImagePointForNormalizedPoint(normalized,
                                                        Int(imageSize.width),
                                                        Int(imageSize.height))
        // For a top-left UIKit context you usually also flip Y:
        // let y = imageSize.height - imagePoint.y
        _ = (jointName, imagePoint)
    }
}

Step 4: Compute a joint angle for the exercise

A rep is a movement, and the cleanest scalar signal for one is the angle at a joint. For a squat, that is the knee angle with hip, knee, and ankle as the three points. Build the two vectors from the vertex and use the atan2 of the cross and dot products; this is numerically robust and returns 0 to 180 degrees without the NaN issues that a bare acos hits near the extremes.

import CoreGraphics

/// Interior angle at vertex B, for points A-B-C, in degrees (0...180).
func angle(_ a: CGPoint, _ b: CGPoint, _ c: CGPoint) -> CGFloat {
    let ba = CGPoint(x: a.x - b.x, y: a.y - b.y)
    let bc = CGPoint(x: c.x - b.x, y: c.y - b.y)
    let dot   = ba.x * bc.x + ba.y * bc.y
    let cross = ba.x * bc.y - ba.y * bc.x
    let rad   = atan2(abs(cross), dot)
    return rad * 180 / .pi
}

// Squat: standing knee angle is roughly 170-180 degrees; parallel is about 90.
// let knee = angle(leftHip, leftKnee, leftAnkle)

Keep x and y in the same units (normalized is fine, the angle is scale-invariant). A 2D angle only matches the true joint angle when the limb is roughly parallel to the camera, so film squats and curls from the side.

Step 5: Count reps with a two-threshold state machine

Rep counting is not machine learning; it is a threshold state machine over that angle. Use two thresholds, not one: the signal must drop below an "enter" value to be counted down, then rise back above a separate "exit" value to be counted up. That gap is hysteresis, and it stops sensor noise from firing dozens of phantom reps at a single threshold. Count on one edge only.

final class RepCounter {
    private let enterDown: CGFloat = 100   // knee angle must go BELOW this (deep)
    private let exitUp:    CGFloat = 160   // must go ABOVE this (standing)
    private let alpha:     CGFloat = 0.3   // EMA smoothing, 0.2...0.4

    private var state = "up"
    private var smoothed: CGFloat?
    private(set) var reps = 0

    func update(kneeAngle raw: CGFloat) {
        // Exponential moving average kills single-frame jitter.
        let s = smoothed.map { alpha * raw + (1 - alpha) * $0 } ?? raw
        smoothed = s

        if state == "up", s < enterDown {
            state = "down"                 // reached the bottom; do NOT count yet
        } else if state == "down", s > exitUp {
            state = "up"
            reps += 1                      // count on the down -> up completion
        }
    }
}

Before calling update, gate on confidence: if the hip, knee, or ankle confidence is below roughly 0.6, skip the frame and hold state rather than feed a garbage angle into the machine.

Step 6: Surface the count in your UI

Vision runs on your background queue, so hop to the main queue before touching UI. Publish the rep count and the current joint angle, and draw the skeleton overlay from the converted image points in Step 3.

DispatchQueue.main.async {
    self.repLabel.text = "\(counter.reps) reps"
}

If a real person is exercising in front of the camera, tell them to clear some space, keep their whole body in frame head-to-ankles, and use even front-ish lighting. Fitted clothing and a side view for squats and curls both measurably improve joint tracking.

Make it reliable

• Confidence gate. Reject joints below about 0.5 to 0.7 confidence; occluded joints jump to garbage positions and can trip a threshold.
• Smooth the signal. The EMA on the angle (alpha 0.2 to 0.4) removes jitter so each threshold is crossed once, cleanly. Too much smoothing lags fast reps.
• Hysteresis, always. A single threshold will double- and triple-count. Keep the enter and exit values separated so the movement has to commit to each extreme.
• Normalize distances. For side-checks like knee-over-toe, scale pixel distances by a body reference (shoulder or hip width) so the rule works at any camera distance.
• One cue at a time. If you add form feedback, surface the single most important failed rule per rep ("go deeper", "chest up"), not five at once.

Alternatives

• Vision 3D (iOS 17+). VNDetectHumanBodyPose3DRequest returns a 17-joint 3D skeleton in meters, camera-relative, via VNHumanBodyPose3DObservation (with an estimated bodyHeight). Use it when a flat 2D angle is not enough and you need metric, out-of-plane joints. The exact 3D JointName case spellings differ from the 2D set; verify them against current Apple docs.
• MediaPipe Tasks for iOS. Google's MediaPipeTasksVision pod exposes a PoseLandmarker class with running modes .image, .video, and .liveStream; live camera work sets runningMode = .liveStream and delivers results to a poseLandmarkerLiveStreamDelegate. It returns 33 landmarks (versus Vision's 19) and runs cross-platform, which matters if you also ship Android. Verify the current pod and class spellings against the MediaPipe docs, as the Tasks API still moves.
• Buy a fitness SDK. A commercial fitness SDK wraps capture, pose, rep counting, and form feedback into a few calls, so you integrate rather than build the angle and state-machine logic yourself. See that vendor's own docs for the exact API; the tradeoff is a license fee and less control over the pipeline.