Multi-Person Pose Tracking: Top-Down vs Bottom-Up

Multi-person pose tracking means estimating the skeletons of several people in the same frame and keeping each person's identity stable from frame to frame. There are two paradigms: top-down (detect each person, then run pose on each box — accurate per person, but cost grows with the number of people) and bottom-up (find all keypoints in the image at once, then group them into people — cost stays roughly constant, but the grouping step is harder and typically less accurate). The single most important thing to know first: most fitness apps are single-user (one person, one camera), and single-person models like MediaPipe/BlazePose or MoveNet are simpler, cheaper, and more accurate for that case. Pick multi-person only if you genuinely track several people at once — group classes, gyms, team or two-person sessions.

For the fuller picture of which model to use, see pose estimation models compared, and for implementation the camera pose tracking guide. Concepts start at what is pose estimation.

Top-down vs bottom-up

The two paradigms differ in the order of operations, and that order drives the cost and accuracy trade-off.

• Top-down first runs a person detector, then estimates pose per detected person (one person per cropped box). Each person gets a clean, well-framed crop, so per-person accuracy is generally higher — but you run the pose model once per person, so inference cost grows with the number of people in the frame.
• Bottom-up detects all keypoints in the whole image at once, then groups them into individuals — for example OpenPose's Part Affinity Fields, which learn the association between joints. Cost is roughly constant regardless of how many people are present, but the association step (deciding which elbow belongs to which shoulder) is harder, and accuracy is typically lower than top-down.

Rule of thumb: top-down favors accuracy in modest crowds, bottom-up favors predictable cost in large crowds. Neither is a universal winner — verify the behavior on your own scene and hardware.

Which models do which

• OpenPose (CMU) — bottom-up, the classic multi-person system, using Part Affinity Fields to associate keypoints into people. It is heavy and GPU-oriented, and its license is non-commercial / academic research — a separate license is required for commercial use, so verify terms before you ship anything commercial.
• YOLO-pose / Ultralytics (YOLO11-pose) — detection-driven multi-person: it produces person boxes and 17 keypoints in one pass (top-down style). It is capable in crowded scenes but is licensed AGPL-3.0 (strong copyleft) with a paid Enterprise License to bypass AGPL — treat the license as a real legal decision point for a commercial app.
• MediaPipe / BlazePose (Google) — single-person focused: it tracks one primary subject and is not designed for crowded multi-person scenes. It predicts 33 landmarks (a superset of the 17 COCO keypoints) and is optimized for on-device real-time use.
• MoveNet / PoseNet — single-person oriented (17 keypoints). MoveNet has a multipose variant, but verify current support in your pipeline before relying on it. PoseNet is legacy — prefer MoveNet or BlazePose for new work.

If you only ever have one person in frame, none of the multi-person machinery above is worth its complexity — a single-person on-device model is the better default.

Tracking IDs across frames

Estimating poses in a single frame is only half the job for multi-person work. You also need identity association across frames — keeping person A labeled as person A from one frame to the next. This is a separate tracking layer on top of per-frame pose estimation:

• Common approaches include box/IoU tracking, ByteTrack-style trackers, or pose-based association.
• Bottom-up grouping only handles association within a single frame; cross-frame ID persistence is added on top.
• Track IDs can be lost or swapped when people occlude each other or cross paths — a real failure mode in group scenes. Verify how your chosen tracker behaves on your camera angle and crowd density.

When a fitness app actually needs this

• Single-user (the common case): one person in front of one camera — a home rep-counting or form-feedback app. Use a single-person model (MediaPipe/BlazePose or MoveNet); skip multi-person and the tracking-ID layer entirely.
• Group classes, gyms, team or two-person sessions, overhead studio cameras: several people tracked at once. Here you need a multi-person model plus a cross-frame ID tracker.

For the multi-person case, choose by crowd size and where per-person accuracy matters:

• Top-down (YOLO-pose) when per-person accuracy matters and the crowd is modest — mind the AGPL-3.0 license.
• Bottom-up (OpenPose) when the crowd is large and you want cost that does not scale with the number of people — mind the non-commercial license.
• Add a dedicated tracking-ID layer for any multi-person case.

The honest limits

Multi-person adds failure modes on top of the usual pose-estimation caveats:

• Association errors. Bottom-up grouping and cross-frame tracking can both mislabel or swap people, especially under occlusion or when paths cross.
• Cost. Top-down inference cost grows with the number of people; a busy frame can blow your latency budget on lower-end hardware.
• The usual degraders still apply. Lighting, occlusion, oblique camera angles, small/distant subjects, baggy clothing, and motion blur all reduce accuracy — and multiple overlapping people make occlusion worse.
• Hardware. A normal RGB camera is enough (no depth sensor or LiDAR needed), but multi-person top-down and heavier bottom-up models are GPU-oriented, which often pushes you toward edge-GPU or server-side deployment — with the privacy and cost trade-offs that streaming video implies.
• Not medical advice. Any form feedback layered on top of pose tracking is a coaching aid, not medical or physical-therapy advice.

Do not build around specific per-person latency or crowd-scaling frame-rate figures you read anywhere, including here — those numbers vary wildly by device and scene. Test on your own hardware.

Before you ship

Confirm you actually need multi-person — if you are single-user, a single-person on-device model is simpler and more accurate. If you do need it, decide top-down vs bottom-up by crowd size and per-person accuracy, add a cross-frame tracking-ID layer, and re-verify the licenses (OpenPose non-commercial, Ultralytics YOLO AGPL-3.0) and current multipose support before committing. Then measure latency and ID stability on your target hardware. When you would rather buy the tracking layer than build it, see AI workout tracking APIs.