Dear friends,
I manually analyze hundreds of competitive swimming race videos to extract per-swimmer biomechanical timing/distance metrics (reaction time, breakout time/distance, underwater length, stroke rate, split times, etc.). We want to train a custom model to reliably detect/track one specific swimmer across a race, including through phases where they are fully submerged and visually similar to 4-9 other swimmers in adjacent lanes. A general-purpose human pose model (MediaPipe Pose) has already failed on the submerged/partially-visible cases. Looking for guidance on the right task type (detection / instance segmentation / pose-keypoint), annotation strategy, and tracking approach for this specific problem.
Project context
We run video analysis of competitive swimming races (50m pool, various strokes: freestyle, backstroke, breaststroke, butterfly, individual medley). Each race has ~5-10 swimmers, of which ~5-7 lanes are simultaneously visible in a given camera shot. Currently this analysis (extracting per-swimmer timing and distance metrics from video) is done manually, frame-by-frame, and we’re trying to build a semi-automated pipeline to scale this to hundreds of videos without losing the precision we currently get by hand (target tolerances below).
Footage characteristics
- Camera positioned at pool-deck level, roughly side-on to the pool (not an elevated/overhead view).
- Camera may be static or may pan/zoom to follow the field along the pool.
- Typical source: 1920x1080 @ 50fps or 1280x720 @ 30fps (broadcast-style footage), sometimes a fixed lane-cam.
- 5-7 lanes visible at once out of ~10 competing swimmers; swimmers in adjacent lanes look visually very similar (same caps/suits in many cases, same body type, same stroke motion in sync).
- A given swimmer cycles through very different visual states within the same race:
- above water (fully visible, normal swimming),
- partially submerged (head/shoulders visible, body underwater),
- fully submerged (underwater kick phase after start/turn — sometimes nearly invisible, just a shadow/blob),
- airborne during the start dive,
- at the wall during a turn (compressed, foreshortened, changing direction),
- breaking the surface again (breakout).
The core computer-vision challenge
This is not a general person-tracking problem. The specific difficulties:
- Long fully-submerged phases (several seconds after start/turn) where the swimmer is a low-contrast, distorted, sometimes near-invisible shape underwater — not a legible human pose.
- Visually near-identical distractors in adjacent lanes, swimming the same stroke in sync — a tracker must not “jump” to a neighboring swimmer when the target briefly disappears or gets occluded by splash.
- Repeated full disappearance/reappearance cycles (every ~15-25m: dive/push-off → underwater → breakout → swim → turn → underwater again), not a single occlusion event.
- We need the tracking to be tied to lane identity (the swimmer’s assigned lane, which has fixed real-world geometry relative to the lane ropes) as a strong prior, not just visual appearance continuity.
What we’ve already tried, and why it’s not enough
- MediaPipe Pose (general-purpose human pose landmark model) for body tracking: works reasonably above water, but fails outright on submerged face-up swimmers (backstroke) — visually confirmed as just an indistinct shadow with no detectable landmarks — and loses tracking within ~1s of entry during butterfly’s long underwater dolphin-kick glide (locks onto the wrong landmark and diverges). Tested across 8 races; underwater distance/position tracking did not converge to a trustworthy result on any of them (acceptance rate of plausible per-frame detections: 11-30%, all below our own ~40% trust threshold).
- Lane geometry (rope detection by color/pattern, re-identified per frame rather than via frame-to-frame continuity) works reliably as a spatial reference and for lane identification, but the underwater swimmer position/extent itself is the unsolved part.
- Above-water timing events (push-off, wall touch, entry-from-splash) are already reliably hand/frame-identified (~0.02-0.06s accuracy) — it’s specifically the submerged phase (position, extent, head location) where the current general-purpose model breaks down.
Data we already have automatically (not something we need the model to produce)
For every race we already have, from the timing/results system and existing manual review database, independent of video analysis:
- athlete identity, lane assignment, discipline, distance, race stage (heat/semi/final)
- official race time, official 50m split times
- reaction time (RT) at the start, where available
- for a subset of races, a full manually-reviewed reference table (time/distance/stroke metrics per race segment) used as ground truth for QC — not derived from our video pipeline
These serve as hard anchors for validation (e.g., every 50m segment must reconcile with the official split; we treat drift between video-derived numbers and these official anchors as a signal to go find the actual mis-identified video event, never as something to redistribute mathematically).
Target output variables (what we want the video pipeline to produce, currently done by hand)
Per swimmer, per race, at ~0.02-0.08s / ~0.1-0.3m target tolerance:
- reaction time cross-check, flight time, flight distance, entry time
- time 0-5m, time 0-15m (and sub-splits 15-25/25-35/35-45/45-50m)
- breakout time, breakout distance
- underwater length, underwater kick count, underwater speed
- turn time, turn 5m-in, turn 5m-out
- stroke count, stroke rate, stroke length
- clean-swimming speed
- full split times per 50m segment (and, for multi-leg races like 200IM, per stroke leg)
Almost all of the above depend on knowing the swimmer’s position (specifically head-center, per our checkpoint definition) at specific moments — which is exactly the part that currently fails underwater.
Current pipeline (Python/OpenCV, local, no cloud dependency at inference time)
Lane-rope detection for spatial calibration → per-frame swimmer localization (currently MediaPipe Pose, the failing piece) → event detection (push-off, breakout, turns) via checkpoint crossing of defined spatial planes → derived metrics. We’re open to replacing the localization stage with a custom-trained YOLO model (detection, segmentation, or pose/keypoint) trained on our own footage, then exporting for local inference (GPU with CPU fallback) — training/annotation is the only part we’d consider doing on Ultralytics Platform’s cloud.
Questions for the community
- For a target that’s a head-center point, not a full body — is a pose/keypoint model (single or few keypoints, custom skeleton) the right task, or would we get more robust underwater behavior from instance segmentation (mask centroid) or plain detection (bbox center)?
- For the fully-submerged, low-contrast, near-invisible case specifically — has anyone trained a model on this kind of “swimmer as blob/shadow” scenario? Any guidance on backbone size / augmentation strategy for very-low-visual-signal targets?
- What’s a reasonable annotation dataset size/composition for this (frames per stroke type, above-water vs. partial vs. fully-submerged split) to get meaningfully better-than-MediaPipe underwater performance?
- For avoiding identity switches between visually near-identical swimmers in adjacent lanes during brief disappearances — is combining a custom YOLO detector with ByteTrack/BoT-SORT plus a hard lane-geometry constraint (never re-acquire outside the assigned lane’s spatial band) a sound approach, or is there a better-suited tracking method for this “repeated full disappearance, strong spatial prior” pattern?
- Is anyone aware of existing swimming-specific public datasets or pretrained models (even partial — e.g. underwater swimmer detection, pool-lane detection) we could start from instead of fully cold-starting annotation?
Thank you for your time and help.
Regards
J.M.