Cutting amateur tennis match video down to the rallies

Every two-week sprint ended with a written research report and a demo on the client's own recordings, so the client saw each decision and the evidence behind it.

The research ran in four lines: ball, court, players and trimming. Each one ended either in the product or in a written reason to drop it. From the fifth month, every release was benchmarked against the one before.

Why it matters: in the first test, most of the false "play" came from a match on the next court. A system that cannot tell its own court from the neighbour's would send coaches clips of strangers.

Classic motion detection found the ball, and also arms, shadows and wind. The fix was to locate the court first and mask out everything around it: neighbouring courts, passers-by and cars.

For full automation we trained a small neural network that finds 21 court keypoints, even when the net or the edge of the frame hides part of the court. We built its training set ourselves from about 1,300 frames with low camera angles, different light and different surfaces. The court is then mapped to a top-down view, so every player position is measured in court units.

Why it matters: the first product put the camera in the player's hands: an iPhone on a tripod at any court, with the cut made on the phone. A model that needs a big GPU could never run there.

We tested six ball detectors on the client's footage. General detectors missed the 2-pixel ball on the far side, and the most accurate one needed a whole GPU. TRexNet, a light model that reads differences between frames, ran about 40 times faster on a CPU and found more of the real balls.

The first trimming version judged each ball path by simple physics: how long it flies, a downward parabola, the distance across the court. With ball detection and trimming together, it ran for a full hour on an iPhone 15 Pro. The iOS app built around it reached a near-final prototype.

Why it matters: a coach notices a missing rally at once, and a clip full of pauses defeats the purpose. Ball-based rules stayed brittle when the ball was hidden or far away, so the final model looks at the players instead. Their movement during a rally is steady and visible in every frame.

The far player is too small to detect in a shrunk frame. So we stack a zoomed crop of the far half on top of the full frame and detect both players in one pass, with no second model and no extra time.

Each change, such as looking ahead in time as well as back or training on 11 recordings instead of one, was measured against the previous release.

A cut only helps if it is trusted. We checked how much of each match goes, whether the rallies stay, whether the system works on a court it has not seen, and what it costs to run.

The startup sells coaching video. With the cut automatic, every recorded match turns into a short review clip, and the service runs on cheap hardware as more clubs join.

All code, trained models and datasets were handed over into the client's own repositories, with documentation for the pipeline, so the client's team can keep developing it.