Solstice

Film tagging

From a game link to who was on the pitch, second by second.

Paste a link to a quadball broadcast. Solstice watches the whole game and works out which player was on the pitch at every second, with no GPS vests, fixed cameras, rosters or models trained for the sport. When it isn't sure, it says so rather than guessing.

Reign v Texas Copperheads · USQ Cup 2026 · Pitch 1

Three minutes drawn at random before the run: 2:43:06 to 2:46:06 of the stream

  • Named at a look
  • Carried between looks
  • Named as someone else
  • On camera, left unnamed

The chart needs JavaScript. In this stretch the pipeline named players for 502 seconds correctly and 18 seconds wrongly, across 14 players.

One row per player the hand-made answer key shows on the pitch. Each column is one second. The pipeline never saw the key; its rules and both full runs were frozen before anything was scored. Most of the red, 11 of 18 seconds, is one repeated mistake: a gallery cell that names Tanner Bonheimer (Copperheads #23) as Mallory Hughes (#8). The official score counts only the seconds between the key's two-second checkpoints: 233 right and 7 wrong here.

What's different

Most sports tracking needs hardware in the venue, a fixed camera or a team of people tagging film. We wanted the opposite: any public recording, and nothing else.

It starts from a link
Give it a YouTube video, even a six-hour tournament stream. It finds the game, the teams and the stretches of live play itself. No roster is supplied; names come from the film.
Nothing is trained for quadball
An off-the-shelf person detector and tracker follow bodies, and general-purpose frontier models do the looking. When next month's model is better, it drops in and gets measured.
A wrong name costs twenty right ones
Every second credited to the wrong player is scored as minus 20. So the pipeline is built to leave a player unnamed rather than guess: a named second has to be right more than 95.2% of the time to help.
Tested blind on a game it had never seen
The rules were frozen, hashed and preregistered before the fresh game was run. In the main window, two independent full runs named 499 of the same seconds and never disagreed about who it was.
It costs model calls, not people
One 58-minute recording took about 6,100 model calls and 120 million tokens, plus 17 minutes of rented GPU: a catalogue of everyone in the game, then recognition, run twice, over two three-minute stretches. Nobody watched the film to help it.

The pipeline, stage by stage

Nine stages turn pixels into people, and people into minutes.

Used in the blind test

Pin the source

Download the exact chapter of the broadcast and fix one clock, so every later observation can be traced back to the right frame.

Receives
A video link, optionally a time range.
Hands on
The full-resolution recording, its checksum and exact frame times.
Runs on
Ordinary code: yt-dlp and FFmpeg.
The trap
Evidence pictures must be the same bytes every time they are cut. A new FFmpeg release changed its output, so the cutter is pinned to a known version.

Game C: 3,495 seconds of a 1080p, 60 fps tournament stream, cut from the chapter "Texas Copperheads vs. Reign".

Problems we had to solve

Each of these broke an earlier version.

Video and time Replays count goals twice

Broadcasts replay plays from a second camera, sometimes minutes later. A model watching the video sees two goals. We match the recording's audio against itself to find replayed stretches, then fold what happens in them back onto the original moment.

One recording replayed four stretches 515 to 521 seconds late. About 8 goals sat in the replays, which explained most of an early overcount.

Video and time The on-screen clock isn't the game clock

Score overlays freeze, reset or show the wrong thing. We read the game's start and end from the play itself, use whistles and commentary when the recording has them, and never require them. Stretches where the clock is unknown stay unknown: minutes become a range plus one consistent best guess.

One broadcast's score panel has a clock field that always reads 00:00. On another, the score matched 18 of 18 goals, but its operator updated it a median 4.2 seconds after the ball went through.

Video and time Seeing someone isn't the same as them playing

Substitutes warm up beside the pitch, and players on the pitch leave the frame. A cheap pitch check removes bench players before naming, and lineups carry players through camera gaps instead of benching them.

On the development game the pitch check removed 128 of 153 bench units and 2 of 80 units in play.

Video and time Fixed-time roll calls land in stoppages

Asking who is on the pitch every 90 seconds put many of the questions in timeouts and goal resets, where the answer is ambiguous. Roll-call times now come from detecting when the scoreboard clock stops.

27 of 80 fixed-time roll calls fell in stoppages. The clock-stop detector found all 17 stoppages in the answer key.

Telling people apart The tracker swaps players when they cross

Carrying a name along a track is how most of the game gets filled in, and also how most errors spread. Names are now carried only between two looks that agree, and a look is dropped when the rest of its track disagrees.

Fill caused 74 of 142 wrong seconds before this rule. With it, the development windows had 1 wrong second in 702.

Telling people apart Teammates look alike

Same kit, same headband, similar build. The comparison works through a fixed checklist of features that can't change during a game: skin, hair, build, sleeves, socks, cleats, braces. Any clear difference means two people. "Same" needs several matches, and two bodies in one frame are always two people.

Where the answer key could check it, a named gallery cell was right 94 to 97% of the time.

Telling people apart Small pieces never reach a person

Earlier versions joined thousands of short track pieces correctly, but the joins landed among other pieces and never on the player's main history, so they earned almost nothing. Recognising players against a fixed gallery at regular moments replaced piece-by-piece joining.

Of 282 correct joins in one test, only 11 reached the player's main identity.

Telling people apart Tracks quietly hold two people

A tracker can follow a referee into a keeper across a camera pan and call it one body. Tracks are now cut wherever the appearance model sees the look of the body change.

About 28% of whole-game track strips held two different people. Cutting on appearance change lowered the swap rate from 6.5% to 1.9% in one window and from 2.9% to 0 in the other.

Telling people apart Headbands change hands

A keeper's headband goes to the next keeper, but models treated it as proof of identity. Comparisons now use a fixed list of features that can't change mid-game, and a headband or number counts only as supporting evidence.

One assembled person absorbed two keepers' stints, costing 443 seconds; readers cited a headband in about a third of their identity answers.

Telling people apart Models are good at "different" and poor at "same"

Shown two crops, a model reliably spots a difference but rarely commits to a match. So differences split people freely, while matches need many agreeing features or a gallery look.

A visible difference was right 32 of 34 times. Only 1 of 122 same-player pairs produced a supported match.

Telling people apart The careful design lost to the simple one

A version built to never join two people without calibrated proof joined almost nobody. The simple catalogue it was meant to beat scored more than three times higher, and recognition against a gallery replaced it.

Value 158 against 533 on the same windows. Its settling step threw away 89.1% of 1.68 million accepted joins.

Telling people apart An off-the-shelf tracker's hidden object limit

Meta's SAM 3.1 tracks at most 48 objects at once. With bench, crowd and officials in frame, tracking silently collapsed. Shorter chunks fixed most of it, but SAM was not adopted.

In 10 of 13 chunks, tracked people fell from about 25 to 2.

Jersey numbers A misread number invents a player

If two confident reads of "13" and "18" each create a person, the game gains a phantom. Now people are assembled from how they look, numbers count as one kind of evidence, and the number is used to name the person at the end. A number read alone never creates anyone.

In one full-game run, 20 of 56 people with a confirmed number had no playing time at all.

Jersey numbers Small digits are unreadable, and look readable

On a wide broadcast shot a number is a few pixels tall, and models answer confidently anyway. Number reads are limited to close-ups at least 100 pixels tall, digits seen in separate looks combine only when appearance ties the looks to one body, and some kits (low-contrast prints, dice-style tilted digits) are handled by appearance alone.

Below 100 pixels, full numbers were right 1 to 3% of the time. One player read as #23 ten times, on digits 8 to 18 pixels tall, was #20.

Jersey numbers Two players share a number

Duplicate numbers happen, and so do reprints. Because identity comes from appearance first, two people with one number stay two people, and a corrected number keeps the person and their history.

Jersey numbers Part of a number outvotes the whole number

A half-hidden 34 reads as 3, and short reads are more common than full ones, so they win the vote. A short number now gives way to the one longer number that contains it when enough independent views read the longer one.

Truncations like 34 to 3, 99 to 9 and 66 to 6 put 3,357 seconds on the wrong player. A roster would have fixed only 25 of 5,024 wrong seconds, because 3, 9 and 6 were real shirts too.

Working with models A second look that sees the first one agrees with it

We tried having a model rewatch film with its own notes. It never changed its mind. Independent looks that never see each other's answers are now the rule, and an earlier pass's questions are used only as places to look.

Across 150 second viewings with notes, the goal list never changed. But first-pass questions fell within 6 seconds of 71% of its misses, against 33% by chance.

Working with models The same question twice gets two answers

Two readings of the same moments moved the score by 154 points. Requiring two separate readings to agree, and checking each look against its own track, turns that noise into abstention instead of error.

Working with models The right model for each job

Each task gets a small benchmark across several models, and goes to the cheapest one that is good enough on the errors that matter. Argon read jersey numbers best on the development game. In the blind test Sol did every job, at low effort for reads and high effort for recognition. Claude Opus was tested as a second reader and never named a wrong player, but abstained too often to move the result.

Working with models The wording of the question changes the answer

A video model counted goals on the same footage under two prompts. Every count now records its prompt, prompts ask what the model saw, and both styles run to give a range.

A prompt mentioning a whistle gave 30 goals; a neutral one gave 24; the truth was 28. Primed with the whistle, it reported 17 goals without seeing the ball go through a hoop.

Working with models Agreement finds errors; confidence doesn't

Each model states confidence on its own scale, so a single threshold means different things for different readers. Two cheap independent passes, with disagreements sent to a third, catch far more.

Where three independent passes agreed on 19 goal questions, all 19 were right; all 7 wrong answers sat in the 4 where they split. Opus said 0.9 or more on 2 to 10% of number reads, Sol on 58 to 84%.

Working with models More thinking made a model worse

Turning Sol up to its maximum reasoning effort made it slower and less accurate on number reads, so reads run at low effort.

At maximum effort Sol got 0 test cases right and abstained 6 times, against 2 right at low and high effort, in about twice the time.

Measuring honestly "Looks right" isn't a measurement

Every result is scored against a hand-made, second-by-second answer key for stretches drawn at random before anyone looks at them. Code that runs during a game can't open the key; scoring code is kept separate.

Measuring honestly Tuning on the test game

Rules chosen after seeing a game's scores are development evidence about that game, nothing more. The fresh test froze and hashed every piece of the method first, preregistered which number counts, and ran the whole thing twice.

All 17 hash checks passed before scoring opened.

Measuring honestly A pilot nobody could score

One 120-question pilot produced no answers the key could check: its questions fell outside the scored stretches. Every new test now pushes perfect answers through the scorer before a single model call.

Measuring honestly The answer key was wrong

Hand-made keys contain mistakes too. When two or more independent sessions agree against the key, the moment goes to a person as a clip question, and both scores are reported.

On the development game, three players' numbers in the key didn't match their shirts. Correcting them moved film-matched players from 12 to 15 of 26.

Measuring honestly #0 and #00 scored as different players

The fresh game's key writes #00 as "0", so the frozen scorer counts correct seconds as wrong. The official score keeps that by contract; a corrected rescore is reported beside it.

14 correct seconds per run were scored wrong. With the alias fixed, the whole-game identity score moved from −281 to 13.

Measuring honestly A number read backwards

A change was announced as a big win because the field was assumed to be a failure rate. It was a pass rate. Every claimed improvement is now checked against the scorer's own definition.

"3.0% to 0.33%" was a 9-times drop in passing, not in failure.

Plumbing A safety flag reordered paid work

Python's isolated mode ignores the fixed hash seed, so two export runs packed questions in a different order and completed answers no longer matched them. Exports now pin the seed, and two model-free exports must match byte for byte.

313 paid model opinions on the fresh game were orphaned.

Plumbing A hidden limit that looked like an outage

Gemini Flash's video mode stops after 23 looks with a generic internal error. Budgeting looks in the prompt and using five-minute clips fixed it.

10 of 10 failed jobs had made exactly 23 looks. The same clip then gave 9 goals instead of nothing.

Plumbing Pauses that didn't pause

Background drivers outlived a requested pause and kept calling a model, and spending caps were checked one call at a time. A pause is now confirmed by checking running processes, and one lock counts money already in flight.

After a pause, detached drivers made 196 more calls over 2 hours.

Results

Value is credited seconds minus 20 times wrong seconds, counted on the seconds between the key's checkpoints. The comparison is the pipeline's own earlier stage, which names players from the assembled catalogue alone.

Why a wrong second costs twenty

A wrong name is worse than no name: someone reads it, and every statistic built on it inherits the mistake. What a level of accuracy is worth, per 100 named seconds:

97.1%: value +39

Break-even is 95.2%. The fresh game's main stretch ran at 97.1%; the development game at 99.9%.

A game it had never seen

Reign v Texas Copperheads. Two three-minute windows drawn at random; window 2 was the preregistered main result. Each run is a full, independent pass with fresh model sessions.

Method Window 2: right / wrong Window 2 value Window 1: right / wrong Window 1 value
Catalogue alone 202 / 8 42 138 / 10 −62
Full pipeline, run 1 (preregistered) 233 / 7 93 120 / 3 60
Full pipeline, run 2 231 / 6 111 117 / 4 37

Overlap with the key (IDF1) rose from 0.374 to 0.556 in window 2. The two runs named 499 of the same seconds and never disagreed on who it was; 6 of run 1's 7 wrong seconds recur identically in run 2, so the errors come from the gallery, not from noise.

The development game

Dice v Heat, two three-minute windows. The combining rules were chosen on these windows, so this shows what the method can do, not a blind test.

Method Right / wrong Value IDF1
Catalogue alone 673 / 7 533 0.470
Looks every 5 seconds, no combining rules 859 / 33 199 0.569
Full pipeline 701 / 1 681 0.614
Perfect answers at the same looks (ceiling) 1,116 / 0 1,116 —

What's still hard

  • Bad gallery cells. The fresh game's errors repeat across independent runs because one cell's pictures were the wrong player. Checking cells against their own views, and auditing cells that win too often, is the work in progress.
  • Margins. On the fresh game, 2.9% of named seconds were wrong, against a break-even of 4.8%. It beat its baseline, by less than on the development game, and almost all of the gain came from one team.
  • Coverage. The pipeline names about half as many boxes as the catalogue alone. Abstaining keeps errors down and leaves time unclaimed.
  • One identity across the whole game. Each stretch scores well on its own. Held to one identity map across both stretches, the fresh game is about break-even once a scoring alias (#0 written as #00) is fixed. The rest of the loss comes from labels that mix two players.
  • Cost. Looking every few seconds across a full game is a lot of model calls. A schedule that looks less often where nothing changes kept only about half to four fifths of the fresh game's value, so it is not used yet.
  • Improvements that don't transfer. A revision with cleaner reference pictures and shorter candidate lists cut the fresh game's wrong time from 10 seconds to 4, but lost two thirds of the development game's value. It was not adopted; the frozen rule stays.

Open source

We're preparing the pipeline for release, including the scorer and the answer-key format, so others can run it on their own sport and check our numbers. Write to austin@solsticeapp.ca to hear when it's out.