The hardest part of verifying sport isn’t computer vision — it’s disagreement

Two reviewers, one clip, different calls. Who is right?

A squash player lunges and flicks the ball at the front wall. It catches the very top of the tin — the metal strip at the base — and drops. Two experienced officials are watching the same rally from the same side of the court. One says it was up, a good shot. The other says it was down, into the tin. Neither is careless. Neither is lying. They watched the same event and saw two different truths, because the event sat exactly on the line where "good" and "down" meet.

Most people assume the hard problem in verifying sport is the computer vision — teaching a machine to see the body and the ball. Vision is hard, but it is a solvable, improving kind of hard. The deeper problem is the one the two officials just demonstrated: after the machine has done its best and flagged the clips it is unsure about, what remains is genuine ambiguity, the kind that makes careful humans disagree. This piece is about how the human layer of the Kinetic Truth Engine™ is built to take that disagreement seriously — to turn it into a result you can defend, and, less obviously, into the single most valuable stream of training data we have.

How we turn disagreement into a result

The instinct, when experts disagree, is to put them in a room and let them talk it out until they converge. That is the wrong instinct, and understanding why is the whole design. When people deliberate, the first confident voice sets an anchor and the rest drift toward it — the anchoring effect documented by Tversky and Kahneman. A group that deliberates does not measure the play; it measures its own social dynamics, and it produces an answer that looks like agreement because the disagreement was pressured out of it.

So we do the opposite. When a session needs human review, it goes to several reviewers who each judge it independently and blind — no shared calls, no running tally, no knowledge of what anyone else decided. Each one sees the clip and the published rule for that challenge and records a judgment alone. Only after the calls are in does the system look at them together. This is the old idea behind Galton's "Vox Populi" and the structured-forecasting Delphi method: independent estimates, aggregated, beat a negotiated one. We call the layer that does this CORE — the Consensus Optimized Reinforcement Engine — a distributed network of independent event labelers.

Two design commitments follow from choosing independence over deliberation, and both cost us convenience. Reviewers must be assigned so they cannot coordinate, which means we forgo the efficiency of a standing panel that gets to know each other's habits. And we must be willing to store disagreement rather than resolve it away, which means carrying more state than a system that simply overwrites conflicting calls with a majority. We pay both costs on purpose: they are exactly what make the resulting agreement mean something.

  INDEPENDENT REVIEWERS        CONSENSUS

     R1 ●╮
         ╰──╮
     R2 ●───●──────▸  ✓ result stands
         ╭──╯
     R3 ●╯
     · · · · · · · · · · · · · · · · ·
     R4 ●╮
         ╰──╮
     R5 ●───●······▸  ✗ escalate + learn
         ╭──╯
     R6 ●╯

  blind calls in · agreement converges

Why a better camera won't fix this

It is tempting to treat the whole human layer as temporary — surely a good enough model eventually makes it unnecessary. We do not believe that, and the reason shapes the entire system. Better perception steadily shrinks the easy ambiguity, the kind caused by blur, a bad angle, or a dropped frame. It does not touch the hard ambiguity, which lives in the rules and the world, not in the pixels. "Did the pin fully settle?" has moments where the pin is genuinely still deciding. "Was the foot behind the line at the instant of release?" has releases where the foot and the line share the same few thousandths of a second. No sensor resolves a question whose answer is not yet determined at the moment it is asked. A sharper camera simply gives more reviewers the same clear view of an event that is itself on the boundary — and honest people still split. The human layer is not a crutch we plan to discard; it is where the irreducible calls get made, and it always will be.

You can see the same ceiling everywhere people have tried to remove judgment with technology. The best officiating tools resolve the measurable questions — was the ball in, did it cross the line — and leave the interpretive ones, which is why even heavily instrumented sports keep humans for the calls that turn on intent, on contact, or on the edge of a rule. We are not trying to abolish that layer. We are trying to give it the cleanest possible evidence and the least possible contamination, and then to learn from every call it makes.

Independence beats deliberation

It is worth being precise about the intervention, because it is testable. Take a batch of genuinely close clips and have them reviewed two ways: once with each reviewer working blind, and once where reviewers can see the calls made before them. The deliberative version produces higher apparent agreement — and that is exactly the problem. The agreement is manufactured by the anchor, not earned by the evidence. The blind version preserves the true spread of opinion, and that spread is information we need: it tells us, honestly, which clips are hard.

Agreement is a signal you can weight, not just count

Once the independent calls are in, the naive move is a majority vote. We do better than that, because not every reviewer is equally reliable on every kind of call, and a raw vote throws that knowledge away. The standard tool here is old and well understood: inter-rater reliability statistics such as Cohen's and Fleiss' kappa measure how much reviewers agree beyond chance, and the Dawid–Skene method (1979) shows how to estimate each reviewer's reliability and the most likely true label at the same time, from the pattern of their calls across many clips.

Concretely: suppose five reviewers see a close bowling clip and three call it a clean strike, two call it a foul on the approach. A majority vote says strike, 3–2, and stops. Our aggregator instead asks how reliable each of those five has proven on this exact kind of call, weights their votes accordingly, and reports not just an outcome but a confidence. If the three who said strike are the ones historically strongest on approach fouls, confidence is high and the result stands. If the split falls along less reliable lines, confidence is low and the clip escalates. The convergence — weighted, calibrated — is the authority. There is no senior official whose vote breaks ties by rank.

Calibrating a stranger

Weighting by reliability only works if we can estimate reliability, which raises the obvious question: how do you trust a reviewer you have never worked with? The answer is standard practice in careful crowdsourcing, and we use it. Seed the stream with clips whose outcome is already well established, mixed in so they are indistinguishable from live work. A reviewer's calls on those known items give an honest, continuous read on their accuracy — per kind of call — without asking anything extra of them. New reviewers start with little weight and earn it; a reviewer who drifts on a particular call type quietly loses weight there and nowhere else. Combined with the Dawid–Skene estimation that reads reliability out of agreement patterns across the whole network, this means the system's trust in a person is always a measured quantity, never an assumed one — and it moves as their demonstrated accuracy moves.

Disagreement is where the information is

Here is the part that compounds, and it is the reason CORE has "reinforcement" in its name rather than just "consensus." The clips that make independent reviewers disagree are, by construction, the most informative clips in the entire system. They sit precisely on the boundaries the computer vision has not yet learned to draw. A thousand easy, unanimous sessions teach a model almost nothing. One genuinely contested session, resolved carefully, teaches it a great deal — which is the core insight of active learning: a model improves fastest when it is trained on the examples it is currently least certain about.

So we keep them all. Every resolved dispute — the clip, the blind calls, and the weighted outcome the network converged on — becomes a labeled example aimed exactly at the model's current blind spots. This closes a loop: the vision surfaces the hard cases, people resolve them, and their resolutions become the training signal that makes the next round of vision better, which shrinks the set of cases that need people at all.

This is why the "R" in CORE is reinforcement, not merely review. A static panel of judges would give you consistent calls and teach you nothing. A system that routes its own hardest disagreements back into training turns the cost of ambiguity into an asset: the more genuinely difficult the terrain, the faster the map of it improves. Over time the boundary the model can draw unaided moves outward, and the human network is left with a smaller, sharper set of truly contested calls — which are, not coincidentally, the calls most worth a human's attention.

We did not take people out of the loop. We built the loop so that people disagree productively — and so that every disagreement makes the machine less likely to need them next time.

The threshold should scale with the stakes

Not every session deserves the same scrutiny, and treating them equally wastes the scarcest resource in the system: careful human attention. So the number of independent reviewers, and the bar a result must clear before it stands, scale with how consequential the session is. A casual practice throw and the final cut of a season are not adjudicated to the same depth, and they should not be. This is a design choice about where to spend judgment, not a claim about accuracy; the figures below are illustrative of the shape, not fixed constants.

Other results

• Reviewer calibration is continuous. Because Dawid–Skene-style estimation reads reliability out of the pattern of calls, reviewers are calibrated as they work, and their weight tracks their demonstrated accuracy on each kind of call rather than a fixed rank.
• Blind assignment doubles as a defense. The same independence that removes anchoring also raises the cost of collusion: it is hard to coordinate an answer with people you cannot see and were assigned at random.
• Human review runs on its own budget. Unlike the on-device pass, adjudication does not have to finish inside the moment; it has a separate, looser deadline, which is precisely why we push only the unsure minority of sessions into it. (See on-device verification for the split.)
• Every resolved case leaves an auditable trail. The clip, the blind calls, and the weighted outcome are retained together, which is what makes the result defensible after the fact and what feeds the training loop.

What's next

The work ahead is to tighten the loop: better reviewer calibration, faster routing of the genuinely hard clips to the people best suited to them, and a cleaner, quicker path from a resolved dispute back into the models. The measure of success is not that the humans disappear — it is that their attention lands, more and more precisely, only on the moments that truly need a human eye, while the machine absorbs everything they teach it. Verifying sport was never mainly a computer-vision problem. It was always a disagreement problem, and disagreement, handled well, is a gift.