Why we verify on the phone, not the cloud
Verification has a deadline, and the cloud misses it.
Stand at the foul line of a bowling alley and watch someone throw. They take four walking steps, the arm swings back, and on the way through the ball leaves the hand and meets the lane. From the top of the backswing to the release is about the time it takes to say the word "release" out loud. The part that actually decides the shot — wrist position, the angle of the hand, the instant the ball leaves the fingers — is a fraction of even that. Blink and you have missed the evidence.
Every sport we work with has a moment like this: short, decisive, and gone almost as soon as it happens. The job of the Kinetic Truth Engine™ is to say something true about that moment. This piece is about a single early decision that shaped everything after it: we run the first pass of verification on the phone, not in a data center. The reason is not fashion and not cost. It is that the event is shorter than a network round trip you can rely on, the cleanest evidence exists only at the lens, and a smaller model that sees the whole moment beats a larger one whose answer arrives after the moment is over. Here is how we arrived there, what the on-device pass does and does not do, and where humans still have to look.
How we measured the deadline
Before deciding where computation should live, we measured the thing that constrains it: time. Verification is not an open-ended analysis; it is a task with a deadline set by physics. The window opens just before the motion and closes the instant the outcome is fixed. Nothing we do can extend it.
Two numbers frame the window. The first is how long the decisive motion lasts — a few hundred milliseconds for a release or a swing, and often much less for the part that matters. That is the same order of magnitude as a human's simple reaction time, which is why these moments feel instantaneous even to the person producing them. The second is how finely we have to sample to describe the motion faithfully. A phone camera captures at 30 or 60 frames per second; the Nyquist–Shannon sampling theorem is the formal reminder that if the thing you are measuring changes faster than roughly half your sample rate, you cannot reconstruct it — you alias, and you get a confident wrong answer. Fast events want more frames, and more frames leave less time between each one for any work to happen.
can be over this fast
captures per second
at 60 fps
Put those together and the budget is unforgiving. At 60 frames per second there are about seventeen milliseconds between frames, and that gap is the entire time available, per frame, before the next one arrives. The tighter you look, the less slack there is.
The per-frame budget is simple arithmetic — one second divided by the frame rate — and it is the whole time the system has, per frame, to keep up with the moment.
The event is shorter than the network
The obvious counter-argument is that you do not have to do the work between frames — you can record the clip and analyze it afterward, in the cloud, where there is effectively unlimited compute. And for many jobs that is exactly right. Our objection is narrower and specific: the round trip to a data center, on a network you do not control, is not built to the deadline the moment sets, and its variability is worse than its average.
The naive fix is "just use a faster connection." But the failure here is not bandwidth, it is the tail. A cloud round trip is a chain — capture, encode, upload, queue, run inference, return — and the slowest link dominates. On a home network the upload leg is the one you least control: it shares an uplink with everything else in the building, and consumer links suffer from bufferbloat, where a momentarily full queue adds unpredictable delay. The distribution has a long tail, and as Google's engineers argued in "The Tail at Scale," once a result depends on a chain of steps, it is the rare slow step, not the typical fast one, that sets what users experience. A system that verifies live sport cannot afford a long tail: the moment does not wait in a queue for you.
So rather than argue about averages, we split the pipeline by what has to happen inside the window versus what can happen after. The first — reading the moment as it occurs — belongs at the source. The second — adjudication, aggregation, storage — can take its time.
Capture → infer
Frames are read and the first pass runs where they are produced. Two steps, no network in the critical path, so the work lands inside the per-frame budget.
Capture → encode → upload → queue → infer → return
Every added step is a place to wait, and the upload leg is the one you least control. The moment is gone before the answer comes back.
The cleanest signal is at the lens
Speed was the reason we started on the device. Signal quality is the reason we stayed. There is a second, quieter cost to shipping the moment to a server first: to send video over a home uplink you have to compress it hard, and compression throws away exactly the high-frequency detail that fast events live in. Compression artifacts — blocking, smearing on fast motion, dropped or duplicated frames when the encoder falls behind — land most heavily on the few frames that carry the release or the contact.
You can test this directly, and we did: run the same perception on the original frames as they come off the sensor, then run it again on a re-encoded, upload-grade copy of the same clip, and compare. The naive expectation is that a bigger model in the cloud more than makes up for slightly worse pixels. What we saw is that the degradation is not spread evenly — it concentrates on the hardest, fastest, most decision-relevant frames, which are the ones a bigger model most needs to be right about. The lens sees the original; everything downstream is working from a copy. Reading the moment where the light actually lands removes a whole class of errors before they can happen.
There is a hardware wrinkle that makes this worse for the cloud and better for the device. Phone cameras use a rolling shutter — they expose the image row by row rather than all at once — so genuinely fast motion is subtly skewed in every frame, in a way the on-device capture pipeline understands and can account for at the source. Once a clip has been re-encoded and shipped, that structure is harder to recover and the distortion is baked in. Reading the moment on the device means reading it with full knowledge of how the sensor produced it, not guessing after the fact from a compressed artifact of it.
A small model that sees everything beats a big one that arrives late
This reframes the usual accuracy-versus-latency trade. The instinct in machine perception is that accuracy scales with model size, so bigger is better. On a deadline, that instinct is incomplete: a model's useful accuracy is zero if its answer arrives after the window has closed, or if it never saw the decisive frames because they were dropped in transit. The right question is not "which model is most accurate in the abstract," but "which model can read the actual moment, in time."
On-device perception has matured to make that a real option. Pose estimators designed to run on phones — the lineage of Google's BlazePose and the MediaPipe perception framework — track the human body in real time on a phone's neural accelerator by exploiting a helpful fact: a body is large in the frame, stays in view, and moves smoothly from one frame to the next, so where it is now strongly predicts where it will be next. When we shrink our own models to fit the device and the window, the body-tracking survives that compression well. What does not survive as gracefully is anything small, fast, and free-flying — a ball crossing a room seen from a single phone.
That result is load-bearing. It is the origin of the scoping rule that runs through the whole platform: the body is verified, and free ball-flight is handled differently. We treat what the device can reliably measure as what counts, and we are explicit about the rest. The full argument — and why home ball-flight tracking stays out of ranked play — is its own piece, "Phone as sensor, TV as stage."
The device has to know when it is wrong
None of this works if the on-device pass is quietly overconfident. The most important thing we built into the first pass is not a score — it is a measure of its own uncertainty. Every candidate result carries a confidence signal, and that signal is what lets us split the labor honestly.
The intervention that proves its worth is simple to describe: add a confidence gate. When the device is clearly right, the result stands, in milliseconds, and nothing else needs to happen — which is the overwhelming majority of sessions. When the device is unsure — an occluded body, an odd camera angle, a genuinely close outcome — the session is routed onward instead of being forced into a confident-looking answer. Turning that gate off (accepting every on-device call) is faster and looks cleaner in a demo, but it launders the hard cases into false certainty. Turning it on trades a little latency on a minority of sessions for the ability to be honest about them.
First pass
Pose, object and event read from the original frames — inside the window.
Candidate + confidence
What it thinks happened, and how sure it is.
Result stands
The common case ends here, in milliseconds. Most sessions never need anything more.
Routed to CORE
Low-confidence and genuinely close sessions go to independent human reviewers.
Where the unsure sessions go is CORE, our distributed network of independent event labelers — real people who review the clip without seeing each other's calls. Machine speed on the common case, human judgment on the hard case: together, on-device computer vision plus distributed human consensus scoring is the Kinetic Truth Engine™. It is AI-powered, and it is human-in-the-loop by design, not by accident. How the human side turns disagreement into a result is the subject of a companion piece on CORE.
On-device CV
Reads the moment at the lens, in the window.
Confidence
Clear calls stand; unsure calls escalate.
Human consensus
Independent reviewers resolve the hard cases.
Verified session
Signed and recorded.
Other results
Putting the first pass on the device produced a set of second-order benefits we did not design for but now depend on:
- It keeps working when the connection does not. A bowling alley, a gym, a driveway — the places sport actually happens — have unreliable networks. On-device verification degrades gracefully to slow or absent connectivity instead of failing, because the answer does not depend on a live round trip.
- The raw video does not have to leave the phone to get a first answer. Reading the moment locally means less has to be shipped anywhere by default — a better posture for privacy, and one that gets its own careful treatment in our integrity work.
- Cost falls out of the architecture. Not uploading and re-processing every second of every session is cheaper than doing so, but that was a consequence, not the goal.
- We engineer to a device floor, not a ceiling. Capture rates, thermals, and accelerators vary widely across phones. We target the capabilities of a reasonable floor device and treat anything better as headroom, so verification does not silently depend on owning the newest phone.
What we haven't solved yet
This is a first design, and it is imperfect in ways worth naming plainly.
The model on a phone is smaller than one in a data center. Fitting the window means trading model size for timeliness. That trade is clearly worth it for body pose; it is a real, ongoing constraint for harder perception, and it sets the boundary of what we currently let count.
Perception degrades with the setup, and unevenly. Low light, a body partly out of frame, a strange tripod angle, motion blur — each erodes the first pass, and not by a uniform amount. Detecting that erosion well enough to escalate honestly is itself a research problem; a confidence signal that is wrong about being confident is the worst failure mode, and calibrating it is never finished.
Object physics remains hard from one camera. A single viewpoint cannot recover everything about a small, fast object in free flight. Until that changes, the scoping rule stands.
We make no claim of perfect accuracy, and never will. The point of the confidence gate and the human loop is that verification is estimation under uncertainty. A number that claimed perfection would be a marketing artifact, not a measurement.
What's next
The direction is steady: push more judgment onto the device without lowering the bar, so the common case gets faster and the set of sessions that truly need a human keeps shrinking to the ones that genuinely deserve one. Concretely, that means better-calibrated confidence so the gate escalates for the right reasons, on-device models that hold their accuracy as they shrink, and wider coverage as we bring the same architecture to more sports. We are live across five sports, and the architecture is built for fifty.
The deadline does not move. The work is to keep meeting it honestly.