Phone as sensor, TV as stage

The screen is in the wrong place, so we moved it.

Set your phone on a tripod so it can see you play. Now notice what just happened to the screen. It is across the room, a few steps away, facing you from the wrong distance and the wrong angle, precisely when you are moving and could most use the feedback. The device that senses you and the device you watch have ended up in the same object, and that object cannot be in two places at once. You can be seen, or you can see — not both, from one phone.

Most fitness and sport apps quietly pretend this problem does not exist. They ask you to hold the phone, glance down at it, or prop it close, and in doing so they surrender the clean, stable camera angle that verification actually needs. We took the constraint at face value: if the sensing device and the viewing device want to be in different places, put them in different places. The phone becomes the sensor; a nearby television becomes the stage. This piece is about why that split is the right architecture for playing in a room — and about a hard rule that falls directly out of the geometry of a single camera: the body is verified, the ball is rendered.

How we split sensing from staging

The mechanism is casting, but with an important twist about what gets cast. The phone, on its tripod, captures the scene and runs the first pass of computer vision locally — the same on-device work described in "Why we verify on the phone, not the cloud." It then renders your motion and casts that rendered view to a screen you can actually see, over the ordinary screen-sharing plumbing that already exists in living rooms (Google Cast, AirPlay, and the like). What lands on the TV is not a passive mirror of the raw camera feed. It is the interpreted version — your body as the system understands it, drawn back to you in real time.

  PHONE · SENSOR                    TV · STAGE

   ┌─────┐                  ┌─────────────┐
   │  ◉  │ ···· cast ····▸  │      O      │
   │     │    sees you      │    ╱│╲  ▓▓  │
   └──┬──┘                  │    ╱ ╲      │
      ╿     tripod          └─────────────┘
                              rendered motion

  body verified  ·  ball rendered

Why we cast the rendering, not the video

One detail in that split does a lot of quiet work: what travels to the TV is the rendered interpretation, not the raw camera feed. Sending raw video to a second screen would be the heavy, laggy option, and it would push private, unprocessed footage around the room by default. Casting the rendered motion instead means shipping a light description of where the body is, which the receiving screen draws. That is cheaper on a home network, easier to keep under the perceptual latency bar, and it keeps the raw capture on the phone where the on-device work already lives. The stage shows you a faithful rendering of your movement without broadcasting a video of your living room to get there.

It also lets the two screens each do one job well. The phone, freed from having to show you anything, can devote itself to capture and computation; the TV, freed from having to see you, can devote itself to being legible and worth watching from across a room. Collapsing both into a single handheld device compromises each — you get a mediocre camera angle and a screen you cannot read. Separating them lets each be good at the one thing it is now responsible for.

You cannot watch yourself move

There is a reason splitting the screen out helps beyond geometry. You have a decent internal sense of where your limbs are — proprioception — but it is coarse, and it is famously unreliable for the fast, trained movements sport is made of. A golfer cannot feel the exact plane of the backswing; a player cannot see their own form mid-rally. An external view, placed where you can actually use it, is the oldest coaching tool there is. The stage is that external view, made live.

The size of that gap is easy to underestimate. Ask a skilled player to reproduce a movement they have done ten thousand times while you film it, then show them the film; they are routinely surprised at what they actually did. The internal sense of a motion and its external reality diverge, and no amount of concentration closes the distance from the inside. A live external stage is the only thing that does — which is why moving the screen to the wall was not a cosmetic choice but the entire point of the split.

For a live stage to feel like a mirror rather than a delayed recording, the loop from motion to rendered feedback has to stay under the thresholds of human perception. Jakob Nielsen's long-standing response-time limits are the ruler we use: about a tenth of a second feels instantaneous, about a second keeps you in flow, and beyond roughly ten seconds you have lost the person's attention. A stage that lags into the second-plus range stops being a mirror and starts being a distraction, so casting latency is a first-class engineering target, not an afterthought.

What one angle is enough for

Before explaining what a single camera cannot do, it is worth being precise about what it can, because the scoping rule is not arbitrary — it tracks a real difference between a body and a ball. Body pose survives one viewpoint for three concrete reasons. Scale: a human body fills a large, contiguous region of the frame, so there is abundant pixel evidence for every joint. Continuity and constraint: bodies move smoothly and obey hard anatomical limits — limbs have fixed lengths, joints bend within known ranges — so a strong prior fills in what any single frame leaves ambiguous, and where a limb was a frame ago sharply narrows where it can be now. Redundancy: the pose is over-determined by the whole visible skeleton, so a briefly hidden joint can be inferred from the rest of the body rather than guessed from nothing. This is precisely the structure that on-device estimators in the BlazePose lineage exploit, and it is why one well-placed phone is enough to stand behind a body-based result. A ball in free flight shares none of these properties — and that is the whole of the next problem.

One camera cannot measure a ball

The split into sensor and stage forces an honest audit of the picture on the TV: of everything shown, what did we actually measure, and what did we draw? The answer is not cosmetic. It comes straight out of the geometry of a single camera.

A camera flattens the world. Under the pinhole model, every point along a ray from the lens projects to the same pixel, so a single image cannot, on its own, tell you how far away something is — recovering three-dimensional structure from one view is underdetermined, which is the entire reason multiple-view geometry exists. For a large object that stays in frame and moves smoothly, priors and continuity carry you a long way: this is why on-device pose can track a human body reliably. For a small object in fast free flight — a ball crossing a room — those priors run out. Serious ball-tracking systems answer the depth problem with hardware: sports officiating tools in the Hawk-Eye lineage triangulate from an array of six or more calibrated, fixed cameras. One phone on a tripod is not that, and no amount of software fully substitutes for the missing viewpoints.

Concretely, the ambiguity is not subtle. A ball rendered small and moving fast across a single frame is consistent with a near ball moving slowly and a far ball moving quickly — the pixels are identical, and only depth tells them apart, and depth is exactly what one camera does not have. Add that a ball can vanish entirely for several frames behind a hand or against a busy background, and the honest verdict is that recovering its path from one phone would be inventing a trajectory, not measuring one.

The fix, when you genuinely need the ball, is geometric and well understood: observe it from two or more calibrated viewpoints at once and triangulate, the way stereo vision recovers depth and the way professional rigs fuse many fixed cameras into a single 3D track. That is a studio solution. It assumes cameras bolted to a venue and calibrated to millimetres — the opposite of a phone a player sets down in a den today and moves tomorrow. We could demand that setup and refuse to launch without it; instead we scoped the product to what one honest camera can actually measure, and were explicit about the rest.

Rendered is honest, as long as it is labeled

So we do not pretend. The body, which the phone can track reliably, is verified. The ball, which one camera cannot pin down in space, is rendered — drawn onto the stage because it makes the experience legible and fun to watch, but never counted as a measurement. The two are held apart by a rule we state plainly and enforce in the product: illustration never masquerades as a verified fact.

The practical consequence is the one that keeps competition honest: home ball-flight tracking stays out of ranked play. A rendered ball can make a rally beautiful on the stage, but it cannot decide a ranked outcome, because we did not measure it. What decides ranked play is what the system verified — the body.

This is a discipline that costs us something in the demo and pays us back in trust. It would be easy, and tempting, to draw a confident ball arc on the stage and quietly let it influence a score; plenty of consumer products blur that line. We draw the arc and then refuse to count it, and we say so out loud, because the moment a rendered thing is allowed to decide a real outcome, "verified" stops meaning anything. The separation of stage from sensor is what makes the separation of drawn from measured enforceable: the TV is free to be expressive precisely because the phone is the only thing that adjudicates.

The stage is where an audience appears

Splitting the display out has a consequence we did not initially set out to build but now treat as central: the moment the thing you watch is a separate screen, it can be watched by more than one person. The same rendered view a player uses for feedback is the one others in the room can follow, and the one a remote viewer can be shown. A phone held at arm's length is a private, single-user object; a stage is a shared one. Moving the screen to the wall did not just fix the geometry of being seen — it turned a solitary measurement into something that can be performed, followed, and competed over, which is most of the point of sport in the first place.

Other results

• The stage rides on plumbing people already own. Casting to a TV, a tablet, or a laptop reuses standard screen-sharing paths, so the second screen is usually a device already in the room rather than new hardware.
• A single well-placed angle is enough for the body. Because body pose tolerates one viewpoint, the sensor requirement stays modest: one phone, positioned once, not a rig.
• The stage is where spectating lives. Separating the display makes room for an audience — the same rendered view a player uses is the one others in the room, or watching a feed, can follow.
• Room variance is handled at setup, not silently. When lighting or placement is poor, the honest move is to tell the player to move the tripod rather than degrade what counts.

What's next

Phone-as-sensor, TV-as-stage is the base pattern for what a home becomes when the body is the controller. The near-term work is a tighter cast and a richer stage — without ever blurring the line between what was measured and what was drawn. That line is not a limitation to be marketed away. It is the thing that makes a Future of Sports result worth trusting, and the subject we turn to next in "Verification you can trust."