The Controller Is Dying: Hand Tracking, Neural Bands and Agents That Watch You Work

Every VR headset VRExtasy ever reviewed came with the same accessory: two controllers, each with a thumbstick, a trigger, a grip button and a ring of infrared LEDs so the cameras could see them. They were brilliant for games and useless for everything else, because nobody wants to hold a wand to read an email. The industry knew it, and spent seven years making the wand disappear. Here is how, in the order it happened, and where it is going next.

2019: the cameras learn to see fingers

In December 2019 Oculus pushed a software update that let the original Quest track your bare hands with its four black-and-white tracking cameras. It was laggy, it lost your fingers when they crossed, and it was the most important update the headset ever got. Menus became something you poked. Developers started building games with no controllers in the box. By the time the Quest 2 arrived in 2020 the feature was good enough to forget it was there, and by the Quest 3 in 2023 it was good enough to type on a virtual keyboard, slowly, out of spite.

Hand tracking's limit was always physics. Cameras need to see the hand. Put it in a pocket, behind your back, or under a desk, and the headset is blind. That is the problem the next two devices solved from opposite directions.

2024: Apple makes looking the click

Vision Pro's answer was eyes. Cameras inside the headset track your gaze precisely enough that whatever you are looking at is already selected; a tiny pinch of thumb and forefinger, seen by downward-facing cameras, confirms it. Your hands never leave your lap. It is the single best thing about a headset with many flaws, and every rival copied it within a year. Samsung's Galaxy XR launched in 2025 with the same look-and-pinch model, and Meta's Quest got eye-tracked selection as an option on the Pro.

Eyes solved the lazy case. They did not solve the hidden case, and they still needed cameras pointed at your face from inside a visor, which is fine in a headset and impossible in glasses.

2024 to 2025: the wristband that reads your nerves

In September 2024 Meta showed Orion, a prototype pair of true AR glasses with a 70-degree field of view, silicon carbide lenses, and a wristband. The glasses were the headline; the wristband was the news. It uses electromyography, EMG, to pick up the electrical signals your brain sends down the arm to move the fingers. It reads the intent to pinch slightly before the pinch happens, and it does not care whether anyone can see your hand. Meta built around ten thousand Orions for internal use and never sold one.

The wristband did ship. In September 2025 Meta launched Ray-Ban Display at $799 with the Neural Band in the box. A year on it has become the quietly radical part of the product: a tap of thumb to finger scrolls, a twist of the wrist turns a dial, and a later update added handwriting, so you can reply to a message by tracing letters on your thigh under a restaurant table. Nobody around you sees a thing. For the first time, the input device is not something you hold or something that watches you; it is something that listens to your muscles.

2027: no controllers in the box

Meta's VR Glasses, announced at Connect in September 2026 for a spring 2027 launch, will be the first Meta headset sold without controllers. Hands, eyes and the band are the interface. It is a bet that seven years of groundwork have made the wand genuinely optional, and it is going to upset exactly one group: people who play shooters. Expect controllers to be sold separately, and expect the launch library to tell you more about the future of VR input than any keynote. We wrote up the glasses themselves here.

The agent in the glasses

Remove the controller and something has to replace the thumbstick's job of telling the computer what you want. The industry's answer is that the computer should already know. Voice assistants were the first draft, and they were reactive: you asked, they answered, badly. The 2026 version is an agent. It has the glasses' cameras, your location, your calendar and your history, and it is expected to act: start the recipe when you open the fridge, pull up the ticket when you reach the gate, translate the menu before you ask.

Meta AI, which does the asking-and-answering part today, rolled out to Quest headsets in the UK, Australia, Japan and three more countries in October 2026, and the same assistant lives in the Ray-Ban glasses. Google is building the equivalent into Android XR, Snap's Specs ship with one, and Apple is conspicuously rebuilding Siri for the job. Whether any of them earn the word "agent" this year is an open question; what is settled is that the input device of 2027 is a sentence, a glance, or nothing at all.

The strange coda: your hands are training data

The same hardware that reads your hands for a menu can record them for a robot, and that is becoming an industry of its own. On 20 October 2026 Skyworth XR, the headset arm of the Chinese electronics group, announced it would demonstrate "full-body ego-centric data capture" at RoboBusiness in Santa Clara. The kit is a family of head-mounted rigs, the two-camera E2, six-camera E6 and nine-camera E8, synchronised with wrist-mounted cameras, so a person doing an ordinary task, folding laundry, loading a dishwasher, is recorded from their own point of view with their hands fully in frame.

Skyworth's argument is that head cameras alone miss the moment that matters, the instant the hand closes on the object, because the hand is usually hiding it. Add the wrist view and the body motion and you have the data an "embodied" AI needs to learn the task. The company says it has shipped more than 10,000 of these units by August 2026, a figure we have only its word for. The direction is not in doubt, though. The decade began with headsets learning to see hands so that we could control computers. It is ending with headsets recording hands so that computers can learn to be us.

Verdict

The controller is not dead; it is retiring to the games room, where it was always happiest. For everything else, the hierarchy is set: eyes to point, fingers to confirm, the wrist to do it in secret, and an agent to do it before you thought of it. The part nobody put on a slide is that every one of those inputs is also an output, a stream of exactly how humans move, and someone is already selling it to the robots.