Most AI works with words and images on a screen. Physical AI has to act in the real world: sensing its surroundings through imperfect hardware and moving real objects through real space. That’s far harder than the polished robot videos suggest. These four demos walk the arc end to end, from why robots fail to how they learn not to. Try the controls.
What does the robot actually know?
Sensor fusion is how a robot combines several imperfect senses, like sight, distance, and touch, into one clear picture of the world. No single sensor is enough on its own, but because each one covers the others’ blind spots, together they add up to something far more reliable than any of them alone.
A robot can only act on what it can sense, and every sensor is partial. A camera sees shape and color but can’t judge distance. A depth sensor measures distance but struggles with plain or reflective surfaces. Touch says nothing until the gripper makes contact, but it is the one cue for how hard the robot is squeezing. And motor feedback, the robot’s sense of its own joints (what researchers call proprioception), says nothing about the outside world. Intelligence here means fusing these flawed signals into one trustworthy estimate. Below, the robot moves packages from the infeed belt to the outfeed chute, aiming each pick at a guess drawn from its belief rather than at the package itself. Toggle the senses and watch the misses pile up and packages get crushed, each failure tagged with the sensor that caused it.
Switch a sense off
Every toggle changes the robot’s belief about where the package is — and where it reaches for it.
Sensors — switch one off to see what it was doing
Simplified illustration. Confidence values are hand-tuned and grasp outcomes are simulated to convey the concept, not measured sensor data.
So the robot acts on a belief, not on reality, and good sensing is what keeps that belief honest. But sensing is only half the problem. Even with perfect perception, the robot only knows how to handle situations it was trained for. On a real line that is the difference between a cell that copes with a new box size and one that stops. Next: what happens at the edges of that training?
The Operating Envelope
Robustness is a robot’s ability to keep working when the real world drifts away from the conditions it learned in. A robot that only ever practiced in a tidy lab tends to stumble the moment something changes, whether a heavier object, a more slippery surface, or a measurement that’s slightly off, even though nothing about the task itself got harder.
A robot’s learned behavior, what researchers call its policy, only holds up within the range of conditions it was trained on: its operating envelope. Push a slider past its green band and the robot starts failing, usually by executing its learned motion perfectly in a situation it was never prepared for. The envelope’s edge is a gradient, not a cliff: just past the band only some attempts fail, deep in the red nearly all do, and the failures get more dramatic the farther you push. Changes take effect on the robot’s next cycle. If more than one condition is out of range, whichever goes wrong first (the lift, the carry, or the placement) is the failure you see.
Then widen the training breadth and watch conditions that used to fail succeed. That’s domain randomization: training across wildly varied conditions so the robot generalizes instead of memorizing one narrow setup.
Push it outside its training
Drag a slider past its green band and the same learned motion starts to fail.
Simplified illustration. Failure thresholds and trained bands are illustrative, not a physics simulation.
Failures aren’t random: they happen when conditions leave the envelope the robot was trained on. Which points straight at the fix: widen that envelope by training on far more varied conditions. This is what happens when a new supplier’s boxes are heavier than the ones the cell was commissioned on. Next: how robots actually learn a more general skill.
Crossing the Reality Gap
Demo 02 showed that a robot fails outside the conditions it was trained on. So the obvious question is how you widen that envelope. One of the dominant answers today is to train the robot in simulation across a huge range of randomized conditions, so the skill it learns survives the jump to messy real hardware.
Both robots below run the same learned pick on a loop. The left one performs in the pristine simulation it trained in, so it succeeds every time. The right one runs the very same policy on real hardware, where the camera is noisy. The box drawn on each parcel is where the robot thinks it is: blue and locked on in simulation, amber and drifting on the real robot. When the policy has only ever been shown a pristine simulation, that estimate lands off the parcel, the arm reaches for it anyway, and the grasp closes on nothing. Then widen the training: as the simulation is randomized across more varied conditions, the policy learns the underlying skill instead of memorizing the pristine sim, and the real-world success rate climbs to meet it.
Train in simulation, run for real
Raise the domain randomization and watch the robot’s estimate of where the parcel is tighten onto the parcel.
Simplified illustration. Transfer outcomes are simulated to convey the concept, not real policy performance.
Randomized simulation is one way to build a general skill. It is also why a vendor’s simulation footage tells you less than their real-world success rate. But it isn’t the only way, and sometimes the fastest way to teach a robot is to show it. Next: learning straight from human demonstration.
Learning From Demonstration
Training in simulation is one way to teach a skill. The other major approach skips the simulator entirely: a person shows the robot what to do, once or twice, and the robot learns to reproduce and generalize the motion. This is how a lot of modern manipulation policies are taught.
Instead of hand-coding a motion or running millions of simulations, you can simply show the robot what to do. Drag the gripper through a pick once or twice, release it over the shipping bin, and the robot learns from what you showed it straight away. The real test is generalization: the part will reappear in a spot you never demonstrated. A policy that merely memorized your path reaches for the old spot and closes on nothing, while one that learned the underlying intent (approach, grasp, carry, release) adapts and succeeds.
Teach it by dragging
Drag the gripper from the part to the shipping bin, then test what it learned on a new position.
Simplified illustration. The learning and generalization are a simplified stand-in for imitation learning, not a trained model.