Google AI Teaches Humanoid Robot to Play Cornhole by Watching Human Videos

Google DeepMind has demonstrated how AI can teach a humanoid robot a new physical skill just by watching a video of a person performing it, instead of having to rely on engineers to program every movement. 

On October 13, in conjunction with the unveiling of the Gemini 4 Argon AI model from Google, the company made public a demonstration of Apptronik’s Apollo humanoid being taught the game of cornhole, where one must throw beanbags at an elevated board with an opening in it. 

The robot was able to land seven straight shots once its AI had done its homework: studying human form and function, some practice in virtual space and then applying those lessons to the machine itself, Google reports. 

Gemini 4 Argon set the whole thing in motion by way of a video analysis; the model would make note of the environment and figure out the body mechanics required for the throw. From there it wrote the simulation code so the robot could have a go at it virtually first. 

Google describes the “Video-to-Sim-to-Real” technique as a means of saving engineers the hours of manual labour that goes into building simulations and robotic programmes; they claim it can cut development time from weeks to a matter of hours. 

There were no instant successes, though. Norman Di Palo of Google DeepMind, who was involved in the experiment, took to social media to say that after the robot’s first efforts, more training was called for on the physical unit. The seven-shot run came only after some tweaking, a testament to the value of feedback from the real world. 

It is an exercise in the kind of embodied reasoning Google is after in its AI systems. On the ERQA benchmark, which puts to the test a system’s grasp of spatial and physical conundrums, Gemini 4 Argon scored 87%, though that is a measure of reasoning, not of a robot’s aptitude in the field. 

To that end, Apptronik, the Texas firm that makes Apollo, has been in collaboration with Google DeepMind, making use of large-scale training and data from the field to further humanoid intelligence. In June, the Austin developer put in an expansion at its Robot Park so the Apollo robots can amass data from their activities for future AI deployments. 

Cornhole is hardly an industrial application but the method has its uses. Should a robot be able to pick up a physical skill off a video, it may be possible for developers to put them to work in all manner of things without having to programme every move. 

For now, these are internal results and Google has yet to determine if the approach holds up in other settings or when the task is more complex. Yet it is indicative of the part that observation and simulation will come to play in schooling a humanoid.