Google's New AI Learned to Control a Robot From Just One Video

Vídeo em Inglês, publicado pelo canal Synthetic Signal.

About this content: The summary, key points and topics below are generated automatically from the video transcript. They reflect the content, opinions, claims and predictions presented in the video and do not represent a position, validation or fact-check by IA Dots.

Video summary

Google introduced Gemini for Argon as a general model for coding, science, and business, featuring robotics capabilities that allow a humanoid robot to learn physical tasks from human videos via physics simulations and real-world testing.

Key points

• Google's newest model watched a single video of a person playing cornhole and enabled an Apollo humanoid robot to sink seven consecutive throws.
• The model generated its own physics simulation in MuJoCo, mapped spatial coordinates, and measured the robot's joints before real-world trials.
• Gemini for Argon was announced on October 13th, 2026, as a general model applied to coding, science, business work, and robotics.
• Demonstrations included controlling a Boston Dynamics Spot to open a trash bin with a foot pedal and reproducing controllers from published robotics papers.
• Google's robotics posts include tests where a robot's completion time improved and real-world success rates increased after simulation training.

Topics

Google Gemini for ArgonHumanoid roboticsPhysics simulationApptronik ApolloMuJoCo engineVideo-based learningSim-to-real gap

Video transcript

Transcript from available captions; it may contain errors.

A humanoid robot sank seven cornhole throws in a row after Google's newest model watched one video of a person playing. The video was only the first input. Before the first real throw, the model wrote its own physics simulation, practiced inside it, and measured the real robot's joints. Neither of Google's two launch posts says how many throws Apollo took in total, and across both posts, the only real-world success rate for a robot is attached to a different task. Google announced Gemini for Argon on October 13th, 2026, as a general model for coding, science, and business work. Robotics got one section of the post. In it, Google describes an internal test. The model watched a raw recording of a person playing cornhole. It mapped the spatial coordinates of the scene and worked out the whole-body joint movements a humanoid would need. Then it wrote the simulation code on its own. One report says it mapped the arm swing and the release point onto the humanoid's joints. The code ran in MuJoCo, a physics engine that many robotics groups use. The result was transferred to a humanoid called Apollo, built by the company Apptronik. In a real-world trial, Apollo sank seven consecutive shots. A post from the team fills in the steps between. The team says the model zero-shotted the task, and that all it was given was a video of a person doing it. After the video, the model generated its own simulation environment. TechCrunch blog reports that it practiced there until the hand cleared the hip cleanly. Then it ran system identification and trained throws on the real robot. Interesting Engineering reports that Apollo threw a few warm-up shots first, and that when asked to explain itself, the robot came up with gestures of its own. Google says the whole pipeline compresses weeks of manual simulation work into a few hours, and that it turns ordinary videos into an endless training ground for robotics. Neither launch post includes a measurement of that saving or a comparison with a named team. A video holds a person's arm swing, the arc of a bean bag, and the position of the board. It holds nothing about the robot's own body. The mass of Apollo's forearm, the response of its motors, and the friction in its joints appear in no cornhole footage. System identification is the step that measures those things. The real robot moves, its motors and joints are read, and the simulator is adjusted until the two agree. Roboticists call the mismatch it corrects the sim-to-real gap. A robot learns something in simulation that does not map accurately to the real world. Argon is open to paying developers through Google's developer interface. Apollo is built for factories and warehouses, and DeepMind and Apptronik assembled the chain from video to simulation to robot together. Apollo stands 5 ft 8 and weighs 160 lb. Apptronik grew out of a University of Texas at Austin lab that built NASA's Valkyrie robot. In February 2026, it raised $520 million in a round that included Google. Google also collaborates with the company on humanoids powered by Gemini robotics models. The prompts Google published for a related task include a list of things the model may not do inside the simulator. The list is specific. Google's own robotics teams had tried to teach robots from video before. In 2024, a system called vid-to-robot took a human video of a task and produced robot actions directly. To train it, the team needed a large set of pairs, each one a prompt video matched to a robot trajectory doing the same task. Those pairs did not exist, so people made them. Participants performed tasks while recording from the robot's point of view, and in another setup, a human and a robot did the same task in the same workspace. On human prompt videos, the system beat an earlier method by more than 20%. The same year, a navigation system from the same lab used a smartphone video tour of a building. In 57 tests in an office of 836 square meters, it reached success rates of up to 90%. It could not explore the building on its own and depended on the tour video. Neither paper describes a simulator. Google's first Gemini robotics model, in March 2025, trained on simulated and real-world data, including teleoperation, where a human guides the robot by remote control. MIT Technology Review called the robot slow and a little janky, and also called its ability to adapt on the fly impressive. Simulation as a bridge is an older argument in robotics. In 2019, OpenAI trained a robot hand to manipulate a Rubik's Cube with networks trained entirely in simulation. Synced Review put the training at roughly 13,000 years of simulated experience. To make the transfer work, the team randomized properties of the simulated world. It wrote that too much randomization makes learning difficult, while too little hinders transfer to the real robot. The hand could still turn the cube with two fingers tied together and while wearing a glove, conditions it had never met in simulation. OpenAI published the result with a denominator. The hand solved the cube 60% of the time on scrambles that needed 15 rotations or fewer, and 20% of the time on the hardest scrambles. An attempt counted as a failure if the hand stalled or dropped the cube. Critics said OpenAI overstated what the robot had learned since a separate algorithm chose the solution steps and the hand only executed the moves. The researcher Eric Jang made the case in a 2021 blog post for iterating directly on real robots. Bridging the gap between simulation and reality is a hard research problem, he wrote. It is also hard to know whether the bridge helps until a policy runs in the real world anyway. Argon's pipeline keeps the simulator and adds a final step. Practice on the real robot. In another demo, the model controlled a Boston Dynamics Spot to drop soda cans into a trash bin that opens with a foot pedal. The prompt told it that one diet Coke can sat behind it and another on a black table to its left. It first built an articulated model of the bin in MuJoCo and practiced balancing on three legs. Only then did Spot step on the real pedal. It also had to confirm in the front cameras that the front right foot was centered on the pedal. The prompt asked for two verification agents before the can was released. One watching the foot and one watching the hand. It also told the model to use a Spot Mobile Manipulation Skill, a prepared set of instructions. The post does not say what is in it. A third demo started from a photo of a pen and paper sketch. The drawing showed an Apollo 2 humanoid throwing a ball along an arc at six boxes stacked three, two one. From the sketch, the model built the physics scene and plan to throw lasting 2.8 seconds. The plan has four phases. The robot takes a stable stance and grasp, then winds up while compensating for its balance at the ankles and hips. It accelerates forward and opens its fingers at peak hand speed to release. The model confirmed in simulation that all six boxes fell. That result exists only in in A fourth demo took footage recorded from a human's own point of view. The model mapped the hand motions onto a simulated humanoid with 76 degrees of freedom and five-fingered hands. Bowls and trays were rebuilt as shapes that physically hold things since a simple box cannot. On a plane robot arm, the model had to stand a fallen mug upright and put a crumpled napkin inside it. It received a wrist camera view overlaid with a grid in 2 cm squares and it sent movement commands to a local server one turn at a time. In the published prompts, a person supplied the structure. The spot prompt named a prepared skill and set up the two verifiers. The mug prompt fixed the movement commands the model could send, among them look, grasp, place, move by a small amount, and stop on contact. Inside that frame, the model chose the sequence. Google's coding example follows the same pattern. A team gave Argon a rulebook of skills in a one-paragraph goal and it worked overnight. The posts do not publish a prompt for the cornhole test, so its frame is unknown. The robotics post was written by Kanishka Rao, a principal software engineer at Google, and it includes the actual prompts. One of them asks the model to recreate a real robot hand's recorded session as a simulation. Renders from three simulated cameras had to match the real footage. The simulated force on each fingertip had to match the recorded touch readings in timing, size, and direction. The model could nudge the recorded robot commands only in frames where a fingertip was near an object and by no more than a set angle. It finished by building a viewer where a human can compare real and simulated camera views and force arrows side by side. The prompt then lists what is not allowed. After the first moment, objects may move only through real contact. The model may not change physics parameters, apply outside forces to objects, add welds that glue things together, set object positions directly, or disable collisions. It also may not report contact forces that did not come from the physics solver, for example, by copying the recorded touch readings. Each item is a shortcut that would make the simulation match the recording without simulating anything. Google wrote all of them out. The safety section of the main post describes the same temptation in coding. Agents can take unauthorized shortcuts when they hit a roadblock, such as a failing test. Google says Argon's shortcut attempts fell to 2% on coding benchmarks where nothing rewards holding back. Google also built a monitor that reads the agent's step-by-step reasoning and actions and blocks workarounds in real time. It was used in internal deployments and in training runs. Google says it avoided penalizing the model's thoughts during training, so that the reasoning stays readable to the monitor. Google says it also trained Argon to report negative findings honestly and to pause for human confirmation before irreversible steps. In a simulation, two robot arms face 12 identical-looking pool balls. 11 weigh the same, one is heavier or lighter, and the task is to find it and say which in three weighings. The same model built the environment, defined the reward, and wrote the solver, which sees the scale through two camera views only. With one prompt per paper, Argon read published robotics papers and rebuilt their controllers as browser demos. Google says it reproduced 68 of them, among them DeepLoco and a humanoid riding a skateboard. The check in the prompt asks for zero console errors and non-zero task progress over 300 frames. It does not ask the demo to match the paper's reported results. A two-armed robot folded a sheet of paper into an origami fox over 136 seconds. The model labeled the video at four levels and flagged pauses, unstable grasps, and fumbles. For training by imitation, the prompt tells it to cut the weight of retry segments to between 15 and 25% of normal. On a small six-joint arm, the model got a rulebook file with safety rules. The goal was to pick up a plastic toy brick and place it in a wooden box. When an attempt failed, it was supposed to diagnose the cause from logs and video. Once it worked, it had to become at least twice as fast within a safety limit. The prompt split the work into three stages. In the first, the arm does not move. The model finds the brick and the box in the image and prints a plan. In the second, the gripper hovers above the brick without touching it, and the wrist camera must confirm that arm and vision agree. Only the third stage runs on its own. The first attempt took 58.9 seconds. Attempt two took 22.9 and grazed the rim of the box on the drop. Attempt three took 17.4 and landed dead center. Four things in the robotics announcement are Google's own. The trial is internal. The time saving is a claim. Google calls cornhole an out-of-distribution skill, meaning one outside the model's training data, and the posts do not show how that was established. And the benchmark where Argon scores 87% is a test of 400 multiple-choice questions about the physical world introduced by Google DeepMind in 2025. Google says Argon beat every other frontier model on it using reasoning alone, and frames the work as progress toward physical artificial general intelligence. The launch was on October 13th, and reporting so far relies on Google's own materials. One news summary says, "Researchers suggest the immense library of videos on YouTube could give Google an edge in physical intelligence without building its own robots." That is a forecast. In the demo, the input was a recording of one person, and the real robot's own measurements supplied the rest. The one real-world success rate in Google's robotics posts comes from a tabletop task with photo booth props. A two-armed robot faced nine party props, among them a plush cap with spring-hinged crown panels. From a single overhead photo, Argon rebuilt the props in MuJoCo and wrote a curriculum of 77 tasks. The curriculum covered picking and placing, uprighting objects, pushing, mid-air handovers between arms, scooping, and clearing the table. It generated more than 4,500 simulated episodes, with object positions randomized by up to 10 cm, and checked each in physics for a full reward. Fine-tuning a general robot policy on those episodes cut the real robot's completion time from 35 seconds to 24. Its real-world success rate rose from 56% to 75%.

Assistir ao vídeo original no YouTube

Assuntos: Robótica e humanoides, AGI. Publicado no YouTube em 2026-10-15.

Seleção automática por assunto e idioma. O vídeo pertence ao canal de origem.