PAPER DEEP DIVE
RT-2 converts pretrained VLMs into robotic policies, enabling zero-shot generalization to novel instructions and objects.
RT-2 converts pretrained VLMs into robotic policies, enabling zero-shot generalization to novel instructions and objects.
SOURCE LINKS
Robot-use Agents (RUAs) have shown promising zero-shot manipulation capabilities, but reliance on frontier, proprietary models limits research into further training. To democratize RUA research, we introduce uni-robot-agent, a unified harness for competitive open-weight robot use that supports further training. Without privileged inputs, an out-of-the-box Qwen3.8-Flash-Next achieves a RoboDojo Score of 31.39, surpassing the published GPT-6 Astra-RoboProbe result of 28.97 and many VLA/WAM baselines, and performs competitively on 19 real-world tasks. Our analysis across model generations, families, and sizes links effective robot use to four general capabilities, suggesting that future RUA training should develop them jointly.
Human video scales easily and does not need robot hardware, but existing methods require depth sensing, overlay the robot onto 2D video, or generate data only for the demonstrated objects. We present Real2Sem2Real (R2Sem2R), a real-to-semantic-to-real pipeline that turns one monocular RGB video of a human demonstration into thousands of robot demonstrations across the semantic object instances of a category. R2Sem2R tracks the SE(3) pose of each object with DINOv3-distilled Gaussian splats, outperforming existing methods and FoundationPose with estimated metric depth. R2Sem2R then aligns digital cousins, each generated from one or a few photographs, to the demonstrated objects with a semantic alignment. This alignment matches whole objects and the functional regions of target objects, such as a coffee machine's tray. R2Sem2R maps the human hand to a feasible robot grasp that extends to the cousins. Robot demonstrations are generated either in IsaacSim, for photorealistic rendering, or MuJoCo, for contact physics, and R2Sem2R can generate over 2,000 episodes in about two hours. On two real robots and four tasks, we train a vision-language-action policy on data from one human video per task, and policies trained on our data match or exceed policies trained on teleoperation across diverse object instances. The project page is available at https://real2sem2real.github.io/.
RoboFoundry freezes the foundation model and treats the surrounding context, skill, and semantic-binding system as the evolving policy, allowing embodied agents to diagnose failures, validate persistent repairs, and transfer improvements across benchmarks and robots.
EgoScale is a human-to-dexterous-manipulation transfer framework. A VLA policy is pretrained on 20,854 hours of action-labeled egocentric human video, revealing a log-linear relationship between data scale and validation loss. Lightweight aligned human-robot mid-training then grounds the learned motor prior in robot control, improving average success by 54% over a no-pretraining baseline on a 22-DoF dexterous hand and transferring to lower-DoF hands.
As an Amazon Associate, we earn from qualifying purchases.
