PAPER DEEP DIVE
Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration
Human video scales easily and does not need robot hardware, but existing methods require depth sensing, overlay the robot onto 2D video, or generate data only for the demonstrated objects. We present Real2Sem2Real (R2Sem2R), a real-to-semantic-to-real pipeline that turns one monocular RGB video of a human demonstration into thousands of robot demonstrations across the semantic object instances of a category. R2Sem2R tracks the SE(3) pose of each object with DINOv3-distilled Gaussian splats, outperforming existing methods and FoundationPose with estimated metric depth. R2Sem2R then aligns digital cousins, each generated from one or a few photographs, to the demonstrated objects with a semantic alignment. This alignment matches whole objects and the functional regions of target objects, such as a coffee machine's tray. R2Sem2R maps the human hand to a feasible robot grasp that extends to the cousins. Robot demonstrations are generated either in IsaacSim, for photorealistic rendering, or MuJoCo, for contact physics, and R2Sem2R can generate over 2,000 episodes in about two hours. On two real robots and four tasks, we train a vision-language-action policy on data from one human video per task, and policies trained on our data match or exceed policies trained on teleoperation across diverse object instances. The project page is available at https://real2sem2real.github.io/.
Authors: Matthew Strong, Ankush K. Dhawan, Julien Poffet, Giuse Pham, Jinho So, and Monroe Kennedy III.
Paper: arXiv:2610.28880, October 20, 2026.
Project page: real2sem2real.github.io. Code status: as of November 4, 2026, the project page still says “Paper and code coming soon.” The public GitHub repository is the project website, not an implementation of the method.
One-Sentence Summary
This paper turns one monocular RGB human demonstration into a robot-data generator across object instances: it recovers the demonstrated 3D object motion, aligns semantically related “digital cousins,” and generates simulation episodes that transfer across object geometries, robot embodiments, and four manipulation tasks.
Why Robot Data Collection Is Stuck Between Scale and Object Coverage
Teleoperation remains a dependable source of robot demonstrations because both actions and contacts occur on the target embodiment. Its bottleneck is human time. The paper reports roughly 48 seconds per demonstration including scene resets, so 150 demonstrations require about two hours of operator effort. Such datasets also tend to cover only a small number of object pairs. A vision-language-action policy can easily learn the appearance of the training mug or coffee machine instead of the task-level relation between them.
Simulation offers parallel data collection, but conventional simulation assumes that assets, trajectories, cameras, and scene layouts already exist. A human video has the opposite profile: it is cheap to capture and requires no robot hardware, but it contains pixels rather than actions, no dependable metric depth, and no complete object model. The challenge is to preserve the physical meaning of a demonstration while replacing the demonstrated instance with other members of the same category.
Previous approaches address only part of this gap. Motion capture and depth sensing recover accurate state but reintroduce hardware. Rendering a robot into an inpainted 2D frame keeps the original viewpoint but ties the data to that scene and makes object motion difficult to change. WARPED-style augmentation can vary texture, translation, scale, and camera pose, yet it does not change object geometry. The resulting policy sees many appearances of the same object pair, not many functional instances.
R2Sem2Real starts from a different abstraction. For placing a mug on a coffee machine, the invariant is not the exact mug or machine mesh. It is the relation between the mug and the machine tray. This observation leads to a three-part pipeline: recover object and hand trajectories from RGB, align category-level cousins by functional regions, and convert the aligned motion into robot grasps and inverse-kinematics targets.
Inputs, Outputs, and the Global Coordinate Story
The system takes one monocular RGB video of a human performing a task, a phone scan of the demonstrated objects, and one or several photographs of each cousin. It outputs thousands of robot demonstrations rather than a single retargeted trajectory. Every episode contains third-person RGB frames, end-effector poses, and commanded gripper opening values suitable for training a vision-language-action policy.
Figure 1 shows how the modules fit together. The human video and phone scan parse the source demonstration. Semantic alignment carries generated cousins to the same task pose. The final stage produces data for four tasks on an xArm7 and a Franka Panda, and the learned policies are evaluated on the real robots.
Figure 1: One RGB demonstration, a phone scan, and cousin photographs become multi-embodiment robot demonstrations.
flowchart LR
A[Single RGB human video] --> B[Object and hand segmentation]
B --> C[DINOv3 Gaussian-splat tracking]
B --> D[HaMeR hand reconstruction]
C --> E[Object trajectory T_t]
D --> F[Parallel-jaw grasp G]
S[Phone scan] --> C
P[Cousin photographs] --> Q[SAM3D or MV-SAM3D]
Q --> R[Semantic ICP]
R --> U[Task-weighted ICP]
U --> V[Cousin trajectory T_t A]
E --> W[Grasp transfer and IK]
F --> W
V --> W
W --> X[IsaacSim or MuJoCo generation]
X --> Y[RGB action episodes]
Y --> Z[LoRA-tuned pi_0.5 policy]
Z --> AA[Real xArm7 and Panda trials]
The central engineering issue is not any one network. It is coordinate consistency. The object, hand, grasp, robot base, camera, and simulation scene must all end up in the same metric frame. The scan supplies shape and an object coordinate system, a measured height supplies metric scale, the video supplies motion, and the robot scene supplies the execution frame.
Stage One: Recovering Objects Without Trusting Monocular Metric Depth
Object tracking estimates a six-degree-of-freedom pose in every video frame. RGB-D trackers such as FoundationPose can exploit direct metric geometry when a depth sensor is available. A monocular pipeline instead has to estimate depth and treat it as if it were sensor depth. Relative ordering may remain useful, but absolute scale, thin structures, reflective surfaces, and hand occlusion make the resulting pose drift.
R2Sem2R uses a phone scan to build a Gaussian-splat object model. DINOv3 features are distilled onto the Gaussians from scan views. The tracker then renders this model at a candidate pose and compares the rendered color, features, silhouette, and depth ordering with the video frame. The important choice is that monocular depth is used only as a ranking signal. It does not have to provide a globally accurate metric surface.
Segmenting the object from the scan
SAM3 segments the object in the first scan image using a text prompt, and the mask is propagated to the remaining views. For a Gaussian with center $\mu_i$, the method computes the fraction of visible views in which the center projects inside the object mask:
$$p_i=|\mathcal{V}_i|^{-1}\sum_{j\in\mathcal{V}_i}M_j(\pi_j(\mu_i))$$Here $\mathcal{V}_i$ is the set of training views that see the Gaussian, $\pi_j$ projects into view $j$, and $M_j$ is the propagated object mask. A Gaussian must be supported by more than one image rather than by a single favorable projection.
The retained set is
$$\mathcal{S}=\{\,i\mid p_i>0.5,\ |\mathcal{V}_i|\geq 3\,\}$$The 0.5 threshold requires agreement from a majority of visible views, while the three-view minimum suppresses isolated floaters caused by one bad mask. RANSAC then fits the table plane, and the scan is scaled to the measured object height. This is how the pipeline obtains a metric coordinate system without assuming that monocular depth can recover absolute scale.
Figure 3: Frozen vision models generate RGB, DINOv3 features, a hand-free mask, and relative depth. The splat is rendered at $T_t$, and four losses update the pose.
Stage Two: Render-and-Compare Tracking
Let $T_t\in SE(3)$ denote the object pose in the static camera frame at frame $t$. The tracker minimizes
$$\mathcal{L}(T_t)=\sum_k\lambda_k\mathcal{L}_k,\qquad k\in\{\mathrm{rgb},\mathrm{feat},\mathrm{sil},\mathrm{rank}\}$$The RGB term captures appearance and texture. The mask-gated DINOv3 term focuses on semantic correspondences when texture is weak. The silhouette term aligns the rendered object boundary with the detected boundary. The ranking term uses Depth Anything V3 only to enforce relative near-far ordering. No term requires the estimated depth to be globally metric.
Initialization exploits the fact that the scanned object was resting on a table. The system enumerates upright orientations about the table normal and optimizes each seed. Candidates are ranked using RGB, DINOv3, and silhouette losses, with silhouette IoU receiving double weight. This reduces the chance that a semantically plausible but spatially displaced seed is selected.
Hand occlusion is handled explicitly. When a person grasps a mug, the hand often covers the handle, which is exactly the asymmetric feature needed to determine yaw. If those pixels remain in the silhouette loss, the renderer can rotate the mug to explain a handle that is not actually visible. The method removes hand pixels from the comparison and tracks later frames from the previous pose with light smoothing.
The tracking results support the design. On the seven objects from the four tasks, the mean per-frame mask IoU is 0.913. WARPED reaches 0.681, RSRD reaches 0.415, FoundationPose reaches 0.823 with depth calibrated once per video, and only 0.629 with raw estimated depth. On YCBInEOAT, the proposed tracker obtains 51.4 ADD-S AUC and 41.3 ADD AUC, both above every monocular baseline.
Stage Three: Turning a Human Grasp into an Executable Gripper Pose
Tracking supplies the object trajectory but not a robot grasp. HaMeR reconstructs the hand in every frame. The hand mesh is approximately metric in its own frame, but monocular scale ambiguity can place it roughly half a meter away from the scene. The method fits an affine mapping from monocular depth to the metric depth rendered by the tracked object splat on object pixels, moves the hand along its viewing rays, and applies a small depth offset found by a one-dimensional search that minimizes fingertip-to-object distance.
The tool center point is the midpoint of the thumb and index fingertips. The closing axis is the line between the fingertips, and the approach direction runs from the wrist toward that midpoint. This converts the human pinch into a parallel-jaw grasp $G\in SE(3)$. Because the grasp is stored in the object frame, the end-effector target at frame $t$ is simply
$$T_tG$$This representation is crucial for transfer. It describes where the gripper sits relative to the object, not where the human hand happened to appear in camera coordinates. A cousin that inherits the object pose can therefore inherit the grasp before local collision checking and refinement.
Not every frame is a useful grasp. A frame is eligible only when the fingertips form a closed pinch, the wrist is approximately rigid in the object frame, and the object has been lifted more than two centimeters above the table. The system samples 100 $SE(3)$ perturbations within two centimeters and $45^\circ$, visits them in order of distance from the human pose, and checks collisions against the gripper, the object TSDF mesh, and the table. The first collision-free antipodal grasp with a fully open gripper is selected. Across eligible frames, it keeps the grasp whose gripper and object normals are most nearly opposed.
Figure 5: A grasp recovered on the demonstrated mug follows semantic alignment to the handles of several cousin mugs.
Stage Four: Semantic Alignment Is More Than Retexturing
Each cousin begins with one or more photographs. SAM3 segments it, and SAM3D or MV-SAM3D generates a splat and textured mesh. The paper reports about seven minutes to generate ten cousins on one RTX 5090. Generated assets do not come with metric scale, so each object is resized to a height obtained from a product specification, a vision-language-model estimate, or a physical measurement.
Whole-object alignment alone is insufficient. A mug and coffee machine may differ substantially in height and silhouette. Bidirectional ICP can minimize overall surface distance while placing the tray, which is the only functionally relevant support, in the wrong location. The method therefore uses three complementary stages. Standard bidirectional ICP produces a coarse pose from several orientation candidates. DINOv3 features are then clustered, clusters are matched by cosine similarity, and ICP correspondences are restricted to matched semantic clusters. Finally, task-weighted ICP gives the functional region twenty times the weight of the rest of the object.
A vision-language model selects the key frame that defines the functional region. For placement, this is usually the settled goal state. A sphere of radius $r$ around the relevant Gaussians is expanded using DINOv3 similarity. For continuous motion such as pouring, SAM3 segments the mug rim and the region is transferred by semantic similarity to the target object. The highlighted regions in Figure 4 are not generic attention maps. They are the geometry that actually anchors alignment.
If $A$ is the rigid alignment from a cousin to the demonstrated object, the transferred trajectory is
$$T_t^{\mathrm{cousin}}=T_tA$$The equation is simple only after the alignment problem has been solved. Whole-shape alignment can map a handle to a body panel; a single-point constraint can push the cousin through the table. The twenty-times functional weight preserves the task relation, while a final vertical ground adjustment makes the starting object sit level on the table without changing the semantic alignment used at the goal.
Figure 4: Functional anchors include the machine tray, poured mug rim, mug handle and tree branch, and drill handle.
Stage Five: Scene Construction, Camera Calibration, and Robot Motion
The aligned assets must enter a simulated scene that shares a coordinate frame with the real robot. A Viser-based tool lets a user place the robot, table, and background board and enter measured offsets. The saved scene exports both to IsaacSim and MuJoCo. MuJoCo decomposes each textured mesh into convex collision hulls whose surfaces remain within three millimeters of the source geometry.
Camera pose is recovered without AprilTags. The robot itself is the calibration object. For simulated camera parameters $\theta$, the renderer produces a robot mask $M_\theta$, which is compared with the SAM3 mask $M^\ast$ from a real image:
$$\mathcal{L}_{\mathrm{cam}}(\theta)=1-\mathrm{IoU}(M_\theta,M^\ast)+\lambda_{\mathrm{rgb}}\mathcal{L}_{\mathrm{rgb}}+\lambda_{\mathrm{feat}}\mathcal{L}_{\mathrm{feat}}$$Silhouette IoU dominates because it is a strong global pose cue. Masked RGB and DINOv3 losses provide a finer signal when silhouette overlap stops improving. Because the simulator renderer is non-differentiable, the optimizer estimates a gradient with simultaneous perturbations, using two renders per step, and runs coarse-to-fine searches around the best candidate.
For a robot base pose $B$, the end-effector target is
$$B^{-1}T_tG$$A collision-free pre-grasp offset is inserted along the approach axis, and jaxmp solves inverse kinematics. Episodes that miss the target or self-collide are discarded. Failed attempts are still included in the reported generation time, so the two-hour figure is not based only on successful episodes.
Figure 6: One Viser scene exports to IsaacSim and MuJoCo. The real setup uses xArm7 and Franka Panda with RGB-only external cameras.
Episode Generation and the Policy Interface
Every episode samples a cousin pair and new initial poses from a uniform range on the table. The demonstrated trajectory is warped from the new start toward the transferred goal. The start offset decays to zero as arc length approaches the goal, which preserves the demonstrated functional motion at the critical moment. Table texture, lighting, skybox, and camera pose are randomized across episodes.
IsaacSim provides photorealistic RGB but carries the object kinematically with the gripper, so it does not simulate finger-object contact during transport. MuJoCo simulates settling, grasping, and release under gravity and retains an episode only when the final object pose is within tolerance, but its renderer is less realistic. The simulator comparison is therefore a tradeoff between visual and physical fidelity, not a claim that one simulator is universally better.
The data logger records RGB, end-effector pose from forward kinematics, and commanded gripper opening at 10 Hz. The policy is initialized from $\pi_{0.5}$ and fine-tuned with LoRA. It receives a $224\times224$ third-person RGB image and an 8-dimensional proprioceptive state containing end-effector position, axis-angle orientation, and gripper opening. It predicts a chunk of ten actions at 10 Hz. Each action has seven dimensions: position change, axis-angle rotation from the current to the next orientation, and a gripper command.
| Stage | Reported scale or cost | Interpretation |
|---|---|---|
| Human capture | About 1 minute | One video and one scan; excludes shared 30-minute robot setup |
| Preprocessing | About 25 minutes of GPU time | Reconstruction, tracking, cousin generation and alignment, motion planning |
| Generate 2,000 episodes | About 2 hours on one RTX 5090 | About 3.6 seconds per episode including discarded attempts |
| Teleoperation baseline | 150 episodes in about 2 operator hours | About 48 seconds per episode including resets |
| Ten cousins | About 7 minutes | SAM3D or MV-SAM3D asset generation |
The cost accounting makes the system’s scalability concrete. R2Sem2R is not free data. It replaces some operator time with GPU time and substantial engineering. Unlike teleoperation, however, its compute can be parallelized across GPUs and the same human capture can cover many objects and robots.
Real-Robot Evaluation
The experiments use a UFactory xArm7 and a Franka Emika Panda, each with its native gripper and an external RealSense D435i used only for RGB. The four tasks are placing a mug on a coffee machine, pouring a mug into a bowl, hanging a mug on a mug tree, and setting a drill upright. Each task is evaluated in 50 real trials per robot, except drill reorientation, which uses 40. Success is defined by the actual task outcome, including a level mug on the tray, a pour directed into the bowl, a handle resting on the tree, or an upright drill at the target orientation.
The teleoperation baseline contains 150 demonstrations across several object instances. R2R2R+ contains 2,000 generated episodes for the single demonstrated object pair. R2Sem2R contains 2,000 generated episodes spanning one reference object plus its cousins. The two generated conditions use the same source video and downstream tracking, grasp, simulation, and IK pipeline, so their difference isolates object diversity as much as possible.
| Robot | Task | Teleoperation, 150 demos | R2R2R+, one pair, 2,000 | R2Sem2R, cousins, 2,000 |
|---|---|---|---|---|
| xArm7 | Mug on coffee machine | 31/50 (62%) | 18/50 (36%) | 45/50 (90%) |
| xArm7 | Pour into bowl | 36/50 (72%) | 23/50 (46%) | 37/50 (74%) |
| xArm7 | Hang on mug tree | 29/50 (58%) | 10/50 (20%) | 31/50 (62%) |
| xArm7 | Drill reorient | 34/40 (85%) | 15/40 (38%) | 34/40 (85%) |
| Franka Panda | Mug on coffee machine | 30/50 (60%) | 20/50 (40%) | 34/50 (68%) |
| Franka Panda | Pour into bowl | 28/50 (56%) | 22/50 (44%) | 38/50 (76%) |
| Franka Panda | Hang on mug tree | 14/50 (28%) | 5/50 (10%) | 18/50 (36%) |
| Franka Panda | Drill reorient | 26/40 (65%) | 12/40 (30%) | 26/40 (65%) |
R2Sem2R exceeds R2R2R+ by 26 to 54 percentage points on every task and robot. Because the two data generators are otherwise matched, this gap is strong evidence that replacing geometry across semantically aligned cousins matters more than merely perturbing poses around one pair. R2Sem2R also matches or exceeds teleoperation on all eight task-robot combinations.
The drill task is especially informative. Both policies train on the same four drills, so the advantage cannot be explained by exposing the policy to entirely unseen test objects. The counterpart still samples poses around one original pair, whereas R2Sem2R changes grasp geometry across cousins. The result suggests that pose augmentation alone is insufficient for learning a robust grasp recovery behavior.
Figure 7: Four human demonstrations and corresponding real executions on an xArm7 and a Franka Panda.
What Actually Drives the Improvement
The data-scaling experiment separates quantity from object diversity. On xArm7 mug placement, a single-pair teleoperation policy remains around 10% to 12% from 50 to 150 demonstrations. R2R2R+ plateaus around 36% to 38% after about 660 episodes. R2Sem2R continues improving, reaches the best teleoperated policy at 1,320 episodes, and surpasses it at 2,000.
A generated episode is therefore worth less than a teleoperated episode at small data sizes. The important difference is that single-pair data has a ceiling: changing poses and appearances cannot create a new object distribution. Cousin data keeps introducing new shapes and functional regions, allowing the policy to learn a category-level relation rather than a particular asset.
The simulator comparison is counterintuitive. With the same 2,000 episodes for xArm7 placement, IsaacSim yields 45/50 while MuJoCo yields 31/50, matching teleoperation. For this RGB-only VLA, visual realism appears more valuable than contact simulation. MuJoCo models contact more faithfully, but the domain gap in rendered images offsets that advantage. The paper therefore treats the simulators as complementary rather than assuming that physics simulation must produce the better training data.
| Tracking method | Mean mask IoU on paper videos | YCBInEOAT ADD-S AUC | YCBInEOAT ADD AUC |
|---|---|---|---|
| R2Sem2R tracker | 0.913 | 51.4 | 41.3 |
| WARPED | 0.681 | 44.0 | 25.1 |
| RSRD | 0.415 | 5.1 | 1.9 |
| FoundationPose, per-video calibrated depth | 0.823 | 30.3 | 22.8 |
| FoundationPose, raw estimated depth | 0.629 | 3.0 | 1.5 |
The tracking table also exposes two failure modes. RSRD collapses on several texture-poor or heavily occluded objects. FoundationPose is highly sensitive to depth calibration: its mask IoU improves from 0.629 to 0.823 when depth is calibrated per video. R2Sem2R still needs a scan and a measured scale, but it does not require a reliable metric-depth sensor for every frame. That is the practical contribution of the render-and-compare design.
Limitations and Open Risks
The authors explicitly require a metric measurement of the demonstrated object. Monocular depth is not suitable for recovering absolute scale. This requirement sounds minor when the object is a mug, but it has broad consequences: an incorrect measurement shifts table height, grasp depth, and every robot trajectory. The method is not fully “video in, data out” because the phone scan and measured height remain part of the input contract.
IsaacSim’s kinematic carry omits finger-object contact. This limits contact-rich manipulation, including tasks that depend on force control, sliding, or precise insertion. MuJoCo can model contact and settling, but the paper’s visual-domain gap reduces real-robot performance. A future system must combine high-fidelity rendering with faithful contact rather than choosing one and sacrificing the other.
The pipeline handles rigid objects only. Soft and deformable objects are outside scope. Articulated objects such as drawers and doors are listed as future work because a single rigid transform cannot represent their internal degrees of freedom. The authors suggest recovering a screw axis from the demonstration and transferring it through semantic alignment, but that extension is not implemented or evaluated.
Grasp recovery assumes that the video contains a moment when the hand stably holds and lifts the object. Pure sliding, pushing, tool use, or two-handed manipulation may not provide the required rigid pinch. Cousin generation also needs at least one real or multi-view photograph. Generating an entire category from text while preserving actionable geometry remains future work.
A separate concern is functional-region selection. The system delegates key-frame and region identification to a vision-language model and feature similarity, but the paper does not report a systematic error rate for this step. The approach is likely reliable for common structures such as coffee-machine trays and mug rims. For rare objects or ambiguous task language, the twenty-times weighting can amplify a mistaken region. A production system should expose a confidence score and allow human verification.
Finally, the reported success rates come from different task and embodiment conditions and should not be collapsed into one unconditional score. Drill trials use 40 attempts, while other tasks use 50, and the visual and motion difficulty varies substantially. The paper’s strongest claim is not that one percentage transfers everywhere; it is that the same single-video pipeline produces consistent gains across four tasks, two object categories, and two robots.
Conclusion
R2Sem2Real moves human-video learning from 2D re-rendering toward 3D semantic augmentation. It does not merely create more videos around one object. It recovers object trajectories, establishes functional correspondences, substitutes genuinely different geometry, and converts the result into executable grasps and arm motion. Data diversity comes from task semantics rather than generic visual noise.
The most useful lesson for robot developers is that same-object trajectory scaling saturates quickly. Semantically aligned cousin instances are a stronger source of generalization than pose augmentation alone. The cost is a nontrivial scan, scale, alignment, and simulation pipeline. The benefit is that one human capture can become a cross-instance, cross-embodiment training set.
The work also gives a practical definition of useful generated data. Photorealism is not enough. The pipeline must preserve functional relations, satisfy execution constraints, and match the visual and physical domains used by the policy. R2Sem2Real’s contribution is the integration of those requirements into a repeatable data-production process.
Key takeaway: scalable demonstration augmentation is not about generating more trajectories around one object; it is about preserving the same functional relation across many objects.
