PAPER DEEP DIVE
StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception
StereoPolicy learns task-conditioned binocular representations directly from synchronized stereo pairs instead of reconstructing explicit depth or point clouds. A Stereo Transformer alternates within-image and cross-image attention over shared 2D features, and the fused representation integrates with diffusion policies, pi0.5, and GR00T-N1.5. Experiments across simulation and real tabletop and bimanual mobile manipulation show consistent gains over RGB, RGB-D, point-cloud, and multi-view baselines, together with practical guidance on stereo baseline and working distance.
Paper Metadata
Title: StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception
Authors: Evans Han, Yunfan Jiang, Yingke Wang, Haoyue Xiao, Huang Huang, Jianwen Xie, Jiajun Wu, Li Fei-Fei, and Ruohan Zhang
Affiliations: Northwestern University, Stanford University, and Lambda, Inc.
Links: arXiv 2605.09989; project page.
Code status: At the time of this review, the project page still labels code as “soon.” Its public GitHub organization contains website assets but no public training code, model implementation, or data-processing pipeline. A third-party Hugging Face repository distributes reproduction weights, but it does not provide a complete auditable implementation. It should therefore not be described as the official code release.
One-sentence summary: StereoPolicy feeds synchronized left-right image pairs directly into visuomotor policies and learns task-conditioned binocular features with cross-attention, improving manipulation without first reconstructing a depth map or point cloud.
Research Context and Motivation
Robot imitation learning has shifted manipulation from hand-engineered state machines toward policies learned from demonstrations. Diffusion policies can represent multimodal action sequences, while vision-language-action models combine visual observations, language instructions, and robot state to generalize across tasks. These systems have become substantially more capable, but the dominant visual interface remains monocular RGB or a collection of independently processed monocular cameras. Such observations provide semantic appearance and broad scene context, yet they do not explicitly encode the metric displacement between a gripper and a narrow slot, a handle, or the surface of an object.
The most common remedy is to add a depth sensor. RGB-D cameras and point-cloud policies can succeed in controlled environments, but their quality depends on material properties, calibration, synchronization, and sensor noise. Transparent cups, shiny metal, thin edges, and small reflective surfaces are especially difficult. A depth image may appear plausible in aggregate while erasing precisely the object boundary that the controller needs. A point cloud may contain a hollow shell, a displaced surface, or no points on the object at all. The resulting geometric representation can therefore be less reliable than the RGB image that a human would inspect.
Stereo vision offers another source of three-dimensional structure. A calibrated pair of cameras observes the same scene from slightly different positions, and the displacement of corresponding pixels constrains relative depth. Classical and learned stereo systems have exploited this signal for decades. In robotics, however, stereo pairs are often used only as the first stage of an explicit reconstruction pipeline: estimate disparity, convert it to depth, back-project to a point cloud, and pass the result to a separately designed policy. This decomposition creates both representation loss and latency.
StereoPolicy asks whether the policy can consume the pair directly and learn which correspondences matter for control. The distinction is important. A generic stereo network is optimized to produce an accurate depth map over the whole image. A manipulation policy needs only enough spatial evidence to align a gripper, distinguish an opening from its background, or determine whether a cup handle is in front of a rack peg. These requirements are task-conditioned and may tolerate errors in regions irrelevant to the action. By training the fusion module through the action objective, the authors turn stereo perception into an internal representation rather than an intermediate artifact that must be visually correct.
The paper organizes its investigation around four questions. First, does direct stereo input outperform monocular RGB, RGB-D, point-cloud, and matched multi-view baselines under comparable training conditions? Second, can the same fusion design improve pretrained vision-language-action models instead of only policies trained from scratch? Third, how do hardware variables such as baseline, working distance, and camera angle affect the result? Fourth, can a general-purpose pretrained stereo model replace task-specific fusion? These questions separate a simple input ablation from a broader claim about representation learning for embodied control.
The motivating hypothesis is that binocular correspondence should be selected by the action loss. If the fusion is trained only for reconstruction, the representation may devote capacity to background surfaces that are easy to measure but irrelevant to grasping. If it is trained only as a monocular semantic encoder, the policy never receives a direct signal comparing the two viewpoints. Alternating within-image and cross-image attention creates an architectural path for both semantic context and relative geometry to enter the controller.
Preliminaries and Problem Setting
At time t, each camera view provides a synchronized pair of images. Let the left and right observations for view i be ILt,i and IRt,i. The policy may receive N such pairs, for example from an external camera and a wrist-mounted camera. These pairs differ from arbitrary multi-view images because each left-right pair has a fixed baseline and an approximately simultaneous exposure. The two pair members can be matched at the pixel level, while different camera pairs observe the robot from different spatial scales and may require separate contextual treatment.
A visuomotor policy maps observations and robot state to a sequence of future actions. For a horizon H, the target is an action chunk containing commands for the next H control steps. Diffusion policies model this chunk as a denoising problem, whereas VLA models predict actions through a pretrained multimodal backbone. StereoPolicy is deliberately designed as a visual representation replacement: it changes how left-right features are formed, but it preserves the action head and training objective of the host policy.
Evaluation uses task success rate. The real tabletop experiments execute each policy 20 times per task with randomized initial conditions. Simulation results aggregate three seeds, and the RoboCasa evaluation computes a macro-average over 24 kitchen tasks so that each task contributes equally. The paper also reports end-to-end latency, parameter count, compute, and peak memory because a perception method that improves success but misses the control cycle is not a complete deployment solution.
Method
1. Shared feature extraction for each image
StereoPolicy first processes every image independently with a ResNet18 feature pyramid network. The left and right branches share weights. This sharing is a geometric prior, not a minor parameter-saving choice: a local patch that depicts the same material or object edge should be encoded in the same feature space regardless of whether it came from the left or right camera. The network can then compare the resulting tokens for appearance and position rather than learning separate vocabularies for the two branches.
The feature pyramid matters because manipulation depends on both broad object identity and fine boundaries. Deep features provide context for recognizing a cup, toaster, or radio. Higher-resolution features retain the thin edge of a handle and the narrow gap of a slot. In the component ablation, removing the FPN lowers the average success rate on ToolHang, Square, and PourWater from 84.0% to 74.0%. This is a larger drop than removing the frozen semantic prior, indicating that stereo fusion needs multi-scale locality.
2. Frozen DINOv2 as an external-view semantic anchor
External camera views receive an additional frozen DINOv2 feature. The feature is downsampled, projected, and concatenated with the convolutional representation. Because DINOv2 is not fine-tuned, it acts as a stable semantic anchor rather than another task-specific encoder. It can help identify object parts when texture repetition or weak contrast makes local stereo matching ambiguous. The projection step aligns the semantic features with the convolutional token dimension:
$$ \tilde f_{t,i}^{\cdot} = \phi\left([f_{t,i}^{\cdot}\|f_{t,i}^{\text{DINO}}]\right) $$Here, ft,i· is the convolutional feature for view i, fDINO is the frozen DINOv2 feature, and φ is the learned dimension alignment. The double bar denotes channel concatenation.
The placement of DINOv2 is part of the finding. The authors apply it to external views but not to wrist views. When they instead add it to wrist observations, average success falls from the full configuration’s 84.0% to 79.3%. They attribute this to the distribution gap between egocentric wrist images and the data used to pretrain DINOv2. The conclusion is not that foundation features are universally harmful, but that a semantic prior should be injected only where its training distribution and the robot’s observation statistics align.
3. Alternating within-image and cross-image attention
The central module is a two-layer Stereo Transformer with eight attention heads. Each layer performs self-attention within the left tokens and within the right tokens, followed by cross-attention between the two images. The cross-attention queries and keys receive two-dimensional rotary position encoding based on pixel coordinates. This positional signal encourages the network to compare tokens at plausible image locations while preserving sensitivity to appearance.
Concatenating all left and right tokens into one long sequence would also permit attention across viewpoints, but it would not explicitly separate the two operations. StereoPolicy’s alternating design gives within-image self-attention the job of building local and object-level context, then uses cross-image attention to search for a corresponding region in the other view. The resulting feature for one view can be written as
$$ \forall i\in\{1,\dots,N\}:\quad z_{t,i} := \operatorname{StereoTrans}_{i} \left(\tilde f_{t,i}^{L},\tilde f_{t,i}^{R}\right) \in \mathbb{R}^{d_z} $$Each camera pair produces a 128-dimensional view feature. Pair modules follow the same structure, but external and wrist views are not forced into one shared spatial scale. This is important because a wrist camera may see the gripper and target at centimeters while an external camera sees the robot, table, and background at meters.
The matched fusion experiment provides evidence that the alternating pattern is useful. On ToolHang, Stereo Transformer reaches 94% with true left-right images, while joint attention over a concatenated sequence reaches 82%. On Square, the two methods reach 88% and 82%, respectively. When both branches receive the same left image, Stereo Transformer falls to 62% on ToolHang. The second view therefore contributes real information, and the architecture is better able to exploit it than a generic joint-attention baseline.
4. Multi-view aggregation and robot state
After each stereo pair is fused, StereoPolicy concatenates the view features with the low-dimensional robot state. The state anchors relative visual geometry to the current configuration of the arm and gripper. The observation vector used by the policy is
$$ z_{\text{obs}} = \left[z_{t,1}\|z_{t,2}\|\cdots\|z_{t,N}\|s_t\right] $$where st is the proprioceptive state and zobs is the final observation representation. The vector no longer corresponds to a human-readable depth image. It is a learned summary of which binocular correspondences and semantic regions are useful for the host policy.
5. Integration with diffusion policy
For StereoPolicy-DP, the policy predicts a continuous action chunk of length H:
$$ \mathbf a_{t:t+H-1} = [\mathbf a_t,\mathbf a_{t+1},\ldots,\mathbf a_{t+H-1}] $$During training, Gaussian noise is added to a demonstration action chunk. The denoising network receives the noisy chunk, the diffusion step, and the stereo observation. Its objective is to recover the injected noise:
$$ \mathcal L_{\mathrm{DP}} = \mathbb E\left[ \left\| \epsilon-\epsilon_\theta \left(x^{(\tau)},\tau,z_{\text{obs}}\right) \right\|_2^2 \right] $$Here, ε is Gaussian noise, x(τ) is the action chunk corrupted at diffusion step τ, and εθ is the denoising network. The vision encoder, Stereo Transformer, and policy backbone are optimized jointly through this action loss. DINOv2 remains frozen. No depth labels, disparity labels, or reconstruction loss are required. At inference, the policy starts from random noise and iteratively produces an action sequence.
6. Integration with pretrained VLA models
For StereoPolicy-VLA, the authors preserve the pretrained vision-language-action backbone and replace its monocular visual embedding with the fused stereo representation. The backbone’s language pathway and action head remain available, so the controller can condition on both the instruction and the robot state:
$$ \mathbf a = \pi_\theta(z_{\text{obs}},x_{\text{lang}},s) $$The paper evaluates this idea on π0.5 and GR00T-N1.5. A critical baseline sends the backbone both images through its native multi-view input. That version receives the same pixels as StereoPolicy but lacks the trainable cross-image fusion module. On RoboCasa, the native two-image model improves only slightly over the monocular model, while StereoPolicy produces a larger and more consistent gain. Extra pixels alone are therefore not enough; the pretrained backbone must be given a representation that makes the binocular relationship explicit.
7. Do the learned features contain measurable geometry?
The authors test whether action-trained stereo features preserve spatial information by freezing the visual representation after policy training and fitting a linear probe. The probe predicts the relative depth between the gripper and target in camera coordinates. Training and test splits are separated by complete trajectories and spatial layouts so that the probe cannot succeed by memorizing a trajectory-specific shortcut. On ToolHang, mean absolute error falls from 3.1 centimeters before stereo training to 1.1 centimeters after training. On PourWater, it falls from 4.0 centimeters to 1.5 centimeters.
This result links action success to an independent geometric measurement. The module does not output a depth map, yet its features become linearly predictive of relative depth. A control that feeds the same left image into both branches does not reproduce the gain. The evidence supports the interpretation that cross-view training creates useful spatial structure rather than merely increasing model capacity.
8. Baseline-to-distance ratio and deployment geometry
Stereo performance depends on more than the presence of two cameras. A pair with a tiny baseline at a long working distance produces weak disparity. A very large baseline at a close distance can reduce overlap and create strong occlusion or perspective changes. The paper argues that the relevant design variable is the ratio
$$ r= \frac{\text{stereo baseline}} {\text{camera-object distance}} $$The hardware study sweeps baselines from 2 to 10 centimeters and working distances from roughly 0.6 to 1.0 meter. The strongest region is approximately r between 0.09 and 0.13, or roughly a 10% baseline-to-distance ratio. For a typical tabletop setup, a 6-centimeter baseline at 0.6 to 0.8 meter is a useful starting point. The paper also holds a policy trained with a 6-centimeter baseline fixed and changes only the test-time baseline. Success falls from 94% to 86% at 4 centimeters and to 88% at 8 centimeters, showing moderate tolerance but confirming that matching the training geometry remains best.
flowchart LR
A[Left RGB] --> B[Shared ResNet18 FPN]
C[Right RGB] --> B
A --> D[Frozen DINOv2 for external view]
C --> D
B --> E[Concatenate and project features]
D --> E
E --> F[Within-image self-attention]
F --> G[Cross-image attention with 2D RoPE]
G --> H[Two-layer Stereo Transformer]
H --> I[128-dimensional feature per view]
I --> J[Concatenate all views and robot state]
J --> K[Diffusion policy denoiser]
J --> L[Pi 0.5 or GR00T-N1.5 action head]
The diagram follows the paper’s actual data path rather than a generic encoder-policy sketch. Left and right observations are paired before multi-view aggregation, cross-image attention is explicitly separated from within-image attention, and the resulting representation is shared across two policy families.
9. Compute and end-to-end latency
The full model has approximately 112 million parameters, including 10 million in the Stereo Transformer. The paper reports 128 GFLOPs, 1.4 GB peak memory, and latency measured on one H100 GPU. Profiling uses batch size one, an observation horizon of two, a prediction horizon of sixteen, seven-dimensional actions, and 100 denoising steps. Median end-to-end latency is 698 milliseconds, with a 95th percentile of 736 milliseconds.
Stereo fusion adds about 3 milliseconds relative to a joint-attention control. The complete StereoPolicy pipeline is 310 milliseconds faster than FoundationStereo combined with RGBD-3DDA and 79 milliseconds faster than Fast FoundationStereo with 3DDA. These comparisons change multiple parts of the perception-control stack at once, so they should not be read as isolating the cost of a single attention layer. They do, however, measure the practical behavior of the complete pipeline used in the experiments.
Experimental Setup and Main Results
The real evaluation spans seven tasks. Five tabletop tasks use a 7-DoF Franka arm and 200 demonstrations per task: Banana Pick-and-Place, Toast Insert, Plastic Cup Hang, Steel Cup Hang, and Glass Cup Hang. The last three vary only the cup material, creating a controlled stress test for depth sensing and visual matching. Two bimanual mobile manipulation tasks use the 24-DoF Galaxea R1 Pro with 75 demonstrations each. One requires approaching a shelf, grasping a radio, inserting the gripper into a handle, and pressing a button. The other requires bimanual toast transfer and placement.
The simulation suite contains three benchmarks. Robomimic includes ToolHang, Square, and Transport. RoboCasa evaluates π0.5 and GR00T-N1.5 on 24 kitchen tasks with 30, 100, and 300 demonstrations. OmniGibson adds four customized long-horizon household tasks. Where possible, diffusion-policy comparisons share the visual backbone, image resolution, training budget, and optimization settings. This matching is necessary because a stereo system that also receives a larger policy or a longer schedule would not establish the benefit of binocular perception.
Real tabletop manipulation
The real-world tabletop result gives the clearest evidence that the second view must be used in the right way. StereoPolicy-DP reaches a five-task average of 59%. Monocular RGB reaches 43%, RGB-D reaches 41%, RGBD-3DDA reaches 45%, and the matched multi-view RGB baseline also reaches 45%. Point-cloud policies perform substantially worse: PointNet averages 14% and DP3 averages 27%. The multi-view comparison is especially informative because it receives more than one camera but does not perform paired cross-attention.
| Method | Banana Pick | Toast Insert | Plastic Cup Hang | Steel Cup Hang | Glass Cup Hang | Average |
|---|---|---|---|---|---|---|
| RGB | 12/20 | 8/20 | 12/20 | 10/20 | 1/20 | 43% |
| RGB-D | 14/20 | 8/20 | 11/20 | 8/20 | 0/20 | 41% |
| RGBD-3DDA | 13/20 | 9/20 | 13/20 | 10/20 | 0/20 | 45% |
| Point cloud with PointNet | 7/20 | 0/20 | 5/20 | 2/20 | 0/20 | 14% |
| Point cloud with DP3 | 11/20 | 3/20 | 8/20 | 5/20 | 0/20 | 27% |
| Multi-view RGB | 13/20 | 9/20 | 13/20 | 9/20 | 1/20 | 45% |
| StereoPolicy-DP | 16/20 | 12/20 | 15/20 | 13/20 | 3/20 | 59% |
The gain is not confined to transparent objects. Toast Insert improves from 8/20 to 12/20, which is consistent with better depth alignment at a narrow slot. Steel Cup Hang improves from 10/20 to 13/20, where reflective surfaces can corrupt explicit depth but the appearance of the handle remains available to stereo matching. Glass Cup Hang improves from 1/20 to 3/20, yet 3/20 is still a low absolute success rate. The paper is transparent about this failure case rather than presenting stereo as a solution to every difficult material.
The poor point-cloud result is not explained by an obviously undersampled representation. The authors fuse calibrated RGB-D observations into a common frame, crop to the workspace, and use farthest-point sampling to retain 4,096 points. Increasing the count to 10,000 does not improve success in the evaluated setting and makes inference more expensive. The authors therefore argue that point density is not the central bottleneck. Missing or deformed task-relevant geometry during reconstruction is a more plausible explanation.
Simulation and data efficiency
The diffusion-policy simulation results continue the same pattern. On OmniGibson Strawberry with 100 demonstrations, RGB reaches 62% and StereoPolicy reaches 82%. PourWater rises from 86% to 100%. On Robomimic ToolHang, StereoPolicy reaches 94% with 100 demonstrations and 96% with 200. Square rises from 88% with 100 demonstrations to 100% with 200. The low-data settings are particularly important because real robot datasets are expensive and a representation that reduces demonstration requirements has direct engineering value.
The paper also contains counterexamples that bound the claim. On TurnOnRadio with 100 demonstrations, RGBD-3DDA reaches 100% while StereoPolicy reaches 57%. On Transport with 100 and 200 demonstrations, RGB-D reaches 94% and StereoPolicy reaches 94% and 96%, respectively. Stereo is therefore not universally superior. Explicit depth can be extremely effective when the simulator and task produce clean geometry. The defensible conclusion is that direct stereo is stronger across most tested tasks and improves data efficiency, not that it dominates every possible sensing modality.
Gains on pretrained vision-language-action models
The 24-task RoboCasa macro-average tests whether StereoPolicy transfers beyond a diffusion policy trained for each task. With π0.5, the monocular success rates at 30, 100, and 300 demonstrations are 48.71%, 67.64%, and 70.31%. Native two-image input gives 49.01%, 68.24%, and 70.81%. StereoPolicy gives 51.72%, 71.54%, and 74.40%. With GR00T-N1.5, the monocular results are 44.98%, 63.58%, and 64.30%; native two-image input gives 45.67%, 65.24%, and 65.31%; StereoPolicy gives 47.12%, 66.17%, and 67.50%.
| Backbone | Visual input | 30 demos | 100 demos | 300 demos |
|---|---|---|---|---|
| π0.5 | Monocular | 48.71 | 67.64 | 70.31 |
| π0.5 | Native two-image | 49.01 | 68.24 | 70.81 |
| π0.5 | StereoPolicy | 51.72 | 71.54 | 74.40 |
| GR00T-N1.5 | Monocular | 44.98 | 63.58 | 64.30 |
| GR00T-N1.5 | Native two-image | 45.67 | 65.24 | 65.31 |
| GR00T-N1.5 | StereoPolicy | 47.12 | 66.17 | 67.50 |
These are moderate but consistent improvements. Their force comes from the comparison with the native two-image model: simply exposing a pretrained backbone to both camera images yields little benefit, while task-trained cross-image fusion improves both backbones at all three data scales. The result suggests that VLA pretraining supplies useful semantic and language grounding but does not automatically learn to exploit binocular displacement.
The mobile manipulation tasks expose where monocular policies fail. In TurnOnRadio, a VLA policy may approach the radio correctly but miss the handle or the button. In Toast PnP, it may locate the object but misalign the gripper during the transfer or final placement. These failures occur in the precise interaction phase, where semantic recognition is already solved and relative geometry remains uncertain. StereoPolicy improves the external-camera estimate of gripper-target displacement and therefore makes the final alignment more reliable.
Ablations and Comparison with Pretrained Stereo Models
The strongest component control does not compare stereo with fewer inputs; it compares true stereo with a duplicated left image. Full StereoPolicy averages 84.0% across ToolHang, Square, and PourWater. Feeding the same left image into both branches lowers the average to 62.7%. This removes the geometric difference between views while keeping the model, fusion layer, and number of image tokens. The large decrease shows that the gain does not come merely from extra computation.
Removing the Stereo Transformer lowers the average to 70.0%; removing the FPN gives 74.0%; removing DINOv2 gives 76.0%; and applying DINOv2 to the wrist view gives 79.3%. The ranking identifies three different roles. Cross-image fusion is the primary mechanism, multi-scale convolutional features preserve the boundaries needed for matching and control, and external-view DINOv2 adds semantic context without replacing task-specific stereo learning.
| Configuration | ToolHang-100 | Square-100 | PourWater-200 | Average |
|---|---|---|---|---|
| StereoPolicy-DP | 94 | 88 | 70 | 84.0 |
| Same image in left and right branches | 62 | 76 | 50 | 62.7 |
| Without Stereo Transformer | 82 | 76 | 52 | 70.0 |
| Without FPN | 78 | 80 | 64 | 74.0 |
| Without DINOv2 | 84 | 82 | 62 | 76.0 |
| DINOv2 on wrist view | 88 | 84 | 66 | 79.3 |
The authors also compare their task-trained fusion with frozen features from a general stereo foundation model. On ToolHang, StereoPolicy reaches 94% with 698-millisecond latency. Frozen FoundationStereo features reach 76% with 740 milliseconds. Using FoundationStereo depth in an RGB-D policy reaches 44% with 995 milliseconds, and using it with RGBD-3DDA reaches 28% with 1,008 milliseconds. The Fast FoundationStereo depth variants reach 38% and 22%, respectively, while remaining slower than StereoPolicy. On real Banana Pick, FoundationStereo plus RGB-D succeeds in 1/20 trials versus 16/20 for StereoPolicy.
This experiment is not a comprehensive indictment of learned stereo in all robotic settings. It uses one benchmark, one task-trained policy family, and specific foundation-model integrations. It does show that a generic reconstruction objective does not necessarily produce the representation a controller needs. The foundation model may estimate depth accurately in regions that dominate its training loss while failing to preserve the small object boundary that determines a grasp. The latency of the explicit depth pipeline further reduces the time available for correcting an error.
Limitations and Independent Assessment
The first limitation is transparent-object manipulation. Glass Cup Hang succeeds in only 3 of 20 trials. This is better than the tested baselines, but it remains a low absolute success rate. Refraction, specular highlights, and weak boundaries can make left-right correspondence ambiguous even when no explicit depth reconstruction is attempted. The paper directly acknowledges this result. Stereo removes one fragile stage from the pipeline, but it cannot manufacture geometric evidence that is absent from both images.
The second limitation is hardware coverage. The real experiments use Franka and Galaxea R1 Pro platforms, while the broad baseline-to-distance sweep is conducted in simulation. Real stereo cameras introduce rolling-shutter effects, exposure differences, mechanical vibration, calibration drift, and synchronization errors that are not systematically varied in the reported experiments. The recommended ratio between 0.09 and 0.13 is a useful design prior, but it should not be treated as a universal constant across robot scales and camera fields of view.
The third limitation is latency. The 698-millisecond median and 736-millisecond 95th percentile include 100 diffusion denoising steps. A robot may execute action chunks asynchronously, but high-frequency closed-loop control still requires careful scheduling or a faster action generator. The Stereo Transformer itself adds little overhead relative to the complete denoising pipeline, yet the entire system is measured on an H100. Deployment on an edge computer with lower memory bandwidth needs separate profiling.
The fourth limitation is reproducibility. No official implementation is presently public, and the project page still says code is coming soon. The paper describes the backbone, token dimension, transformer depth, attention heads, image processing, training budgets, and hardware, but real reproduction also requires synchronized capture, rectification, camera calibration, action-space alignment, data augmentation, and robot-specific deployment details. Third-party weights are useful evidence of activity, not a substitute for an auditable official release.
From an independent perspective, the paper’s strongest contribution is the experimental elimination of weak explanations. The gain is not just “more cameras,” because a matched multi-view baseline is lower. It is not just more parameters, because duplicating the left image collapses performance. It is not just a foundation model, because frozen FoundationStereo features trail the task-trained fusion. It is also not merely a VLA artifact, because the same trend appears in diffusion policies and a linear depth probe. The evidence forms a coherent chain from architecture to representation to control.
The VLA gains should still be interpreted carefully. At 100 demonstrations, π0.5 improves from 67.64% to 71.54%, and GR00T-N1.5 improves from 63.58% to 66.17%. These are meaningful and consistent, but the margins are much smaller than the real-tabletop gap between RGB and stereo. Practical deployment should validate the method on the target camera geometry and object distribution. A promising next step would combine stereo with tactile or force feedback, use uncertainty to decide when stereo should be trusted, and actively select camera viewpoints instead of treating the pair as fixed.
Conclusion and Outlook
StereoPolicy turns a change in visual input into a representation-learning problem. The left and right images are not converted into a depth map. They are processed by a shared 2D encoder, fused through alternating within-image and cross-image attention, concatenated with robot state, and passed to a diffusion or VLA policy. The action objective trains the fusion to retain the spatial relationships useful for manipulation rather than the depth values that best reconstruct every visible surface.
The empirical case is broad. Real tabletop success rises from 43% for monocular RGB to 59% for StereoPolicy-DP. Low-data simulation tasks and 24-task RoboCasa macro-averages show consistent gains. Ablations demonstrate the importance of true left-right input, the Stereo Transformer, multi-scale features, and selective external-view semantics. A linear probe shows that the learned representation becomes more predictive of relative depth, even though no depth output is trained. The paper also reports failures and counterexamples, including transparent objects and tasks where RGB-D remains stronger.
The engineering conclusion is to design stereo sensing around both the task and the hardware geometry. A baseline near one tenth of the working distance is a defensible starting point, and training and deployment should use matching configurations when possible. For ordinary tabletop grasping, insertion, and mobile manipulation, direct stereo fusion offers a lighter and more robust alternative to an explicit 3D reconstruction pipeline while remaining more spatially informative than monocular RGB.
The important output of a stereo policy is not a depth map; it is a task-conditioned representation of which left-right correspondences the action depends on.
SOURCE LINKS
