PAPER DEEP DIVE
Vision as Proprioception: Estimating Humanoid Robot Pose from Video
HumanoidPose combines frozen human-pose features with robot kinematics to recover full-body joint configurations and world-space motion from monocular video across embodiments.
Paper Deep Dive
Authors: Jiaxi Jiang, Matthias Heyrman, René Zurbrügg, Marco Hutter, Christian Holz
Affiliation: ETH Zürich
Paper: https://arxiv.org/abs/2610.25288
Project page: https://siplab.org/projects/HumanoidPose
Code: https://github.com/eth-siplab/HumanoidPose. As of October 23, 2026, the repository contains a README, license, and media assets, but its code and models are marked as coming soon. This review therefore separates claims supported by the paper from implementation details that cannot yet be inspected.
One-Sentence Summary
HumanoidPose treats an ordinary monocular video as an external proprioceptive signal: synthetic renders teach a kinematics-conditioned decoder to recover robot joint angles and root pose, online-video self-training adapts the detector to real appearances, and camera-motion composition turns the result into a world-space trajectory.
Why Robot Pose Has to Come from the Outside
A humanoid normally knows its configuration through encoders, an inertial measurement unit, and a software interface tied to its control stack. Those signals are usually unavailable to an outside observer. A researcher may be watching a robot from another laboratory, a camera may record a demonstration before any state channel is exposed, or a policy evaluation may intentionally withhold internal data. A person can still infer much of the motion by looking at the robot, which raises the central question of this paper: can a vision system recover the same mechanical state from pixels?
The requested output is more specific than a human mesh or a collection of image landmarks. It must contain the controlled joint coordinates of the observed robot, its root transform in the camera frame, and a short trajectory in world coordinates. Those values have to respect the robot's link tree, joint order, joint limits, and axis conventions. Predicting a plausible skeleton is insufficient when the result is meant to enter a controller, a motion-analysis tool, or a robot-learning dataset.
Monocular geometry makes this difficult. Depth is underconstrained, and a thin limb can change apparent length under perspective. Occlusion can remove evidence for a wrist or ankle for several frames. Humanoids add another source of variation: a small research platform may have 19 driven joints, while a larger machine may have 43, with different proportions, axes, meshes, and mechanical stops. Training a separate network for every robot would repeat both annotation and optimization work as the fleet grows.
Existing tools solve neighboring problems rather than this one. Articulated-object trackers often assume a known object and strong temporal continuity. Two-dimensional humanoid keypoint detectors locate visible body points but do not determine three-dimensional joint angles. Human mesh recovery contains rich body priors, but human bones are not robot bones. A pipeline that predicts a human and then retargets it to a robot adds an avoidable intermediate representation and propagates its errors into the controller coordinates.
The data problem is equally restrictive. Real supervision would ideally pair thousands of synchronized videos with encoder logs across viewpoints, motions, and appearances. That scale is impractical to collect for each embodiment. The paper instead builds paired data in simulation by retargeting motion-capture sequences to many robot models and rendering them with randomized scenes. Real video is still necessary, but it is used mainly to close the appearance gap rather than to obtain 3D pose labels for every frame.
Task Definition and System Architecture
For a video containing frames $I_t$, the model receives the target robot's MJCF or URDF description. If the robot has $n$ actuated joints, it predicts $\hat{\mathbf{q}}_t\in\mathbb{R}^{n}$ and the transform $\hat{\mathbf{T}}_{C\leftarrow R,t}$ from the robot root frame $R$ to the camera frame $C$. The kinematic model is part of the input, not a hidden choice permanently encoded into one output layer. This is what allows one set of weights to serve multiple robot types.
The pipeline has three cooperating stages. A synthetic data stage converts human motion into labeled robot motion under randomized rendering. A detection stage starts with rendered robots and then adapts to real internet footage through self-training. A pose stage combines a frozen human-pretrained visual encoder with joint queries derived from the robot description. Finally, a separate camera-motion path converts per-frame root predictions into world-space movement.
Figure 1: Paper overview. Synthetic supervision supplies robot state, self-training adapts the detector, and the shared decoder predicts body configuration, root pose, and world motion.
The architectural choice matters for more than parameter sharing. The human-pretrained encoder supplies a visual prior about bodies, but the output remains in robot space. The network never needs to generate a detailed human surface and then solve a second retargeting problem. The ablation later shows that removing this direct robot path causes the largest degradation, which is strong evidence that the design distinction is substantive rather than cosmetic.
flowchart LR
V[Monocular video] --> D[YOLO11 robot detector]
D --> C[Square robot crop]
C --> E[Frozen human-pretrained encoder]
M[Joint limits and tree] --> J[27D joint descriptor]
J --> Q[Shared joint queries]
E --> T[32 by 32 image tokens]
Q --> P[Eight-head shared decoder]
T --> P
P --> QP[Joint angles and camera-frame root]
V --> S[DROID-SLAM camera path]
D --> S
S --> W[World-space composition]
QP --> W
W --> O[Robot trajectory in world]
Synthetic Supervision across Sixteen Embodiments
AMASS motions are retargeted with GMR to sixteen humanoid models. Their standing heights span 0.60 to 1.78 meters and their controlled degrees of freedom span 19 to 43. The resulting corpus contains 128,000 images rendered in Unreal Engine with randomized scenes, lighting, camera viewpoints, and robot appearance. Each frame records the exact robot configuration, camera parameters, root pose, and projected landmarks used to generate it.
This construction avoids a common weakness of image-labeled datasets: the target is not inferred after rendering by an imperfect annotator. Joint values are part of the simulator state, and forward kinematics can evaluate the same configuration that produced the image. The supervision therefore connects pixels, joint-space control, and body-space geometry without an intermediate human parameterization.
Figure 2: The corpus spans sixteen humanoids with roughly a threefold height range and 19 to 43 actuated joints. The robot description remains available for every rendered image.
Variation across the sixteen systems is useful only if the network can interpret it through a common interface. The paper obtains that interface from joint axes, anchor points, link-tree depth, limits, and robot scale. A single learned map from those quantities to joint queries lets the decoder see the meaning of each output slot instead of memorizing a fixed list of joint indices.
Rendering does not eliminate the reality gap. Real cameras add motion blur, rolling-shutter artifacts, sensor noise, and unfamiliar backgrounds. Materials and mechanical tolerances also differ from simulation. The paper addresses the visible-object mismatch through detector adaptation, while relying on the synthetic corpus for the three-dimensional pose prior. That division is more realistic than assuming photorealistic rendering alone will transfer everywhere.
Detector Adaptation on Unlabeled Video
YOLO11-pose detects the robot and fifteen shared body landmarks: pelvis, neck, head, hips, knees, ankles, shoulders, elbows, and wrists. These points are not required to match every actuated joint. Their role is to localize the body consistently across robot types and to anchor the crop used by pose estimation. The box is expanded to a square and normalized so that the robot occupies a fixed fraction $\rho=0.55$ of the crop height.
The initial model begins from human keypoint weights and is fine-tuned on rendered robots. It performs well in the synthetic domain, but coverage on real G1, PM01, and N1 video is uneven. The paper mines additional frames from online videos, accepts detections with high confidence, interpolates short temporal gaps, filters for single-robot clips, and mixes the selected frames with synthetic supervision. Two self-training rounds are performed, and clips used for evaluation are excluded.
After the second round, detector coverage reaches 98.8 percent on G1, 96.8 percent on PM01, and 99.4 percent on N1. Coverage and landmark quality have different consequences for the final system. A missed frame yields no pose at all, while a detected but poorly aligned body changes the crop and can bias the decoder. The paper evaluates the latter on frames detected in all stages so that changing coverage does not silently alter the comparison set.
This adaptation remains weakly supervised with respect to 3D motion. It teaches the detector where the robot and its body landmarks are in real video, but it does not claim that unlabeled clips provide ground-truth joint angles. The decoder's pose training still comes from synthetic paired data. That distinction is important when interpreting the reported gains in real-world accuracy.
Kinematic Conditioning: A Robot-Agnostic Identity for Each Joint
A shared decoder cannot assign a permanent learned vector to every joint and then add new vectors for each robot, because the set of joints is open-ended. Instead, the paper describes each joint with geometry taken from the robot model. For joint $j$, it uses a local axis $\mathbf{a}_j$, local anchor $\mathbf{c}_j$, rest offset $\mathbf{o}_j$, rest quaternion $\mathbf{r}_j$, lower and upper limits, tree depth, side marker, robot scale, and quantities expressed in the root frame.
The paper concatenates these values into a 27-dimensional descriptor. The descriptor tells the network how the joint is mounted, where it lies relative to the root, which directions it can rotate, whether it has finite limits, how deep it sits in the chain, and how large the robot is. A shared multilayer perceptron maps the descriptor to a joint query:
$$ \begin{aligned} \mathbf{v}_j=[&\mathbf{a}_j,\mathbf{c}_j/s_R,\mathbf{o}_j/s_R,\mathbf{r}_j, \|\mathbf{o}_j\|_2/s_R,\ell_j,u_j,u_j-\ell_j,\\ &d_j/10,\eta_j,\log s_R,\bar{\mathbf{c}}_j/s_R,\bar{\mathbf{a}}_j,m_j] \end{aligned} $$Here $[\ell_j,u_j]$ is the allowed joint interval, $d_j$ is the compiled link-tree depth, $s_R$ is the robot's vertical scale at the rest configuration, and $m_j$ indicates whether the joint has finite limits. Machine-specific values remain in the descriptor, but the function that interprets them is shared.
The decoder input contains a root query, an auxiliary query, and one query per actuated joint:
$$ \mathbf{H}^{0}=[\mathbf{e}_{root},\mathbf{e}_{aux}, \mathbf{t}_1,\ldots,\mathbf{t}_n] $$Because each query has meaning in the target robot's kinematic model, the decoder can use attention to connect an image region with the corresponding mechanical degree of freedom. A longer sequence represents a more complex robot without changing the network definition. The root-frame anchor, axis, and side marker also prevent two visually similar joints from becoming indistinguishable merely because their local transforms look alike.
A Frozen Body Encoder and a Shared Pose Decoder
The image branch follows PromptHMR and keeps its image and camera encoders frozen. A square robot crop is transformed into $32\times32$ spatial image tokens. Keeping a grid, rather than pooling the whole crop into one vector, gives the decoder access to where visible limbs and torso parts occur. Human-pretrained weights provide the body-level visual prior needed to interpret unfamiliar robot appearance.
The shared pose decoder has width 256 and eight attention heads. The root, auxiliary, and joint queries attend to the image tokens and produce robot parameters. Approximately 4.55 million parameters are active in this decoding path. Training only this smaller component preserves the pretrained visual representation and keeps the experiment tractable, while the paper leaves open whether a carefully adapted visual backbone could improve results further.
Joint outputs are parameterized differently according to the robot model. A joint with finite limits predicts a normalized value and maps it through a sigmoid into its mechanical range:
$$ \hat{q}_j=\ell_j+(u_j-\ell_j)\operatorname{sigmoid}(z_j) $$A joint without finite limits regresses an angle and applies periodic wrapping:
$$ \hat{q}_j=\operatorname{wrap}(z_j) $$This parameterization prevents limited joints from drifting outside physically meaningful values while preserving the periodic structure of unrestricted rotational coordinates. It is a small implementation detail with large downstream relevance because the decoder's output can be consumed as a joint configuration rather than only visualized as an approximate skeleton.
Pose inference is frame-wise. Temporal information appears in detector gap filling and camera tracking, but no recurrent state or learned trajectory smoother is part of the main pose decoder. This simplifies supervision and avoids forcing the model to learn long-horizon dynamics from the available data. The cost is that per-frame noise and occlusion-induced jumps may remain in the recovered motion.
Metric Root Estimation from a Known Robot Height
Predicting arbitrary camera-space coordinates would force the network to learn metric scale from images alone. The paper instead combines the known robot height with the fixed crop construction. For root depth, it forms a scale from focal length, robot height, crop reference size, and the $0.55$ height ratio, then applies a bounded learned correction:
$$ \hat{z}=\frac{f_xH_R}{S\rho} \exp(\operatorname{clip}(p_z,-3,3)) $$The camera-frame root translation in image coordinates is then derived from the projected crop center and learned offsets:
$$ \begin{aligned} \hat{t}_x&=\left(\frac{u_0-c_x}{f_x}+p_x\right)\hat{z},\\ \hat{t}_y&=\left(\frac{u_0-c_y}{f_y}+p_y\right)\hat{z} \end{aligned} $$The initial depth follows from perspective and known size, while $p_z,p_x,p_y$ correct the estimate when a bounding box no longer corresponds to a standing pose. A crouched robot occupies a different fraction of the crop even when its distance is unchanged. A purely size-based baseline therefore has a systematic failure mode that the learned correction can reduce. This distinction becomes visible in the root-trajectory experiment.
The crop uses $S=480$ as its reference size. The detector supplies the camera-frame crop, and the pose branch predicts root orientation and translation along with joint values. The approach does not estimate the robot's physical dimensions from appearance; it reads them from the supplied model. Model inaccuracies can therefore propagate into metric depth, a limitation that should be considered in deployment.
Losses That Respect Joints, Links, and the Root
Joint-angle error alone can leave ambiguity. Two configurations of connected joints may produce similar positions for the next link, and changing the orientation of a terminal link may not move a body origin at all. The paper therefore supervises the controlled angles, forward-kinematics body positions, terminal orientation, root rotation, and root translation.
The joint loss averages wrapped angular error over all actuated joints:
$$ \mathcal{L}_{q}=\frac{1}{n}\sum_j \left|\operatorname{wrap}(\hat{q}_j-q_j)\right| $$The forward-kinematics term compares body positions generated by predicted and target joint configurations:
$$ \mathcal{L}_{FK}=\frac{1}{|\mathcal{B}_R|} \sum_{b\in\mathcal{B}_R} \left\|P_R(\hat{\mathbf{q}})_b-P_R(\mathbf{q})_b\right\|_2 $$Differentiable forward kinematics lets this geometric error update the joint predictions. A terminal-orientation loss uses $1-\langle\hat{\mathbf{u}}_b,\mathbf{u}_b\rangle^2$ for relevant terminal links, while root rotation and translation are also supervised. The full objective is
$$ \mathcal{L}=\mathcal{L}_{q}+\lambda_{FK}\mathcal{L}_{FK} +\lambda_{end}\mathcal{L}_{end} +\lambda_{root}\left(\mathcal{L}_{rot} +\frac{\|\hat{\mathbf{t}}-\mathbf{t}\|_1}{5}\right) $$The weights are $\lambda_{FK}=1$, $\lambda_{end}=0.1$, and $\lambda_{root}=0.5$. Training uses AdamW with a learning rate of $4\times10^{-4}$, an effective batch size of 48, 8,000 rendered images per embodiment, and 20 epochs. The combination gives the network direct control-space targets and body-space geometric targets without requiring the same loss to compare incomparable units.
From Camera Coordinates to a World Trajectory
A camera-frame root answers where the robot appears in one image. A world trajectory must also account for movement of the observer. With $\mathbf{T}_{A\leftarrow B}$ denoting a transform from frame $B$ to frame $A$, the robot's world root is obtained by composing the camera-to-world and robot-to-camera transforms:
$$ \hat{\mathbf{T}}_{W\leftarrow R,t} =\mathbf{T}_{W\leftarrow C,t} \hat{\mathbf{T}}_{C\leftarrow R,t} $$The camera path is estimated with PromptHMR's camera procedure and DROID-SLAM, while robot pixels are masked so that the moving robot is less likely to be mistaken for static-scene motion. Metric robot depth supplies a scale cue. Each continuous shot receives its own world origin. The output is therefore a relative trajectory within a shot, not an automatically merged reconstruction of several disconnected rooms or recording sessions.
This coordinate decomposition exposes the error chain directly. Joint errors alter the robot's shape, root errors alter its placement relative to the camera, and camera errors alter the observer's path. Even an accurate per-frame silhouette can produce a drifting world trajectory when the scene motion estimate is poor. The paper evaluates root motion separately on a robot with motion-capture reference, which is more informative than judging world-space output only from rendered overlays.
Real-Platform Pose Accuracy
The main real evaluation uses Unitree G1, EngineAI PM01, and Fourier N1 recordings. Clips include walking, arm movement, crouching, and supported body poses. Joint references are synchronized with encoders. G1 also has optical motion capture for root evaluation. Synthetic results are averaged with equal weight across the sixteen embodiments so that robots with more frames do not dominate.
The baselines include landmark fitting, PromptHMR followed by GMR retargeting, a synthetic-fine-tuned version of that pipeline, HoRoPose, and RoboPEPP. Since a standard benchmark for monocular full-body humanoid proprioception does not yet exist, several baselines are adapted specifically for this task. The comparison holds detection, crops, and frame sets as constant as possible. HumanoidPose uses one model for all sixteen robots, whereas multiple baselines require separate per-robot training.
| Platform | HumanoidPose angle | Best baseline angle | HumanoidPose position | Best baseline position |
|---|---|---|---|---|
| G1 | 7.10° | 10.46° | 3.81 cm | 4.88 cm |
| PM01 | 5.65° | 6.99° | 2.60 cm | 6.02 cm |
| N1 | 8.62° | 10.52° | 5.18 cm | 5.63 cm |
| Synthetic, 16 models | 9.48° | 14.49° | 4.49 cm | 8.68 cm |
The main empirical point is that a shared model beats the best per-platform baseline on all three real robots. Relative to those baselines, real joint-angle error falls by 18 to 32 percent, with the largest reduction on G1. On synthetic data, angle error is about 35 percent lower than the strongest listed baseline. Position error also improves, although N1's margin over its best baseline is modest.
Figure 3: Real-image predictions for Unitree G1, EngineAI PM01, and Fourier N1. The examples cover walking, raised arms, crouching, and supported poses.
The qualitative examples reveal why cross-embodiment conditioning is useful. The robots differ in limb length, torso shape, and external appearance, and some views are far from a frontal presentation. Yet one decoder can assign image evidence to the corresponding model-specific joints. The figure supports breadth of behavior, but it does not establish robustness to every occlusion, rapid motion, or extreme joint configuration.
Root Motion on G1
Root-trajectory evaluation is limited to G1 because it is the only platform with motion-capture reference in this study. Each trajectory receives one fixed-scale rigid alignment rather than a separate correction per frame. Alignment is refined using the 90 percent of frames with the lowest initial residual. This protocol rewards a coherent trajectory instead of allowing each point to be shifted independently onto the reference.
| Method | Root mean square | Median | P90 |
|---|---|---|---|
| Landmark fitting | 42.2 | 21.3 | 56.3 |
| PromptHMR | 66.2 | 37.4 | 96.2 |
| PromptHMR with height scaling | 39.0 | 16.8 | 51.1 |
| HumanoidPose | 23.4 | 11.3 | 21.7 |
Compared with height-scaled PromptHMR, HumanoidPose lowers root RMSE from 39.0 to 23.4 centimeters, a reduction of roughly 40 percent. The median falls from 16.8 to 11.3 centimeters and P90 falls from 51.1 to 21.7 centimeters. The improvement is therefore not caused by a few large errors disappearing from the mean; the tail of the distribution becomes substantially shorter.
The result also exposes where simple grounding assumptions fail. A fixed height-to-depth rule is reasonable for an upright robot, but a crouch or supported pose changes the observed extent without changing the camera distance. Learned root offsets can use the body configuration to compensate. The evidence remains platform-specific, however. Joint accuracy on PM01 and N1 does not prove that their world trajectories are equally reliable.
Ablation: Which Design Choices Carry the Result?
The paper removes four components while holding the evaluation data fixed. The no-joint-identity variant drops root-frame anchors, axes, and side markers. The no-direct-robot-decoding variant predicts a human pose and retargets it. The no-kinematic-conditioning variant replaces geometric descriptors with per-robot learnable embeddings. The no-human-pretrained-encoder variant substitutes DINOv2. Real numbers are frame-weighted across the three robots.
| Variant | Synthetic angle | Synthetic position | Real angle | Real position |
|---|---|---|---|---|
| Full model | 9.48° | 4.49 cm | 7.09° | 3.82 cm |
| No joint identity | 10.94° | 5.05 cm | 7.15° | 4.02 cm |
| No direct robot decoding | 16.82° | 7.98 cm | 12.12° | 6.19 cm |
| No kinematic conditioning | 12.43° | 6.67 cm | 7.62° | 4.15 cm |
| No human-pretrained encoder | 11.14° | 5.57 cm | 7.95° | 4.65 cm |
Direct robot decoding has the largest effect. Passing through a human representation and retargeting raises synthetic angle error from 9.48 to 16.82 degrees and real error from 7.09 to 12.12 degrees. This supports the paper's claim that human priors are valuable for visual features but harmful as a bottleneck for the final robot state.
Kinematic conditioning is the second major factor. Replacing geometric descriptors with per-robot learned embeddings raises synthetic angle error to 12.43 degrees. Such embeddings can memorize the robots seen during training, but they do not expose a common geometric meaning for a new set of joint axes and limits. The descriptor gives a new robot a route into the same decoder without allocating a fresh model or a fresh embedding table.
Joint identity and the human-pretrained encoder show smaller but consistent gains. Their real-domain margins are modest, so the ablation should not be overread as proof that these modules are equally important on every platform. The full-model real angle of 7.09 degrees differs slightly from the equal-platform mean in Table 1 because the ablation uses frame weighting. The two values answer related but not identical aggregation questions.
Held-Out Robots and Few-Shot Adaptation
The transfer study removes one or two embodiments from decoder training. G1 variants or PM01 can be held out while the remaining fourteen or fifteen robots remain available. The detector can still be trained with synthetic images including the target structure and with public real video. At pose-test time, the held-out robot enters through its MJCF-derived queries; no new joint-embedding parameters are learned.
On held-out real data, the aggregated joint-angle errors are 11.27 degrees for G1 and 11.44 degrees for PM01. Median root errors across sequences fall between 8 and 16 centimeters. These results are worse than those for robots represented during decoder training, which is expected, but they remain usable without any real joint-angle supervision for the target decoder.
A second experiment fine-tunes a held-out G1 decoder on increasing amounts of G1 synthetic data. Synthetic test error decreases as the amount grows, while the visual encoder stays frozen and real joint labels remain absent. This supports a practical adaptation recipe: begin with the robot description, add a modest target-specific render set, and avoid assuming that real encoder synchronization is the only path to a new embodiment.
What Detector Self-Training Actually Improves
| Stage | Synthetic keypoint accuracy | G1 coverage and angle | PM01 coverage and angle | N1 coverage and angle |
|---|---|---|---|---|
| Initial YOLO11 | 38.6% | 98.6%, 7.82° | 96.5%, 5.89° | 77.8%, 17.65° |
| With synthetic fine-tuning | 93.6% | 81.1%, 8.65° | 75.6%, 5.87° | 99.3%, 8.98° |
| With two self-training rounds | 92.1% | 98.8%, 7.11° | 96.8%, 5.36° | 99.4%, 8.90° |
The table shows a tradeoff hidden by a single average. Synthetic fine-tuning raises keypoint accuracy from 38.6 to 93.6 percent and greatly improves N1, but real coverage drops on G1 and PM01. Self-training restores both to above 96 percent while reducing fixed-decoder angle error on G1 and PM01. N1 coverage is already high, and its error changes only slightly.
For a deployable perception system, these effects must be separated. A detector that never fires produces no 3D output, regardless of decoder quality. A detector that finds the robot but places the crop or landmarks poorly produces a less reliable pose. The shared-frame evaluation is therefore a useful control: self-training contributes both more usable footage and better geometric conditioning, with the coverage gain being the larger change.
Why the Design Is Useful beyond a Benchmark
The first practical advantage is output compatibility. Joint coordinates and a root transform are closer to the values used by forward kinematics, motion analysis, and imitation-learning pipelines than a human-body mesh is. The system removes one conversion between observing a body and consuming a robot state. The second advantage is model reuse. A robot-specific description changes the decoder query, but it does not require a new visual network for every new platform.
The third advantage is a clean split between simulated pose learning and real appearance adaptation. Rendering supplies exact mechanical labels, while unlabeled internet video supplies visual diversity. The fourth is metric grounding through a known robot height and camera model. The fifth is explicit compensation for observer motion, without which a moving camera can be mistaken for robot translation.
These choices could support external supervision of demonstrations, recovery of motion from old robot videos, filtering of generated robot footage, and inspection of machines whose state interface is unavailable. Those are plausible applications suggested by the capabilities, not tasks evaluated in the paper. A task-level claim would require showing that downstream controllers or policies improve when using the estimated pose.
Limitations and Open Risks
The authors explicitly limit the evaluated setting to a single robot. Multiple robots introduce identity assignment, mutual occlusion, and longer evidence gaps. Applying the detector to every box is not enough, because the system also needs to preserve which trajectory belongs to which robot. The paper does not present a multi-robot tracking result.
The authors also note that downstream interaction and learning from demonstration are not evaluated. Lower joint error is a useful intermediate result, but it does not prove that a contact-rich manipulation policy or a human-robot collaboration system will perform better. Metrics should be chosen for the task: end-effector error may matter more than average joint error in manipulation, while torso stability may matter more for locomotion.
The decoder is frame-wise. Temporal information helps detection and camera estimation, but there is no explicit learned smoother or dynamic model for the 3D joint trajectory. Real videos can therefore contain frame-to-frame jitter, short discontinuities, or inconsistencies after occlusion. The method should be viewed as a strong per-frame pose estimator with trajectory composition, not as a complete temporal motion-capture engine.
The empirical platform set is narrow. Only three physical robots are used, and world-root ground truth exists only for G1. Deployment on different foot designs, wheeled-legged robots, dexterous hands, or substantially different heights remains to be tested. The method also depends on accurate MJCF or URDF geometry; incorrect axes, limits, or link scales affect both the joint descriptor and the forward-kinematics loss.
Finally, the repository currently does not release code or model weights. The paper explains the method and evaluation well enough for external review, but implementation choices, exact preprocessing, training cost, latency, and deployment robustness cannot be independently checked yet. A coming-soon notice is useful provenance, not a substitute for a reproducible artifact.
Conclusion
HumanoidPose is not primarily a new visual backbone. Its contribution is the organization of four resources: a human-pretrained visual encoder, robot-specific kinematics, synthetic paired supervision, and unlabeled real video for detector adaptation. The first supplies a body prior, the second supplies a common interface across embodiments, the third supplies exact state labels, and the fourth reduces the appearance gap.
The resulting system recovers full-body joint coordinates and camera-frame root pose from ordinary video, then composes them with camera motion for relative world-space trajectories. Its real joint-angle errors range from 5.65 to 8.62 degrees and its position errors from 2.60 to 5.18 centimeters. G1 root RMSE reaches 23.4 centimeters, about 40 percent lower than height-scaled human-model retargeting.
The next meaningful advances are temporal tracking, multi-robot identity, physically feasible motion constraints, and task-level evaluation. If those are connected to the current cross-embodiment decoder, video could become an independent observation channel for robot state rather than a weak substitute for onboard sensing.
Quote
When a robot's internal state is unavailable, video is not merely a visual record; it can become an external proprioceptive signal for recovering joint configuration, balance, and intended motion.
