OmniEgoCap: Device-Agnostic Sequence-Level Egocentric Motion Reconstruction

Authors: Kyungwon Cho, Jeonghyeon Na, Hanbyul Joo

Paper: arXiv:2512.19283　 Project: kyungwoncho.github.io/OmniEgoCap　 Code: Official GitHub repository

Code status: Public. The repository provides preprocessing, training, inference, device adapters, and OmniEgoDB evaluation scripts. The checkpoint is released under CC BY-NC 4.0.

Executive Summary

OmniEgoCap changes egocentric motion capture from frame-by-frame body guessing into a sequence problem that estimates one body shape, one camera-visibility context, and a coherent motion trajectory at the same time. From a head trajectory and intermittent wrist observations, it can reconstruct SMPL-H motion across Aria, GoPro, Quest, and simulated camera configurations without device-specific retraining.

Why the Problem Is Difficult

An ego camera follows the wearer's head, so most of the body is outside the field of view. Head pose is usually available, and a hand tracker may recover one or both wrists for part of the sequence. The missing torso, elbows, legs, and feet must come from a learned human prior. This is not only a single-frame completion task: wrist depth is ambiguous, hand visibility changes abruptly, and a pose that fits one frame can contradict the body shape inferred from the next.

Existing head-centric methods often encode an implicit camera model in their training distribution. A model trained with a 120-degree field of view may learn that an invisible hand belongs near a particular image boundary. It may then pull a genuinely occluded hand back into view or hallucinate it at the edge. Narrower lenses, tilted mounts, non-square sensors, and different wrist estimators violate that learned boundary. The resulting error is architectural, not merely a matter of insufficient expression capacity.

Temporal inconsistency creates a second failure mode. A per-frame system can fit the currently visible wrist very closely while changing body scale and root orientation between neighboring frames. If the wrist disappears for a few frames, a local extrapolator may continue its last velocity and generate impossible arm motion. A sequence model can instead treat the missing interval as evidence and use later observations to resolve earlier ambiguity.

The empirical problem is also a benchmarking problem. Most real ego-motion datasets use one dominant capture device, so a claimed cross-device result may only reflect one camera's optical geometry and tracking errors. The authors introduce OmniEgoDB to pair Xsens motion capture with Aria, GoPro, and Quest recordings. It contains 50 sequences from four subjects and 177.2 minutes of motion. No compared method is trained on this benchmark, which makes it a test of transfer rather than in-domain fitting.

The central design thesis follows directly from these failures. The representation should remove as much device-specific coordinate structure as possible, while training should still expose the network to many physically plausible visibility boundaries. Either ingredient alone is insufficient: invariant features without varied visibility can overfit one camera, and augmentation without sequence context cannot resolve which missing observations belong to the same body.

Representation and Generative Prior

The output is a sequence of SMPL-H poses and shape parameters. A frame contains 16 shape coefficients, a root orientation relative to a gravity-aligned frame, and local transforms for 22 joints. The network receives head transforms and left/right wrist transforms in head coordinates. Each wrist also has a visibility flag that separates a measured observation from a placeholder.

To keep device geometry from dominating the representation, the authors derive head deltas, a gravity-aligned canonical frame, relative yaw, head height, and wrist poses expressed relative to the head. This does not discard global motion. It separates the body's articulation from the particular camera mount, leaving root orientation and trajectory available for later recovery.

The generative prior is a diffusion model over the full motion sequence. During training, Gaussian noise is added to ground-truth canonical poses and the decoder learns to predict the clean sequence. At inference, the sample starts from noise and is refined with DDIM. Sequence tokens represent time, so self-attention can use recurring motion patterns, transitions between visible and hidden hands, and the rhythmic relation between head trajectory and body support.

Method in Detail

1. Build observations that follow the body rather than the camera

The reconstruction problem is written as a mapping from the head trajectory and wrist observations over time to a complete body sequence:

$$\mathbf{M}_{1:T}=\operatorname{OmniEgoCap}\left(\mathbf{T}^{\mathrm{head}}_{0:T},\mathbf{H}_{0:T}\right).$$

Here $\mathbf{M}_{1:T}$ is the full motion, $\mathbf{T}^{\mathrm{head}}$ is head pose over time, and $\mathbf{H}$ contains wrist transforms and visibility. The notation deliberately makes the entire interval the object of inference rather than a collection of unrelated frames.

For each time step, the system forms a condition vector from head motion, canonical orientation, head height, canonical yaw change, and head-relative hands:

$$\boldsymbol{\Omega}_{t}=\Gamma\left(\Delta\mathbf{T}^{\mathrm{head}}_{t},\mathbf{R}^{\mathrm{head}\to\mathrm{cano}}_{t},h_{t},\Delta\mathbf{R}^{\mathrm{cano}}_{t},\mathbf{T}^{\mathrm{hand}\to\mathrm{head}}_{t}\right).$$

$\Gamma$ denotes feature construction and embedding, while $h_t$ is head height. The implementation in omniegocap/core/features.py:42-52 computes the head delta, canonical frame, yaw delta, and wrist transforms in head coordinates exactly along this path. The choice matters because raw world-space wrist positions mix body movement with camera installation and SLAM orientation.

Missing hands are represented by learned null tokens, not zeros:

left_emb = torch.where(
    visible[:, :, 0:1],
    self.left_embedder(left_cond),
    self.left_null_embed,
)
right_emb = torch.where(
    visible[:, :, 1:2],
    self.right_embedder(right_cond),
    self.right_null_embed,
)

The two branches in omniegocap/core/model.py:88-101 use separate null embeddings for the left and right sides. A zero coordinate would imply a physical wrist at the head origin. A null token instead lets the network learn an absence state that can participate in the sequence prior without pretending to be an observation.

2. Separate the person from the observation process

The encoder has three functional branches. One compresses the sequence into a single body shape, one learns a global visibility and camera context, and one produces per-frame local evidence conditioned on that context:

$$\hat{\beta}=\mathcal{E}_{\mathrm{shape}}(\boldsymbol{\Omega}_{1:T}),\quad \mathcal{G}=\mathcal{E}_{\mathrm{global}}(\boldsymbol{\Omega}_{1:T}),\quad \mathcal{S}_{1:T}=\mathcal{E}_{\mathrm{local}}(\boldsymbol{\Omega}_{1:T},\mathcal{G}).$$

$\hat{\beta}$ is the SMPL-H shape shared by the sequence, $\mathcal{G}$ summarizes what the sequence can see, and $\mathcal{S}_{1:T}$ is the decoder's local condition. The official code at omniegocap/core/model.py:230-255 uses attention pooling for the 16 shape dimensions and injects the global embedding into the local encoder through adaptive normalization or concatenation.

This decomposition addresses two time scales. Shape should remain nearly constant across a short action, but visibility can change from frame to frame. A per-frame shape estimator allows noisy hands to make the person grow or shrink. A single global feature for everything loses the exact moments at which a wrist appears. OmniEgoCap gives the decoder both a stable identity variable and a dense temporal condition.

3. Condition every denoising layer on the inferred body

The decoder receives a noisy canonical pose, diffusion timestep, predicted shape, and local sequence condition:

$$\hat{\mathbf{X}}_{1:T}^{0}= \mathcal{D}(\mathbf{X}_{1:T}^{n},n,\hat{\beta},\mathcal{S}_{1:T}).$$

Timestep and shape embeddings jointly modulate the decoder, while local conditioning enters through cross-attention. This is stronger than appending shape as one side-channel vector: the inferred proportions influence each layer throughout denoising. The implementation at omniegocap/core/model.py:264-325 mirrors this arrangement, and the paper's ablation confirms that removing shape conditioning increases both body and hand error.

Inference uses 30 DDIM steps rather than the 1000 training steps. An optional timestep-adaptive guidance step corrects the current prediction toward visible wrist observations. The source at omniegocap/core/inference.py:113-161 first encodes the whole sequence, then performs denoising, optional wrist optimization, and reconstruction of the next DDIM sample. The method therefore uses guidance as a correction inside the learned sequence prior, not as a replacement for that prior.

4. Synthesize camera geometry instead of one fixed visibility mask

The visibility augmentation defines whether a wrist direction lies inside a generalized ellipse:

$$\left|\psi_{x}/\gamma_{x}\right|^{p} +\left|\psi_{y}/\gamma_{y}\right|^{p}\leq 1.$$

$\psi_x$ and $\psi_y$ are yaw and pitch after center and tilt offsets, $\gamma_x$ and $\gamma_y$ are horizontal and vertical half-angles, and $p$ changes the boundary from an ellipse toward a more rectangular contour. The sampling procedure varies these parameters across batches, producing realistic dog-bone regions for different sensors and mounts.

u, v = yaw / yaw_half, pitch / pitch_half
val = torch.abs(u) ** p + torch.abs(v) ** p
mask = val <= 1.0

The visibility test appears at omniegocap/training/augment.py:67-70. The full augmentation also samples temporal tracking drops with Poisson starts and log-normal lengths, and perturbs wrist transforms. These disturbances model geometry, occlusions, tracking failures, and upstream pose noise together instead of relying only on random 20-percent masking.

5. Train denoising, shape, and auxiliary signals together

The objective combines the diffusion prediction loss with shape supervision and additional intermediate constraints:

$$\mathcal{L}=\mathcal{L}_{\mathrm{simple}} +\lambda_{\mathrm{shape}}\mathcal{L}_{\mathrm{shape}} +\mathcal{L}_{\mathrm{aux}}.$$

$\mathcal{L}_{\mathrm{simple}}$ is the clean-pose denoising objective, $\mathcal{L}_{\mathrm{shape}}$ stabilizes body proportions, and $\mathcal{L}_{\mathrm{aux}}$ supports representation learning. The appendix reports that removing the auxiliary loss changes jerk from 0.117 to 0.163, showing that the auxiliary term affects temporal smoothness rather than only optimization speed.

flowchart TB
    A[Head trajectory T 0:T] --> B[Delta and canonical features]
    C[Wrist poses and visibility] --> D[Left and right null-aware embeddings]
    B --> D
    D --> E[Shape encoder]
    E --> F[One beta for sequence]
    D --> G[Global visibility encoder]
    G --> H[Local frame encoder]
    H --> I[Diffusion decoder]
    F --> I
    I --> J[DDIM sampling]
    J --> K[Timestep-adaptive wrist guidance]
    K --> L[Coherent SMPL-H motion]

This diagram follows the implementation rather than a generic diffusion template. The global representation modifies local encoding, the shape vector conditions the denoiser at every layer, and guidance operates inside the DDIM loop only when enabled.

Experiments and Evidence

The simulation study trains on AMASS and evaluates several fields of view: 90 by 90, 120 by 120, 180 by 180, 120 by 60, and 120 by 120 with a 30-degree downward tilt. The diverse benchmark also changes visibility boundaries, temporal dropout, and wrist perturbations. Metrics cover joint position, joint velocity, acceleration, all-hand error, and error on currently visible hands.

Table 1 shows why the sequence design matters. OmniEgoCap without guidance reaches 81.35 mm MPJPE, and lightweight guidance lowers it to 80.45 mm. The augmented EgoAllo variant with optimization reaches 94.00 mm. The gain is not just a better fit to visible hands: the sequence model preserves a plausible whole-body state when hand observations disappear.

Method	MPJPE / mm	MPJVE / mm	Jerk	Hand PE / mm	VisHandPE / mm
EgoAllo with optimization	99.32	31.66	0.152	164.80	41.49
EgoAllo† with optimization	94.00	26.13	0.120	142.24	25.74
OmniEgoCap	81.35	25.81	0.117	146.71	56.98
OmniEgoCap with optimization	80.45	25.48	0.120	132.74	29.34

The hand numbers contain an important tradeoff. EgoAllo† has the lowest error on visible hands, but its overall hand error is higher. It follows observations aggressively and becomes unstable when they disappear. OmniEgoCap is worse on visible hands before guidance because it favors sequence plausibility, but its total hand error is already lower. Guidance then brings visible-hand accuracy down without destroying the global trajectory.

Figure 4 provides the corresponding failure cases. In a dance sequence, EgoAllo produces abrupt, unstable motion when hands leave the camera; OmniEgoCap remains rhythmically and anatomically plausible. In a kick, EgoAllo hallucinates the hidden hand near the learned field-of-view boundary, while OmniEgoCap keeps it outside the observed region. The visual evidence directly tests the paper's claim that visibility should be modeled as camera geometry rather than memorized as a fixed mask.

Figure 4: Qualitative comparison under varied visibility; the kick case probes hidden-hand hallucination.

Camera-setup generalization isolates the same issue without changing the body model. OmniEgoCap reports MPJPE values of 93.65, 85.70, 72.22, 75.70, and 98.00 mm across the five test fields of view. The wide 120 by 60 setup is hardest, but the model still beats baselines trained around a fixed boundary. EgoAllo† can even become worse at 180 degrees, which shows that seeing more of the scene is not enough when the learned visibility prior is wrong.

Figure 3: Calibrated fields of view and geometry-aware augmentation boundaries across realistic camera shapes and tilts.

Real Devices and In-the-Wild Sequences

OmniEgoDB adds Aria, GoPro, and Quest recordings with synchronized Xsens ground truth. It contains 21 Aria sequences totaling 67.1 minutes, 20 GoPro sequences totaling 68.1 minutes, and 9 Quest sequences totaling 42.0 minutes. Aria can use stereo or RGB-only hand inputs, while GoPro depends more heavily on RGB head and hand estimation. The benchmark therefore varies optics, mounting, trajectory length, and upstream tracking errors at the same time.

Real track	MPJPE / mm	MPJVE / mm	Jerk	Hand PE / mm
Aria stereo	106.61	21.29	0.077	142.28
Aria RGB	118.11	23.25	0.083	173.83
GoPro	111.68	21.81	0.073	166.20
Quest	110.49	18.16	0.062	135.86

The real-device errors remain around ten centimeters, so the benchmark does not claim that egocentric capture is solved. Its result is relative robustness without device-specific training. Aria RGB is worse than stereo, consistent with the difficulty of monocular hand depth. GoPro has lower jerk than Aria RGB despite a similar joint-velocity error, showing that different devices fail in different ways and that one aggregate score hides important structure.

Figure 6 overlays GoPro and Aria RGB predictions with Xsens ground truth. The improvement is visible in long-range torso and lower-body coherence, not only in isolated wrist frames. Figure 7 turns to EgoExo4D cooking, where EgoAllo's body shape fluctuates under noisy real observations while OmniEgoCap remains stable during a mostly static activity. Stable stature is useful for robot learning because downstream retargeting can otherwise interpret a camera-tracking artifact as a change in body scale.

Figure 6: GoPro and Aria RGB reconstruction on OmniEgoDB, with Xsens ground truth in orange.

Figure 7: On EgoExo4D cooking, OmniEgoCap maintains body shape more reliably than EgoAllo.

Ablation: What Actually Carries the Gain

Table 4 removes the sequence-level components one at a time. Without shape and global context, MPJPE is 97.06 mm. Removing only shape conditioning raises it to 86.55 mm; removing only global context gives 83.53 mm; replacing adaptive modulation with concatenation gives 81.60 mm. The full model reaches 81.35 mm. The result establishes that both stable shape and visibility context matter, although jerk is slightly lower in some reduced variants, so the best joint-error model is not automatically the smoothest model.

Variant	MPJPE / mm	MPJVE / mm	Jerk	Hand PE / mm	VisHandPE / mm
No shape or global context	97.06	25.17	0.104	161.25	79.70
No shape conditioning	86.55	26.59	0.111	156.09	65.05
No global context	83.53	26.82	0.116	152.73	65.99
Concatenation instead of modulation	81.60	26.13	0.112	149.92	61.42
Default model	81.35	25.81	0.117	146.71	56.98

A linear probe asks whether the global code merely memorizes device identity. Shuffled labels achieve 0.447 visible-region IoU, a mean training camera reaches 0.582, coarse visibility statistics reach 0.662, and the learned context reaches 0.740. In a second probe, the context reduces stature error to 9.19 cm, while explicit shape parameters reach 3.05 cm. The two variables are complementary: context predicts what the camera can observe, and shape carries the body's metric identity.

Training with a fixed 120-degree boundary is not equivalent to sampling geometry. Fixed masks can fit the nominal setup but transfer poorly to narrow, wide, tilted, or non-square cameras. Geometry-aware augmentation spans the process that generates those masks. The model therefore learns a relation between direction, field of view, and missing observations rather than one boundary coordinate.

Implementation Notes

The public repository separates adapters, model code, diffusion utilities, and evaluation scripts. Aria can be prepared from MPS output, while other devices use a trajectory adapter that accepts camera-to-head calibration, head trajectories, wrist poses, and visibility. Device independence therefore means no retraining by device; it does not mean the model accepts raw uncalibrated video.

Hydra configuration files expose the model, diffusion, and augmentation settings. Geometry augmentation lives under conf/augment/geometry.yaml; scripts/sample.py and scripts/visualize.py provide the inference and inspection path. Users must separately obtain SMPL-H and AMASS. The MIT code license and CC BY-NC 4.0 checkpoint license should be checked before commercial deployment.

The most fragile practical assumption is coordinate consistency. Head and wrist inputs must share one world frame, and the camera-to-head transform must be calibrated. A calibration error can look like body motion to the network. The paper also reports degradation when SLAM or hand tracking becomes unreliable, confirming that OmniEgoCap is a consumer of upstream tracking rather than a direct RGB replacement.

Limitations

The authors explicitly identify seated poses as ambiguous. Similar head and hand trajectories can correspond to squatting, sitting on the floor, or sitting on a chair, and an ego view contains little contact information to distinguish them. Near-lying poses also destabilize the head-centered canonical frame and can produce minor foot floating.

Reconstruction quality drops when SLAM or hand tracking is unreliable, and the model assumes a flat ground plane. It does not reconstruct finger articulation or model hand-object contact, so its output is not yet a complete manipulation demonstration. The method is also offline and sequence-level; accelerated DDIM does not make it a streaming real-time system.

From an independent deployment perspective, OmniEgoDB covers only four subjects and a limited range of locations and activities. Real-device validation is concentrated on Aria, GoPro, and Quest, so generalization to phones, ordinary glasses, or arbitrary headsets remains unproven. A new device still needs trustworthy calibration, synchronized modalities, and validation of its tracker errors.

Conclusion

OmniEgoCap's durable idea is to separate three sources of uncertainty. A sequence-level shape variable represents the person, a sequence-level visibility variable represents the camera, and a diffusion decoder resolves sparse local observations into motion. Geometry-aware augmentation gives those variables many observation regimes to explain. The result is not just lower average error, but less contradictory behavior when the camera changes or the hands disappear.

For robotics, the method offers a practical route from human ego video to full-body motion data. The next useful step is to add ground contact, object contact, and finger state to the same temporal model. That extension would move the system from plausible human reconstruction toward demonstrations that can be retargeted and executed. The current release is best understood as a stable offline body estimator, not a finished embodied-data platform.

“The model should not memorize where one camera hides a hand; it should infer how one body remains coherent while the observations disappear.”