A Unified Inertial Encoder for Human and Humanoid Motion

A Unified Inertial Encoder for Human and Humanoid Motion

Paper record: Yutian Chen, Shibo Zhao, Rocky Duan, Karen Liu, Carmelo Sferrazza, Guanya Shi, Yuheng Qiu, and Koushil Sreenath. arXiv 2026, 2611.10137. Code is available at amazon-far/Unified-Inertial-Encoder under Apache-2.0, with the reviewed commit 462b0f1. The project page is unified-inertial-encoder.github.io.

One-sentence summary: The paper turns human and G1 motion into timestamped measurements at twelve corresponding body sites, then trains one continuous-time encoder for velocity, trajectory consistency, sensor denoising, and language retrieval, allowing human motion data to reach a physical humanoid without real-robot training trajectories or fine-tuning.

Why a new inertial interface is needed

Learned inertial odometry is usually trained where it will be deployed. A pedestrian model sees pedestrian recordings; a drone model sees aggressive flight; a humanoid model sees motion that its current controller can execute. This creates a coverage trap for humanoids. Reliable control is needed to collect diverse robot motion, but the estimator itself is supposed to help control remain reliable when the robot encounters uncommon movement. Training only on successful controller rollouts narrows the motion distribution exactly where generalization matters.

Human motion datasets break that circular dependency. They contain walking, turning, jumping, dancing, sitting, interacting, and other whole-body behaviors that are expensive or unsafe to request from a humanoid controller. Some datasets also provide natural-language descriptions, which can supervise motion semantics in addition to geometry. The opportunity is not merely “more hours.” It is access to action categories and speed profiles that a single robot platform may never have recorded.

The obstacle is embodiment mismatch. A human and a G1 have different limb lengths, joint layouts, mass distributions, and sensor placements. Their accelerometers and gyroscopes may use different coordinate conventions. A synthetic signal also lacks contact impulses, structural vibration, mounting compliance, and the electrical imperfections of a physical IMU. Naively pooling human and robot recordings therefore leaves the model to discover both the physical correspondence and the domain gap from undifferentiated data.

Most prior multi-IMU models are optimized for human activity recognition or human motion retrieval. Most inertial odometry systems are specialized to one platform or dataset. The paper occupies the intersection: it asks whether a common full-body inertial representation can serve both metric state estimation and language-grounded motion understanding across two embodiments.

The authors make a deliberate choice about where transfer should occur. They do not first learn a humanoid control policy in simulation and then collect enough G1 data to train a robot-specific estimator. They transfer motion knowledge at the sensing layer. A kinematic mapping converts available G1 signals into the same anatomical interface used by human data, and one encoder learns from both. This makes the central claim testable: if the interface is meaningful, human motion should improve G1 odometry and semantic retrieval even without G1 training trajectories.

The twelve-site representation

The interface contains the pelvis, head, left and right upper arms, forearms, upper legs, lower legs, and feet. Human streams can be real wearable IMUs or virtual sensors derived from motion capture. G1 streams at non-pelvis sites are synthesized from one pelvis IMU, joint positions, joint velocities, and the robot's kinematic model. For each site $i$, the input is a variable-length timestamped sequence rather than a fixed-rate array:

$$\mathcal{X}_{i}=\left\{(t_{i,j},\mathbf{a}_{i,j},\boldsymbol{\omega}_{i,j})\right\}_{j=1}^{T_{i}},\qquad i\in\mathcal{S}.$$

Here $\mathbf{a}_{i,j}$ is specific force including gravity, and $\boldsymbol{\omega}_{i,j}$ is angular velocity in the local sensor frame. The model predicts body-frame linear velocity for every site and uses pelvis velocity as root velocity. A separate [CLS] embedding is aligned with a frozen text encoder for motion-description retrieval. The same trunk therefore has to preserve both metric quantities and semantic motion cues.

The representation is shared in meaning, not in raw scale. Human and G1 sites have the same anatomical role, but their geometry is allowed to differ. The conversion below inserts those geometric differences into the signal before learning begins, which is a stronger inductive bias than asking a generic sequence model to infer them from embodiment labels.

From one pelvis sensor to twelve virtual sensors

G1 hardware supplies one base IMU plus joint encoders. The paper transports the base angular velocity and acceleration to each selected link using URDF forward kinematics and Jacobians. With joint angle vector $\mathbf{q}$, link angular Jacobian $\mathbf{J}_{\omega,i}$, and the relative pelvis-to-link rotation $\mathbf{R}_{bi}$, the synthesized angular velocity is:

$$\boldsymbol{\omega}_{i}=\mathbf{R}_{bi}^{\mathsf{T}}\left(\boldsymbol{\omega}_{b}+\mathbf{J}_{\omega,i}\dot{\mathbf{q}}\right).$$

The acceleration transport is more involved because an IMU mounted away from the pelvis experiences tangential, centripetal, Coriolis, and relative-link acceleration. Let $\mathbf{r}_{i}$ be the mounting-point position relative to the pelvis, with first and second derivatives $\dot{\mathbf{r}}_{i}$ and $\ddot{\mathbf{r}}_{i}$. Then:

$$\mathbf{a}_{i}=\mathbf{R}_{bi}^{\mathsf{T}}\left[\mathbf{a}_{b}+\dot{\boldsymbol{\omega}}_{b}\times\mathbf{r}_{i}+\boldsymbol{\omega}_{b}\times\left(\boldsymbol{\omega}_{b}\times\mathbf{r}_{i}\right)+2\boldsymbol{\omega}_{b}\times\dot{\mathbf{r}}_{i}+\ddot{\mathbf{r}}_{i}\right].$$

This equation explains why the method is not a trivial duplication of the pelvis measurement. During torso rotation, the head, hands, and feet follow different arcs and therefore observe different accelerations. Knee flexion changes the motion of the lower leg and foot. The synthesis uses only the pelvis IMU, joint state, and robot geometry; it does not require a world pose or root velocity. That property is what makes the same mapping usable for retargeted training motion and for live hardware input.

The implementation exposes the correspondence rather than hiding it inside a neural module. src/interface/data_humanoid.py:147-177 defines the human-to-G1 site map; 217-224 computes relative position, velocity, and acceleration; src/utility/g1_mounting.py:1-21 stores the fixed G1-to-Xsens mounting rotations. These are concrete geometric assumptions that can be audited when the method is moved to another robot.

Coordinate alignment is part of the model

Matching anatomical names is insufficient if the sensor axes differ. The authors align every stream to the Xsens human convention. For each site they estimate a fixed rotation from the dataset's native axes to the reference convention and apply it to orientation, acceleration, gyroscope, and velocity. Let $\mathbf{M}_{i}$ be that correction and $\mathbf{x}_{i}$ any of the three-dimensional motion quantities:

$$\mathbf{R}_{wi}^{\mathrm{can}}=\mathbf{R}_{wi}\mathbf{M}_{i},\qquad \mathbf{x}_{i}^{\mathrm{can}}=\mathbf{M}_{i}^{\mathsf{T}}\mathbf{x}_{i}.$$

World positions remain unchanged. Only the expression of orientation and local motion vectors changes. The same correction is applied to retargeted and physical G1 streams, reducing the chance that a hardware-versus-simulation gap is actually an avoidable axis convention error.

This choice also creates a dependency. The correction relies on an anatomical frame estimate and an average mounting orientation. A loose sensor, a different rigid mount, or a large calibration error can violate the assumption. Motion randomization is not a substitute for calibration because it perturbs the signal statistically rather than identifying the actual installation transform.

Continuous convolution preserves physical time

The datasets do not share one sampling rate. Human motion may arrive at 120 or 240 Hz, while G1 evaluation uses 200 Hz. Fixed-sample patches would map different physical durations to the same sequence length. Resampling to a common grid would introduce interpolation artifacts and could erase timing irregularities that are useful for robustness.

The model instead performs a learned hierarchical continuous convolution over native timestamps. Within a window starting at $\tau_w$ and lasting $\Delta$, feature $j$ at time $t_j$ receives normalized time $u_{w,j}=(t_j-\tau_w)/\Delta$. With linear B-spline kernels $\phi_k$, normalized kernel aggregation, learned weights $\mathbf{W}_{k}$, and bias $\mathbf{b}$:

$$\bar{\mathbf{h}}_{w,k}=\frac{\sum_{j\in\mathcal{J}_{w}}\phi_{k}(u_{w,j})\mathbf{h}_{j}}{\max\left(\epsilon,\sum_{j\in\mathcal{J}_{w}}\phi_{k}(u_{w,j})\right)},\qquad \mathbf{y}_{w}=\mathbf{b}+\sum_{k=1}^{K}\mathbf{W}_{k}\bar{\mathbf{h}}_{w,k}.$$

Normalization keeps the feature scale stable when a window contains a different number of samples. Time-based rotary position encoding then preserves elapsed time between the resulting tokens. Each token corresponds to roughly 125 ms of motion, but its construction follows the real measurement times instead of an assumed sampling grid.

src/model/tokenizer.py:249-303 implements the timestamp-aware tokenizer, and 339-423 contains the replay and decoder path. The ablation makes the engineering importance unusually clear: replacing continuous encoding with fixed-sample patches raises real-G1 ATE-GT from 3.153 m to 17.37 m. This is not a marginal architectural preference; it is the component that makes heterogeneous timing and physical duration comparable.

Alternating attention across body and time

The encoder has six transformer layers, width 768, and eight attention heads. Each layer alternates attention within a body site and attention across all sites and times. Site embeddings identify the twelve anatomical locations, while RoPE encodes elapsed time between tokens. The [CLS] token summarizes the motion for language alignment; four [REG] tokens support regression and denoising.

The ordering has a physical interpretation. Local attention first models the dynamics of one limb or sensor. Global attention then exchanges information between limbs and the torso. A model that immediately mixes every token can blur very different local dynamics, especially when some streams have fewer samples or temporary gaps. Alternating layers give the representation separate opportunities to learn within-site and cross-body structure.

flowchart LR
  A["Human real or virtual IMUs"] --> C["Canonical 12-site inertial interface"]
  B["G1 pelvis IMU plus joint encoders"] --> D["URDF transport of acceleration and angular velocity"]
  D --> C
  C --> E["Continuous convolution over native timestamps"]
  E --> F["125 ms tokens with time-aware rotary positions"]
  F --> G["Six alternating-attention layers width 768"]
  G --> H["Denoising head for clean acceleration and angular velocity"]
  G --> I["Per-site velocity head and pelvis root velocity"]
  G --> J["CLS embedding aligned to frozen CLIP text"]
  H --> K["Strapdown attitude integration"]
  I --> L["Trapezoidal trajectory integration"]
  K --> L
  J --> M["Motion-description retrieval"]
  L --> N["Zero-shot physical G1 odometry"]
    

The repository matches the paper's architecture. src/config/model/trunk/full_aa_transformer.yaml:1-10 specifies the six-layer, 768-wide alternating-attention trunk, and src/model/transformer.py:53-60 implements the switch between site-level and full attention. This is useful evidence for reproduction because the descriptive term “alternating attention” corresponds to an explicit code path rather than an inferred diagram.

A multi-task loss with different physical roles

The shared trunk is trained with four objectives: velocity estimation, short-horizon trajectory consistency, language alignment, and denoising. Their weighted sum is:

$$\mathcal{L}=\lambda_{\mathrm{vel}}\mathcal{L}_{\mathrm{vel}}+\lambda_{\mathrm{traj}}\mathcal{L}_{\mathrm{traj}}+\lambda_{\mathrm{lang}}\mathcal{L}_{\mathrm{lang}}+\lambda_{\mathrm{den}}\mathcal{L}_{\mathrm{den}}.$$

Velocity supervision is the mean Euclidean error between predicted and ground-truth body-frame velocity at every site. It is useful for immediate motion estimation but does not directly penalize accumulated displacement. The trajectory loss integrates predicted velocities over 0.25, 1, and 2 second intervals and compares the relative displacement with ground truth in the starting sensor frame:

$$\mathcal{L}_{\mathrm{traj}}=\frac{1}{|\mathcal{I}|}\sum_{(s,e)\in\mathcal{I}}\left\|\frac{\Delta\mathbf{p}_{se}^{\mathrm{pred}}-\Delta\mathbf{p}_{se}^{\mathrm{gt}}}{t_e-t_s}\right\|_2^2.$$

The integration has two stages. Denoised angular velocity is integrated with fourth-order Runge-Kutta to estimate sensor-to-world rotation, then world-frame velocity is integrated with the trapezoidal rule. Gradients through the rotation path are stopped, so the trajectory objective shapes velocity estimation without back-propagating through attitude reconstruction. This separates two failure modes: an inaccurate local velocity estimate and an accurate velocity that drifts after being rotated by a bad attitude estimate.

Language supervision uses a frozen CLIP ViT-L/14@336px text encoder. A motion window that is fully covered by one annotation is mapped to a normalized 768-dimensional target $\mathbf{z}$, and the [CLS] feature $\mathbf{f}_{\mathrm{Lang}}$ is optimized with cosine distance:

$$\mathcal{L}_{\mathrm{lang}}=1-\cos(\mathbf{f}_{\mathrm{Lang}},\mathbf{z}).$$

Partially covered windows are excluded from language loss but remain usable for the other objectives. This avoids attaching a whole-sentence label to a window that contains only half of the described action. The denoising target is gravity-free acceleration and clean angular velocity from the underlying kinematics, while the input receives gravity, sensor noise, vibration, impacts, and timing variation. The encoder must therefore preserve motion structure while discarding the disturbances used to corrupt it.

Velocity and trajectory weights are adapted from the gradient norm of each task relative to denoising at a shared feature layer, with exponential moving averages and clipping. Language and denoising weights are fixed at one. The code paths are visible in src/model/loss.py:257-303 for cosine language alignment, src/model/loss.py:576-621 for denoising, and src/model/integrator.py:246-269 for attitude and velocity integration.

Randomization bridges kinematics and hardware

Kinematic synthesis produces geometrically consistent but physically incomplete signals. It does not reproduce foot impacts, motor vibration, frame flex, magnetic disturbance, or a real accelerometer's bias behavior. The authors therefore randomize gravity magnitude, body scale, motion speed, human sampling rate, timestamp jitter, acceleration and gyroscope noise, vibration, and impacts. Scale and timing changes are applied consistently to inputs and targets, while denoising targets remain free of gravity and sensor noise.

Training uses 4096 ms windows with batch size 1024 and bfloat16 precision for 20 reported epochs. Muon updates matrix parameters at $2\times10^{-3}$, AdamW updates the remaining parameters at $5\times10^{-3}$, and the reported results use an EMA weight of 0.95. These details matter because the method combines many data sources; the claim is not that a larger generic transformer replaces physical modeling, but that synthesized geometry, native-time encoding, and randomized supervision train one model across domains.

Evidence ladder: from human data to a physical G1

The experiments use five domains: Nymeria, Natural Motion, BONES-Seed human, BONES-Seed retargeted G1, and a real-G1 set reserved entirely for evaluation. The real recordings total about 0.7 hours and are never used for training or fine-tuning. Velocity is measured by absolute velocity error, and trajectory accuracy by absolute trajectory error with either ground-truth attitude or gyro-integrated attitude.

$${\mathrm{AVE}}=\frac{1}{T}\sum_{t=1}^{T}\left\|\hat{\mathbf{v}}_{t}-\mathbf{v}_{t}\right\|_2,\qquad {\mathrm{ATE}}=\sqrt{\frac{1}{T}\sum_{t=1}^{T}\left\|\hat{\mathbf{p}}_{t}-\mathbf{p}_{t}\right\|_2^2}.$$

The strongest odometry result is not a single average but the pattern across domains. The full model has the lowest AVE on all five datasets: 0.026 m/s on Nymeria, 0.049 on Natural Motion, 0.031 on BONES human, 0.024 on retargeted G1, and 0.099 on real G1. On the physical robot, ATE-GT is 3.153 m. A humanoid-only model reaches 5.901 m, a pelvis-only input reaches 8.114 m, and pelvis plus joints reaches 3.513 m. The progression supports both human-to-humanoid transfer and the value of representing the extra signals as a common inertial interface rather than as an unstructured joint vector.

Two qualifications prevent overreading this table. First, the real-G1 ATE-Pred is 11.50 m while ATE-GT is 3.153 m. The velocity estimator is strong enough to produce a useful trajectory prior, but attitude drift still dominates long-horizon position error. Second, several baselines were trained separately per dataset, while the unified model shares weights. That makes the broad cross-domain result notable, but it does not eliminate differences in input count, tuning, and training budget.

Motion-language retrieval tests whether the shared representation carries semantics rather than only speed. IMU windows and natural-language descriptions are embedded into a common space and ranked by cosine similarity. On G1 categories seen during retargeted training, the model reaches 63.4% R@1 and 32.9% mAP. On unseen G1 categories that appear only on the human side during training, it still reaches 53.8% R@1 and 11.3% mAP, versus 17.3% and 0.5% for IMU2CLIP.

The unseen-category result is the more informative cross-embodiment evidence because the G1 side has no training examples for those motion classes. It suggests that language-aligned human motion features remain usable after retargeting and kinematic IMU synthesis. The result is not universal: IMU2CLIP has higher Nymeria recall at one, 16.2% versus 14.9%, while the unified model has a higher, but still low, mAP of 0.6% versus 0.4%. Keeping this exception visible matters because retrieval labels and dataset statistics differ, and a single headline metric would hide that the method is not uniformly better on every human benchmark.

Runtime and physical deployment

The hardware experiment tracks a G1 descending stairs and retrieves descriptions that change from walking to standing when the robot stops. This is a useful qualitative demonstration because it combines odometry with semantic output on the same physical sequence. The paper does not turn that demonstration into a closed-loop control experiment, so it should not be read as proof that the velocity estimate alone stabilizes stair descent.

Runtime is evaluated separately. On an NVIDIA L40S, TensorRT processes twelve 200 Hz IMUs over a four-second window in 1.78 ms. Without that optimization the model takes 9.23 ms. Since new measurements arrive every 5 ms at 200 Hz, the optimized path can keep pace with the input stream. The comparison with AirIO at 1.19 ms and TartanIMU at 3.84 ms must account for input scale: AirIO uses one IMU and a four-second window, while TartanIMU uses one IMU and a ten-second window. The appropriate conclusion is that the twelve-site encoder can be made real-time, not that it is cheaper than every single-IMU model.

Ablations identify the load-bearing assumptions

The ablations separate three effects that could otherwise be conflated with model capacity. Joint human and G1 training improves real-G1 ATE-GT from 5.901 m for a humanoid specialist to 3.153 m. Replacing the multi-site interface with pelvis-only input worsens it to 8.114 m, and giving the model pelvis plus joints only reaches 3.513 m. The architecture therefore benefits from both broader human coverage and an input construction that encodes robot geometry in sensor space.

The continuous-time encoder is the largest single design factor. Removing it raises real-G1 ATE-GT from 3.153 m to 17.37 m and retargeted-G1 ATE-GT from 0.101 m to 1.189 m. This fits the paper's premise: heterogeneous sampling and physical duration are not incidental preprocessing details. Alternating attention gives a smaller gain, with real-G1 ATE-GT changing from 3.153 m to 3.530 m when removed, so the evidence supports it as a useful but not equally decisive component.

The data-diversity study complicates a simple “more data is always better” story. Real-G1 error improves from 4.369 m at one sixty-fourth of the data to 3.716 m at one quarter and 3.153 m at the full set. Retargeted G1 is best at one quarter, 0.099 m, slightly ahead of the full-data 0.101 m. The authors interpret this as evidence that broader human motion benefits hardware transfer more than same-domain retargeted accuracy. It is a plausible explanation, but the non-monotonic result also shows that data allocation remains an empirical tuning problem.

The scaling analysis fits a Chinchilla-style relation $L(N,D)=E+A/N^{\alpha}+B/D^{\beta}$, where $L$ is shared-site validation velocity error, $N$ is parameter count, and $D$ is processed tokens. The fit yields optimal parameter count proportional to $C^{0.52}$ and optimal token count proportional to $C^{0.48}$ under the compute proxy $C=6ND$. Nearly equal exponents argue for scaling capacity and exposure together. The fitted surface includes extrapolation and optimizes validation error, so it is guidance for compute allocation rather than a guarantee of best hardware transfer.

What the evidence supports, and what it does not

The authors explicitly limit hardware coverage to one humanoid platform. A successful G1 result does not establish the same performance on robots with different leg proportions, foot geometry, sensor mounting, or control behavior. The kinematic mapping contains platform-specific assumptions, so each new humanoid requires at least a new URDF interface, mounting calibration, and validation of the twelve-site correspondence.

The authors also state that language supervision is partly synthetic and has a limited vocabulary. The model performs well on the annotated motion categories, but real user descriptions can include direction, speed, object interaction, intent, and compound events that may not appear in the training captions. The unseen-category retrieval result demonstrates transfer beyond G1 training labels; it does not demonstrate open-ended language understanding.

Orientation drift is the third limitation acknowledged by the paper. The gap between 3.153 m ATE-GT and 11.50 m ATE-Pred on real G1 shows that denoised gyroscope integration remains a major error source. A deployable system would still need filtering, external localization, contact constraints, or visual measurements over longer horizons. The learned encoder supplies a velocity prior; it does not replace the full state-estimation stack.

A further independent limitation is the physical fidelity of kinematic synthesis. Contact impulses, joint compliance, frame vibration, magnetic disturbances, and sensor faults are not reproduced by URDF kinematics. Randomization covers a chosen family of perturbations, but the real-G1 evidence is only about 0.7 hours and does not include long-duration closed-loop navigation. The paper demonstrates zero-shot transfer under its hardware protocol; it does not prove unrestricted cross-platform robustness.

Conclusion

The contribution is best understood as a sensing-level interface rather than a single odometry network. URDF transport converts one pelvis IMU and joint encoders into twelve virtual inertial sites. A fixed coordinate correction makes human and G1 axes comparable. Continuous convolution preserves physical time, alternating attention models local and whole-body dynamics, and the multi-task loss couples velocity, trajectory, denoising, and language semantics. Each component addresses a specific mismatch in the human-to-humanoid transfer problem.

The evidence is strongest where the paper makes its narrowest claims. Human data improves both retargeted and physical G1 trajectory estimates, the full model achieves 0.099 m/s AVE on real G1 without real-robot training or fine-tuning, and G1 motion categories absent from robot training can still be retrieved through human-side language supervision. The next meaningful tests are multiple humanoid platforms, longer closed-loop deployments, stronger attitude estimation, and language data that is less synthetic and more diverse.

Golden quote: What transfers is not a fixed humanoid motion distribution, but an inertial interface through which human and robot motion can become mutually legible.