PAPER DEEP DIVE
Dense Temporal Motion Retargeting for Legged Robots
DTMR jointly optimizes reference phase increments and PD joint targets, deforming motion only where robot dynamics require it. Across 999 clips it improves retargeting precision while running about 19–21 times faster than STMR.
One-Sentence Summary
DTMR treats the increment of reference-motion phase as an additional control input, optimizes it together with PD joint targets through GPU-parallel annealed MPPI, and therefore changes timing only where the robot cannot physically follow the source motion.
Why This Problem Matters
Human motion capture provides a broad and semantically rich source of demonstrations for humanoid robots. A captured walk, jump, or dance contains coordination patterns that would be difficult to author by hand. The source human and the target robot nevertheless differ in limb length, mass distribution, joint limits, actuator strength, and contact dynamics.
A conventional pipeline first retargets body keypoints or joint angles to the robot and then asks a whole-body controller to track the resulting trajectory. Geometry is handled in the first stage, while dynamics are handled in the second. The source time axis is usually left unchanged throughout this process.
That fixed time axis creates a specific failure mode. If a robot needs extra time to crouch before a high jump, a tracker cannot manufacture the missing preparation by increasing gains. It either falls behind the reference or sacrifices another part of the body. Many failures are therefore not errors in target pose, but errors in when the target becomes reachable.
Existing temporal retargeting methods can rescale a complete clip or treat static and dynamic segments differently. Such changes improve feasibility, but they distribute the modification too broadly. A single difficult takeoff may force the entire clip to become slower, and a dance may lose the beat even though only a few frames required adjustment.
DTMR starts from a different premise: timing should be resolved densely over the trajectory and jointly with motor control. The phase increment can vary at every control step, so a difficult local transition receives more time while an easy section can retain its original pace. Timing becomes part of the physical search rather than a preprocessing label.
Method Preliminaries
The controller builds on model predictive control with path-integral-style sampling. At each control step, it samples perturbations around a nominal future plan, rolls the candidates through a dynamics model, scores them, and updates the nominal plan using a cost-weighted average.
The core update is the MPPI rule below. Low-cost trajectories receive larger weights, but the exponential transform keeps several plausible candidates in the update instead of committing to one sampled extremum.
$$\mathbf{U}\leftarrow\mathbf{U}+\sum_{n=1}^{N_W}w_n[W]_n,\qquad w_n=\frac{e^{-J_n/\lambda}}{\sum_m e^{-J_m/\lambda}}$$
Here $\mathbf{U}$ is the nominal plan, $[W]_n$ is one sampled noise sequence, $J_n$ is its rollout cost, and $\lambda$ controls the softness of the weighting. GPU execution is essential because thousands of independent physics rollouts are evaluated for every control step.
The reference motion is represented by body keypoints $\bar{\mathbf{p}}$, and a phase variable selects the portion of that motion currently being requested. Advancing the phase at the source rate reproduces source timing. Advancing it faster or slower changes local tempo while preserving the same underlying motion path.
How DTMR Works
1. The original problem searches controls and phase separately
The robot state is $\mathbf{x}_k=(\mathbf{q}_k,\dot{\mathbf{q}}_k)$, where $\mathbf{q}_k$ contains the root pose and joint angles. The state evolves under dynamics $f$ and motor control $\mathbf{u}_k$. A phase trajectory determines which reference frame the robot should match at each step.
A direct temporal retargeting objective can be written as follows:
$$\min_{\mathbf{u},\boldsymbol{\phi}}\frac{1}{2T_\phi}\sum_{k=0}^{T_\phi}\left\|\mathrm{FK}(\mathbf{x}_k)-\mathrm{LI}(\phi_k;\bar{\mathbf{p}})\right\|_Q^2$$
The problem is constrained by robot dynamics and a monotone phase path:
$$\mathbf{x}_{k+1}=f(\mathbf{x}_k,\mathbf{u}_k),\quad\phi_0=0,\quad\phi_k\leq\phi_{k+1},\quad\phi_{T_\phi}=1$$
$\mathrm{FK}$ denotes forward kinematics, $\mathrm{LI}$ interpolates reference keypoints at the selected phase, $Q$ weights the tracking error, and $T_\phi$ is the number of control steps needed to reach the end. This formulation is conceptually clean, but it leaves phase as a separate trajectory that is awkward to sample inside a standard motor controller.
2. Phase increments become part of the control vector
DTMR converts the bilevel-looking problem into a single augmented control problem:
$$\tilde{\mathbf{x}}_k=(\mathbf{x}_k,\phi_k),\qquad\tilde{\mathbf{u}}_k=(\mathbf{u}_k,d\phi_k),\qquad\tilde{\mathbf{x}}_{k+1}=\tilde{f}(\tilde{\mathbf{x}}_k,\tilde{\mathbf{u}}_k)$$
The robot part of the state follows the original dynamics, while the phase advances according to $\phi_{k+1}=\phi_k+d\phi_k$. Each sampled control now answers two questions at once: which joint targets should the robot execute, and how much of the reference should it consume during the next interval?
To make tempo changes comparable across clips, the paper defines a deformation rate relative to source speed:
$$r_k=\log_2(d\phi_k/d\phi^{\mathrm{src}}),\qquad\langle|r|\rangle=\frac{1}{T_\phi}\sum_k|r_k|$$
A value of $r_k=-1$ means the local motion is twice as slow as the source, while $r_k=1$ means it is twice as fast. The average absolute rate $\langle|r|\rangle$ summarizes how much timing was changed over a complete motion.
3. Tracking and temporal deformation share one running cost
After state augmentation, the trajectory objective becomes:
$$\min_{\tilde{\mathbf{u}}}\frac{1}{T_\phi}\sum_{k=0}^{T_\phi}c(\tilde{\mathbf{x}}_k,\tilde{\mathbf{u}}_k)$$
The cost separates tracking, timing, and regularization:
$$c(\tilde{\mathbf{x}}_k,\tilde{\mathbf{u}}_k)=c_{\mathrm{track}}+w_{d\phi}c_\phi+c_{\mathrm{reg}}$$
The tracking term compares root position, link position, link orientation, joint velocity, and torque behavior against the reference selected by the current phase. The phase term penalizes temporal deformation, and the regularizer discourages abrupt or high-energy controls. The scalar $w_{d\phi}$ determines how strongly source tempo is protected.
This cost has an important practical consequence. The controller does not need a classifier that labels an action as static or dynamic. At each step it compares the physical tracking benefit of slowing down against the timing cost. A high jump may retain speed during flight and slow down only before takeoff, when the robot needs more time to load its legs.
4. Sampling is compressed to control nodes
Independently sampling every joint target and phase increment at every horizon step would create a very high-dimensional search space. DTMR instead samples a small set of control nodes and expands them into the full horizon. The main experiment uses a horizon of 40 control steps and only 8 nodes, while evaluating 4096 parallel rollouts per annealed iteration.
The sampling variance follows a schedule inherited from DIAL-MPC:
$$\sigma_{i,h}=\sigma_0\beta^{i-1}\beta_h^{H-h}$$
Later iterations search closer to the current plan, and near-term controls are refined more aggressively than distant ones. The work also applies robust norms to tracking residuals so that one large component does not dominate the ranking of an otherwise useful rollout:
$$\rho(\mathbf{e})=\sum_i\left[(|e_i|^q+\delta^q)^{1/q}-\delta\right]$$
Root and link errors use different values of $\delta$ and $q$. The root is treated more like a direct spatial constraint, while link orientation and position receive a smoother robust penalty. This distinction matters because a small root drift often changes the global task more than a local limb deviation.
5. Rollouts advance phase before physics
The initial plan is built from the reference joint trajectory and the source phase speed. For each sampled control sequence, the controller advances phase first, sets the selected PD target, performs the physics step, and evaluates the tracking cost at the newly selected reference frame. It then updates the plan with annealed MPPI, executes only the first control, and shifts the warm-start plan forward.
The public implementation exposes the same ordering. The controller extracts the final action dimension as the phase-rate command, updates phase, launches control and physics, and evaluates plan cost:
for t in range(H):
extract_rdphi(actions, rdphi, t)
phase_update(rdphi, phi, dphi, frame_idx, frame_alpha)
launch_ctrl(state, actions, t)
step_physics(state, actions, t)
launch_plan_cost(state, t)
This logic is implemented in src/dtmr/tracker/controller.py:327-395. Getting the ordering wrong would compare a robot state to a reference frame selected one step later, which is especially harmful around contact events.
6. The released implementation has two reproducibility details
The paper describes linear interpolation between control nodes, but the current repository builds the interpolation matrix with a quadratic spline: make_interp_spline(..., k=2) in src/dtmr/tracker/splines.py:10-17. A quadratic spline can produce smoother transitions between nodes, but the paper does not include an ablation isolating this implementation choice.
The repository also contains debug defaults that are smaller than the paper's main evaluation setting of 4096 samples, 40 horizon steps, and 8 nodes. Results reported in the paper should therefore be reproduced with the release configuration and table values, not by assuming that every code default corresponds to the final experiment.
7. Dense timing is produced by receding-horizon feedback
Only the first action of each optimized plan is executed. The next state is simulated or measured, and the entire phase-control problem is solved again. This receding-horizon loop lets timing respond to deviations: if a landing impact changes the robot's vertical velocity more than expected, the next phase increment can allocate more time to recovery.
The cost also determines where deformation is concentrated. Spreading a small slowdown over many easy frames may be cheaper than making one difficult frame impossible to track, but the reverse can occur when a local transition dominates the failure. The paper's high-jump example demonstrates the second case: the robot delays takeoff and shortens flight instead of uniformly stretching the entire approach.
The following diagram summarizes the paper-specific rollout loop rather than a generic model pipeline.
flowchart LR
A[Source keypoint motion] --> B[Phase phi in 0 to 1]
B --> C[Plan nodes: PD targets + dphi]
C --> D[Sample 4096 controls]
D --> E[Advance phase before physics]
E --> F[MuJoCo Warp rollout]
F --> G[Tracking + timing + torque cost]
G --> H[Annealed MPPI elite update]
H --> I[Execute first control]
I --> J[Shift warm start plan]
J --> D
8. Why joint optimization changes the result
If timing is fixed first, a controller can only report that a rapid motion is infeasible. If timing is optimized separately, the resulting curve may not match what the motor controller can realize. DTMR compares timing and joint targets inside the same physics rollout, so the selected phase rate reflects the actual cost of following it.
This explains why the method can improve both feasibility and semantic tracking. A dynamic segment receives the extra time needed to reduce large pose errors, while a static segment can remain close to source speed. Timing is used as a continuous resource that lowers physical cost, not as an instruction to slow down uniformly.
Experiment Design
The authors retarget a two-hour human-motion dataset into 999 clips, including 496 locomotion clips. Humanoid experiments cover Unitree G1, R1, and H1-2 as well as Booster T1. A smaller quadruped evaluation uses Unitree Go1. The resulting references are evaluated in simulation and then used to train policies for real-robot deployment.
The paper reports two complementary errors. Phase-aligned retargeting error compares the robot and reference at the same normalized time, so it exposes failures caused by physical infeasibility. Dynamic time warping error first allows a nonlinear temporal alignment, then measures shape mismatch. The first metric asks whether the motion happens on schedule; the second asks whether its semantic structure survives.
The tracking-cost weights are listed below. They are not a single global scale factor: root position, links, orientation, velocity, phase, torque, and action smoothness have separate roles.
| Cost term | Weight | Purpose |
|---|---|---|
| Root position | 28 | Preserve global translation |
| Link position | 14 | Track whole-body keypoints |
| Link orientation | 5 | Preserve limb orientation |
| Joint velocity | 0.01 | Light smoothness constraint |
| Phase deformation | 5 | Control timing changes |
| Torque energy | 0.05 | Discourage excessive effort |
| Action rate | 1.0 | Reduce target jumps |
The optimization settings below come from the paper's principal experiments. They are important because runtime and success rates change substantially with the number of rollouts, horizon length, and annealing iterations.
| Parameter | Value | Parameter | Value |
|---|---|---|---|
| Samples | 4096 | Horizon | 40 steps |
| Control nodes | 8 | Cold / warm annealing | 5 / 2 iterations |
| Initial noise | 0.15 | Iteration decay | 0.6 |
| Horizon envelope | 0.95 | Sampling temperature | 0.3 |
| Elite fraction | 0.1 | Phase noise | 0.3 |
| Root robust norm | 0.1 m, q=2 | Link robust norm | 0.2 m, q=4 |
| Control period | 20 ms | Physics step | 5 ms |
Results
1. Dynamic feasibility improves on all four humanoids
DTMR is applied on top of SoMa-RT and compared with GMR, PHUMA, OmniRetarget, and the original SoMa-RT output. The median phase-aligned errors improve on every humanoid: G1 falls from 103.7 mm to 81.4 mm, R1 from 83.9 mm to 77.6 mm, H1-2 from 151.8 mm to 136.3 mm, and T1 from 132.0 mm to 112.8 mm.
| Platform | SoMa-RT median error | DTMR median error | Relative improvement |
|---|---|---|---|
| Unitree G1 | 103.7 mm | 81.4 mm | 21.5% |
| Unitree R1 | 83.9 mm | 77.6 mm | 7.5% |
| Unitree H1-2 | 151.8 mm | 136.3 mm | 10.2% |
| Booster T1 | 132.0 mm | 112.8 mm | 14.5% |
| Four-platform average | 13.4% |
The variation across robots is informative. R1 already has relatively low baseline error, so there is less room to improve. G1 and T1 gain more from local timing changes, suggesting that actuator limits and morphology alter where physical infeasibility appears.
2. Most gains come from the hardest quartile
The lower quartile improves by only 1.8% on average, while the median improves by 13.4% and the upper quartile by 14.9%. Easy clips therefore remain close to their original timing, while difficult clips receive the largest benefit. This pattern is inconsistent with a method that simply slows every motion by a constant factor.
Dynamic time warping error shows the same direction. Median warped error improves by 12.9% across the four humanoids. Since this metric already tolerates nonlinear time alignment, the improvement indicates that DTMR preserves motion shape while changing local tempo rather than replacing semantic structure with a smoother but different action.
The dance study exposes the tuning tradeoff. A larger phase weight protects the source rhythm, while a smaller weight allows more deformation. When $w_{d\phi}$ decreases from 5 to 0.5, average absolute deformation increases from 0.20 to 0.23 and error drops by about 30 mm. At weight 10, error is about 202 mm; at 0.5, it is about 170 mm, with the range 164 to 170 mm for values at or below 0.5.
Figure 1: Paper Figure 4. Larger phase weights preserve source tempo, while smaller weights permit more timing deformation. Image from the paper.
3. DTMR is more precise under the same deformation budget
To separate the benefit of dense timing from the benefit of unrestricted deformation, the authors compare against STMR using the same deformation budget. DTMR is about 22% more precise. On ten G1 clips, total runtime falls from 45.4 hours to 2.4 hours, about 19.2 times faster; average time per clip falls from 4.5 hours to 14 minutes, about 18.9 times faster. Two Go1 clips fall from 2.4 hours to 7 minutes, about 20.8 times faster.
| Setting | Baseline | DTMR | Outcome |
|---|---|---|---|
| Ten G1 clips, total | 45.4 h | 2.4 h | About 19.2x faster |
| Average G1 clip | 4.5 h | 14 min | About 18.9x faster |
| Two Go1 clips | 2.4 h | 7 min | About 20.8x faster |
| Same deformation budget | STMR | DTMR | About 22% more precise |
The speedup comes from solving a single sampling problem instead of repeatedly optimizing a separate temporal trajectory. The complete two-hour dataset is estimated to require about 200 GPU hours, or 80 to 100 times the motion duration. This remains a substantial compute cost, but it shifts large-scale dataset generation from impractical to feasible on a well-equipped machine.
4. Jump and flip clips reveal the effect of localized timing
Table V gives the clearest failure cases. The third jump clips improve from 556.0 mm to 82.2 mm, the flip from 1213.6 mm to 111.6 mm, and the first catch from 165.3 mm to 55.3 mm. These motions require large center-of-mass changes and brief flight phases; when timing is fixed, tracking error accumulates before landing and cannot be recovered.
| Motion | STMR error | DTMR error | Change |
|---|---|---|---|
| Jump-3 | 556.0 mm | 82.2 mm | 85.2% lower |
| Flip | 1213.6 mm | 111.6 mm | 90.8% lower |
| Catch-1 | 165.3 mm | 55.3 mm | 66.5% lower |
| Side steps | 83.2 mm | 76.7 mm | 7.8% lower |
| Brushing | 11.6 mm | 12.3 mm | 6.0% higher |
Static actions do not improve monotonically. Brushing increases slightly, showing that additional timing freedom can introduce small deviations when the original timing was already suitable. The result supports a nuanced conclusion: DTMR is valuable for difficult dynamics, but not every clip should be expected to benefit.
Figure 2: Paper Figure 1. DTMR delays the jump preparation and reduces flight time while preserving the recognizable high-jump structure. Image from the paper.
Figure 3: Paper Figure 3(a). STMR fails during the high jump, while DTMR succeeds after changing takeoff timing. Image from the paper.
Figure 4: Paper Figure 3(b). STMR loses the landing, while DTMR redistributes flight and landing timing. Image from the paper.
5. Pelvis timelines show that timing is changed locally
Figure 5 uses pelvis height over time for a static catch. STMR applies a broader timing deformation, while DTMR remains closer to the source rhythm and deviates mainly when balance requires it. Figure 6 shows the opposite case: the high jump is delayed before takeoff and then compressed during flight.
Figure 5: Paper Figure 6(a). Pelvis timelines for a static catch show that DTMR does not uniformly stretch the easy portion. Image from the paper.
Figure 6: Paper Figure 6(b). The high-jump timeline changes takeoff and landing timing while preserving local flight structure. Image from the paper.
These plots matter because they connect the optimization objective to an observable behavior. The controller returns toward source speed when tracking is affordable and changes phase more aggressively when the physical constraints require it. The average error reduction is therefore produced by a sequence of local timing decisions.
6. Real-robot deployment provides an initial execution check
The authors train policies on DTMR references and deploy them on a Unitree G1. Tracking reward improves from 40.8 to 42.5 for a foot-shuffle dance and from 38.9 to 41.7 for a weight-shift dance. In the second motion, the arm is slowed to 0.84 times the source speed, allowing the robot to maintain the broader rhythm while avoiding an arm motion it cannot execute comfortably.
The real-robot evidence is encouraging but not exhaustive. The paper does not establish long-term reliability across many robots or environments. It does show that the timing changes survive the transition from simulation to actuated hardware and do not only fit a narrow simulator state distribution.
Implementation and Reproducibility
The repository supports several humanoids and quadrupeds and provides an end-to-end path from imported motion to tracking, scoring, replay, and video rendering. The code's phase exponent base is set to 2.0, matching the paper's definition of deformation rate in log base two.
dphi = (dphi_exp_base ** rdphi) * dt / motion_duration
This corresponds to src/dtmr/env/kernels.py:745-764, where the last action dimension is restored to a phase increment. The controller then advances phase before the physics step. A reimplementation that updates phase after physics will evaluate the tracking cost against the wrong source frame.
The code also captures the rollout sequence in a CUDA graph to reduce launch overhead. That engineering choice is part of why thousands of parallel rollouts are usable in practice. At the same time, the quadratic interpolation and debug defaults should be documented when comparing runtimes or reproducing a particular table entry.
Discussion
The main methodological contribution is a change in what counts as a control resource. Timing is not a fixed label attached to a reference clip; it is a variable that determines whether the robot can satisfy the next physical state. This is especially appropriate for legged systems, where contact timing and actuator saturation are central to feasibility.
The work also changes how robot motion datasets can be produced. Instead of copying human timing and hoping a tracker can follow it, the pipeline can generate references that already contain physically informed local re-timing. That may improve downstream imitation learning because the learner receives trajectories closer to the controller's reachable set.
Physical feasibility is still not semantic correctness. A motion can be dynamically stable and still be wrong for the task. Music-driven dance, human-robot collaboration, and object manipulation all contain event constraints that cannot be inferred from tracking cost alone.
Limitations
The authors explicitly note that DTMR does not know which parts of a motion must remain semantically fixed. A dance can become physically easier while losing the musical beat. The phase weight limits the amount of deformation, but it does not encode that a specific footfall, gesture, or accent must occur at a specific time. Event or musical constraints would be needed for those tasks.
The authors also leave object interaction open. Catching, placing, or throwing requires the robot and object to reach a contact state simultaneously. Retiming the robot alone can break the object's trajectory or contact schedule. A future formulation should optimize robot phase, controller, object state, and contact events together.
The method is compute-intensive. A complete two-hour dataset requires roughly 200 GPU hours, and each robot or contact model may require new tuning. This is much better than the baseline pipeline, but it is still a barrier for groups without parallel physics simulation.
Tracking weights determine which body parts are allowed to absorb error. A high root weight preserves global translation but may push error into the arms or joints. The paper studies several weight settings, yet real tasks still require a human decision about which errors are acceptable.
Finally, the public code and paper differ in interpolation and some default parameters. The quadratic spline may be preferable, but it is not isolated by an ablation. Exact reproduction should record the repository commit, release configuration, and whether CUDA graph execution is active.
Conclusion
DTMR makes reference phase an explicit control input and optimizes it with PD targets inside one GPU-parallel physics rollout. This allows the controller to slow down only the segments that require more time, speed up easy segments, and preserve the overall motion structure. The method is not merely a post-hoc smoother; it changes the formulation of temporal motion retargeting.
Across four humanoids, the paper reports lower phase-aligned and time-warped error, particularly on difficult dynamic clips. Under the same deformation budget, DTMR is more precise than a separate temporal baseline and roughly nineteen times faster. Real-robot experiments provide an initial execution check.
The next step is to make timing semantically aware. Music beats, object contacts, and task events should become explicit constraints around the physically optimized phase. If that happens, dense temporal retargeting can move from making a motion physically possible to making it happen at the moment the task requires.
Golden line: A robot often does not need a different pose; it needs the correct pose placed on a physically reachable timeline.



