Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching

Paper Metadata

Title: Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching (abbreviated PHP throughout)

Authors: Zhen Wu*, Xiaoyu Huang*, Lujie Yang* (equal contribution), Yuanhang Zhang, Xi Chen, Pieter Abbeel†, Rocky Duan†, Angjoo Kanazawa†, Carmelo Sferrazza†, Guanya Shi†, C. Karen Liu† (* equal contribution, † co-lead of the Amazon FAR team; affiliations: Amazon FAR, UC Berkeley, CMU, Stanford University)

Links: arXiv:2602.15827 (v1 2026-02-17, v2 2026-05-06, RSS 2026) · arXiv HTML full text · project page · video

Code status: released. github.com/amazon-far/php_parkour ships the pipeline in three parts: motion_matching/ is the motion-matching data-generation package (G1 skeleton, skill registry, scenes, visualizer); wbt_training/ holds the IsaacSim teacher-training and student-distillation configurations against a pinned amazon-far/holosoma submodule; and scripts/download_assets.py fetches the motion library, the example motion/terrain datasets and the sanitized depth_backbone.onnx + student.onnx, verifying each artifact against the SHA-256 manifest in release-assets.json so the released pair can be run directly in MuJoCo for sim-to-sim checks. The raw .pt files and the teacher checkpoints are deliberately withheld; a community mjlab port exists.

Background and Motivation

Bipedal locomotion on humanoids has largely stopped being news: blind walking handles slopes and stairs, and once perception is added the robot can pick sparse footholds and cross gaps. What remains open is agility — the human capacity to select and chain whole-body dynamic skills rapidly enough to cross terrain whose scale varies by an order of magnitude within a few strides. The authors adopt parkour as a self-contained testbed for exactly that capacity, and the choice is pragmatic rather than decorative: parkour saturates three axes at once (dynamic intensity, coupling to perception, and number of distinct skills), and relaxing any one of them means the goal has not actually been met.

The paper separates parkour into three entangled difficulties. First, the skills themselves are highly dynamic and contact-rich: climbing a wall at or above the robot's own height, or vaulting an obstacle within a few tenths of a second, both demand effective control inside the very high-dimensional action space of a humanoid. Second, these skills must be tightly coupled to exteroception, otherwise the robot cannot adapt to a changed environment or react quickly to an unexpected disturbance. Third, leaving isolated movements behind and running a whole obstacle course requires compressing many high-dynamic skills into a single visuomotor policy, and that requirement grows harder fast as the skill set widens.

Human motion data is a prerequisite for this class of behaviour: jumping, rolling and flipping cannot realistically be written down as reward shaping. Yet highly dynamic data is intrinsically scarce. Capturing fast, contact-rich movement needs dedicated rigs and careful curation, and the resulting datasets typically carry one or two demonstrations per skill, a few seconds each. Scarcity of this kind is not specific to parkour; it applies to every dynamic human skill. Long-horizon tasks are awkward precisely because they need both ends of the spectrum at once: enough variation within a skill to cope with different approaches, and smooth, natural transitions between skills. A seconds-long isolated clip supplies neither.

Existing routes handle both ends poorly. On quadrupeds, parkour can be learned from scratch by reward shaping, but the recipe does not carry over to humanoids, where whole-body control makes the dimensionality explode; consequently most perceptive humanoids today stay on low-dynamic terrain — stairs, sparse footholds, low platforms. To keep exploration tractable the dominant pattern is teacher-student: train an expert on privileged state, then distil a vision student with DAgger. This paper keeps that pattern but reports that pure DAgger is not enough for high-dynamic humanoid skills, and therefore augments it with an RL objective. That is a different use of RL from Extreme Parkour, which first distils a strong DAgger policy and then fine-tunes it separately to reach unseen terrain. AMP takes another road: a single policy learns a distribution over skills and transitions emerge from RL exploration, at the cost of replacing hand-written rewards with a style reward learned from motion data. It works in animation and on quadrupeds, but humanoid hardware demonstrations so far are confined to less agile skills such as walking, stepping and box carrying. A further family synthesizes in-between transitions online with a generative kinematic model (MDM being the canonical example) and hands them to a DeepMimic-style tracking controller; the trouble is that these models degrade markedly in the low-data regime parkour lives in, and recovering quality requires either expensive iterative co-training or receding-horizon replanning — and replanning with perception in the loop is costly in real time.

The choice made here is to use motion matching as the source of kinematic references. The technique is mature in games and character animation, and it is simple, controllable and high-quality, yet in robotics it had previously been applied only to comparatively simple quadruped behaviours. The authors' argument is that retrieval plus stitching densifies a sparse motion library: it produces varied transitions across approach distances, headings and timings while preserving the realism of the captured clips themselves. Three contributions follow from that premise — an efficient pipeline for composing kinematic skills, a scalable training framework that distils several experts into one visuomotor policy with seamless transitions, and a depth policy that transfers zero-shot to a physical humanoid and performs high-dynamic parkour over multiple obstacle types.

Preliminaries

Motion matching. The technique was designed for interactive character control. At every frame, or at designated transition points, it selects from a large motion database the single frame whose motion feature best matches the current pose together with the intended future behaviour, then moves the playhead there and continues forward playback. Only three ingredients are involved: a feature space, one nearest-neighbour query, and a short blend at the switch. Nothing is learned and nothing is optimized, which is why the method is nearly indifferent to dataset size — the price is that features must be hand-designed and transition windows hand-annotated.

Reference-based reinforcement learning. DeepMimic-style tracking rewards express deviation from a reference motion as a set of Gaussian kernels, measuring separately the position, orientation, linear velocity and angular velocity error of tracked body keypoints. BeyondMimic and OmniRetarget turn that idea into engineering: the former supplies the reward, termination and randomization recipe for humanoid tracking, the latter retargets human motion onto a concrete robot skeleton. The expert training in this paper adopts both settings as-is and departs from them in exactly one place, described below, where global tracking is switched on so that recovery behaviour becomes learnable.

Teacher-student distillation. Simulation offers quantities that hardware does not, cheaply: the global pose of the pelvis and a terrain height map. The standard move is therefore to train a high-quality expert under privileged observation, then let a student that sees only proprioception and depth query that expert repeatedly through DAgger, cloning its behaviour state by state. The structural weakness of DAgger is that its loss is stepwise imitation and never looks at whether the episode as a whole succeeded. That weakness is exactly what this paper sets out to repair.

Method

The system is a three-stage modular pipeline, laid out in Fig. 2 of the paper. Motion matching composes atomic parkour skills with walking and running into long-horizon kinematic reference trajectories; each skill then receives its own single-skill teacher trained by RL motion tracking under privileged information; finally the teachers are distilled into one depth-based student through a hybrid DAgger-and-RL objective, and that student is deployed zero-shot on hardware. Each stage is replaceable on its own, which is what the authors mean by modular.

flowchart TB
    A["Human mocap clips"] --> B["OmniRetarget to 29-DoF Unitree G1"]
    B --> C["Atomic skill library + locomotion database
each clip paired with a box terrain"]
    C --> D["Manual annotation: start s_k, end e_k, entry window H_k"]
    D --> E["Motion matching synthesis
nearest-neighbor search in 27-D feature space"]
    E --> F["Long-horizon references: locomotion to skill to locomotion
paired velocity commands + terrain variants"]
    F --> G["Per-skill expert policies
privileged state + 0.7 m x 0.7 m height scan
adaptive sampling of failure regions"]
    G --> H["Distillation: DAgger + PPO hybrid loss
curriculum on lambda_D, termination relaxed 0.5 m to 1 m"]
    H --> I["Single depth-based multi-skill student
3-layer CNN + 5-layer MLP, 16384 envs, 20K iters"]
    I --> J["Zero-shot sim-to-real on physical G1
depth camera 30 Hz + discrete 2D velocity command"]
    J --> K["Autonomous skill selection: step over, climb on, vault, drop off
closed-loop adaptation to displaced obstacles"]
    F -.-> L["Randomization: approach distance, obstacle width up to 1.5 m,
dims +-5 cm, yaw +-45 deg, distractor boxes"]
    L -.-> H

Fig. 1: the three-stage PHP pipeline, drawn from Fig. 2 and Section III of the paper. The left data branch produces motion-terrain pairs by motion matching, the centre holds the privileged experts, the right the depth student obtained from the hybrid objective; dashed edges are the randomization terms that cut across both data and training.

1. Feature Space and Constrained Nearest-Neighbour Retrieval

Every clip is first retargeted with OmniRetarget onto the 29-degree-of-freedom G1 and stored frame by frame as a robot configuration $\bm{q}_{i}=(\bm{p}_{i},\bm{r}_{i},\bm{\theta}_{i})$, where the root translation is $\bm{p}_{i}\in\mathbb{R}^{3}$, the root quaternion $\bm{r}_{i}\in\mathbb{R}^{4}$ and the joint angles $\bm{\theta}_{i}\in\mathbb{R}^{29}$. Each frame additionally carries a precomputed matching feature $\bm{x}_{i}\in\mathbb{R}^{27}$ expressed entirely in the character's own frame and assembled from three blocks: a future root trajectory $\bm{t}_{i}\in\mathbb{R}^{12}$ giving planar position and heading at 0.33 s, 0.67 s and 1 s ahead; local foot state $\bm{f}_{i}\in\mathbb{R}^{12}$, the positions and linear velocities of both feet in the root frame; and root velocity $\bm{h}_{i}\in\mathbb{R}^{3}$. Coverage is widened by mirroring every clip, and each parkour clip is paired with a hand-fitted box terrain aligned to it.

At run time the query feature $\hat{\bm{x}}_{t}$ is built from the current configuration $\bm{q}_{t}$ plus a 2D velocity command: foot state and root velocity are read straight off the current state, while the future-trajectory block is computed from the command. Retrieval is then a single constrained nearest-neighbour search,

$$i_{t}^{\star}=\arg\min_{i\in\mathcal{C}_{t}}\;\|\hat{\bm{x}}_{t}-\bm{x}_{i}\|^{2}, \tag{1}$$

in which $\mathcal{C}_{t}$ is the search window associated with the skill the user has asked to be executed next. The query runs every $M$ frames, and is additionally fired immediately whenever the commanded velocity changes significantly. Once $i_{t}^{\star}$ is found the playhead jumps to that frame and runs forward until the next query; a short blend window is applied at the switch so the output stays continuous.

Turning a 2D command into a future trajectory is done with a critically damped spring. Writing the spring position as $\bm{s}$, its velocity as $\dot{\bm{s}}$, the goal as $\bm{s}_{\text{goal}}$ and the damping parameter as $y>0$, and defining $\bm{j}_{0}=\bm{s}_{0}-\bm{s}_{\text{goal}}$ together with $\bm{j}_{1}=\dot{\bm{s}}_{0}+y\,\bm{j}_{0}$, the state at any future time $\tau$ has the closed form

$$\bm{s}(\tau)=e^{-y\tau}\big(\bm{j}_{0}+\tau\bm{j}_{1}\big)+\bm{s}_{\text{goal}}. \tag{2}$$

Planar displacement uses the same spring in velocity space — the spring position stands for planar velocity and its derivative for acceleration — and the closed-form velocity is integrated to give the future root position

$$\bm{p}(\tau)=\bm{p}_{0}-\frac{\bm{j}_{1}}{y^{2}}e^{-y\tau}+\frac{-\bm{j}_{0}-\tau\bm{j}_{1}}{y}e^{-y\tau}+\frac{\bm{j}_{1}}{y^{2}}+\frac{\bm{j}_{0}}{y}+\bm{u}^{\text{cmd}}_{t}\,\tau, \tag{3}$$

applied component-wise in the plane. Heading is simpler: the spring acts directly on the yaw angle $\psi$ and converges to $\psi^{\text{cmd}}_{t}=\mathrm{atan2}(u^{\text{cmd}}_{t,y},u^{\text{cmd}}_{t,x})$, so Eq. (2) is evaluated at the same three horizons and no integration is needed. Blending at a switch uses inertialization: at the instant of the switch the offset between the currently playing motion and the target motion is computed, that offset is added to the output afterwards to preserve continuity, and the same critically damped spring — now with a zero goal — decays it away.

2. Long-Horizon Synthesis: Locomotion as the Universal Connector

The usage here differs from games in one important respect: motion matching is an offline motion-synthesis module whose job is to emit a large batch of long-horizon references in one pass. Every composed clip takes the fixed form walk/run, then a parkour skill, then walk/run again. Routing all skills through a shared locomotion manifold is what makes the scheme scale, because walking and running are the most general and most reusable connectors available: heterogeneous skills then receive consistent transitions without a single transition clip being captured for any specific pair of skills.

Structurally the system keeps one locomotion database $\mathcal{D}_{\text{loco}}=\{(\bm{x}_{i}^{\text{loco}},\bm{q}_{i}^{\text{loco}})\}$ and a set of skill databases $\{\mathcal{D}_{k}\}$, one per skill $k$. Each skill clip owns a corresponding terrain asset and carries hand-annotated start and end frames $(s_{k},e_{k})$. On top of that the authors define an entry window whose length depends on the skill:

$$\mathcal{E}_{k}:=[\,s_{k}-H_{k},\;s_{k}\,], \tag{4}$$

covering the approach phase immediately before the main contact event — cutting into the clip is only meaningful inside this range. For a vault, $\mathcal{E}_{k}$ spans the last few steps before take-off, which rules out transitions that would enter at an unintended phase. The three operating modes follow cleanly. During locomotion, $\mathcal{C}_{t}=\mathcal{D}_{\text{loco}}$ and the system runs ordinary retrieval with forward playback. When skill $k$ is requested, $\mathcal{C}_{t}$ is narrowed to $\mathcal{E}_{k}$, Eq. (1) matches an entry frame, the playhead switches to it, and the paired terrain is placed using the terrain-to-root offset recorded at that matched frame so it lands on the robot's current root pose. Playback then runs to the annotated end frame $e_{k}$ with motion matching disabled, preserving the contact-rich human motion verbatim; after $e_{k}$, $\mathcal{C}_{t}$ returns to $\mathcal{D}_{\text{loco}}$ and retrieval resumes.

The dataset is produced by rolling this procedure out repeatedly. Each trajectory starts from standing and enters a locomotion phase driven by a 2D velocity command: speed takes one of two values (1 m/s low, 2 m/s high) and turning one of five headings ($-90^{\circ}$, $-45^{\circ}$, $0^{\circ}$, $45^{\circ}$, $90^{\circ}$). The system then cuts into skill $k$ and plays the clip, during which the command is set to straight ahead ($0^{\circ}$) while retaining the speed of the preceding locomotion segment. After $e_{k}$ it returns to locomotion and walks for a further 2 s before stopping. Throughout, the per-timestep velocity command is logged alongside the generated kinematic reference $\{\bm{q}_{t}\}$, producing the paired data that downstream policy training consumes.

3. Transition Density: Making Sure the Barely-in-Time Step Gets Sampled

Transition density is treated as a first-class quantity rather than a by-product. Because motion matching lets a skill be entered from any locomotion state that is nearby in feature space, the same skill acquires a family of entries spanning different approach distances and gait phases — one extra step before take-off, or a vault launched from a different point of the running cycle. This substantially thickens the distribution of pre-entry states. The example given in the paper is concrete: moving the initial approach distance from 3.9 m to 4.8 m forces the retrieval engine to pick a different footstep sequence, which in turn lands the robot in a distinctly different entry posture, left leg leading in one case and right leg leading in the other.

To stop the policy from exploiting non-causal shortcuts — predicting a skill trigger from how many seconds it has already walked, or how many steps it has taken — the duration of the pre-entry locomotion segment is drawn uniformly from $[0.1,3]$ s with a mean spacing of 0.3 s. Diverse motion-terrain pairs of this kind force the policy to react to the present context, and that is the precondition for triggering the correct skill reliably across distances and timings.

Terrain randomization raises robustness while keeping the reference feasible: obstacle width is sampled anywhere from the minimum the reference requires up to 1.5 m, all other dimensions are perturbed within $\pm 5$ cm, and obstacle yaw is randomized within $\pm 45^{\circ}$. Distractor boxes of random size and pose are additionally placed near the reference trajectory, which improves robustness to irrelevant objects and reduces overfitting on the physical robot.

4. Expert Policies: Motion Tracking with Privileged Global Observation

The expert observes reference joint positions and velocities, the reference pelvis pose error, pelvis linear and angular velocity, joint positions and velocities, and the previous action, plus a 0.7 m × 0.7 m height scan that lets it cope with the terrain randomization described above. Unlike OmniRetarget, this work enables global tracking: the expert receives the global position and velocity of the pelvis and can therefore learn recovery behaviour. The motivation is specific — reference motion and terrain are tightly coupled, so a small drift or a timing error accumulates quickly and must be corrected if the robot is to stay on the intended trajectory. These privileged quantities are unavailable on hardware, but the student can infer them from visual input.

The second component that cannot be dropped is adaptive sampling, which preferentially draws rollout start states from regions where failure is more frequent. The authors state plainly that without it the high-wall climbing expert never converges to meaningful behaviour. Rewards, terminations and domain randomization follow BeyondMimic: DeepMimic-style tracking rewards, penalties on action rate of change, joint limits and collisions, early termination driven by tracking error, and light randomization. The tracking terms are Gaussian kernels; the body position and orientation terms read

$$r_{\text{pos}}=\exp\!\Big(-\big(\tfrac{1}{|\mathcal{B}_{\mathrm{target}}|}\textstyle\sum_{b\in\mathcal{B}_{\mathrm{target}}}\|\mathbf{p}^{\mathrm{des}}_{b}-\mathbf{p}_{b}\|^{2}\big)/0.3^{2}\Big),\quad r_{\text{ori}}=\exp\!\Big(-\big(\tfrac{1}{|\mathcal{B}_{\mathrm{target}}|}\textstyle\sum_{b\in\mathcal{B}_{\mathrm{target}}}\|\log(R^{\mathrm{des}}_{b}R_{b}^{\top})\|^{2}\big)/0.4^{2}\Big), \tag{5}$$

where $\mathcal{B}_{\mathrm{target}}$ is the set of tracked body parts, $\mathbf{p}^{\mathrm{des}}_{b}$ and $R^{\mathrm{des}}_{b}$ are the reference quantities, and $\log(\cdot)$ returns the axis-angle representation of the rotation error. The linear- and angular-velocity terms have the same form with scales $1.0^{2}$ and $3.14^{2}$ respectively, and there are two further anchor terms for position and orientation (scales $0.3^{2}$ and $0.4^{2}$); all six are weighted 1.0. Regularizers comprise action smoothness $\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\|^{2}$ with weight $-0.1$, joint-limit violation with $-10.0$, and undesired self-contact $\sum_{b\notin\mathcal{B}_{\mathrm{ee}}}\mathbf{1}[\|f^{\mathrm{self}}_{b}\|>1\,\text{N}]$ with $-0.5$. Actions are joint PD targets divided by a fixed action scale; because RL exploration is hard here, the action scale is set to 1 for every expert instead of using the BeyondMimic heuristics. Domain randomization covers static friction $\mu_{\text{static}}\sim\mathcal{U}[0.4,1.3]$, kinetic friction $\mathcal{U}[0.4,1.1]$, restitution $\mathcal{U}[0,0.5]$, torso centre-of-mass offset ($\pm 2.5$ cm along $x$, $\pm 5$ cm along $y$ and $z$), and root velocity pushes lasting 1 to 3 s.

5. Distillation: Adding an RL Term Alongside DAgger

Learning one unified policy from several experts would normally be done by DAgger-style imitation. That works for easier motions such as stepping, but not for highly dynamic skills like climbing and vaulting. The diagnosis the authors give is persuasive: these skills depend on brief, high-amplitude bursts of torque, and a stepwise imitation objective never looks at episode outcome, so it has no reason to prefer such bursts. Their illustration is sharp — two actions that leave the root slightly high or slightly low, symmetric about the reference, can incur an identical DAgger loss, yet only the high one actually clears the obstacle. On a 76 cm obstacle the pure-DAgger student reliably stalls in the pull-up phase: hand placement is accurate, but the burst that lifts the torso never materializes.

The remedy is to run a curriculum-scheduled PPO objective in parallel with DAgger:

$$\mathcal{L}=\lambda_{\text{PPO}}\,\mathcal{L}_{\text{PPO}}+\lambda_{D}\,\mathcal{L}_{D},\qquad \lambda_{\text{PPO}}+\lambda_{D}=1, \tag{6}$$

and the authors are careful about what PPO is for. It supplies a success-driven exploitation signal that pushes the policy to make full use of behaviour the expert already demonstrates (that torque burst, for instance); it is not there to explore beyond the expert's skill distribution. The curriculum has three parts. First, $\lambda_{D}$ is ramped down linearly over the first half of training and floored at 0.1:

$$\lambda_{D}(k)=\max\!\left(0.1,\;1-\frac{k}{K/2}\right), \tag{7}$$

with $k$ the current iteration and $K$ the total. Second, left-right symmetry creates multimodality: many skills admit two equally correct mirrored executions (a hurdle can be led with either leg) while the reference records only one of them, so a student that performs the mirror has in fact completed the skill yet shows a large tracking error and would be terminated incorrectly — a spurious failure signal that inflates the variance of the PPO reward. The countermeasure is to relax the termination threshold from the expert's 0.5 m to 1 m along the same linear schedule, so mirrored executions are not killed early. Third, adaptive learning rate and KL-based exploration control are enabled only once $\lambda_{\text{PPO}}$ exceeds 0.1, which keeps a noisy early PPO from dragging distillation off course.

One further trade-off is easy to miss but consequential: adaptive sampling of rollout starts is switched off for the student. It helps the expert focus on failure regions, but for the student it undersamples exactly those borderline fragments that do not fail in simulation yet shake during execution — the fragments that carry the largest sim-to-real gap. To avoid imbalance across skills, every skill is sampled with equal probability and uniformly within itself. Relaxing termination has a side effect too: the student reaches states outside the expert's training distribution, where the expert (trained under the original threshold) cannot give a meaningful action. So whenever the student violates the expert's original termination condition while still satisfying the relaxed one, the DAgger loss at those timesteps is masked off entirely and only the PPO objective remains, preventing corrupted supervision from entering the gradient.

The authors also separate their use of RL from the Extreme Parkour style of post-processing: that work first trains a strong DAgger policy and then runs an independent RL stage to adapt to unseen terrain, whereas here RL corrects the conservatism induced by imitation during distillation itself. A converse finding is equally important — the DAgger term must stay online for the entire run. Dropping the DAgger loss after the curriculum ends and continuing with pure RL tends to produce jittery, unnatural behaviour, which indicates that in a high-dimensional action space the behaviour-cloning objective is a key regularizer for RL rather than a scaffold to be removed.

6. Vision, Camera Modelling and Sim-to-Real

The student observes proprioception (pelvis gravity vector and angular velocity, joint positions and velocities, previous action), a depth image, and the 2D velocity command. Depth is rendered with Nvidia WARP to keep training throughput up. Camera calibration is refreshingly practical: the visible parts of the robot itself are compared between simulation and hardware across a set of poses and aligned by ROI overlap; the extrinsics are then randomized around the calibrated value — 2.5 cm in translation and 2.5° in rotation — to absorb viewpoint shift and mounting variation. Depth noise follows prior work, but Gaussian blur is deliberately excluded, because at speed blur erases the obstacle. Observation latency is randomized between 60 ms and 80 ms to model hardware jitter, and a random depth offset of $\pm 3$ cm plus independent Gaussian noise with 3 cm standard deviation is added on top. The onboard depth camera runs at 30 Hz.

Training configuration: a 3-layer CNN as the depth backbone (input resolution $58\times 87$, 32-dimensional output, globally average-pooled and concatenated with proprioception), a 5-layer MLP with hidden widths $[2048,1024,512,256,128]$ and ELU activations, 16384 parallel environments, and 20K iterations for both expert and student. The student's initial action noise standard deviation is only 0.01 (the expert uses 1.0), its learning rate is $3\times10^{-4}$ (the expert $1\times10^{-3}$), and its mini-batch is 96 (the expert 4); the distillation loss is MSE with a DAgger coefficient of 10.0, and the curriculum completes at epoch 10000. Taken together these numbers say something clear: the distillation stage relies almost nothing on stochastic exploration and instead on expert labels plus a success signal.

7. Where the Method Lands in the Released Code

Equations (1) and (4) exist in the repository as one declarative table plus one masked nearest-neighbour function. In motion_matching/motion_matching/scenarios/skill_registry.py, SKILL_REGISTRY writes each skill's search_ranges as triplets [(start_searching_frame, end_searching_frame, end_of_motion_frame), ...]: the first two entries are $\mathcal{E}_{k}=[s_{k}-H_{k},s_{k}]$ and the third is the annotated $e_{k}$. For instance climb_76_up is [(70, 95, 411), (744, 769, 1085)], where the second triplet is the mirrored copy of the same clip (offset +674), which is the paper's mirror augmentation made literal. Each skill also carries a height_range (for example (0.0, CLIMB_76_TOP=0.76382022)) and terrains (box.obj and box_mirror.obj), i.e. the hand-fitted paired terrain.

The retrieval itself lives in motion_matching/motion_matching/core/database.py::motion_matching_search_w_custom_ranges. It sets the whole search_mask false and then opens it segment by segment according to custom_search_ranges, blanks the last frames of every clip via ignore_range_end=20, and computes feature_costs = ((features - query_normalized) ** 2).sum(axis=1), followed by costs[~search_mask] = np.inf and np.argmin — a vectorized implementation of Eq. (1). The helper immediately after it, _prefer_earlier_mirror_on_tie, deserves a note: when the best frame and its mirror-phase counterpart differ in cost by less than MIRROR_TIE_EPSILON = 1e-4, the globally smaller frame index is forced, and the comment explains that real query cost gaps are of order 1e-2 or larger, so the epsilon exists purely to remove the numerical ambiguity introduced by mirroring and keep results reproducible. run.py::_step_transitions implements terrain placement at the locomotion-to-skill instant: it derives t_rot and t_pos from the current root pose and the matched frame's root pose and applies them to the terrain metadata's pos and quat, matching the paper's placement rule word for word. self.search_time = 0.1 (five frames at 50 FPS) is the retrieval period $M$ of Eq. (1), and the force_search branch is the immediate query on significant command change.

The student-side correspondences concentrate in wbt_training/config_values/depth_distillation.py. The depth term WarpDepthImageObsTerm is parameterized with resize=(58, 87), latency_frame=(3, 4), enable_holes=True and buffer_len=6; the first two are precisely the $58\times 87$ input and the 60 to 80 ms latency randomization of Table VI. The teacher observation group carries one extra routing item, which_motion, whose comment states that off-track experts are flagged as $-1$ so the core emits a zero teacher action for those rows, and PhpDistillationPPO._dagger_loss then masks them out — this is the implementation of Appendix B3's rule that the DAgger loss is disabled at timesteps violating the expert's original termination condition. Threshold relaxation is likewise a curriculum object, RelaxTerminationThresholds, with curriculum_start_step=24_000 and curriculum_end_step=240_000. The repository goes one step beyond the paper: rather than a single scalar schedule, the PPO weight uses residual gating (ppo_gate_enabled, ppo_gate_low=0.5, ppo_gate_quantile_lo/hi=0.50/0.95, ppo_gate_ema_alpha=0.001, ppo_gate_warmup_samples=5e5), allocating PPO weight by comparing imitation residuals per phase and thereby refining the global curriculum of Eq. (6) into per-skill-phase weighting. The example training datasets also give a sense of scale: 100 motion/terrain pairs for locomotion (118.3 MB), 56 for low-speed stepping, 60 for high-speed stepping, and 40 each for low- and high-speed climb-76, consistent with the five-teacher setup.

Experiments

Fig. 2 (paper Fig. 1): four instants of high-dynamic, long-horizon parkour executed by the G1 under onboard perception alone — (a) a cat vault over a low obstacle at roughly 3 m/s immediately followed by a dash vault over a taller one; (b) climbing a 1.25 m wall, 96% of the robot's own height, and dropping down the far side; (c) a speed vault at about 3 m/s; (d) a 60 s continuous traverse of a complex course in which the policy picks its own skills and blends them without pausing.

Everything is evaluated on a 1.3 m, 29-degree-of-freedom Unitree G1. The protocol warrants its own paragraph, because it dictates how the numbers should be read. For each task the robot advances at a fixed commanded speed (1.0 or 2.0 m/s) towards a single obstacle of prescribed height, with $20^{\circ}$ of yaw randomization. The approach condition is varied by drawing the initial distance uniformly from 1.5–3.0 m for the 1 m/s tasks and from 3.0–4.5 m for the 2 m/s tasks. Every task is scored over 100 obstacle instances with 5 trials each, so 500 trials per cell, and a trial counts as a success only if the robot clears the obstacle and then advances a further 1.5 m without falling inside the fixed horizon.

1. On Hardware: Reducing "Human-Level Agility" to Stopwatch Readings

Fig. 3 (paper Fig. 4): the high-wall climb, robot and human side by side. Timed from toe-off, the robot reaches the top of the 1.25 m wall in 3.63 s.

The most persuasive result in the paper is the human-versus-robot comparison on the high-wall climb. The authors set the robot against a human traceur taken from public footage performing the same skill. A wall climb counts as a foundational parkour technique, yet it demands considerable upper-body strength and precise whole-body coordination, and an untrained person will usually fail it. Timed from toe-off, the robot reaches the top of a 1.25 m wall — 96% of its standing height — in 3.63 s, a pace comparable to the human reference, and the paper aligns the key events segment by segment: toe-off, pull-up, swing and stable stand all occur at closely matched times. What matters here is not simply that the wall was climbed. Tracking error could easily have dragged the motion into a slow crawl, and it did not, which means the dynamics present in the retargeted reference survive into the deployed policy.

Fig. 4 (paper Fig. 5): hardware results. (a) cat vault; (b) landing after a drop from a 1.25 m platform; (c) a 48 s continuous terrain traverse during which obstacles are moved in real time and the policy reroutes online.

The remaining dynamic skills are reported with the same quantitative care. In the cat vault the robot clears an obstacle 0.4 m high and 0.5 m long, spending only 0.8 s between toe-off and toe-on while covering more than 2 m forward — 154% of its height — at a peak forward speed of 3.41 m/s and a mean of 2.53 m/s. Those readings indicate that momentum is carried through contact instead of being paid for again at every step. When dropping from the 1.25 m platform, the robot flexes its leg joints to absorb the impact and settles back into a stable posture.

Long-horizon behaviour is where the framework makes its real claim, because the training data contain single-obstacle traversals only, yet the policy generalizes to multi-obstacle courses. The authors attribute this to the way motion matching composes: skills are chained through shared locomotion segments, and the synthesized set exposes a wide spread of approach distances and timings, so the student learns to execute reliably across obstacle sequences without ever being supervised on one. On hardware the robot links stepping with low- and high-wall climbs into a single continuous run, the visuomotor policy generating the transitions online. In a harder test, several obstacles are displaced by roughly 0.5 m while the robot is already moving, and it still completes a 48 s traverse by adjusting its approach path and the timing of each skill. Skill selection is entirely autonomous; the human supplies only the simple 2-D velocity command used for navigation.

2. Simulation Baselines: What Each Component Contributes

Three baselines decompose the contribution of human reference motion, of motion-matching composition, and of the two-stage teacher–student framework. Velocity Tracking is the standard IsaacLab recipe for G1 velocity tracking over rough terrain — pure reward shaping with no human reference at all — equipped with a ten-step terrain curriculum from 0.3 m to 1.0 m and a 2 m run-up, and given the privileged heightmap directly. Uncomposed Data removes motion matching and trains on uncomposed walking data together with the atomic skill clips. End-to-end Depth removes distillation and trains a depth policy from scratch under the same observation as the student and the same tracking reward as the expert.

Method	1.0 m/s · 36 cm	1.0 m/s · 58 cm	1.0 m/s · 76 cm	2.0 m/s · 36 cm	2.0 m/s · 58 cm	2.0 m/s · 76 cm
Velocity Tracking (privileged heightmap)	1.00	0.00	0.00	1.00	0.00	0.00
Uncomposed Data (no motion matching)	0.06	0.02	0.00	0.37	0.27	0.07
End-to-end Depth (no distillation)	0.95	0.07	0.08	0.78	0.19	0.14
PHP (this paper)	1.00	0.99	0.95	1.00	0.99	0.95

Table 1: success rate across commanded speeds and obstacle heights (paper Table I; 500 trials per cell).

The three baselines fail in distinct ways, and the failure modes carry more information than the numbers do. Velocity Tracking clears 36 cm but goes straight to zero at 58 cm and 76 cm: it relies almost entirely on footholds and never discovers the whole-body, arm-supported climb, so even a privileged heightmap plus a terrain curriculum cannot reach that class of motion — consistent with what has been reported for reward-shaping approaches. Uncomposed Data stays poor despite having access to the atomic skills (0.06/0.02/0.00 at 1 m/s); the characteristic failure is walking right up to the obstacle and then neither climbing it nor jumping it, because without explicit long-horizon composition the policy has never experienced a skill transition in training and has never learned to observe the obstacle during the walking phase in preparation for what comes next. End-to-end Depth handles the low obstacle (0.95/0.78) and collapses on the hard ones (0.07/0.08 and 0.19/0.14), which places the bottleneck squarely in exploration when training from scratch. PHP scores 1.00/0.99/0.95 and 1.00/0.99/0.95 on those same six cells.

3. Ablations: Data Density, Scalability and the Role of RL

Variant	1.0 m/s · 58 cm	1.0 m/s · 76 cm	1.0 m/s · 94 cm	2.0 m/s · 36 cm	2.0 m/s · 58 cm	2.0 m/s · 76 cm
Extreme Distances (keep only nearest/farthest)	0.99	0.62	0.64	0.98	0.60	0.58
Half Density (random half retained)	0.95	0.32	0.57	0.99	0.85	0.81
DAgger Only (RL removed)	0.16	0.03	0.12	0.63	0.09	0.10
DAgger + Alive Reward	1.00	0.90	0.96	0.94	0.91	0.84
DAgger + Root Tracking	1.00	0.79	0.75	1.00	0.92	0.87
1/4 Training Envs	0.97	0.00	0.59	0.94	0.65	0.58
1/2 Training Envs	0.94	0.60	0.68	0.97	0.79	0.75
3-layer MLP	0.99	0.02	0.00	0.98	0.89	0.81
4-layer MLP	1.00	0.94	0.08	1.00	0.94	0.88
PHP (full)	0.99	0.95	1.00	1.00	0.98	0.90

Table 2: ablations over motion-matching data density, the RL objective used during distillation, training scale and network capacity (paper Table II).

The data density rows are the direct evidence for the transition-density hypothesis. Retaining only the shortest and the longest approach distances (Extreme Distances) lowers success across the board, since the policy cannot generalize to the intermediate distances where contact timing matters most. Keeping a random half (Half Density) hurts the difficult skills harder, especially when the surviving samples lean towards one end of the distance range: at 1.0 m/s the 76 cm and 94 cm climbs fall to 0.32 and 0.57, because insufficient local density makes hand-placement timing unreliable. Only the full dataset, which covers approach conditions densely, executes stably at every distance.

The role of RL is the sharpest conclusion in the paper. Removing it and keeping DAgger alone drops the scores from 0.99/0.95/1.00 to 0.16/0.03/0.12, and the failure the authors observed on the 76 cm obstacle is very specific: the student places its hands correctly, then stalls during the pull-up and never produces the burst of torque, because that decisive burst spans only a handful of timesteps and a stepwise imitation loss barely penalizes an action that undershoots it. Conversely, the RL term proves unexpectedly insensitive to reward design: an alive/progress reward with no tracking term reaches 1.00/0.90/0.96 and 0.94/0.91/0.84, and a root-tracking-only reward reaches 1.00/0.79/0.75 and 1.00/0.92/0.87, both close to the full whole-body tracking scheme. The authors read this as evidence that, when trained alongside DAgger, RL mainly acts as a success-driven exploitation signal compensating for what DAgger underestimates, rather than depending on careful task-specific shaping — which suggests that a simple alive reward may well suffice as the skill set grows.

The scalability rows move in the opposite direction from training from scratch. At one quarter of the environments the 76 cm climb drops to 0.00; at one half it recovers to 0.60; at the full 16384 environments it reaches 0.95. Capacity behaves the same way: a 3-layer MLP gives 0.02/0.00, a 4-layer MLP 0.94/0.08, and the paper's 5 layers 0.95/1.00, with the hardest tasks benefiting most. The conclusion the authors draw is worth repeating for anyone planning a larger skill set — when training from scratch, adding rollouts usually returns diminishing gains because exploration is the binding constraint, whereas the distillation framework scales well along both model capacity and rollout throughput.

4. The Skill Library: A Quantitative Picture of Scarcity

Fig. 5 (paper Fig. 3): the variety that motion matching synthesizes out of the atomic library. (a) different approach distances trigger different gait phases and entry postures; (b) varied speeds, directions and durations; (c) randomized terrain poses and shapes.

Skill category	Clip	Total duration (s)
Locomotion (shared transition manifold)	stand / walk / run, commanded speeds 0.8–3.5 m/s	495.5
Parkour skills @ 1.0 m/s	Step (36 cm)	2.2
	Climb (58 cm)	12.1
	Climb (76 cm)	8.8
	Climb (94 cm)	10.3
Parkour skills @ 2.0 m/s	Step (36 cm)	1.6
	Climb (58 cm)	6.1
	Climb (76 cm)	4.4
	Climb (94 cm)	5.2
	Climb (125 cm)	5.9
	Dash Vault	5.0
	Speed Vault	3.1
Parkour skill @ 3.0 m/s	Cat Vault	1.5

Table 3: composition of the motion library (paper Table III). The locomotion clips total 495.5 s, while each atomic skill lasts only 1.5–12.1 s — that ratio is the quantitative picture of dynamic-motion scarcity, and it is why the library has to be enlarged by composition rather than by capture.

Fig. 6 (paper Fig. 2): system overview. Atomic skills are composed into long-horizon references by motion matching; single-skill teachers perform RL motion tracking under privileged information; the teachers are distilled into one depth student through the hybrid DAgger-and-RL objective, which transfers to hardware zero-shot.

Limitations

No semantic scene understanding (author-stated). Every decision in this system rests on geometry. The depth image tells the policy that something of a given height is in front of it; it does not tell the policy that this object is a table one may press down on, or that one is a rail that can be gripped. The authors argue that richer conditioning signals, language among them, would allow finer control over the diversity and the style of the resulting motion.

Perception and hardware jointly cap the demonstrated ability (author-stated). A short-range, narrow-field-of-view camera cannot take in the full obstacle geometry while the robot is running at speed, so the policy is forced to commit early under perceptual ambiguity. The hardware also lacks hands or grippers strong enough to interact with ledges and bars, which is why climbs above the robot's own height and hanging-type skills were never tested at all. Put differently, 1.25 m is not the ceiling of the method; it is the ceiling of this particular G1's hands.

Manual annotation is the hidden cost of the pipeline (reviewer judgment). Every skill needs a hand-labelled start and end frame $(s_{k},e_{k})$, a prescribed entry-window length $H_{k}$, and a hand-fitted box terrain aligned to that clip. The 30 search_ranges entries in SKILL_REGISTRY and the per-skill terrain/*.obj files in the repository are exactly that invoice. Motion matching removes the cost of learning a transition model, but it transfers the cost to annotation, and annotation does not scale automatically as the number of skills grows.

The quantitative evidence covers single obstacles only (reviewer judgment). Every figure in Table 1 and Table 2 is a success rate over one obstacle and 500 trials, while the most striking multi-obstacle traverses (48 s and 60 s) are shown qualitatively on hardware, with no success rate, no repeat count and no multi-obstacle baseline. The reference trajectories are also purely kinematic — no dynamics, no contact forces — and the paper never verifies that they remain dynamically feasible for this G1's mass distribution after retargeting. The evaluation metric is binary success, with no dimension for motion naturalness or energy cost. Finally, all comparisons are in-house ablations: there is no head-to-head against the strongest current perceptive humanoid systems, and the AMP baseline that was implemented (in MimicKit; it walked steadily but mostly failed the obstacles) is excluded from the formal tables because its tuning space was left unexplored. The authors are candid about this, but readers should discount accordingly.

Conclusion and Outlook

PHP's contribution compresses into a single sentence: the way to make a motion library dense is not to capture more data but to let the clips you already have lead into one another. Motion matching turns that into one windowed nearest-neighbour query, stitching atomic skills a few seconds long into long-horizon references that span different approach distances, gait phases and timings. In doing so it dissolves the hardest problem on the human-motion-imitation route — how skills transition into each other — because no transition has to be captured per skill pair: walking and running already are the shared connector. The training-side conclusion is equally clean. The expert owns motion quality, distillation owns the job of compressing that ability into a policy that sees nothing but depth, and distillation has to be a hybrid of DAgger and RL: pure imitation cannot produce the decisive torque burst, while pure RL shakes itself into unnatural motion. The details that look like engineering trivia — data density, relaxing the termination threshold, mirroring for multimodality, switching off adaptive sampling in the student stage — each leave a measurable trace in the ablations or on hardware.

Looking forward, the authors name two directions: semantic and linguistic conditioning, so that the choice of skill is no longer decided by geometry alone and style and diversity become controllable; and stronger sensing and hand hardware, so that climbing can be pushed above body height and hanging or bar interactions can enter the library. For teams doing robot learning, the most reusable part of this recipe is not parkour itself but the general rule underneath it: when a skill has only a few seconds of data, first retrieve it onto a shared manifold to make it dense, and only then talk about policy learning. And when the student turns out to be a little short of force on a few critical frames, the problem usually is not too little imitation data but a loss function that cannot see success and failure.