PAPER DEEP DIVE
KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skills
A physics-based whole-body control framework that combines contact-aware motion filtering, differentiable IK retargeting, and adaptive RL tracking to teach Unitree G1 highly dynamic skills from human motion.
Authors: Weiji Xie, Jinrui Han, Jiakun Zheng, Huanyu Li, Xinzhe Liu, Jiyuan Shi, Weinan Zhang, Chenjia Bai, and Xuelong Li
Paper and project: arXiv:2506.12851 · Project page
Code status: The official implementation is public at TeleHuman/PBHC. This analysis checked commit ffac5cded61fe78b39b051f09ac2ed0f6a1ccea0. The repository is distributed under CC BY-NC 4.0 and explicitly restricts commercial use.
One-Sentence Summary
KungfuBot separates humanoid skill learning into two engineering problems that must agree: first turn a human video into a reference motion that obeys support and ground-contact constraints, then train one physics-based policy per motion with a tracking tolerance that tightens as the robot's actual error decreases.
Why Highly Dynamic Motion Is Different
Humanoid robots can inherit a remarkable range of skills from human demonstrations, but imitation is not a matter of copying joint angles. Boxing, kicks, spins, jumps, and dance require the robot to coordinate contact, center of mass, angular momentum, actuator limits, friction, and impact recovery. A movement that looks natural in a video may be physically impossible for the robot because the support foot slides, the center of mass leaves the feasible region, a knee torque saturates, or the landing cannot be stabilized.
Existing motion-tracking systems are strongest on smooth, relatively slow behavior. Walk, turn, wave, and crouch can be learned with carefully shaped rewards and hand-designed curricula. Highly dynamic actions create two additional failures. The reference trajectory itself may be physically inconsistent because monocular human-motion recovery produces floating feet, ground penetration, or incorrect support phases. Even after cleanup, a fixed reward tolerance cannot serve both slow Tai Chi and a rapid roundhouse kick. A loose tolerance stabilizes early learning but stops rewarding precision; a tight tolerance gives almost no useful feedback when initial errors are large.
The paper attacks these problems separately rather than expecting a larger policy network to absorb both. A multi-stage motion-processing pipeline creates a physically plausible robot reference, and an adaptive tracking mechanism changes the reward sensitivity during training. Figure 1 connects the two halves: video recovery, physics filtering, contact-aware correction, inverse-kinematics retargeting, reinforcement learning, and deployment on the Unitree G1.
The control problem is represented as a Markov decision process with an explicit reference-state component:
$$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{S}^{\rm ref},\gamma,r,P)$$Here, $\mathcal{S}$ contains robot state, $\mathcal{A}$ contains joint actions, $\mathcal{S}^{\rm ref}$ contains the target motion, $\gamma$ is the discount factor, $r$ is the reward, and $P$ is the environment transition. The reference is not just a sequence of joint angles. It also carries root motion, body positions, velocities, and accelerations, so temporal errors in a fast strike or landing are visible to both the policy and the value function.
From Video to a Robot-Executable Motion
Recovering a Human Mesh
The process starts with ordinary RGB video. A human-motion recovery model, in the GVHMR family, estimates SMPL parameters for each frame. The resulting human mesh is determined by body shape, pose, and global translation:
$$\mathcal{V}=M(\bm{\beta},\bm{\theta},\bm{\psi})\in\mathbb{R}^{6890\times3}$$$\bm{\beta}$ controls body proportions, $\bm{\theta}$ controls joint rotations, and $\bm{\psi}$ places the body in world space. A mesh is more useful than sparse 2D landmarks because it supports volume-aware center-of-mass estimation, foot-ground distance, and surface-based contact correction. It does not, however, guarantee physical consistency. Camera scale, occlusion, fast motion, and ambiguous ground cues can all produce a sequence that is visually plausible yet impossible to execute.
The paper therefore adds an explicit acceptance layer. Raw recovered motion is not sent directly to reinforcement learning. Instead, each sequence must pass a physics-based stability test and a contact-aware correction stage before retargeting.
Filtering with Center of Mass and Center of Pressure
The stability test compares the horizontal center of mass with the center of pressure estimated from the mesh. A frame is considered stable when their separation remains below a threshold:
$$\Delta d_t=\|\bar{\bm p}_t^{\rm CoM}-\bar{\bm p}_t^{\rm CoP}\|_2<\epsilon_{\rm stab}$$The experiment uses $\epsilon_{\rm stab}=0.1$. The released implementation computes per-part volume from the SMPL mesh, samples a high-density surface, obtains a volume-weighted center of mass, and assigns pressure weights to vertices according to their height above the ground. This is a useful distinction from a purely kinematic filter: support is inferred from body distribution and proximity to the floor, not only from foot positions.
A sequence is not rejected merely because a few frames are unstable. The algorithm requires stable boundary frames and limits the longest consecutive unstable interval to $\epsilon_N=100$. That allowance matters for jumps and kicks, where static support is intentionally broken. What the filter rejects is a long unresolved loss of support that the motion never recovers from.
Estimating Foot Contact from Velocity and Height
For the left ankle, contact is detected by combining low inter-frame displacement with low absolute height:
$$c_t^{\rm left}=\mathbb I\left[\|p_{t+1}^{\rm l-ankle}-p_t^{\rm l-ankle}\|_2^2<\epsilon_{\rm vel}\right]\mathbb I\left[p_{t,z}^{\rm l-ankle}<\epsilon_{\rm height}\right]$$The right ankle uses the symmetric test, with $\epsilon_{\rm vel}=0.002$ and $\epsilon_{\rm height}=0.2$. A frame is considered supported when either foot is in contact. The paper evaluates this estimator on ten manually labeled sequences and reports 91.4 percent accuracy. This is an important intermediate result because an incorrect contact mask propagates directly into the global height correction.
The two conditions solve complementary false positives. A fast-swinging foot may pass near the ground, so height alone is insufficient. A floating foot may pause briefly, so velocity alone is insufficient. Their conjunction produces a more selective support signal. The evaluation is still small, and the thresholds may need recalibration for different shoes, camera angles, body proportions, or recovery errors.
Correcting the Ground and Retargeting to G1
When contact is present, the system measures the lowest mesh point relative to the floor and shifts the entire global translation by the resulting offset:
$$\psi_{t,z}^{\rm corr}=\psi_{t,z}-\Delta h_t$$Frames without contact inherit the offset from the preceding contact frame. The corrected vertical trajectory is then smoothed with an exponential moving average. The public configuration uses $\alpha=0.3$:
$$\hat x\leftarrow(1-\alpha)\hat x+\alpha x$$This operation corrects global floating without rewriting joint rotations, so the expressive content of the motion remains intact. The cleaned SMPL sequence is passed to differentiable inverse kinematics, which maps human end-effectors and body targets onto the robot while respecting its kinematic constraints. The output controls 23 G1 degrees of freedom.
The implementation detail matches the paper's hardware scope. In smpl_retarget/mink_retarget/convert_fit_motion.py, the joint vector is formed from dof[:, :19] and dof[:, 22:26]. The result has 23 dimensions and excludes the three hand degrees of freedom in each wrist. That is appropriate for whole-body kicks, punches, spins, and dance, but it is not a dexterous manipulation system.
flowchart LR
Video[Monocular video] --> HMR[Recover SMPL motion]
HMR --> Physics[CoM and CoP stability filter]
Physics -->|accept| Contact[Ankle velocity and height mask]
Physics -->|reject| Drop[Keep out of training set]
Contact --> Floor[Contact-aware ground correction]
Floor --> Smooth[EMA global translation]
Smooth --> IK[Differentiable IK retargeting]
IK --> Ref[23-DoF G1 reference]
Ref --> Sigma[Adaptive sigma update]
Sigma --> PPO[Asymmetric actor-critic]
PPO --> Sim[MuJoCo transfer check]
Sim --> Robot[G1 deployment]
Adaptive Motion Tracking
The Scale Problem in Exponential Rewards
Motion-tracking rewards commonly use an exponential kernel:
$$r(x)=\exp(-x/\sigma)$$Error $x$ can represent body position, joint position, velocity, or acceleration, while $\sigma$ controls the interval over which the reward is sensitive. If $\sigma$ is much larger than typical error, every attempt receives a reward near one and the policy has little reason to improve. If $\sigma$ is too small, early errors drive the reward toward zero and reduce the learning signal. A single hand-tuned value is therefore fragile across motions with different speed and error scales.
Figure 2 illustrates the effect on the right hand during a horse-stance punch. With adaptive tolerance, the hand trajectory begins inside a wider acceptance band and tightens as tracking improves. The policy is not asked to reproduce a high-speed target exactly before it can reliably move the arm toward that target.
A Bi-Level View of the Tolerance
The authors formulate tolerance selection as a bi-level problem. The inner problem trains a policy for a given $\sigma$, and the outer problem evaluates the resulting solution for accurate tracking:
$$\max_{\sigma\in\mathbb R_+}J^{\rm ex}(x^*)\quad \text{s.t. }x^*\in\arg\max_x J^{\rm in}(x,\sigma)+R(x)$$$J^{\rm in}$ is the shaped inner objective, $R(x)$ is a regularizer, and $J^{\rm ex}$ measures the accuracy achieved after convergence. Under the paper's simplified derivation, the optimal scale is related to the average converged error:
$$\sigma^*=\frac{1}{N}\sum_{i=1}^N x_i^*$$This result formalizes the intuition that the sensitive region should match the error scale the policy can currently achieve. It does not mean that a single average is a complete model of every motion. It provides a principled starting point for an online approximation.
Turning the Solution into a Training Rule
At each update, the implementation maintains an exponential moving average of the per-term tracking error:
$$\hat x\leftarrow(1-\alpha)\hat x+\alpha x$$It then applies the estimate while forbidding the tolerance from increasing:
$$\sigma\leftarrow\min(\sigma,\hat x)$$Training begins with a relatively large $\sigma$. As errors fall, the reward becomes increasingly selective. The non-increasing constraint prevents one noisy batch from loosening the curriculum and reintroducing an easy regime after the policy has already improved. This detail is visible in humanoidverse/envs/motion_tracking/motion_tracking.py, where each reward term keeps its own error average and the origin update takes the minimum of the estimate and the current sigma.
Figure 3 shows the resulting feedback loop. The policy first receives a broad reward region that supports exploration, then the tolerance tightens as observed error decreases. There is no sudden switch at a hand-selected training step; the scale is updated continuously from measured behavior. Improvement changes the objective, and the changed objective asks for further improvement.
Compared with a manually staged curriculum, this rule moves the schedule into the reward function. Position, joint, velocity, and acceleration channels can tighten independently according to their observed errors. That flexibility is useful for mixed motion, where translational error may stabilize earlier than body acceleration or joint velocity.
What the Derivation Does and Does Not Establish
The derivation assumes an idealized inner optimization, reduces error behavior to a mean, and replaces the converged solution with an online sampled estimate. It explains why adaptive scaling should help and gives a coherent update rule. It does not prove convergence for arbitrary contacts, impacts, actuator limits, and real-world disturbances. The experimental evidence should therefore be read as validation on a bounded motion set and robot platform, not as a universal stability theorem.
The training system also uses an asymmetric actor-critic. The actor receives observations available or estimable on the robot. The critic additionally receives base linear velocity, reference body positions, current-to-reference position differences, and randomized physical parameters. Privileged information improves value estimation during training without creating an impossible deployment requirement.
Reference-state initialization reduces wasted rollout time at the beginning of each episode. Reward vectorization preserves separate tracking channels for adaptive scaling. Domain randomization covers friction, PD gains, link mass, ankle inertia, base center-of-mass offset, actuator delay, and external pushes. The paper reports that training stabilizes after roughly 20,000 steps and takes about 27 hours per policy on one RTX 4090.
Data and Experimental Protocol
The dataset contains 13 highly dynamic motions grouped into easy, medium, and hard tiers. Sources include monocular video processed by the proposed pipeline, selected AMASS sequences, and LAFAN. All sequences are filtered, contact-corrected, retargeted, and padded with short linear transitions from a default pose. This gives the policies a common robot representation while preserving the different dynamics of martial arts, acrobatics, stretching, dance, and Tai Chi.
Figure 4 shows ten representative motions. Color encodes time, but the display understates the difficulty differences. A quick kick combines single-foot support with high angular acceleration, a spin requires continuous angular momentum, a deep squat stresses knee torque and center-of-mass control, and Tai Chi stresses sustained balance at low speed. A method that succeeds on this set must cover a wider range than ordinary locomotion benchmarks.
Simulation uses IsaacGym with three random seeds and 1,000 rollout episodes per setting. The main comparison includes deployable OmniH2O and ExBody2 policies, MaskedMimic as a character-animation reference, and an oracle PBHC variant that relaxes deployment constraints. The oracle is useful for estimating the cost of partial observability and control regularization, but it is not a deployable baseline in the same sense.
A separate filtering study tests ten motions. Six are accepted and four are rejected. Rejected motions reach a maximum episode-length ratio of 54 percent, while accepted motions sustain much longer rollouts. Figure 5 plots the two distributions. The gap supports the claim that the physics metric predicts learnability, although the study is not a broad calibration of every threshold.
Simulation Results
Table 1 reports the non-oracle PBHC errors at each difficulty level. The columns are global body position, root-relative body position, joint position, body velocity, body acceleration, and joint velocity. Lower values are better, but the columns use different units and should not be summed into a single score.
| Difficulty | Global body position | Root-relative body position | Joint position | Body velocity | Body acceleration | Joint velocity |
|---|---|---|---|---|---|---|
| Easy | 53.25±17.60 | 28.16±6.127 | 725.62±16.20 | 4.41±0.312 | 4.65±0.140 | 81.28±2.052 |
| Medium | 126.48±27.01 | 48.87±7.550 | 1043.30±104.4 | 6.62±0.412 | 7.19±0.254 | 105.30±5.941 |
| Hard | 290.36±139.1 | 124.61±53.54 | 1326.60±378.9 | 11.93±2.622 | 12.36±2.401 | 135.05±16.43 |
The progression from easy to hard is substantial. Global body-position error rises by more than a factor of five, and the hard-tier standard deviation reaches 139.1. That spread suggests that individual fast motions remain sensitive to initialization or seed, even when the aggregate result is strong. The paper reports PBHC outperforming the deployable baselines on every metric, but the practical reading is not that all hard motions are equally reliable.
Ablation results isolate the value of adaptive tolerance. For the jab punch, PBHC records a global body-position error of 44.38±7.118 versus 63.95±6.680 for a fixed coarse setting. The roundhouse kick reports 52.53±2.106 versus 76.81±2.863. On the Charleston dance, PBHC reaches 94.81±14.18 while the lower-bound setting produces 358.82±10.35. Slow and fast motions require different reward sensitivity, and the adaptive schedule searches that sensitivity during learning.
Contact correction has an even clearer effect on motions with flight phases. On the Charleston, full PBHC has a mean contact error of 217.82±47.97, compared with 633.91±49.74 without the contact mask. For the jump kick, global body-position error is 42.58±8.126 with the mask and 170.28±97.29 without it. A wrong ground offset moves the entire reference body through space, so the policy loses reward even when local joint coordination is close.
| Motion | Metric | Full PBHC | Ablation |
|---|---|---|---|
| Charleston dance | Mean contact error | 217.82±47.97 | 633.91±49.74 |
| Jump kick | Global body-position error | 42.58±8.126 | 170.28±97.29 |
Real-Robot Evidence
The real G1 demonstrations include jabs, hooks, horse-stance punches, front kicks, jump kicks, side kicks, back kicks, spinning roundhouse kicks, a full 360-degree spin, deep squats, stretching, dance, and Tai Chi. These videos establish breadth that a single metric cannot capture. Jumping and spinning motions require the controller to tolerate a temporary loss of support and recover balance after landing, making them stronger transfer tests than a static pose.
The quantitative real-world evaluation is narrower. The authors run ten Tai Chi trials and compare onboard sensor metrics with MuJoCo. Table 2 lists local body-position, joint-position, body-velocity, body-acceleration, and joint-velocity errors. Real local body position is 36.64±2.592 versus 33.18±2.720 in simulation, and joint position is 1130.05±9.478 versus 1061.24±83.27. The values remain in a similar range.
| Platform | Local body position | Joint position | Body velocity | Body acceleration | Joint velocity |
|---|---|---|---|---|---|
| MuJoCo | 33.18±2.720 | 1061.24±83.27 | 2.96±0.342 | 2.90±0.498 | 67.71±6.747 |
| Real G1 | 36.64±2.592 | 1130.05±9.478 | 3.01±0.126 | 3.12±0.056 | 65.68±1.972 |
This is meaningful evidence of sim-to-real transfer, but its scope should remain explicit. Only Tai Chi is quantified. The root is fixed at the origin on the real robot because global localization is unavailable, so world-space displacement is excluded from the real table. Ten trials can reveal a large transfer failure, but they cannot estimate long-run reliability across floor surfaces, battery states, actuator temperatures, repeated impacts, or operator conditions.
The defensible conclusion is therefore narrower than the demonstration reel. PBHC has been shown to execute many dynamic skills on hardware and to track Tai Chi with simulation-like quantitative error. It has not yet shown statistically robust real-world success rates for the complete set of jumps, spins, and high-speed kicks.
Limitations and Open Risks
The authors identify two major limitations. First, the robot has no terrain perception or obstacle avoidance. The policy assumes a predictable flat floor and does not consume depth, material, or obstacle geometry, so the results do not establish autonomous operation in complex environments. Second, every motion receives a separate policy. There is no single general controller that can switch among the full repertoire or compose learned skills online, and training cost grows with the number of motions.
This analysis adds three risks. The physics filter uses manually chosen stability, velocity, height, and maximum-gap thresholds, but its acceptance study contains only ten motions. The contact estimator reaches 91.4 percent accuracy on only ten labeled samples. Both may shift under different clothing, body shapes, camera angles, or floors. Separately, the real-robot quantitative study covers ten Tai Chi trials and does not report repeated success rates, impact loads, joint heating, or safety recovery for the most dynamic skills.
The adaptive rule also compresses each error channel into a mean. It does not model temporal correlation or identify a small but difficult segment inside an otherwise easy motion. If a few rapid transitions dominate failure while most frames are easy, the mean tolerance can remain too loose for the critical interval. A future version could use quantiles, phase-aware tolerance, or localized error statistics rather than one global average per reward term.
Finally, the paper's evidence is platform-specific. The hand-selected 23 controlled joints, actuator model, PD gains, mass ranges, and contact behavior all belong to the G1 setup. The high-level recipe should transfer to other humanoids, but the numerical thresholds and randomization ranges should not be copied without revalidation.
Reproducibility and Code Audit
The public repository exposes the major pieces of the paper. The retargeting module loads SMPL motion and emits the 23-DoF G1 trajectory. The motion-filter module computes center of mass and center of pressure from mesh volume and contact pressure. The HumanoidVerse environment implements adaptive tracking error averages and non-increasing sigma updates. These are direct code-level correspondences to the method rather than merely repository names attached to the paper.
The license requires care. PBHC is released under CC BY-NC 4.0, not a permissive software license. Research and other non-commercial use are allowed within the terms, while commercial deployment or redistribution requires separate authorization. A serious reproduction also needs to inspect the licenses and provenance of motion datasets, SMPL dependencies, the simulator, and robot-vendor software; public policy code does not make the entire stack commercially reusable.
The most transferable ideas are interfaces rather than constants. Human motion should pass an explicit physical acceptance test before policy training. Contact assumptions should connect data cleanup, reference construction, and rewards instead of appearing only as a foot-slip penalty. Reward scale should be derived from the policy's current error distribution. These principles can survive a change of robot, simulator, or network architecture.
Conclusion
KungfuBot presents a complete path from video to highly dynamic humanoid behavior. Physics-based filtering rejects motions that cannot maintain support, contact-aware correction restores a consistent ground relationship, differentiable inverse kinematics produces a G1 reference, and adaptive exponential rewards tighten the learning objective as tracking improves. An asymmetric actor-critic with domain randomization then transfers the resulting policies to hardware.
The key shift is that imitation is no longer treated as a visual matching problem alone. A human video proposes a motion, the physics filter decides whether that proposal can persist, contact correction places it on the ground, and reinforcement learning discovers the actuation needed to realize it under robot dynamics. This causal chain is the paper's most durable contribution.
Golden quote: A highly dynamic robot skill is not a human trajectory copied onto hardware; it is a physically grounded intention that the controller must re-earn at every contact, flight phase, and landing.
