PAPER DEEP DIVE
PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface
PanTact treats contact, not joint trajectories, as the transfer currency from people to humanoids. Wearable patches are assembled magnetically from 44x44 mm FSR modules; a custom 50x50 mm PCB streams 16 channels at 1 kHz over a CRC-checked RS-485 daisy chain, one Vive tracker per patch maps sensor centers into the world frame, and contact states are binarized against a per-channel adaptive baseline with threshold max(0.35 beta, r_min). The policy observes body-masked 3D egocentric point clouds, both wrist images and proprioception, and at 10 Hz predicts an action chunk of H=16 steps containing a look-at point, EEF poses with gripper widths, and per-sensor contact goals (binary state plus 3D position). A soft-minimum proximity d(p,q) with T = 0.02 m hands each active contact to whichever robot surface is currently nearest, and active goals are offset by delta along the pressing direction to form virtual goals that generate pressing force, so the G1 holds objects without any tactile sensor on its body. Three control formulations trade off differently: training-free upper-body IK, WBIK whose dense reference is executed by SONIC, and a sparse contact tracker trained with PPO over 4096 simulation environments for 30k iterations (50 Hz, 29 joints, S=8 upper-body / S=16 whole-body contact slots). On a 29-DoF Unitree G1: Grasp-and-Embrace reaches 85% with upper-body IK and 80% with the sparse tracker against 0% for Humanoid UMI; Sweep-and-Stow reaches 90.63% end to end while Humanoid UMI++ scores 18.75% on each stage and 0% end to end; Carry-and-Push is 90% versus 40%. In simulation replay, adding contact goals lowers contact distance for all three controllers at almost no cost in EEF accuracy.
Paper: PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface (arXiv:2610.12558, arxiv.org/abs/2610.12558, v1 submitted 2026-10-08, primary category cs.RO)
Authors: Xiaomeng Xu, Han Zhang, Zhian Ruan, Xiaofeng Guo, Jinkun Cao, Ruihan Yang, Carmelo Sferrazza, Shuran Song, Guanya Shi (Stanford University, Amazon FAR, Carnegie Mellon University; Ruihan Yang, Carmelo Sferrazza, Shuran Song and Guanya Shi are marked Equal Advising)
Code status: Not open-sourced. The paper and the project page pantact.github.io publish videos and hardware notes only; at the time of this reading there is no public code repository, and a GitHub search on the title plus author names turns up no official implementation. The surrounding components each have their own provenance: the collection interface extends the UMI and Humanoid UMI (HoMMI) hardware line, the dense whole-body tracker reuses SONIC, and the sparse contact tracker is trained from scratch in simulation by this paper. Equation numbers, hyperparameters and experimental figures below all correspond to the arXiv v1 full text and appendix.
Figure 1: paper Fig. 1. (a) The wearable collection interface records whole-body contact alongside egocentric vision and motion; (b) contact as a cross-embodiment interface: the policy predicts contact goals and the whole-body controller picks which robot surface realizes them; (c) whole-body dexterous skills deployed on a Unitree G1, including embracing, sweeping and pushing.
One-Sentence Summary
PanTact makes contact, rather than joint trajectories, the currency of human-to-robot transfer: wearable contact patches record where and when a person touched the world, a visuomotor policy predicts switchable contact goals, and a whole-body controller decides on its own which robot surface realizes them, so a Unitree G1 with no body-mounted sensors at all learns embracing, sweeping, knee pushing and loaded walking, all manipulation that exceeds what two grippers can do.
Background and Motivation
People never manipulate with their hands alone. Carrying something bulky, the arms and torso wrap around it together; clearing a table, a forearm sweeps several objects into one pile; with both hands full, a knee nudges a box out of the way. The authors name this capability whole-body dexterity: accomplishing manipulation with any suitable body surface. Humanoids share morphology with humans and own dozens of actuated degrees of freedom plus potential contact surfaces over the entire body, yet mainstream loco-manipulation research restricts manipulation to the two hands, leaves the rest of the body to locomotion and balance, and deliberately avoids contact between body and environment. That idles most of the manipulation potential a humanoid body actually has.
Using that potential requires whole-body motion and contact sequences across body surfaces to be coordinated, and human demonstrations supply exactly such coordinated samples. The question then becomes: can a humanoid learn whole-body dexterity directly from human demonstrations? The authors observe that existing imitation-learning pipelines collect and imitate end-effector (EEF) trajectories or joint angles, so contact information is never carried explicitly through any of the three stages of demonstration collection, policy prediction and robot execution. Extending to whole-body dexterity therefore runs into three interlocking difficulties.
The first is collection: existing interfaces record only EEF motion or joint angles and never measure where and when body contact occurs. The second is transfer: EEF goals alone leave body contact completely undefined, while retargeting whole-body joints onto a robot changes contact locations because human and robot shapes and proportions differ. The third is execution: controllers designed to track motion references or EEF goals have no explicit objective of achieving contact goals on the body, yet they must insert those goals while still maintaining motion tracking and balance.
The key observation is that contact is manipulation-centric information that can be decoupled from embodiment-specific motion. Contact location and timing describe the task-level physical interaction, whereas which body surface and which joint configuration produce it belong to the robot. Contact also marks which body motions in a demonstration must be preserved, letting the robot learn only task-critical contact instead of reproducing free-space body postures. The authors therefore use contact as a cross-embodiment interface: measure contact on the human, have the policy predict its desired state and location, and have the controller track those contact goals that are active. Because an active goal is not bound to a particular robot link, the controller is free to choose a configuration that suits its own body and keeps it balanced.
PanTact is the complete system built around that interface, with three components: a contact-complete robot-free collection system (reconfigurable modular contact patches layered on top of egocentric sensing and bimanual UMI grippers); a contact-predicting whole-body manipulation policy (body regions masked geometrically in the observation, jointly predicting EEF trajectories and per-sensor contact goals); and contact-aware whole-body controllers (three formulations: upper-body contact-aware IK, whole-body IK executed by a dense tracker, and direct sparse contact tracking). The system is deployed on a 29-DoF Unitree G1 with no additional body contact sensors on the robot side.
Preliminaries: Three Threads Converge
The first thread is robot-free demonstration collection. UMI removed the dependence on a robot body with a handheld gripper; follow-up work added a head-mounted camera to form egocentric observations, and the humanoid version further records body or keypoint trajectories. The shared blind spot of these interfaces is that none measures body contact, and the PanTact patches fill precisely that gap.
The second thread is diffusion policies and egocentric representation. Diffusion policy emits a chunk of future actions, and paired with a DINO-style visual backbone it learns actions end to end from images; work such as HoMMI goes further by using 3D egocentric point clouds to reduce sensitivity to viewpoint differences. PanTact keeps this architecture but geometrically masks the parts of the point cloud that belong to the body, narrowing the appearance and viewpoint gap between human and robot.
The third thread is whole-body control. One family maps task-space commands to joint velocities by optimization, typically a quadratic program over differential IK; another family trains reinforcement-learning trackers that execute dense whole-body joint references, such as SONIC. The former is interpretable and training-free but bounded by model accuracy; the latter is robust but requires the reference to be reasonable. The three control formulations in PanTact are exactly the different ways of feeding contact goals into these two kinds of executor.
Method in Detail
Modular contact patches. Existing whole-body tactile skins are usually custom-built for one robot, while PanTact must serve different demonstrators and different body regions, so the sensing surface is made into small assemblable modules. Each module is a 44x44 mm cell carrying a single force-sensitive resistor (FSR); a custom 50x50 mm PCB streams up to 16 FSR channels simultaneously at 1 kHz. Every FSR forms a voltage divider with a fixed 1 kOhm resistor and is read by a 12-bit ADC with 8x oversampling. Modules communicate over a multi-drop daisy-chained RS-485 bus using a request-response protocol with CRC checking, and direct USB readout is also supported. Each module has connection slots on all four sides, and press-fit magnetic connectors let modules be assembled into patches of arbitrary shape and area; three connector types are designed: a full connector bridging adjacent sensing modules, a strap connector that fixes the patch to the body and provides cable strain relief, and a tracker connector that mounts an HTC Vive tracker to record the patch pose in 6 DoF. The paper shows arm and knee patch configurations.
Figure 2: paper Fig. 2. Sensor modules, connector modules and strap modules snap together magnetically into a patch of any shape and extent; the figure shows a UMI gripper combined with a Vive tracker, plus arm and knee patch assemblies, with magnets seated in circular recesses.
Contact detection and localization. FSR resistance drifts with temperature and mounting pressure, so the authors avoid an absolute threshold and binarize each channel against its own adaptive baseline. With r the raw ADC reading of channel i and beta its running baseline, the contact state during a demonstration is
$$c_{t,i}=\mathbf{1}\!\left[\,\beta_{i}-r_{t,i}>\max(0.35\,\beta_{i},\,r_{\min})\,\right] \tag{8}$$
where r_min is a fixed noise floor and the 0.35 beta term makes the threshold adapt to the baseline. Each patch carries one tracker, and a rigid transform maps every sensor center into the tracking world frame; at each synchronized timestep an N x 4 contact array is stored, with N the number of FSRs and each row holding that sensor's binary contact state and 3D position. These sequences are the supervision for the policy's contact-goal prediction, and the binary state tells the controller when to track a contact location.
Synchronization and data format. The head stereo camera (ZED X Mini) and two fisheye wrist cameras (ZED X One S) record images at 60 fps, the HTC Vive trackers on head and both wrists record poses at 250 Hz, and gripper width is recorded at 1 kHz; all streams share one monotonic clock and are recorded on an NVIDIA Jetson AGX Orin. In post-processing everything is aligned to the 60 Hz camera timeline and then sampled at 20 Hz for training. Camera resolution is uniformly 960x600, stereo depth is computed offline by ZED's own neural depth model, and the back-projected point cloud then goes through the body masking described below. One demonstration ends up as the synchronized sequence of Eq. (7): egocentric image and point cloud, both wrist images, head pose, both EEF poses with gripper widths, and the per-sensor contact triplets.
An embodiment-agnostic observation space. Whole-body contact often happens behind the grippers, outside the wrist cameras' field of view. Egocentric vision sees a wider scene and the whole-body interaction, but at the cost of a larger human-robot gap: the demonstrator's body repeatedly enters the frame, creating an appearance mismatch between training and deployment, and the height difference between person and robot adds viewpoint misalignment. The authors keep HoMMI's 3D egocentric representation and mask body regions geometrically: each gripper frame and each contact patch defines a half-space facing the body (the arm one starts behind the gripper, the body-surface one starts behind the patch surface), the intersection of these half-spaces approximates the body region, and the corresponding points are masked inside the attention layers of the egocentric visual encoder. The construction needs no appearance model of either party and scales to any number of patches. The full policy observation is
$$\mathbf{o}_{t}=\left(P^{\mathrm{ego}}_{t-h+1:t},\;I^{L}_{t-h+1:t},\;I^{R}_{t-h+1:t},\;\mathbf{x}^{L}_{t-h+1:t},\;\mathbf{x}^{R}_{t-h+1:t}\right) \tag{1}$$
with h=2 history steps, P the masked egocentric point cloud, I the two wrist images, and each x holding one EEF pose plus gripper width. In implementation the authors finetune a pretrained DINO-v3 vision transformer and attach sinusoidal encodings of the 3D coordinates to the point-cloud patch features.
Figure 3: paper Fig. 3. The contact-predicting whole-body manipulation policy takes body-masked 3D egocentric observations, wrist images and proprioception, and predicts EEF trajectories plus contact goals (position with a binary state); the contact-aware whole-body controller coordinates whole-body motion, tracks the EEF commands and realizes active contact goals with suitable robot surfaces.
The contact-predicting action space. The policy uses a diffusion policy architecture, runs at 10 Hz and predicts an action chunk of H=16 steps, where each step is
$$\mathbf{a}_{t}=\Big(\boldsymbol{\lambda}_{t},\;\mathbf{x}^{L}_{t},\;\mathbf{x}^{R}_{t},\;\{(\hat{c}_{t,i},\;\hat{\mathbf{p}}_{t,i})\}_{i=1}^{N}\Big) \tag{2}$$
containing the head 3D look-at point lambda, the target EEF poses and gripper widths x, and the set of contact goals. The binary state c-hat decides whether the contact position p-hat is a tracking target: only an active goal (c-hat = 1) requires the robot to be in contact with the world at that position, while an inactive goal (c-hat = 0) does not require the robot to reproduce the corresponding sensor's free-space trajectory, which keeps the controller from being over-constrained by irrelevant contact. Point clouds, proprioceptive poses, EEF poses and contact positions inside the observation history and the action chunk are all expressed in one frame, relative to the most recently observed left-gripper pose, reducing sensitivity to human-robot kinematic differences. The denoising head is a DiT trained with epsilon prediction and sampled with 16 DDIM steps at inference.
An any-surface contact objective. The whole-body controller must turn the policy's EEF trajectories and contact goals into whole-body robot motion, tracking contact positions only when c-hat = 1. Let q be the robot configuration, B the set of candidate robot links, and d_b(p,q) the distance from point p to the collision surface of link b. The proximity of a commanded contact p to the robot is defined as a soft minimum
$$\mathrm{d}(\mathbf{p},q)=-T\log\sum_{b\in\mathcal{B}}\exp\!\big(-\mathrm{d}_{b}(\mathbf{p},q)/T\big) \tag{3}$$
with T = 0.02 m controlling smoothness. For an active contact the controller drives d(p-hat, q) toward zero. What matters about this soft-minimum form is that it never specifies which surface makes the contact: the gradient naturally flows to the few currently nearest surfaces, leaving the choice of body part to the controller.
Virtual contact goals. Reaching a contact position alone does not specify pressing force. The authors follow the virtual-goal principle: push the controller setpoint into the object along the pressing direction, so that when the object constrains the robot surface at the contact, the residual tracking error is converted into pressing force by the low-level controller. At deployment an active goal is offset as
$$\tilde{\mathbf{p}}_{i}=\hat{\mathbf{p}}_{i}+\hat{c}_{i}\,\delta\,\mathbf{n}_{i},\qquad \delta=\min\!\left(\frac{F_{\mathrm{des}}}{k_{\mathrm{virt}}},\;\delta_{\max}\right) \tag{4}$$
where n is the unit pressing direction pointing into the object, F_des the nominal single-point pressing force, k_virt the design stiffness and delta_max an offset limit. Tracking p-tilde encourages the controller to generate pressing force, which is the mechanism that lets the robot hold on without any real tactile sensor on its body.
Formulation I: upper-body contact-aware IK. The most direct option adds the contact task to the differential-IK quadratic program:
$$\min_{\dot{q}}\;\;\sum_{k\in\mathcal{T}}w_{k}\big\|J_{k}(q)\dot{q}-K_{k}e_{k}(q)\big\|^{2}+\varepsilon\|\dot{q}\|^{2}\quad\text{s.t.}\quad \dot{q}\in\mathcal{C} \tag{5}$$
The task set T contains EEF position and orientation tracking, a nominal posture regularizer and active contact tracking; epsilon penalizes joint velocity and C imposes joint position and velocity limits. For each active contact p-hat the solver finds the closest point s_b on every candidate surface, keeps the three nearest surfaces, sets their contact task weights proportional to exp(-d_b/T) normalized over the selected surfaces, and attaches to each a position task pulling s_b toward the goal. The appendix gives concrete weights: EEF position 500, orientation 200, contact task 250, and posture regularization 1 per joint group; joint velocity is capped by velocity damping with gain 0.95, at 1 rad/s for arms and waist and 2 rad/s for legs. The solved joint velocities are integrated into arm joint position targets while the legs go to a standing controller. This formulation needs neither training data nor simulation, but it handles upper-body tasks only, with the legs fixed.
Formulation II: WBIK plus SONIC. To extend contact tracking to the lower body, the authors expand the QP to whole-body degrees of freedom and reuse the existing SONIC dense motion tracker for execution. When the EEF and contact goals are reachable within the current support polygon both feet stay on the ground and the solver adjusts posture by squatting and leaning; otherwise a step is commanded. Stepping triggers when the pelvis drifts more than 0.12 m horizontally from the support midpoint for 0.2 s, then releases the SE(3) constraint of the trailing foot and tracks a 0.4 s cycloidal swing with 0.06 m peak lift, choosing a stride of 0.10 to 0.25 m along the pelvis drift direction. During the swing the position and orientation weights are 1000 and 100; after touchdown that foot is fixed and a 0.4 s cooldown is enforced. The resulting whole-body configuration trajectory is handed to SONIC as a dense reference tracking all joint targets. This option enables lower-body contact and walking, but a dense reference pins down a great deal of task-irrelevant postural detail, and tracking it competes with EEF and contact accuracy; moreover the IK solver does not model SONIC's execution limits, so it can produce dynamically infeasible references.
Formulation III: the sparse contact tracker. To keep whole-body coordination free at execution time, the authors train a controller that tracks sparse EEF and contact goals directly, with no intermediate dense reference. The tracker runs at 50 Hz and outputs 29 joint position targets to PD control. Its input is proprioception (joint positions and velocities, root orientation and angular velocity, previous action) plus a goal block: masked future joint targets, EEF targets, and S contact slots. The upper-body version uses S=8 (four per forearm) and the whole-body version S=16 (four more per knee); each slot holds a 3D point in the robot root frame with a contact-state flag, and a validity mask says which slots this episode uses. At deployment the controller receives only EEF targets and active contact goals, with the remaining targets left masked.
Hindsight contact labeling. Training such a controller needs (goal, motion) pairs in which the contact goals are actually reachable, and standard motion-capture data does not provide them. The construction is this: replay the reference motion in simulation with an expert tracker, then sample goal points uniformly on the collision surfaces of body parts that can make contact, along the trajectory actually achieved. The same labeling is applied both to the general mocap dataset used to train SONIC, which supplies broad whole-body motion coverage, and to this paper's human demonstrations, executed with WBIK plus SONIC under trajectory perturbation, which supplies task-specific coverage. During training each episode draws S slots from the segment's candidate pool, randomly decides how many are active, and randomly labels each selected slot as in-contact (tight tracking) or pre-contact (soft approach). The contact goals produced by replay thus become the commands the tracker learns to satisfy, giving supervision without any manual annotation.
Training objective and randomization. The reward is r = r_track + r_contact + r_reg. Over the valid slot set V, the contact reward is
$$r_{\mathrm{contact}}=w\sum_{i\in\mathcal{V}}\left[0.3\,c_{i}\,e^{-\mathrm{d}_{i}^{2}/\sigma_{a}^{2}}+0.1\,(1-c_{i})\,e^{-\mathrm{d}_{i}^{2}/\sigma_{s}^{2}}\right] \tag{6}$$
where d_i = d(p-hat_i, q), c_i distinguishes in-contact from pre-contact goals, sigma_a = 0.05 m applies to tight tracking and sigma_s = 0.25 m to soft approach, and the weight w ramps up during training so contact tracking is introduced only after basic motion tracking is in place. r_track rewards body position, orientation, velocity and EEF tracking, while r_reg penalizes action jumps, joint limit violations and uncommanded contact. Training uses PPO over 4096 parallel simulation environments for 30,000 iterations. Domain randomization covers static friction [0.3, 1.6], dynamic friction [0.3, 1.2], restitution [0, 0.5], base center-of-mass offset up to (2.5, 5, 5) cm, joint zero-position bias of +/-0.015 rad, PD gains of +/-10%, link mass multipliers [0.8, 1.5] and base mass offset [-1, 3] kg, plus pelvis velocity perturbations up to 0.4 m/s every 1.5 to 3 s. This formulation learns whole-body contact realization directly, at the price of simulation training and dependence on the training distribution.
flowchart TD H[Human demonstrator
head stereo and wrist fisheye Vive 250Hz UMI gripper 1kHz] --> P[Modular contact patches
44mm FSR magnetic assembly RS-485 1kHz] H --> S[Jetson AGX Orin synchronized logging
monotonic clock aligned 60Hz training sample 20Hz] P --> B[Adaptive baseline binarization
N x 4 contact array state plus 3D position] S --> B B --> O[Masked egocentric point cloud
half-space intersection masked in attention] O --> D[Diffusion policy 10Hz DiT denoising
history h=2 action chunk H=16] D --> A[Action chunk
look-at EEF pose gripper width contact goals] A --> C{Whole-body controller choose one} C -->|training free| K1[Upper-Body IK QP
three nearest surfaces weight exp of minus d over T] C -->|dense reference| K2[WBIK plus SONIC
whole-body QP reference stepping logic execution] C -->|sparse direct| K3[Sparse Contact Tracker
50Hz 29 joints PPO 4096 envs] K1 --> V[Virtual contact goals
offset delta along pressing direction creates force] K2 --> V K3 --> V V --> G[Unitree G1 29-DoF
no body sensors async inference resampled to 50Hz]
The deployment loop. Policy inference and low-level control run asynchronously: each predicted action chunk is stamped with the time of its source observation, the scheduler drops expired steps, replaces overlapping targets with the newest prediction, and resamples commands to the tracker's 50 Hz control rate. Policy actions are expressed in the left-gripper frame of the source observation and are transformed into the world frame at execution time using pose estimates from Super Odometry, which exploits the head-mounted LiDAR-inertial unit the G1 already carries, so the robot runs in real spaces without external motion capture.
Experimental Results
Evaluation is organized around four capabilities: cross-embodiment contact transfer (needed by every task), non-prehensile upper-body manipulation (embracing and sweeping), lower-body loco-manipulation (walking, pushing and balancing at once), and whole-body contact timing (establishing, maintaining and releasing contact on different body regions as a task progresses). The two baselines are pose interfaces with no explicit contact goals: Humanoid UMI, where the policy predicts only EEF actions and the controller tracks only EEF trajectories, and Humanoid UMI++, which additionally predicts and continuously tracks a knee pose with each pose goal bound to a fixed link. All hardware experiments use a 29-DoF Unitree G1 with grippers and cameras, and every method is trained on the same demonstrations and evaluated from matched test configurations.
Figure 4: paper Fig. 4, Grasp-and-Embrace. (a) Autonomous rollout with Upper-Body IK, grasping a small object first and then embracing a large one with both arms; (b) the overlaid test configurations; (c) a typical Humanoid UMI failure, insufficient arm contact so the large object is not supported; (d) success-rate bar chart; (e) tracking curves from a real rollout.
Grasp-and-Embrace. The robot first picks a small object off the table, then embraces a large object with both arms and moves the two together; 100 demonstrations, 20 rollouts. PanTact with Upper-Body IK reaches 85% success and PanTact with the Sparse Contact Tracker 80%, the occasional failure coming from the gripper missing the small object. Humanoid UMI never supports the large object: its controller tracks only EEF commands, leaving arm posture completely undefined. The authors also observe the baseline's grippers extending forward until they approach the arm joint limits, possibly accumulated tracking error pushing the observation outside the policy's training distribution. The tracking curves show that Upper-Body IK is accurate on both EEF and active contact tracking, while the sparse tracker has slightly larger errors and a tracking dead zone.
Figure 5: paper Fig. 5, Sweep-and-Stow. (a) WBIK plus SONIC rollout: the arm sweeps objects on the table while the lower body pushes and walks; (b) test configurations varying seen and unseen object combinations, door opening angle and robot-to-cabinet distance; (c) a typical Humanoid UMI++ failure; (d) per-stage and end-to-end success rates.
Sweep-and-Stow. The robot uses its arm to sweep one or more upright objects from the table into an open storage bin, then closes the cabinet door with a lower-body surface. The sweep score counts objects that entered the bin, the door score gives 50% for less than half closed and 100% for more than half, and the end-to-end score is the product of the two stage scores. With 150 demonstrations and 16 rollouts split evenly across single-seen, double-seen, single-unseen and double-unseen conditions, PanTact (WBIK plus Dense Tracker) reaches 90.63% end to end, while Humanoid UMI++ gets 18.75% on sweeping and 18.75% on door closing and 0% end to end. What stands out is not the number but the way it generalizes: the robot sweeps with the wrist, forearm or the side of the gripper and closes the door with a knee or a foot, going beyond the forearm and left-knee contacts that appear in the demonstrations. The occasional half-closed door happens when the controller abandons a contact goal to stay balanced. UMI++ instead misses sweeps or knocks objects out of the bin for lack of arm regularization, drives its arms toward joint limits more often, and fails to step or close the door, which the authors attribute to human-robot proportion differences: tracking the demonstrated hand and knee poses simultaneously can demand a relative arrangement the robot cannot realize.
Figure 6: paper Fig. 6, Carry-and-Push. (a) WBIK plus SONIC rollout: walking while carrying a load with both arms and pushing a large box on the floor with the lower body; (b) the test scene; (c) a Humanoid UMI++ failure where insufficient contact support drops the carried box; (d) success rates; (e) tracking curves from a real rollout.
Carry-and-Push. The robot carries a box with both arms, walks to a larger box on the floor and pushes it forward more than 20 cm with its lower body; success requires keeping the carried box throughout the walk. With 110 demonstrations and 10 rollouts (five with the floor box on the left, five on the right), PanTact (WBIK plus Dense Tracker) reaches 90%, the Sparse Tracker 60% and Humanoid UMI++ 40%; failures are mostly dropping the carried box. The tracking curves make the point clearly: UMI++ tracks the knee pose continuously and therefore has large knee pose error, whereas PanTact applies leg contact goals only during active contact phases and follows the commanded contact positions noticeably better in exactly those phases.
| Task | Demos / rollouts | Baseline | PanTact configuration | Success rate (%) |
|---|---|---|---|---|
| Grasp-and-Embrace | 100 / 20 | Humanoid UMI | - | 0 (never supports the large object) |
| Grasp-and-Embrace | 100 / 20 | - | Upper-Body IK | 85 |
| Grasp-and-Embrace | 100 / 20 | - | Sparse Contact Tracker | 80 |
| Sweep-and-Stow sweep | 150 / 16 | Humanoid UMI++ | - | 18.75 |
| Sweep-and-Stow sweep | 150 / 16 | - | WBIK + Dense Tracker | 100 |
| Sweep-and-Stow door | 150 / 16 | Humanoid UMI++ | - | 18.75 |
| Sweep-and-Stow door | 150 / 16 | - | WBIK + Dense Tracker | 90.63 |
| Sweep-and-Stow end-to-end | 150 / 16 | Humanoid UMI++ | - | 0 |
| Sweep-and-Stow end-to-end | 150 / 16 | - | WBIK + Dense Tracker | 90.63 |
| Carry-and-Push | 110 / 10 | Humanoid UMI++ | - | 40 |
| Carry-and-Push | 110 / 10 | - | WBIK + Dense Tracker | 90 |
| Carry-and-Push | 110 / 10 | - | Sparse Contact Tracker | 60 |
Table 1: success rates on the three real-robot tasks, taken from the paper Sections 6.1 to 6.3 and the bar charts in Figs. 4d, 5d and 6d; the Sweep-and-Stow end-to-end score is the product of the sweep and door stage scores.
Controller ablation (simulation replay). The authors also replay demonstration trajectories onto the three control formulations in simulation, comparing settings with and without contact goals: 10 Grasp-and-Embrace trajectories for the upper-body evaluation and 20 for the whole-body evaluation (half Sweep-and-Stow, half Carry-and-Push), reporting the median of per-trajectory median errors. Adding contact goals lowers the contact surface distance for every controller at almost no cost in EEF tracking accuracy. With contact goals on, Upper-Body IK has the lowest upper-body EEF error, while the Sparse Contact Tracker has the smallest contact distance and beats WBIK plus SONIC on both metrics. On whole-body tasks the two trackers trade off: the sparse tracker has lower contact distance but higher EEF error. This ablation separates two questions that are easily conflated, whether contact goals are worth having and which executor suits them better.
| Control formulation | Needs training / simulation | Usable body range | Execution | Shortcoming stated by the authors |
|---|---|---|---|---|
| I. Upper-Body IK | No / No | Upper body, legs fixed | QP solves joint velocities, integrated into arm position targets; legs run a standing controller | Upper-body tasks only |
| II. WBIK + SONIC | No / No (reuses a trained tracker) | Whole body, including stepping | Whole-body QP produces a dense reference that SONIC tracks on all joints | Dense reference competes with EEF and contact accuracy; the reference may be dynamically infeasible |
| III. Sparse Contact Tracker | Yes / Yes (PPO, 4096 envs, 30k iterations) | Whole body, including stepping | Outputs 29 joint targets directly at 50 Hz | Depends on simulation training and on the training distribution |
Table 2: comparison of the three contact-aware whole-body control formulations, drawn from paper Sections 5.2 and 6.4.
| Hyperparameter | Value | Hyperparameter | Value |
|---|---|---|---|
| Transformer depth / width / heads | 10 / 768 / 12 | Training diffusion steps / inference DDIM steps | 100 / 16 |
| Training epochs | 500 | Effective batch size | 48 |
| Backbone learning rate | 7.5e-5 | Vision encoder learning rate | 7.5e-6 |
| Learning rate schedule | cosine decay with warmup | Validation split | 5% |
| Policy rate / action chunk / history | 10 Hz / H=16 / h=2 | Tracker rate / joint count | 50 Hz / 29 |
| Contact slots S (upper body / whole body) | 8 / 16 | Contact reward sigma_a / sigma_s | 0.05 m / 0.25 m |
Table 3: key policy and controller hyperparameters, taken from appendix Tables 1 and 2 (appendix B and C) and the main text.
Limitations
The limitations the authors state cluster around the three control formulations. Upper-Body IK needs no training and no simulation but is confined to upper-body tasks with the legs fixed. WBIK plus SONIC enables lower-body contact and walking, yet the dense reference pins down task-irrelevant postural detail whose tracking competes with EEF and contact accuracy, and because the IK solver does not model SONIC's execution limits the reference can be dynamically infeasible. The Sparse Contact Tracker learns whole-body contact realization directly, at the cost of simulation training and dependence on the training distribution, so contact configurations outside the demonstrated coverage are not necessarily reliable.
A second limitation the authors acknowledge shows up in the experimental observations. With no body contact sensors on the robot, whether contact is truly achieved can only be judged indirectly through the control loop, via tracking curves and task success. The failure modes are related: the gripper missing during an embrace, the controller completing only a half-close to preserve balance, and dropping the carried box during loaded walking, the last two being more frequent with the Sparse Tracker and with Humanoid UMI++. Virtual contact goals rest the pressing force on a modeled stiffness, so variation in the compliance of real objects changes the achieved pressing effect directly, and the system has no closed-loop observation of that.
From an evaluation-design standpoint the three tasks use 20, 16 and 10 rollouts, unseen conditions cover only a few new objects and layouts, and the statistical power of the conclusions is correspondingly limited. All experiments run on one G1 in an indoor bench-top scene, and on the demonstration side the contact patches cover only the forearms and knees; contact interaction on the torso, back and head has not yet been collected or validated. Bringing those regions into the interface is an assembly problem in hardware, but a new training problem in data and controller distribution.
Summary and Outlook
The contribution of PanTact is a complete system that treats contact as a first-class citizen. Modular, reconfigurable, low-cost wearable contact patches solve the problem of not being able to measure; an embodiment-agnostic visuomotor policy that predicts contact goals solves the problem of not being able to transfer; three contact-aware whole-body control formulations solve the problem of not being able to execute. All three share the same contact interface, so where and when a person touched during a demonstration can be reproduced on the robot in the robot's own way, with no extra sensors on the robot side.
The open problems this route leaves behind are equally clear: closed-loop verification of contact goals (cheap tactile or vision-based contact estimation on the robot), sparse-tracker behavior on contact configurations outside the training distribution, and data scale under larger contact coverage and more task forms. If contact really can serve as a general cross-embodiment interface, then contact data harvested from everyday human activity could turn directly into a whole-body skill library for humanoids, which is the direction the authors hope the hardware, the learning framework and the control formulations will push.
Golden Quote
"Contact locations and timing specify the interactions needed to accomplish a task, while the robot chooses how to coordinate its body to realize them within its own physical constraints."