DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations

Paper metadata

Title: DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations

Authors: Mingze Sun, Ziang Cao, Yuan Fei, Qiankuan Tao, Zihan Wang, Tianqi Liu, Junhao Chen, Haiwen Diao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu (all affiliated with Ropedia; Ziwei Liu is the corresponding author)

Links: arXiv:2610.29997v1 [cs.RO], 2026-10-21; full text https://arxiv.org/abs/2610.29997 ; project page https://ropedia.github.io/DONIC ; code https://github.com/Ropedia/DONIC

Code status: The repository currently releases only bihoi_pipeline, the multi-view reconstruction and annotation pipeline (license CC BY-NC 4.0), covering masks and metric depth, metric object reconstruction and 6-DoF tracking, joint hand-object optimization, and retargeted export. The training code, checkpoints, and the BiHOI data itself are marked "coming soon" in the README. What is reproducible today is therefore the path from multi-view recordings to robot motion references; the policy side rests on the paper and appendices. Numbers below that concern the data pipeline were checked line by line against the released scripts.

One-line summary

DONIC pairs an object-centered, per-hand reward with clip-centered advantages so that a single policy trains jointly over 3,420 heterogeneous bimanual references; one checkpoint, without fine-tuning, reaches 86.79% success on unseen trajectories and 57.32% on unseen objects, beating the strongest bimanual baseline by 77.36 and 45.73 percentage points.

Figure 1: DONIC overview (paper Fig. 1). BiHOI supplies multi-view bimanual demonstrations that are reconstructed and retargeted to robot hands; DONIC learns a shared policy for diverse bimanual reference-tracking tasks.

Background and motivation

Dexterous manipulation underpins household assistance, industrial assembly, and assistive care, but it demands precise coordination of finger motion and changing hand-object contacts: grasping, reorienting, and repositioning all live in contact timing. Retargeting human demonstrations to robot hands with different kinematics is already hard; generalizing across objects and behaviors is harder. Simulation-ready assets and synthetic trajectories scale up interaction data yet may miss the structure of natural human manipulation, so recent trackers learn from human hand-object motion but remain largely single-handed.

Everyday tasks, however, are often asymmetric bimanual coordination: one hand stabilizes the object while the other opens, rotates, or repositions it. That motivates a generalist bimanual reference tracker that learns from diverse human motion and controls different objects and tasks with one policy. The paper states its goal with restraint: not end-to-end video-to-real, but tracking retargeted bimanual references in simulation, evaluated without fine-tuning on held-out trajectories and held-out objects.

Two hard obstacles stand in the way. First, data: public datasets contain bimanual manipulation, but fully tracked sequences with sustained two-hand contact are scarce, limiting the coordination patterns available for training. Second, optimization: a shared policy faces changing hand roles and contact timing. Hand-pose accuracy does not ensure manipulation success, yet requiring both hands to satisfy contact conditions can mask one hand's progress; unequal clip difficulty also leaves reference-dependent offsets in the advantage estimates, so relatively successful transitions in hard clips get consistently negative advantages.

DONIC answers with three pieces: an object-centered per-hand reward (object-motion feedback stays online throughout, while each hand's contact is supervised only when the reference requires it); Clip-Centered Advantages (CCA, subtracting each clip's own mean advantage after GAE, removing the offset while preserving within-clip differences); and a PPO plus behavioral cloning feedback loop that recycles the shared policy's verified successes into demonstrations. BiHOI closes the data side: 1,575 multi-view sequences, 229 objects, 14.77 hours, emphasizing sustained bimanual contact.

Prerequisites

Reference-conditioned PD control interface. The policy does not emit joint targets directly; it emits bounded corrections added to the next reference frame, clamped to joint limits, and executed by a PD controller. Corrections re-anchor to the fresh reference every step and never accumulate. This interface is inherited from the DexTrack and Dexplore line of work; DONIC keeps it and innovates in reward and training.

PPO plus behavioral cloning feedback loop. Trajectory-specific specialists optimize the references retained by replay screening, and their successful rollouts enter the demonstration library after replay verification. The shared generalist is evaluated periodically on the training pool, and its verified successes also join the library for subsequent PPO+BC updates. Demonstrations come only from training references; held-out references stay isolated throughout.

MANO and kinematic retargeting. Human hands are represented with the MANO model (shape plus pose). Retargeting is position-based kinematic fitting: hand keypoints are fitted frame-wise to robot links with joint limits as hard bounds, initialized from the previous frame's solution and regularized across frames. Smoothing happens only before retargeting.

Open-loop replay screening. Each candidate reference is replayed open-loop with zero policy correction and classified into six outcomes: immediate object loss (A), object ejection (B), contact without lifting (C), partial execution (D), tracking with incorrect release or ending (E), and complete execution (F). A and B are excluded; C through F remain for closed-loop training, because failing to lift under open-loop replay does not rule out recovery by feedback control. Screening decides reference eligibility; it is a different gate from the verification that admits demonstrations.

BiHOI: a multi-view dataset built for two hands

The capture rig is a ring of 32 calibrated RGB cameras around a tabletop workspace at 1920x1232 pixels and 30 fps. Complementary views support reconstruction under occlusion, and calibration provides a common metric frame. BiHOI contains 1,575 interaction sequences over 229 objects and 14.77 hours of unique interaction time (about 1.60M synchronized frames), spanning household items, containers, utensils, tools, and electronics; participants perform natural object-centric interactions rather than fixed motion templates. Recordings include symmetric bimanual manipulation, asymmetric coordination, and coordinated multi-object manipulation.

The selling point is not scale but the density of simultaneous two-hand contact. The paper defines the denominator as frames with at least one hand in contact; on that measure, 65.91% of BiHOI's contact-active time has both hands in contact, exceeding H2O, the highest analyzed public subset at 38.57%, by 27.34 percentage points. For contrast, the public training pool measures 31.40% on the same measure, and only 22.56% if the denominator is all frames. Stating the denominator is what makes the number comparable, and the paper is more careful here than most dataset papers.

Source	Capture	Sequences	Frames	Objects	Bimanual contact (%)
BiHOI (ours)	MV RGB	1,575	approx. 1.60M	229	65.91
GRAB (2020)	Marker MoCap	1,334	1.622M	51	32.21
OakInk2 (2024)	MV RGB + MoCap	627	4.01M	75	24.97
TACO V1 (2024)	MV RGB + ego RGB-D	2.5K	5.2M	206	15.69
GigaHands (2025)	MV RGB	-	183M	417	34.74
H2O (2021)	MV RGB-D	-	571.6K	8	38.57
HOI4D (2022)	Ego RGB-D	4,000	2.4M	800	-
HO-Cap rev. (2025)	MV RGB-D	64	656K	64	-

Table 1: Dataset overview (paper Table 1). Bimanual contact is both-hand-contact frames divided by frames with at least one hand in contact, pooled over analyzed recordings; BiHOI's 14.77 hours count synchronized time steps once at 30 fps.

Turning recordings into robot references takes five stages: masks and metric depth, metric object reconstruction and tracking, participant-specific hand calibration, hand-motion reconstruction, and joint refinement. The object branch uses SAM3 in two stages (first frames from all cameras form an ordered cross-camera sequence initialized by a single object prompt, then per-camera temporal tracking), Depth Anything 3 for spatially consistent depth anchored to metric scale by extrinsic-scale alignment, MV-SAM3D for a canonical mesh whose physical scale comes from back-projected depth, and FoundationPose for 6-DoF object motion with multi-view pose consensus handling occlusion and re-registration when consensus fails.

Figure 2: Reconstruction and annotation pipeline (paper Fig. S1). Synchronized multi-view RGB yields SAM3 masks and metric depth; the object branch performs metric reconstruction and robust multi-view 6D tracking, the hand branch performs participant-specific calibration and multi-view hand-pose reconstruction, and a depth-guided joint optimization closes the loop.

Object scale estimation deliberately avoids the ambiguous canonical orientation: robust principal-component extents are computed for the depth cloud and the mesh samples, radial outliers removed, 1-99 percentile extents sorted along the three principal axes, divided per axis, and combined by median:

$$s_{\mathrm{obj}}=\operatorname{median}_{k\in\{1,2,3\}}\frac{e_{k}^{\mathrm{depth}}}{e_{k}^{\mathrm{mesh}}}.\tag{B.2}$$

Only scale transfers; the canonical mesh keeps its own orientation. The median keeps one occluded or misestimated axis from dragging the whole estimate. Object tracking initializes translation by triangulating centers of minimum-area rectangles fitted to masks and orientation from the previous accepted pose, refines the ten views with the largest visible masks independently with FoundationPose, groups candidate poses by geometric consensus, then repairs translation spikes with Kalman filtering and rotation spikes on SO(3), and finally refines the full trajectory against fused metric depth with alternating ICP, rejecting correspondences beyond 3 cm; the geometric term weights point-to-plane and point-to-point residuals 0.7 and 0.3, and corrections stay within 3 cm and 20 degrees of the input trajectory.

The hand branch starts with participant-specific calibration: a flat-hand sequence provides metric triangulated positions for a 21-joint skeleton, calibrating an initial MANO shape and a physical scale from the ratio of 20 bone lengths (MANO shape coefficients are not a global scale). The calibration is reused across sequences. WiLoR supplies 21 image keypoints per hand per view; per joint, valid two-view pairs generate triangulation hypotheses, inliers are reprojection residuals below 50 pixels with at least three views, and the selected hypothesis is re-triangulated with all inliers. Note that reconstruction runs at a 10 fps sampling rate, distinct from the 30 fps capture rate; each joint is filtered with a constant-velocity state, measurement and acceleration-process standard deviations 0.01 m and 0.12 m/s squared, prediction-only on missing observations. MANO fitting keeps shape fixed and estimates pose and global motion.

Hands and object are then refined jointly against shared metric depth, with per-frame hand pose and global transforms, object pose, and one sequence-shared object scale variable as unknowns:

$$\mathcal{L}=10\,\mathcal{L}_{\mathrm{SDF}}+\mathcal{L}_{\mathrm{reg}}^{h}+\mathcal{L}_{\mathrm{reg}}^{o}+\mathcal{L}_{\mathrm{smooth}}^{h}+\mathcal{L}_{\mathrm{smooth}}^{o}.\tag{B.5}$$

The SDF term aligns reconstructed geometry to a signed distance field built from depth (MeshSDF formulation following HO-Cap, 32 depth views in the interaction region, 3 mm voxels); the two regularization terms preserve hand and object initialization and the two smoothness terms regularize temporal evolution. Optimization proceeds in three phases: priors and smoothness first, then SDF alignment, then object scale with poses fixed. The paper states explicitly that there are no explicit contact, penetration, collision, or snapping losses here, so the hand-object relationship is constrained only indirectly through the depth field, which is one source of the contact-proxy error discussed later.

The standardized export uses a right-handed, +Z-up world frame and records its 4x4 transform from the reconstruction frame; the tabletop defines z=0, estimated from the fifth percentile of the object bottom; MANO uses 45 axis-angle pose coordinates without PCA, a flat-hand mean, and no added mean-pose offset; object vertices keep canonical coordinates without recentering, axis flips, or mirroring. Retargeting fits fifteen targets per hand (five MANO fingertips, five additional finger keypoints, and the wrist repeated five times) to Wuji-hand links with a SmoothL1 position loss at transition scale 0.02 m, NLopt optimization with joint limits as hard bounds, tolerance 1e-3, previous-frame initialization, and inter-frame regularization norm_delta of 4e-3. Source motion is interpreted at 10 fps and converted to 30 fps control references; pre-retargeting cleanup uses a 0.10 m per-frame displacement threshold at 30 fps with at most three despiking passes. Open-loop replay screening then assigns each reference to categories A through F.

Method

4.1 Task definition: one policy, two hands, one object

Given a reference for two hands and one object, a single policy controls both robot hands to reproduce the object motion through physical interaction. For a clip spanning T control steps the reference is

$$\tau=\big\{(\hat{\bm{q}}^{L}_{t},\hat{\bm{q}}^{R}_{t},\hat{\bm{p}}^{o}_{t},\hat{\bm{R}}^{o}_{t})\big\}_{t=0}^{T}.\tag{1}$$

Here t is the frame index, h denotes left or right, and hats mark reference quantities. Each hand configuration contains wrist coordinates and finger-joint angles, d controlled coordinates in total; object position and orientation are a 3-vector and an SO(3) rotation. The two hand configurations concatenate to 2d dimensions; unhatted counterparts are simulated states. The reference object linear velocity comes from finite differences. Units are metres, radians, and metres per second. Training follows DexTrack's PPO plus behavioral cloning recipe, and at test time the policy tracks unseen trajectories and objects without fine-tuning.

4.2 Reference-conditioned bimanual control: 3,016-dim observation, 52-dim action

A single stochastic policy predicts joint actions for both hands from the observation. The observation concatenates two per-hand blocks containing proprioception, object pose and velocity, upcoming reference targets, reference-state errors, and fingertip-to-object distance vectors, with spatial features expressed in each wrist's heading frame. It appends a basis point set shape encoding, object scale and centroid, hand identities, and the fixed left-hand reward weight; forty contact-force channels are zeroed, because simulated forces supervise rewards only and never enter the policy input. The appendix gives the dimension budget: 3,016 total, equal to 2,884 base features plus a 132-dimensional object descriptor (128 BPS features plus scale plus centroid); each hand contributes six wrist coordinates and twenty finger joints, so the two-hand action is 52-dimensional.

The policy outputs bounded corrections added to the next reference, forming a PD target:

$$\bm{q}^{\mathrm{pd}}_{t}=\operatorname{clamp}\!\left(\hat{\bm{q}}_{t+1}+\bm{s}\odot\bm{a}_{t},\ \bm{q}_{\min},\bm{q}_{\max}\right),\qquad \bm{a}_{t}\in[-1,1]^{52}.\tag{2}$$

The vector s sets per-coordinate correction ranges: 0.1 m for wrist translation, 1.0 rad for wrist rotation, and 0.15 rad for finger joints. Corrections anchor to the next reference at every step and do not accumulate; the PD controller executes the target and the resulting state supplies the next observation.

4.3 Object-centered per-hand reward

Hand-pose accuracy guarantees neither contact nor object motion, and the two hands may engage asynchronously with different roles. The reward therefore keeps object-motion feedback online throughout while supervising each hand's contact only when the reference requires it. The object term checks position, orientation, and linear velocity jointly:

$$r_{t}^{\mathrm{obj}}=e^{-\kappa_{p}\|\bm{p}^{o}_{t}-\hat{\bm{p}}^{o}_{t}\|_{2}}\,e^{-\kappa_{R}\angle(\bm{R}^{o}_{t},\hat{\bm{R}}^{o}_{t})}\,e^{-\kappa_{v}\|\bm{v}^{o}_{t}-\hat{\bm{v}}^{o}_{t}\|_{2}}.\tag{3}$$

Three positive coefficients set sensitivity to position, rotation, and velocity error, implemented as (9, 2, 14). The product rewards joint agreement, and the velocity factor penalizes stationary solutions to moving references; the term is independent of the contact gate.

Contact supervision reads from the reference which fingertips must touch now: let C be that set, and for fingertip i let the distance to the object surface and the simulated net contact-force magnitude define a contact proxy through a double indicator:

$$\alpha_{i,t}^{h}=\bm{1}[f_{i,t}^{h}>f_{\min}]\ \bm{1}[\delta_{i,t}^{h}<\delta_{\max}],\tag{4}$$

with thresholds f_min of 0.5 N and delta_max of 0.03 m. The distance condition reduces false attribution of forces from other collision partners, but the paper concedes that net force does not identify the collision partner, so this remains a proxy. For a hand with at least one required fingertip, proximity and instantaneous coverage give:

$$r_{t}^{\mathrm{cont},h}=\frac{1}{|\mathcal{C}_{t}^{h}|}\sum_{i\in\mathcal{C}_{t}^{h}}\left[\zeta\, e^{-\kappa_{c}\delta_{i,t}^{h}}+(1-\zeta)\,\alpha_{i,t}^{h}\right],\qquad r_{t}^{\mathrm{hold},h}=\frac{1}{|\mathcal{C}_{t}^{h}|}\sum_{i\in\mathcal{C}_{t}^{h}}\alpha_{i,t}^{h}.\tag{5}$$

Kappa_c is 30 per metre and zeta is 0.5. Proximity pulls fingertips toward the object before contact; the hold term measures how much of the reference-required contact is covered right now, allowing valid sliding and rolling, so it is not a no-slip condition.

Contact rewards aggregate over the engaged-hand set (hands whose reference requires contact now) as a fixed-weight mean, and as zero when the set is empty:

$$r_{t}^{u}=\begin{cases}\dfrac{\sum_{h\in\mathcal{E}_{t}}w_{h}\,r_{t}^{u,h}}{\sum_{h\in\mathcal{E}_{t}}w_{h}},&\mathcal{E}_{t}\neq\varnothing,\\[4pt] 0,&\mathcal{E}_{t}=\varnothing,\end{cases}\tag{6}$$

Hand weights are fixed at 0.5 for left and 1 for right, with adaptive weighting disabled. Averaging rather than multiplying is the key choice: when one hand loses contact the other hand's feedback survives; excluding inactive hands avoids rewarding attachment during reference release.

Finally the terms combine with the inherited hand-tracking reward (Dexplore's per-hand multiplicative chain over wrist, finger-joint, and keypoint agreement, with its object-position and object-rotation subterms zeroed to avoid double counting with the object term):

$$r_{t}=(1+w_{L})\left[\beta_{\mathrm{hand}}\,r_{t}^{\mathrm{hand}}+\beta_{\mathrm{obj}}\,r_{t}^{\mathrm{obj}}+\beta_{\mathrm{cont}}\,r_{t}^{\mathrm{cont}}+\beta_{\mathrm{hold}}\,r_{t}^{\mathrm{hold}}\right],\tag{7}$$

Beta is (0.25, 0.45, 0.15, 0.15), summing to one; the (1+w_L) factor only sets reward scale. The object term carries the largest weight, consistent with the method's stance: object motion is the final judge.

4.4 Learning a generalist from verified demonstrations: CCA and PPO+BC

A shared PPO batch mixes clips with different contact patterns and coordination demands. GAE accumulates temporally weighted value residuals into advantages, but uneven critic accuracy across references leaves clip-dependent offsets: relatively successful transitions in hard clips can receive consistently negative advantages. CCA is deliberately plain, grouping transitions by reference identity and subtracting the group mean:

$$\widetilde{A}_{j}=A_{j}^{\mathrm{GAE}}-\frac{1}{|\mathcal{G}_{m(j)}|}\sum_{k\in\mathcal{G}_{m(j)}}A_{k}^{\mathrm{GAE}}.\tag{8}$$

Groups are defined by reference identity rather than environment instance or temporal phase, so parallel rollouts of the same reference contribute to a common baseline. Only the mean is subtracted, never a per-clip standard deviation, preserving the scale of within-clip advantage differences; the adjustment happens after GAE and leaves the critic, the temporal advantage estimate, and the reward untouched. The paper itself notes this is a mean shift whose algebraic identities do not establish an unbiased policy-gradient estimator.

The actor loss plugs the centered advantages into the standard clipped PPO objective and adds a state-paired regression on the demonstration library:

$$\mathcal{L}_{\mathrm{actor}}(\theta)=\mathcal{L}_{\mathrm{PPO}}(\theta;\widetilde{A})+\lambda_{\mathrm{BC}}\,\mathbb{E}_{(\bm{o}^{D},\bm{a}^{D})\sim\mathcal{D}}\left\|\mu_{\theta}(\bm{o}^{D})-\bm{a}^{D}\right\|_{2}^{2},\qquad \lambda_{\mathrm{BC}}=50.\tag{9}$$

Demonstrations use a separate sampling budget (4,096 pairs per minibatch) and never enter the PPO probability ratio or GAE; the critic keeps its standard value-regression objective. Admission to the library passes two gates: an exported rollout must succeed again on replay, and verification checks object displacement plus lifting when the reference requires it. Training runs Isaac Gym with 8,192 parallel environments, Adam at constant learning rate 2e-5, rollout horizon 64, PPO minibatch 16,384, six epochs per rollout, clipping radius 0.2, discount 0.99, GAE trace parameter 0.95, yielding 524,288 environment transitions per iteration, with checkpoints every 25 iterations.

4.5 Method flow overview

flowchart TD
  subgraph DATA["BiHOI and public data"]
    CAM["32 calibrated RGB cameras at 30 fps"] --> MASK["SAM3 masks plus Depth Anything 3 metric depth"]
    MASK --> OBJ["MV-SAM3D metric reconstruction plus FoundationPose 6-DoF tracking"]
    MASK --> HAND["WiLoR keypoints plus participant-specific MANO calibration and fitting"]
    OBJ --> JOINT["Depth-guided joint hand-object refinement with 10x SDF weight"]
    HAND --> JOINT
    JOINT --> RET["Retarget to Wuji hands plus open-loop replay screening A to F"]
    RET --> POOL["Training pool 1845 public clips plus 1575 BiHOI clips"]
  end
  subgraph LOOP["PPO plus BC feedback loop"]
    POOL --> ROLL["8192 parallel environment rollouts"]
    ROLL --> REW["Object-centered per-hand reward object plus contact plus hold"]
    REW --> GAE["GAE advantages"]
    GAE --> CCA["CCA subtract per-clip mean"]
    CCA --> PPO["Clipped PPO objective"]
    ROLL --> VER["Verified successes displacement plus lift plus replay"]
    VER --> LIB["Demonstration library D specialist plus generalist successes"]
    LIB --> BC["BC regression coefficient 50 with 4096 pairs per batch"]
    PPO --> ACTOR["Actor loss"]
    BC --> ACTOR
    ACTOR --> POL["Shared bimanual policy"]
    POL --> ROLL
  end

Figure 3: DONIC method flow. Top: multi-view recordings pass through masks and metric depth, the object and hand reconstruction branches, joint refinement, retargeting, and screening into the training pool. Bottom: rollouts produce rewards and GAE advantages, CCA centers them before PPO, verified successes enter the demonstration library for BC, and both terms form the actor loss updating the shared policy.

Experiments

5.1 Setup, metrics, and baselines

Training references are curated from GRAB, GigaHands, OakInk2, TACO, and H2O: clips where both hands and the manipulated object are tracked and each hand contacts the object at some point (simultaneity not required), harmonized, retargeted to the Wuji Hand V1, and replay-screened, yielding 1,845 public training clips; Ours+BiHOI adds all 1,575 BiHOI sequences for a pool of 3,420. Evaluation reserves two splits: unseen-traj with 53 unseen trajectories of training objects, and unseen-obj with 164 clips involving held-out objects. Neither split contributes to policy optimization, specialist training, or demonstration collection; all methods use the same splits, no fine-tuning, one checkpoint per method.

Metrics extend AdaDexTrack's trajectory-fidelity protocol to two hands: object translation and rotation error, wrist translation and rotation error, and finger-joint error averaged over 20 joints. Wrist errors use robot forward kinematics against retargeted references; reported hand errors take the worse of the two trajectory-averaged hand errors, then average across clips. Success requires both hands to independently pass three gates on trajectory-averaged errors:

$$\mathcal{K}(x^{o},x^{h})=\bm{1}\!\left[\frac{x_{p}^{o}+x_{p}^{h}}{2}\leq 10\ \mathrm{cm}\right]\bm{1}\!\left[\frac{x_{R}^{o}+x_{R}^{h}}{2}\leq\frac{\pi}{3}\right]\bm{1}\!\left[x_{f}^{h}\leq\frac{2\pi}{9}\right].\tag{D.6}$$

The angular gates pi/3 and 2pi/9 correspond to 60 and 40 degrees. SR uses trajectory-mean errors with a logical AND across hands; Comp applies the same tests per frame and averages, so later valid frames still count after an earlier violation, measuring the fraction of satisfying frames rather than the prefix before first failure. Baselines are DexTrack-bi and Dexplore-bi: released code with observation and action interfaces adapted to two Wuji hands, sharing DONIC's public training pool, Isaac Gym simulator, robot model, and held-out splits.

Parameter	Setting
Reward weights beta (hand, obj, cont, hold)	(0.25, 0.45, 0.15, 0.15)
Object sensitivities (position, rotation, velocity)	(9, 2, 14)
Contact proxy thresholds (force, distance)	(0.5 N, 0.03 m)
Contact distance sensitivity and proximity weight	(30 per metre, 0.5)
Fixed hand weights (left, right)	(0.5, 1)
Parallel environments / BC coefficient / demo pairs per batch	8,192 / 50 / 4,096
Recovery and object perturbation curricula	Disabled

Table 2: Reward and generalist-training settings (paper Table S2). Distances in reward exponents use metres, rotations radians, velocities metres per second.

5.2 Main results: one checkpoint, both splits

Split	Method	Obj trans (cm)	Obj rot (rad)	Wrist trans (cm)	Wrist rot (rad)	Finger (rad)	SR (%)	Comp (%)
Unseen trajectory	DexTrack-bi	41.86	2.3340	23.03	0.8296	0.6301	0.00	1.30
	Dexplore-bi	14.73	0.8597	21.88	0.8412	0.3166	9.43	27.87
	w/o CCA	11.27	0.9585	4.34	0.2980	0.0433	81.13	62.72
	w/o contact gate	10.26	0.9589	4.00	0.3013	0.0338	81.13	65.03
	DONIC	10.27	0.9277	3.96	0.2905	0.0372	86.79	67.62
	DONIC+BiHOI	9.74	0.8637	3.32	0.2818	0.0338	88.68	68.72
Unseen object	DexTrack-bi	127.87	2.3532	72.83	1.0629	0.6455	0.00	0.88
	Dexplore-bi	17.92	1.1431	19.46	0.9509	0.3685	11.59	26.37
	w/o CCA	16.83	1.1464	5.52	0.3235	0.0378	48.17	43.89
	w/o contact gate	17.61	1.2041	4.97	0.3386	0.0371	48.78	49.52
	DONIC	15.74	1.1310	4.58	0.3226	0.0373	57.32	54.69
	DONIC+BiHOI	13.96	1.1157	4.54	0.3206	0.0361	59.76	55.35

Table 3: Reference tracking without fine-tuning (paper Table 2). One checkpoint per method across both splits; lower is better for errors, higher for SR and Comp.

On unseen-traj DONIC reaches 86.79% SR and 67.62% Comp, exceeding the stronger baseline Dexplore-bi by 77.36 and 39.75 percentage points; on unseen-obj it reaches 57.32% and 54.69% against 11.59% and 26.37%. Hand tracking improves as well: wrist translation error falls from 21.88 to 3.96 cm on unseen trajectories and from 19.46 to 4.58 cm on unseen objects. Except for object rotation on unseen-traj, DONIC reduces every reported error relative to both baselines. Because SR and Comp require both hands to pass, these gains reflect coordinated bimanual execution rather than one accurate hand.

The baselines' failure modes are instructive: Dexplore-bi moves its hands but produces limited object motion, its reward and reference-scoped exploration failing to transmit force into the object in bimanual settings; DexTrack-bi loses hand-object alignment, its specialist-distillation interface mismatching two hands, with object translation error reaching 127.87 cm on unseen objects. In qualitative comparisons DONIC coordinates both hands and reproduces reference object motion, while the baselines either leave the object still or decouple hand from object.

Figure 4: Qualitative comparison on two held-out object trajectories (paper Fig. 3). Each group shows four rows, reference, Dexplore-bi, DexTrack-bi, and DONIC, with time running left to right; DONIC reproduces the reference hand-object coordination and object motion.

5.3 Ablations: both learning signals matter, BiHOI is a net gain

Removing CCA drops SR from 86.79% to 81.13% on unseen trajectories and from 57.32% to 48.17% on unseen objects, with lower Comp and all five tracking errors worse. Removing the contact-ownership gate (replacing the force-and-distance proxy by a force-only test above 0.5 N) drops SR to 81.13% and 48.78%; interestingly this variant shows lower individual errors in some cases but lower Comp on both splits, so reading errors alone would give the wrong conclusion when joint bimanual success is the target. Both learning signals are therefore supported by ablation.

Adding all 1,575 BiHOI sequences expands training from 1,845 to 3,420 clips and improves all seven metrics on both splits: SR reaches 88.68% and 59.76%, Comp 68.72% and 55.35%, and unseen-object translation error falls 11.3% from 15.74 to 13.96 cm, ranking first on every unseen-object metric. The training inventory and contact proportions below show why: within the public pool H2O has the highest bimanual contact (38.57%) and TACO the lowest (15.69%), while BiHOI's 65.91% is nearly double the pool average.

Source	Training sequences	Simultaneous bimanual contact (%)
GRAB	505	32.21
GigaHands	592	34.74
OakInk2	449	24.97
H2O	127	38.57
TACO	172	15.69
Public pool total	1,845	31.40 (pooled measure)
BiHOI	1,575	65.91
Public pool + BiHOI	3,420	-

Table 4: Training inventory and simultaneous bimanual contact (paper Table S1). Proportions divide both-hand-contact frames by frames with at least one hand in contact, pooled across analyzed recordings; over all frames the public pool would measure 22.56%.

Figure 5: Qualitative bimanual tracking comparisons on five sequences (paper Fig. S2). Within each group the four rows are reference, Dexplore-bi, DexTrack-bi, and DONIC; baselines show limited object motion or lost hand-object alignment.

Code correspondence

The released bihoi_pipeline (CC BY-NC 4.0) maps onto Appendix B.2 line by line, a rare case of formulas-as-code. Object metric scale lives in run_mv_sam3d_metric_scale.py as estimate_metric_scale: robust principal-component extents at the 1 and 99 percentiles (robust_point_extent with q=0.01), per-axis ratios, median, exactly Eq B.2:

mesh_extent, mesh_center_robust, _ = robust_point_extent(mesh_pts, q=0.01)
ratios = np.asarray(real_extent_m, dtype=np.float64) / np.asarray(
    mesh_extent, dtype=np.float64
)
# Sorted PCA extents are orientation-independent up to axis permutation.
# Median across the three dimensions is robust to one imperfect/occluded axis.
scale = float(np.median(ratios))

The export convention "tabletop z=0 from the fifth percentile of the object bottom" matches estimate_table_height_from_object in export_standardized_mano_object_table_4rings_first500.py (default percentile=5.0, exposed as --table_bottom_percentile), and MANO is exported with 45 axis-angle coordinates, no PCA, flat-hand mean, as Appendix B.2 states. The weights and voxel size of Eq B.5 appear verbatim as command-line arguments of the joint optimization script:

--joint_total_steps 160 \
--joint_sdf_steps 60 \
--w_sdf 10.0 \
--sdf_voxel_size 0.003 \
--sdf_expected_cam_count "$SDF_CAM_COUNT"

The 10.0 is the tenfold SDF weight of Eq B.5, 0.003 is the 3 mm voxel, and the 160 and 60 steps implement the staged optimization (priors and smoothness first, SDF alignment later). One bookkeeping difference is worth recording: the paper describes 32 capture cameras and 32 depth views in the interaction region for joint refinement, while this script's camera list and the naming in rename_videos_31cam.py default to 31 cameras, which --sdf_expected_cam_count also validates; the pipeline default excludes one camera from SDF construction. Read it as an implementation convention (capture 32, SDF default 31), not as a paper typo or a code bug. Training code, checkpoints, and BiHOI data are not released yet, so every policy-side number currently rests on the paper alone.

Limitations

The authors state (Appendix D.6): quantitative experiments evaluate reference-conditioned, single-rigid-object tracking in simulation; unconstrained goal-conditioned manipulation and held-out operation-category generalization are outside the evaluation; the capture inventory includes broader object types than the single-rigid-object representation (articulated and deformable objects are controlled as a fixed mesh with rigid pose); contact supervision is a net-force and proximity proxy; reconstruction has no explicit contact or collision loss.

Four further observations of our own. First, BiHOI appears in the experiments only as a training increment (the Ours+BiHOI row); there is no held-out BiHOI test split, so the dataset's own generalization value is never measured independently, and the 65.91% contact proportion is descriptive statistics, not causal evidence. Second, CCA is an empirical mean shift and the paper concedes the algebraic identities do not establish an unbiased gradient estimator; its group means are only stable at the scale of 8,192 parallel environments and 524,288 transitions per iteration, so at smaller compute budgets, with fewer samples per clip per batch, the benefit may shrink and the design's transferability is open. Third, all conclusions rest on Isaac Gym and the Wuji hand with no real-robot validation; the 0.5 N and 3 cm contact-proxy thresholds were tuned against simulated contacts, and their validity under real sensor noise and contact estimation error is unknown. Fourth, the both-hands SR gate is strict, but failures are not decomposed: for the 42.68% of unseen-object clips that fail, the paper reports no statistics on which hand or which phase (approach, grasp, lift, release) fails, leaving readers unable to localize the remaining error.

Conclusion and outlook

DONIC's contribution is not a new network but a decomposition of "bimanual" into three independently verifiable designs: data density of sustained two-hand contact (BiHOI), reward decoupling of object motion from per-hand contact supervision (object-centered per-hand reward), and optimization removing clip-difficulty offsets from advantages (CCA). Stacked together, one checkpoint substantially beats bimanual-adapted baselines on two held-out splits without fine-tuning, and ablations support each design individually. It also charts a pragmatic route: first turn human demonstrations into executable simulation references through multi-view reconstruction and retargeting, then let reinforcement learning turn references into real manipulation in physics.

Three things are worth watching next: an independent held-out BiHOI evaluation plus longer-horizon and multi-object extensions; transfer to physical bimanual platforms, especially recalibrating contact-proxy thresholds under real sensing; and CCA's behavior at small batch and few-environment scales, since if it is merely a large-batch dividend its generality discounts. For teams working on bimanual dexterous manipulation, the released reconstruction pipeline is usable today; the policy side waits on training code and data.

Quotable lines

"Hand-pose accuracy is not manipulation success: whether the object moves is the final judge of bimanual manipulation."

"Relatively successful actions in hard clips should not be uniformly punished because the clip is hard; subtract the clip's own mean and keep the differences inside it."

"When one hand loses contact, the other hand's feedback must survive; when the reference releases the object, attachment must not be rewarded."