GRAFT: Grounded and Efficient Online Reinforcement Adaptation for Fine-Grained Robot Manipulation

Observation and action interface

At time $t$, the policy receives

$$s_t=(I_t^s,I_t^w,l,q_t),$$

where $I_t^s$ and $I_t^w$ are side-view and wrist-view RGB observations, $l$ is the language instruction, and $q_t\in\mathbb{R}^8$ is adapter state. It predicts a chunk of ten low-level actions:

$$a_{t:t+H-1}\in\mathbb{R}^{7H},\qquad H=10.$$

Each action contains translation, rotation, and gripper commands. A frozen VLM prefix maps each view into visual tokens $X_t^v\in\mathbb{R}^{P_v\times d}$, a pooled context $c_t\in\mathbb{R}^{d}$, and key-value prefix states for $v\in\{s,w\}$. Training observations may also contain patch-aligned masks, but those masks are not inputs to the deployed policy.

The chunking choice trades reactivity against computation. Pipette Tip Attachment replans after every three executed actions, while the other tasks execute all ten actions before replanning. Frequent replanning can correct local alignment during a contact-rich task, but it increases the number of expensive inference calls. A longer open-loop block saves computation but requires the chunk to remain appropriate as contact conditions change.

View-specialized anchors

GRAFT keeps eight anchor templates for the side view and eight for the wrist view while sharing the context and cross-attention projections. The pooled context first modulates each template:

$$\widetilde e_i^v=e_i^v+\operatorname{NormScale}(W_c c_t).$$

An anchor then reads only patches from its assigned view. Raw attention and the resulting visual anchor are

$$A_{i,p}^v=\operatorname{softmax}_p\left(\frac{\langle\operatorname{norm}(W_q\widetilde e_i^v),\operatorname{norm}(W_kX_{t,p}^v)\rangle}{\tau_a}\right),$$

$$z_i^v=W_o\sum_{p=1}^{P_v}A_{i,p}^vW_vX_{t,p}^v+b_{\mathrm{view}}^v.$$

The anchors have no fixed proposal identity. An anchor is not permanently assigned to a Petri dish, a tube, or a gripper. Supervision asks the set of anchors to cover task-relevant regions in the current frame, but matching is reconstructed for every frame and view. This is useful when the same anchor set must support different objects and when the role of a region changes across task stages.

Global context is also used to reweight anchors within a view:

$$g_i^v=\operatorname{softmax}_{i\in v}\left(\frac{W_2\operatorname{swish}(W_1[z_i^v;c_t])}{T_g}\right),$$

$$\bar z_i^v=\left[(1-\alpha_g)+\alpha_gN_vg_i^v\right]z_i^v.$$

The base term prevents context gating from entirely suppressing an anchor, while the normalized gate can emphasize anchors whose content is relevant to the current observation and instruction. GRAFT uses $\tau_a=0.5$, $T_g=1.5$, $\alpha_g=0.1$, and $\beta_g=0.04$. The side and wrist branches are not assigned hand-coded image coordinates; their complementary behavior emerges from separate view inputs, separate anchor templates, and training supervision.

Multi-region FreeDice supervision

Each training frame may provide several valid proposal masks $\mathcal{M}_t^v=\{M_{t,k}^v\}_{k\in\mathcal{V}_v}$ for a view. GRAFT compares an anchor attention map and a proposal through a soft Dice cost:

$$C_{i,k}^v=1-\frac{2\langle A_i^v,M_k^v\rangle+\epsilon}{\max(\lVert A_i^v\rVert_1+\lVert M_k^v\rVert_1,\epsilon)}.$$

A direct one-to-one assignment would recreate persistent object identities and make anchors brittle when the scene changes. Instead, FreeDice uses a soft minimum to match regions in both directions. For a set of costs $c_j$,

$$\operatorname{smin}_{\tau}\{c_j\}=\sum_j[\operatorname{softmax}(-\mathbf c/\tau)]_jc_j.$$

The first term asks every valid proposal to be covered by at least one anchor. The reverse term asks every anchor to be close to at least one valid proposal:

$$\mathcal{L}_{\mathrm{FD}}^v=\frac{1}{|\mathcal{V}_v|}\sum_{k\in\mathcal{V}_v}\operatorname{smin}_{\tau_D}\{C_{i,k}^v\}_{i=1}^{N_v}+\frac{\lambda_{\mathrm{anc}}}{N_v}\sum_{i=1}^{N_v}\operatorname{smin}_{\tau_D}\{C_{i,k}^v\}_{k\in\mathcal{V}_v}.$$

This objective differs from supervising the union of all proposals. A union mask says that a collection of pixels is important but does not force individual anchors to specialize across the collection. FreeDice encourages proposal coverage while letting different anchors attend to different regions. It is especially relevant to centrifuge tube loading, where the policy must relate the tube, gripper, and loading target rather than focus only on the largest object.

To prevent anchor collapse, a redundancy term applies a margin to pairwise overlap after attention is renormalized over the valid-proposal union:

$$\mathcal{L}_{\mathrm{dup}}=\operatorname{mean}_{v,(i,j)\in\mathcal{P}_v}\left[\langle\widetilde A_i^v,\widetilde A_j^v\rangle-m_{\mathrm{dup}}\right]_+,$$

$$\mathcal{L}_{\mathrm{ground}}=\alpha(\mathcal{L}_{\mathrm{FD}}^s+\mathcal{L}_{\mathrm{FD}}^w)+\beta\mathcal{L}_{\mathrm{dup}}.$$

The paper fixes $\lambda_{\mathrm{anc}}=0.25$, $m_{\mathrm{dup}}=0.90$, $\alpha=0.4$, and $\beta=0.5$. Views without a valid proposal contribute zero grounding loss rather than treating an empty mask as evidence that nothing matters.

Single-step actions and prefix caching

The first efficiency mechanism replaces iterative flow-matching generation with a single-step consistency policy. Given an observation, the model samples noise and produces the whole chunk in one pass:

$$w\sim\mathcal{N}(0,\sigma^2I),\qquad a_{t:t+H-1}\sim\pi_\theta(\cdot\mid s_t).$$

The sampled action remains stochastic, preserving exploration, but the policy does not perform a sequence of denoising steps for every chunk. The second mechanism exploits the frozen VLM prefix. For replayed observations, GRAFT caches prefix KV states, pooled summaries, and visual tokens. During learner updates it reuses those states and recomputes only the trainable anchor, action, and critic pathways. Live observations still run a fresh prefix forward pass.

These optimizations target different serial bottlenecks. Single-step generation reduces the number of action-decoding stages. Prefix reuse avoids repeatedly encoding identical replay images. The throughput experiment shows that both matter, but that caching contributes the larger multiplier in the tested learner configuration. This is an important systems observation: when online learning is constrained by wall-clock time, optimizing robot inference alone may leave a substantial cost inside replay optimization.

Asynchronous actor–learner optimization

The actor uploads interaction windows and human corrections at episode boundaries. Once online replay contains 100 valid windows, the learner samples equal batches from online and demonstration data. Intervention windows are also added to the demonstration buffer, so a correction can improve future behavior cloning rather than only rescuing the current episode.

A transition is written as $(s,a,r,s',m)$, where $a\in\mathbb{R}^{7H}$ is the executed window, $s'=s_{t+H}$, $r$ is return, and $m$ is a bootstrap mask. With twin critics, the next action and Bellman target are

$$a'\sim\pi_\theta(\cdot\mid s'),\qquad y=r+\gamma m\min_iQ_{\bar\phi_i}(s',a'),$$

with $\gamma=0.98$. Each critic minimizes

$$\mathcal{L}_{\mathrm{critic}}=\mathbb{E}_{(s,a,r,s',m)\sim\mathcal{D}}\left[(Q_\phi(s,a)-y)^2\right].$$

The actor balances imitation and value maximization:

$$\mathcal{L}_{\mathrm{actor}}=\lambda_{\mathrm{bc}}\mathcal{L}_{\mathrm{BC}}+\lambda_q\mathcal{L}_Q,$$

$$\mathcal{L}_{\mathrm{BC}}=\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[\lVert a-\pi_\theta(s)\rVert_2^2\right],\qquad\mathcal{L}_Q=-\mathbb{E}_{s\sim\mathcal{D}}\left[Q_\phi(s,\pi_\theta(s))\right].$$

The reported weights are $\lambda_{\mathrm{bc}}=0.5$ and $\lambda_q=1$. Each iteration performs two critic updates and one actor update. Every ten iterations, proposal-valid demonstration windows add an actor-only grounding update. This schedule keeps value learning frequent and stable, while grounding supervision is periodic and depends on the availability of valid masks.

flowchart TD
    A[Side RGB wrist RGB language and adapter state] --> B[Frozen VLM prefix]
    B --> C[Eight side-view anchors]
    B --> D[Eight wrist-view anchors]
    C --> E[Context gating and anchor fusion]
    D --> E
    E --> F[Single-step consistency head]
    F --> G[Ten-action chunk executed for 3 or 10 steps]
    G --> H[Robot interaction and human correction]
    H --> I[Online replay and demonstration buffers]
    I --> J[Reuse frozen-prefix KV cache]
    J --> K[Twin-critic Bellman update]
    K --> L[Behavior cloning plus Q-maximization]
    L --> M{Grounding period with valid proposals}
    M -->|Yes| N[Training-only FreeDice update]
    M -->|No| O[Publish latest policy]
    N --> O
    O --> G
    P[Offline masks and external proposal source] --> N
    P -. Training only .-> Q[Removed at deployment]
  

Four real-robot tasks

The experiments use a JAKA arm on four biomedical manipulation tasks: Petri Dish De-lidding, Centrifuge Tube Loading, Precision Liquid Transfer, and Pipette Tip Attachment. Each task has ten teleoperated demonstrations. Every method receives the same 45-minute online adaptation budget. Episodes start from a fixed robot configuration, while objects are randomly perturbed within $\pm30$ millimeters of their nominal planar positions. Task-specific criteria define success, and intervened episodes are not counted as autonomous successes.

The comparison isolates two design choices. RL-FM and RL-CP use the same protocol but differ in multi-step flow matching versus a single-step consistency head. GRAFT-Union and GRAFT both use view-specialized anchors, context reweighting, redundancy regularization, and the same remaining architecture, but GRAFT-Union supervises a proposal union while GRAFT uses identity-free multi-region FreeDice. The second pair therefore provides the cleaner test of the paper’s central grounding objective.

Failures shown in Figure 4 are concrete and informative: release ahead, low height, collision, insertion error, spray early, and tip drop. These modes are not interchangeable. A policy that reduces collisions may still release at the wrong moment, and a policy that improves insertion can still lose the pipette tip. The task suite therefore probes more than one kind of local geometry, although all tasks remain within a related laboratory tabletop setting.

Final online success

Table I reports rolling success over the final ten available episodes of each online trace. It is not a separate evaluation of a frozen policy on an unseen test set. The numbers answer whether one method finishes the matched 45-minute adaptation run with stronger recent behavior; they do not establish unbiased generalization to new scenes.

Original Table I: rolling success in the final ten episodes of each available online trace.

Policy	Petri Dish	Centrifuge Tube	Precision Liquid	Pipette Tip	Overall
RL-FM	2/10	0/10	0/10	2/10	4/40, 10.0%
RL-CP	3/10	3/10	9/10	5/10	20/40, 50.0%
GRAFT-Union	7/10	2/10	8/10	6/10	23/40, 57.5%
GRAFT	9/10	9/10	8/10	7/10	33/40, 82.5%

GRAFT reaches 82.5% overall, 25 percentage points above GRAFT-Union and 32.5 points above RL-CP under the matched budget. The largest change is Centrifuge Tube Loading, from 2/10 for GRAFT-Union to 9/10 for GRAFT. This task requires simultaneous localization of the tube, gripper, and loading target, so the gain is consistent with the intended benefit of supervising several local regions without a fixed identity assignment.

The result is not a uniform win. Precision Liquid Transfer is the clear counterexample: RL-CP reaches 9/10, whereas GRAFT reaches 8/10. GRAFT therefore improves the overall endpoint and prevents a severe failure on centrifuge loading, but it does not converge faster on every task. An honest reading distinguishes aggregate robustness from per-task superiority.

Figure 5 adds trajectory evidence that the endpoint table hides. Petri Dish De-lidding drops during adaptation before recovering to 90%, Centrifuge Tube Loading separates from alternatives late in training, and Pipette Tip Attachment improves at the end. Precision Liquid Transfer shows the opposite ordering, with RL-CP achieving high success earlier. The paper’s conclusion is best framed as stronger final performance under a fixed wall-clock interaction budget rather than uniformly better online dynamics.

Learner throughput and prefix reuse

The efficiency study uses Centrifuge Tube Loading, ten demonstrations, 946 replay chunks, and 1,000 learner steps for each configuration. It varies the number of action-generation steps and whether frozen-prefix KV states are cached. These measurements isolate learner-update cost rather than full robot control latency.

Original Table II: offline learner-update throughput on Centrifuge Tube Loading.

Policy formulation	Prefix KV cache	Total learner seconds	Steady steps/s	Steady seconds/step
RL-FM, K=10	Off	597.989	1.990	0.503
RL-FM, K=10	On	236.405	6.300	0.159
RL-CP, K=1	Off	542.047	2.210	0.452
RL-CP, K=1	On	119.216	21.960	0.0455

Without caching, RL-FM and RL-CP run at 1.99 and 2.21 learner steps per second. Prefix reuse raises these rates to 6.30 and 21.96, speedups of 3.17 and 9.94 times respectively. With caching enabled, the consistency policy is another 3.49 times faster than the flow-matching policy. The result supports the claim that action generation and replay encoding create complementary costs.

The table does not measure end-to-end robot throughput. It excludes live actor inference, network transfer, robot motion, human intervention, and policy publication. A cached learner may use more memory than an uncached one, and the best batch configuration can vary with hardware. The relative speedup is therefore strong evidence for the mechanism, but not a complete deployment profile.

What the attention visualizations show

Figure 6 compares spatial attention on Pipette Tip Attachment, Centrifuge Tube Loading, and Precision Liquid Transfer. RL-FM and RL-CP visualize final-layer action-token-to-patch attention. GRAFT-Union and GRAFT visualize raw anchor-to-patch attention. These are different quantities, so comparisons across the two families remain qualitative. The GRAFT-Union versus GRAFT comparison is more direct because the visualized quantity is the same and only the grounding objective changes.

In that controlled pair, GRAFT produces more concentrated responses around pipette tips, tubes, and liquid-transfer targets. Side-view attention tends toward global approach and object alignment, while wrist-view attention concentrates on local contact and insertion. This is consistent with the architecture, but it is not a quantitative localization score. The paper does not report intersection-over-union against task regions, multi-seed attention averages, or an ablation in which attention is deliberately perturbed while all other factors remain fixed.

Limitations and residual uncertainty

The author-stated limitation is that GRAFT is reactive. It does not explicitly track previous actions or task progress, so it may struggle when the relevant target changes over a longer horizon or when recovery depends on remembering an earlier stage. Temporal context is a plausible extension, but the paper does not test it.

The empirical scope is narrow: four related biomedical tabletop tasks, one JAKA arm, ten demonstrations per task, and one real-world protocol. Ten final episodes mean that one success changes a per-task result by ten percentage points. The study is large enough to demonstrate a substantial aggregate difference, but not broad enough to establish robustness across embodiments, camera layouts, reflective transparent objects, or longer task sequences.

Training still depends on offline proposals. Deployment is proposal-free, but the method does not eliminate external region supervision from the learning pipeline. The paper does not quantify how missed detections, inaccurate boundaries, or missing regions affect the learned anchors. A complete robustness study would degrade the proposal source and measure success, attention concentration, and final policy variance.

The main table is an online rolling result rather than a frozen-policy test. This is appropriate for comparing methods under a wall-clock interaction budget, but it leaves open the question of generalization after training. A separate test should freeze the final policy and evaluate repeated trials on held-out object positions, lighting conditions, and task configurations.

The attention evidence remains qualitative. The authors explicitly note that cross-family heatmaps visualize different quantities, which is an important caveat. Even the controlled GRAFT-Union comparison would benefit from region-level metrics, multiple seeds, and failure attribution that connects a missed anchor to a specific collision, release, or insertion failure.

Finally, no public implementation was found. The exact VLM version, proposal generation procedure, cache implementation, actor–learner communication, reward definitions, and task-specific success checks cannot be audited from code. The paper supplies enough detail to understand the design, but not enough to guarantee independent reproduction.

What the work changes

GRAFT’s main idea is to move visual grounding into training rather than leaving a detector in the deployment loop. Multiple proposal regions shape identity-free anchors through bidirectional soft matching. At deployment, only the learned visual pathway remains. Compared with a permanent object-detection pipeline, this reduces inference dependencies. Compared with learning solely from task rewards, it injects information about which local regions matter without requiring a human to annotate a rigid object-to-anchor assignment.

The learner-side contribution is equally practical. A frozen visual-language prefix need not be recomputed for every replay sample, and a single-step consistency head removes iterative action generation. The reported 9.94-times cache speedup for the consistency policy and 3.49-times advantage over the cached flow-matching version show that system-level reuse can materially change the amount of learning completed within a real-robot time budget.

The defensible empirical claim is bounded but useful: across four biomedical manipulation tasks, one robot, ten demonstrations per task, and a matched 45-minute online budget, FreeDice grounding improves final rolling success over proposal-union supervision by 25 percentage points, while the efficiency mechanisms substantially increase learner throughput. The unresolved questions are equally clear: one task still favors a simpler policy, final online episodes are not an independent generalization test, proposal quality is not ablated, and the code is unavailable.

Conclusion

GRAFT combines grounded visual adaptation, online reinforcement learning, and computational reuse in a coherent real-robot system. Side and wrist anchors specialize without permanent proposal identities, FreeDice supervises multiple local regions only during training, and deployment removes the proposal branch. A single-step action head and frozen-prefix KV cache reduce the cost of acting and replay updates. On four fine-grained tasks, GRAFT reaches 33 successes in the final 40 online episodes, including a jump from 2/10 to 9/10 on centrifuge tube loading.

The next evidence should come from a frozen-policy test on held-out scenes, proposal-quality ablations, temporal context for changing targets, and replication on another robot or camera configuration. Those experiments would determine whether the main gain comes from a generally useful grounding mechanism or from a favorable interaction with this particular biomedical setup.

Key takeaway: GRAFT teaches a small set of anchors where to look during training, then leaves the proposal generator behind when the robot starts acting.