PAPER DEEP DIVE
T-WLA: Bridging Slow Vision and Fast Touch
T-WLA shares a history-aware visual memory between a slow vision-language-action policy and a fast tactile residual editor. The visual branch produces an action chunk and memory at 39.7 Hz; the lightweight tactile editor reuses them to correct only unexecuted actions at 119.1 Hz. It improves tactile-conditioned hand-motion prediction and contact-rich manipulation, reaching 50.0% average success on four real-robot tasks with 30 demonstrations each.
Paper Metadata
Title: T-WLA: Bridging Slow Vision and Fast Touch
Authors: Kaichen Zhou, Yuzhen Chen, Hannah Yao, Chonghe Jiang, Rao Fu, Xihang Yu, Wei Dai, Bashar Kabbarah, Haoyu Chen, Ishita Goluguri, Zhiyang Dou, Haonan Chen, Fangneng Zhan, Mengyu Wang, Wojciech Matusik, Antonio Torralba, Paul Pu Liang (MIT, Harvard University, UC Davis, UC Berkeley; the first three authors contributed equally)
Links: arXiv:2610.28814v1 [cs.RO], 2026-10-20; full text https://arxiv.org/abs/2610.28814 ; project page https://anonymous-submission-20.github.io/T-WLA.github.io/
Code status: the abstract says that "Videos and codes are available", and the project page does provide an overview video, four real-robot demonstrations, and a hand-pose example. As of this read, however, the project page has no repository link, and a GitHub search by title and method name returned no public implementation. The method and training recipe are therefore documented, but code and weights are not yet available. This page separates the paper's claim from the current state of the project page.
One-Sentence Summary
T-WLA treats the visual policy as a low-rate planner that produces an action chunk plus a reusable history-aware visual memory at 39.7 Hz, then lets a lightweight 119.1 Hz tactile editor rewrite only the unexecuted actions between visual updates; this raises average success on four real contact-rich tasks from 17.0% to 50.0% and cuts finger-joint MPJPE by about 30%.
Fig. 1: T-WLA overview (paper Fig. 1). (a) Human demonstrations with and without touch; (b) future hand-pose prediction; (c) real-robot tactile manipulation, where touch corrects grasp and lift while the tactile-free version lets the egg slip.
Background and Motivation
Contact-rich manipulation has a built-in timescale mismatch. A visual model sees a frame that is already tens of milliseconds old, while contact can change within a few milliseconds. When a plug first touches a socket, a paper cup starts to slip, or a thin sheet is pinched, the camera may not resolve the event; touch already reports the change in pressure and contact location.
Most vision-language-action models run on a single visual clock. Every new image triggers scene encoding, instruction understanding, and generation of a whole action sequence. If contact changes before the next frame, the policy must either wait for the next visual inference or hold the old plan open-loop. Rerunning the visual backbone adds latency, while touch alone lacks task semantics and scene context: it knows that contact happened, but not which step the robot is executing or where the motion should be corrected.
Action chunking reduces the decision rate by predicting several actions at once, but it also locks responsiveness inside the chunk. The actions were generated from an older observation; if contact changes, the remaining actions must be revised without regenerating the entire plan. Real-time chunking, bidirectional decoding, and streaming diffusion accelerate or reselect chunks, but they do not directly answer a finer question: how can touch rewrite only the unexecuted portion of the current chunk while the visual branch stays frozen?
Tactile foundation models and multi-rate control provide the other half. T-Rex assigns vision and touch to experts running at different rates; ViTaR attaches a tactile residual branch to a frozen policy; Reactive Diffusion Policy pairs a slow visual diffusion policy with a fast latent tactile controller. T-WLA differs in the object that is shared: the visual branch outputs not only actions but also a history-aware visual memory, and that same memory becomes the interface through which the tactile editor corrects the plan.
The central question can therefore be stated as follows. When vision is slow and touch is fast, can touch act as a local residual correction on an existing plan, preserving task semantics while gaining millisecond-level responsiveness? T-WLA answers yes, but with an explicit boundary condition: the base action chunk must remain a useful reference between visual updates.
Preliminaries
Action chunking. The policy predicts H actions at once and executes several before replanning. This lowers the cost of running a large model but creates the problem of actions becoming stale inside the chunk.
Conditional flow matching. The visual action expert and the predictive heads learn a velocity field that transports noise toward actions or future latents, conditioned on observations and instructions.
FiLM modulation. Feature-wise Linear Modulation uses a condition vector to produce per-channel scale and shift, injecting one modality into another modality's features.
Multi-rate control. The visual branch and tactile branch run at different frequencies: vision handles semantics and long-horizon plans, while touch handles local contact and short-horizon corrections. The hard part is not having two clocks; it is sharing a representation without letting the slow branch throttle the fast one.
Method
3.1 System Overview: One Memory, Two Timelines
Let t index visual updates and tau index tactile updates between them. The visual branch receives the current RGB image It, task instruction, field-of-view information, and robot state, and produces a history-aware visual memory MVt. An action expert then generates an H-step base action chunk:
$$A_{t}^{\mathrm{base}}=f_{\mathrm{act}}\left(M_{\mathrm{V}}^{t}\right)\tag{1}$$
The tactile branch receives a higher-frequency pressure stream Tt,tau. Between visual updates, MVt and Atbase are cached, and the tactile editor computes only a residual that is added to the base chunk:
$$\widehat{A}_{t,\tau}=A_{t}^{\mathrm{base}}+\Delta A_{t,\tau}\tag{2}$$
Delta A is not a newly generated plan. It is applied only to valid, unexecuted action positions. Every tactile update is computed relative to the same original base chunk, not accumulated on top of the previous correction. This prevents correction errors from compounding across tactile updates.
Fig. 1: T-WLA dual-rate architecture (paper Fig. 2). The visual branch produces a base action chunk and visual memory at RGB updates; the tactile branch reuses the cached memory and edits only the remaining actions at tactile updates.
3.2 History-Aware Visual Memory
The current image may hide information that was visible earlier, and it may leave the recent approach to contact ambiguous. T-WLA does not re-encode the whole image sequence. Instead, it keeps a compact memory bank of the last K visual output-memory sets, each containing S memory slots. At a visual update, learned query tokens QV retrieve input memory from the bank BVt-1:
$$M_{\mathrm{V},\mathrm{in}}^{t}=\mathcal{M}_{\mathrm{V}}\left(\mathcal{B}_{\mathrm{V}}^{t-1},Q_{\mathrm{V}}\right)\tag{3}$$
The retrieved history is combined with current image tokens Et, instruction tokens Lell, field-of-view tokens ct, and state tokens St to produce the current visual memory:
$$M_{\mathrm{V}}^{t}=f_{\mathrm{V}}\left(E_{t},L_{\ell},c_{t},S_{t},M_{\mathrm{V},\mathrm{in}}^{t}\right)\tag{4}$$
The bank then appends the new memory and keeps only the K most recent sets. The implementation uses S=8, d=960, and K=8. Retrieval uses two pre-normalized cross-attention blocks with 15 heads and a feed-forward width of 2560, plus slot embeddings and age embeddings that tell the model which memory slot it is reading and how many visual updates ago that memory was written. Age is measured in visual updates, not tactile events, so the high tactile rate cannot distort the visual history's time scale.
Fig. 2: Memory-to-editor interface (paper Fig. 3). (a) Learned queries retrieve history and fuse it with the current image and instruction. (b) Tactile history is encoded relative to a fixed reference. (c) Tactile context modulates action features through FiLM, and a residual head outputs the correction.
3.3 Tactile Context: Encode Change, Not Absolute Pressure
Each hand uses the most recent 8 frames of 21x21 pressure maps. Invalid time steps are zeroed, and the left-hand maps are flipped along the horizontal axis so both hands share a spatial convention. Pressure and the sensor-validity mask form two input channels. A shared CNN uses three 5x5 convolutions with 32 channels, padding 2, ReLU, and 2x2 max pooling, producing 128-dimensional frame features. These enter a two-layer bidirectional GRU with hidden width 120 per direction; the final forward and initial backward states are concatenated to 240 dimensions, projected to 64, and L2-normalized.
The key operation is reference subtraction. The same encoder processes the current tactile history Ht,taur and a fixed hand-specific reference history Hrefr, and the reference-centered tactile context is their difference:
$$Z_{t,\tau}^{r}=v_{t,\tau}^{r}\left[F_{\mathrm{T}}\left(\mathcal{H}_{t,\tau}^{r}\right)-F_{\mathrm{T}}\left(\mathcal{H}_{\mathrm{ref}}^{r}\right)\right]\tag{5}$$
Here v is a validity gate that is one only when the encoded history passes its check and the latest tactile frame is valid; otherwise the whole hand representation is zero. The reference is not a cached embedding: it is a raw pressure window encoded on every forward pass, so the reference path also receives gradients. The difference is not normalized again, preserving the magnitude of tactile change.
Why a temporal window? A single pressure map can say where contact is, but not how contact is evolving. Eight frames expose slipping, pressure buildup, and the speed of contact formation. The bidirectional GRU operates only inside the observed history window and never reads future tactile frames, so the representation remains causal.
3.4 Residual Editor: FiLM on Action Features
The editor projects visual memory and base-action tokens to width 256 and adds modality, action-position, and elapsed-time embeddings. The time input is 10 times Delta t in seconds, so the model can distinguish the second action of the chunk from the tenth and know which actions are already executed. The editor is a two-layer pre-normalized Transformer with 8 heads, feed-forward width 1024, GELU, and zero dropout. Only the action-position outputs Hact enter the residual readout.
For each hand, the 64-dimensional tactile context passes through a bias-free linear layer that produces FiLM scale and shift. The modulated action features are mapped back to action space by a residual head:
$$U^{r}=W_{\mathrm{out}}^{r}\left[\gamma^{r}\left(Z^{r}\right)\odot H_{\mathrm{act}}+\beta^{r}\left(Z^{r}\right)\right]\tag{6}$$
$$\Delta A_{t,\tau}=\sum_{r\in\{\mathrm{L},\mathrm{R}\}}m_{\mathrm{edit}}^{r}\odot U^{r}\tag{7}$$
The mask meditr intersects editable action positions, that hand's action channels, and hand validity. A hand without tactile input receives zero correction, and executed actions are never rewritten. The output projection is zero-initialized, so training starts with Delta A = 0: the model begins by fully trusting the base action chunk and gradually learns where and how much to correct.
This structure confines touch to local action editing rather than replanning. Visual memory supplies task and scene context, the base chunk says what is being executed, elapsed time identifies the remaining actions, and touch answers a narrower question: on this plan, how should the next few actions be shifted?
3.5 Causal Contact Gate and Action Masks
The paper also describes an optional contact-filtered variant that enables tactile correction only when the observed pressure history actually contains contact. For each hand r, it averages the five largest valid pressure values in each frame and checks whether any history frame crosses the threshold eta = 0.25:
$$c^{r}=\mathbf{1}\left[\max_{j\in\mathcal{J}_{r}}\frac{1}{5}\sum_{u\in\operatorname{Top5}\left(P_{j,r}|_{\mathcal{S}_{r}}\right)}P_{j,r,u}\geq\eta\right]\tag{8}$$
The gate uses only observed pressure, never a future contact label. A gated-out hand keeps its base action, but its entries are excluded from the zero-error denominator, so "no contact" is not treated as "correct correction." The same gate is intersected with the action-supervision mask and the future-pressure-validity mask.
3.6 Predictive Supervision: Future Image and Future Pressure Are Training-Only
Action supervision alone may teach a representation that reproduces demonstrations without understanding contact consequences. T-WLA therefore adds two auxiliary objectives: future-image prediction and future-pressure prediction. The future-image target is a latent produced by a frozen image encoder. Tactile context passes through a learned adapter that modifies only the prediction condition, not the cached visual memory used for control.
Future-pressure prediction is conditioned on visual memory, tactile context, and a window of upcoming edited actions. The action window is detached so the pressure loss cannot backpropagate through the action readout, while gradients through tactile context still update the shared tactile encoder. The target covers 441 pressure sites and is standardized with training-set per-site mean and standard deviation:
$$y_{u}^{r}=\frac{P_{u}^{+,r}-\mu_{u}^{r}}{\max\left(\sigma_{u}^{r},10^{-3}\right)}\tag{9}$$
Both prediction heads are removed at inference, so they add no deployment latency. Their role is not "imagine the future and then plan"; it is to force visual memory and tactile features to contain interaction dynamics that matter for the next contact.
3.7 Staged Training and Losses
Training has three stages. First, the visual action expert is trained with conditional flow matching, using action loss and future-image loss. Second, the tactile encoder is pretrained on synchronized tactile and hand-pose histories from EgoTouch with a symmetric contrastive objective: similarity is Sij = alpha uiT pj, synchronized pairs are positives, and other pairs in the minibatch are negatives. Third, the visual branch is frozen; its visual memory and base action chunks are cached, and the tactile encoder, residual editor, tactile-conditioning adapter, and pressure predictor are trained.
The visual action loss is group-balanced mean squared error: compute a mean squared error within each action-component group, then equally average groups that have valid samples. This prevents palm-local joints, wrist translation, and wrist rotation from being dominated by groups with different channel counts. The editor loss first averages Smooth L1 within each nonempty group of each sample, then averages supervised samples equally:
$$\mathcal{L}_{\mathrm{edit}}=\frac{1}{|\mathcal{B}_{+}|}\sum_{b\in\mathcal{B}_{+}}\frac{1}{|\mathcal{G}_{b}|}\sum_{g\in\mathcal{G}_{b}}\frac{\sum_{j}\sum_{d\in\mathcal{D}_{g}}m_{b,j,d}^{\mathcal{E}}\rho\left(e_{b,j,d}\right)}{n_{b,g}}\tag{10}$$
Here e is the corrected-action error, rho is Smooth L1, and m is the effective editable mask. Samples with no valid group are excluded from the denominator. A residual-magnitude penalty discourages the editor from hiding problems in the base chunk behind large corrections:
$$\widehat{\mathcal{R}}_{\Delta}=\frac{\sum_{b,j,d}m_{b,j,d}^{\mathcal{E}}\left(\Delta A_{b,j,d}\right)^{2}}{\sum_{b,j,d}m_{b,j,d}^{\mathcal{E}}}\tag{11}$$
The editor objective can be summarized as accurate correction, small correction, and informative future-image/future-pressure representations. Freezing the visual branch prevents tactile training from erasing the visual prior; the cost is that the two branches are not jointly optimized. The authors list joint fine-tuning as a future extension.
3.8 Action Layout and Temporal Alignment
The hand-action chunk has H=16 positions and 192 channels per position. Of these, 138 channels contain motion and 54 are padding. Each hand has 69 motion channels: wrist translation increment (3), wrist Euler rotation increment (3), and palm-local joint coordinates (63). The visual branch runs at 39.7 Hz and the tactile editor at 119.1 Hz. Elapsed time is measured from the start of the active chunk rather than inferred from the tactile update index, so corrections align to the right action position even when the two clocks differ.
Future-pressure conditioning uses the next eight edited actions. The window starts at k0 = ceil(fcmd Delta t), where fcmd is the action command rate. Positions beyond the chunk are zero-padded and marked invalid. This window is auxiliary supervision only; it does not change how the controller executes actions.
3.9 Pipeline Overview
flowchart TD
subgraph VIS["Visual branch 39.7 Hz"]
RGB["Current RGB image"] --> VE["Vision encoder"]
LANG["Task instruction"] --> TX["Text tokenizer"]
STATE["Robot state and FoV"] --> VLM
VE --> VLM["Vision-language model"]
BANK["Visual context bank last K sets"] --> RET["Learned-query retrieval 8 slots 960 dim"]
RET --> VLM
VLM --> VM["Visual memory M_V"]
VM --> AE["Action expert conditional flow matching"]
AE --> BASE["Base action chunk H=16"]
VM --> BANK
end
subgraph TAC["Tactile branch 119.1 Hz"]
TOUCH["Past 8 frames 21x21 pressure maps both hands"] --> ENC["Shared CNN plus BiGRU"]
REF["Fixed reference pressure window"] --> ENC
ENC --> Z["Reference-subtracted tactile context Z"]
BASE --> EDIT
VM --> EDIT["Residual editor Transformer plus FiLM"]
Z --> EDIT
TIME["Elapsed time and action position"] --> EDIT
EDIT --> DA["Action residual only on unexecuted positions"]
DA --> FINAL["Corrected remaining action chunk"]
BASE --> FINAL
end
VIS -. "Training-only future-image prediction" .-> AUX["Auxiliary prediction heads removed at inference"]
TAC -. "Training-only future-pressure prediction" .-> AUX
Fig. 3: T-WLA pipeline. The visual branch produces and reuses visual memory and a base action chunk; between visual updates, the tactile branch uses tactile context, cached memory, the base chunk, and elapsed time to edit only the remaining actions.
Experiments
4.1 Tactile-Conditioned Hand-Motion Prediction
The paper first evaluates future hand-motion prediction on EgoTouch and Tachin: given vision and touch at time t, predict the hand pose 100 ms later. EgoTouch is reported over its five native environments and as a macro-average; Tachin uses its complete held-out test set. All methods share the same prediction horizon and the same tactile encoder, prediction head, and optimization budget; T-Rex and T-WLA keep their native multi-rate interfaces.
T-WLA achieves the lowest error on all three metrics, in every EgoTouch environment, and on Tachin. The clearest gain is finger articulation: wrist-relative MPJPE falls by about 30.0% on EgoTouch and 27.0% on Tachin relative to each dataset's strongest baseline. Wrist translation and rotation also improve, but more modestly. This matches the intended division of labor: the tactile editor refines finger and contact detail rather than replacing the visual prediction of the global hand trajectory.
| Method | EgoTouch trans. (cm) | EgoTouch rot. (deg) | EgoTouch finger MPJPE (cm) | Tachin trans. (cm) | Tachin rot. (deg) | Tachin finger MPJPE (cm) | Tactile rate (Hz) |
|---|---|---|---|---|---|---|---|
| VITRA + tactile | 4.08 | 6.27 | 0.50 | 1.48 | 4.55 | 0.63 | 58.2 |
| SmolVLA + tactile | 4.10 | 6.28 | 0.50 | 1.50 | 4.20 | 1.28 | 69.0 |
| MemoryVLA + tactile | 4.18 | 6.34 | 1.34 | 1.55 | 4.39 | 1.87 | 15.0 |
| T-Rex | 4.16 | 6.29 | 1.14 | 1.54 | 4.25 | 1.08 | 17.16 |
| T-WLA | 3.96 | 6.10 | 0.35 | 1.39 | 4.17 | 0.46 | 119.1 |
Table 1: Tactile-conditioned future hand-motion prediction (paper Table 1). EgoTouch is a five-environment macro-average; Tachin is the complete held-out test set. Lower is better.
Crucially, T-WLA does not buy accuracy by slowing down touch. Its effective tactile-correction rate is 119.1 Hz, while the fastest reported baseline reaches 69.0 Hz. It combines more frequent correction with lower error, which is the intended advantage of a lightweight residual path over rerunning the visual model.
Fig. 4: Representative real-robot trials for cup stacking and egg transfer (paper Fig. 5). Each column shows start, contact, and goal; the top row is a successful T-WLA trial and the bottom row is a failed baseline trial.
4.2 Real-Robot Contact-Rich Manipulation
The robot platform is a dual-arm Linker A7black, with a 7-DoF arm and a Linker Hand L6 dexterous hand on each side. The fingertips provide tactile sensing, and an Intel RealSense D435i on a custom bracket supplies an egocentric view. Demonstrations are collected with Wuji data gloves for finger articulation and HTC VIVE Tracker 3.0 devices at the wrists, tracked by two SteamVR Base Station 2.0 units; no VR headset is required. Each task has only 30 teleoperated episodes.
The four tasks are egg transfer, single-sheet pickup, paper-cup nesting, and paper-bag dragging. Egg transfer requires detecting small in-hand slip that is hard to see. Single-sheet pickup requires a pinch strong enough to prevent slipping but light enough not to crumple the sheet. Cup nesting requires a secure grasp without crushing the cup, plus correct alignment and placement. Bag dragging requires holding the loaded bag's rim as fingertip loads change. Together they test whether touch can supply what vision misses: grasp stability, force regulation, and the need to keep pushing or back off.
| Controller | Egg transfer | Single-sheet pickup | Paper-cup nesting | Paper-bag dragging | Average |
|---|---|---|---|---|---|
| SmolVLA | 0 | 28 | 0 | 39 | 16.75 |
| SmolVLA + tactile | 0 | 30 | 1 | 37 | 17.00 |
| T-Rex | 4 | 3 | 22 | 63 | 23.00 |
| T-WLA without tactile | 18 | 72 | 17 | 82 | 47.25 |
| T-WLA | 21 | 78 | 17 | 84 | 50.00 |
Table 2: Success rates on four real contact-rich tasks (%, paper Table 2), with 30 demonstrations per task. Average is unweighted across tasks.
Several numbers deserve separation. First, the visual-only T-WLA already reaches 47.25%, far above SmolVLA's 16.75%, so history-aware visual memory and the action expert are strong on their own; the tactile gain of 2.75 points is an increment on a strong baseline, not a rescue of a weak policy. Second, the largest tactile gain is single-sheet pickup, from 72 to 78, consistent with the sensitivity of a thin sheet to pinch force. Third, cup nesting stays at 17%, showing that a secure grasp is only part of the task and cannot fix alignment or placement failures. Fourth, VITRA and MemoryVLA achieve no successful trials under the same 30-demonstration budget, which shows that predicting hand poses is not the same as closing the loop on manipulation.
Fig. 5: Real-robot platform (paper Fig. 6a): dual Linker A7black arms, Linker Hand L6 dexterous hands, and a RealSense D435i egocentric camera.
Fig. 6: Wuji data glove used for teleoperation (paper Fig. 6b). Finger articulation comes from the glove; wrist pose comes from VIVE Trackers.
4.3 Ablations: How Much Do Touch, Memory, and Prediction Contribute?
The ablations isolate each component under matched settings. Removing tactile input produces the largest increase in finger-joint error: 22.9% on EgoTouch and 8.7% on Tachin. Removing recurrent memory hurts wrist prediction more, especially on Tachin, where translation and rotation errors rise by 21.6% and 9.4%. Shortening memory from K=8 to K=4 increases the six-metric average error by 2.3%. Removing both world-prediction objectives increases it by 1.0%.
| Variant | EgoTouch trans. | EgoTouch rot. | EgoTouch joint | Tachin trans. | Tachin rot. | Tachin joint |
|---|---|---|---|---|---|---|
| Without tactile input | 4.010 | 6.200 | 0.430 | 1.450 | 4.400 | 0.500 |
| Without recurrent memory | 4.070 | 6.280 | 0.370 | 1.690 | 4.560 | 0.490 |
| Shorter memory K=4 | 3.980 | 6.150 | 0.360 | 1.480 | 4.310 | 0.460 |
| Without world prediction (T+V) | 3.963 | 6.113 | 0.353 | 1.413 | 4.283 | 0.463 |
| Without world prediction (T only) | 3.962 | 6.112 | 0.352 | 1.412 | 4.263 | 0.462 |
| T-WLA | 3.960 | 6.100 | 0.350 | 1.390 | 4.170 | 0.460 |
Table 3: Ablation of tactile input, memory, and predictive supervision (paper Table 3). Lower is better on every metric.
The pattern is informative. Removing memory raises EgoTouch joint error from 0.350 to 0.370 and Tachin from 0.460 to 0.490, smaller than the tactile effect, but it raises Tachin wrist translation from 1.39 to 1.69. Memory matters more for whole-hand spatial continuity; touch matters more for local finger contact. They are not redundant views of the same capability.
4.4 The Visual Branch Still Works Without Touch
The paper also runs a visual-only experiment: touch is removed from training and evaluation, leaving RGB, instruction, pose history, recurrent memory, and the visual action expert. On GigaHands, T-WLA averages 0.65 cm translation, 3.64 degrees rotation, and 0.43 cm joint error, reducing the corresponding SmolVLA errors by 25.3%, 15.9%, and 54.7%. On OakInk2 it reaches 0.44 cm, 1.38 degrees, and 0.37 cm. On TACO it has the lowest rotation and joint errors; its translation of 0.75 cm is slightly above SmolVLA's 0.72 cm but still 22.7% below VITRA.
This matters because it shows that T-WLA is not a system that deliberately weakens vision and then relies on touch to recover. Visual memory and predictive supervision already provide a strong action prior; touch is a local correction on top. That is also why the real-robot tactile-free version reaches 47.25% rather than collapsing into a controller that can only wait for touch.
4.5 Key Numbers
| Item | Value |
|---|---|
| Visual / tactile update rate | 39.7 Hz / 119.1 Hz |
| Action chunk | H=16; 192 channels per position, 138 motion + 54 padding |
| Visual memory | S=8 slots, d=960, K=8 history sets; retrieval 15 heads, FFN 2560 |
| Tactile encoder | 8 frames of 21x21 per hand; 3 CNN layers with 32 channels; BiGRU 120x2; 64-dim output |
| Residual editor | 256 wide, 8 heads, FFN 1024, FiLM, zero-initialized output |
| Real-robot demonstrations | 30 per task, 4 tasks |
| Real-robot average success | 50.0% (47.25% without touch; SmolVLA 16.75%) |
| Hand-prediction gain | Finger MPJPE down 30.0% on EgoTouch / 27.0% on Tachin |
Limitations
Boundaries stated by the paper: the method assumes that the base action chunk remains a useful reference between visual updates. If visual planning sends the hand to the wrong place, the tactile editor can only patch a wrong plan; the observed SmolVLA failures often close the hand away from the object, leaving no contact to correct. The second boundary is that touch only solves grasp-related failures. Cup nesting stays at 17%, showing that alignment, insertion, and placement are not repaired by fingertip pressure alone.
Our assessment: first, 50.0% average success is a large improvement over 17.0% but still far from solving general dexterous manipulation, and the evidence covers four tasks, 30 demonstrations per task, and one dual-arm platform. Second, the method depends on the Linker Hand L6 fingertip array and its 441 pressure sites; the paper does not test whether reference subtraction and the contact gate transfer across sensors, hand morphologies, or missing tactile channels. Third, staged training with a frozen visual branch is stable but may leave joint-optimization gains on the table; the authors themselves present joint fine-tuning only as a proposed extension. Fourth, the paper says code is available, but the project page currently has no repository entry, so independent reproduction remains difficult.
Summary and Outlook
T-WLA's contribution is not merely adding a tactile branch. It turns visual memory into a shared interface between two timescales: slow vision uses it to generate the base action, fast touch uses it to make residual corrections. The base chunk supplies task semantics and long-horizon planning; tactile context supplies contact change; FiLM and a residual head connect them. Future-image and future-pressure prediction supervise the representations only during training and disappear at inference. Empirically, the 30% finger-joint improvement and 50.0% average real-robot success support this shared-memory plus asynchronous-residual design.
The next steps worth watching are: joint fine-tuning so tactile corrections can also update visual memory; evaluation across more sensors and hand types to test reference subtraction and the contact gate; and handling the two failure classes touch cannot fix alone, namely a wrong base plan and failures in non-grasp stages. Those may require touch to do more than edit actions: it may need to trigger replanning or a skill switch.
Quotes
"Vision supplies task and scene; touch supplies the instantaneous change in contact. What is missing is not another modality, but an interface that lets both share the same memory."
"The tactile editor does not regenerate the plan; it rewrites only the part of the base action chunk that has not yet been executed."
"When the base action takes the hand to the wrong place, even fast touch can only patch the wrong plan."
