Temporal Visuo-Tactile Learning for Dexterous Grasp Stability
Most robotic grasping research picks a pose from vision and closes a parallel gripper, while humans rely on fingertip touch and almost never drop things. The LASR Lab at TU Dresden reframes the question as "is this multi-fingered grasp stable before we lift?" and answers it with data. They mount four Digit 360 sensors on the fingertips of a 16-DoF Tilburg Hand on a 7-DoF xArm7 and automatically collect 10,000 grasp trials over 200 objects (105 rigid, 95 deformable, 1.9-244 g), recording external RGB-D, proprioception and four tactile streams (camera, audio, IMU, pressure) throughout each grasp. Stability labels come from SAM 3 measuring post-lift object height and agree with manual verification on 92.34% of trials. The predictor consumes a 3-second window anchored at grasp initiation: 16-frame RGB and tactile clips go through pretrained VideoMAEv2-Base, 100-step proprioception and 224-step audio/IMU/pressure go through Transformers, fingertip features are fused across modalities, time and fingers by weight-shared factorized attention, and everything is aggregated by attention fusion into a post-lift stability probability. Under 5-fold object-disjoint cross-validation, vision+proprioception+touch reaches 83.55% and removing touch costs 3.99 points (79.56%). Tactile spatial resolution matters monotonically (79.40% at 1x1 up to 83.55% at 112x112), and freezing touch to its final frame costs 2.07 points, so the useful signal is high-resolution and dynamic. Deployed as an online lift-or-regrasp gate (threshold 0.95, up to five regrasps) on 20 objects excluded from training, the visuo-tactile gate reaches 82.0% success among executed lifts, 10.5 points above a non-tactile gate. The ~1.2 TB dataset is public on OPARA.