SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation
SERF represents the environment and the articulated robot body as neural points in one shared latent space and updates them online. Environment points follow object-level rigid registration, while robot surface points are posed by forward kinematics. A shared residual decoder maps local point features to DINOv3 patch embeddings, and a two-stage Point Transformer produces eight map tokens from three robot-centered radii, both end effectors, robot-only, environment-only, and global branches. These tokens condition a VLA policy together with images, proprioception, and language. Across three long-horizon BEHAVIOR-1K tasks, SERF raises average progress from 44.0% for fine-tuned PI0.5 to 58.7%, and the paper also reports gains when labels and base poses are estimated onboard, when object-drop recovery is required, and in unvisited regions.