FACTR 2: Learning External Force Sensing for Commodity Robot Arms Improves Policy Learning

FIRST: Turning Force Into a Better Data Distribution

The Failure Is Often Just Before Contact

The policy study starts from an empirical observation: failures in contact-rich tasks do not occur uniformly over a demonstration. Free-space motion may be easy, but the final approach requires precise alignment. Once contact begins, the policy must make fine corrections, detect a jam, and decide when to stop. Standard behavior cloning samples timesteps from the dataset according to their natural frequency. In a long demonstration, those difficult transitions can be a small fraction of every training batch.

FIRST uses the estimated external torque to assign each demonstration timestep to one of three phases: free space, pre-contact, or contact. The phase labels are not produced by manually annotated videos or task-specific event detectors. They are inferred from the magnitude of the learned residual. This is why NEXT and FIRST form a coherent system rather than two unrelated contributions.

Contact Score and Hysteresis

The scalar contact score is the L1 norm of the estimated external torque:

A single threshold would turn ordinary estimation noise into rapid boundary flicker. FIRST therefore uses high and low thresholds. Contact switches on above $T_{\mathrm{high}}$, switches off below $T_{\mathrm{low}}$, and otherwise retains its previous state:

A transition from zero to one defines a contact onset:

The phase rule is then spatial in trajectory time rather than semantic in object space. Contact timesteps are labeled contact. The one-second window immediately before each onset is labeled pre-contact. Every remaining timestep is free space. If $F$ is the control frequency, the labels can be summarized as

The definition is intentionally simple, but it contains assumptions that matter in deployment. A contact event that produces almost no external torque may be missed. A collision during transit may create a false onset. The threshold pair is shared across a task in the reported experiments, yet the magnitude of external torque depends on the arm and interaction. FIRST therefore inherits both the strengths and the calibration sensitivity of NEXT.

flowchart LR
    A[10 min contact-free trajectories] --> B[Joint position, velocity, command error history]
    B --> C[Stateless two-layer LSTM]
    C --> D[Predicted free-space torque]
    E[Motor current times torque constant] --> F[Measured motor torque]
    D --> G[External torque residual]
    F --> G
    G --> H[L1 contact score]
    H --> I[Hysteresis thresholding]
    I --> J[Free space]
    I --> K[One-second pre-contact]
    I --> L[Contact]
    J --> M[Weighted behavior-cloning sampler]
    K --> M
    L --> M
    M --> N[Vision-language-action policy]
    

Resampling Instead of Reweighting the Loss

Let $w_F$, $w_{PC}$, and $w_C$ be the sampling weights for free-space, pre-contact, and contact examples. FIRST samples a training point with probability

Most tasks use $1:5:1$, which gives pre-contact samples five times the sampling weight of the other phases. Cap Screwing uses $1:3:3$, because sustained interaction and the decision to stop are central to success. The original action-regression objective remains unchanged. Only the composition of each minibatch changes.

The paper reports that this distinction is practical, not cosmetic. A weighted-regression baseline changes the loss contribution of rare phases and performs worse or provides only limited improvement. FIRST keeps the regression target intact while increasing how often critical transitions appear. The result is a form of data curriculum driven by interaction state rather than by training time. It avoids assigning a huge gradient to a small set of hard samples, an effect that can destabilize optimization or encourage overfitting to a narrow segment.

The policy itself can be either a flow-matching Diffusion Transformer or an Action Chunking Transformer. Visual observations are encoded with DINOv3-Base, proprioceptive and force inputs are tokenized with MLP projections, and the action horizon is 30. The flow-matching objective trains a velocity field between noise and an expert action chunk:

Here $\mathbf{x}_1$ is an expert action chunk, $\mathbf{x}_0$ is Gaussian noise, and $\mathbf{x}_\alpha$ is an interpolation. FIRST is orthogonal to that backbone choice because it operates on the data sampler, not on the generative model.

What the Public Implementation Confirms

The public repository is useful because it lets readers check the paper's narrowest technical claims independently. At commit 0df02c5b26e33911078152b6c587d6e8a160d279, the dataset builder constructs the same three input channels described in the paper and supervises the final timestep of each history window with measured joint torque:

# src/factr2_next/factr2_next/training/dataset.py
x_step = np.concatenate(
    [pos[:n], vel[:n], cmd[:n] - pos[:n]], axis=1
)
return x_step, torque[:n]

The recurrent regressor also matches the architectural description. A two-layer LSTM consumes the sequence, and the final hidden representation is passed to a two-layer MLP head:

# src/factr2_next/factr2_next/training/models.py
self.lstm = nn.LSTM(
    input_size=input_size,
    hidden_size=hidden_size,
    num_layers=num_layers,
    batch_first=True,
    bidirectional=bidirectional,
)

def forward(self, x):
    y, _ = self.lstm(x)
    return self.head(y[:, -1])

Training uses PyTorch mean-squared error, and inference subtracts the predicted free-space torque from the measured torque. The inference node calculates an L1 contact magnitude and applies the same high/low hysteresis logic. The repository also exposes the mechanical form of the residual:

# src/factr2_next/factr2_next/inference/inference_node.py
tau_free = self._predict_free_torque()
tau_ext_raw = measured - tau_free

The code boundary is equally important. The public repository mainly documents NEXT. It does not provide the complete FIRST training stack, task environments, or the exact experiments that produced the five-task policy results. The README also distinguishes runtime safeguards such as filtering, gating, ramps, and clipping from the reported research setup. Those safeguards are useful deployment features, but they are not evidence that the bare estimator has already solved stability and safety on every arm.

Evidence: Does the Residual Behave Like Force?

Franka Has a Reference Signal

Franka is used because it provides a built-in external-torque estimate that can serve as the contact reference. The authors collect five minutes of held-out contact data while a human applies forces, and another five minutes of held-out free-space motion where the true external torque is zero. The comparison includes FILIC, a disturbance observer, and, in free space, Franka's own sensor signal.

In contact, NEXT's 0.547 Nm error is 87.6% lower than FILIC and 62.8% lower than the disturbance observer. The free-space comparison is more revealing. The true external torque is zero, yet the factory sensor has a mean residual of 0.449 Nm. NEXT's residual is 0.414 Nm. The learned model is not only tracking a reference sensor; it is exploiting the known absence of contact to reduce hardware-specific noise that the reference itself retains.

This result should not be read as proof that a learned residual is universally more accurate than a torque sensor. The contact reference is still Franka's own estimate, not an independent metrology-grade ground truth. The strongest claim supported by the experiment is conditional: on this robot, controller, dataset, and comparison protocol, NEXT approximates the factory estimate well while producing a cleaner free-space signal.

Piper Tests the Commodity-Arm Case

The AgileX Piper has no dedicated joint-torque sensor, so a direct contact-force ground truth is unavailable. Free space still provides a meaningful test because the true external torque is exactly zero. On held-out Piper motion, NEXT reaches 0.018 Nm mean L1 error, compared with 0.435 Nm for FILIC and 0.248 Nm for the disturbance observer. The large gap is evidence that learning the individual actuator and transmission behavior matters on low-cost hardware.

The experiment also marks the paper's evidence boundary. Piper contact behavior cannot be scored against a joint-torque sensor because the robot does not have one. The authors evaluate contact usefulness through teleoperation and policy outcomes instead. Those are important downstream tests, but they mix estimator error with controller gains, operator adaptation, task difficulty, and policy learning. They do not establish contact-torque accuracy in newton-meters.

Teleoperation As a Closed-Loop Test

A static regression score can miss the properties that matter in a feedback loop. The user study therefore asks participants to perform a wiping task with five conditions: no feedback, disturbance-observer feedback, position-position feedback, FACTR Teleop using Franka's built-in torque sensor, and FACTR Teleop using NEXT. Both FACTR versions scale external torque into leader-arm torque:

Twenty participants rate ease of use from one to five while the system records the joint torque they apply. NEXT is rated easier than the baselines and requires less unnecessary exertion, close to the sensorized FACTR condition. The Piper study repeats the comparison without the factory-sensor condition. NEXT again outperforms the available baselines.

This is a better test than free-space error alone because the estimate is used online. It still does not isolate every cause. A feedback controller can be tuned to a noisy signal, and subjects can adapt to a particular feel. The proper conclusion is that the learned residual is useful enough to support the tested bilateral teleoperation interface, not that it is interchangeable with a calibrated sensor in every control law.

Evidence: Does Better Force Data Improve Policies?

The policy study uses a bimanual Piper, four RGB cameras, 250 demonstrations per task, and 20 rollouts per policy. The five tasks are LEGO Assembly, NIST Belt Assembly, NIST Insertion, Tool Clean Up, and Cap Screwing. All are long-horizon and contain multiple stages. The reported metric is task progress rather than a binary final success, which preserves information about policies that solve most of a sequence before failing at a contact transition.

Pre-Contact Data Is the Surprise

The phase ablation is the strongest conceptual result in the policy section. Upsampling only contact samples reaches an average progress of 0.670, below the two mixed strategies. Upsampling only the one-second pre-contact window reaches 0.818. Upsampling pre-contact and contact reaches 0.811, but wins on Cap Screwing, where sustained rotation and correct termination matter more.

The interpretation is straightforward but easy to miss. Contact is where the consequence appears; pre-contact is where many policies decide the pose and velocity that determine whether the contact succeeds. A demonstration that spends much more time moving freely can dominate batches precisely when the model needs more examples of final alignment. FIRST rebalances exposure to that scarce causal interval.

The result also warns against treating force as a scalar difficulty score. A task with sustained interaction does not necessarily benefit most from more contact frames. Cap Screwing benefits when both pre-contact and contact are increased, while the other tasks prefer a heavier pre-contact bias. The best phase mixture remains task-dependent in the paper, and the authors rely on task progress to select it.

Why Not Feed the Raw Current?

Commodity arms already expose current, so an obvious baseline is to give current directly to the policy. The paper compares this with NEXT external torque on NIST Insertion and Cap Screwing. Raw current contains free-space effort, gravity, friction, payload, and contact in one signal. A policy conditioned on a single current observation must implicitly subtract all the nuisance terms while also solving the manipulation task.

Raw current improves NIST Insertion but does not move Cap Screwing. NEXT torque improves both. The attention analysis is consistent with the intended mechanism: the torque-conditioned policy attends more strongly to the force input around tightening, while the current-conditioned policy shows weaker event alignment. Attention is diagnostic rather than causal, but the combination of task outcome and temporal alignment supports the claim that removing the learned free-space baseline produces a cleaner policy input.

An alternative would be to give the policy the same long proprioceptive history used by NEXT and let the downstream model perform the decomposition itself. The authors argue against that route for practical reasons: it increases the policy's optimization burden, may encourage trajectory-specific overfitting, and duplicates a specialized computation inside an already multimodal model. NEXT acts as an explicit interface that isolates force estimation from visuomotor control.

What Generalizes From This Paper

The first reusable idea is supervision through environmental control. NEXT does not need manually labeled external forces because the data collector controls the contact condition. The target becomes exact by construction: if the arm is genuinely free, the external torque is zero. This is a useful pattern for other hardware-specific latent quantities that are expensive to measure but easy to constrain during collection.

The second idea is to expose an intermediate variable instead of hiding it in one end-to-end objective. A joint-torque residual can drive force feedback, provide an observation to a learned policy, segment demonstrations, or support diagnostics. This makes failures easier to localize. A poor insertion policy might be caused by a weak visual encoder, an inaccurate force estimate, an unhelpful sampling ratio, or an unstable controller. FACTR 2 does not solve all four, but it makes them separately testable.

The third idea is that data selection can be a form of force-aware learning without changing the policy architecture. FIRST uses force to identify when learning is likely to matter, then changes sample frequency while preserving the original regression objective. This is attractive for modern action-chunking policies because the same sampler can be attached to ACT, a flow-matching transformer, or a future backbone.

The engineering claim is also intentionally narrow. Ten minutes of free-motion data and one minute of training are enough to learn the baseline on the tested robot, not enough to infer a universal transferable dynamics model. The paper demonstrates several robots and downstream uses, but the estimator remains hardware-specific. That is a reasonable tradeoff for commodity manipulation: cheap per-robot adaptation can be preferable to an expensive, fragile universal calibration.

Limitations and Evidence Boundaries

The first author-stated limitation is the motor torque constant $K$. NEXT learns the free-space torque pattern, but it does not remove the initial conversion from current to torque. If the manufacturer's constant or current-torque curve is inaccurate, the absolute external-torque scale will be inaccurate as well. The paper argues that force feedback is often rescaled for operator comfort and policy inputs are normalized, so the practical effect may be small in the reported tasks. It is still a real limit for metrology, force-limit enforcement, or precise interaction control.

The second author-stated limitation is robot specificity. Friction, actuator dynamics, controller behavior, mounting, and payload all shape the free-space torque distribution. NEXT may need to be retrained for each arm. The training cost is low, but retraining is not the same as calibration-free generalization. The paper does not establish that one model transfers zero-shot across grippers, payloads, wear states, or mechanically similar arms from different manufacturers.

A third boundary follows from the evaluation design. Piper has no contact-torque reference, so its contact accuracy is inferred from user studies and policy outcomes rather than measured against an independent sensor. Those tests show usefulness, but they do not separate estimator error from feedback gain, policy capacity, or task-specific sampling. Conversely, the Franka contact experiment compares against a built-in external-torque estimate, which is a strong practical reference but not an external metrology standard.

FIRST introduces its own tuning dependencies. Contact onset depends on the L1 score and two thresholds. The pre-contact horizon is fixed at one second. The best sampling ratio is selected per task: $1:5:1$ for most tasks and $1:3:3$ for Cap Screwing. A new task may have a different contact duration, a slower approach, or multiple touches that the simple onset rule does not represent well. The public code primarily covers NEXT, so independent reproduction of the full policy study is limited.

Safety remains outside the core evidence. A force estimate is not a certified force limit, and runtime filters, ramps, gates, and clipping can change the closed-loop response. The repository distinguishes these deployment guards from the research experiments. Any real system should treat NEXT as one sensor-like input with its own failure modes, not as a replacement for an emergency stop or a validated safety layer.

Conclusion

FACTR 2 makes a compelling case that force-aware manipulation can begin with a cheap question: what torque does this arm need when it is touching nothing? NEXT answers that question from free-motion data and subtracts the learned answer from measured motor torque. The residue is accurate enough on Franka to approach the factory signal, useful enough on Piper to drive teleoperation, and informative enough to improve five contact-rich policies when FIRST uses it to reshape training data.

The best reported policy configuration reaches 0.818 average task progress, with the specific lesson that the one second before contact is often more valuable to oversample than contact alone. That result gives the paper value beyond its sensor-cost story. It describes a practical data-selection primitive for imitation learning: use an interaction signal to find the scarce transitions, then expose the policy to them without distorting the regression objective.

The next technical questions are clear. Can a free-space model adapt online when payload or wear changes? Can a shared representation reduce the cost of retraining across arms while preserving the hardware-specific residual? Can contact thresholds and phase ratios be estimated from failure data instead of selected per task? FACTR 2 does not answer those questions, but it defines a useful interface on which they can be studied.

Learn what the arm should feel like when it touches nothing; the remaining torque is where contact-aware teleoperation and policy learning begin.