FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvement

FAR: Learning from Your Own Mistakes at Test Time, and Keeping the Lessons

Paper: FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvement | Authors: Haoran Hao, Shahram Najam Syed, Jeffrey Ichnowski, Jeff Schneider | Affiliation: Robotics Institute, Carnegie Mellon University | Link: https://arxiv.org/abs/2607.01111 | Code: https://github.com/Hoar012/FAR (MIT license, official CoRL 2026 implementation) | Status: accepted at CoRL 2026, v2 (2026-10-07)

One-paragraph summary

FAR lets a deployed robot policy learn from its own failures without resetting the scene and without calling a human. After a failed attempt, an IQL-style conservative critic attributes the failure to the few action chunks whose value dropped the most; the policy then samples candidate chunks, distance-filters and critic-ranks them into positive alternatives, and runs only 10 steps of diffusion preference optimization (1.5-3 s on a single A5000) before retrying with a temporally smoothed Gaussian perturbation for local exploration. Successful recovery trajectories flow back into three replay buffers and drive an advantage-weighted continual-improvement loop. Across nine simulated manipulation tasks FAR beats standard diffusion policy by 17.6 points on average, and on three real xArm tasks by 11.7 points (80.0 vs 68.3), while also cutting the environment resets that online learning normally burns.

Background and motivation

Policies behavior-cloned from offline expert demonstrations look excellent in evaluation and then fail in deployment: the grasp lands a few millimeters off, the can gets knocked over, the bread slides to the table edge. These failure states are out-of-distribution for the pretrained policy because they barely appear in the demonstration data, so re-executing the same policy simply replays the same mistake. The policy has no new information at the failure state, and its sampled actions are nearly identical to the ones that just failed.

Existing remedies each carry a price. Human-in-the-loop correction (DAgger and its successors) works but converts deployment into another expensive data-collection campaign. VLM-based failure detection and language-level correction add inference latency and still have to translate advice into actions. Training a separate recovery policy with reinforcement learning concedes that the main policy cannot save itself and adds a second module with its own training budget.

The closest family of prior work is retry-based recovery, especially Bellman-Guided Retrials (BGR), which compares expected against observed task progress and skews action sampling away from previously failed actions during retries. BGR is non-parametric: no weights are updated at test time, only a failure memory is maintained. FAR attacks exactly this design. A failed interaction usually changes the environment (the can is now somewhere else), so a memory of old states becomes a poor reference once the scene has been disturbed.

FAR starts from a simpler premise: a failure is itself a supervision signal, and the cheapest kind available. Two obstacles remain. First, credit assignment: which part of a long failed trajectory deserves the blame? Second, exploration: the policy's action distribution comes from offline demonstrations, so even knowing what to avoid may not be enough to sample a better action in an out-of-distribution state. FAR answers the first with value-based attribution plus preference learning, and the second with lightweight action perturbations during retries.

Framework overview

Figure 1: overall framework. After a failure, FAR identifies failure-inducing actions via value estimation, updates the policy with failure examples and alternative positives, and adds the collected trajectories to the replay buffer for continual improvement.

The deployment protocol is the hidden protagonist of the paper. When an attempt fails, the environment is not reset: only the arm returns to a predefined start pose while objects and scene stay exactly as the failure left them, and the robot gets up to N retries (5 in simulation, 3 on the real robot). This protocol matches real deployment, where nobody re-stages the scene, and it forces the policy to finish tasks in messes of its own making. Reward signals are used only to train the critic; test-time adaptation needs nothing but the binary outcome of each attempt plus recent interaction history.

Conservative value estimation: learn not to be optimistic

Failure attribution needs a value function, and failure states are precisely where a standard critic overestimates values through extrapolation error. FAR therefore trains a chunk-conditioned critic with the Implicit Q-Learning (IQL) objective: a state-value network $V_{\psi}$ and an action-chunk-value network $Q_{\phi}$,

$$\mathcal{L}_{V}=\mathbb{E}_{(s,\mathbf{a})\sim\mathcal{D}}\left[L_{2}^{\tau}\left(Q_{\phi}(s,\mathbf{a})-V_{\psi}(s)\right)\right],$$

$$\mathcal{L}_{Q}=\mathbb{E}_{(s,\mathbf{a},s^{\prime})\sim\mathcal{D}}\left[\left(r(s,\tilde{a})+\gamma V_{\psi}(s^{\prime})-Q_{\phi}(s,\mathbf{a})\right)^{2}\right],$$

where $L_{2}^{\tau}(u)=|\tau-\mathbf{1}(u<0)|u^{2}$ is the expectile regression loss with $\tau=0.7$, and $\mathbf{a}=a_{1:H}$ is an action chunk of horizon $H=8$. The environment executes only the temporally aggregated single action $\tilde{a}=g(\mathbf{a})$, so the TD target is defined with $\tilde{a}$ while $Q_{\phi}(s,\mathbf{a})$ serves as a scorer for candidate chunks. IQL's conservatism comes from learning values mainly on actions supported by the offline data, which is exactly the property failure states demand.

Failure attribution: rank chunks by value drop

Given a failed trajectory $\xi=\{(s_t,a_t)\}_{t=0}^{T}$, each chunk $\mathbf{a}_t=a_{t:t+H-1}$ is scored by the value change it causes, $\Delta V_t=V_{\psi}(s_{t+H})-V_{\psi}(s_t)$. Rewards are negative before task completion, so values normally rise as the task progresses; a pronounced drop marks unstable or failure-inducing behavior. Instead of a fixed threshold, FAR ranks all chunks of the failed trajectory and takes those at or below the $\rho$-th percentile (default $\rho=10\%$) as negatives:

$$\xi^{-}=\left\{(s_t,\mathbf{a}_t^{-})\mid \Delta V_t \leq \mathrm{Percentile}_{\rho}(\{\Delta V_j\})\right\}.$$

The percentile is a pragmatic choice: thresholds need per-task tuning and value scales are not comparable across tasks, while a percentile guarantees a fixed fraction of negatives per failed trajectory and hence stable gradient magnitudes. The ablation agrees: the fixed-threshold variant averages 73.0 and treating the whole trajectory as negative averages 72.0, both below the percentile version's 74.9.

Contrastive sample construction: avoid the bad, know the good

Learning from negatives alone leaves the update direction undefined: suppressing the failed action pushes probability mass somewhere, possibly distorting the policy's multi-modal structure. FAR therefore pairs every negative with positives. It samples $M=64$ candidate chunks from the current policy, $\mathcal{C}(s_t)=\{\mathbf{a}_t^{(i)}\}$, and keeps only candidates at a moderate $L^2$ distance from the negative:

$$\mathcal{C}_{\mathrm{safe}}(s_t)=\left\{\mathbf{a}_t^{(i)}\in\mathcal{C}(s_t)\mid d_{\min}\leq d(\mathbf{a}_t^{(i)},\mathbf{a}_t^{-})\leq d_{\max}\right\},$$

Candidates too close to the failed action teach nothing; candidates too far leave the policy's support, where the critic's scores cannot be trusted. In practice $d_{\min}$ is set to the mean distance $\bar{d}$ between candidates and the negative, and $d_{\max}=1.0$. Survivors are ranked by the critic and the top $K=8$ become positives:

$$\mathcal{A}_t^{+}=\operatorname{TopK}_{\mathbf{a}\in\mathcal{C}_{\mathrm{safe}}(s_t)}Q_{\phi}(s_t,\mathbf{a}).$$

Each failed state thus yields preference pairs $\{(\mathbf{a}_t^{+},\mathbf{a}_t^{-})\}$ stating that, in this state, this chunk should be preferred over that one. Replacing critic ranking with random selection drops the average from 74.9 to 72.3; $K=1$ drops to 71.4 and $K=16$ to 68.0, because too few positives give unstable guidance while too many inject noisy positives with conflicting optimization directions.

FCPA: ten gradient steps of preference optimization

Diffusion policies have no tractable action likelihood, so DPO cannot be applied directly. Following preference optimization for diffusion models, FAR uses the negative denoising loss as a surrogate preference score: for chunk $\mathbf{a}$ and state $s$, $\ell_{\mathrm{diff}}(\mathbf{a},s;\theta)=\mathbb{E}_{k,\boldsymbol{\epsilon}}\left[\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}(\mathbf{a}_k,s,k)\|^{2}\right]$, where $\mathbf{a}_k$ is $\mathbf{a}$ noised to diffusion step $k$. The pairwise objective is

$$\mathcal{L}_{\mathrm{pref}}=-\log\sigma\left(\beta\left[\ell_{\mathrm{diff}}(\mathbf{a}_t^{-},s_t;\theta)-\ell_{\mathrm{diff}}(\mathbf{a}_t^{+},s_t;\theta)\right]\right),$$

with sigmoid $\sigma$ and preference strength $\beta=0.1$. Minimizing it assigns lower denoising error to the preferred chunk than to the failed one, shifting sampling mass from failed behavior toward recovery behavior. The entire test-time adaptation runs $n_{\mathrm{tta}}=10$ AdamW steps at learning rate $1\times10^{-5}$, taking roughly 1.5-3 seconds on a single A5000, cheaper than executing one more retry. A step-count sweep shows that too many steps hurt, which the authors attribute to overfitting the local adaptation data.

Retry perturbation: explore, but do not shake the robot

FCPA removes the old mistake but cannot widen the policy's support: in an out-of-distribution state the policy may simply never sample the action that routes around the obstacle. FAR therefore perturbs the executed action. With probability $\epsilon_{\mathrm{explore}}=0.5$ it draws a target perturbation $\boldsymbol{\delta}^{\star}\sim\mathcal{N}(\mathbf{0},\Sigma)$ (per-dimension std 0.1) and holds it for a few steps, otherwise zero; the applied perturbation is exponentially smoothed, $\boldsymbol{\delta}_t=\alpha\boldsymbol{\delta}_{t-1}+(1-\alpha)\boldsymbol{\delta}^{\star}$ with $\alpha=0.3$, and the executed action is

$$\tilde{a}_t=\hat{a}_t+\boldsymbol{\delta}_t,$$

where $\hat{a}_t=g(\mathbf{a}_t)$ is the temporally aggregated action. The smoothing is not cosmetic: high-frequency noise makes real arms jitter and can damage hardware, so on the robot the perturbation is applied only to the translational components of the end-effector delta action. The ablation finds $\epsilon_{\mathrm{explore}}=0.5$ the best trade-off (average 74.9); at 0.0 long-horizon tasks such as Door survive but short-horizon tasks stay stuck in local optima, while at 1.0 short tasks benefit and long tasks get pushed out of the training distribution. The smoothed variant scores close to pure Gaussian noise in simulation (73.8 vs 74.9) but is far friendlier to real hardware, a rare honest trade of simulation points for deployability.

flowchart TB
    START[Attempt i: roll out policy in the kept scene] --> JUDGE{Success?}
    JUDGE -- yes --> SUCC[Trajectory to D_succ, episode ends]
    JUDGE -- no --> ATTR[Score every chunk by value drop dV_t = V(s_t+H) - V(s_t)]
    ATTR --> NEG[Negatives: chunks at or below the rho=10% percentile of dV]
    NEG --> CAND[Sample M=64 candidate chunks from current policy]
    CAND --> FILT[Keep candidates whose L2 distance to the negative lies in d_min to d_max]
    FILT --> TOPK[Critic ranks survivors, keep top-K=8 positives]
    TOPK --> PREF[10 AdamW steps on the diffusion preference loss, about 1.5-3 s]
    PREF --> PERT[Retry with temporally smoothed Gaussian perturbation, eps=0.5]
    PERT --> START
    SUCC --> CPI[Critic update on D_exp + D_succ + D_fail]
    CPI --> AWR[Advantage-weighted denoising update on D_exp + D_succ only]

Figure 2 (drawn for this review): control flow of one FAR episode. The test-time loop updates only the policy weights for 10 steps; the continual-improvement loop updates critic and actor from the three replay buffers after the episode ends.

From recovery trajectories to continual policy improvement

A retry that succeeds produces the most valuable data in the paper: a trajectory that starts from a state where the initial policy reliably fails and still completes the task, thereby labeling the current capability boundary. FAR keeps three buffers: offline demonstrations $\mathcal{D}_{\mathrm{exp}}$, successful online trajectories including recovered ones $\mathcal{D}_{\mathrm{succ}}$, and failed trajectories $\mathcal{D}_{\mathrm{fail}}$. Each learning epoch collects 20 episodes, first updates the critic on the aggregated set $\mathcal{D}=\mathcal{D}_{\mathrm{exp}}\cup\mathcal{D}_{\mathrm{succ}}\cup\mathcal{D}_{\mathrm{fail}}$ (failures help the critic recognize bad states), then computes advantages $A(s,\mathbf{a})=Q_{\phi}(s,\mathbf{a})-V_{\psi}(s)$ and weights $w(s,\mathbf{a})=\exp(A(s,\mathbf{a})/\eta)$ with $\eta=1.0$, but updates the actor only on $\mathcal{D}_{\mathrm{exp}}\cup\mathcal{D}_{\mathrm{succ}}$:

$$\mathcal{L}_{\pi}=\mathbb{E}_{(s,\mathbf{a}_0)\sim\mathcal{D}_{\mathrm{exp}}\cup\mathcal{D}_{\mathrm{succ}},\,k,\,\boldsymbol{\epsilon}}\left[w(s,\mathbf{a}_0)\left\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}(\mathbf{a}_k,s,k)\right\|^{2}\right].$$

This asymmetry, critic sees everything while the actor sees only the good data, prevents behavior cloning from being polluted by stagnation and partial failures. The payoff shows up in data efficiency: counted in environment resets, each FAR episode yields more usable data; counted in environment timesteps, FAR spends its extra steps on hard cases near the capability boundary, so improvement per step is higher as well. Re-running continual improvement with a sparse reward (3 on completion, 0 otherwise) still improves steadily, because attribution relies on relative value changes rather than on the absolute shape of the reward.

Experimental setup

Simulation covers nine vision-based manipulation tasks across ManiSkill, RoboSuite and RoboMimic, with 50 evaluation episodes per task, up to 5 attempts per episode, and mean plus standard deviation over three seeds. The real robot is a 7-DoF xArm with end-effector delta-position control at 30 Hz, a third-person ZED Mini camera, 20 episodes per task, up to 3 attempts, and success judged by a human observer. Demonstration counts are deliberately tiny (3 to 100) to make failures frequent. All experiments run on a single NVIDIA A5000 GPU.

Four baselines are compared: DP, the behavior-cloned diffusion policy with standard rollout; DP-NR, the same policy allowed multiple attempts without updates or resets (naive retry); DP-BGR, the authors' re-implementation of Bellman-Guided Retrials adapted to this setting; and DP-FAR, the proposed method. In policy-improvement experiments all baselines receive Gaussian action noise during data collection so exploration is fair.

Simulation results: retries help, but only FAR turns retries into learning

The most instructive row in Table 1 is not FAR's but DP-NR's: merely allowing retries without resets lifts the average from 47.0 to 59.1 (+12.1), which means the evaluation protocol itself dominates the score of any retry-based method and cross-paper comparisons must align attempt counts first. On top of that, FAR adds another +4.3 (63.4) and halves the standard deviation from 8.0 to 4.5: it is not only more successful but more consistent. Door is the clearest case. DP manages 46.7, retry methods jump above 80, and FAR reaches 86.7 with half the variance, because after a failed door attempt the handle position has changed, the non-parametric BGR memory is mismatched, while FAR re-fits its weights to the new state.

Table 2 turns the demonstration count into the independent variable, and the pattern is monotone in scarcity: with only 5 demonstrations on Lift, DP scores 50.0 while FAR scores 84.7 (+34.7); on Square with 50 demonstrations it is 19.3 versus 38.0, nearly a doubling. The intuition is direct: less data puts the capability boundary closer, failures become frequent, and test-time learning has more information to extract. The single exception is Can with 20 demonstrations, where BGR's 90.7 edges out FAR's 89.3; the authors do not hide it, and the standard deviations (1.2 versus 6.1) put the gap inside noise.

Real-robot results: +11.7 points on an xArm

Figure 3: task setups. The three left columns are the nine simulated tasks in ManiSkill, RoboSuite and RoboMimic; the right column shows the three real xArm tasks Drawer, Pot and Tea.

The real-world numbers (read off the paper's bar chart by axis calibration) show a clean difficulty gradient. Drawer is a short pushing task where DP already reaches 90 and retry methods saturate at 95, leaving FAR no room to shine. Pot and Tea are where recovery actually matters: once the lid sits crooked or the teapot angle is wrong, the policy must identify which action caused the failure and try something different. FAR scores 80.0 versus 60.0 on Pot and 65.0 versus 55.0 on Tea, averaging 80.0 against 68.3, which is the +11.7 points claimed in the abstract. On the robot the perturbation touches only translational components and is temporally smoothed, and success labels for the 20 episodes per task come from a human observer.

Ablations: every component earns its place

The most striking row is "without positives": the average collapses from 74.9 to 61.3, and Door drops to 44.0±22.0, worse than not learning at all. Suppressing negatives alone warps the policy's probability structure, which is exactly why FCPA insists on paired supervision. Swapping the preference objective for advantage-weighted regression yields only 68.7, because AWR mainly exploits positive updates from failure trajectories and lacks an explicit signal to avoid the failed behavior. On attribution, the percentile rule (74.9) beats a fixed threshold (73.0) and whole-trajectory negatives (72.0), confirming that not every transition deserves blame for a failure.

Extra baselines and an instructive negative result

CCIL and V-GPS remain constrained by offline data and never learn from the out-of-distribution failures actually encountered at test time, so both fall below FAR on average. More instructive is DP-FAR-Realtime, which compares $V(s_t)$ and $V(s_{t+H})$ during rollout and intervenes as soon as value drops. It averages only 56.2, worse than no intervention, because online intervention keeps sending the robot back to previously visited states and strangles task progress: continuing the rollout may risk failure, but it also allows progress. Hindsight labeling after each attempt turns out to be the more reliable attribution. This ablation spares future readers a trap that looks entirely natural from the outside.

Cost and qualitative behavior

The cost accounting is clean. The default 10 adaptation steps take about 1.5-3 seconds on a single A5000, the same order as executing one retry; offline training of critic and policy (15k-30k and 30k iterations) happens beforehand. A sweep over adaptation steps shows too few cannot learn and too many overfit the local preference data, with 10 steps the empirical optimum.

Figure 4: qualitative results on Can. Red points mark positions FAR flags as potentially failure-inducing, blue points normal ones; FAR localizes the failure to the contact phase where the gripper pushes the can away, and the second attempt succeeds. BGR's repulsion from the stored failed action loses its reference once the can has moved.

Figure 5: appendix qualitative results on Lift and Square with the same red-blue annotation; flagged failure points concentrate in contact and insertion phases.

Limitations

The authors state three. First, experiments cover only diffusion policies trained from scratch in single-task settings; evaluating the method on large-scale pretrained policies such as VLA models, which carry richer priors and broader generalization, is left as future work. Second, the framework has no principled failure-detection mechanism and relies on environment feedback and time limits (or a human observer on the real robot); they plan to incorporate pretrained models such as VLMs for task tracking and failure detection. Third, the safety of exploration during recovery needs dedicated effort, since perturbed exploration on physical hardware is not risk-free.

Several further caveats deserve the reader's own weighing. First, the real-robot study uses 20 episodes per task with human success labels; with 20 Bernoulli trials the standard error is around 9 points, so the 80.0 versus 68.3 gap should be read as directional evidence rather than a precise effect size. Second, attribution quality rests entirely on a critic trained only on offline demonstrations; IQL's conservatism mitigates overestimation in failure states, but whether the ranking of value drops identifies causally responsible actions is supported only by qualitative visualization, with no controlled attribution-accuracy evaluation. Third, the deployment protocol assumes the scene can be kept after a failure, yet many real failures (objects on the floor, spilled liquid) do not permit in-place retries, and the boundary of the protocol's applicability is not discussed. Fourth, preference-pair construction depends on six hyperparameters, and the ablations sweep only some of them individually, leaving joint sensitivity unknown. Fifth, the BGR baseline is the authors' own re-implementation, so cross-paper comparisons should be treated with care.

Conclusion and outlook

FAR's contribution is not a new loss function but a redefinition of failure: from the end of an evaluation to the beginning of a learning step. The package is a no-reset deployment protocol, a conservative-critic pipeline for attribution and candidate screening, a ten-step test-time update, and a loop that recycles recovery trajectories into training. The reporting deserves praise as well: the +12.1 contributed by the retry protocol alone is shown explicitly, the negative result on realtime detection is reported honestly, and sparse-reward continual improvement is included in the appendix, so readers can see where each point of gain comes from.

Two roads are now clearly marked for follow-up work. One is porting FAR's attribution and preference adaptation onto large pretrained policies such as VLA models, testing whether test-time learning can exploit stronger priors. The other is replacing the human success label with automatic failure detection from VLMs or proprioception, closing the loop without any human in it. If both hold, the operations paradigm of deployment-as-training gains one of its missing pieces.

Golden quote