In-Context Learning for Robots: Methods and Applications

Paper: In-Context Learning for Robots: Methods and Applications (arXiv:2609.36012, arxiv.org/abs/2609.36012, submitted 2026-09-28, cs.RO)
Authors: Haojian Huang, Zexi Li, Junhao Guo, Yehang Zhang, Wenxuan Peng, Bohan Zhou, Weilin Ruan, Leyi Wu, Chenxu Wang, Jianchong Su and 29 others, 39 authors in total (Knowin AI; HKUST Guangzhou; CUHK; Tongji University; Tsinghua University; Harbin Institute of Technology, Shenzhen; Westlake University; Tianjin University. Corresponding authors: Ying-Cong Chen and Yinchuan Li)
Code and resources: a project page and the GitHub repository awesome-robots-icl accompany the survey, together with Supplement S1, which ships the item-level corpus census and the reported-comparison records. This is a literature review, so there is no trainable model code.

Figure 1: the six learning horizons. S1 encodes control explicitly and S2 learns reusable policies. With neural parameters frozen at deployment, S3 adapts to interaction outcomes while S4 infers task requirements from teaching. S5 improves how subsequent tasks get learned, and S6 extends that process into validated knowledge exchange across embodiments.

In One Sentence

This roughly 100-page, 412-reference survey defines in-context learning for robots as using demonstrations and interaction to steer competence that already exists, with neural parameters held fixed during deployment; it sorts the field into four method families according to which intermediate quantity carries context into action, and it pairs that taxonomy with evaluation controls that separate being responsive to teaching from transferring physically and from benefiting from retained experience, before pointing the agenda toward compositional task acquisition and physical recursive self-improvement.

Background and Motivation

A robot can be able to move and still not know what to do. A palletizing arm may already grasp every part in the cell reliably, yet still need a demonstration to fix the assembly order, a history to confirm which placements are already complete, and contact evidence to adapt to a tighter fit than it was built for. The same visible scene can therefore license different actions. The authors call this gap decision-relevant ambiguity, and their framing point is that it cannot be closed by better motor competence alone: pretraining supplies reusable perception and control, but the present scene may leave several programs feasible, may conceal the events that led up to it, and may say almost nothing about how an unfamiliar material will respond.

Fine-tuning and in-context learning are the two routes for acting on new evidence. Fine-tuning updates pretrained parameters into task parameters using task data. In-context learning instead puts the evidence into the inference-time input and leaves the neural parameters untouched during deployment. The difference is not merely an engineering preference. Where task information is stored determines acquisition cost, persistence, and reset behaviour, and whether a retained correction keeps working on the next attempt depends on whether it lives in context, in an external artifact, or in weights.

Two research threads support the deliberately broad notion of context used here. The first is one-shot imitation, which infers intended behaviour from a demonstration. The second is meta-reinforcement learning, which infers a task or a dynamics model from outcomes. Sequence models retain trajectories and learning histories; multimodal prompts join a task specification to sensorimotor examples. As the prior widens, teaching can direct a generalist's actions, predict futures, or emit executable programs, while physical experiments supply evidence for correcting execution. Language participates throughout, as instructions, examples, corrections, and retained summaries.

Existing surveys each organize a neighbouring slice of this literature: demonstration learning by teaching interface and learned policy; ICL and in-context RL by example and interaction; human-video-to-control transfer by the information a video carries; manipulation ICL by content, inference target, adaptation mechanism, and transfer; VLA integrations by monolithic versus hierarchical design. This survey's organizing question is different. It asks how new evidence resolves what existing competence leaves undetermined, and how that resolution survives physical execution. The four families place that burden on different intermediate quantities, and correspondence, training, and memory decide whether those intermediates can retain the information an application actually needs.

Three contributions follow. First, a taxonomy of four families plus the correspondence and memory mechanisms they share. Second, an account of how training relationships establish context use, and how object substitution, unfamiliar environments, and execution conditions bound transfer, which is what separates broader motor competence from a stronger ability to learn from teaching. Third, a synthesis of reported comparisons and evaluation controls that isolates context dependence, transfer, and the benefit of retained experience as three distinct claims, and from there motivates compositional learning and improved teachability.

Preliminaries: Six Learning Horizons and One Shared Interface

The authors organize the relation between prior competence, contextual adaptation, and learning from retained experience using six learning horizons. S1, explicit control programming, encodes task logic, control laws, and planning models. S2, neural policy learning, acquires reusable behaviour through imitation and reinforcement. S3, history-based adaptation, infers task or dynamics information from interaction memory. S4, in-context task learning, infers new task requirements from supplied teaching such as instructions, demonstrations, textual examples, and corrections. S3 and S4 are the two complementary evidence sources of ICL, and both hold neural parameters fixed during deployment. S5, physical recursive self-improvement, means experience improves how the next task is learned; S6, collective knowledge evolution, extends that learning loop across a group of robots.

Formally, at decision step $t$ of the current attempt there is a history $h_t=(o_1,a_1,\ldots,a_{t-1},o_t)$, where an observation $o_t$ may carry images, proprioception, force, tactile or inertial measurements, and feedback messages. The task-relevant context $C_t$ may carry an instruction, a goal reference, multimodal support demonstrations $D^{\mathrm s}$, textual input-output examples, corrections, retrieved segments, or an interaction memory. The policy proposes an action block $A_t$ of length $H$. With deployment parameters $\theta$:

$$A_t \sim \pi_\theta(\,\cdot \mid h_t, C_t). \tag{1}$$

Only actions that are actually executed enter the subsequent history, and a new observation can supersede the unexecuted remainder of an action block. When $C_t$ itself carries state that later decisions need, a memory update folds in an action together with its observed consequence:

$$C_{t+1}=U_\theta(C_t,o_t,a_t,o_{t+1}). \tag{2}$$

The defining constraint of ICL is that parameters do not move at deployment:

$$\theta_{t+1}=\theta_t=\theta_0,\qquad h_{t+1}=(h_t,a_t,o_{t+1}). \tag{3}$$

Equation (3) is the watershed of the whole survey. Any method that changes $\theta$ during deployment, including online fine-tuning and meta-learned fast-weight updates, is not strictly ICL and appears here only as a contrast case. Figure 2 draws exactly that contrast: fine-tuning uses task data to move $\theta$ to $\theta_{\mathrm{task}}$, whereas ICL supplies the support demonstration $D^{\mathrm s}$ as $C_t$ with $\theta$ fixed, and both then predict the next action block from the observation-action history.

Figure 2: fine-tuning versus in-context learning. (a) Task data updates $\theta$ into $\theta_{\mathrm{task}}$. (b) A demonstration supplies $C_t=D^{\mathrm s}$ with $\theta$ frozen. Both routes predict the next action block $A_t$ from the observation-action history $h_t$.

Adaptation can live in three kinds of places: context state, retained external artifacts, or neural parameters. Figure 3 lays the five locations and their timescales side by side. In (a) to (c), supplied context, recurrent state, and retrieved segments condition a policy whose parameters are frozen. In (d), fast weights change while the base stays put. In (e), updated weights persist into later tasks. Blue frames and snowflake marks denote frozen parameters; red frames denote updated ones.

Figure 3: five locations and timescales of adaptation. (a) to (c) context, recurrent state, and retrieved segments condition the policy under frozen parameters; (d) fast weights update while the base is unchanged; (e) updated weights are retained for subsequent tasks.

Horizon	What changes in the system	Evidence source	Representative mechanisms
S1 explicit control programming	task logic / control laws / planning model	manual encoding	STRIPS, subsumption, operational space control, behaviour trees
S2 neural policy learning	policy weights	offline training data	visuomotor policies, imitation and reinforcement learning
S3 history-based adaptation	context / recurrent state	interaction outcomes	meta-RL, prompted decision transformers
S4 in-context task learning	context / retrieved artifacts	teaching: instructions, demonstrations, examples, corrections	VIMA, Vid2Robot, UniSkill
S5 physical recursive self-improvement	the acquisition process for later learning	cross-session experience	LMPC successor training, program revision, autonomous practice
S6 collective knowledge evolution	knowledge shared across embodiments	validated exchange between robots	shared progress representations, morphology-aware transfer, pooled policies
Common ground of S3 and S4	$\theta$ frozen at deployment	—	the constraint in Equation (3)

Methods: Four Interfaces from Context to Action

The organizing axis is not which modality a method uses, but which intermediate quantity turns context into action. Figure 4 shows the four technical paradigms: (a) direct action inference, (b) geometric motion-reference transfer, (c) prediction-conditioned control, and (d) skill or tool execution. The representative implementations pictured are Vid2Robot, SemAnCorr, Zero-WAM, and UniSkill. Across the 260 method-class references in the corpus, the four families hold 123, 24, 22, and 91 entries respectively.

Figure 4: how the four technical paradigms turn context into action. (a) Context-conditioned policy: cross-attention plus an action decoder. (b) Geometric demonstration transfer: surface correspondence, contact-frame transfer, motion executor. (c) World-model control: a video model predicts the robot's future, which an action model then consumes. (d) Skill and agent execution: an inverse skill model produces a skill sequence handed to a robot policy.

Family One: Context-Conditioned Policies (123 references)

This family encodes context straight into the conditioning of the policy. The generic form first compresses history and context into a task representation with an encoder, then decodes that representation with an action head:

$$r_t=E_\theta^{\mathrm c}(h_t,C_t),\qquad \pi_\theta(A_t\mid h_t,C_t)=q_\theta^{\mathrm a}(A_t\mid h_t,r_t). \tag{4}$$

Retrieval variants write the action distribution as a mixture over candidate action blocks:

$$\pi_\theta(\,\cdot\mid h_t,C_t)=\sum_{j\in J_t}\beta_{tj}\,\delta_{\bar A_j},\qquad \beta_{tj}\geq 0,\ \sum_{j\in J_t}\beta_{tj}=1. \tag{5}$$

Here $\bar A_j$ is a retrieved candidate action block and $\beta_{tj}$ its mixture weight. Attention variants weight the support demonstrations directly:

$$\alpha_{ti}=\frac{\exp s_\theta(h_t,d_i^{\mathrm s})}{\sum_{k=1}^{L}\exp s_\theta(h_t,d_k^{\mathrm s})},\qquad r_t=\sum_{i=1}^{L}\alpha_{ti}\,e_\theta(d_i^{\mathrm s}). \tag{6}$$

Equation (6) exposes the family's central assumption: the similarity $s_\theta$ must rank demonstrations that are close in task semantics, not close in scene appearance, to the front. The LIBERO-CF finding lands precisely here. When the scene by itself reveals task identity, tested VLAs tend to keep performing the scene-associated task rather than the one the teaching specifies. For demonstration-based ICL this means a substituted video has to change the intended order or relation, not merely perturb appearance, or the test cannot separate reading the teaching from recognizing the scene.

The same family also carries the navigation cases, which shows how wide the interface is. NOLO combines a scene video with optical-flow-derived action labels, the current view, and a goal image to predict actions, training the interpreter offline with reinforcement learning and then reading a new scene video at deployment with weights fixed. The computational interface is still contextual action inference; only the content of the context changes, from a route to an environment.

Family Two: Geometric Demonstration Transfer (24 references)

This family does not learn a semantic representation at all. It treats the demonstration as a retargetable geometric reference: extract a motion reference from the demonstration, retarget it against the current observation and body, then execute it with a controller.

$$\zeta_{1:L}=E_\theta^{\mathrm m}(D^{\mathrm s}),\qquad g=\operatorname{Retarget}(\zeta_{1:L},o_t,\mathcal B),\qquad A_t=\kappa(g,o_t). \tag{7}$$

The rigid relative transform is the classical building block here, converting an end-effector trajectory expressed in the demonstration frame into the frame of the target object:

$${}^{W}T_{E,b}^{\mathrm q}(\sigma)={}^{W}T_{O_b}^{\mathrm q}\bigl({}^{W}T_{O_b}^{\mathrm s}\bigr)^{-1}\,{}^{W}T_{E,b}^{\mathrm s}(\sigma). \tag{8}$$

Visual servoing then closes a geometric error into a velocity command:

$$\boldsymbol e_t=\boldsymbol s_t-\boldsymbol s_t^\star,\qquad \dot{\boldsymbol q}_t=-k_{\mathrm{vs}}\,J_{\mathrm{vis},t}^{\dagger}\,\boldsymbol e_t. \tag{9}$$

The transfer assumption of this family is the most explicit and the most brittle of the four: interaction geometry must be available and correspondable. A handleless vessel can satisfy the goal of pouring while being unable to preserve an explicitly required handle grasp. Geometric transfer is therefore good at exposing contact and motion constraints and poor at expressing order and preconditions. The part-based transfer study quantifies the granularity question directly: switching from whole-object warping to part-wise warping, with cross-category part correspondences supplied, moves mug-to-rack placement from 11/27 to 23/27 successful object-pair evaluations, using one demonstrated end-effector transform with four RGB-D views.

Geometric one-shot methods also separate initial alignment from recovery during execution, which matters for reuse. DOME and ODIL first establish a relative starting configuration, while staged alignment can re-establish it at later subtask boundaries. A transport segment additionally depends on the object remaining grasped: after a slip, geometric alignment can persist while the segment's entry condition fails. Entry condition and intended completion therefore carry different information, the first establishing whether a recorded motion can begin and the second whether the next segment should follow.

Family Three: World-Model-Based Control (22 references)

This family predicts consequences first and acts on them second. Prediction and action are decoupled into two distributions:

$$v^+\sim p_\theta^{\mathrm v}(\,\cdot\mid C_t,h_t),\qquad A_t\sim q_\theta^{\mathrm v}(\,\cdot\mid v^+,C_t,h_t). \tag{10}$$

Marginalizing over the prediction yields an end-to-end policy:

$$\pi_\theta(A_t\mid h_t,C_t)=\int q_\theta^{\mathrm v}(A_t\mid v^+,C_t,h_t)\,p_\theta^{\mathrm v}(v^+\mid C_t,h_t)\,\mathrm dv^+. \tag{11}$$

Model-predictive variants instead maximize expected return over candidate actions:

$$A_t^\star\in\arg\max_{\widehat A_t\in\mathcal A_t}\ \mathbb E_{v^+\sim p_\theta^{\mathrm d}(\cdot\mid h_t,\widehat A_t,C_t)}\bigl[\mathcal R(v^+,C_t)\bigr]. \tag{12}$$

Training typically uses a joint objective of action loss plus prediction loss:

$$\theta^\star\in\arg\min_\theta\bigl[\mathcal L_a(\theta)+\lambda\,\mathcal L_v(\theta)\bigr]. \tag{13}$$

The payoff of this family depends on whether predictions are realizable: a predicted future must lie inside the reachable set of the executor. Zero-WAM makes that interface explicit by training the video and action branches, freezing both, and then replacing recorded future video with generated video, which pins down what prediction owes to execution. Demo-JEPA adds a nuance worth keeping, because the preferred realization changes with task familiarity: an action head wins on familiar real-world tasks while latent planning wins on unseen-task averages in both simulation and the real domain, and the simulation table does not show a familiar-task head advantage at all.

Family Four: Skill- and Agent-Based Execution (91 references)

This family translates context into an executable specification and hands that specification to an existing executor:

$$g=f_\theta(C_t,h_t),\qquad A_t=\kappa(g,o_t). \tag{14}$$

The specification $g$ can be a skill sequence or a synthesized program. Skill composition takes $g=(z_1,\ldots,z_{N_g})$ and executes $\kappa(z_{\ell_t},o_t)$ step by step; program synthesis takes $g=f_\theta(C_t,h_t)$ and runs $\operatorname{Run}(g,h_t)$. GPT-Policy is a concrete instance of this interface: a general VLM receives teaching, requests Cartesian targets or gripper commands through fixed robot tools, and reads back execution feedback. Its plug-reinsertion experiment quantifies the value of correspondence bluntly. Video alone succeeds in 0 of 3 trials; the same video augmented with aligned end-effector poses, gripper states, and commands succeeds in 2 of 3. Same backbone, same task, only the correspondence content of the evidence changed.

The agentic branch also revises its own artifacts rather than only selecting among them. Textual feedback can revise a plan, intermediate observations can revise a plan and a Python controller separately, and program checks can make some responses executable without another model call. Constraint violations narrow revisions to code and sampling bounds while primitive-level errors narrow them to control settings. Those interfaces determine the available edit space, which is a different question from which edit an observed failure actually justifies.

A plausible procedure can still request an action the current scene cannot support. Skill-affordance estimates address that gap before execution by coupling linguistic relevance to physical feasibility, and scene or execution feedback addresses it after a transition by revising what should happen next. When feasibility depends on an unknown property, targeted measurements such as proprioceptive estimates of mass or stiffness can change the manipulation strategy. These are complementary uses of evidence: predict applicability, observe the outcome, then acquire whatever information neither supplied.

Cross-Cutting Mechanisms: Correspondence, Memory, and External Revision

Correspondence and memory run across all four families. Memory assembles short-term state, a long-term bank, a retrieval result, and the task context into what the policy actually sees:

$$m_{t+1}=u_\theta(m_t,x_t),\quad M_{t+1}=w_\theta(M_t,x_t),\quad R_t=\rho_\theta(h_t,M_t),\quad C_t=b_\theta(C_t^{\mathrm{task}},m_t,R_t). \tag{15}$$

Budgeted retrieval decides which segments are allowed into the context at all:

$$J_t\in\arg\max_{J\subseteq\{1,\ldots,N_t\},\,|J|\leq K}\ \sum_{j\in J}\bigl[s_\theta(h_t,\xi_j)-\eta\bigr],\qquad R_t=(\xi_j)_{j\in J_t}. \tag{16}$$

Here $\eta$ is the per-item cost and $K$ the budget. External knowledge revision then writes candidate-artifact-plus-validation as a single guarded update:

$$\widetilde B_{n+1}\sim F_\theta(\cdot\mid B_n,\mathcal T_n),\quad B_{n+1}=\begin{cases}\widetilde B_{n+1},&\chi_n=1\\ B_n,&\chi_n=0\end{cases},\quad \theta=\theta_0. \tag{17}$$

The load-bearing symbol in Equation (17) is $\chi_n$: a revision committed without verification can be worse than no revision at all. FARE's always-search ablation is the empirical version of that warning, since invoking recovery search at every step scores 91.0% on RoboTwin Easy, below both the full selective system at 93.2% and the base at 91.5%, with the base checkpoint frozen and only the search computation differing. RAPID points the same way from the other side: switching feasible-scene-variant verification on raises mean success from 53.2% to 75.9% across eight non-prehensile simulation tasks with 50 novel scenes each and three trials.

RoboRSI shows the same distinction inside one system. It organizes diagnosis and revision through a task-skill tree and retains successful workflows as callable code; its code-on versus code-off comparison reports 174/600 against 129/600 successes over 120 tasks with five layouts each. That tests retained routines. Its separate policy-training case consolidates corrective trajectories into neural parameters instead, and the retained object is what distinguishes the two forms of reuse.

Figure 5: the complete robot ICL workflow. Prior training supplies competence; at deployment evidence is acquired, context is constructed, and actions are generated through the four method families. Outcome assessment feeds execution online, and retained experience feeds back into later context construction.

flowchart TD
  EV[Deployment evidence
instruction / demo D^s / correction / interaction history] --> CC[Context construction C_t
retrieval + memory update U_theta]
  CC --> F1[Family 1 context-conditioned policy
r_t then action head q^a]
  CC --> F2[Family 2 geometric transfer
zeta then Retarget then kappa]
  CC --> F3[Family 3 world-model control
predict v+ then act, or MPC argmax]
  CC --> F4[Family 4 skill / agent execution
g then kappa or Run]
  F1 --> AT[Action block A_t
theta frozen at deployment]
  F2 --> AT
  F3 --> AT
  F4 --> AT
  AT --> OA[Outcome assessment
success / contact / progress]
  OA -->|online feedback| CC
  OA --> MEM[Retained experience
m and M and artifacts B_n]
  MEM -->|later context construction| CC

Figure 6: the four context-to-action interfaces and their feedback loops, drawn from Figures 5 and 4. $\theta$ stays frozen at deployment; outcome assessment both corrects execution online and writes reusable experience back into context construction.

Data: Where Context Comes From

Context does not appear out of nowhere; it is manufactured by a training relationship. The paired objective binds a support demonstration and a query trajectory to the same task specification:

$$\mathcal L(\theta)=\mathbb E_{(D^{\mathrm s},\tau^{\mathrm q})\sim P_{\mathrm{pair}}}\Bigl[\frac{1}{|I^{\mathrm q}|}\sum_{t\in I^{\mathrm q}}\ell_\theta\bigl(A_t^{\mathrm q};h_t^{\mathrm q},D^{\mathrm s}\bigr)\Bigr]. \tag{18}$$

The practical reading of Equation (18) is that one should count distinguishable teaching problems rather than trajectories. If the current scene already determines every training action, the loss is minimized without ever reading the demonstration; only similar scenes with different valid procedures make teaching predictive. BPP's controlled allocation experiment quantifies the trade-off between diversity and quantity: holding 10,000 drawing demonstrations fixed, mean Chamfer error is 9.5 pixels for 500 tasks with 20 demonstrations each, 5.1 pixels for 1,000 tasks with ten each, and 3.4 pixels for 2,000 tasks with five each. More tasks with fewer demonstrations each learned better, because each example had to resolve more.

Scale tells a complementary story. AgiBot World's March 2025 release spans 1,001,552 trajectories, 2,976.4 hours, 217 tasks, and 106 scenes, and it includes subtask and recovery records. Those annotations matter for ICL specifically, because linking an attempt to its correction preserves the evidence that makes a different continuation appropriate. EgoScale connects 20,854 hours of action-labeled human data to robot behaviour through aligned mid-training and downstream post-training, and EgoDex contributes 829 hours of contemporaneous hand and finger tracking that exposes motion otherwise requiring reconstruction from RGB.

Two further resources bracket the acquisition-cost question. RoboCasa365 adds 365 tasks and 2,500 kitchen scenes with 612 hours of human-operated demonstrations in simulation plus 1,615 hours generated with MimicGen, which lets a procedure be held fixed across environments or varied within a matched environment. XR-2 mixes 531.7 hours of robot teleoperation with roughly 1,000 hours of dual-UMI demonstrations, and by separating policy actions from human takeovers it improves cloth folding from 58% to 93% across three retraining rounds. That separation expands the repairs represented in weights, which is exactly what successful demonstrations alone leave underrepresented.

S1's scaling experiment supplies the caution that belongs next to all of this. Across 1k to 100k training hours, with data, compute, and architecture matched and only the prompt embedding varied, demonstration prompting on unseen tasks rises from 1% to 66% while language prompting rises from 0% to 9%. Similar seen-task performance can therefore conceal entirely different returns to new-task teaching. The cumulative per-step metric also includes human recovery, mainly for the language baseline, so the gap is not an artifact of a stricter ruler.

Resource or experiment	Scale or contrast	Key reading
AgiBot World (2025-03)	trajectories / hours / tasks / scenes	1,001,552 / 2,976.4 / 217 / 106
EgoScale	action-labeled human data	20,854 hours
EgoDex	hand-object tracking	829 hours
RoboCasa365	tasks / kitchen scenes / human demos / MimicGen	365 / 2,500 / 612 h / 1,615 h
XR-2	teleoperation plus dual-UMI; three retraining rounds on folding	531.7 h + about 1,000 h; 58% to 93%
BPP diversity allocation	500 / 1,000 / 2,000 tasks at 10,000 demonstrations total	Chamfer 9.5 / 5.1 / 3.4 pixels
S1 scaling, unseen tasks	demonstration versus language prompting, 1k to 100k h	1% to 66% versus 0% to 9%
GPT-Policy plug reinsertion	video only versus video plus aligned state and action	0/3 versus 2/3
RoboRSI code ablation	retained routines on versus off	174/600 versus 129/600 over 120 tasks

Applications: What Must Context Resolve

The applications are organized by what context has to resolve rather than by which venue it appears in. In manipulation the uncertainty is that the intended interaction is unspecified: a demonstration may have to convey an object-purpose relation, an order, or a contact. In navigation the uncertainty is that the route or the local spatial structure is hidden. In long-horizon work the uncertainty is that a completed prerequisite is hidden. With an unfamiliar body the uncertainty is that its response to action is unknown. Any of the four families can serve several of these demands, provided its intermediate preserves what the application requires.

On the manipulation side, the value of a demonstration depends on what the user needs to specify. A final drawing leaves stroke order, direction, and tool approach unresolved, and BPP's DrawAnything study connects broader drawing-task diversity to better demonstration-conditioned generalization. In its three-task laundry study, by contrast, language is more reliable than behaviour demonstrations, and some demonstration-conditioned failures execute the wrong fold. Those two settings together identify an application condition rather than a ranking: video is especially useful when it communicates a choice that a short task label or a goal image leaves open. A goal image can specify the task when only a visible endpoint matters; a demonstration additionally conveys a required path or order.

Personalized behaviour adds persistent conventions to the specification. A request to place objects quietly should govern later placements, whereas a correction to one object's location should not. Such applications require both an acceptable endpoint and adherence to a continuing constraint, which is a different failure surface from getting the motion right once. Replacing objects makes the same distinction concrete: a new workpiece can preserve an assembly relation while changing its grasp, a new fixture can preserve the goal while changing insertion geometry, and replacing both requires correspondence at both ends of the interaction.

On the navigation side the authors sort evidence into four context types: route demonstrations for where to go in what order, environment observations for where places and objects are, instruction-decision examples for how to interpret a request, and outcome feedback for revising the next attempt. The four retain different things and fail at different boundaries. A new goal can reuse a map but replaces a route; a new building invalidates local positions while interpretation conventions remain useful; changed terrain can invalidate a previous traversability estimate. NOLO's context-removal experiment and ReLIC's cross-episode attention comparison probe the scene-preview and within-environment-reuse contributions respectively, and ReLIC's own numbers show success rising from 23% on the first navigation episode to 43% on the fifteenth in the same home layout across changing goals and start positions.

Long-horizon work turns the teaching problem from choosing an action into maintaining a dependency structure. An unfinished step, a completed step, and a failed precondition can demand different continuations even when the current image looks the same. Irreversibility constrains recovery further: repeating an insertion and repeating a dose are not the same mistake, so the informative next interaction must respect both the taught dependency structure and the state of intermediate products. Embodiment and dynamics adaptation isolates the physical response instead, where a direction or velocity command already specifies the goal and recent action consequences reveal how this body and this terrain currently respond.

Task changeover combines these demands with acquisition and reuse cost. Two ingredient orders can share parts and grasps yet require different placement sequences; across batches the chosen order persists while completed-step memory resets; changing a tool demands a new physical estimate while the sequence may remain valid. Changeover therefore measures the whole acquisition process including the evidence needed to validate it, which is why the survey treats it as an application rather than as a benchmark footnote.

Grounding and Recovery: When Context Stops Applying

Objects move, grasps fail, and resistance changes after a task has been interpreted correctly. Recovery aims to preserve the taught requirement while revising the state estimate and the continuation. The authors separate three failures that call for three different kinds of evidence: visual occlusion needs a view that restores the missing geometry, a lost grasp needs contact checked before transport resumes, and a wrong target needs the instruction-demonstration correspondence revisited. Each repair addresses a different missing fact. Contact-intensive tasks add a further ambiguity, since a stalled insertion can reflect misalignment, unfamiliar resistance, or excessive force; a bounded probe can distinguish physical responses, whereas an irreversible operation may require inspection or clarification before another attempt.

Recovery proceeds from assessment to diagnosis to correction. Assessment needs a reference for what should have happened, and for ICL that reference comes from the current teaching, which lets an unfamiliar but valid order define expected progress. Progress and physical acceptability can still disagree: a connector can reach its apparent destination with an unsuitable contact force, just as a kit can reach the right inventory in the wrong order. Diagnosis then turns failure detection into a repair choice, because misalignment and an unseen obstruction need different interventions; uncertainty about intent is reduced by clarification and reasoning over alternative plans, while uncertainty about resistance needs sensing or interaction, including tactile evidence absent from RGB.

Correction granularity should match the cause. A wrong target requires changing task selection; a displaced approach point requires a spatial update; insufficient displacement may require only a magnitude adjustment. Iterative in-context policy improvement links an observed error directly to a motion primitive's settings, with a frozen language model proposing an adjustment from primitive parameters, execution errors, and correction examples, and favouring relative errors over raw states and targets. Neural parameters stay fixed while the generated control settings change within the primitive's available range. When a frozen action policy simply cannot continue from the failure state, CoRe uses history and counterfactual continuations to choose a rejoin state and realigns displaced objects before returning control.

Recovery can instead revise the history that conditions prediction. FARE compares a full causal cache, a recovered prefix, and a reset cache under the same latest observation, selecting their predicted continuations with a frozen world-action model. This changes the model's retained evidence rather than physically restoring the scene, so a compatible historical prefix must still be re-grounded in the current state. Its ablations find that always invoking recovery search, or committing an unverified prefix, can underperform the base policy, which is the direct support for selective intervention with verification.

Reuse retains a conditional lesson: which intervention corrected which failure, and under what conditions. Different lessons require different reset rules. Language-correction memory retains task constraints whereas action-effect history retains physical response; a workcell reset clears completed-step memory, while a changed connector or tool can invalidate a response estimate and leave the assembly order intact. For a jammed connector, an adjustment justified by measured resistance should not be reused when misalignment is the cause. Retaining the observation, intervention, and outcome lets later attempts reassess that condition instead of replaying the repair unconditionally.

Evaluation: When Is Context Actually Useful

Evaluation follows the contribution chain from a change in evidence to a physical success. A context-intervention test probes sensitivity to teaching: with the checkpoint, the scene, and the executor held fixed, the examples are varied and one asks whether the inferred task requirement moves the behaviour. Clearing the history under matched physical states instead probes access to prior evidence. Intermediate interventions localize a failure along the four interfaces: supply a relevant demonstration segment to a context-conditioned policy, a correct correspondence and reference to a geometric transfer pipeline, a task-consistent future to a predictive controller, and a correct executable specification to an agent.

The gain that accrues across attempts is measured by a single controlled difference. For episodic-memory adaptation, the reuse gain on attempt $n$ is:

$$\Delta_n^{\mathrm{reuse}}=S_n^{\mathrm{keep}}-S_n^{\mathrm{reset}}. \tag{19}$$

Here $S_n$ is the same pre-declared success or compliance rate and the superscript says whether memory was retained or cleared. The comparison holds the task distribution, the physical initialization, the executor, any other retained artifact, the number of prior attempts, and the execution budget constant, varying only access to the evidence those attempts produced. A per-attempt ratio isolates the benefit of retained evidence, whereas a cumulative success count also profits from independent retries; the two must not be mixed.

Supplement S1 records fourteen reported comparisons item by item, and they are the densest factual layer of the survey. They fall into three groups. The first is training and task evidence: an ICRT variant trained on a DROID subset alone makes no progress on the held-out test tasks; LMAct's sweep from 0 to 512 expert segments often buys very little; Show-Harness, given arbitrary action names and no written convention, succeeds on 1 of 20 episodes and maps actions correctly 23.3% of the time; and removing NOLO's scene-preview video drops Habitat success from 43.65%±1.09% to 33.58%±0.92% over three seeds. The second group covers transfer and physical realization: replacing a whole-body warp with part-level deformation raises successful object pairs on the cup-and-rack task from 11/27 to 23/27, and switching on RAPID's feasible-scene-variant verification lifts mean success from 53.2% to 75.9%. The third covers retained experience and subsequent learning: Zeva loses 10 to 20 points across five real tasks when its persistent interaction memory is reduced to a short trace; RTCF moves from a frozen base at 86.4% to memory-aligned correction at 88.4%; TraceFlow goes from 21/50 to 39/50 on a single real ordered-palletizing task; FARE's base sits at 91.5% and the full system at 93.2%, while an always-search ablation reaches only 91.0%; and LMPC's trained successor raises test success from 39.4% to 66.3% while mean dialogue turns in successful sessions fall from 2.4 to 1.9.

Figure 7: two tests of contextual transfer. (a) Holding the robot scene fixed and varying only the demonstration changes the continuation UniSkill predicts. (b) A LIBERO-PRO object swap tests goal selection. Together the two axes demand a new test design: vary teaching and object identity independently while measuring both the inferred task requirement and its physical realization.

Study	Control	Reading	Setting
ICRT	DROID-only versus multi-task training	DROID-only: no progress	shared test tasks
LMAct	0 to 512 expert segments	often a very small gain	game and simulated control
Show-Harness	arbitrary names, no convention	1/20 success; 23.3% mapping	fixed harness
NOLO	no scene video to scene video	SR 33.58 to 43.65	Habitat; 3 seeds
Part-level transfer	whole-body to part deformation	11/27 to 23/27	cup-and-rack; part hints given by a human
RAPID	scene-variant verification off to on	SR 53.2 to 75.9	8 tasks, 50 scenes, 3 trials each
Zeva	persistent memory off to on	+10 to 20 points	5 real tasks
RTCF	memory correction off to on	SR 86.4 to 88.4	fixed library; 2,000 segments per variant
TraceFlow	base to trajectory-guided	ordered 21/50 to 39/50	1 real palletizing task
FARE	base to selective revision; always-search	91.5 to 93.2; always-search 91.0	RoboTwin Easy; base checkpoint frozen
LMPC	base to successor	SR 39.4 to 66.3; turns 2.4 to 1.9	held-out teaching tasks

flowchart LR
  A[Change teaching
same scene and executor] -->|does behavior follow evidence| B[Context dependence]
  B --> C[Substitute objects / environments / bodies] --> D[Transfer boundary]
  D --> E[Clear vs retain memory
matched physical resets] --> F[Retained-experience gain]
  F --> G[Successor comparison
same new teaching] --> H[Improved teachability S5]

Figure 8: the evaluation control ladder, drawn from Section 7. From being sensitive to teaching, to transferring physically, to gaining from retained experience, to teaching a successor better, each rung needs a different intervention and a different reading.

Three patterns matter more than any single entry. Large effects cluster where context supplies something the weights genuinely lack: grounding an action name in a perception stack, a scene the policy cannot see, a feasible-variant check it cannot infer, or an object that has to be warped into the current frame. Small effects cluster where context is redundant with pretraining, as in ICRT's DROID-only variant and LMAct's length sweep. And the single rung that reaches improved teachability is also the one with the cleanest control, since LMPC varies only what the successor was trained on and reports turns alongside success. Attribution remains the recurring weakness: several studies change more than one thing between conditions, so a gain cannot be assigned to context with confidence, and the survey flags this per row instead of in one global caveat.

Corpus Census: The Evidence Base Behind the Survey

The survey codes all 412 numbered references item by item, of which 260 are method-class. Within that method subset the four families divide as 123 context-conditioned policies, 24 geometric demonstration transfer, 22 world-model control, and 91 skill- and agent-based execution. The remaining 152 non-method references carry the roles of data resource (44), evaluation (39), foundations and prior surveys (40), and training and representation (29). The temporal distribution is strongly right-skewed: 24 references from 2022, 33 from 2023, 47 from 2024, 53 from 2025, and 197 from a partial 2026 counted through 2026-09-29, so 72.1% of the corpus was published in 2024 or later. That says both how young the field is and that every family share will move at the next refresh.

Figure 9: composition of the surveyed literature. (a) First publication year for all 412 references. (b) The four-family split of the 260 method-class references. The first bin reaches back to 2015; the dagger marks the partial year as of 2026-09-29.

Role	References	Share of 412
Context-conditioned policy	123	29.9%
Skill and agent execution	91	22.1%
Geometric demonstration transfer	24	5.8%
World-model control	22	5.3%
Data resource	44	10.7%
Foundations and surveys	40	9.7%
Evaluation	39	9.5%
Training and representation	29	7.0%

Coding was done by the authors against a fixed protocol, and the branch-level placements that a reader might contest are disclosed rather than buried: ReCAP's Cosmos Policy branch is coded as predictive world-action modelling, ENPIRE as program revision, and GPT-Policy as in-context tool use. Listing those decisions is what makes the census reproducible, since a different rule can be substituted and the shift measured.

Limitations

The first class of limitation is the authors' own and concerns corpus freshness and coding granularity. With 72.1% of references from 2024 onward and 197 from a year that has not finished, both the family shares and the annual curve move with every refresh. Branch-level coding also forces single assignments onto systems that span categories, and those assignments are themselves arguable judgements rather than measurements.

The second class comes from the evidence. All fourteen reported comparisons are within-study: conditions differ across studies and the readings cannot be added sideways into a ranking. Several lack any uncertainty, FARE's three success rates among them. LMAct's flat curve is compatible with early saturation, weak context use, or insufficient control, and the curve alone cannot distinguish the three. That is why the evaluation chapter offers proposed controls rather than a cross-study ranking, and why the confidence of each row stays visible next to its number.

The third is our own reading. Organizing by interface costs depth on individual systems: a reader who wants one VLA's absolute numbers on one benchmark still has to go back to the source paper. And within the six horizons, the evidence for S5 and S6 rests on a handful of works such as LMPC, RoboRSI, and REVOLVE. Physical recursive self-improvement is at present closer to an agenda than to a reproduced phenomenon, and the survey is right to present it that way.

Future Directions and Conclusion

The research agenda has four linked objectives: infer unfamiliar requirements, retain binding physical constraints through abstraction, revise retained corrections when their conditions change, and improve the way subsequent tasks get learned. Figure 10 draws them as inferring a taught relation, composing task steps, preserving function under physical change, and using corrective feedback. The authors single out one testable hypothesis: coverage of task relations and coverage of executable motions do not contribute equally to contextual transfer. Broader pretraining may support unfamiliar task requirements while grounded paired training builds precise action correspondence, and the two can be separated by varying pretraining scale and explicit pairing independently while holding the executor and the task information fixed.

Figure 10: the four research directions. (a) Inferring the taught relation. (b) Composing task steps. (c) Preserving function under physical change. (d) Using corrective feedback. Together they push in-context learning toward interpreting, transferring, and retaining what a task requires.

A kitting task makes the hypothesis observable in three ordered cases. First, the plate-first order and the required quantities are given, so only realization is tested, and nearly everything in the corpus solves it. Second, the destination changes to tray B mid-episode, which separates a system that re-binds a relation from one that replays a trajectory. Third, the instruction is ambiguous about which tray, which separates a system that knows it lacks information from one that does not. Case one is saturated, case two is where interpretation shows up, and case three is where the field has almost no measurements at all.

Over longer learning cycles the criterion for S5 is what a successor learns from new teaching, not whether the current solver got stronger. LMPC's cross-session successor training, raising test success from 39.4% to 66.3%, is the representative evidence so far; TraceFlow's stacking gain peaks before its final round and HarnessPAI's public sequence stops at policy refinement, and both show that one stronger round is not the same as a working loop. S6, collective knowledge evolution, additionally requires two-way metric transfer: an arm demonstrates a placement relation, a humanoid realizes it under different kinematic constraints, and the receiver returns a validated execution or a correction. The exchanged lesson has to preserve task order and object relations while allowing grasp and balance control to change.

Summary and Outlook

Robot in-context learning addresses the gap between possessing reusable competence and knowing how to apply it in the present task, with teaching and interaction closing that gap while deployment parameters stay frozen. Four interfaces carry information into action: contextual action inference, geometric reference transfer, consequence prediction, and skill or program execution. Their respective payoffs depend on a compatible representation, available interaction geometry, realizable prediction, and sufficient actuator capability. Correspondence and memory connect evidence across situations and across time, and grounding establishes whether that evidence still applies.

The central synthesis is that in-context learning depends on carrying a task-relevant distinction from evidence all the way to execution. Losing it at selection, at interpretation, at realization, or at reuse each points to a different data, representation, or control capability. The next challenge is to make those dependencies transferable: infer unfamiliar requirement compositions, revise only the constraints that feedback actually touches, and discard experience whose conditions no longer hold. Physical recursive self-improvement and collective knowledge evolution will in the end be judged by what a successor learned and by the total teaching, interaction, and training cost of getting it there.

The value of in-context learning is not in how large the model is, but in whether it can carry a task-relevant distinction from the evidence all the way to execution. Losing it at selection, interpretation, realization, or reuse is four different failures, and each one calls for a different capability.