PAPER DEEP DIVE
RoboFoundry: System-as-Policy Evolution for Self-Learning Embodied Agents
RoboFoundry freezes the foundation model and treats the surrounding context, skill, and semantic-binding system as the evolving policy, allowing embodied agents to diagnose failures, validate persistent repairs, and transfer improvements across benchmarks and robots.
One-sentence summary: RoboFoundry freezes the foundation model and treats the surrounding context, skill, binding, and filesystem machinery as the evolving embodied policy, turning execution failures into validated and persistent system edits.
1. Why a Foundation Model Is Not Yet an Agent
An embodied agent is not defined by a single call to a multimodal model. The executable policy also determines what the model observes, which history survives, how tools are exposed, how skills compose, how failures are recovered, and how semantic decisions are translated into robot-specific control. RoboFoundry starts from the observation that the same foundation model can behave very differently when this surrounding structure changes. The relevant policy is therefore the model-system pair, not the model alone.
Early embodied systems commonly used language models as high-level planners with a fixed action vocabulary. Code-as-policy systems pushed the model closer to the control loop by letting it compose perception and manipulation primitives into executable programs. This expanded the behavioral range, but it also made failure attribution harder. A long-horizon failure may come from a missed object, an incorrect spatial relation, a bad subtask order, missing task state, stale memory, or a semantic goal bound to the wrong robot interface.
Most prior systems optimize one component at a time: a memory module, a context compressor, a skill library, or an action adapter. Each component can improve its local metric while the complete agent remains fragile. For example, a grasp failure may be caused not by the gripper primitive but by an outdated object location retrieved from memory. A navigation plan may be semantically correct but impossible for the current robot because the execution binding ignores reachable floor area.
The paper also rejects the assumption that more interaction automatically produces self-improvement. A system can repeat the same mistake for many trials, or preserve a lucky success with the wrong explanation. Experience becomes capability only when failures are attributed to identifiable operations, edits are tested in subsequent rollouts, and recurring repairs survive evaluation on related tasks. RoboFoundry organizes these requirements into an Act-Reflect-Repair-Promote loop.
This design is aimed at robots that operate over days rather than demos that run once. A deployed system will encounter new rooms, objects, embodiments, and task stages. Rebuilding it through human engineering after every failure does not scale, while direct weight updates can cause forgetting and are difficult to audit. RoboFoundry takes a middle path: the foundation model stays frozen, but readable files, skills, context policies, and bindings can evolve and be rolled back.
2. System-as-Policy Formulation
Let $M$ denote the frozen foundation model and $H$ the supporting system around it. A task instance $x$ is executed under a general system $H_g$ and a task-level system $H_t$. The execution produces a trace containing tool calls, observations, robot states, actions, outcomes, and feedback:
$$\tau\sim\mathcal{T}(H_g,H_t,x)$$
The equation matters because of its dependency structure. The trace depends on both system levels, so a failure can expose a task-local context error as well as a reusable skill or binding defect. Evolution optimizes $H_t$ and $H_g$, not the weights of $M$. The task objective is:
$$H_t^*=\arg\max_{H_t\in S_t}\mathbb{E}_{\tau}\left[r_x(\Delta H_t,\tau)\right]$$
Here $S_t$ is the editable task-system space, $\Delta H_t$ is a candidate edit, and $r_x$ scores the resulting trajectory on task $x$. This objective asks only whether the observed failure was repaired. A repair may overfit one obstacle or one object arrangement, so the paper adds a second objective over the persistent general system:
$$H_g^*=\arg\max_{H_g\in S_g}\mathbb{E}_{x\sim\mathcal{X}_{\mathcal{A}},\,\tau\sim\mathcal{T}(H_g,H_t^*(x),x)}\left[r_x(\Delta H_g,\tau)\right]$$
The distribution $\mathcal{X}_{\mathcal{A}}$ contains tasks associated with a shared capability gap. A general edit must remain useful after task-specific systems have already adapted. This two-level formulation separates “fix this case” from “acquire a reusable capability,” which is the central distinction between RoboFoundry and unconstrained prompt reflection.
3. The Inner and Outer Loops
The inner loop executes the current system. The model receives a task, observations, and selected context; it calls skills and produces actions. The environment returns state changes and feedback, which are logged as a trace. The outer loop diagnoses the failure, proposes an edit, and tests the edit in later rollouts. Successful task edits may remain local. Only edits that explain recurring failures and pass a held-out capability check are promoted to the general system.
RoboFoundry diagnoses failures across six capability categories: perception, reasoning, planning, saving, retrieval, and utilization. These categories correspond to different repair surfaces. A perception failure changes how observations are parsed; a reasoning failure changes relations or constraints; a planning failure changes subtask order; a saving failure changes what evidence enters persistent memory; a retrieval failure changes when that evidence is recalled; and a utilization failure changes how retrieved evidence is presented to the decision model.
flowchart TB
A[Task instance] --> B[Run H_g and H_t]
B --> C[Store execution trace]
C --> D[Attribute failure
perception reason plan
save retrieve utilize]
D --> E[Propose task edit]
E --> F{Held-in repair
succeeds?}
F -->|No| G[Discard and retain score]
F -->|Yes| H[Commit task system]
H --> I[Abstract reusable edit]
I --> J{Held-out transfer
non-negative?}
J -->|No| K[Keep task local]
J -->|Yes| L[Promote to general system]
L --> B
K --> B
This process differs from a generic reflection agent in two ways. First, a candidate is represented as an executable change to a file, context rule, skill, or binding. Second, a candidate has an explicit rejection path. A plausible diagnosis that does not improve execution is discarded instead of being accumulated in a growing prompt. The system therefore learns not only what worked, but also which proposed repairs failed.
Figure 2 shows the architecture. The inner loop combines the general and task systems. The context surface manages active context and persistent filesystem memory. The skill surface contains atomic skills, compositions, and failure-conditioned recovery. Semantic and execution bindings separate cross-embodiment decisions from robot-specific actions. The outer loop converts capability failures into candidates and evaluates them through held-in repair and held-out retention.
Figure 2: RoboFoundry combines inner-loop execution, two system levels, context and skill surfaces, and an outer loop for diagnosis and promotion.
4. Context Evolution: Memory Must Be Used, Not Merely Stored
The context system contains active context and a persistent filesystem memory. Instead of rewriting memory at every step, the system monitors task progress and saves evidence when an event warrants persistence. The context operation space is deliberately small:
$$S_t^{\mathrm{ctx}}=\{\mathrm{SAVE},\mathrm{RETRIEVE},\mathrm{UTILIZE}\}$$
Saving chooses task-relevant evidence such as completed subtasks, object relations, pre-occlusion states, or failure conditions. Retrieval brings evidence back at the appropriate stage. Utilization converts retrieved content into active context:
$$c_t=\mathrm{Utilize}(\mathrm{Retrieve}(x,z_t,\mathcal{M}_t),z_t)$$
Here $z_t$ is the current observation and robot state, $\mathcal{M}_t$ is persistent memory, and $c_t$ is the active context supplied to the model. The resulting semantic decision is:
$$p_t=M(x,z_t,c_t;H_g,H_t^s)$$
This decomposition creates three testable failure modes. If evidence never entered $\mathcal{M}_t$, the saving policy is wrong. If it exists but is not retrieved, the retrieval condition is wrong. If it is present in $c_t$ and the decision is still wrong, the failure is in utilization or reasoning. Diagnosis becomes an operation-level question rather than a vague claim that the robot forgot.
The chemistry task in Figure 6 gives a concrete example. Six dependent stages require a funnel, a dish, taring, powder, a returned spoon, and a final pour. The system records funnel readiness and scale taring during execution, then retrieves that state at the final pouring step. The last image alone does not reveal whether the funnel is ready. The filesystem record carries the prerequisite across the long horizon.
Figure 6: A task-state record connects six chemistry subtasks and retrieves early funnel readiness during the final pouring phase.
5. Skill Evolution and Failure-Conditioned Recovery
The skill surface is hierarchical rather than a flat list of tools. Atomic skills support basic actions such as moving, grasping, placing, and opening. Composite skills encode reusable sequences with semantic boundaries. Recovery skills activate under observed failure conditions, such as a grasp that produces no expected displacement or a target hidden by occlusion.
This hierarchy reduces the search burden on the decision model. For a long task, the model does not need to regenerate every low-level call. It can select a tested composition and switch to a recovery when the expected condition fails. The composition still retains semantic parameters, so the system can bind object identities, target relations, and stage conditions rather than hard-coding one scene.
LIBERO-PRO case studies show the system abstracting repeated manipulation code into helpers for gripping and placing, correcting task-generation templates, and distinguishing object-centered targets from TCP-centered targets. In a cream-cheese recovery example, a tangential push fails; the system checks planar displacement, re-grasps, and then places the object. The recovery remains trial-local, while the shared composite skill is persisted.
The paper also reports a failure of abstraction. In a bowl-related case, shorter execution code does not fix incorrect spatial grounding. This is an important limitation: a cleaner skill interface cannot compensate for a wrong relation between object, support surface, and target position. RoboFoundry therefore scores the outcome instead of treating code reuse itself as evidence of progress.
6. Semantic and Execution Bindings
Cross-embodiment transfer requires different abstractions for task meaning and robot control. “Find drinking water” requires identifying a dispenser and planning an approach, but a wheeled base, quadruped, and humanoid need different locomotion and sensing primitives. If the evolved system directly stores low-level actions for one robot, the resulting knowledge is difficult to transfer. RoboFoundry projects the same task system through two bindings:
$$H_t^s(M)=\mathrm{BIND}_s(H_t(M)),\qquad H_t^e(M)=\mathrm{BIND}_e(H_t(M))$$
The semantic binding $H_t^s$ carries task decomposition, object relations, and embodiment-invariant decisions. The execution binding $H_t^e$ grounds those decisions in a specific robot's kinematics, tools, and action API. Feedback from execution returns to the semantic layer:
$$(z_{t+1},f_t)=\mathrm{Execute}(H_t^e(p_t))$$
The separation supports diagnosis and transfer. A mismatch between the semantic action and the resulting observation can be attributed either to the high-level decision or to the execution binding. An evolved skill can retain its semantic structure while replacing the robot-specific implementation. The Unitree G1 navigation demonstration uses this mechanism to map a semantic water target to a dispenser and then to embodiment-specific navigation actions.
Figure 5: Real-world evolution and transfer across nesting dolls, unseen towel colors, and semantic navigation with Unitree G1.
The towel experiment gives evidence for the semantic part of the claim. RoboFoundry evolves on a blue towel and transfers directly to green and pink towels with different appearance, geometry, and material properties. It exceeds 80% success without further adaptation, while standalone $\pi_{0.5}$ remains below 20%. This supports transfer of the fold structure, but it remains transfer within a related object family rather than arbitrary deformable-object generalization.
7. How Task Repairs Become General Capabilities
A failed trace first produces a candidate task edit. The candidate is deployed in the current task and evaluated on later rollouts. If performance improves, it is committed:
$$H_t^*\leftarrow H_t\oplus\Delta H_t^*$$
The operator $\oplus$ applies an executable change to the system. The edit may alter a task description, a context-saving rule, a retrieval condition, a skill function, a recovery branch, or an execution parameter. The corresponding evidence and outcome are retained so the change can be audited later.
A successful task edit is then abstracted with $\Delta H_g\leftarrow\mathrm{Abstract}(\Delta H_t^*,\tau_x;H_g)$. Abstraction removes task-specific names and extracts a capability-level rule. A red-star insertion repair becomes “keep the gripper closed after grasp until the object enters the opening.” An occlusion repair becomes “treat the current frame as authoritative and history as a reminder.” The abstraction is still a hypothesis and must be tested on other traces.
Figure 3 traces one EB-Habitat spatial run. Held-in success starts at 21.4%. Three committed edits raise it to 50.0%; two candidates are discarded without changing the system; the final candidate reaches 78.0% and passes the promotion check. The rejected candidates matter because autonomous evolution must be able to avoid permanent damage from plausible but ineffective repairs.
Figure 3: Evolution trace on EB-Habitat spatial tasks, with three commits, two discarded candidates, and one promotion after held-out checking.
8. Capability-Conditioned Promotion
For a general candidate, RoboFoundry retrieves successful traces from the same capability category but different task instances:
$$\mathcal{T}_a^{\mathrm{out}}(x)=\{\tau_i\in\mathcal{T}:a_i=a,\ x_i\neq x\}$$
Each trace contributes only the pre-repair task, observations, state, and history. The candidate general rule proposes a new task edit $\Delta H_{t,i}^{\prime}$, which is compared with the recorded successful edit $\Delta H_{t,i}^{*}$. The capability-conditioned transfer score is:
$$\Delta_{\mathrm{cap}}(\Delta H_g)=\frac{1}{|\mathcal{T}_a^{\mathrm{out}}(x)|}\sum_{\tau_i\in\mathcal{T}_a^{\mathrm{out}}(x)}\left[r_x(\Delta H_{t,i}^{\prime},\tau_i)-r_x(\Delta H_{t,i}^{*},\tau_i)\right]$$
The edit is promoted only if $\Delta_{\mathrm{cap}}(\Delta H_g^*)\geq 0$:
$$H_g^*\leftarrow H_g\oplus\Delta H_g^*$$
This gate requires the general rule to preserve or improve existing successful repairs on related traces. It is more meaningful than judging whether a rule sounds general. The zero threshold is conservative in one sense and permissive in another: it rejects harmful transfer, but it may accept a neutral rule and cannot prove robustness outside the available trace distribution.
Figure 4 tests data efficiency. With 50 fixed held-out tasks per suite, one quarter of the held-in data already reaches 78.0% on EB-ALFRED and 66.0% on EB-Habitat. Three quarters of the held-in data brings EB-Habitat to 74.0%. The result suggests that evolution extracts high-value rules from a relatively small set of informative failures instead of requiring large-scale retraining.
Figure 4: Held-in data scaling on EmbodiedBench. A quarter of the data captures much of the eventual improvement.
9. Filesystem Organization and Auditability
RoboFoundry stores the executable support system in files rather than hiding it in model weights or opaque vector state. The organization separates executable skills and task templates from evidence, traces, and trial-local recovery records. The model can inspect files with ordinary read/search operations and modify them through add, modify, and delete operations.
This design makes edits visible. A committed change can be compared with its previous version and linked to the trace that motivated it. Trial-local recovery can be inspected without automatically becoming a general skill. The paper's LIBERO-PRO examples persist task-generation prompts, coordinate semantics, reusable helpers, and startup logic while keeping a specific cream-cheese recovery local.
Auditability does not remove complexity risk. Persisted rules can conflict with later edits, encode a shortcut, or become stale after an environment change. The current paper emphasizes selection and promotion, but production systems would also need dependency checks, regression suites, rollback policies, and periodic cleanup. These concerns are not fully evaluated in the reported experiments.
10. Benchmark Results
EmbodiedBench evaluates ALFRED, Habitat, Navigation, and Manipulation behavior. RoboFoundry improves GPT-5.5 from 56.9% to 72.7%, a gain of 15.8 percentage points. The 27.8% in the abstract is a relative improvement and should not be confused with the absolute point gain. GPT-6 Astra rises from 71.9% to 78.0%, while Qwen3.7-Plus rises from 60.0% to 70.3% and approaches GPT-5.5's 72.7%.
| Configuration | Average success | Point gain | Interpretation |
|---|---|---|---|
| Base GPT-5.5 | 56.9% | — | Foundation model with the standard agent stack |
| RoboFoundry + GPT-5.5 | 72.7% | +15.8 pp | Full two-level evolution |
| RoboFoundry-Lite + GPT-5.5 | 65.5% | +8.6 pp | A reduced system still yields a sizeable gain |
| RoboFoundry + Qwen3.7-Plus | 70.3% | +10.4 pp | Closes much of the gap to the larger model |
| RoboFoundry + GPT-6 Astra | 78.0% | +6.1 pp | Stronger base model leaves less headroom |
RoboMemArena tests transfer, occlusion, counting, and sequence memory. RoboFoundry reaches 53.5% task success and 72.8% condition success, ahead of PrediMem at 38.5%/55.2% and MemER at 27.3%/49.1%. The gap matters because full task success requires completing the whole sequence, not merely recognizing a later stage.
| Method | Task success rate | Conditional success rate |
|---|---|---|
| RoboFoundry | 53.5% | 72.8% |
| PrediMem | 38.5% | 55.2% |
| MemER | 27.3% | 49.1% |
| HiF-VLA | 16.9% | 39.8% |
| $\pi_{0.5}$ | 21.5% | 38.7% |
| MemoryVLA | 15.0% | 35.3% |
Context evolution also improves standalone systems. $\pi_{0.5}$ rises from 26.8%/45.6% to 56.2%/72.2%, and PrediMem rises from 42.0%/60.6% to 57.7%/75.9%. Because the base action model is unchanged, these results support the claim that a substantial part of the gain comes from how evidence is saved, retrieved, and presented.
LIBERO-PRO adds object, goal, and spatial perturbations. Without privileged object poses, RoboFoundry reaches 96.0%/98.0% on Object, 88.0%/86.0% on Goal, and 92.0%/91.5% on Spatial in TSR/CSR. With privileged poses, the same settings rise to 99.0%/100.0%, 96.0%/100.0%, and 98.0%/99.2%. The improvement is useful evidence for the system-level method, but the jump also identifies perception as a remaining bottleneck.
| LIBERO-PRO split | RoboFoundry TSR / CSR | CaP-Agent0 TSR / CSR | With privileged poses |
|---|---|---|---|
| Object | 96.0 / 98.0 | 21.8 / 18.2 | 99.0 / 100.0 |
| Goal | 88.0 / 86.0 | 25.6 / 16.8 | 96.0 / 100.0 |
| Spatial | 92.0 / 91.5 | 11.8 / 14.0 | 98.0 / 99.2 |
11. What the Case Studies Reveal
In the cleanser task, the initial system emits long sequences of up to twenty actions. Evolution compresses repeated behavior into one or two high-level steps. The gain comes from a reusable abstraction that reduces the need to re-derive the same sequence at every step.
In the red-star insertion task, fifteen valid actions still fail because the object slips before entering the opening. The repaired skill keeps the gripper closed after grasping until insertion is achieved, completing the task on the sixth action. The rule is then abstracted as a grasp recovery rather than stored as a task-specific sentence.
The newspaper case separates the semantic object from its physical support: the robot must move the laptop while respecting the sofa as the surface beneath it. The navigation case repairs reachable-floor reasoning and reduces target distance from 3.9 meters to 0.9 meters. These examples show that the system can repair different layers without changing the foundation model.
RoboMemArena provides additional memory-specific evidence. Explicit next-stage records raise butter-and-cream-cheese task success from 40% to 70%. The occlusion case teaches the agent to treat the current frame as authoritative and an attempted command as different from a successful action; stage completion rises from 50% to 75%. Stage-conditioned context improves counting from 60% to 80%. The sequence case improves intermediate stages but does not solve the full task, a limitation the paper reports rather than hides.
12. Limitations
The authors identify perception as a primary limitation. Privileged object poses produce a large improvement on LIBERO-PRO, which means the evolved system cannot fully compensate for incorrect visual localization or state estimation. Real robots add calibration drift, sensor noise, occlusion, and contact variability, so semantic repairs cannot replace a reliable perception pipeline.
A second limitation follows from freezing the foundation model. RoboFoundry can improve context, skills, and interfaces, but it cannot add new visual priors or low-level manipulation competence that the base model lacks. The different results across base models show that system evolution complements model capability rather than eliminating its ceiling.
Third, self-generated repair and self-verification can contain verifier bias. A scoring rule may reward a short-term success while missing a long-term side effect. Held-out traces still share benchmark assumptions and collection procedures with the training traces. Reaching a non-negative transfer score on those traces is evidence of retention, not proof of open-world generalization.
Finally, the real-robot evaluation uses roughly ten trials per task and a limited set of scenes. It demonstrates that the mechanism can operate outside simulation, but it does not estimate long-tail failure rates. Persistent edits can also accumulate conflicts or shortcuts, requiring versioning, rollback, dependency checks, and regular regression. The current repository does not yet release the core benchmark implementation, which limits independent reproduction.
13. Conclusion
The paper's central contribution is a change of optimization target. Instead of asking how to retrain a foundation model after every failure, RoboFoundry asks which part of the surrounding system should change, how that change can be tested, and when it deserves to become persistent. Context, skills, semantic bindings, execution bindings, and filesystem records together form the effective embodiment-specific policy.
The next research step is not simply a larger skill library. It is a more reliable learning system: calibrated verification, open-world held-out sets, perception-aware diagnosis, safe cross-embodiment transfer, and long-term governance of persistent files. RoboFoundry provides a coherent architecture for that direction, while its reported limitations show how much engineering remains between benchmark self-improvement and dependable autonomous robots.
Key sentence: A robot policy becomes self-improving only when failure produces a tested, reversible, transferable change in the system that surrounds the model.
