D&A: Decoupling Description and Assessment Enables VLMs as Zero-Shot Robotic Reward Models

Paper metadata: Marius Memmel, Shawn Yin, Urvi Rutia, Angad Wadhwa, Eldon Li, Gavin Angeloff, Ankit Goyal, Jesse Zhang, Dieter Fox, Fabio Ramos, Anqi Li, and Abhishek Gupta. The work is associated with the University of Washington and NVIDIA. Resources: arXiv 2611.03622, project page, and official code. The repository is MIT-licensed; this review inspected commit 8f728ef.

One-sentence summary

The paper separates robotic reward estimation into an objective description of what happened and an assessment against an explicit success criterion, reducing false positive rewards on failed rollouts without robotics-specific training.

Background and motivation

A robot learning system needs a reward signal that distinguishes progress from motion. Real-world trajectories rarely include dense, reliable human labels, so a practical reward model should extract useful supervision directly from video and a language instruction. The desired output is a sequence that stays near zero when the task has not started, approaches one only when the task is complete, and can move downward when the robot undoes progress.

Directly asking a vision-language model for this signal exposes a confirmation bias. Once the instruction says that the robot is performing a task, the model tends to interpret plausible motion as evidence of completion. Figure 1 of the paper shows failed trajectories receiving high scores and successful trajectories overlapping with failures. Such a reward is worse than merely noisy: it can teach a policy that the wrong object or an incomplete placement is desirable.

Post-trained reward models offer an alternative. RoboReward and Robometer learn progress, success, or preference targets from curated robot data and improve in-distribution accuracy. Their dependence on labeled failures is expensive, and their learned weights are tied to a particular vision-language backbone. Every change in the base model can require another training cycle, which conflicts with the rapid release cadence of frontier models.

Existing zero-shot alternatives also take indirect shortcuts. GVL ranks shuffled frames, while TOPReward extracts a success-oriented token probability. Both methods infer a trajectory-level property from local visual plausibility or relative ordering. Neither first states the observable condition that would make the instruction true. D&A starts from this missing semantic anchor.

The paper's broader observation is that a reward model must answer two different questions. The first is descriptive: what objects changed, how did the robot move, and what state was reached? The second is evaluative: does that state satisfy the instruction? Collapsing these questions into one forward pass lets the intended answer contaminate perception.

Problem formulation

Given a video $o_{1:T}=\{o_1,o_2,\ldots,o_T\}$ and a natural-language instruction $l$, the model predicts a reward for every timestep. The reward is bounded by $[0,1]$, where zero denotes no progress and one denotes complete success. The conventional end-to-end formulation asks one model to learn $$ R:(o_{1:T},l)\mapsto r_{1:T}. $$ This mapping combines entity recognition, temporal reasoning, instruction following, and scalar calibration without exposing the intermediate evidence.

D&A instead composes a describer $D$ with an assessor $A$: $$ R(o_{1:T},l)=A\bigl(l,D(o_{1:T})\bigr). $$ The describer is not given the instruction. The assessor sees the resulting trajectory text and the instruction, but not the raw video. The separation is behavioral rather than architectural: the two stages use fresh prompts and do not share conversational context.

The two outputs can be written as $$ d_{1:T}=D(o_{1:T}),\qquad r_{1:T}=A(l,d_{1:T}). $$ This text bottleneck makes the reward auditable. A developer can inspect whether a missing object is absent from the scene or merely omitted from the description. It also creates a real information loss, which becomes the method's principal technical limitation.

For cost control, the implementation samples $H$ frames from the trajectory before inference. The paper often uses $H=8$, with $H\ll T$, and linearly interpolates the resulting rewards back to the original length. The interpolation changes temporal resolution, not the absolute definition of completion.

flowchart TB
    V[Full robot video] --> S[Sample H key frames]
    I[Language instruction] --> E[Entity extractor]
    S --> E
    E --> O[All observed objects]
    S --> D[Grounded describer]
    O --> D
    D --> X[Per-frame text description]
    I --> C[Generate explicit success criterion]
    X --> C
    C --> A[Assessor]
    X --> A
    A --> R[Rewards in 0 to 1]
    A --> F[Failure analysis and corrective feedback]
    R --> P[Online RL or offline advantage]
    F --> G[Next planning attempt]
    

Figure 1: Method flow reconstructed from Figure 3 of the paper. The entity extractor receives the instruction, but the describer receives the full object list rather than a task-relevance label.

Method

Whole-trajectory description

A causal estimator computes $r_t=R(o_{1:t},l)$ from a prefix of the video. D&A supplies the complete sequence to the describer in one pass. The later frames let the model distinguish an object moving through a target region from an object that remains there after release. The same context helps describe reversals instead of forcing every reward to increase.

The method does not assume monotonic progress. A bowl can be lifted and then dropped, so its reward should fall. Temporally consistent language is intended to capture this behavior. Yet the description itself is still generated by a model with a finite context, so long videos must be sampled and summarized before assessment.

Entity grounding without revealing task relevance

A describer with no instruction may ignore small but consequential objects. For example, a task may refer to a book with an orange spine, while a generic description says only that a book is on the table. The paper therefore introduces an entity extractor $E$ that reads both video and instruction and lists every object in the scene. A grounded describer consumes the full set: $$ d_{1:T}=D_{\mathrm{desc}}\bigl(o_{1:T},E(o_{1:T},l)\bigr). $$ The extractor can preserve useful names and attributes, while withholding the subset marked as task-relevant prevents the describer from assuming that those objects were manipulated correctly.

This is instruction-blind in effect, not in the strict information-theoretic sense. Because the extractor has already seen the task, its vocabulary may favor distinctions that matter for the task. That residual channel is a useful point for independent scrutiny when interpreting the paper's zero-shot claim.

Explicit success criteria

An instruction describes an action, but it does not always define the observable completion state. “Put the fruit in the bowl” leaves open whether the gripper must be released, whether the fruit must remain inside after a short delay, and whether touching the bowl counts. Asking for a score against the instruction alone leaves this boundary underspecified.

The assessor first constructs a natural-language criterion $c=C(l,d_{1:T})$ and then scores the description against it: $$ r_{1:T}=A_{\mathrm{score}}\bigl(d_{1:T},C(l,d_{1:T})\bigr). $$ The question changes from “does this look like the requested action?” to “how closely does the observed state satisfy the explicit completion condition?” In the released implementation, criterion construction and scoring occur in a single assessor call, with a prompt that asks the model to articulate completion before producing the progress list.

This absolute anchor differs from temporal ranking. If every video in a batch is a failure, a relative model can still assign a high score to the least bad member. An explicit criterion supports a zero score when the required object is absent and partial scores when some conditions are met.

Failure diagnosis and corrective feedback

A scalar reward can tell a policy that an action was poor, but not why. The feedback variant introduces a failure function $F$: $$ A(l,d_{1:T})=A_{\mathrm{score}}\bigl(d_{1:T},C(l,d_{1:T}),F(l,d_{1:T})\bigr). $$ For an incomplete trajectory, the assessor reports the failure and a correction, such as using the wrong arm, missing the grasp, or stopping before placing the object. A code-as-policies planner can then append this diagnosis to the history for the next attempt.

The feedback is not a separate reward head. It reuses the semantic rubric built for scoring, which keeps diagnosis and numerical assessment tied to the same completion condition. Its effect is evaluated separately because a better explanation need not automatically produce a better policy.

Rewards for policy improvement

Online RL uses $r_t$ directly as a dense environment reward. The paper applies latent-space reinforcement learning to a frozen policy and evaluates whether the signal can suppress a repeated wrong-object grasp. The reward must therefore distinguish object identity, not merely detect that the gripper is moving toward some object.

Offline RL converts absolute progress into an advantage. With action-chunk size $C$, one practical estimator is $$ A_t=r_{\min(t+C,T)}-r_t. $$ A positive value means that the chunk advanced the task, while zero or negative values identify stagnant or reversing chunks. The advantage can filter a dataset or condition a policy during fine-tuning.

The appendix also presents a reward-difference form. Define $$ \tilde r_t=r_t-r_{t-1} $$ as local progress and $$ G_t=\sum_{k=t}^{T}\gamma^{k-t}\tilde r_k $$ as its discounted accumulation. Combining this return with a value baseline gives $A(s_t,a_t)=G_t-V$. These expressions show how a bounded language-model score is converted into a standard learning signal rather than used as a direct action generator.

Implementation mapping

The public API exposes three modes. dna performs grounding, description, and assessment; dna_feedback adds failure analysis; naive sends the video directly to a progress prompt. The aliases in dna/classes/main.py:21-53 preserve the connection between the paper terminology and the executable pipeline.

_METHOD_ALIASES = {
    "dna": "dna",
    "video_grounded_hierarchy_single": "dna",
    "dna_feedback": "dna_feedback",
    "video_grounded_hierarchy_failure": "dna_feedback",
    "naive": "naive",
    "video_endtoend_single": "naive",
}

The orchestration in dna/classes/main.py:192-247 first obtains the grounded description, then dispatches to the ordinary or feedback assessor. The returned progress vector is validated against the frame count in _sample_once and divided by one hundred, matching the paper's zero-to-one convention.

description_raw = await get_description_from_video_grounded(
    frames, task, model_id=self.model,
    thinking_level=self.thinking
)
description = {i: d["description"]
               for i, d in enumerate(description_raw)}

if self.method == "dna_feedback":
    results = await get_progress_from_description_failure(
        description, task, model_id=self.model,
        thinking_level=self.thinking
    )
else:
    results = await get_progress_from_description(
        description, task, model_id=self.model,
        thinking_level=self.thinking
    )

Two prompt files make the paper's safeguards explicit. dna/functions/description.py:64-146 tells the describer not to infer the robot's goal and to describe the scene objectively. dna/functions/progress.py:10-87 states that the instruction fixes which objects count and that the scene description is the only evidence about their presence. The assessor is forbidden from renaming a different object to satisfy the task.

Experimental design and benchmark results

D&A-Bench contains 14 manipulation tasks collected on a bimanual YAM platform, with 27 successful and 27 failed trajectories. Each task is paired with one handwritten instruction and nine paraphrases. The resulting evaluation contains 540 video-instruction pairs. An appendix sentence uses the arithmetic of four episodes per task and states 560, but the released trajectory count is 54; the main text and the actual count support 540, which is the figure used here.

The main ordering metric is Kendall $\tau_\alpha$ on the final reward, with a range of $[-1,1]$. It measures whether success, suboptimality, and failure are ordered correctly. RBM-EVAL-OOD tests transfer to an external distribution. RoboRewardBenchmark reports mean absolute error against discrete end-of-episode progress labels, so lower is better.

Benchmark	Model	Metric	Value
D&A-Bench	D&A with Gemini 3 Flash	Kendall τ (table 0.672)	0.67
D&A-Bench	Robometer	Kendall τ	0.56
D&A-Bench	RoboReward	Kendall τ	0.43
D&A-Bench	GVL / TOPReward	Kendall τ	0.27 / 0.24
D&A-Bench	D&A with GPT-6 Astra	Kendall τ	0.790
RBM-EVAL-OOD	D&A with Gemini 3 Flash	Kendall τ	0.626
RBM-EVAL-OOD	Robometer	Kendall τ	0.648

Table 1 shows the central tradeoff. On D&A-Bench, the Gemini 3 Flash version reaches 0.67 and beats post-trained Robometer at 0.56 and RoboReward at 0.43. On RBM-EVAL-OOD, the same configuration is 0.626 and trails Robometer's 0.648. The claim is therefore not that one prompt dominates every external benchmark. It is that zero-shot decomposition is competitive while remaining model-agnostic, and that stronger bases continue to improve it.

Model	RoboRewardBenchmark MAE ↓	Training requirement
Naive VLM	0.92	No robotics data
GVL	1.34	No robotics data
TOPReward	1.80	No robotics data
RoboReward-4B	0.85	Post-trained
Robometer	0.75	Post-trained
D&A	0.70	Zero-shot prompting

Table 2 reports an MAE of 0.70 for D&A, compared with 0.75 for Robometer, 0.85 for RoboReward-4B, and 0.92 for the direct prompt. The improvement over post-training is important, but the operational interpretation must include inference expense. D&A is cheap in robotics data, not free in compute.

This failure clarifies the distinction between motion and task progress. A describer that does not know the instruction can record that the shirt is being folded. An assessor that sees “unfold the shirt” can then reject that state. The direct prompt instead uses the instruction as a frame for interpreting every visual change, which is exactly the bias the method is designed to interrupt.

The diagram also exposes the method's debugging surface. If a correct trajectory receives a low score, one can inspect the description for missing evidence. If an incorrect trajectory receives a high score, one can inspect the generated completion condition for an overly permissive rubric. A single end-to-end score offers no comparable check.

Real-world policy learning

In the online setting, a $\pi_0$ policy trained on DROID attempts to put fruit in a bowl. The initial success rate is 10%, largely because the policy repeatedly grasps an ice cream cone instead of the banana. TOPReward gives high reward to both object interactions and the success rate falls to 5%. D&A gives low reward to the wrong-object trajectory and reaches 100% in roughly 90 minutes.

This experiment is a useful stress test because both trajectories contain a successful-looking grasp and transport phase. The reward must bind the noun in the instruction to the object in the description. A model that rewards generic manipulation cannot solve the task, even if its scores look smooth.

For offline learning, the authors collect 50 optimal, 50 suboptimal, and 50 failed demonstrations on a bimanual YAM. Behavior cloning on the mixed data reaches 20% success. D&A advantages condition a $\pi_{0.5}$ policy and improve over that baseline by 80%. In a separate advantage-filtered experiment, the D&A-trained policy completes the task in $18.8\pm1.8$ seconds, while the baseline requires $28.5\pm9.2$ seconds.

The reduction in variance is as relevant as the mean speedup. A policy that occasionally finishes quickly but often stalls is difficult to deploy. D&A's advantage ranking appears to preserve chunks that complete the intended sequence and reject superficially similar alternatives. The result is still limited to the tasks and reward scales shown in the study.

The scaling study supports the claim that decomposition helps across model families. Gemini 2.5 Flash rises from 0.280 under naive prompting to 0.418 with D&A at low thinking; Gemini 3.5 Flash rises from 0.499 to 0.609; Gemini 3.1 Pro rises from 0.515 to 0.620 at medium thinking. The non-monotonic response to thinking levels is evidence that extra test-time reasoning is not the only source of improvement.

Ablation and feedback analysis

The RBM-EVAL-OOD ablation isolates the design choices. A success criterion without decoupling improves the score from 0.486 to 0.561. Decoupling without grounding drops it to 0.343, likely because the describer omits objects needed by the rubric. Adding entity grounding recovers the score to 0.546, and the full method reaches 0.626.

This negative result is one of the paper's strongest pieces of evidence. Splitting a model call is not inherently beneficial. The benefit depends on maintaining a sufficiently informative description while removing the evaluative hypothesis from the observer. A pipeline that hides both the instruction and the relevant entities merely shifts the failure from confirmation bias to omission.

Cost measurements reinforce that base-model choice is not a simple quality ranking. Gemini 3.1 Pro uses 0.035 dollars, 23.7 seconds, and 7,358 tokens per trajectory. Gemini 3.5 Flash uses 0.039 dollars, 25.9 seconds, and 8,842 tokens. Gemini 2.5 Pro uses 0.060 dollars, 58.0 seconds, and 10,843 tokens. Gemini 2.5 Flash costs 0.015 dollars but takes 36.9 seconds and 11,177 tokens. A deployment must choose among accuracy, latency, and provider constraints rather than optimizing one scalar.

The feedback experiment isolates the value of language beyond reward. A code-as-policies planner on LIBERO improves from 11.4% with no history to 40.6% with code and binary-success history. Appending D&A failure feedback raises success to 51.4%, with a reported $p=0.00076$ over the history baseline. The feedback therefore contributes information that scalar success does not encode.

Discussion

D&A is best suited to tasks whose completion can be expressed as observable object and robot states. Picking a named object, placing it in a region, sorting objects by a rule, or assembling a structure all fit the abstraction. Tasks requiring sub-millimeter geometry, continuous force regulation, or contact dynamics expose the weakness of a purely linguistic interface.

The method can also serve as a data-selection layer. An absolute reward model can rank action chunks across episodes, estimate advantages, and retain behavior that moves toward an explicit criterion. This reduces the need for perfect labels, but it does not remove the need for validation. A biased describer or permissive criterion can systematically admit bad data.

The engineering interface is deliberately minimal. A caller supplies frames and a task string and receives a vector bounded by zero and one. Retries and length validation handle malformed responses, and the same wrapper can route to Gemini, GPT, Claude, or other providers. That portability is useful, but server-side model changes can alter behavior without changing the repository commit.

Limitations

The authors identify the main system constraints directly. Frontier APIs impose latency, token cost, and context limits; closed providers introduce stochasticity and limited control; occasional semantic hallucinations can corrupt both the description and the downstream reward. The paper also states that natural-language granularity limits rewards requiring precise spatial relationships or absolute geometric distances.

An independent limitation is the residual task leak in entity extraction. The extractor reads the instruction and supplies all objects to the describer. Although the task-relevant subset is hidden, the chosen granularity can reflect the task. A second concern is benchmark contamination: the authors suggest that Gemini 4 Argon may have seen RBM-EVAL-OOD during training, which weakens a cross-model out-of-distribution comparison. Closed APIs also make exact reproduction dependent on provider versions and availability.

Conclusion

D&A changes the order in which a vision-language model is asked to reason. It describes observed behavior before it knows what the evaluator hopes to see, then constructs an explicit success condition before assigning progress. Entity grounding preserves task-relevant detail, while a text bottleneck keeps the description inspectable. The resulting rewards support online RL, offline advantage estimation, data filtering, and feedback-driven planning.

The paper's broader lesson is that zero-shot reward quality depends on information routing, not only model scale. A stronger model can still confirm the premise embedded in a prompt. Separating observation from assessment gives failures a chance to be recognized as failures, while leaving open the harder problem of representing geometric and dynamic task constraints that natural language cannot reliably capture.

A reward model earns the right to recognize success only after it can describe failure without looking away.