Skip to content
←Back to Papers

PAPER DEEP DIVE

Robot LearningAtomic ActionsSubtask Annotation

RoboAtom: Large-scale Atomic-Action Supervision for Robot Learning

Fine-grained action annotations can support skill reuse and compositional generalization in robot policies. However, robot datasets typically provide task-level instructions without subtask annotations. Although some include subtask labels, inconsistent formats and conventions hinder cross-source joint training. We propose RoboAtom-Data, which adds unified, time-aligned atomic-action annotations to existing data from multiple sources, covering 1.45 million trajectories, 6.88 million skill-labeled segments, and 16,882.1 hours. Our pipeline combines a unified protocol, large-scale manual annotation, propagation across repeated executions with the same ordered skill sequence, and multi-stage quality control. We evaluate the dataset through automatic atomic-action annotation and robot control. We construct RoboAtom-Bench to standardize evaluation of skill recognition and temporal boundary localization. Trained on the annotated data, RoboAtom-Anno achieves better automatic annotation performance than all evaluated open- and closed-source models on both RoboAtom-Bench and the external WGO-Bench. For robot control, atomic-action supervision paired with hierarchical execution improves task success over task-level baselines on LIBERO-Long, LIBERO-Plus, and RoboTwin. We will release the data, benchmark, model, and construction pipeline to support further research on skill-level robot learning.

Paper: arXiv:2610.40207

Authors: Xueyuan Wei, Xuanxu Wang, Xudong Li, Yuping He, Wanshui Gan, Jiaqi Peng, Wenqi Guo, Jianjun Zhou, Junting Dong, Chunhua Shen, Haoyu Guo, Mulin Yu, and Tao Lu

Code: InternRobotics/RoboAtom, public and reviewed at commit 3bf97a0156b4db6d234c8d98a8f00c4943a34fba

Resources: annotation tools, propagation code, an evaluation protocol, and 9B and 27B annotation models. The current data card exposes 12 downloadable subsets, while the archived paper statistics cover 16 sources.

One-sentence summary: RoboAtom turns heterogeneous robot trajectories into a shared supervision interface composed of skill categories, object-specific action descriptions, and time-aligned boundaries, then scales that interface through physical-signal propagation, dedicated annotation models, and hierarchical policy execution.

Background: The missing layer is reusable intermediate structure

Most robot policies are trained from a task-level language instruction such as "put the cup in the drawer." That sentence specifies a goal, but it does not expose the ordered operations required to reach it: approach the cup, grasp it, move to the drawer, open the drawer, place the cup, and release it. A monolithic policy can imitate the whole sequence, yet the absence of intermediate structure makes demonstrations difficult to compare, reuse, repair, or combine across datasets.

An atomic action in RoboAtom is a time interval with three linked fields: a skill category shared across data sources, a natural-language description that retains object and motion details, and a start and end frame. "Pick up the red block" and "pick up the blue block" can therefore share the category Pick while remaining distinguishable in language. This representation supports two consumers at once. It provides temporal evidence for video annotation and executable intermediate instructions for a robot policy.

Existing robot corpora are not devoid of subtask labels. AgiBot World, Galaxea Open-World, RoboInter-Data, BEHAVIOR-1K, and other collections contain useful local conventions. The problem is that their vocabularies, segmentation granularity, temporal coverage, and counting rules do not agree. One source may merge several pick-and-place cycles into a single segment; another may label only the full task; a third may use categories that cannot be mapped one-to-one. Concatenating these files does not create a coherent training set, because a category name without a common temporal meaning is not portable supervision.

Full manual correction is economically unattractive. The paper estimates roughly USD 100 per source-video hour for trajectory-by-trajectory refinement. Across more than 1.4 million trajectories, that process cannot be the default path. General video-language models offer another route, but they are strongest at describing visible events and weaker at locating brief physical transitions. A grasp, release, contact break, or object departure may occupy only a few frames. Producing a plausible sentence is not the same as identifying both endpoints of the event.

RoboAtom therefore combines human seed annotation with propagation across repeated executions of the same task. A small set of reviewed references establishes the ordered skill sequence. PhysExtend transfers that sequence to new trajectories using robot state-action evidence, duration priors, and skill-dependent physical events. Later stages perform model checks, cross-seed consistency analysis, and human audits. The design is deliberately different from asking a multimodal model to regenerate every label. The model may propose candidates or inspect semantics, while temporal localization is grounded in measurable physical signals.

The contribution has three layers. RoboAtom-Data contains 1,446,974 trajectories, 6,876,282 non-empty skill segments, and 16,882.1 hours across 16 sources and 3,334 tasks. RoboAtom-Bench contains 500 expert-reviewed trajectories and 4,018 ground-truth segments, with separate measures for temporal overlap, skill recognition, and sequence editing. The downstream evaluation trains RoboAtom-Anno for automatic annotation and tests whether the same atomic-action interface improves LIBERO-Long, LIBERO-Plus, and RoboTwin control.

The work is best understood as infrastructure for the data layer of robot foundation models. It does not introduce one narrow segmentation head or one new policy architecture. It defines the intermediate unit that data ingestion, annotation, evaluation, policy conditioning, and online scheduling can share. If robot systems are expected to compose skills, recover from failures, or train jointly across embodiments, this common unit matters before model scaling begins.

Overview of RoboAtom data, benchmark, automatic annotation, and robot control

Figure 1. RoboAtom links dataset construction and evaluation with two downstream uses: automatic atomic-action annotation and hierarchical policy execution.

Data definition: A common ontology without erasing object-level language

The annotation protocol starts a new segment only when the dominant skill changes. Preparation, arm re-positioning, and short transitions are absorbed into an adjacent semantic segment. This convention is small in appearance but essential for cross-source comparison. If one pipeline treats "approach the object" as a separate action while another includes it in "grasp," both may call their output atomic, yet their boundaries and segment counts cannot be aligned.

The released vocabulary contains 29 skills, including close, fold, hand over, move, open, pick, place, plug, pour, press, pull, push, screw, sweep, wipe, and additional long-tail operations. Semantically close verbs may share a skill category while preserving their original verb in the description. For example, an unfolding operation can be represented under the broader fold family, and a loosening operation can remain distinct in language while mapping to screw. The category enables aggregate learning; the description preserves the concrete intent.

Training and primary benchmarking use a 15-class closed vocabulary: Close, Fold, Handover, Move, Open, Pick, Place, Plug, Pour, Press, Pull, Push, Screw, Sweep, and Wipe. These classes cover 6,819,538 segments, or 99.2% of the released set. The remaining 14 classes remain available for finer-grained research, but the paper excludes several long-tail distinctions from the main exact-match metric because actions such as stir versus shake, or cut versus peel, are often adjacent in time and inconsistently separated across sources.

This split between a complete ontology and a controlled evaluation ontology addresses two competing requirements. Keeping all 29 classes avoids deleting rare behaviors from the resource. Restricting the main metric to 15 classes prevents a precise category score from being dominated by sparse labels with ambiguous boundaries. The distinction also clarifies what RoboAtom-Anno predicts: it must output a shared skill category, a descriptive subtask string, and interval endpoints. It is not merely translating a video into a free-form paragraph.

Data are normalized to LeRobot v2.1 while retaining repeated demonstrations of the same task. Existing annotations are not copied without inspection. Source rules are mapped into the unified taxonomy, compound segments are split when multiple interactions are present, and human reviewers inspect examples. RH20T illustrates why this step is needed: a source segment can combine several pick-and-place rounds over different colored blocks, even though the execution contains multiple dominant-action transitions.

Construction: Seeds, physical propagation, and layered quality control

Seed annotation begins with Gemini 3.6 Flash proposing an ordered action list and initial boundaries. Human annotators inspect the original video, correct categories and descriptions, adjust step order and segmentation granularity, and refine start and end frames. The model receives a fixed 3-by-4 contact sheet rather than a dense frame stream, reducing visual cost while retaining samples across the episode. A second model, Qwen3-VL-235B-A22B-Instruct, checks spelling and semantic consistency; the annotator decides whether each suggestion should be accepted.

PhysExtend applies only when a target execution can be paired with references that have the same ordered skill sequence. Let the target trajectory contain frames from 0 through T and let the known sequence contain M skills. The boundary vector must define a complete, non-overlapping partition:

$$0=b_0 < b_1 < \cdots < b_{M-1} < b_M=T$$

This constraint converts generic video segmentation into ordered partition alignment. The method cannot invent an extra skill or swap two stages. It searches for plausible transition locations under the reference sequence. The objective contains a reference and motion-progress cost, a duration penalty, and skill-specific physical-event terms:

$$\hat{\mathbf b}=\arg\min_{\mathbf b}\sum_{m=1}^{M}\left[C_m(b_{m-1},b_m)+\lambda_dD_m(b_{m-1},b_m)\right]+\lambda_q\sum_{m=1}^{M-1}Q_m(b_m)$$

Here, C measures deviation from the reference position and cumulative motion progress, D favors plausible segment durations, and Q rewards consistency with gripper events associated with a skill. Candidate boundaries are generated from reference durations, motion progress, and events. Joint dynamic programming selects an ordered path, after which reliable pick and place transitions can correct the result. Candidate extension, fusion, and monotonic projection address the possibility that one reference is noisy or that independently decoded boundaries cross.

Physical evidence includes motion intensity, pauses, direction changes, gripper state, holding-state consistency, and chassis movement. Different embodiments require explicit adaptation. Agilex dual-arm recordings use dynamic time warping over joint trajectories to transfer boundaries according to temporal variation. Mobile dual-arm platforms include chassis state because arm signals alone cannot identify a movement phase. When a robot's grasp and release encodings have a stable offset from the reference, the system calibrates that offset instead of assuming identical sensor values.

Quality control samples 20 trajectories per source-by-embodiment group and checks skill order, boundaries, descriptions, systematic reference errors, and agreement among seeds. The release gate requires at least 85% subtask segmentation accuracy and at least 75% episode-level accuracy, where an episode passes only if every subtask has correct semantics and event-aligned boundaries. In the reported checks, consistent seeds produce 87.6% segmentation accuracy and 80.5% episode accuracy. Inconsistent seeds fall to 79.5% and 57.4%.

The gap is not a small annotation detail. It means a cheap propagation system can multiply an erroneous reference across many executions of the same task. RoboAtom therefore treats sequence consistency as a risk signal and performs a second manual review when seeds disagree. Tasks with insufficient reference coverage receive additional manual annotations. The scaling mechanism expands reviewed structure; it does not assume that scale itself repairs a bad structure.

flowchart LR
    A["16 sources
real and simulated"] --> B["LeRobot v2.1
task filtering"] B --> C["Gemini candidates
contact-sheet prelabel"] C --> D["Human seed review
category text boundaries"] D --> E["PhysExtend
ordered same-task transfer"] E --> F["Joint DP
motion gripper chassis"] F --> G["Reference fusion
monotonic closed coverage"] G --> H["Model checks and human audit
semantic and temporal QC"] H --> I["RoboAtom-Data
1.45M trajectories"] H --> J["RoboAtom-Bench
500 reviewed trajectories"] I --> K["RoboAtom-Anno
mid-train SFT GRPO"] I --> L["Atomic-action prior
for System 2 scheduling"] K --> M["Predicted atomic segments"] L --> N["System 1 pi0.5 execution"]

The diagram reflects the actual dependency structure. PhysExtend does not infer an unknown sequence from a bare video. RoboAtom-Anno and the control policy are separate consumers of the annotated data. The annotation model is not placed inside the real-time control loop. System 2 schedules an intermediate instruction; System 1 executes it with robot actions.

Dataset scale and composition

RoboAtom-Data contains approximately 9,375 real-robot hours and 7,508 simulated hours. It spans 16 sources, 3,334 tasks, and 32 embodiment metadata categories. Task counts are deduplicated using a source plus normalized task identifier so that the same task is not repeatedly counted across batches, policy splits, or embodiments. Segment statistics include only segments with non-empty skill labels, excluding empty, idle, and recovery intervals from the supervision count.

DatasetDomainHoursTrajectoriesSegmentsSkillsTasks
AgiBot World BetaReal2,976.4141,5631,001,55277217
AgiBot World 2026Real591.628,331327,3554337
Galaxea Open-WorldReal500.019,183108,77558150
RoboInter-DataReal2,417.1235,880752,38815232
BEHAVIOR-1K 2026Simulation1,952.920,000406,34135100
RoboAtom-DataMixed16,882.11,446,9746,876,282293,334

The final row is the archived corpus total, not the arithmetic sum of the five illustrative rows. The paper compares RoboAtom with datasets that already provide some subtask annotations and reports a larger trajectory, segment, hour, and task footprint. It also characterizes episode duration, segment count, task text, and action-description distributions. The corpus includes short interactions and episodes with more than 50 segments, which is important because supervision difficulty changes sharply with action density.

Coverage alone does not establish annotation quality. The archive combines public and private sources with different collection procedures, and a downloadable data card cannot be assumed to expose every archived trajectory. The repository currently describes 12 downloadable subsets, whereas the paper statistics include 16 sources. Anyone reproducing the training mix should inspect per-dataset counts, licenses, and access conditions rather than treating 1.45 million as the size of a single public download.

RoboAtom dataset scale, episode duration, segment counts, and language distributions

Figure 4. Dataset diversity and scale. The panels report episode duration, segment count, comparisons with existing annotated datasets, and distributions of task and action descriptions.

RoboAtom-Bench: Separate localization, recognition, and sequence structure

An atomic-action prediction can fail in at least three independent ways. Its boundaries may be misplaced while the skill category is correct; its category may be wrong despite a plausible interval; or it may omit, insert, or reorder a skill and thereby change the execution plan. A single segmentation score conflates these failures. RoboAtom-Bench therefore reports temporal mean IoU, temporal F1, Skill F1, and an Edit score.

For ground-truth interval G_i and prediction P_j, temporal overlap and mean IoU are defined as:

$$\operatorname{tIoU}(G_i,P_j)=\frac{|G_i\cap P_j|}{|G_i\cup P_j|},\qquad\operatorname{mIoU}=\frac{1}{N_G}\sum_{i=1}^{N_G}\operatorname{tIoU}(G_i,P_{\pi(i)})$$

The mapping pi is an order-preserving one-to-one alignment that maximizes overlap without considering skill labels. An unmatched ground-truth segment contributes zero. The F1 scores aggregate true positives, false positives, and false negatives over episodes before taking Micro F1:

$$F1=\frac{2TP}{2TP+FP+FN}$$

Temporal F1 first maximizes the number of pairs above an IoU threshold and then maximizes their total overlap. It does not require equal skills. Skill F1 processes predictions in temporal order and permits matches only to unassigned ground-truth segments with the same category. If the best same-skill overlap misses the threshold, the prediction becomes a false positive rather than falling back to a lower-quality interval. The metric is therefore sensitive to both semantic and temporal correctness.

The Edit score removes timestamps and compares skill sequences after collapsing consecutive duplicate labels. For predicted sequence S_P and ground-truth sequence S_G,

$$\operatorname{Edit}=100\left(1-\frac{d_{\mathrm{Lev}}(S_P,S_G)}{\max(|S_P|,|S_G|)}\right)$$

Here d_Lev is Levenshtein distance with unit insertion, deletion, and substitution costs. Edit can expose missing or reordered operations, but it ignores boundary placement and can hide changes in the internal split of a repeated skill. It is used as a complementary diagnostic rather than a replacement for the temporal measures.

SubsetTrajectoriesMean GT segmentsRoboAtom-Anno-9B Skill F1@0.7Difficulty condition
Short2002.0671.11 to 4 actions
Medium1006.3443.75 to 8 actions
Long10023.0217.39 to 62 actions
Multiview1006.7133.9Synchronized cameras
Overall5008.0430.14,018 GT segments

RoboAtom-Bench includes 500 expert-reviewed trajectories, 721 videos, and 4,018 ground-truth segments. Long trajectories average 99 seconds, compared with 13 seconds for the short subset. Actions shorter than two seconds account for 21.8% of long-set actions and 11.7% of short-set actions. Under a shared input policy of 2 frames per second and at most 448 frames, a brief action occupies only a small number of samples in a long episode. This directly explains part of the performance drop from 71.1 Skill F1 on short trajectories to 17.3 on long ones.

RoboAtom-Bench temporal metrics and segment count distributions

Figure 5. Definitions of temporal mIoU, temporal F1, Skill F1, and Edit, followed by ground-truth segment distributions and predicted-to-GT segment ratios across benchmark subsets.

PhysExtend: Physical evidence changes the boundary solution

The extension study contains 100 tasks with five human-reviewed trajectories per task. The number of references ranges from one to four, and every reference set within a task shares the same ordered skill sequence. PhysExtend is compared with Cortex, which also transfers reference segmentations using state-action and visual features but fits reference feature distributions. RoboAtom's variant combines skill-specific events, motion progress, chassis state, duration priors, ordered dynamic programming, event correction, multi-reference fusion, and monotonic projection.

MethodmIoUF1@0.7Skill F1@0.7Episode accuracyUSD per source-video hour
GPT-4o48.220.711.110%0.888
Gemini 3.1 Pro61.144.939.739%2.877
GPT-5.6-sol58.937.835.046%3.510
Qwen3.8-27B46.125.419.125%0.051
Cortex66.549.848.537%4.8 x 10-4
PhysExtend76.871.671.069%2.8 x 10-5

With one reference, PhysExtend reaches 71.0 Skill F1@0.7 versus 48.5 for Cortex. With four references it improves to 79.3. The stricter episode score is more informative: PhysExtend reaches 69%, Cortex 37%, and GPT-5.6-sol 46%. A high aggregate IoU can still conceal a boundary offset inside a long segment, whereas the episode score requires every subtask to have the right semantics and an event-aligned interval.

On one CPU with one reference, PhysExtend processes 15,952 trajectories per hour, about 17.1 times the throughput of Cortex. The reported compute cost is approximately USD 0.000028 per source-video hour for RoboAtom and USD 0.000481 for Cortex. This is inference computation, not the total cost of data acquisition, seed annotation, model training, and human review. The result supports cheap propagation after a reviewed reference exists; it does not prove that the whole corpus can be annotated without human labor.

PhysExtend evidence from gripper state, chassis motion, skill features, and reference count

Figure 3. Skill-aware physical evidence and extension performance. The panels align gripper signals with pick and place, chassis motion with move, summarize skill-specific features, and compare correctness as references increase.

Code correspondence: Boundaries are aligned, not narrated

The public implementation confirms that PhysExtend is not a prompt-only video segmenter. In seeded_decoder.py, sensor_cut() finds sensor channels shared by the seed and target, constructs state, velocity, acceleration, and time-context features, and fits the physical model from the independently annotated seed. The visual placeholder is retained only for interface compatibility, and its weight is explicitly set to zero.

# src/roboatom/physextend/decoders/seeded_decoder.py:144-165
def sensor_cut(row, config):
    seed, target = row["seed"], row["target"]
    common = sorted(
        name for name in set(seed_raw) & set(target_raw)
        if seed_raw[name].shape[1] == target_raw[name].shape[1]
    )
    seed_features = sensor_episode(seed, common)
    target_features = sensor_episode(target, common)
    model = sensor.fit_model([label], {seed["episode_key"]: seed_features})
    costs = sensor.episode_costs(
        target_features, model, state_weight=1.0, visual_weight=0.0
    )
    boundaries, confidence, expected = sensor.solve_boundaries(
        features=target_features,
        cost=costs,
        model=model,
        duration_weight=float(config.get("duration_weight", 0.85)),
        boundary_motion_weight=float(config.get("motion_weight", 0.25)),
    )

Multi-reference fusion is also more constrained than averaging. reference_ordered_projection() computes a minimum feasible duration for each segment from all references and then applies a pool-adjacent-violators style projection. Nearby candidate boundaries can be averaged, while larger disagreements can use a weighted median. Regardless of the fusion outcome, the projected cuts must remain ordered and must leave enough room for every segment.

# src/roboatom/physextend/decoders/fusion.py:220-252
def reference_ordered_projection(values, paths, length):
    durations = np.diff(
        np.column_stack([
            np.zeros(len(paths)), paths, np.full(len(paths), length)
        ]),
        axis=1,
    )
    original_minimum = max(2, round(length * 0.012))
    minima = np.maximum(
        1,
        np.minimum(original_minimum, np.floor(0.5 * np.min(durations, axis=0))),
    ).astype(int)
    # Pool-adjacent-violators projection keeps all cuts ordered.
    offsets = np.cumsum(minima)[:-1]
    slack = length - int(np.sum(minima))
    # ... block averaging and clipping ...
    return tuple(int(frame) for frame in projected + offsets)

The grasp-specific path in group_methods.py first verifies that the reference is a clean pick/place sequence and that gripper events recovered from the reference sensor agree with the reference labels. Only then does it combine target gripper events with a calibrated reference offset. Even if the target transcript differs from the reference, the result must pass checks on event-count changes and boundary feasibility before it replaces the candidate transcript.

joint_dtw_transfer() handles variation in joint trajectories when the seed and target share the same ordered skills. It calculates a dynamic-warping path over joint motion, transfers reference boundaries to the target timeline, and limits both the path cost and the allowed shift. The final cuts still have to be strictly increasing; candidates that collapse a segment are rejected rather than repaired by silently dropping a skill.

RoboAtom-Anno: Domain mid-training, SFT, and verifiable rewards

RoboAtom-Anno starts from Qwen3.5-9B. Robot-domain mid-training covers gripper-state question answering, object grounding, spatial reasoning, planning, and image or frame-sequence operations. The paper reports 2,841,942 samples from 27 subsets, 48 GPUs, and 3,084.1 GPU hours for this stage. Supervised fine-tuning then uses a task-sampled RoboAtom subset, 8 GPUs, and 27.7 GPU hours. GRPO consumes another 16 GPUs and 364.5 GPU hours. The three stages total approximately 3,476.3 GPU hours.

SFT teaches the model to emit a JSON segment list, but cross entropy alone does not directly optimize ordered matching, coverage, or sequence editing. GRPO combines several verifiable rewards:

$$R=\lambda_{\mathrm{temporal}}R_{\mathrm{temporal}}+\lambda_{\mathrm{soft}}R_{\mathrm{soft}}+\lambda_{\mathrm{edit}}R_{\mathrm{edit}}+\lambda_{\mathrm{format}}R_{\mathrm{format}}$$

The main experiment uses these weights:

$$R=0.35R_{\mathrm{temporal}}+0.25R_{\mathrm{soft}}+0.30R_{\mathrm{edit}}+0.10R_{\mathrm{format}}$$

The temporal term uses order-preserving F1 at an IoU threshold of 0.5. The soft term divides matched overlap by the larger of predicted and ground-truth segment counts. The edit term measures sequence similarity, and the format term checks JSON parsing, required fields, and interval coverage. An unparsable output zeros the first three terms and assigns -1 to the format reward, producing a total of -0.02. The model samples eight candidates for a prompt and normalizes their rewards within the group:

$$A_i=\frac{r_i-\bar r}{s_r+10^{-4}}$$

Here r_i is the reward of candidate i, r-bar is the group mean, and s_r is the group standard deviation. The reward makes segment count, boundary coverage, and ordered skill structure explicit optimization targets. Before GRPO, the predicted-to-ground-truth segment-count ratio is 1.20; afterward it is 0.99. The model moves from a tendency to over-segment toward output granularity closer to the reviewed annotations.

The inference code fixes a reproducible input contract. sample_indices() targets 2 frames per second, caps the input at 448 frames, and includes both endpoints. A synchronized multiview trajectory is assembled as one mosaic for each sampled time. The prompt requires zero-based closed intervals, complete coverage, no gaps, no overlaps, a first start at frame 0, each next start equal to the previous end plus one, and a final end at the last frame.

# src/roboatom/inference/video.py:15-26
def sample_indices(total_frames: int, duration: float) -> list[int]:
    count = min(total_frames, 448, max(1, math.floor(duration * 2.0)))
    return np.linspace(0, total_frames - 1, count).astype(np.int32).tolist()

# src/roboatom/inference/prompt.py:73-81
"Intervals are closed and inclusive: [start_frame, end_frame]."
"Every frame must be covered exactly once with no gaps or overlaps."
"The first start_frame must be 0."
"Each next start_frame must equal the previous end_frame plus 1."
"The final end_frame must equal total_frames minus 1."

These constraints matter for evaluation because the benchmark scores complete partitions, not independent snippets. They also expose a scaling weakness. A two-hour episode cannot retain 2 frames per second under a 448-frame cap. The pipeline chooses broad temporal coverage over local boundary resolution, which benefits long-horizon context but compresses brief actions. Section 5.3 reports the consequence directly in the long-subset score.

Automatic annotation results: Better quality at much lower marginal cost

RoboAtom-Bench is task-disjoint from training. WGO-Bench Robot provides an external open-vocabulary check. Reference models recover much of the ordered action structure but miss precise endpoints. GPT-5.6-sol reaches 63.4 Edit on RoboAtom-Bench yet only 28.0 temporal F1@0.7. This pattern suggests that video-level descriptions teach a model what is happening more effectively than when each physical transition begins and ends.

ModelRoboAtom mIoURoboAtom temporal F1@0.7Skill F1@0.7RoboAtom EditWGO Robot F1@0.7USD per video hour
GPT-4o31.513.97.247.714.60.971
GPT-5.6-sol44.028.021.463.431.34.330
Gemini 3.1 Pro42.524.416.161.829.64.516
Qwen3.8-27B38.420.110.956.624.30.103
Qwen3.5-9B9.66.23.528.71.70.008
RoboAtom-Anno-9B55.141.030.176.748.30.006
RoboAtom-Anno-27B56.443.233.876.849.20.078

RoboAtom-Anno-9B reaches 30.1 Skill F1@0.7, compared with 21.4 for GPT-5.6-sol. On WGO-Bench Robot, its temporal F1@0.7 is 48.3 versus the strongest reference score of 31.3. Its reported inference cost is USD 0.006 per video hour, almost three orders of magnitude below GPT-5.6-sol at USD 4.330. The 27B variant improves Skill F1 to 33.8 and WGO F1 to 49.2, and raises WGO Ego F1 from 21.1 to 28.3.

The training progression is more diagnostic than the final aggregate. The base model starts at 3.5 Skill F1. SFT on noisy open-source annotations moves it only to 6.6. Vocabulary alignment with ambiguity filtering reaches 17.1. Replacing the supervision with RoboAtom data and training a language-only LoRA reaches 23.0. The best SFT configuration reaches 24.2, robot-domain mid-training reaches 26.9, and GRPO reaches 30.1. Semantic consistency and reward optimization contribute at different stages rather than one intervention explaining the whole gain.

The benchmark subsets expose the unresolved regime. Skill F1 falls from 71.1 on short trajectories to 43.7 on medium, 33.9 on multiview, and 17.3 on long trajectories. Predicted segment counts are close to the correct total overall after GRPO, but total-count calibration is not enough to place every boundary in a dense sequence. The model needs stronger local temporal evidence without discarding the global context supplied by a long video.

Hierarchical control: The same annotation becomes an execution interface

The control study binds each observation-action pair to the skill category and description that cover its timestamp. System 1, a pi-0.5 policy fine-tuned with OpenPI, is trained with the local atomic instruction instead of only the full task string. At deployment, System 2 receives the task instruction, task knowledge, current observation, and execution history. It selects the next intermediate instruction from a constrained plan. System 1 generates an action chunk, and the next observation determines whether System 2 stays on the current skill or advances.

The architecture is not valuable merely because it contains a second language model. Its important property is that training and execution use the same granularity. If System 2 emitted categories or descriptions outside the annotation vocabulary, System 1 would receive an out-of-distribution condition. The task plan prior narrows the candidate space, while observation and history handle progress, failure, and retry. Task-level and atomic-action policies use the same post-training budget, observation space, and action space within each benchmark.

RoboAtom dual-system architecture with System 2 scheduling and System 1 execution

Figure 6. System 2 combines task priors, observations, history, and external knowledge to select an atomic action. System 1 executes the action-conditioned instruction, after which new observations update the schedule.

MethodLIBERO-LongLIBERO-Plus overallLIBERO-Plus objectLIBERO-Plus goalRoboTwin CleanRoboTwin RandomRoboTwin average
Task-level pi-0.595.0084.0585.2379.9380.7048.0464.37
RoboAtom dual system98.2088.1891.8686.3482.0250.3266.17

LIBERO-Long improves by 3.20 percentage points, which is consistent with the claim that local supervision helps a long-horizon policy maintain intermediate progress. LIBERO-Plus overall improves by 4.13 points, with gains of 6.63 under object change and 6.41 under goal change. RoboTwin improves by 1.32 in Clean and 2.28 in Random, raising the average by 1.80. The larger randomized gain may indicate that a local skill interface constrains bimanual execution more effectively than repeating the full task instruction.

The control evidence remains bounded. All three benchmarks are simulated, and real-world wear, occlusion, latency, unknown objects, and long-term recovery are not covered. The comparison also changes more than the supervision string: the dual system adds planning, task priors, and repeated model calls. The result supports the combined execution recipe, not a claim that intermediate instructions are the sole cause of every observed gain. Runtime latency and System 2 error propagation are separate deployment questions.

Analysis: Constraint propagation is the durable idea

The strongest technical idea is not the low inference cost by itself. It is the decomposition of propagation into a discrete ordered sequence and continuous boundary refinement. Repeated executions often preserve the list of skills even when pauses, speed, retries, and durations change. PhysExtend treats the order as a reference constraint and asks where the known transitions occur in the target. A failure in the continuous solver is less likely to corrupt the skill vocabulary, and a sequence mismatch can be rejected before expensive boundary optimization.

The distinction is useful operationally. It provides a natural review unit: confirm the skill order, confirm the object-specific descriptions, then inspect temporal endpoints. It also explains why the method needs robot state or a suitable proxy. A monocular video without proprioception does not expose a reliable gripper channel or chassis motion in every dataset. RoboAtom scales by adapting many sensor streams and embodiments, not by claiming that one embodiment-independent visual algorithm solves all modalities.

The annotation model changes the economics only after training. Total model training is about 3,476.3 GPU hours before data construction and human review are counted. The reported USD 0.006 per video hour is an inference estimate, not a complete replacement cost for manual annotation at USD 100 per hour. A fair conclusion is narrower and still important: once reviewed supervision and a trained model exist, the marginal cost of processing additional video falls sharply while benchmark accuracy remains above the tested general models.

Seed quality sets an upper bound on propagation. Consistent seeds yield 87.6% segmentation accuracy and 80.5% episode accuracy, while inconsistent seeds yield 79.5% and 57.4%. Scale can therefore amplify a systematic category mistake or temporal convention error. A production pipeline should reject conflicting seeds, cross-check event counts, and route uncertain cases to human review before propagation. Sampling failures after the fact is weaker than making consistency a precondition.

The benchmark's multi-metric design is also a reminder that no one score describes annotation quality. Skill F1 requires exact category matching, Edit ignores time, and temporal overlap ignores semantics. Their combination exposes more failure modes, but the 29-to-15 vocabulary reduction leaves open-vocabulary recognition outside the primary result. Repeated consecutive skills and long-tail distinctions remain difficult to validate with exact-match segmentation alone.

Limitations: Author-stated scope and independent risks

The authors explicitly state that atomic-action supervision has not yet been validated in large-scale policy pretraining. Their control evidence comes from fine-tuning pi-0.5 and evaluating three simulated benchmarks. It does not establish that adding RoboAtom labels to a general robot foundation-model pretraining mixture would produce the same gain. This distinction matters because downstream fine-tuning can exploit the annotation interface more directly than a heterogeneous pretraining objective.

The authors also exclude egocentric human video from the archived data. Human manipulation is much larger than robot trajectory data, but human actions are often brief and fragmented. Some last less than one second, and the current 2 FPS, 448-frame input does not provide enough temporal detail for every boundary. Extending the method to human video may require coarser action definitions, adaptive sampling, or a changed sequence representation. The same sampling limit affects long robot videos as action density rises.

An independent limitation is that PhysExtend depends on consistent ordered references and usable sensor-action signals. The second manual check reduces risk, but it also restores human cost for ambiguous tasks. New embodiments, missing state channels, different gripper encodings, and legitimate execution variants can force the pipeline back to rules or manual annotation. The method is scalable within the region where the reference sequence transfers; it is not universally automatic.

The paper does not compare System 2 planning latency, monetary cost, and failure recovery against the task-level policy in the main control table. A dual system may improve success while adding online inference delay and another source of semantic error. Real-world long-horizon operation would also test whether the skill schedule remains stable under occlusion, communication lag, object uncertainty, and repeated recovery attempts.

Finally, the archived paper statistics and the public release should not be conflated. The paper covers 16 sources, while the repository's current data card describes 12 downloadable subsets. A reproduction should inspect the specific counts, modalities, access restrictions, and licenses in that card. The architecture and benchmark can be studied independently, but the exact training mixture cannot be assumed from the headline total alone.

Conclusion: From a larger dataset to a shared action interface

RoboAtom advances robot data work from collecting more trajectories to defining a reusable supervision interface. The resource includes a vocabulary, segmentation protocol, reviewed seeds, physical propagation, quality gates, benchmark metrics, annotation models, and a policy-conditioning scheme. The meaning of 1,446,974 trajectories and 6,876,282 segments depends on whether those segments preserve a comparable meaning across sources and downstream tasks. Much of the paper's engineering exists to make that condition true.

PhysExtend demonstrates a credible low-cost path after human seeds: hold the ordered skill sequence fixed and use the robot's motion, gripper, and chassis signals to locate transitions. RoboAtom-Anno demonstrates that the resulting supervision can train a specialized, cheaper annotator that transfers to an external benchmark. The dual-system experiments connect the same label definition to online instruction selection. The common value is temporal and semantic alignment across all three uses.

The next questions are not only about adding more trajectories. A stronger system should estimate seed-consistency risk automatically, preserve local evidence in dense long videos, and test whether atomic-action supervision improves general robot pretraining rather than only task-specific fine-tuning. Those steps would determine whether RoboAtom remains a large high-quality annotation resource or becomes a stable component of a broader robot-learning paradigm.

Golden line: The scarce resource is not a sentence attached to a video; it is a skill interval whose meaning can be compared, propagated, and executed across trajectories.

Related Papers

AIRoA-0: Learning Long-Horizon, Precise Cross-Embodiment Manipulation from Handheld Data

AIRoA-0: Learning Long-Horizon, Precise Cross-Embodiment Manipulation from Handheld Data

AIRoA-0 turns 16,151 hours of bimanual handheld-gripper demonstrations into a vision-language-action policy that uses anchored tool motion, selective wrist perception, stage prediction, proprioceptive memory, and streaming decoding to carry one set of weights across four robot embodiments.

Yueh-Hua Wu, Kei Ota, Hanlong Li · 279 views · Nov 2, 2026
Bimanual ManipulationHandheld Demonstrations
Nov 2, 2026
SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation

SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation

SERF represents the environment and the articulated robot body as neural points in one shared latent space and updates them online. Environment points follow object-level rigid registration, while robot surface points are posed by forward kinematics. A shared residual decoder maps local point features to DINOv3 patch embeddings, and a two-stage Point Transformer produces eight map tokens from three robot-centered radii, both end effectors, robot-only, environment-only, and global branches. These tokens condition a VLA policy together with images, proprioception, and language. Across three long-horizon BEHAVIOR-1K tasks, SERF raises average progress from 44.0% for fine-tuned PI0.5 to 58.7%, and the paper also reports gains when labels and base poses are estimated onboard, when object-drop recovery is required, and in unvisited regions.

Sunghwan Kim, Byeonghyun Pak, Kehan Long · 284 views · Jun 11, 2026
VLAMobile Manipulation
Jun 11, 2026
KlaskTron: An Open-Source Platform for Physical Adversarial Multi-Agent RL

KlaskTron: An Open-Source Platform for Physical Adversarial Multi-Agent RL

Progress in robot learning, particularly physical multi-agent reinforcement learning (MARL), is currently challenged by a limited availability of accessible, standardized benchmarks. While simulation-based MARL has produced remarkable emergent behaviors, from coordinated team play to complex tool use, translating these advances to physical systems remains difficult. A key barrier is infrastructural: physical platforms are often prohibitively expensive, require specialized facilities, or lack open mechanisms and designs for reproducibility. We introduce KlaskTron, an open-source, low-cost (<$2500 USD), desktop-scale robotic testbed for adversarial MARL based on the dynamic tabletop game KLASK. The platform features dual CoreXY gantries with high-torque brushless motors, enabling the high-speed, precise manipulation required for competitive play. Crucially, we provide a complete ecosystem: hardware designs (CAD, BOM), a GPU-native digital twin in NVIDIA Isaac Lab for high-fidelity modeling and simulation, and a validated sim-to-real baseline using a learned neural actuator model. We demonstrate that this baseline enables zero-shot policy transfer, with trained agents exhibiting emergent adversarial behaviors. To ensure full reproducibility of our results, all project assets are released publicly: hardware CAD/BOM and assembly documentation at [https://idscethzurich.github.io/klask_hardware], inference software stack at [https://github.com/IDSCETHZurich/klask_software] and the simulation and MARL setup at [https://github.com/IDSCETHZurich/klask_rl].

Aswin Ramachandran, Jona Schulz, Maurus Derungs · 296 views · Nov 2, 2026
Physical Multi-Agent RLRobot Learning
Nov 2, 2026
D&A: Decoupling Description and Assessment Enables VLMs as Zero-Shot Robotic Reward Models

D&A: Decoupling Description and Assessment Enables VLMs as Zero-Shot Robotic Reward Models

Vision-Language Models (VLMs) hold promise for zero-shot video task-progress estimation in robot learning, yet existing frameworks frequently suffer from confirmation bias, hallucinating progress in failed executions due to implicit success assumptions or relative temporal ranking designs. Naive prompting of frontier models collapses into a binary success detector that fails to capture nuanced, non-linear progress and failures. While post-training approaches can mitigate this through explicit training, they demand expensive real-world robotics data and require computationally intensive post-training for every individual base model. To resolve these limitations, we introduce D&A (Describe & Assess): our approach decouples progress estimation into an objective, task-grounded textual description of scene dynamics followed by an assessment phase using explicit rubrics. By compelling the VLM to synthesize an explicit, text-based definition of absolute success before scoring, the framework anchors evaluations to an external, global criterion rather than the video's internal timeline. Empirical results demonstrate that our method calibrates zero-shot estimates and mitigates confirmation bias, providing fine-grained, partial progress metrics that accurately penalize failed executions while correctly interpolating successful runs. Being purely prompt-based, D&A generalizes across tasks, embodiments, and benchmarks—matching or surpassing post-trained reward models—and its reward estimates enable efficient downstream reinforcement learning.

Marius Memmel, Shawn Yin, Urvi Rutia · 278 views · Nov 2, 2026
VLMRobot Reward Modeling
Nov 2, 2026

As an Amazon Associate, we earn from qualifying purchases.