Omega-0 Official Code: Training, Inference and Unitree G1 Deployment for a Latent Predictive World-Action Model

In one line: paper, code, weights and real-robot data all released together

ω-0 (OMEGA-0) is the official implementation of the August 2026 paper "ω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation" by NTU's MARS Lab with Peking University, BAAI and HKUST(GZ). The target problem is concurrent loco-manipulation: given a language instruction, a stream of visual observations and proprioceptive state, the model emits whole-body action latents that the SONIC controller can execute directly, so walking and manipulating come from one policy instead of a locomotion stack bolted to a manipulation stack.

The repository went public on 2026-09-29 (initial commit 2e5c03b) with six README revisions the same day. The last commit, 2ca35a3 on 2026-10-13, ticks the "Publish pretrained checkpoints" TODO and adds the Hugging Face checkpoint badge. That update supplies the last missing piece of the open-source loop: the 7.42 GB pretrained checkpoint, uploaded to Hugging Face on 2026-10-12. The data piece was already in place, because the 4,828-episode ω-HOME dataset was ticked off and linked from the README on day one, 2026-09-29, and its files carry an even earlier timestamp (2026-08-17). MIT-licensed code, weights and data are now all public. For a 56-star early repo, that completeness is rare among humanoid paper codebases.

Repository baseline: small first-party code, heavy thirdparty

Metric	Reading (2026-10-15)
stars / forks / open issues	56 / 3 / 0
created / last commit	2026-09-29 / 2026-10-13 (README ticks checkpoint release)
language / license	Python / MIT (GitHub API reports NOASSERTION; see the license section)
first-party code	src/ 7,232 lines + real/ 6,062 lines + tools/ 260 lines (Python and YAML, ~13.6k total)
repo composition	6,790 blobs in the GitHub tree, 6,688 of them under thirdparty/ (G1 meshes and STLs, a 38 MB arm64 .deb, the Orin video sender, asio docs)
companion assets	HF checkpoint keycharon/Omega-zero (7.42 GB), HF dataset keycharon/omega-HOME (4,828 episodes)

Two readings deserve a pause. First, roughly 13.6k first-party lines cover five jobs: training, inference, an HTTP server, teleoperation collection and robot deployment. The abstraction is deliberately thin: src/ owns learning, real/ owns the robot, and the two talk only through a versioned protocol. Second, thirdparty holds 98.5% of the file count. Reading the project means reading src/ and real/, but deploying it means going through thirdparty/gear_sonic_deploy, the NVIDIA GR00T-WholeBodyControl deployment backend patched with Inspire hand support.

Architecture: conditioning, latent prediction, diffusion action head

The training and inference model is ACVJEPAModel in src/models/wam.py, and its forward pass has three stages. Stage one is conditioning: a Qwen3-VL backbone (default Qwen/Qwen3-VL-2B-Instruct) consumes a chat template of view tag, current frame and lowercased instruction, and the pooled last hidden state becomes the VLM feature; in parallel a frozen T5 encoder (t5-base by default) mask-mean-pools the instruction into the text conditioning for the diffusion head. Stage two is the visual latent space: a ViT-g-class frame encoder (hidden 1408, depth 40, 16 heads, mlp_ratio 48/11, RoPE, patch 16, 256x256 input) turns history frames into token sequences and stays frozen during fine-tuning; future-observation supervision never touches pixels. Instead the next video segment goes through the 48-channel VAE of Wan2.2 TI2V-5B (fast_wan_vae_encode: 256x256 area cap, patch [1,2,2], vae_stride [4,16,16]) and the resulting latent grid is the prediction target. That is the code form of "latent predictive" in the title: predict future visual representations, not future RGB.

Stage three is ACVJEPAPredictor: six stacked ACBlock layers, each maintaining three token streams. The prefix stream carries the VLM feature, T5 text, view embedding and current-frame tokens; the motion stream is 30 learnable action queries; the video stream is 1,792 learnable video queries. Within a stream each applies ACRoPE self-attention (causal for video); across streams there are three cross-attentions (motion reads prefix, video reads prefix, motion reads video). The video stream ends in predictor_proj, projecting back to 48 channels to meet the Wan latent target; the motion stream feeds the dit_ddim_xl action head, a DiT with adaLN-Zero modulation plus a flow-matching transport and cosine noise schedule, producing a 30-step by 66-dim whole-body action chunk. View conditioning uses two learnable embeddings, view_1 and view_3, with ego encoded as 1 and exo as 0 during training.

Stage	Config	Trained / frozen	Action representation	Key hyperparameters
Action-token pretraining	src/configs/vlm_pretrain.yaml	Qwen3-VL LLM and mm projector trained; vision tower frozen	FAST tokenizer with 2,048 bins, 75-dim SMPL, chunk=1, action 3 frames after the observation	lr 2e-5, 1M steps, bf16, flash_attention_2, 8192 sequence cap
WAM fine-tuning	src/configs/finetune.yaml	VLM and frame encoder frozen; only predictor and action head train (strict_predictor_only)	66-dim continuous actions, chunk=30, RTC on (max_delay 8)	lr 1e-4 cosine, warmup 1000, 80k steps, batch 32 per GPU, bf16

The objective lives in FinetuneObjective in src/training/losses.py and is refreshingly plain: total loss = jepa weight x MSE(predicted latent, Wan VAE target latent) + action weight x per-dimension weighted L1(actions), optionally plus an intra-chunk temporal-delta alignment term that discourages a flat, collapsed chunk. RTC (real-time chunking) shows up in training as a per-sample random delay whose first steps get sigma 0, telling flow matching that this prefix is already known; on the inference side src/inference/model.py writes the previous chunk's actions into the noise prefix before sampling. One easily missed default: ac_vjepa_force_null_state: true with state dropout 0.1, i.e. the headline recipe is a vision-only policy and the state channel exists for ablations.

Inference server: stdlib HTTP, one session, one in-flight prediction

src/inference/server.py skips FastAPI and builds on ThreadingHTTPServer plus a Service state machine with exactly three routes: /health, /reset, /act. The constraints are hard on purpose: one session and one in-flight inference at a time (lock contention returns 429 Model is busy); request_id deduplication rejects repeats with 400; a session replaced by /reset enters a retired set (cap 4096) and can never resume, and in-flight responses are invalidated immediately through a generation counter; request bodies cap at 128 MB and sockets time out after 2 seconds. For a robot control loop these refusal boundaries matter more than throughput: it is better to make the client retry than to let two command streams reach the robot.

WAMInference turns protocol observations into model inputs: an image history deque (default history_size 1), <EGO_VIEW>/<EXO_VIEW> view tags, lowercased instructions, a cached T5 encoding that only re-runs when the instruction changes, and sampling noise from a fixed seed 1039. The denormalized output is a (30, 66) float32 action chunk at a 33,333,333 ns period, i.e. a 30 Hz control cadence.

The real-robot stack: versioned protocol, module runtime, SONIC adapter

real/ is the most engineering-heavy part of the release. real/core/protocol.py defines a VERSION=1 msgpack host contract: TensorSpec validates dtype, shape and finiteness with a 64 MB per-tensor cap, and CommandMetadata carries session_id, sequence and a valid_until_ns deadline, so stale commands are dropped in the adapter layer. That is the first safety gate. real/core/runtime.py (988 lines) implements module workers, bounded StreamQueues that drop and count new samples when full, Ports with name-ordered lock acquisition, and event bindings that map key presses to actions.

Module (deploy.yaml)	Type	Rate	Responsibility
telemetry	ZmqReceiver	100 Hz	Subscribes SONIC g1_debug telemetry: joints, base quaternion, both hand states
camera	JpegZmqCamera	30 Hz	JPEG over ZMQ 5555, OPEN_CAMERA over TCP 13579, 640x360 left-eye crop
lifecycle	Lifecycle	50 Hz	Start/stop/planner transitions, 2.5 s controller settling, key event source
model	OmegaInferenceClient	100 Hz (inference every 8 steps)	Calls the inference server, RTC scheduling (overlap 9, 5 s response age cap), emits named SONIC commands
sonic	SonicCommandAdapter	100 Hz	Trigger interpolation to six-channel Inspire open/close targets, limit checks, model-vs-XR arbitration
sender	ZmqSender	100 Hz	Publishes pose v4 and command v1 to SONIC on ZMQ 5556

OmegaInferenceClient infers one of four action layouts from the server's action shape: latent66 (first 64 dims are token_state fed straight to SONIC, last two are hand triggers), smpl71, smpl75 and joints86. The latter three require a calibrated skeleton; the client runs forward kinematics to convert SMPL poses into 24 joint positions plus wrist targets before assembling SONIC's named inputs. A layout mismatch or missing skeleton raises at bind time, not after the robot starts moving. On the collection side, collect.yaml drives Pico body tracking through XRoboToolkit: three-point calibration, SMPL ordering for the first 22 joints, Inspire trigger interpolation, and recordings of state_action.hdf5 plus ego.mp4 (optionally exo.mp4 and exo_depth/), matching the ω-HOME directory layout exactly.

Data and weights: 4,828 episodes, an 8-GPU training archive

The ω-HOME dataset (HF keycharon/omega-HOME, MIT) is organized by task folder: 24 task directories, 4,828 episodes and 19,991 files in total. Each episode is a four-piece set of ego.mp4, exo.mp4, state_action.hdf5 and session_meta.json, and the paper reports 40+ hours overall. Counting session_meta.json files gives 4,828 episodes while the top level holds only 4,827 episode directories: one episode under push_chair was accidentally nested inside another (episode_0014_20260611_230102/episode_0016_20260611_230210/), a small packaging blemish that does not affect the data itself. The task distribution is remarkably even (187-210 episodes per task) and spans hang_clothes, clean_bed, mop_floor, move_table_with_human, retrieve_from_the_upper_fridge and more household loco-manipulation tasks, wider than the 11 tasks evaluated in the experiments section: the dataset is the collection surface, the experiments are the evaluation surface.

The checkpoint repo keycharon/Omega-zero was updated on 2026-10-12. model.safetensors is 7.42 GB of bf16 weights with architecture Qwen3VLForConditionalGeneration (text hidden 2048, 28 layers) and a vocabulary of 153,792: the extra entries over the native Qwen3-VL vocabulary are exactly the 2,048 <|a_i|> FAST action tokens plus the action start/end specials added during pretraining. The directory also holds eight random_states_*.pkl files, latest and zero_to_fp32.py, the signature of an 8-GPU Accelerate/DeepSpeed archive, and run_config.json records exp "Wan+JEPA", wandb group finetune, global batch 256 and lr 1e-4 cosine. The README's serve.yaml notes line up: this checkpoint loads through the original-format path, while exports produced by this repository's own training code load through model.artifact.

Reproduction cost and license boundaries

The environment side uses uv: uv sync --locked --extra train --extra deploy --python 3.10 --inexact, then flash-attn compiled from source. The robot side has a longer chain: SONIC runs inside the gear_sonic_deploy TensorRT container, Inspire hands need their own SDK environment, the ZED Mini needs the ZED SDK plus a C++ build of OrinVideoSender_jpeg on the robot, and the workstation needs Pico's xrobotoolkit_sdk. Training additionally requires five external assets: Qwen3-VL, T5, a V-JEPA frame-encoder checkpoint, the Wan2.2 VAE and a predictor initialization checkpoint, all referenced through ${VLM_PATH}-style environment placeholders. One conversion step still sits between the dataset and the training reader: ω-HOME ships raw recordings under two-level task/episode directories, while FinetuneDataset._build_episode_list globs annotation/*.hdf5 non-recursively and pairs each stem with an ego video in first/, so reproducing means first running tools/export_training_data.py per episode (it reads state_action.hdf5 and session_meta.json through EpisodeReader) to resample everything into the flat annotation/ + first/ layout. In short, inference and deployment are a closed loop; training from scratch means assembling a small zoo of large-model artifacts yourself.

The license needs three readings. The repository LICENSE is standard MIT plus a paragraph stating that third-party code keeps its original headers, and pyproject declares license = "MIT" with src/models/WAN_LICENSE.txt in license-files; that extra paragraph is precisely why GitHub's detector reports NOASSERTION. The HF checkpoint card is tagged apache-2.0, which differs from the code repository's MIT, so commercial use requires checking both. Everything under thirdparty/, including gear_sonic_deploy and XRoboToolkit-Orin-Video-Sender, keeps its upstream license and README.

Limits and verdict

Three explicit boundaries. First, there is no tests/ directory even though the pyproject pytest config points at one: the careful protocol and runtime design (lock ordering, deadline drops, generation invalidation) is currently self-attested by code with no regression net. Second, the hardware coupling is deep: Unitree G1 plus Inspire hands plus ZED Mini plus Pico teleoperation is the default assumption baked into configs and geometry code, and swapping embodiments means editing skeleton, URDF and crop parameters in three places. Third, the checkpoint is a research artifact; its own card states it should not run unsupervised without hardware and software safety mechanisms.

The case for inclusion is equally clear. This is one of the few humanoid repositories that releases paper method, training recipes, inference server, real-robot runtime, teleoperation collection, dataset and weights in a single drop, and the versioned protocol, session invalidation and RTC scheduling in real/ show product-grade discipline. For readers tracking whole-body control and world action models, it is both a reproducible baseline and an engineering reference worth stealing from.