Skip to content
←Back to Blog
After 600 experiments, here's what I discovered about JEPA world models
JEPAWorld ModelsLeWM

After 600 experiments, here's what I discovered about JEPA world models

Six hundred Push-T experiments show how 25-step subgoals, jump-cost latent memory, symmetry augmentation, and multi-goal scoring improve JEPA world-model planning without retraining.

Haddy AlchaerNovember 11, 202613 min read22 views
ZH

There is a serious debate in AI about what a world model should be. Generative world models predict the future by producing it, usually in pixels. Yann LeCun and many other researchers argue that this spends too much effort on details that do not matter for action. Most pixels are irrelevant to a decision, and many of them cannot be predicted reliably at all.

The alternative is JEPA, the joint-embedding predictive architecture. A JEPA world model has three parts: an encoder, a predictor, and a planner. The encoder turns each observation into a compact learned embedding. The predictor takes that embedding and an action, then predicts the embedding of the next state. The planner samples many action sequences through the predictor and selects the one whose predicted final embedding is closest to the goal. Because this loop runs in representation space, the model can ignore visual details that are not useful for control.

A robot observes a tower of blocks and the scene is encoded as a numeric vector
Perception. The JEPA encoder compresses an observation into an embedding rather than reconstructing every pixel.

The same representation is used to predict possible next states and to plan through them. The diagram below separates the three operations, but in a deployed system they form one loop: observe, predict in latent space, score candidate plans, act, and observe again.

An action changes a block-tower embedding into a predicted future embedding
Prediction. The predictor conditions the current embedding on a candidate action and estimates the embedding of the resulting scene.
Many candidate action sequences branch into the future and the planner selects one that reaches the goal
Planning. Candidate action sequences are rolled out inside the learned predictor, and the planner chooses the sequence whose predicted endpoint best matches the goal representation.

This year, a team including LeCun released LeWM, a JEPA world model evaluated primarily on Push-T. Push-T is a simulated manipulation task: a robot must push a gray T-shaped block across a table until the T occupies the green target pose shown in a goal image. Haddy Alchaer ran 600 experiments on LeWM to answer a practical question: once a small JEPA has learned useful dynamics, where do the next gains come from?

Push-T goal image containing a gray T block, a blue pusher, and a green target T
A Push-T goal image is not just a target for the block. It also contains the pusher at one particular position. That supposedly incidental detail turned out to affect success more than replacing LeWM's predictor with exact simulated physics.

Prediction is good enough; planning is the bottleneck

The first experiment isolated the predictor. On 250 long-horizon tasks with the goal 75 steps away, both versions received the same expert subgoals: frames from an expert demonstration every 25 steps. LeWM with its learned predictor succeeded 78% of the time. A version whose predictor was replaced by the simulator's exact physics succeeded 77%. Along the kinds of trajectories represented in its training data, LeWM was already as useful as perfect physics for planning.

That result changes where to look for improvement. If the predictor is not the immediate bottleneck, better control can come from changing how the trained architecture is used. Four planning changes produced substantial gains without one gradient step of retraining.

Planning condition Goal distance Success
LeWM, all 75 steps planned at once 75 steps 22%
Exact physics, all 75 steps planned at once 75 steps 42%
LeWM with an expert subgoal every 25 steps 75 steps, chained 80%
LeWM, direct 25-step tasks 25 steps 89.7%
Exact physics, direct 25-step tasks 25 steps 90.5%

1. Chain subgoals roughly 25 steps apart

Used exactly as published, LeWM succeeds on only 22% of tasks whose goal is 75 steps away. At 25 steps, it succeeds 89.7% of the time, within one point of the exact-physics planner's 90.5%. The model is accurate over the shorter distance and degrades over the longer one.

The usual explanation is compounding prediction error: small mistakes accumulate, so the model needs to be more accurate. The controlled comparison points to a different remedy. Exact physics on the same 75-step task improves success from 22% to 42%, a gain of 20 points. Giving the unchanged LeWM a new expert subgoal every 25 steps raises success to 80%, a gain of 58 points.

Comparison showing 22 percent for direct planning, 42 percent with exact physics, and 80 percent with subgoals
On 250 long-horizon tasks, shorter planning segments provide almost three times the gain of replacing the learned dynamics with exact physics.

This is hierarchical planning in a very concrete form. A high-level process selects a nearby subgoal, and LeWM performs the short-horizon search that it already handles well. The experiment establishes a useful planning range of roughly 25 steps and shows that longer tasks should be built by chaining searches inside that range.

One 75-step search succeeds 22 percent of the time while three 25-step searches reach 80 percent
One long search distributes probability across too many possible branches. Three short searches keep each prediction inside the horizon where LeWM remains accurate.

More search is not a substitute for better decomposition. Running the same planner four times and keeping the lowest-cost result consumed four times the compute but added less than one percentage point. The four attempts tended to succeed or fail together because repeated sampling did not repair the structure of a 75-step plan.

The expert subgoals prove that hierarchical decomposition is valuable, but a deployed robot cannot assume that an expert will supply a frame every 25 steps. The next method removes that requirement.

2. Build a latent memory graph with a cost on every jump

LeWM was trained on expert demonstrations. Its encoder can embed every frame, producing roughly 460,000 latent states. Alchaer connected these states into a graph. Each state links to the states that follow it in the same demonstration and to nearby states from other demonstrations in embedding space. To solve a long-horizon task, the planner searches this graph for a path from the current state to the goal and uses states along that path as subgoals.

This only works if latent distance has physical meaning. It does. Given one state that the robot can reach within 25 steps and one that it cannot, LeWM's embedding distance ranked the reachable state as closer 87% of the time. The representation therefore acts as an approximate reachability map, not merely a visual summary.

The first graph planner had a structural flaw. Jumping between nearby states in two different demonstrations cost the same as following a real transition inside one demonstration. The search exploited that shortcut and produced paths the block could not physically follow. In one analysis, 318 of 1,015 plans used six or more cross-demonstration jumps.

Adding a cost to every cross-demonstration jump fixed the problem. Using only LeWM's embeddings of past demonstrations, success on the same 250 long-horizon tasks rose from 62% to 85%. That matched the planner that was handed expert subgoals, without requiring an expert to choose subgoals at test time.

Demonstration trajectories converge on the same region while harder goals lie outside the demonstrated flow
The standard Push-T benchmark selects goals along trajectories already present in the demonstrations. A harder test takes the start from one demonstration and the goal from another.

The standard Push-T test is easier than it looks

The benchmark's protocol creates a hidden advantage. It starts partway through a demonstration and chooses a goal a fixed number of steps farther along the same demonstration. Every Push-T demonstration also ends with the block in roughly the same place. Standard goals therefore lie on paths the demonstrations already travel, heading toward a common endpoint. For 84% of standard goals, the memory graph contained a nearly identical state from another demonstration.

The sample is also small. On the official 50 episodes with a goal 50 steps away, LeWM succeeded 68% of the time. On 250 fresh episodes drawn by the same procedure, it succeeded only 44%. Changing only the planner's random seed flipped 38 of 250 episodes between success and failure. Those results are a warning against treating a small benchmark score as a stable estimate.

Alchaer's harder test takes each start from one demonstration and each goal from another, so no single demonstration connects the two states. A success requires the block to remain on target for the final 10 steps. The memory-graph planner scores 75% on standard tasks and 45% on this harder split. The remaining methods target that gap.

3. Fill gaps in memory with transformed demonstrations

A memory graph is only as useful as its coverage. When every stored state near the goal was deleted, success fell from 93% to 32%. Planning through past examples is powerful when a relevant example exists and breaks down when the goal lies outside the data.

Push-T offers a useful symmetry. The physics of pushing the block does not fundamentally change when the whole table is shifted, rotated, or mirrored. If the graph has no state near a goal, it can take a demonstration segment from elsewhere, transform it into the missing region, and plan through the transformed states.

With transformed segments, success returned from 32% to 85%. The transformation also makes a small dataset go farther: a graph built from 1% of the demonstrations plus transformed copies performed about as well as a graph built from 10%. On the harder cross-demonstration test, transformed segments raised success from 45% to 56%.

Demonstration segments are rotated and mirrored into an empty region near a target
When no stored trajectory reaches the goal region, transformed demonstration segments can fill the gap. Deleting nearby memory drops success to 32%; transformed copies bring it back to 85%.

The benefit depends on the direction of the missing goal. When the goal lies with the flow of the demonstrations, the plain memory graph already succeeds 69% of the time. When it lies across that flow, plain memory succeeds 39% of the time and transformed memory raises that to 54%. When the goal points against the data, plain memory scores 27% and transformed memory raises that to 46%.

Success with plain and transformed memory varies by whether goals follow, cross, or oppose demonstration direction
Transformed memories help most when demonstrations are thinnest or the requested goal points against the direction of the training data.

This is not general reasoning beyond the data. It is a targeted use of a known symmetry to create useful latent states. The method should transfer when a task has transformations that preserve its dynamics; it does not solve arbitrary extrapolation.

4. The goal image was scoring the wrong thing

The most surprising failure came from how a goal is represented. In an interactive Push-T demo, the operator places the block and chooses a goal image. To make a goal image, the pusher must appear somewhere, so it was placed just behind the desired block pose. In one run, the robot moved the block to within 14 pixels of the target, close enough to count as solved. It then pushed the block away while steering the pusher toward the position shown in the goal image.

This was not random behavior. Experts' demonstration frames include both the block's target pose and the pusher's incidental pose at that moment. A model that compares its predicted scene with the whole goal image learns to match both objects. On this task, matching the pusher exactly makes the block objective harder.

To remove predictor error from the question, exact physics was run twice on the same 50 harder tasks. The only difference was whether the cost included the pusher's position as well as the block's. Thirty tasks went from failure to success when the pusher was excluded, and none went the other way. A split this lopsided is expected by chance about twice in a billion. Since the physics was ground truth, the roughly 4x difference came entirely from how the goal was scored.

The planner that included the pusher did what it was asked. On the 30 flipped tasks, it placed the pusher within half a pixel of the goal while losing the block on the way. Requiring both the final T pose and the pusher's exact incidental pose is a substantially harder constraint.

LeWM cannot simply be told to ignore the pusher after training. Its distance is computed between embeddings, and the embedding of a goal image entangles the block and the pusher. There is no labeled "pusher" component to switch off. A practical workaround is to compare a prediction against many versions of the same goal: keep the block fixed and move the pusher to positions observed in stored demonstrations. Selecting the closest of 16 goal variants doubled success on the harder test, from 11% to 23%, without retraining.

A prediction is compared with sixteen goal images that keep the block fixed and vary the pusher position
Scoring against a set of valid end states avoids forcing the planner to reproduce one arbitrary pusher pose from a single goal image.

The principle applies beyond Push-T. A goal should describe a set of successful end states, not demand a pixel-level copy of one demonstration frame. "Put the cup on the shelf" is satisfied wherever the robot arm comes to rest after the cup is safely placed. A single goal image instead picks one arm position and asks the planner to match it. A good goal accepts every end state that solves the task while still encoding constraints the task genuinely requires, such as coming to rest at the end.

Large demonstration datasets already provide much of this set. The same object may appear in the same place many times while the robot occupies different poses. Scoring against the closest compatible goal broadens the acceptable endpoint without changing the world model.

What remains open

None of these four findings changes LeWM's weights. They are inference-time changes to planning, memory construction, data augmentation, and goal scoring. The results suggest that larger JEPA world models which plan in the same way can inherit similar gains without extra training cost.

The main open problem is generalization beyond perfect demonstrations. On the harder Push-T test, the best planner holds the block on target 56% of the time. Most remaining failures ask for goals that run against the direction of the data. Closing that gap requires a higher-level system that can propose its own subgoals outside the demonstrated distribution. Existing learned hierarchies such as H-JEPA have not yet solved this: on Push-T they match or only slightly exceed flat LeWM.

The broader lesson is that a small JEPA trained in a few hours can predict well enough to rival the simulator along familiar trajectories. Its immediate bottleneck is not always representation quality. The way a planner decomposes time, navigates memory, uses symmetry, and defines a goal can matter more. What LeWM still cannot do is choose useful subgoals where no demonstration has gone before.

Source: Haddy Alchaer, "After 600 experiments, here's what I discovered about JEPA world models" (November 11, 2026).


References

  1. Y. LeCun, A Path Towards Autonomous Machine Intelligence. OpenReview, 2022.
  2. L. Maes, Q. Le Lidec, D. Scieur, Y. LeCun, and R. Balestriero, LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. arXiv:2603.19312, 2026.
  3. C. Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS, 2023.
  4. W. Zhang, B. Terver, M. Rabbat, Y. LeCun, and R. Balestriero, H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning. arXiv:2610.06805, 2026.

Source:Haddy Alchaerhttps://haddyalchaer.com/jepa

As an Amazon Associate, we earn from qualifying purchases.