DyCraft: An Agentic System for Long-Horizon 4D Scene Generation

Authors: Jeff Tan, Hang Dai, Zeqian Long, Yanzhe Lyu, R. Kenny Jones, Shangzhe Wu, Chen Geng, and Jiajun Wu

Paper and project: arXiv:2610.40754 · Project page

Code status: The official implementation is public at jefftan969/dycraft. This analysis checked commit 4d0fd0d, released under Apache-2.0.

One-Sentence Summary

DyCraft turns long-horizon animation from a single code-generation request into a staged workflow: expand the brief, build an explicit scene, approve key 3D states, author motion between them, verify every frame, and render only after the required checks pass.

Why This Problem Is Hard

A four-dimensional scene combines persistent geometry with time-varying motion. For film, games, and robot learning, the result must preserve object identity, camera choices, articulation, contacts, and editable structure. Pixel-based video generators can produce convincing frames, but they offer weak handles for correcting one object, joint, or time interval. DyCraft therefore represents the film as Blender code: an editable artifact whose scene graph, animation curves, simulations, and cameras can be inspected directly.

Code alone does not make long-horizon authoring reliable. A general coding agent tends to script the entire performance in one pass. That works for a short object turntable, but not for a minute-long sequence with multiple shots and interacting characters. Layout errors cause grasp failures, grasp failures create floating objects, and motion interpolation can hide the original cause. The agent sees the final frames, while the failure is distributed across scene construction, pose selection, timing, and physical constraints.

The paper locates the bottleneck in the workflow rather than in the agent's ability to write Blender Python. Bare agents produce interpenetration, missing contact, failed grasps, discontinuous motion, unusable layouts, and fake dynamics. Figure 2 makes the pattern concrete: adding more visual spot checks does not reliably reveal these errors because a defect may occupy only a few frames and may be hidden by a camera angle.

DyCraft borrows pose-to-pose animation from traditional production. An animator first establishes key poses, staging, and state transitions, then fills the intervals. The paper transfers that practice to 3D coding agents: long motion is decomposed into static 3D states that can be rendered and checked individually, followed by continuous motion through approved states. Temporal constraints become geometric, support, articulation, and event constraints that can be diagnosed locally.

The system is not a new end-to-end 4D network. It extends a general coding-agent harness with planning, CAD, storyboard, motion, and verification operations while retaining native file editing, Python, shell, and rendering. This hybrid design leaves repair decisions to the agent, but it also imposes hard gates where unverified completion would otherwise be accepted.

System Formulation

The paper does not present numbered display equations. The expressions below are operational reconstructions of the method description and the public scorer, not quotations of numbered paper equations. A task can be described by its prompt, required states, objects, and events:

$$\mathcal{T}=\left(q,\{s_k\}_{k=1}^{K},\{o_i\}_{i=1}^{N},\{e_j\}_{j=1}^{M}\right)$$

Here $q$ is the sparse text prompt, $s_k$ is a shot or storyboard state, $o_i$ is an object requiring construction or control, and $e_j$ is an event that must occur. The final film is produced through a chain of transformations:

$$\mathcal{F}=\operatorname{Render}\left(\operatorname{Motion}\left(\operatorname{Storyboard}\left(\operatorname{SceneIR}\left(\operatorname{Plan}(q)\right)\right)\right)\right)$$

This chain is not a one-way waterfall. Verifiers return diagnostics to the storyboard and motion layers, and the agent may repair the scene, a state, or an interval before rerunning checks. Figure 3 illustrates the loop with a robot carrying a wide panel through a narrow doorway. The system expands the prompt, builds the scene, checks a key carrying pose, detects an arm passing through the torso, and then repairs the motion before rendering.

flowchart TB
    Prompt[User prompt] --> SceneRewrite[Fresh-context scene rewrite]
    Prompt --> MotionRewrite[Fresh-context motion rewrite]
    SceneRewrite --> Plan[Shot and event plan]
    MotionRewrite --> Plan
    Plan --> SceneIR[CAD and SceneIR authoring]
    SceneIR --> Boards[Accepted 3D storyboard states]
    Boards --> Gate{All required checks pass?}
    Gate -->|no| Repair[Repair scene or board]
    Repair --> SceneIR
    Gate -->|yes| Motion[IK simulation and motion tools]
    Motion --> Verify[Per-frame verifiers]
    Verify -->|fail| RepairMotion[Repair motion or storyboard]
    RepairMotion --> Motion
    Verify -->|pass| Render[Final Blender render]

Three design choices shape the engineering behavior. Storyboards are executable prerequisites rather than documentation. Verification produces actionable evidence such as penetration depth, separating axes, failing joints, and unmet events. The main agent may delegate object, scene, or motion work to helpers, but it remains responsible for integration and final acceptance.

Method in Detail

Fresh-Context Prompt Rewriting

Sparse prompts describe outcomes but omit production details. DyCraft therefore uses separate fresh-context rewriters for scene and motion. The first selects an organizing principle and fills in the objects, materials, geometry, and spatial relationships needed by the scene. The second expands the action into believable transitions, contacts, and timing. Their relation can be summarized as:

$$s,m=\left(\operatorname{SceneRewrite}(q),\operatorname{MotionRewrite}(q,s)\right)$$

The rewritten briefs are staged with the original instruction and system prompt. The main agent then produces the action plan, shot sequence, object set, motion requirements, and event list. The ablation that removes rewriting still has access to all later tools, but produces less detailed scenes and motion. This is evidence that rewriting contributes planning information rather than merely making the prompt sound more elaborate.

CAD and Explicit Scene Structure

Bare agents often model complex objects as collections of cubes, cylinders, and hand-written surfaces. Such approximations may look acceptable in one frame, but a character that must grasp, open, or pass through them exposes every gap and overlap. DyCraft adds a boundary-representation CAD interface and extends it with parts, articulated joints, named coordinate frames, posed instances, and joint-limit inspection.

The benefit is structural, not just geometric. Named frames turn a semantic instruction such as “align the gripper with the handle” into a computation over local axes. Articulated joints expose parent-child relations and rotation limits. Solids provide the geometry required by collision and support checks. Figure 4 compares a marble run: the DyCraft version has continuous tracks, a funnel, gears, and a lift, while the bare-agent version contains empty shells, gaps, and incorrectly assembled parts.

The CAD layer is additive. Blender remains available for skinned characters, cloth, fluids, and other deformable effects that are difficult to express in CAD. The agent can choose the appropriate representation for each object while keeping the scene, cameras, animation, and verification inside one project.

A Hard Storyboard Gate

Before editing continuous motion, the agent must author key states in 3D. The harness renders those states through the film's cameras and runs verifiers on their geometry. Motion editing for a segment is blocked until every relevant state has passed checks and been visually accepted. The gate is a conjunction:

$$G_{\mathrm{board}}(\mathcal{S})=\prod_{s\in\mathcal{S}}\mathbf{1}\left[\operatorname{review}(s)=1\land\operatorname{checks}(s)=1\right]$$

If any state is missing, stale, or failing, the gate is zero. This prevents interpolation across an unchecked pose. It is cheaper to discover an unreachable handle or an occluded camera in a static state than in the final film, because the static state has no sampling error across the interval.

The public implementation matches the paper's requirement. At animation/storyboard.py:17, require_storyboards() identifies missing states, design-stale states, and objects added since review, then rejects animation editing:

def require_storyboards(self, workspace, check=None):
    plan, build = self.plan_of(workspace), self.build_of(workspace)
    boards = read_json(workspace / "storyboards.json", {})
    missing = missing_states(
        plan, boards, build["design_id"], build.get("design_objects")
    )
    appeared = self.appeared_since_review(workspace, check)
    missing = sorted(set(missing) | set(appeared))
    if missing:
        raise ValueError(
            "these states need a passing, accepted storyboard before an "
            "animation edit: " + ", ".join(missing)
        )

The important detail is dependency invalidation. A scene repair that adds an object visible in an earlier shot invalidates the affected board. Without that check, a long-running agent could approve a state, make a local scene edit hours later, and continue animating against a state it never reviewed.

Domain Motion Tools

Approved states define anchors, not natural motion. DyCraft supplies humanoid motion guidance, footstep planning, joint-direction constraints, inverse kinematics, rigid-body simulation, fluid simulation, and Blender baking. Figure 5 shows four common failures of hand-authored motion: the character's center of mass leaves the support region, a limb stretches beyond its length, physical bodies interpenetrate, and a drink behaves like a rigid cylinder.

The tools do not make every creative decision. The agent still chooses shots, solvers, contact targets, and repair strategies. The harness converts recurring physical constraints into callable operations and measurable evidence. “Move the foot before shifting the torso” is not left as vague prose; it becomes part of inverse-kinematics targets, balance, contact, and verification.

Required Checks and Advisory Evidence

DyCraft separates required checks from advisory diagnostics. Animation well-formedness, joint limits, penetration, and event satisfaction can block clean completion. Continuity, support margin, contact slip, and anatomical limits provide additional evidence for the agent's judgment. The public code defines the required set as follows:

REQUIRED_VERIFIERS = (
    "AnimationWellFormed",
    "JointLimits",
    "NoPenetration",
    "EventsSatisfied",
)

def required_checks_pass(reports):
    by_name = {r["name"]: r for r in reports}
    return set(REQUIRED_VERIFIERS).issubset(by_name) and all(
        by_name[n]["verdict"] in {"pass", "not_applicable"}
        for n in REQUIRED_VERIFIERS
    )

This implementation appears at animation/planning.py:412. The report set must contain every required verifier, so a crashed or missing check cannot silently pass. not_applicable is reserved for a verifier whose target set is empty by construction, such as a scene with no articulated joints. Missing evidence and legitimate non-applicability are treated differently.

Figure 6 shows repairs for body-furniture penetration, floating objects, and an elbow bending in the wrong direction. These are exactly the errors that a few sampled camera frames can miss. Per-frame geometry and joint trajectories expose them, and the agent uses the diagnostics to revise a state or motion segment before compiling again.

How the Public Scorer Defines Physical Error

The reported penetration rate is the fraction of evaluated frames containing at least one counted dynamic mesh penetration, averaged over the 24 films:

$$P_m=\frac{1}{T_m}\sum_{t=1}^{T_m}\mathbf{1}\left[\exists(A,B):C_{ABt}=1\right]$$ $$P=\frac{1}{24}\sum_{m=1}^{24}P_m$$

$C_{ABt}$ marks a counted penetrating pair at frame $t$, and $T_m$ is the number of evaluated frames in film $m$. A fixed 5 mm threshold would miss a thin card, while a fixed 1 mm threshold would flag mesh noise. The public scorer therefore adapts the threshold to the thinner body:

$$\tau(A,B)=\max\left(1\text{ mm},\min\left(5\text{ mm},0.5\min\{\ell_A,\ell_B\}\right)\right)$$

The scorer also excludes persistent construction overlaps, directly connected structural parts, liquids and effects, and consumed food. It handles initial overlap, self-collision, movement, rigid coupling, and joint adjacency. The paper gives a compact definition in Section 4.4; the released scoring code contains the full operational boundary conditions and should be used for reproduction.

Floating is measured through a per-frame support graph. A moving body must have a legal path to ground through resting contact, pivots, structural attachment, stable carrying, or a deformable carrier. Self-propelled or buoyant exemptions are defined per task. The rate is:

$$F_m=\frac{\sum_{b,t}D_{bt}\mathbf{1}\left[\neg\operatorname{Reach}_G(b,t)\right]}{\sum_{b,t}D_{bt}}$$

$D_{bt}$ identifies an evaluated dynamic body-frame, and $\operatorname{Reach}_G(b,t)$ indicates whether that body reaches a legal support source in the current graph. This is not a visual gap heuristic: contact tolerances and graph edges must support the body.

Joint violations can be expressed as a frame-level out-of-range indicator:

$$J_{j,t}=\mathbf{1}\left[q_{j,t}\notin[\ell_j,u_j]\right]$$

$q_{j,t}$ is the joint value, and $[\ell_j,u_j]$ is its allowed interval. The arithmetic is simple, but correct joint axes and parent-child structure are essential. A motion can remain inside a numerical range and still look impossible if the axis itself is wrong.

Experiments

DyBench contains 24 long-horizon, multi-shot prompts, split evenly across Unitree G1 humanoid tasks, human or animal characters, and physical mechanisms. The tasks require multiple events and interactions rather than a single isolated action. Examples include cooking eggs, assembling an office chair, loading a kiln, mopping toward an exit, carrying a wide panel through a doorway, and operating a bottling line or clock escapement.

The baselines are a bare agent using the same GPT-6-Astra Codex harness and VIGA, which uses a code-render-inspect loop. The bare agent receives a condensed description of the quality standards but not DyCraft's prompt rewriters, plans, storyboard gate, domain tools, or verifiers. The comparison therefore tests the workflow rather than a stronger foundation model.

Preference aspect	DyCraft vs. VIGA	DyCraft vs. bare agent
Overall 4D realism	92.0%	75.6%
Physical plausibility	88.5%	70.1%
Motion quality	88.5%	71.0%
Overall preference	89.7%	72.3%

The user study recruited 109 Prolific participants. Each person compared five film pairs and answered three binary questions per pair, producing 1,538 judgments. The questions separately address whether an animation resembles a real filmed scene, maintains contact and support without penetration or floating, and shows natural motion. DyCraft wins over VIGA in approximately 88.5% to 92.0% of judgments across the first three aspects.

The comparison with the bare agent is especially informative because both systems use the same base model. DyCraft is preferred overall in 72.3% of judgments and has a 19.4-point advantage in physical plausibility and a 17.5-point advantage in motion quality. The result supports the paper's claim that the harness, not merely model strength, accounts for a substantial part of the improvement.

Physical metric, lower is better	Bare agent	VIGA	DyCraft
Penetration rate	80.1%	74.9%	13.0%
Floating rate	15.7%	26.1%	2.7%

The automated metrics show a larger gap. Across the 24 tasks, DyCraft reduces penetration from 80.1% for the bare agent and 74.9% for VIGA to 13.0%, and floating from 15.7% and 26.1% to 2.7%. The remaining penetration is not negligible. The authors report that most residual cases occur between characters and held objects, such as a dog's mouth entering a newspaper, or between layered parts such as stitching on a weight belt.

These metrics should be read as targeted physical checks, not a total quality score. They do not measure event semantics, visual composition, character appeal, or material realism. Their value is that they expose failures which sampled visual inspection can miss and which the system can often repair automatically. Human preference supplies the broader quality judgment.

Qualitative results explain the source of the preference. VIGA often produces short clips and omits events, while the bare agent attempts the full brief in a simplified scene with kinematic motion. DyCraft preserves multiple shots and event order while simulating the relevant dynamics. Figure 1 shows office-chair assembly, piano moving, and parcel sorting, indicating that the system is not limited to a single body type.

The ablation removes prompt rewriting, 3D storyboarding, domain motion tools, or external verifiers one at a time. Every variant reveals planning or physical-plausibility errors, while the full system performs best. The paper presents this ablation mainly qualitatively, so it does not estimate each component's separate effect on penetration and floating. The system-level result is strong, but the quantitative attribution among components remains less complete.

Cost and Practical Implications

The released README gives a useful production-cost signal that is not emphasized in the paper. With GPT-6-Astra in Codex, a one-minute film has a median cost of about 11 agent-hours for the main agent and 15 agent-hours when helper agents are counted. The default run stops after 12 hours, so an overlong prompt can fail before completion. Scoring the 24 DyCraft films takes roughly 19 CPU-hours, compared with about 8 for the 24 bare-agent films, because per-frame geometry dominates the cost.

Engineering measure	Public value	Interpretation
Film authoring time	~11 agent-hours/minute	Median, main agent only
Including helpers	~15 agent-hours/minute	Total time after delegation
Default run limit	12 hours	Long briefs require splitting or a higher limit
Scoring geometry	~19 CPU-hours / 24 films	Frame-by-frame collision and support

DyCraft is therefore closer to a high-value authoring system than a real-time generator. It fits robot simulation, explanatory animation, and mechanism visualization where editable geometry, contact, and persistent structure matter. A system aiming only for a few atmospheric seconds may still prefer a video model. The paper's claim is more specific: when control and physical consistency dominate, explicit code and staged verification are a viable generation route.

Limitations

The authors explicitly report residual penetration between characters and held objects and between layered components. These cases involve deformation, local contact tolerance, and semantic notions of acceptable overlap. The system blocks clean completion when required verifiers fail, but it also supports delivery with exceptions as a fallback. Relationships outside the verifier's model can therefore survive into the final film.

The evaluation also relies primarily on human preference because reliable automated metrics for dynamic scene quality remain difficult. The 109-person study provides a strong comparative signal, but penetration and floating do not cover narrative order, object identity, camera quality, material behavior, or whether every event is semantically correct.

A third limitation is dependence on the underlying model, Blender, available character assets, and long agent runs. All experiments use GPT-6-Astra. A weaker agent may still have access to the CAD and verifier APIs, yet fail to interpret complex diagnostics or perform multi-step repair. The median hours per minute of film show that quality is produced through extensive search and iteration, not a single generation call.

Finally, DyBench contains 24 tasks. It covers humanoid, character, and mechanism domains, but not the full complexity of production animation. The paper does not establish how the system behaves with many interacting characters, long dialogue-driven scenes, unusual rigs, or adversarial asset failures. Broader production deployment would require testing those conditions and measuring how often a correct repair is found within a fixed budget.

Conclusion

DyCraft's main contribution is a decomposition of long-horizon 4D generation into steps that a coding agent can plan and verify. Prompt rewriting supplies detail, CAD supplies explicit structure, storyboards convert temporal reasoning into static checks, domain tools generate physical motion, and verifiers prevent unchecked output from being treated as complete.

The experiments support this decomposition with strong preference gains and large reductions in penetration and floating. More broadly, the work shows that generated 4D content need not exist only as a video tensor. It can be a persistent combination of scenes, objects, joints, events, cameras, simulations, and code. That representation is particularly relevant to robotics, where editable scenes and physical constraints are part of the target artifact rather than post-processing details.

The deeper lesson is not that an agent can write Blender code. It is that long-horizon generation becomes tractable when every key state can be observed, verified, and revised before time is filled in.