Text/image-to-video, physical and temporal coherence
Video generation adds one dimension to images: time. A bad still can be re-rolled, but clipping geometry, fingers multiplying, or a background melting under a camera push are all evidence about how well the model understands the physical world as it evolves - which is exactly the question world models try to answer and which embodied agents must internalise on real hardware. So progress here cannot be judged on picture quality alone. Temporal consistency (an object staying identical across frames), physical plausibility (gravity, collisions, occlusion) and controllability (camera moves, duration, first/last-frame constraints) are what decide whether this is a special-effects toy or a production tool. The link to AGI is clearest in this domain: a model that predicts "what happens next" accurately enough is one action interface away from a model that can plan in a real environment.
Gemini Omni: moving the unit of video delivery from one attempt to a conversational editing process
Google DeepMind's video and multimodal generation model, officially create anything from any input, starting with video: it edits video through step-by-step conversation, each edit building on the last and keeping the scene coherent, raising the unit of delivery from one shot to footage you can keep revising; the other two surfaces are real-world knowledge and arbitrary reference composition. Read the level with its votes: on the Artificial Analysis arena (2026-09-22) gemini-omni-1.1-flash is #1 for text-to-video at Elo 1516 with only 1,784 votes and a ±15 CI, while #2 is the sibling at 1513 with 26,576 votes and ±9, a 3-point gap far smaller than either CI, so statistically indistinguishable: the honest statement is that the Omni family ties for the top. For image-to-video minimax-h3 leads (1495, 57,112 votes) with Omni #2 at 1488 on 3,734 votes. Pricing: $1.50 per million input tokens, $9.00 text and $17.50 video output, 720p about $0.10 per second and a 30-second clip roughly $3. GA to paid-tier developers, none on the free tier. Closed, not self-hostable, not fine-tunable. Graded A (confirmed), the first A-grade video asset; A means the reading is credible and checkable, not undisputed first place.
1516Arena T2V Elo (1,784 votes)Confirmed · 2026-09
Kling 3.0 series: native audio, 4K frames, and parameterised camera language as the moat
Kuaishou Kling was among the first Chinese video lines to reach a level where it can take commercial work, and its lasting strength is not raw text-to-video but image-to-video plus camera control: lock composition and character in the image stage, then let the video model interpolate from that fixed keyframe, with first/last-frame constraints and push/pull/pan/truck/orbit/follow parameters producing predictable clips. The 3.0 series (official release-note banner: API fully available) brings three substantive jumps - audio generated natively (5s/10s/12s audio tiers), native 2K/4K for both image and video, and storyboarding plus element reference turning multi-shot coherence from an editing problem into something the model controls; the line-up is 3.0 / 3.0 Omni / 3.0 Turbo, Kling Image 3.0 and -omni, Native 4K Video and Motion Control. The arena reading, stated plainly: image-to-video kling-v3-pro is #19 (Elo 1354), behind MiniMax h3, gemini-omni, wan3.0 and seedance-2.5. No authoritative release date for the 3.0 series could be obtained, so the metric date is recorded only at the verifiable month, 2026-09. Closed, vendor API only; not independently benchmarked, graded C.
3.0 / Omni / TurboCurrent flagship seriesVendor Claim · 2026-09
Seedance 2.5: raising the unit of video delivery from one shot to a 30-second story beat
ByteDance Seed line for video generation. The 1.x tiers were silent short clips from text or image; from 2.0 the architecture is a unified multimodal audio-video joint generator taking text, image, audio and video as inputs; 2.5 raises the unit of delivery to a 30-second story beat with two further extensions, white-model control, green-screen editing, professional camera work and performance direction. Its third-party reading is the strongest part: on the Artificial Analysis arena (2026-09-22), dreamina-seedance-2.5-720p is #4 for image-to-video (Elo 1477), with 2.0 #5 (1479) and 2.5 #7 (1474) for text-to-video. Closed, reachable only through Dreamina / Volcano Engine / BytePlus, no self-hosting or fine-tuning; SeedVideoBench-2.0 is internal and cannot be reproduced. Not independently benchmarked; graded C (vendor-stated).
30sNarrative duration per generationVendor Claim · 2026-09
Wan: the open-weights baseline on the video side, letting the whole ecosystem iterate on its own hardware
Alibaba Tongyi's Wan releases video generation weights under Apache-2.0, making it the most important public baseline in the video domain. Its value differs from a closed service: not "best-looking output today" but that it is fine-tunable, reproducible and runnable on your own inference stack - the whole ecosystem of LoRAs, ControlNet-style conditioning, quantisation and few-step distillation builds on it. For an intelligence site, an open-weights baseline means the question "what can video generation do today" finally has a reference can be verified directly instead of trusting vendor-selected samples. The licence and the public repository, verified; picture quality and physical plausibility remain C-grade vendor claims.
FLUX: the line that moved image generation from "produce a picture" to controllable editing with open weights
Black Forest Labs' FLUX family generates images with a rectified-flow transformer and ships Apache-2.0 dev weights, making it the most important open baseline on the image side; the Kontext tier turned "edit this part of the image on instruction without disturbing the rest" into a usable capability. The collected record also covers the FLUX 3 launch, which the vendor positions as a world model unifying image, video and audio - a claim log as unverified.
Doubao Work: ByteDance's productivity agent, from decks to computer control
ByteDance's August 2026 productivity agent: PPT, documents and sheets, deep research, scheduled tasks — plus reading and writing local files, filling forms across pages, and phone-based task dispatch.
OpenMontage: Turn Your AI Coding Assistant Into a Full Video Production Studio
Open-source agentic video production system with no code orchestrator in sight: your AI coding assistant is the orchestrator, driven by YAML pipeline manifests, Markdown stage-director skills, 100+ registered tools and a seven-dimension scored provider selector. Research, proposal, script, scene plan, assets, edit and compose run as gated stages with creative approvals, budget governance and an auditable decision trail; with zero API keys it still cuts real-footage videos from Piper TTS, Archive.org, NASA, Wikimedia and free stock corpora, while the Backlot board shows every stage, storyboard gate and dollar live.
HyperFrames: Write HTML, Render Video — the Agent-Native Deterministic Video Engine
HeyGen open-sourced HyperFrames defines a video as HTML with data-* timing attributes: no build step, an index.html that plays and previews as-is. At render time headless Chrome seeks and captures frame by frame while FFmpeg encodes, so the same input always yields the same MP4. With 21 agent skills, the frame.md design-system translation layer and a reusable Catalog, coding agents such as Claude Code, Codex and Cursor can run the full plan-write-lint-preview-render production loop; an AWS Lambda path covers distributed renders. Apache-2.0 with no per-render fees.
Toonflow: open-source AI video generation workbench for short dramas and motion comics, from script and storyboard to shots on one infinite canvas
Toonflow productizes short-drama and motion-comic production as infinite-canvas nodes: text nodes write script and storyboard, image-generation nodes cut character turnarounds and scene plates, video-generation nodes render each shot from its references, while audio nodes and a 3D director stage carry dubbing and camera blocking. The model layer stays open - third-party APIs, local ComfyUI and local LLMs all plug in - and agents, MCP and a plugin market open the same canvas to machines. MIT licensed, deployable as desktop, Android, Docker or a plain server.
Wan2GP: run frontier generative models on a 6 GB GPU
A one-stop local generation app for video, image, audio and speech models, built on MMGP: block swapping, VRAM preloading and seven memory profiles let models larger than your VRAM run, starting at 6 GB.
LiveTalking: a real-time streaming digital-human engine
LiveTalking connects LLMs, TTS and three lip-sync backends into a real-time digital-human pipeline with WebRTC, RTMP, virtual-camera output, interruption, recording and multi-session support.
Pika H3 Flex: synchronized video and audio in six steps, driven by up to 12 references
An open adaptation of MiniMax-H3 for synchronized video and stereo audio, with endpoint-frame and up to 12 multimodal references, six joint evaluations, and explicit compute and licensing limits.
LiveKit Agents: Realtime Voice AI Agents as Programmable Participants
LiveKit Agents is an open-source framework for realtime voice AI agents (Apache-2.0, 15.0k stars): the agent joins a WebRTC room as a server-side programmable participant that can hear, see and speak. Four abstractions - Agent, AgentSession, entrypoint, AgentServer - sit on 83 pluggable STT/LLM/TTS/Realtime plugins and LiveKit Inference model strings, while conversation robustness is handled in-framework by a transformer turn detector, adaptive interruption detection and AMD answering-machine detection; MCP support, multi-agent handoffs, pytest event-stream assertions with LLM judges, and console/dev/start run modes connect it to production dispatch and the telephony stack.
The boundaries here are harder than in image generation, and harder to work around. Duration: single-pass generation generally stops at 5-10 seconds; longer pieces are stitched from first/last-frame relays or multiple segments, and the style drift and identity breaks at the seams need human selection - "one prompt, one coherent minute" does not exist yet. Physics: fluids, cloth, multi-person interaction and hand-object contact remain frequent failure points; the model has learned texture statistics that look physical, not conservation laws. Audio: most output is silent, voice and score come from a separate pipeline, and lip sync is only usable in dedicated products. Controllability: camera and storyboard instructions are followed poorly, so professional workflows lock composition via image-to-video (render keyframes first, let the model interpolate), which means real authorial control still sits upstream in the image stage. Evaluation is distorted: dimension-wise scores like VBench correlate weakly with "can I ship this clip", and vendor demos are always cherry-picked, so nearly every conclusion in this domain carries a vendor-claim grade. Cost and latency: a 5-10 second 720p/1080p clip typically takes tens of seconds to minutes of queueing, APIs are priced per second, and a single clip costs one to two orders of magnitude more than an image. Re-rolling three to ten times per shipped clip is normal - budget accordingly.
As an Amazon Associate, we earn from qualifying purchases.
