Open-H3-Regenerate-2K: High-Resolution Video Refinement with Few-Step Diffusion

High-resolution video generation is often organized as two separate jobs. A diffusion model first settles the scene, motion and broad composition at a manageable resolution. A second model then redraws that draft at full size, adding the fine structure that a low-resolution sampler could not afford to compute. MiniMax's H3-Regenerate-2K performs this second job behind a hosted API, but its weights are not open. The open H3 base model can create a draft; it cannot perform the proprietary refinement step locally.

Open-H3-Regenerate-2K closes that gap. It keeps the H3 transformer and H3 video VAE frozen, trains one refiner LoRA on real high-resolution clips paired with degraded copies of themselves, and performs the entire refinement in three diffusion steps. The input can be a draft from H3 or another video model, including LTX-2 and SANA-Video. The output can be 768P, 1080P, 2K (2560 x 1440), or 4K (3840 x 2160). Clips from 5 to 15 seconds are processed in one pass.

The project comes from NVIDIA and the SoL-Engine effort. Its central claim is deliberately narrower than "open video generation now matches a closed service in every respect." It is that a frozen, open H3 stack plus a small trained refiner can reproduce this particular draft-to-high-resolution stage, accept a wider range of inputs and output sizes, and run quickly enough to be useful on local hardware. The authors report quality and speed measurements for that stage and release the inference path around it.

Why H3 needs an open second stage

H3 separates video generation into a cheap low-resolution stage and an expensive high-resolution stage for a practical reason. Sampling at 2K or 4K throughout generation would repeatedly evaluate a large transformer at enormous token counts. Instead, the base model can decide what is in the scene and how it moves on a smaller latent grid. A refinement model then receives the draft, preserves its content and temporal structure, and spends computation on the details that become visible only at the final size.

The hosted H3-Regenerate-2K API exposes the second stage but also constrains it. The authors state that the API accepts unmodified H3 768P output and returns 2K. It does not supply weights, so users cannot run the refiner on their own GPUs, combine it with a different draft model, extend it to other output resolutions, or study its internals. Open-H3-Regenerate-2K provides an open implementation of the same functional role while changing those operational limits.

This distinction also clarifies what is new here. The H3 transformer, the H3 VAE and the base model's ability to form a coherent draft remain upstream components. The contribution is the refiner around them: a target-grid latent upscaler, a trained LoRA, a three-step noise schedule, and an optimized inference stack. The work does not claim that a three-step refiner replaces H3's original content-generation process.

Property	Official H3-Regenerate-2K API	Open-H3-Regenerate-2K
Availability	Hosted service; model weights are not released	Open inference path built around H3
Accepted draft	Unmodified H3 768P output	H3, LTX-2, SANA-Video and other low-resolution drafts
Draft sizes shown	1344 x 768	672 x 384, 896 x 512, 1024 x 576, 1344 x 768 and other sizes
Native output	2560 x 1440	1344 x 768, 1920 x 1080, 2560 x 1440 and 3840 x 2160
Refinement steps	Not specified as an open implementation	Three full-resolution Euler steps
Clip length	Service-defined	5 to 15 seconds in a single pass

The four-stage refinement pipeline

The refiner wraps a frozen H3 transformer and video VAE. It receives the draft video and text prompt, then produces the same scene and motion at a larger target resolution.

1. Encode once. The H3 VAE encodes the low-resolution draft into its latent representation.
2. Enlarge in latent space. H3-Latent-Upscaler, a community 3D convolutional network, maps that latent directly onto the target grid.
3. Partially re-noise. The enlarged latent is corrupted only enough to let the refiner reconstruct missing detail without allowing the draft's composition to drift.
4. Run three steps and decode. The frozen H3 transformer, adapted by the refiner LoRA, performs three full-resolution Euler steps. The H3 VAE decoder returns the result to pixels; the fast path uses a 24-layer decoder from StreamRender-H3.

The key qualification is that latent upscaling alone does not add credible high-frequency detail. It carries layout, color, rough edges and motion onto the larger grid. The three diffusion steps are responsible for drawing the surfaces that now need to occupy many more pixels. This is why the system is called a regeneration model rather than a video super-resolution network that simply predicts a sharpened resize.

What the visual comparisons show

The published comparisons pair the same frame and crop. In the first examples, the input is an H3 base-model draft. The two output videos are both 2560 x 1440: the official H3-Regenerate-2K API result and the open model's result. The crop isolates the type of detail the refiner is expected to recover.

The site cannot reproduce the interactive magnifier used by the project page, so the local transcript includes three short clips from a representative H3 draft comparison: the input, the official API output and the open result. They are presented separately rather than as a synchronized player. The purpose is to show the type of temporal evidence being compared, not to create an alternate benchmark viewer.

The project also shows drafts from LTX-2 and SANA-Video, prepared at 1024 x 576. That experiment matters because it tests whether refinement is tied to the exact H3 base-image statistics. The authors report that the model preserves the alternate draft's scene while applying the learned detail distribution. The engineering implication is that the refiner can sit after different low-resolution generators as long as their latent layouts and content are compatible with the conversion path.

From any low-resolution draft to native 4K

The model is not tied to one input resolution or one output size. Low-resolution examples include 672 x 384, 896 x 512 and 1024 x 576. The draft is encoded once, its latent is enlarged to the selected target grid, and all three diffusion steps run at that target size. Native outputs listed by the authors are 1344 x 768, 1920 x 1080, 2560 x 1440 and 3840 x 2160.

For 2K, the reported latent grid grows from $36 \times 64$ to $90 \times 160$, a factor of 2.5 on each side, while retaining 24 channels and 37 latent frames. For 4K, H3-Latent-Upscaler is applied twice, each time within the range for which it was trained. The important alternative is a naive 4K resize: it makes the frame larger but cannot invent the detail required by the new pixel budget.

Stage	Examples used by the project	What changes
Input draft	672 x 384; 896 x 512; 1024 x 576; 1344 x 768	Content and motion are already decided
Native 768P	1344 x 768	One refiner pass at the 768P grid
Native 1080P	1920 x 1080	Latent and diffusion work move to the 1080P grid
Native 2K	2560 x 1440	Latent grows from 36 x 64 to 90 x 160
Native 4K	3840 x 2160	The latent upscaler is applied twice

Long clips without stitching

Open-H3-Regenerate-2K supports 5 to 15 second clips in one pass. The 15-second example contains 360 frames and is regenerated at 2K in a single run. This avoids the seam and identity-management problems introduced by splitting a shot into short windows, refining each window independently and blending the results. It does not mean that arbitrary-length video is supported; the reported range is the operating envelope.

Training: learn what degradation removed

The refiner follows Stage I of SoL-Refiner, the project's high-resolution continual-training recipe, with H3 taking the place of the original base model. Each training pair begins with a real high-quality clip at either 1280 x 704 or 2560 x 1440. The original clip becomes the target. A degraded copy becomes the condition: it is shrunk by a scale factor between 8 and 64 with an antialiased bilinear filter, then resized back to the original dimensions. Both versions are encoded by the H3 VAE.

For a high-resolution latent $z_{\mathrm{hq}}$, scale factor $s$, antialiased bilinear downsampler $D_s$, and resize operator $U_{1/s}$, the condition can be written as

$$ z_{\mathrm{cond}} =E_{\mathrm{VAE}}\!\left( U_{1/s}\!\left(D_s(x_{\mathrm{hq}})\right) \right), \qquad s\in\{8,16,32,64\}. $$

The H3 transformer is frozen. Only a refiner LoRA is optimized, using a flow-matching velocity loss over noise levels starting at $\sigma=0.91$. That starting level is aligned with the inference schedule, where the upscaled draft is re-noised to $\sigma_0=0.909$. At inference, the trained refiner LoRA is merged with a community distillation LoRA.

SoL-Refiner describes three stages: high-resolution continual training, reinforcement-learning post-training with ReFL, and DMD distillation to a one-step refiner. Open-H3-Regenerate-2K uses Stage I only. The later ReFL and DMD stages are explicitly listed as not used yet. This boundary matters when interpreting the results: the three-step model is not the endpoint of the full SoL-Refiner program.

Recipe element	Open-H3-Regenerate-2K setting
Trainable components	Refiner LoRA only; H3 transformer and VAE stay frozen
High-quality training sizes	1280 x 704 or 2560 x 1440
Condition construction	Antialiased bilinear down/up sampling by 8x to 64x
Training objective	Flow-matching velocity loss from $\sigma=0.91$ downward
Refiner LoRA	Rank 384, trained with a 5,000-step EMA
Inference LoRAs merged	Refiner LoRA, LightX2V FL2VA Turbo and LMS Sharp
SoL-Refiner stages used	Stage I only; ReFL and DMD are not used yet

For the continuous latent $z_0$, noise $\epsilon\sim\mathcal{N}(0,I)$, and noise level $\sigma$, the flow-matching construction and target are

$$ z_\sigma=(1-\sigma)z_0+\sigma\epsilon, \qquad v_\mathrm{target}=\epsilon-z_0. $$

The refiner learns a velocity field $v_\theta(z_\sigma,\sigma,c)$ conditioned on the enlarged draft and text context $c$. Training on degraded inputs teaches it to predict the direction back toward the real high-resolution latent, rather than merely reconstructing the low-resolution pixels.

Why upscaling happens in latent space

The refiner operates on the target grid, so the draft must first reach the output size. Pixel-space upscaling would require a costly VAE round trip: decode every low-resolution frame, resize it to 2K or 4K, and encode every frame again. At those dimensions, even moving the video through the VAE is a major part of latency.

H3-Latent-Upscaler avoids that round trip. It is a learned 3D convolutional network that maps the draft latent directly to a larger spatial grid while leaving the channel count and temporal length unchanged. At 2K, the reported transformation is

$$ z_{\mathrm{draft}}\in\mathbb{R}^{37\times 24\times 36\times 64} \longrightarrow z_{\mathrm{up}}\in\mathbb{R}^{37\times 24\times 90\times 160}, $$

where the dimensions denote frames, channels, height and width in the project's layout. For 4K, the same learned upscaler is invoked twice. The refiner then performs all three denoising steps at the enlarged resolution, so it is genuinely generating detail on the final grid.

Three steps instead of dozens

The speed comes from combining the draft prior with a distilled sampling path. LightX2V's FL2VA Turbo is a LoRA that lets H3 generate in four steps rather than dozens. Open-H3-Regenerate-2K merges Turbo together with the community LMS Sharp adapter and its own refiner LoRA. Unlike a four-step generation run that starts from pure noise, refinement starts from a draft that already contains layout, color and motion.

The enlarged latent is partly re-noised at $\sigma_0=0.909$. Given standard Gaussian noise $\epsilon$,

$$ z_{\sigma_0}=(1-\sigma_0)z_{\mathrm{up}}+\sigma_0\epsilon. $$

Three Euler updates then follow the schedule

$$ 0.909\longrightarrow0.857\longrightarrow0.667\longrightarrow0, $$ $$ z_{k+1}=z_k+(\sigma_{k+1}-\sigma_k)\, v_\theta(z_k,\sigma_k,c). $$

Classifier-free guidance is off. Each step is therefore one transformer call at the target resolution rather than a conditional and unconditional pair. The draft has already removed the need to decide what the scene contains, so the full-resolution model can spend its limited steps on detail and coherence.

Where the 3.9x speedup comes from

The authors measure one B200 from a 1344 x 768 H3 latent through latent upscaling, three refinement steps and VAE decoding to a 5-second 2K video. Generating the original draft, loading the model, text encoding and writing the MP4 are excluded. The same warm configuration takes 87.26 seconds with dense BF16 attention and the released VAE, and 22.45 seconds with SoL-Engine plus the faster StreamRender-H3 decoder. The ratio is 3.9x.

Inference configuration	Time for a 5-second 2K clip	Cumulative speedup	Main change
Dense attention, BF16, released VAE	87.26 s	1.0x	Baseline refinement path
Lossless kernels	81.38 s	1.1x	Strided cuDNN and torch.compile
Sol-Attn	41.38 s	2.1x	Sparse attention; saves 39.99 s
MXFP8 linear layers	38.12 s	2.3x	FP8 GEMMs, re-quantized per stage
Fused SwiGLU with MXFP8	37.85 s	2.3x	One-pass activation quantization
CUDA graphs	35.87 s	2.4x	One captured graph per refinement step
Fast 24-layer H3 VAE decoder	22.45 s	3.9x	Non-causal decoder distilled from the 36-layer VAE

Sparse attention is the largest individual saving, removing 39.99 seconds. Replacing the released 36-layer VAE decoder with the 24-layer StreamRender-H3 decoder cuts decoding from 14.99 to 1.58 seconds. Lossless kernels, MXFP8 linear layers with a fused SwiGLU quantizer and per-step CUDA graphs account for the remaining gain.

Stage	Baseline	With SoL-Engine and fast decoder
Refine: three steps at 2560 x 1440	71.27 s	19.75 s
VAE decode: 121 frames at 2560 x 1440	14.99 s	1.58 s
Latent upscale: 768P to 2K	1.01 s	1.05 s
Total	87.26 s	22.45 s

Do not merge this number with the separate 5.0x claim on the project page. The 3.9x result compares two configurations of the same open pipeline while holding the latent input and target fixed. The 5.0x figure compares the optimized open system with the official API on a single B200 under the authors' service-level comparison. They answer different questions and should remain separate.

Benchmarks: quality gains, but only on the authors' test set

The quantitative comparison uses RefinerBench-150, inherited from the SoL-Refiner work. It contains 150 videos spanning everyday scenes, sports and dance, animals, vehicles, landscapes, portraits and fast camera motion. Both systems receive the same input and produce 2560 x 1440 output. All numbers below are reported by the Open-H3-Regenerate-2K authors; this article has not run an independent reproduction.

VBench

VBench evaluates six dimensions on a zero-to-one scale. Following SoL-Refiner, the reported average is the unweighted mean of those six dimensions rather than a leaderboard-specific aggregate. Subject consistency tracks identity over time, background consistency tracks scene stability, motion smoothness and dynamic degree describe motion, and aesthetic and imaging quality assess the appearance of individual frames.

VBench dimension	Official H3-Regenerate-2K	Open-H3-Regenerate-2K	Difference
Unweighted average	0.802	0.814	+0.012
Subject consistency	0.918	0.924	+0.006
Background consistency	0.950	0.950	0.000
Motion smoothness	0.984	0.981	-0.003
Dynamic degree	0.765	0.752	-0.013
Aesthetic quality	0.569	0.604	+0.035
Imaging quality	0.629	0.676	+0.047

The open model is higher on aesthetic quality, imaging quality and subject consistency. Background consistency is equal. The official API is slightly ahead on motion smoothness and dynamic degree. In compact form, the reported average is

$$ \mathrm{VBench}_{\mathrm{avg}} =\frac{1}{6}\sum_{j=1}^{6}s_j, \qquad 0.814 >\;0.802. $$

This is not a clean sweep, and the pattern is informative. The largest gains are in image-level aesthetics and technical quality, which is consistent with training on real high-resolution pairs. Motion-related scores are essentially tied, with small losses of 0.003 and 0.013. A three-step regeneration model that starts from a draft can improve detail without improving, and may slightly attenuate, dynamic content.

UniPercept

UniPercept scores perceived quality from zero to 100 along three axes: aesthetics (IAA), technical image quality (IQA), and structure and texture (ISTA). The average is the mean of the three axes.

UniPercept axis	Official H3-Regenerate-2K	Open-H3-Regenerate-2K	Difference
Average	53.4	60.0	+6.6
Aesthetics (IAA)	59.1	67.3	+8.2
Quality (IQA)	59.5	67.8	+8.3
Structure and texture (ISTA)	41.6	44.8	+3.2

The open model is higher on all three axes, by about eight points on aesthetics and image quality and just over three points on structure and texture. The result supports the authors' claim that the refiner adds useful high-frequency structure, not only a pleasing global color treatment. It still covers only this benchmark and these two systems at 2K.

Blinded user preference

The user study contains 50 cases: 25 RefinerBench-150 videos and 25 prompts shared online, mostly on X. Each trial shows the low-resolution input as a reference and places the two 2560 x 1440 outputs in random A/B order. Participants choose the result they find better while considering clarity, detail, naturalness, artifacts and temporal stability. Each session covers ten cases. Twenty completed sessions yield 200 judgments.

System	Preferred judgments	Share
Open-H3-Regenerate-2K	174 / 200	87%
Official H3-Regenerate-2K	26 / 200	13%

The preference margin is large, but it should be read with the study size and composition in view. The authors report 20 completed sessions, not a broad demographic sample, and the task is a two-way forced choice involving mostly benchmark cases and online prompts. The study measures preference for visual results; it does not test controllability, factual consistency, long-form editing or downstream use in a production pipeline.

What the evidence supports, and what remains open

First, the project demonstrates a concrete route around a closed refinement service. A frozen H3 model plus a small refiner, an open latent upscaler and optimized inference can accept several draft sources, generate from 768P through 4K, and refine long clips without stitching. That is a meaningful systems contribution even before considering the benchmark scores.

Second, the results support a narrow quality claim on RefinerBench-150: the reported open configuration scores higher than the official API on several perceptual and image-quality measures and wins the authors' blinded comparison. They do not establish that the model is universally better. Motion dynamics, temporal artifacts outside this benchmark, unusual domains and human evaluation at other output sizes remain separate questions.

Third, the model is explicitly an intermediate version of the SoL-Refiner program. Only Stage I high-resolution continual training is used. ReFL reinforcement post-training and DMD one-step distillation are not included yet. A future version could therefore differ in both quality and speed; the current three-step result should not be described as the one-step endpoint.

Fourth, the speed measurements are warm, stage-specific and carefully bounded. The 3.9x gain covers latent upscaling, three refinement steps and VAE decoding; it excludes draft generation, text encoding, model loading and MP4 writing. The separate 5.0x comparison to the official service also needs its own conditions preserved. This kind of accounting is useful because high-resolution video latency is often dominated by decode and transfer as much as by the diffusion transformer.

For robotics and physical-AI teams, the practical value is in the infrastructure. Generated or simulated video used for training data, scene inspection, demonstrations or presentation can begin with an inexpensive draft and be refined only where distribution or publication requires full resolution. A 15-second single-pass path reduces the bookkeeping and seam risk of clip-by-clip processing. It does not turn the model into a robot policy, a physics simulator or evidence that synthesized video preserves contact and object state correctly; those properties still need task-specific validation.

The larger significance of Open-H3-Regenerate-2K is architectural rather than promotional. It shows how a closed high-resolution stage can be decomposed into reusable pieces: frozen base model, latent-space enlargement, small trained adaptation, short denoising schedule and system-level acceleration. That decomposition makes the model easier to inspect, deploy on local hardware and extend to different draft sources and output sizes.

Authors and attribution

Open-H3-Regenerate-2K is by Sensen Gao*, Haozhe Liu*, Yitong Li, Haopeng Li, Tian Ye, Jia-Wang Bian, Song Han and Enze Xie. Asterisks indicate equal contribution. The project is published through NVIDIA and the SoL-Engine research effort. Benchmark values, speed measurements, qualitative examples and the user study are reported by the authors; they are not independent evaluations by RobotWorld.

Source: Open-H3-Regenerate-2K: High-Resolution Video Refinement with Few-Step Diffusion, NVIDIA / SoL-Engine, October 2026.