From One Prompt to a Rigged 3D Performance: Inside prompt2rig's Video-to-Mixamo Pipeline

Why the app generates a performer first

The browser starts with a generated photograph of one young adult in a neutral T-pose-like stance: full body visible, light-grey shirt, dark trousers, plain studio background, soft even lighting. This image is used as the first frame for image-to-video generation. The same character and framing can then be reused across prompts, while stream mode uses the previous clip's final frame instead.

That choice makes the downstream problem easier in three ways. It gives the video model a stable identity, it makes the person large enough in frame for pose crops, and it avoids scene cuts or camera motion. The prompt builder also appends constraints such as "both arms clearly visible and held away from the torso", "no turning around", "no flips", and "static locked-off camera". These are not cosmetic instructions. They prevent exactly the ambiguities that a monocular estimator cannot reliably resolve.

Live path: useful latency, limited geometry

MediaPipe Pose runs in the browser while the generated video plays. It returns 33 body landmarks and a set of world-space landmarks. The app converts those coordinates into three.js space, filters them with an adaptive One Euro filter, estimates torso scale from 2D shoulder and hip positions, drives the Mixamo rig, and records snapshots into a live animation track.

The strength of this route is latency. There is no server round trip and no multi-gigabyte checkpoint to start. The user sees a moving character almost immediately after the clip is ready. The weakness is geometric quality: monocular landmarks can jitter, depth is uncertain, limbs can exchange identity when they cross, and the world-space points are not a full body model. The app exposes smoothing controls and stabilizes root motion, but no filter can invent the missing depth.

HQ path: recover a body model over the whole clip

The HQ worker is deliberately shaped around clips generated by prompt2rig. A single-person, static-camera clip does not benefit from expensive multi-object tracking, so the worker batches a YOLO11m detector, keeps the largest person in each frame, fills short detection gaps, and smooths the crop boxes. Frames are reused between detection and cropping instead of being decoded twice. The long side is capped at 854 pixels because GVHMR ultimately operates on small person crops, and the clip is resampled to 30 fps.

GVHMR then recovers a sequence of SMPL parameters with a fixed-proportion body and a gravity-aware root trajectory. The worker returns the global orientation, the first 63 body-pose values per frame, translation, and a measured leg length. It also returns the first frame's orientation relative to the camera so chained clips can preserve a consistent heading. Because GVHMR looks at the clip as a sequence rather than solving each frame independently, it can reduce the limb stretching and forearm twisting that appear in frame-by-frame landmark retargeting.

The implementation contains several application-specific speedups. Batched detection replaces frame-by-frame tracking, ViTPose runs without its mirror flip-test by default, CUDA work stays on one warm thread, and the input resolution is reduced before body recovery. The README reports roughly 10 seconds for the stock GVHMR path and about 3.4 seconds after these changes for a 5-second clip on an RTX 4090.

Hands are tracked as a separate geometric problem

Body models usually stop at the wrist. prompt2rig therefore runs a hand detector and pose model on the full-resolution source clip while GVHMR processes the body. It supports WiLoR with MANO, or MediaPipe Hand Landmarker as a lighter fallback. Each detected hand is represented by 21 keypoints in the MediaPipe and OpenPose ordering: wrist, four thumb points, and four points for each remaining finger.

Assigning those hands to the body is part of the work. The hand model reports handedness inconsistently, while the body keypoints provide a more stable notion of left and right. The code assigns each detected hand to the nearest body wrist, mirrors left-hand crops for WiLoR's right-hand prior, un-mirrors the resulting pose, and resamples the track to the body motion's frame rate.

Camera coordinates also have to be rotated into GVHMR's world frame. The worker constructs a per-frame mapping from the world and camera-space pelvis orientations:

$$R_{w \leftarrow h}(t) = R_{w \leftarrow c}(t)R_{c \leftarrow h}(t)^T$$

Here the two source rotations describe the body's pelvis in world and camera coordinates. The resulting matrix is applied to wrist-relative hand points, which allows the fingers to move with the recovered 3D world while keeping the palm attached to the wrist.

Missing hand detections are tolerated rather than hidden. Gaps up to 0.4 seconds can be interpolated, longer invalid stretches remain marked as invalid, and each valid run is smoothed independently with a Gaussian filter using a standard deviation of 1.2 frames at 30 fps. When a hand is absent, the retargeter produces a relaxed curl instead of a rigid flat hand.

The retargeter drives 15 bones per hand, three for each finger, by aiming each bone at the measured segment direction. It does not force every finger to fold along one palm normal. Instead, it constrains backward bending and caps knuckle and joint bends at anatomically plausible angles. WiLoR's MANO output is treated as already valid and bypasses the stronger MediaPipe joint limits.

Retargeting: rotations first, dimensions second

The body path maps the 22 SMPL joints directly onto Mixamo names such as Hips, Spine, LeftArm, RightUpLeg, and LeftToeBase. The target character's rest pose supplies a hip basis and a chest basis, and the source's world rotations are converted into local bone quaternions. The hips path is translated into the target's coordinate frame and scaled by the ratio between the character's leg length and the recovered performer's leg length.

$$s = \frac{L_{\mathrm{rig}}}{L_{\mathrm{human}}}$$

After retargeting, the code checks the lowest foot against the rest-pose floor and raises the hips if a foot would penetrate it. This is a useful correction, but it is not a contact solver: sliding, penetration during a jump, and pressure distribution are still not modeled.

For the live path, the One Euro filter smooths MediaPipe landmarks before solving the rig. The HQ path is already smoother, so the browser simply samples the recovered track and cross-fades from the live animation to HQ over 600 milliseconds. Character switches use a shorter 250-millisecond blend. Animated GLB export samples the best available track into local rotation keys plus a hips position track, then uses three.js GLTFExporter to write a binary GLB.

Stream mode and its cumulative risk

Stream mode turns separate generations into an approximate continuous performance. Each generation uses the previous clip's final frame as its image-to-video conditioning. The app holds the last pose while the next HQ result is pending, starts another take on the live pass if the queue is starving, and inserts short idle clips between prompts. Horizontal root offset is carried from one take to the next and capped at 2.5 meters.

This produces a much more coherent demo than restarting from the same neutral image every time, but it inherits two sources of drift. The video model can gradually change identity, costume, or body proportions, and the previous last frame does not guarantee a physically continuous next action. Stream mode is therefore best understood as visual stitching with motion-aware playback, not as a long-horizon world model or a guarantee of seamless contact.

Measured speed

Operation	Configuration	Reported time
Fast video generation	H3 Max Turbo, 480P, 5 seconds	About 2 to 3 seconds wall time, 0.5 seconds GPU
Higher-fidelity video	FLUX 3 Max, 720p, 5 seconds	About 8 to 9 seconds wall time, 4.5 seconds GPU
HQ body recovery	GVHMR, RTX 4090	About 3.5 to 4 seconds
Finger tracking	WiLoR, parallel with GVHMR	About 0.3 seconds additional

These are author-reported timings for generated, single-person clips on a tuned Windows 11 machine with an RTX 4090. They should not be read as general throughput numbers. FLUX 3 has a minimum five-second duration in the server wrapper, while H3 accepts shorter clips; model-provider queueing, download time, and first-run CUDA initialization are separate from the stated GPU inference figures.

Where the pipeline is likely to fail

The central limitation is not the neural network choice but the input assumption. The system expects one person, a static camera, a clean background, a visible full body, and limbs that remain separable. Turning, spinning, flips, floor work, crossed arms, occluded hands, loose clothing, and unusual camera angles all stress the wrong parts of the stack. The prompt template avoids those cases because pose estimators can swap left and right, lose depth, or hallucinate a limb behind the torso.

Correct-looking rotations also do not imply physical validity. prompt2rig grounds the lowest foot and applies finger hinge limits, but it does not solve contact, torque, center of mass, or hardware joint limits. There is no published quantitative benchmark for natural long-form motion, hand accuracy, foot sliding, or sim-to-real transfer. The demos establish that the pipeline can produce compelling short clips; they do not establish the accuracy of a motion-capture product.

The Windows and NVIDIA requirement is another practical boundary. The browser half runs with Node alone, but the recommended HQ pass depends on a local CUDA environment, roughly 5 GB of GVHMR checkpoints, a registered SMPL-X body model, and optionally MANO and WiLoR weights. This is not a zero-install web application.

Why robotics teams may still care

The most relevant robotics pattern is the separation between motion generation, motion recovery, retargeting, and execution. A robot learning stack might replace the generated video with a human demonstration, replace Mixamo with a humanoid, and replace the export step with a trajectory and controller. The project demonstrates a practical front end for turning a video into articulated motion while keeping the intermediate body representation inspectable.

It also shows the boundary that remains after mocap. A humanoid needs contact schedules, center-of-mass consistency, joint velocity and torque limits, collision-aware retargeting, and a controller that can recover when the planned motion is infeasible. prompt2rig is not a robot-control system, a dynamics model, or a validated sim-to-real pipeline. It is a browser-native content and prototyping tool whose retargeting ideas may be useful upstream of those systems.

Running it

git clone https://github.com/blendi-remade/prompt2rig
cd prompt2rig
cp .env.example .env
# add FAL_KEY to .env
node server.mjs

That launches the web app and the MediaPipe live path at http://localhost:5177. For the HQ path on Windows, run hq\setup.ps1, install SMPLX_NEUTRAL.npz under hq/body_models/smplx/, optionally install MANO through hq\setup_hands.py, then start hq\.venv\Scripts\python hq\worker.py. The worker listens on 127.0.0.1:5178 and stays warm across requests.

Licensing is the most important deployment constraint

Component	License status	Practical implication
prompt2rig code	Apache-2.0	The project's own code is permissive
GVHMR	Educational, research, and non-profit use only	The HQ body path is not cleared for commercial use
SMPL-X	Non-commercial registration; commercial license separate	A commercial product must license the body model
WiLoR	CC-BY-NC-ND	The HQ finger path is non-commercial
MANO	Non-commercial registration	Commercial use requires a separate license
Ultralytics YOLO	AGPL-3.0 or commercial license	Network deployment can trigger copyleft obligations

The README is unusually explicit about this split: the web interface, live MediaPipe preview, and hosted video generation can be commercially usable under their respective terms, but the recommended HQ body-and-hand stack is research-oriented. A company planning to ship this pipeline must replace or separately license GVHMR, SMPL-X, WiLoR, MANO, and the detector. Calling the repository "Apache-2.0" without that qualification would be misleading.

What prompt2rig demonstrates

prompt2rig is convincing because it connects mature pieces with clear contracts. Video generation supplies an expressive but unreliable observation. MediaPipe supplies speed. GVHMR supplies a coherent body model. WiLoR supplies fingers. The retargeter supplies a common skeleton interface, and GLB export makes the result useful outside the demo. The engineering work is mostly in the seams between those stages: staging, coordinate frames, handedness, scale, smoothing, blending, and invalid-data handling.

That is also the right way to read the demo. It is not evidence that text alone now produces production-grade motion capture. It is evidence that a generated clip can become an editable, rigged 3D performance with a relatively small orchestration layer, provided the scene is constrained and the underlying model licenses fit the intended use.

---

Source: GitHubDaily on X, "One prompt to full-body 3D motion with fingers", and the blendi-remade/prompt2rig repository. The article was written from the linked repository's README, server, retargeting, HQ worker, hand-tracking, and Blender export source. The public video remains available in RobotWorld's existing X demo record; this article adds the full bilingual technical analysis rather than duplicating the short video card.