Skip to content
←Back to Open Source

OPEN SOURCE DEEP DIVE

VideoAnalysisReverseEngineeringAIGCVLM

rea-video: Reverse Engineer How Every Shot Was Made

A shot-by-shot forensic pipeline that identifies live action, AI video, renders, or code and turns the evidence into a remake plan.

JuneYaooo/rea-video9.5kPythonApache-2.08 min read

rea-video is not a one-click face-swap or reposting tool. It is a shot-by-shot forensic system for understanding how a finished video was assembled. It separates picture, sound, and packaging, estimates whether each shot came from live action, AI video, 3D rendering, code-driven motion, screen capture, a moving still, or a mixture, and then turns that evidence into a filming, generation, coding, or asset-preparation plan. The repository is open source under Apache-2.0.

A fourteen-shot live-action blender advertisement above an eight-shot AI remake, with method, confidence, and timing labels on every frame
A real comparison: the upper block is a fourteen-shot phone-shot original; the lower block is an eight-shot AI remake. The system resolves both to individual shots while preserving method, confidence, and timing labels.

Collect evidence before naming a production method

The central idea is not to ask a vision model to guess a category from one frame. The pipeline first performs deterministic measurement. It detects shot changes from color-histogram distance and pixel difference, combines global and adaptive local thresholds, handles jump cuts, fades, dissolves, wipes, and flashes, and attempts to infer transition direction. Hands, faces, text, physical contact, camera motion, lighting, color treatment, frame cadence, sensor noise, and container metadata all become possible evidence.

On the audio side, short-time Fourier analysis and harmonic-percussive source separation provide clues about speech, music, and effects. The system estimates tempo, key, sections, and whether cuts land on beats. The packaging pass runs OCR at roughly two frames per second, samples densely around cue boundaries, and reconstructs caption timing, color, outline, and entrance style. It also looks for stickers, picture-in-picture regions, embedded assets, and synchronization with sound events. A small logo or screen panel can therefore be treated as its own evidence rather than being averaged into the whole frame.

Evidence layer Primary measurements Usable output
Shots and picture Histogram change, pixel difference, jump cuts, transitions, frame cadence, sensor noise, whole-frame planar motion Shot boundaries, production-method candidates, camera and lighting clues
Sound Voice and accompaniment separation, tempo, key, sections, transient events Speech and music structure, beat-synced cuts, low-confidence effect candidates
Packaging Dense OCR, style extraction, stickers, picture-in-picture, temporal alignment Caption and styled-text timeline, placement and style, audio-visual sync
Visual review Optional vision-language model or an image-capable agent checking hands, text, physics, and cross-shot consistency Extra evidence when measurements are ambiguous, plus a remake prompt for each shot

Fusing hard evidence with visual judgment

Not all signals carry equal weight. Generative metadata, frame timing, repeated-frame patterns, and whole-frame affine motion can override a vague impression that footage “looks synthetic.” Platform re-encoding can destroy noise and cadence evidence, so the system identifies common transcoding signatures and reduces the weight of rules that those signatures invalidate. Conflicting candidates are resolved with temporal continuity, scene membership, neighboring shots, and confidence rather than a simple majority vote. Shots that remain uncertain can be sent to a vision model or an agent for review.

The analysis also keeps production method separate from material provenance. A shot may contain live footage with a code-rendered title, or an AI-generated subject composited with a real screen recording, product image, or screenshot. A credible remake plan must know which pixels are imported originals, which are licensed references, which must be generated again, and which the creator must supply. Without that distinction, the result can copy a surface style while losing the actual production chain and rights boundary.

VIDEO.md turns a long report into a layered interface

Each breakdown can emit a report, timeline, shot files, storyboard, extracted frames, audio stems, and remake assets. The raw structured result may approach one hundred ninety thousand tokens, which is useful to software but wasteful to load into an agent. The project therefore writes VIDEO.md in layers: a short card, an overview and method map of roughly two thousand tokens, a one-line-per-shot timeline of roughly four thousand tokens, and separate files for only the shots that need deeper inspection.

The format keeps direct observations distinct from tool inferences, so a guess is not silently presented as something visible in the frame. An agent answering a camera-motion question can read only the relevant timeline entry. A prompt-building task can descend into that shot's evidence, characters, setting, props, and sound. Human readers receive an HTML or PDF report, while agents consume the same underlying evidence through a compact machine-readable structure.

A real case: live action to AI remake

The repository's blender example starts with a twenty-second, first-person phone advertisement. The original has fourteen shots across one kitchen set and two camera positions. It opens with a conclusion, then rapidly cuts through ingredients, milk, protein powder, fruit, closing the lid, blending, pouring, and drinking. The remake is an eight-shot, twenty-one-second AI version. The system does not flatten both into a generic “product advertisement” category; it reconstructs live action for one and generation for the other.

The comparison also makes the generated version's failure modes visible. Brand lettering turns into gibberish across shots, whole fruit becomes a smoothie without a plausible blending interval, and the glass changes shape between adjacent views. The remake report groups the eight shots into three generation units and specifies which first frame, last frame, or reference image each unit needs to preserve product identity and action continuity. What gets reused is structure, pacing, and technique, not the original pixels.

Read the evaluation numbers with their conditions

Evaluation condition Production-method accuracy How to interpret it
Tools only, tuning set 98.3% Rules were designed and adjusted on these fifteen videos
Tools only, first holdout 99.6% Rules were revised after initial failures, so it is not fully independent
Tools only, second holdout 98.9% Full-decode result from the first run after the production rules were fixed
Tools only, re-encoded development set 88.6% Platform compression removes useful evidence and is the harder realistic case
With a vision model, re-encoded development set 99.6% A development result produced after rules had seen this data
With agent image review, re-encoded development set 100% An upper-bound demonstration on that set, not a claim about arbitrary new footage

Shot-boundary evaluation reports ninety-five percent recall and ninety-four percent precision. Cached-feature results differ from full-decode results, so a number cannot be quoted responsibly without its measurement path. Sound-effect detection is a clear deterministic weakness: without a listening vision-language model, signal features often cannot separate effects from the attacks of speech or music. Experimental precision was about 0.2, so the tools should emit only low-confidence candidates. Music provenance is similarly uncertain, because an AI-generated instrumental inside an otherwise ordinary video can be indistinguishable from library music.

Other limits belong in the operating procedure. 2D animation, stop motion, and footage edited from film or television generally require visual review. Subtitle timing and styling are suitable for deterministic recovery, while sticker recall remains partial. A fixed-camera phone shot after aggressive transcoding is one of the hardest current cases. The tools, a vision model, and an agent can also share the same mistake. The published figures are therefore conditional results on named sets, not a guarantee that every new video can be classified automatically.

How a breakdown becomes a remake

The system routes every shot toward the method that can rebuild it. Live-action shots become a scene, camera, lighting, performance, and acquisition plan. AI shots are grouped around the target model's maximum clip length, multi-shot support, and reference-image budget. Code and interface shots become a Remotion project. Software demonstrations require fresh screen capture. Still images can receive controlled pushes and pulls. 3D and 2D segments stay with an appropriate renderer or animation tool.

Only part of this is automated. With capsule-cinema installed, AI shots can be generated and placed back on the original timeline. Code shots can be written as a Remotion project by an agent. Live action still has to be filmed from the plan. The project does not pretend that one video model replaces every production method, and it does not turn differently licensed material into a publish-ready composite.

Requirements and copyright boundary

The project is designed for an agent that can execute commands. It requires Python 3.10 or newer and ffmpeg. Core dependencies include OpenCV, Pillow, ONNX Runtime, and the VIDEO.md validator. Voice-separation and local speech-recognition models are downloaded on first use. Tools-only analysis works offline. Configuring Gemini, Doubao, or another image-capable endpoint improves judgments about live action, AI video, 3D, and sound effects, with model costs paid by the user. Automated generation starts with a free dry run and should call paid models only after confirmation.

rea-video is intended for learning a video's structure, pacing, camera language, and production technique, then rebuilding it with material the user has the rights to use. Reposting another creator's footage, music, characters, or likeness is not permitted. Platform rules, local AI-content labeling requirements, and asset licenses remain the publisher's responsibility. Its practical value is the step from “this video works” to “I can show why each shot works and rebuild the method lawfully.”

As an Amazon Associate, we earn from qualifying purchases.