PAPER DEEP DIVE
VideoJev: System One Model for Video Understanding and Interaction
VideoJev casts temporal grounding and multiple-choice video QA as candidate scoring: a shared head scores temporal intervals and textual options against video evidence without autoregressive decoding, while Cached and Joint inference reuse the video prefix, giving +6.7/+10.8 mean mIoU over backbones with 2.8-3.1x and >100x measured speedups.
One-line summary
VideoJev recasts video understanding and interaction as structured decision tasks: a shared scoring head replaces the autoregressive generation head of a pretrained MLLM, so temporal grounding and multiple-choice video QA are read out as probabilities in a single forward pass, reaching or beating Thinking-mode generative models at one to three hundredths of the inference latency.
Background and motivation
A visual agent must repeatedly judge the same stream of observations: locate when something happened, verify whether a precondition holds, or pick the most suitable action among candidates. An assistant following a cooking procedure may first check that an ingredient was added before choosing the next step. Such judgments share a property: their output is compact and structured, ready for software to consume, rather than a paragraph of free text.
Two existing routes each fall short. Discriminative vision models predict efficiently but lean on task-specific designs and fixed label spaces, requiring a new head per task. Video multimodal large language models (MLLMs) offer broad semantics and a flexible language interface, yet they express every decision through autoregressive generation: emitting timestamps token by token for temporal grounding, or answer text for question answering, even when the answer is just an interval or a letter.
The System One paradigm supplies the missing idea. The Jev line of work produces typed probabilistic judgments that software can use directly, without generating answers token by token. The paradigm had been extended to images (Visual Jev, PixelJev), but video adds two requirements: decisions must reflect temporal structure, and successive queries must share an expensive visual context.
As an interaction unfolds, new goals and intermediate results keep triggering further questions about the same visual history. The marginal cost of each additional judgment then decides whether the interaction feels responsive. Re-encoding the whole video and regenerating an answer per query stacks latency linearly and blocks real-time agents.
VideoJev is therefore built on two principles: direct candidate scoring and reusable visual computation. The former lets temporal intervals and textual options share one scoring architecture, with task-specific supervision and output rules preserving their distinct structures; the latter encodes the video prefix once and reuses it through Cached inference for arriving queries and Joint inference for queries processed together.
Prelude: from generation to a decision readout
The starting point is an observation that pre-MLP features of a pretrained MLLM already support semantic similarity well (as in FreeRet and related work). The authors therefore delete the final MLP branch and replace the language-model output head with a candidate scorer. The scorer generates no tokens; it projects and normalizes hidden states at designated marker positions and takes a cosine similarity to produce logits.
To carry the two readout kinds, two new special tokens <|option_end|> and <|decide|> are added, both initialized from the pretrained <|im_end|> embedding. Every candidate ends with <|option_end|> and the candidate list ends with a single <|decide|>. Causal attention lets each candidate readout see the video, the query, and preceding candidates, while the decision readout sees the full list.
The scoring form is: project the decision state and each candidate state to 512 dimensions, L2-normalize, take cosine similarity, multiply by a fixed scale of 10 and add a learned scalar bias to obtain the logit:
$$u=\frac{W_{q}h_{d}+b_{q}}{\|W_{q}h_{d}+b_{q}\|_{2}},\qquad v_{i}=\frac{W_{k}h_{i}+b_{k}}{\|W_{k}h_{i}+b_{k}\|_{2}},\qquad s_{i}=10\,u^{\top}v_{i}+b \tag{1}$$
Here $h_{d}$ and $h_{i}$ are the decision and candidate states, $W_{q},W_{k},b_{q},b_{k}$ are learned projection parameters, the scale is fixed at 10 and the bias $b$ is learned. This equation is the model's decision interface: temporal and option tasks share it, differing only in how logits become outputs.
Method in detail
A unified candidate representation
Given a video $V$, a language query $q$, and $K$ candidates $C=\{c_{1},\ldots,c_{K}\}$, the sequence places video tokens first, followed by the task instruction, the query, and the candidate list. The instruction names the task as temporal grounding or multiple-choice video QA. Temporal candidates are timestamp texts copied from the video prefix into the list, each identifying a frame-pair interval; option candidates are answer texts. Candidates plus output rules specify the task while the architecture stays fixed.
Because causal modeling makes the video prefix independent of later queries, its computation can be shared. In Cached inference the video is encoded and prefilled once; each arriving query branches from the cached prefix and needs one language-model forward over its query, candidates, and readout tokens. In Joint inference multiple queries and their candidate lists follow a single video prefix, each with its own <|decide|> readout, and one causal forward yields all query scores. Training uses the same concatenated layout so visual computation is shared across questions.
Temporal grounding: interval membership plus coverage supervision
The backbone's twofold temporal compression groups frames into pairs, so the timeline is partitioned into $K$ candidate intervals, one per pair. For interval $B_{i}=[\ell_{i},r_{i}]$, $t_{i}$ is the mean of its two frame timestamps; internal boundaries sit halfway between adjacent pairs, with $\ell_{1}=0$ and $r_{K}$ equal to the video duration. Let $G$ be the union of annotated event intervals; the candidate label is $y_{i}=\mathds{1}\{t_{i}\in G\}$ and the membership probability is $p_{i}=\sigma(s_{i})$. Since one event may cover several candidates, multi-label binary cross-entropy supervises each independently:
$$\mathcal{L}_{\mathrm{BCE}}=-\frac{1}{K}\sum_{i=1}^{K}\left[y_{i}\log p_{i}+(1-y_{i})\log(1-p_{i})\right] \tag{2}$$
BCE supervises local membership but does not reward covering the event as a whole, so a duration-weighted soft IoU loss is added. With interval duration $d_{i}=r_{i}-\ell_{i}$:
$$\mathcal{L}_{\mathrm{IoU}}=1-\frac{\sum_{i}d_{i}p_{i}y_{i}}{\sum_{i}d_{i}\left[y_{i}+p_{i}(1-y_{i})\right]} \tag{3}$$
The numerator rewards coverage of the event while the denominator penalizes responses outside it; weighting by $d_{i}$ handles unequal interval lengths. Soft IoU is omitted when no midpoint is positive, and BCE uses only valid candidates.
To assemble interval scores into an event span, logits are shifted by a probability threshold $\tau$ and the nonempty contiguous range with the largest sum is selected, with $\tau=0.25$ and smoothing window 1 fixed everywhere:
$$(a^{\star},b^{\star})=\arg\max_{1\leq a\leq b\leq K}\sum_{i=a}^{b}\left[s_{i}-\log\frac{\tau}{1-\tau}\right] \tag{4}$$
This maximum-subsegment problem is solved by Kadane's algorithm in $O(K)$ time; the selected candidates' boundaries give the continuous prediction $[\ell_{a^{\star}},r_{b^{\star}}]$. If all shifted logits are negative the decoder falls back to the strongest single candidate interval, followed by lightweight boundary postprocessing. In the code, timejev/temporal.py::decode implements Eq. (4) verbatim via z = z - np.log(threshold/(1-threshold)) and the following Kadane loop, with a docstring noting that all-negative sequences return their strongest single frame without using ground truth.
Video QA: softmax preferences and position-bias suppression
Multiple-choice QA selects one of $K$ textual options. The shared head assigns logit $s_{i}$ to option $i$; let $y$ be the correct index. Because exactly one answer is correct, logits are normalized with softmax and trained with cross-entropy:
$$p_{i}=\frac{\exp(s_{i})}{\sum_{j=1}^{K}\exp(s_{j})},\qquad \mathcal{L}_{\mathrm{CE}}=-\log p_{y} \tag{5}$$
To discourage shortcuts based on option position, each eligible training question receives a reproducible, answer-independent permutation of its options with the correct index updated accordingly; questions that refer to option positions or earlier turns keep their original order. At inference the supplied order is kept and the option with the highest $p_{i}$ is selected. The code in timejev/qa_option_permutation.py records PROTOCOL='qa_option_permutation_sha256_v1' and reports permutation_independent_of_correct_answer=True and evaluation_permuted=False, i.e. evaluation is never permuted.
Two-stage training recipe
Temporal membership is trained on TimeLens-100K and TimeLens2-93K, converting annotated intervals into candidate-level labels. Option selection uses an internally curated corpus of about 240K multiple-choice questions over 19K videos spanning gaming, sports, cooking, music and dance, outdoor activities, and crafts; questions cover entity and action recognition, spatiotemporal reasoning, information retrieval, event understanding, counting, OCR, and summarization; videos run from 4 minutes to nearly 2 hours with four or five options each. Queries from the same video are grouped around a shared prefix.
Inspired by projector alignment in MLLMs, the scorer is first trained against frozen pretrained features, then the backbone is adapted. Stage one trains only the head on 30% of videos with the backbone and new marker embeddings frozen:
$$\mathcal{L}_{\mathrm{TG}}=\mathcal{L}_{\mathrm{BCE}},\qquad \mathcal{L}_{\mathrm{QA}}=\mathcal{L}_{\mathrm{CE}} \tag{6}$$
Stage two trains the language backbone, marker embeddings, and aligned head on the remaining 70% of videos; soft IoU adds event-level coverage to grounding while QA keeps cross-entropy:
$$\mathcal{L}_{\mathrm{TG}}=\mathcal{L}_{\mathrm{BCE}}+2\,\mathcal{L}_{\mathrm{IoU}} \tag{7}$$
$$\mathcal{L}_{\mathrm{QA}}=\mathcal{L}_{\mathrm{CE}} \tag{8}$$
Task losses are averaged equally over questions; the visual encoder and merger stay frozen throughout; each stage runs one epoch with its own optimizer and schedule. The two stages run 160 and 375 optimizer updates on 64 NVIDIA H200 GPUs with BF16, global batch 64 packed sequences, training videos at 2 fps, max sequence 144K tokens, and an 80% visual-token budget; AdamW with weight decay 0.01 and gradient clipping 1.0, 5% warm-up then cosine decay to 10% of peak; head learning rate 3e-4 during alignment and 1e-4 during joint adaptation, backbone learning rate 5e-6 for 4B and 2e-6 for 35B-A3B; DeepSpeed ZeRO-2 for 4B and ZeRO-3 for 35B-A3B. The code in timejev/staged_training.py enforces the handoff via a timejev_head_stage_v1 checkpoint format and creates a fresh AdamW per stage.
Method flow overview
flowchart TD V["video V sampled at 2fps frames grouped in pairs"] --> ENC["visual encoder + merger frozen throughout"] ENC --> PREF["shared video prefix encoded once"] Q["language query q and task instruction"] --> PREF C["K candidates timestamp intervals or text options"] --> SEQ["concatenate video + instruction + q + candidates + option_end + decide"] PREF --> SEQ SEQ --> ATT["causal attention final layer attention only MLP removed"] ATT --> HEAD["shared scorer cosine similarity x10 + learned bias"] HEAD --> TG["temporal branch sigmoid membership p_i"] HEAD --> QA["option branch softmax p_i"] TG --> KAD["Kadane O K max contiguous sum after tau shift tau=0.25"] KAD --> SPAN["emit event interval"] QA --> ARG["argmax p_i"] ARG --> ANS["emit answer option"] PREF --> CI["Cached inference queries arrive one by one reuse cache"] PREF --> JI["Joint inference many queries one forward separate decide readouts"]
Where the code matches the paper
Removing the final MLP appears in timejev/readout.py as reclassifying the last layer and deleting two submodules:
# timejev/readout.py::adapt_readout
final.__class__ = AttentionOnlyDecoderLayer
del final.mlp
del final.post_attention_layernormInitializing the new markers from the <|im_end|> embedding and fixing the logit scale at 10 with a trainable bias appears in timejev/decision_model.py:
# timejev/decision_model.py
im_end = tokenizer.convert_tokens_to_ids('<|im_end|>')
initial = embedding.weight[im_end].detach().clone()
for token in SPECIAL_TOKENS: # ['<|decide|>', '<|option_end|>']
embedding.weight[tokenizer.convert_tokens_to_ids(token)].copy_(initial)
# head: similarity='cosine', logit_scale=10.0, trainable_scale=False, trainable_bias=TrueThe duration-weighted soft IoU is computed in timejev/decision_model.py::_duration_soft_iou_loss as intersection = (durations*probabilities*targets).sum(1) and union = (durations*(targets+probabilities*(1-targets))).sum(1), keeping pure BCE when no midpoint is positive; the task-loss combination in timejev/losses.py::decision_losses reads per_query = where(choice, ce, bce) + soft_iou_weight*iou, matching Eqs. (6)-(8).
Experiments
Figure 1: the unified candidate-scoring architecture. (a) temporal intervals and textual options share an input format; (b) Cached and Joint inference share the backbone and scorer; (c) temporal scores decode an event span while option scores select an answer.
Temporal grounding: best accuracy at the lowest latency
On the TimeLens-Bench refined annotations of Charades-STA, ActivityNet Captions, and QVHighlights, VideoJev-4B reaches 65.3 mean mIoU at a measured 0.276 s per query, improving on its Qwen3.5-4B backbone by 6.7 points and on the specialist TimeLens2-4B by 3.3 points; under the stricter R1@0.7 it beats TimeLens2-4B by 2.8-4.8 points, indicating genuinely tighter temporal boundaries rather than looser spans gaming mIoU. VideoJev-35B-A3B pushes mean mIoU to 67.7, 10.8 points above Qwen3.6-35B-A3B; scaling from 4B adds 2.4 points for only a 22% latency increase. Both sizes lead every reported metric in their open-source groups at the lowest measured latency, with 2.8-3.1x speedups over their backbones; because the generative baselines also share video prefill, the speedup comes purely from not autoregressively generating timestamps.
| Model | s/query | Charades mIoU | ActivityNet mIoU | QVHighlights mIoU | Mean mIoU |
|---|---|---|---|---|---|
| VideoJev-4B | 0.276 | 60.1 | 62.8 | 73.2 | 65.3 |
| Qwen3.5-4B (backbone, Instruct) | 0.763 | 52.3 | 55.6 | 67.9 | 58.6 |
| TimeLens2-4B | 0.646 | 57.7 | 59.0 | 69.3 | 62.0 |
| VideoChat3-4B | 1.067 | 56.1 | 54.6 | 67.0 | 59.2 |
| VideoJev-35B-A3B | 0.336 | 63.7 | 65.5 | 74.0 | 67.7 |
| Qwen3.6-35B-A3B (backbone, Instruct) | 1.054 | 47.3 | 56.0 | 67.5 | 56.9 |
| Gemini-2.5-Pro | - | 52.8 | 58.1 | 70.4 | 60.4 |
Video QA: matching Thinking mode at a hundredth of the latency
Across LVBench, Video-MME, LongVideoBench, and MLVU, VideoJev-4B and 35B-A3B improve the four-benchmark mean over their Instruct backbones by 8.0 and 5.7 points, reaching 74.3 and 79.5 with gains on every benchmark. The sharper comparison is against Thinking mode: VideoJev-4B at 74.3 trails Qwen3.5-4B Thinking's 74.9 by only 0.6 points, while VideoJev-35B-A3B at 79.5 exceeds Qwen3.6-35B-A3B Thinking's 78.8 by 0.7 points; both beat their Thinking counterparts on LVBench. On cost, the Thinking baselines measure 144.929 s and 149.777 s per question versus 1.123 s and 1.493 s for VideoJev, i.e. 129x and 100x speedups. Thinking keeps an edge only on Video-MME and MLVU, suggesting explicit reasoning still helps some long-chain items while direct scoring suffices for most structured judgments.
| Model | s/question | LVBench | Video-MME | LongVideoBench | MLVU | 4-bench mean |
|---|---|---|---|---|---|---|
| VideoJev-4B | 1.123 | 67.9 | 75.0 | 73.6 | 80.6 | 74.3 |
| Qwen3.5-4B Instruct | 1.355 | 60.0 | 70.6 | 61.2 | 73.3 | 66.3 |
| Qwen3.5-4B Thinking | 144.929 | 66.4 | 76.9 | 73.6 | 82.8 | 74.9 |
| VideoJev-35B-A3B | 1.493 | 73.4 | 81.8 | 77.1 | 85.9 | 79.5 |
| Qwen3.6-35B-A3B Instruct | 2.379 | 66.0 | 77.1 | 72.4 | 79.5 | 73.8 |
| Qwen3.6-35B-A3B Thinking | 149.777 | 71.4 | 82.5 | 74.9 | 86.2 | 78.8 |
Figure 2: teaser. Top: shared candidate scoring with reusable visual computation. Bottom: prediction quality versus model forward latency at both scales; VideoJev sits in the high-quality, low-latency corner.
Ablations: the gains are architectural, not just data
To rule out a data-only explanation, the authors fine-tune an ordinary Qwen3.5-4B autoregressive model that keeps its native final MLP and language-model head on exactly the same post-training data, single-stage. VideoJev-4B still leads by 1.3 points in mean grounding mIoU and 1.9 points in mean QA, showing that directly supervising temporal membership and option preferences aligns the readout with the required decisions. Other ablations agree: Joint inference improves mean grounding/QA over Cached inference by 0.8/0.5 points at 4B and 0.3/0.7 at 35B-A3B (13 of 14 model-benchmark comparisons improve), consistent with the joint training layout; joint TG+QA training beats single-task training on all seven benchmarks (+0.7 grounding, +1.3 QA, with +1.9 on LongVideoBench); soft IoU lifts grounding mean from 60.7 to 65.3 (+4.6) and QA from 73.9 to 74.3; removing the final MLP costs nothing and gains about 0.2 points, so the simpler readout is adopted; two-stage adaptation matches single-stage quality while updating the backbone on only 70% of videos.
| Setting (VideoJev-4B) | Grounding mean mIoU | QA 4-bench mean | Change vs baseline |
|---|---|---|---|
| Full model (Joint + soft IoU + no MLP + two-stage) | 65.3 | 74.3 | baseline |
| Autoregressive SFT on identical data | 64.1 (vs 65.4 single-stage) | 72.8 (vs 74.7) | architectural gain +1.3 / +1.9 |
| Cached inference instead of Joint | 64.5 | 73.8 | -0.8 / -0.5 |
| No soft IoU (BCE/CE only) | 60.7 | 73.9 | -4.6 / -0.4 |
| Retain final-layer MLP | 65.1 | 74.1 | -0.2 / -0.2 |
| Single-stage (100% of videos) | 65.4 | 74.7 | +0.1 / +0.4 (but backbone updated on all data) |
Multi-query scaling and confidence calibration
Figure 3: mean forward time for 1-10 queries per video. (a) Charades grounding; (b) Video-Holmes QA. Joint inference (dashed) stays nearly flat as queries grow, while Cached and multi-token generation (dotted) climb steeply.
Growing from 1 to 10 queries about the same video, Joint inference adds only 55 ms for grounding and 31 ms for QA; at ten queries forward times are 0.39 s and 1.64 s versus 0.85 s and 2.17 s under Cached inference, while independently measured Charades mIoU and LVBench accuracy improve by 1.5 and 0.8 points. By contrast, the autoregressive Direct QA baseline drops from 60.0% to 58.7% accuracy under Joint (VideoJev holds 67.9% at comparable latency); Qwen3.5-4B Instruct needs 5.98 s for ten grounding queries at 52.3 mIoU; Qwen Thinking needs 1,393 s for ten QA queries at 66.4% accuracy. Near-zero marginal cost without quality loss is the core selling point of VideoJev as a composable decision interface.
Figure 4: confidence-accuracy calibration on four benchmarks. High-confidence bins are markedly more accurate, but probabilities are overconfident overall.
Candidate scores also signal when to trust a prediction. Taking the highest option probability $\max_{i}p_{i}$, the 35B-A3B model reaches 95.8% accuracy on its 591 LVBench predictions with confidence at least 99%, versus 73.4% overall; this separation opens the door to selectively accepting confident answers and escalating uncertain ones. The authors concede calibration is imperfect, however: four-benchmark mean expected calibration error (ECE, ten equal-width bins) is 8.65 and 8.13 percentage points for 4B and 35B-A3B, with probabilities systematically overestimating correctness because VideoJev receives no calibration-specific training (Jev optimizes calibration explicitly via reinforcement learning).
Figure 5: Joint inference answers six Charades queries in one forward; green marks ground truth. Both sizes match one-second transitions, and 35B-A3B localizes adjacent actions more precisely.
Limitations
First (author-stated): decisions are confined to closed candidate sets. VideoJev can only score supplied intervals or supplied options and cannot generate openly; the Future Work section states that a higher-level System Two agent must propose candidates and organize the decision space to cover open-ended goals. Second (author-stated): probabilities are overconfident and receive no calibration training, with ECE of 8-9 percentage points needing temperature scaling or calibration-aware training; streaming inputs are also unsupported, requiring incremental visual encoding and memory management. Third (our assessment): evaluation covers only multiple-choice format, and the internal 240K-question corpus is not released, so third parties cannot reproduce the source of the QA gains; the fixed $\tau=0.25$ and smoothing window 1 are never tuned per dataset or size, which guarantees comparability but may be suboptimal on some distributions. Fourth (our assessment): Joint inference lets later queries attend to earlier queries and candidates; although the authors verify no answer leakage, query order can still influence scores, a sensitivity worth watching when the model is exposed as an external interface.
Conclusion and outlook
VideoJev's contribution is rewriting video understanding from a generation problem into a decision problem: one shared scoring head carries both temporal membership and option preference, and Cached plus Joint prefix reuse drive the marginal cost of repeated judgments toward zero. Results on seven benchmarks, together with the same-data autoregressive control, show the gains come from architecture as well as data, and that the two tasks reinforce each other under joint training. Looking ahead, the authors plan to extend the candidate interface to action recognition, event detection, scene-state classification, and visual condition verification, letting a higher-level agent decompose open goals into visual checks via standard operating procedures or generated plans while System One evaluates each against visual evidence; they also target streaming encoding, calibration-aware training, and selective prediction to decide when to accept a System One judgment versus deferring to higher-level reasoning. For robot developers, this route means the perception layer can return confidence-bearing structured judgments instead of free text, ready to plug into planning and execution loops.
Golden quotes
"Structured visual judgments can support open-ended behavior without requiring open-ended generation at every step."
"When successive questions share the same observations, the cost of each additional judgment becomes central to responsive interaction."
"System Two organizes the decision space, while System One evaluates it against visual evidence."
