Skip to content
←Back to Open Source

OPEN SOURCE DEEP DIVE

VLX-SeekVision-language modelsVisual GroundingEmbodied AI

VLX-Seek 1.5: Turning Candidate Regions into Referable Visual Entities

VLX-Seek 1.5 turns open-vocabulary visual grounding into region selection: WeDetect-Base-Uni proposes boxes, a hybrid Qwen3.5/C-RADIOv4 encoder makes each region addressable, and a 10B language model returns object IDs that resolve to pixels. The release covers seven inference tasks and 1.5-10B weights for conditional inspection perception, but proposal recall bounds localization, training and robot-control code are absent, and the OmAI Model License v1.0 includes a revenue threshold.

om-ai-lab/VLX-Seek2.2kPythonNOASSERTION9 min read

VLX-Seek 1.5 separates fine-grained perception into two auditable stages: first recall candidate regions that may contain objects, then let the language model select, compare, and refer to those regions. The repository currently releases inference code and weights for the 1.5-10B model. Its central design choice is to stop asking the language model to spell out box coordinates and instead turn every candidate box into an addressable visual entity.

VLX-Seek pipeline from visual input and candidate regions to region tokens and grounded output
Candidate regions are numbered and encoded before language reasoning. Region IDs in the answer map back to pixel coordinates.

Localization without coordinate strings

A conventional vision-language model often represents a box as four generated numbers. That works for one object, but the output becomes long and fragile when several instances must be returned. A single digit, ordering error, or malformed separator can invalidate the whole result. VLX-Seek changes the interface on both sides. In the prompt, candidate regions receive special tokens such as <obj0> and <obj1>, each followed by a region-feature placeholder. In the answer, the model emits only the IDs of the regions it selects.

For a query about people wearing red, a grounded response can have the form <ground>people wearing red</ground><objects><obj2><obj5></objects>. The inference program then resolves those IDs against the caller-provided boxes and restores pixel coordinates. Semantic matching remains a language-model task, while geometry stays in a structured input that does not need to survive free-text parsing.

The interface also makes absence an explicit answer. If no candidate matches the request, the model can return an empty result instead of forcing a box into the scene. The project reports that version 1.5 uses hard negatives and an explicit empty-result format to reduce this failure mode. In an inspection workflow, a reliable statement that the target is absent can be just as important as a positive detection.

Candidate recall sets the ceiling

VLX-Seek does not regress one global box directly from the whole image. It first depends on a proposal generator to recall plausible regions. The default open-source path invokes WeDetect-Base-Uni and produces the top 100 proposals when no boxes are supplied. A caller may instead provide pixel-coordinate boxes from another detector. This separates recall from semantic selection and allows deployments to trade detector latency, hardware cost, and recall independently from the language model.

The separation has a hard limit. If a target never enters the candidate set, the language model cannot select it. Heavy occlusion, small or distant objects, ambiguous boundaries, and overlapping instances still depend on proposal quality. VLX-Seek is therefore not an unconstrained end-to-end detector whose localization quality is independent of its upstream region generator.

Turning a box into a readable region

A geometric box does not tell the model what it contains. The repository combines two visual pathways in a hybrid fine-grained region encoder. The main vision tower preserves semantic features aligned with the language model, while an auxiliary tower supplies local structure and detail. In the current implementation, the main path applies region-of-interest alignment to Qwen3.5 vision-tower features. The auxiliary path uses C-RADIOv4 features and a multi-scale feature structure to handle objects at different sizes.

Each candidate box is also converted into a normalized sinusoidal positional encoding that preserves center, width, height, and spatial relationships. Aligned region features are concatenated or fused, pooled, and projected into the language model's embedding space. The current configuration uses a region alignment resolution of 14, produces a 6,656-dimensional fused region representation, and projects it to the model's 4,096-dimensional embedding width. The candidate box is then more than a hint: it is an element the language model can address.

Key interfaces in the region-reference path
Stage Input Output Engineering consequence
Proposal generation Source image Pixel-coordinate boxes WeDetect or an external detector determines which targets can be selected
Region encoding Image, boxes, box positions Region-level visual features Combines semantic context, local detail, and multi-scale spatial cues
Language reasoning Global image tokens, text, region tokens Natural-language answer and region IDs Produces a compact localization result that resolves to original coordinates

Seven tasks on one region interface

The command-line program exposes seven tasks: general visual question answering, detection, single-target grounding, counting, reasoning detection, region captioning, and region OCR. Every task except general visual question answering needs proposals. Region captioning and OCR additionally require the indexes of the regions being referenced. The Python worker can keep the model resident and process multiple requests without reloading the checkpoint.

These tasks are not seven separate models. Detection, grounding, counting, and reasoning detection share the operation of selecting objects from a candidate set. Captioning and OCR treat specified region IDs as visual evidence. General visual question answering can omit proposals and helps preserve whole-image understanding. A unified contract reduces the number of model stacks that a deployment must maintain.

Released tasks and their additional inputs
Task Primary use Proposals Additional input
Detection Find every instance matching one or more text categories Required Text categories
Single-target grounding Select one instance from a complex description Required Referring expression
Counting Select matching objects and report the total Required Target description
Reasoning detection Compare regional evidence before returning objects Required Target description or category
Region captioning Describe the details of selected candidates Required Target region indexes
Region OCR Read text inside selected candidates Required Target region indexes
General VQA Answer whole-image questions without region selection Not used Free-form question

Conditional search from robot viewpoints

The qualitative examples resemble field instructions rather than isolated object categories. Queries such as water bottles that are not on the cardboard box, flammable or explosive objects, and a person riding an electric bike require category understanding together with spatial relations or exclusion conditions. For a robot on inspection duty, these conditional descriptions are often more useful than asking whether a common object is present somewhere in the frame.

VLX-Seek qualitative result for water bottles that are not on the cardboard box
A conditional query: the model must distinguish bottles from nearby boxes and resolve their spatial relation.
VLX-Seek qualitative result for flammable or explosive objects
Safety-related recognition still depends on the visible scene and does not replace certified sensors or hazard procedures.
VLX-Seek qualitative result for a person riding an electric bike
An aerial view with people, vehicles, and traffic context demonstrates region selection under multi-object relations.

These examples show behavior on selected inputs, not proof that the model is reliable in every closed-loop robot deployment. A real system still has to handle viewpoint changes, motion blur, temporal sensor fusion, calibration errors, network latency, and safety rules. VLX-Seek supplies visual perception and regional references. It does not include motion planning, manipulator control, or task policy.

How to read the published result figures

The result charts in the repository report the original VLX-Seek 3B model, not the 1.5-10B checkpoint released in this update. They are useful for understanding the benchmark families targeted by the region-reference approach, but their numbers should not be treated as reproduced results for the current 10B weights. Results for the full 1.5 line are reported in the project blog. Training scripts and a complete benchmark reproduction pipeline are not included in this repository release.

Original VLX-Seek 3B results on the ODinW13 open-vocabulary detection benchmarks
Open-vocabulary detection results for the original 3B model across external data distributions.
Original VLX-Seek 3B results on the RefCOCO family of referring-expression benchmarks
Referring-expression results showing the selection of instances that match a language description.

Release scope and license boundary

The current repository releases inference for the 1.5-10B model. The planned 0.6B and 3B checkpoints are not public. The internally trained OPN described in the project material is withheld under company policy, and WeDetect-Base-Uni is supplied as the default alternative. Training, fine-tuning, data construction, and a robot-control stack are also outside this release. The package is therefore a replaceable region-perception component, not a complete machine-vision platform.

The actual boundary of the current open-source package
Capability Status Explanation
1.5-10B inference code and weights Available Supports command-line and resident-process use across seven visual tasks
0.6B and 3B checkpoints Not available The model family is planned, but no matching public weights are included
Internal proposal network Not available Use WeDetect-Base-Uni or provide boxes from another source
Training and fine-tuning code Not available The repository alone cannot reproduce the two-stage training process
Robot control and policy Out of scope The output is visual understanding and region localization, not actions

The license needs separate attention. GitHub's metadata does not map the custom repository license to a standard identifier and returns an unrecognized value. The actual file is the OmAI Model License v1.0. Its commercial-use clause permits use below US$10 million in worldwide annual revenue and requires a separate written commercial license when that threshold is reached or exceeded. Non-commercial research and evaluation are permitted regardless of revenue. The project should not be described as Apache-licensed.

Deployment cost and fit

The 1.5-10B checkpoint has roughly 10.02 billion parameters. A sixteen-bit copy of the weights alone is nearly twenty gigabytes, before activations, image processing, and cached runtime state. Linear-attention layers, acceleration kernels, and scripts for NVIDIA, Ascend, Apple Silicon, and processor execution improve portability or inference cost, but they do not turn a 10B model into a lightweight detector. Supported hardware does not imply equal latency across those paths.

A conventional detector is usually cheaper when the task is a fixed category inventory. A region-reference vision-language model becomes useful when the workload needs open descriptions, relational conditions, regional explanations, and explicit rejection. The safer deployment pattern is to control proposal count with an external detector and then measure recall, end-to-end latency, and false positives on representative data. For manipulation or mobile robots, perception must also feed an independent safety and policy layer.

The important contribution of VLX-Seek 1.5 is not merely another visual-language checkpoint. It changes the localization interface from coordinate generation to region-entity selection. That makes outputs shorter and easier to resolve, while also exposing proposal recall as a variable the system cannot hide. Production suitability depends on candidate quality, license terms, compute budget, and whether the task can make good use of an explicit absent-target answer, not on the qualitative examples alone.

As an Amazon Associate, we earn from qualifying purchases.