Route Anything: A Study of Model and Policy Routing for General-Purpose AI Systems

Paper: Route Anything: A Study of Model and Policy Routing for General-Purpose AI Systems
Authors: Pengfei Zhou, Yixing Ma, Hexin Wang, Weidong Tang, Zhenglin Wan, Wangbo Zhao, and Yang You
Publication: arXiv 2611.05275, 3 November 2026, Computer Vision and Pattern Recognition
Paper: https://arxiv.org/abs/2611.05275
Code: https://github.com/InfRec-Cardinal/route-anything
Project page: https://github.com/mikeeeeeeeeeeeeee/RouteAnything
Release boundary: the MIT-licensed repository exposes router code, quality-floor certification, and some code and text replay data. It does not release the paper's 25-benchmark omni-modal trajectory matrix, task-level derived tables, split files, embeddings, media, or trained checkpoints. The implementation can therefore be inspected and parts of it can be replayed, but the complete empirical panel cannot be reproduced from public artifacts alone.

1. The unit of decision is one input, not one model leaderboard

A general-purpose AI system is increasingly an orchestration layer over specialized components. One model answers text, another handles a long document, an image model renders a prompt, a video model produces a sequence, a speech model synthesizes audio, a coding agent invokes tools, and a vision-language-action policy controls a robot. Their quality, latency, and price all depend on the input. A system that always calls the strongest model wastes money on easy requests, while a system that always uses a cheap model accepts avoidable failures on hard ones.

The paper calls every callable model, tool, policy, or complete execution plan a candidate. A router observes a state derived from the input and selects a candidate. Because a candidate may itself be a cascade, retry policy, verifier, or escalation plan, the formulation covers much more than a simple choice between two language models. This abstraction is useful because model selection, tool use, and robot policy selection have the same statistical shape: each option has an input-dependent value and a resource cost.

Prior work has mostly studied language-model routing. It has asked whether a small model can answer easy questions and a larger model can handle difficult ones, or whether a cascade can stop once a verifier accepts an answer. Route Anything widens the question to text, images, video, speech, code agents, workplace agents, and embodied policies. It then asks three separate questions that are often conflated: how large is the routing headroom, how well do learned routers recover it, and when does a cheaper route remain good enough?

The answer requires both a theory and a common evaluation substrate. A router can look accurate on a classification task while producing no quality improvement, and a cheaper route can look efficient while silently violating a quality requirement. The paper therefore separates the attainable oracle from the learned router, and separates average efficiency from the quality floor.

2. A common language for value, cost, information, and certification

For input x, let U(x) be the available candidate set. Running candidate u produces a normalized value V_u and consumes resource D_u, such as dollars, seconds of inference, or robot policy time. A router observes s=σ(x) and chooses g(s). The state may be as coarse as the benchmark name, as structured as a capability type, or as rich as the complete input and a robot camera frame.

At resource price λ, the router maximizes expected value net of priced cost:

$$J_{\lambda}(g)=\mathbb{E}[V_{g}]-\lambda\mathbb{E}[D_{g}]$$

When no natural price exists, the paper compares a router with a named reference policy. If the reference produces Y_B at cost C_B, define the quality change, saving, efficiency, and quality tolerance as:

$$Q=\mathbb{E}[V_{g}-Y_{B}],\qquad \mathcal{S}=\mathbb{E}[C_{B}-D_{g}],\qquad \eta=\frac{\mathbb{E}[V_{g}]}{\mathbb{E}[D_{g}]},\qquad \eta_{B}=\frac{\mathbb{E}[Y_{B}]}{\mathbb{E}[C_{B}]},\qquad Q\geq-\epsilon$$

The four quantities are linked by an exact identity:

$$\eta-\eta_{B}=\frac{Q+\eta_{B}\mathcal{S}}{\mathbb{E}[D_{g}]}$$

This identity prevents a common accounting error. A route improves efficiency only when the quality change plus the cost saving valued at the reference efficiency is positive. Yet efficiency alone is insufficient. A primary model that answers 15% of tasks correctly at one tenth of the reference cost can have 1.5 times the reference efficiency while losing 0.85 in quality. The floor Q ≥ -ε excludes that route regardless of its attractive cost.

Under finite states and positive costs, the optimal priced router chooses in each state the candidate maximizing v(s,u)-λc(s,u). The highest efficiency price is the unique root of:

$$\Phi(\lambda)=\sum_{s}\pi(s)\max_{u}[v(s,u)-\lambda c(s,u)]$$

The result connects budgets, quality constraints, and resource prices. Once a reference policy and tolerance are fixed, a quality-constrained optimum can still be represented as routing at a price. That price is determined by the reference and the floor rather than selected by hand. This is the theoretical basis for the paper's price paths and fixed-sequence certification.

Information determines what a router can achieve. Let W_u=V_u-λD_u. For a state σ, define the best attainable expected return as V*(σ). If σ₁ is coarser than σ₂, the information ladder is:

$$V^{*}(\sigma_{1})\leq V^{*}(\sigma_{2})\leq V^{*}(x)\leq\mathbb{E}\left[\max_{u}W_{u}\right]$$

The left endpoint is the best fixed candidate for a coarse state such as the benchmark identity. The right endpoint is the per-input oracle, which knows every realized outcome in advance. Intermediate rungs correspond to task types, capability leaves, candidate forecasts, camera frames, or other observations. A finer state cannot reduce the theoretical ceiling, but collecting it has a cost, and irrelevant detail can add noise without separating candidates.

flowchart LR
    A[Text, image, video, audio, or robot observation] --> B[State encoder]
    B --> C[Predicted value and cost per candidate]
    C --> D{Routing rule}
    D -->|Quality only| E[Highest predicted value]
    D -->|Quality floor| F[Price-path router]
    F --> G[Certification on independent inputs]
    G -->|Pass| H[Deploy route]
    G -->|Fail| I[Use reference policy]
    E --> J[Candidate outcome log]
    H --> J
    J --> C

The feedback loop matters because the paper insists on matched inputs. A candidate's outcome is useful only when another candidate has an outcome on the same task under the same fallback rule. Otherwise a comparison can measure data completeness instead of model capability.

3. The omni-modal matrix and the size of the opportunity

The empirical substrate contains 25 benchmarks in seven groups: text QA, image-text QA, image generation, video generation, speech generation, code and workplace agents, and robot manipulation. It records 374,401 scored runs from 452 candidate configurations over 103,307 tasks and 104,050 decisions. A candidate configuration includes the base model, harness, effort level, and execution plan, so the comparison is not limited to model names.

Inputs are split 70/10/20 by source group, preventing items from the same source from crossing training, development, and test sets. RoboTwin 2.0 contributes 2,000 test scenes from ten task families that the router never sees during training. Half are clean and half are domain-randomized. This is a stronger test of embodied generalization than randomly splitting frames from the same task.

On 19,814 test inputs for which every candidate has a score, the best fixed candidate per benchmark reaches 82.60 and the per-input oracle reaches 87.98. The 5.38-point gap is a ceiling, not a learned-router result. SciCode is excluded from this panel because later steps depend on code generated by the same model earlier, making the state dependent on the router's own previous choice.

Figure 1: The paper's overview. A router selects a candidate per input; oracle routing shows 5.4 points of average headroom over the best fixed candidate, while priced routing raises efficiency over a reference model.

Group	Benchmarks	Candidates	Test inputs	Best fixed	Oracle	Gap
Text QA	7	3–4	9,482	83.27	87.56	4.28
Image-text QA	3	3	5,708	85.32	88.02	2.70
Image generation	5	4–6	1,515	82.93	87.92	4.99
Video generation	1	3	158	62.43	81.36	18.93
Speech generation	2	3	608	98.62	99.30	0.68
Code and workplace agents	5	4	343	69.17	77.29	8.11
Robot manipulation	1	7	2,000	70.45	88.85	18.40
Overall panel	24	—	19,814	82.60	87.98	5.38

The opportunity is highly uneven. Speech generation is already close to saturation, leaving only 0.68 points of oracle headroom. Video generation and robot manipulation have 18–19 points of room because no candidate dominates across inputs. Those are also output types for which calls are expensive, so a small improvement in candidate selection can have a large economic effect. Text QA has less headroom per input, but its much larger volume means that even a modest gain applies to many requests.

Knowing the benchmark name explains very little of the gap. Choosing the best fixed candidate per benchmark with test-set hindsight moves 82.60 to only 82.62. A rule-based capability tree recovers about 8% of the oracle gap, and a robot scene-condition lookup recovers about 23%. Most useful variation therefore sits inside a task family, in the prompt, image, scene, camera frame, or robot state. Coarse task labels are a starting point, not a solution.

Figure 2: Most oracle headroom remains inside coarse task types. A router needs state information that actually separates candidates rather than a more detailed benchmark label.

4. Learned routers improve modestly, with one strong embodied result

The paper compares several learning strategies. A capability-tree router uses a Qwen3.5-4B classifier to assign an input to a modality, task, and capability leaf, then looks up the best candidate for that leaf. An outcome router predicts each candidate's result on each input and selects the highest predicted value. A winner-label router learns only which candidate won. The last approach is cheaper to label in some settings, but it discards most of the information in a multi-candidate outcome matrix.

Across 16 benchmarks and 3,444 common test inputs, the capability-tree router scores 79.98 against 79.68 for the best fixed candidate: a 0.30-point gain and about 6% of the oracle headroom. It loses 0.17 points on text QA, gains 0.77 on code and workplace agents, gains 0.25 on image generation, gains 2.12 on video generation, and exactly matches the fixed candidate on speech. The result is useful precisely because it is not uniformly positive. Cross-modal routing is not a universal free lunch; it works where candidates complement one another.

Evaluation group	Inputs	Fixed	Router	Router minus fixed	Oracle	Share recovered
Text QA, 3 benchmarks	588	65.14	64.97	-0.17	71.43	-3%
Code and workplace, 5 benchmarks	519	69.24	70.01	+0.77	74.78	14%
Image generation, 5 benchmarks	1,559	82.96	83.21	+0.25	87.92	5%
Video generation, 1 benchmark	161	62.17	64.28	+2.12	80.22	12%
Speech generation, 2 benchmarks	617	98.63	98.63	0.00	99.30	0%
All 16 benchmarks	3,444	79.68	79.98	+0.30	84.80	6%
Robot manipulation, outcome router	2,000	70.45	71.55	+1.10	88.85	6%

The capability classifier is 86–89% accurate at the leaf level, yet 83–86% of inputs still go to the fixed candidate. Within many benchmarks, one candidate wins nearly every type. A high type-classification accuracy can therefore produce almost no routing gain. The paper's image-routing suite makes the point even sharper: 31 designs and 124 configurations on GenEval2 and Qwen-Image-Bench produce no configuration whose interval is above zero relative to the fixed generator, while 44 are significantly worse. An untrained Qwen3.5-4B that chooses a candidate directly scores 1.15 points below fixed routing.

Robots are the strongest learned-routing case. The router reads the instruction, three 240×320 camera frames, and the robot state, then predicts the success of seven vision-language-action policies. It is trained on 49,000 candidate outcomes and tested on 2,000 scenes from ten unseen tasks. ME-Dex, the best single policy, succeeds on 70.45% of scenes. The router reaches 71.55% while reducing policy inference time from 14.51 seconds to 11.05 seconds per scene, a 24% reduction and 1.33 times the success per policy-second.

The behavior tracks a real change in scene conditions. The router sends 929 of 1,000 clean scenes to PatchWAM and 842 of 1,000 randomized scenes to ME-Dex, with another 141 randomized scenes going to GigaBrain. It has not learned that one policy is globally superior. It has learned that the ordering changes when visual conditions change, which is exactly the kind of input-dependent state described by the information ladder.

Figure 3: Robot policy routing by scene. Clean and randomized scenes favor different policies; camera-conditioned routing gains 1.1 points and saves 24% of policy time.

Training targets matter more than architecture in this experiment. With the same inputs, backbone, and training scenes, the winner-label classifier succeeds on 42.8% of robot scenes, while per-candidate outcome regression succeeds on 71.6%. The labels explain the failure. Several policies succeed on most scenes, and a latency tie-break decides 94% of the winner labels. As a result, 24,556 of 31,408 successful training outcomes become wrong-class targets. The classifier learns to choose fast policies rather than policies likely to succeed.

If candidate u has conditional expected value m_u(x)=E[V_u|x], then at λ=0 the optimal router selects the largest conditional expectation. Outcome regression estimates that quantity directly. A winner probability p_u(x) estimates a different event and can prefer a low-mean candidate that occasionally wins ties. The formal distinction is simple:

$$m_{u}(x)=\mathbb{E}[V_{u}\mid x],\qquad g^{\star}(x)=\arg\max_{u}m_{u}(x)$$

Outside robotics, the gains are less reliable. On 11 text, agent, and image benchmarks, the outcome router scores 0.40 points below the matched fixed candidate with an interval crossing zero. A 68-type classifier is 75% accurate but maps 93% of predictions to the same candidate as the true type and finishes 1.32 points lower. The evidence supports a narrow conclusion: learned routing is worthwhile when candidates are truly complementary and the supervision preserves every candidate's outcome.

Figure 4: Learn outcomes, not winners. When several policies succeed, winner labels are rewritten by latency or fixed tie-breaking and discard most of the useful supervision.

The paper also identifies a baseline artifact caused by missing evaluations. Suppose a fixed candidate is admitted only when it has a score on every training input. Opus 5.5 is then dropped from benchmarks where only a few results are missing. On HLE, missing 11 of 1,122 results changes the fixed baseline from 58.24 to 52.90. Once the fixed baseline is evaluated under the same fallback as the router, the reported 0.40-point router deficit becomes a 2.38-point gain. A matched-input baseline is therefore not a cosmetic choice; it can reverse the sign of the result.

Figure 5: Missing scores can flip a comparison. The fixed baseline and router must use the same fallback policy on the same inputs.

5. Priced routes and certified quality floors

The most practical analysis appears on the cost side. When candidates have similar quality but very different prices, the main value of routing is choosing a sufficiently good cheap candidate. The paper uses token prices in CodeRouterBench and RouterBench, image and speech prices in generative tasks, and inference seconds for robot policies. It does not add dollars and seconds into one fictitious unit; dollar efficiency and time efficiency are reported separately.

On CodeRouterBench, each state offers 22 candidates: eight direct model calls, or one of seven primaries followed by the reference Opus 4.6 if verification fails. For a verifier decision H, primary value Y_P, and the cost A of calling the primary plus verifier, the route delivers:

$$V=HY_{P}+(1-H)Y_{B},\qquad D=A+(1-H)C_{B}$$

A price path starts with the best-quality candidate and switches to cheaper candidates at value-cost breakpoints as λ increases. The fitted floor is not accepted as a deployment guarantee. Instead, routes on the path are tested in a fixed order on independent data. The one-sided p-value reported in the paper has the form:

$$p_{k}=\exp\left\{-\frac{n(\hat{Q}_{k}+\epsilon)_{+}^{2}}{2}\right\},\qquad \mathbb{P}(Q_{\hat{g}}<-\epsilon\mid\mathcal{A})\leq\alpha$$

Here Q̂_k is the average quality change of route k on the certification sample. Testing begins at the high-quality end and stops at the first route that cannot reject the null hypothesis of violating the floor. The procedure then deploys the last route that passed, or the reference if the first route fails. Fixed-sequence testing avoids a full multiplicity correction because the order is fixed before looking at certification outcomes.

Testbed and resource	Reference	Route	Test quality change	Efficiency versus reference
Code, dollars	Always Opus 4.6	Fit, zero-loss floor	-0.44 points	25.5 times
Code, dollars	Always Opus 4.6	Certified, zero-loss floor	Floor held in 200/200 splits	19.4 times median
Text QA, dollars	Always GPT-4	Fit, zero-loss floor	-0.40 points	7.1 times
Text QA, dollars	Always GPT-4	Certified, zero-loss floor	Floor held in 200/200 splits	6.7 times median
Image generation, dollars	Fixed generator	Fit, 0.5-point tolerance	-0.47 points	1.35 times
Image generation, dollars	Fixed generator	Fit, 2-point tolerance	-1.56 points	2.03 times
Robot manipulation, seconds	Fixed policy	Outcome router	+1.10 points	1.33 times

Across the four testbeds, the average efficiency multiplier is 8.82. This is an average of four resource-specific ratios, not a claim that dollars and robot seconds can be pooled. The fitted code route reaches 25.5 times the reference efficiency with a 0.44-point test loss. Its certified version keeps the zero-loss floor in all 200 random halvings and reaches a median 19.4 times. Text QA follows the same pattern: 7.1 times for the fitted route and 6.7 times after certification. The gap between fitted and certified efficiency is the price paid for a finite-sample guarantee.

The certification experiment exposes a large difference between fitting and deployment. When a zero-loss route is fitted on half the probing tasks and tested without a separate certification split, only 12.5% of 200 splits satisfy the floor. When the other half is used for empirical-Bernstein certification, all 200 satisfy it, with a median efficiency of 19.4. Training-set compliance is therefore weak evidence when the underlying quality difference is close to zero.

The theory also answers when the reference model is worth calling. Let λ^B be the largest price at which a candidate that calls the reference can be optimal. If it lies below the efficiency-optimal price, the best zero-loss route never calls the reference. In CodeRouterBench, Opus has an admission price of 100 score per dollar, while the zero-loss floor price is 144. The route therefore dispatches every task to a cheaper model. RouterBench is different: the best route that avoids the reference is 9.7 points below it, so a zero-loss route must use cheap primaries and escalate to GPT-4.

Figure 6: When a reference model is worth calling. The shaded range is the price interval in which the reference remains admissible on the optimal path.

Figure 7: Fixed-sequence certification. Routes are tested from the high-quality end toward cheaper routes; the first failure ends the sequence and the previous certified route is deployed.

Certification has real costs and a limited scope. The guarantee is aggregate rather than per-task: the certified route loses quality in a median eight of 21 test cells. It also requires outcomes for every candidate on the certification path and for the reference on every certification input. On CodeRouterBench, certification costs about 0.0158 dollars per task, compared with 0.0124 dollars for always calling Opus. The quality guarantee is therefore a resource-consuming part of the system, not a free post-processing step.

6. What the public code actually implements

The release does not include the paper's private 25-modal matrix, labels, split files, embeddings, media, or checkpoints. It does expose several code paths that make the algorithmic claims inspectable. In experiments/quality_floor/certification.py, price_path constructs route candidates from value-cost breakpoints. This is the implementation of the ordered path that Section 5 certifies.

The same file contains pvalues for Hoeffding and empirical-Bernstein one-sided tests and fixed_sequence for stopping at the first failure. These functions are central to the paper's finite-sample claim. A reader should verify the direction of the null hypothesis, the paired differences, the independence assumptions, and the sample index used for each route account, not only the final efficiency number.

In experiments/learned_price/ridge.py, fit_predict selects regularization by out-of-fold mean squared error and then fits a full model for test prediction. compose reconstructs value and cost for direct calls and reference-fallback cascades from predicted scores, costs, hit probabilities, and accepted values. This code corresponds to the learned quality-cost routes and the price-based action used in the public replay experiments.

The robot path is visible in experiments/omnimodal/robot/train.py. quality_targets returns every observed solver result, success_loss averages over the seven policy heads before averaging scenes, and prediction_record stores the selected and fixed outcomes, probabilities, oracle score, and latency. experiments/omnimodal/robot/model.py defines RouterClassifier, which shares a multimodal Qwen backbone and adds seven independent success logits. This mapping explains why the paper treats candidate outcomes rather than winner labels as the primary learning target.

7. Limitations and conditions for use

The first limitation is the fixed-state assumption. The theory takes the law of states as given and does not cover states that depend on the router's own earlier choices. SciCode is the clearest example: later steps contain code generated by whichever model handled earlier steps, so choosing a model can alter the next input distribution. Long-horizon agents need step-level policies, not just a single task-level choice.

The second limitation is certification data. A quality-floor guarantee needs the reference strategy's outcome on every certification input, and the certification sample must be independent of fitting. In real deployments, reference answers may be more expensive than candidate calls, and for open-ended images, video, or manipulation the evaluation itself may be uncertain. The guarantee is relative to a named reference, a named metric, and a tolerance; it is not a universal statement about human value.

The third limitation is long-horizon step-level routing. The matrix mostly models a task as one decision, with cascades, retries, and verification represented as complete candidates. A deployed agent may choose among tools and models after every observation, with changing state, delayed credit, and verification costs. That problem has a much larger decision space and no straightforward certification guarantee from the present theory.

The release boundary is equally important. The public repository can replay some code and text studies and inspect the router, certification, and composition implementations, but it cannot recreate the 374,401-run omni-modal matrix. The missing matrix is not a packaging oversight that a reader can repair with a larger download; the README states that the matrix and trained artifacts are proprietary. Reproducing the main empirical table requires separately authorized data.

Finally, the evaluation panels are not interchangeable. The fixed candidate, reference, input set, and metric differ from table to table. A score from the robot panel cannot be compared directly with a score from text QA, and a fitted floor is not the same object as an independently certified floor. Deployment should repeat the matched-input comparison, re-estimate the fixed baseline under the deployed fallback, and certify the selected route on data that was not used to choose it.

8. Conclusion: routing is an accountable system choice

Route Anything does not establish that a learned router dominates the strongest fixed model everywhere. The evidence says the opposite in several places: the 16-benchmark gain is only 0.30 points, many image-router designs fail to beat a fixed generator, and some text and image panels show small losses. The useful contribution is a method for locating where routing can work and for stating the condition under which a cheaper route is acceptable.

The robot result demonstrates the strongest case. The router does not make an individual policy more capable; it recognizes that clean and randomized scenes favor different policies, then improves success by 1.1 points while cutting policy time by 24%. The code and text experiments show a different but equally practical case: when quality is close and prices are far apart, a reference policy and a tolerance define a price, and independent certification turns “cheaper” into a quality commitment with a finite-sample error bound.

Three engineering rules follow. Compare candidates on identical inputs and under identical fallback behavior. Train on each candidate's outcome instead of reducing a rich matrix to winner labels. Fix the reference and tolerance before ranking routes, then certify the selected route on independent data. These rules are less exciting than a universal router, but they are what make routing accountable in a production system.

For Agientry's LLM coverage, the paper is relevant because it treats model selection as infrastructure rather than a leaderboard trick. The durable takeaway is not “train one router for every modality.” It is that value, cost, information, and validation must be designed together. That is the step from a routing demo to a service that can explain when it is safe to spend less.