RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

Auto-research agents — LLM systems that read code, propose hypotheses, write patches, launch training, read the metrics, and decide what to try next — have been demoed repeatedly, from FunSearch to AlphaEvolve to the AI Scientist line. But most demonstrations live in the comfortable zone of small models and cheap, deterministic evaluators. Enter a mature production codebase, where each candidate change costs hours of GPU training and a single silent defect — test-set leakage, a missing layer norm, a disconnected gradient, an unwired train/eval flag — burns dozens of accelerator-hours and lets the whole iteration roll on from a wrong conclusion. Four Meta researchers (Zheng Chen, Linfeng Liu, Hong Li, Hong Yan) laid this real scenario bare and built RankEvolve, a multi-agent auto-research framework purpose-designed for evolving ranking models, deployed for twelve full iterations on the open-source HSTU generative recommender.

The paper's real interest lies not in the slogan of "agents doing research" but in its quantitative attitude toward reliability engineering: instead of stopping at an architecture diagram, the authors built an oracle-based benchmark and turned "having two coding-agent products from different vendors cross-check each other" into a falsifiable experiment with controls, matched budgets, and pre-registered hypotheses.

Three layers: turning a research procedure into a runtime-enforced protocol

RankEvolve has three layers, each making a deliberately different claim.

Layer one: the Executable Operating Protocol (EOP). The authors' first assertion: author the research procedure once as a versioned, semi-structured protocol — declaring phases, dependencies, tools, human-confirmation gates, branches, and loops — then compile it into a state machine enforced by the runtime. Every marker (dependency, branch, join, goto, loop budget, tool allowlist) is either statically checked or enforced at runtime; the model performs each step's work, but control of the process stays with the framework. Runtime state is formalized as $\sigma=(q,V,A,G,B,R,T)$ — active phase, protocol variables, committed artifacts, gate status, live branches, attempt/budget counters, append-only event trace — advanced by typed events $\delta(\sigma,e)\rightarrow\sigma'$: node-complete, gate-approved, branch-failed, budget-exhausted. The authors honestly bound the guarantee: dependencies, gates, branch cardinality, loop budgets, tool allowlists, and checkpoint/restart are enforced, while semantic code correctness is expressly not guaranteed.

Why runtime enforcement? A pilot experiment: ten runs received the whole procedure as a single in-context playbook, and none was fully faithful — all ten executed only a subset of proposals, four stopped early, four skipped a gate, three called tools off-spec. Figure 2 shows violations accumulating: rare in iteration 1, proliferating from iteration 2 as accumulated context crowds out the current-step instruction. That is long-horizon drift: a protocol written in a prompt gets diluted by context; compiled into a state machine, it doesn't.

Layer two: a meta-meta-harness. The sharpest design choice in the paper. Each commercial coding product (Claude Code, Codex, OpenHands) is already a harness over a raw model — with its own planner, context management, tools, and sandbox. RankEvolve "harnesses these harnesses," standing two harnessing levels above the model and binding complete products as work nodes of an execution graph rather than issuing raw model calls. The binding is an adapter contract: in go the task, repository snapshot, allowed tools, budget, and prior artifacts; out come a patch, terminal status, event trace, and usage record. Sandboxing, non-interactive invocation, timeout, and usage collection belong to the runtime; the product's internal planning is untouched. This makes the three products substitutable at coding or review nodes while routing, aggregation, gates, and metrics remain deterministic runtime functions.

The runtime instantiates three execution patterns (Figure 3): Linear, a chain with loop-back; Dual, a propose→review→fix consensus loop — in effect internal peer review — repeating until the reviewer approves or residual severity drops below a threshold; and BTA (Breakdown-then-Aggregate), a diamond splitting a task into bounded-concurrency sub-tasks and synthesizing results. These compose into flows such as Plan-then-Implement. The recovery contract is specific: state and artifacts checkpoint at node boundaries; transient invocations get bounded retries; timed-out nodes fail closed; resumed runs start from the last committed boundary.

Layer three: the knowledge layer. A long-horizon agent that forgets re-derives the same dead ends. The knowledge layer does two things: an artifact/evidence store records, per proposal, repository and environment hashes, prompts, EOP and product versions, patches, test outputs, training configuration, splits, checkpoints, metrics, cost, and the promotion decision — with negative outcomes as first-class leaderboard rows. And an experiential memory writes self-contained notes beside the code they describe, abstracts transferable lessons into a central wiki that cites those notes as evidence, and links both tiers through an entity graph. A reconciliation agent keeps the wiki the single source of truth, while a per-phase read path injects only the few relevant lessons, preserving the EOP's per-step context discipline.

Twelve iterations on HSTU: winning on execution, losing on novelty

The target is HSTU, Meta's generative recommender published at ICML 2024; the authors use its public implementation and public MovieLens data, so all results sit at public-benchmark scale. HSTU replaces the Transformer block with SiLU-normalized attention, a fused UVQK projection, and a learnable relative bucketed time-and-position bias — the last being exactly why explicit time-decay add-ons later prove redundant. Published anchors: 0.1895 (BASE) and 0.2098 (LARGE) on ML-20M. The evaluation discipline is one of the paper's finest details: canonical full-test NDCG@10 (leave-one-out, full-corpus ranking) is strictly separated from a faster subset evaluation that ran 0.005–0.010 higher; checkpoint evaluations within one run are not independent seeds, so "best checkpoint" values are descriptive history only. The authors state plainly that without this discipline, the system repeatedly chased phantom "gains" from subset and checkpoint noise. Compute scale: roughly 55 leaderboard entries across 49 experiment directories on 1–8 NVIDIA H200s.

Stage	Iters	Intervention	Outcome	Finding and claim scope
Anchor	0	Published HSTU anchor	0.2098	Published comparison point
I — compose known components	1–2	SSD-style input compression (IC) + PRISM conditioning	0.2140	+2.0%, measured historical endpoint
	3–4	+ multi-token scoring head; preference optimization (DPO/IPO/SimPO)	0.2161 / DPO −35.9%	SYNAPSE +3.0%; preference direction rejected
II — refine	5–7	FLUID time decay; failure-overlap diagnostic; focal/tail reweighting	tie / 75.2% overlap / no lift	decay redundant; diagnostic descriptive only; reweighting rejected
	8	Deep failure analysis + MPTA	0.2154	genre Jaccard ≈0.236; MPTA within noise, rejected
III — push the ceiling	9	Leak-free genre side feature (UDK)	0.2192	Largest endpoint +4.48%, after fixing the v1 leak; a metadata gain, not architectural novelty
	10–11	Position-averaged TTA; UDK channels	≈0.2190 / ≈0.2192	no gain; all three channels saturate at the ≈0.219 ceiling
	12	Inference-time boost-last-K and new methods	±0.0003	within checkpoint noise; no new endpoint counted

The twelve iterations summarize as three stages, with an almost anti-climactic honesty:

Stage I — composing known components (SYNAPSE). The most architectural work tunes and composes three known blocks on the HSTU backbone: SSD-style tiered input compression (IC), an $O(N)$ state compression of long history in the spirit of structured state-space duality; PRISM user-conditioning, a feature-wise FiLM modulation $\gamma\odot\text{item}+\beta$; and a multi-token scoring head that pools the sequence into a "recent intent" and a "long-term preference" token, scored by summed inner products that collapse to a single ANN call under per-facet normalization. None is new — but the tuned composition lifts NDCG@10 from 0.2098 to 0.2161 (+3.0%). The stage's boldest bet — RLHF-style preference optimization — failed across the board: DPO collapsed 35.9% relative to its own 0.2191 reference (0.1405), IPO −8.2%, SimPO −17.9%, while all three reached near-1.0 preference accuracy — a textbook reward-over-optimization signature. The agent's own root-cause analysis at iteration 8 is the key insight: the "rejected" top-K items share ≈0.236 genre Jaccard with the target — they are partially co-relevant, and the preference objective systematically devalues valid candidates. A leak-free re-derivation (mining reject pairs only from internal positions) no longer collapsed but yielded Δ≈0: preference-pair fine-tuning is structurally mismatched to this retrieval objective.

Stage II — refinements with diminishing returns. A FLUID elapsed-gap time-decay add-on tied SYNAPSE (HSTU already carries a learnable relative time bias, making explicit decay redundant); focal/tail reweighting survived three bug fixes and nine runs without gain (HSTU's sampled-softmax already balances per-item gradients — importing a classification technique into ranking is a mismatch); multi-position training augmentation (MPTA) added +0.7% within checkpoint noise. The most valuable output was the iteration-6 failure-overlap diagnostic: define the failure set $F_{T}=\{u:\mathrm{rank}_{u}(i_{n})>10\}$, re-evaluate the frozen model at an internal position for the same users to get $F_{S}$, and compute the overlap $O(F_{T},F_{S})=\frac{|F_{T}\cap F_{S}|}{|F_{T}|}$. On a 0.2127 checkpoint, 75.2% of last-position failures overlapped second-last failures and 88.8% overlapped the union of positions 2/3/4 — failures persist across adjacent positions, so they are addressable from the training side. The diagnostic exposes no test label ($i_{n-1}$ is an ordinary seen training label, used only to locate persistent failures and never reported as a metric), making it a safe post-hoc analysis against any checkpoint. It naturally suggested the training-side boost-last-$K$ reweighting $w_{t}^{\prime}=w_{t}\cdot[1+(\beta-1)\,\mathbf{1}\{n-K-1\leq t\}]$ — a one-function, two-parameter change the agent implemented and smoke-tested without human help.

Stage III — pushing the ceiling. The largest number came from the cheapest move: a leak-free genre side feature (the ML-20M instantiation of UDK). A learnable embedding table $E_{g}$ maps an item's genres to a vector, fused additively on the encoder input only: $\tilde{x}_{t}=x_{t}+\alpha\cdot\mathrm{meanpool}(E_{g}[\mathrm{genres}(i_{t})])$, with supervision computed from raw item embeddings so positives and sampled negatives are treated symmetrically. Here hides the paper's best teaching case: v1 injected genre into the item-embedding lookup used by the loss, so positives carried genre information that negatives did not — training loss collapsed to zero and eval spiked (0.2163) before degrading, a textbook feature leak; the agent detected and corrected it itself. The leak-free v2 robustly achieved 0.2192 (+4.48%), stable across $\alpha\in[0.05,0.20]$, and required fine-tuning from the 0.2140 stack (from-scratch training reached only 0.1911). But genre, year, and popularity channels then all saturated at ≈0.219, and inference-time last-K boosting and position-averaged test-time augmentation added nothing — an apparent 0.22 ceiling.

The authors call this arc the inversion: the only architectural work gave a solid but bounded +3.0%, the boldest objective bet collapsed −35.9%, and the largest number came from the cheapest, least novel move — a metadata feature that then saturated, leaving genuine novelty unproven. Their reading deserves to be remembered by everyone watching auto-research: contemporary auto-research agents are most reliable at disciplined execution, failure diagnosis, and pragmatic information-adding — and least reliable at exactly the modeling novelty we most want from them.

Cross-dataset transfer provides the placebo check: replaying the two exported recipes (SYNAPSE, then UDK) on Foursquare-TKY/NYC, Gowalla, and Yelp with no per-dataset re-tuning beats vanilla HSTU on all four, cumulatively up to +25% (Yelp 0.0321→0.0382) — the discovered levers are portable, though UDK adds little on Foursquare-TKY (0.0200→0.0202).

The core experiment: does product composition buy execution correctness?

The case study proves the system runs. The paper's scientific question is harder: does spending the budget on composing two different products buy execution correctness beyond spending the same budget on a single product? The authors turned the deployment's logged incidents into the ExecML benchmark: 96 private tasks per repository (the public HSTU codebase and LitGPT, a non-recommender training codebase), balancing six incident families — data provenance and leakage, tensor routing, gradient flow, train/eval mode, metric semantics, configuration wiring — half new implementation, half repair. Every item pairs an immutable snapshot with a hidden executable oracle: a patch passes only when it simultaneously satisfies the regression suite, task-specific behavioral checks, scientific-safety invariants, and evaluator-integrity checks. The primary endpoint is all-oracle execution accuracy (EA); the silent critical-defect rate (CDR) is reported separately — an oracle-confirmed fault that leaves the patch runnable but can invalidate a scientific conclusion. All conditions share a frozen resource envelope $B$ (same three roles, per-node limits, tool permissions, action caps, wall-clock and spend caps); timeouts and overruns score as failures and stay in the denominator. CC runs Claude Opus 4.8, Codex runs GPT-5.6, both at maximum reasoning effort. The primary hypothesis was pre-registered: only if CC→Codex→CC exceeds all budget-matched baselines with a task-clustered 95% interval excluding zero may the authors say heterogeneous composition buys execution accuracy; the LitGPT transfer contrast is published whatever its outcome.

Condition	HSTU EA↑	HSTU CDR↓	LitGPT EA↑	LitGPT CDR↓	Cost/task↓
CC, one pass	22.9 ± 10.5	35.4 ± 12.0	20.8 ± 10.1	37.5 ± 12.1	$0.41
CC, extended budget B	33.3 ± 11.8	27.1 ± 11.1	29.2 ± 11.4	29.2 ± 11.4	$0.98
CC best-of-N + blind selection	43.8 ± 12.4	18.8 ± 9.8	39.6 ± 12.2	20.8 ± 10.1	$0.95
CC→CC→CC	45.8 ± 12.5	16.7 ± 9.3	43.8 ± 12.4	18.8 ± 9.8	$0.97
CC→Codex→CC	62.5 ± 12.1	10.4 ± 7.6	56.2 ± 12.4	12.5 ± 8.3	$1.02
Codex→CC→Codex	56.2 ± 12.4	12.5 ± 8.3	50.0 ± 12.5	14.6 ± 8.8	$1.00
Plan-merge (CC∥Codex), stage 2 †	70.8 ± 11.4	6.2 ± 6.0	64.6 ± 12.0	8.3 ± 6.9	$2.40

The result is strikingly clean: the heterogeneous pair CC→Codex→CC reaches 62.5% EA, above every budget-matched baseline (one-pass 22.9, extended-budget 33.3, blind best-of-N 43.8, homogeneous CC→CC→CC 45.8); the paired contrast against the strongest baseline is +16.7 points (95% CI [6.6, 26.7]). The silent critical-defect rate drops from 35.4% to 10.4%. The pre-registered LitGPT transfer contrast holds as well: 56.2% vs 43.8%, paired +12.5 (CI [3.0, 22.0], p=0.008) — the effect replicates on a non-recommender codebase. The five envelope-B conditions run at $0.95–$1.02 per task and 90–109 tool actions, so the improvement is not bought with budget. The reversed pair also beats all homogeneous conditions (56.2%) but trails the forward pair by 6.2 points — which product implements versus reviews matters.

The second-stage deployed topology, plan-merge (CC∥Codex dual planning lanes), posts the table's highest EA (70.8%) and lowest CDR (6.2%) — but its paired lift of +8.3 points over a contemporaneously re-run comparator (McNemar p=0.077, CI [−0.7, 17.3]) is not statistically distinguishable; capped back to envelope B it ties CC→Codex→CC (paired 0.0). The conclusion is crisp: the controlled contrast establishes why it works — pairing two different products — while plan-merge is what we deploy; its advantage comes from the extra budget it is worth spending ($2.40/task, ≈2.4× the envelope), and the inference cost is negligible beside the multi-GPU training it protects: one slipped defect burns hours-to-days of accelerators and contaminates every downstream iteration, an averted loss on the order of 10³× the spend.

The mechanism analysis (RQ3) nails the "why": defining directional complementarity $D_{a\leftarrow b}=P(E_{a}{=}1,E_{b}{=}0)$ — the error mass available for the reviewer to rescue — the heterogeneous pair CC←Codex shows low error correlation (ρ=0.21), large complementary mass (D=0.18), and positive realized gain after subtracting errors the review itself introduces (G=0.06); the homogeneous pair CC←CC has correlation 0.58, little to rescue (D=0.10), and nets ≈0. The regression slope $\hat{\beta}_{1}{=}0.34$ (CI [0.12, 0.56]) excludes zero: the mechanism is not "diversity is good" but complementary error × conversion efficiency. The context-scope ablation (RQ4) completes the EOP evidence: with runtime, state, tools, and product all fixed, toggling only "inject the active phase's body" vs "inject the full protocol" yields paired EA gains of +2.1 early, +6.2 middle, and +10.4 late — monotone growth — because the state machine carries the process, and inactive instructions in the prompt are just distraction.

Honest boundaries

The limitations statement is equally worth quoting: the evidence base is one ranking codebase plus LitGPT, both Python/PyTorch, so cross-domain transfer is not fully established; pretraining may include the public repositories (though tasks, hidden tests, and reference patches are private); oracles are incomplete — mutation testing and audits reduce, not eliminate, false passes; the products are black boxes, so observable tokens/actions/time/cost are matched while provider-side FLOPs cannot be; the knowledge layer was active in deployment but lacks a memory-on/off ablation, so its marginal contribution to proposal quality is not isolated; and the HSTU case study had a human operator at the protocol's gates who also steered the run, so the agent's contribution cannot be separated from the operator's (the ExecML conditions have no human in the loop). The authors even add a footnote disambiguating their system from the like-named RankEvolve that evolves retrieval algorithms (arXiv 2602.16932) — academic hygiene rarely seen in agent papers.

Conclusion: a process immune system for agents

Read back into robotics and embodied intelligence, RankEvolve's three layers state a proposition common to all long-horizon agent systems: when every step is expensive and every failure is silent, reliability is not a function of model capability but of system architecture. The EOP moves the procedure from the prompt into a state machine, curing context drift; heterogeneous product cross-review cures silent defects — neither weapon is new, but quantifying each weapon's effect with budget-matched hidden oracles is the paper's scarcest contribution. The counter-intuitive findings are equally sharp: agents are already fairly reliable at wielding known levers correctly (even catching their own feature leaks), and still unreliable at inventing genuinely new modeling ideas. For teams looking to accelerate the experiment loop with auto-research agents, this paper's deployment recipe — protocols compiled into runtimes, products from different vendors cross-checking each other, negative results on the leaderboard — is worth far more than any "agents will replace researchers" narrative.

The paper (arXiv:2609.39551) says it best in its title: this is a reliable auto-research harness — and reliable, precisely, was engineered rather than hoped for.

flowchart LR
    A[Investigate
code + data] --> B[Research & Propose
BTA parallel workers]
    B --> C[Implement & Experiment
PTI branches · Dual review loop]
    C --> D[Summarize & Evolve
LLM critic reranking]
    D -- "promote → next iteration" --> A
    B -. human gate .-> HUMAN[human]
    D -. human gate .-> HUMAN
    subgraph K[Knowledge layer · persists across iterations]
      L[run state + manifests + leaderboard]
    end
    C -- record hypotheses/patches/negatives --> L
    L -- inject only relevant lessons --> B

No public code is provided for the RankEvolve framework itself. The public HSTU implementation evaluated in the paper corresponds to Meta's open-source facebookresearch/generative-recommenders (method-to-code correspondences in this article: the leave-one-out / ignore_last_n split logic maps to ignore_last_n in generative_recommenders/research/data/dataset.py; the learnable relative time-position bias maps to add_timestamp_positional_embeddings in generative_recommenders/modules/positional_encoder.py; the "dropout 0.2→0.1 was the only clean BASE win" maps to train_fn.dropout_rate = 0.2 and hstu_encoder.linear_dropout_rate = 0.2 in configs/ml-20m/hstu-sampled-softmax-n128-final.gin; the sampled-softmax loss maps to SampledSoftmaxLoss in the same configs; the LARGE 16-block/8-head configuration maps to hstu-sampled-softmax-n128-large-final.gin).