Skip to content
←Back to Papers

PAPER DEEP DIVE

LLMReinforcement LearningMulti-Agent Systems

AutoRL: A Multi-Agent System towards Autonomous Reinforcement Learning

AutoRL keeps reinforcement-learning training under continuous multi-agent supervision: four perceptors detect failures or plateaus, three proposers diagnose and debate interventions, and an orchestrator applies online hot updates or checkpoint-based cold resumptions. Across Qwen3 and LongCat-Flash-Lite, it outperforms static GRPO recipes and a general coding agent.

AutoRL: A Multi-Agent System towards Autonomous Reinforcement Learning

Paper information: Simon Tang, Wenjie Shi, Yongjing Yin, Keheng Wang, Jialin Liu, Zheng Zhang, Linvo Lin, Xiangyuan Xue, Kan Ren, Jingang Wang, and Xunliang Cai. arXiv, 2026, 2611.17790. Searches by exact title, title plus representative authors, full author list, and topic found no public implementation at the time of review. The analysis therefore concerns the system described in the paper, not a source-level reconstruction of unreleased code.

One-sentence summary: AutoRL turns reinforcement-learning training into an auditable control loop in which perceivers detect drift, proposers diagnose the same frozen evidence and debate repairs, and an orchestrator executes either a hot update or a checkpoint-based cold restart.

The engineering problem: an RL recipe is a hypothesis, not a permanent controller

Long-running reinforcement learning is usually launched with a static recipe. Learning rate, group size, clipping, reward implementation, KL regularization, data mixture, and evaluation cadence are fixed before the run begins. This is convenient for experiment management, but it assumes that the best controls remain stable while the policy, generated responses, and failure modes change. That assumption often fails. A reward parser that works on short answers may mis-score a new response format. A healthy entropy level may later indicate collapse. A numerical configuration that survives early updates may generate NaN after asynchronous workers drift apart.

Human researchers do not normally wait for an entire run to end before noticing these problems. They inspect gradients, reward distributions, response length, throughput, evaluation traces, and logs. A failed parser may lead back to code; a plateau may motivate a new entropy schedule; a communication fault may require a checkpoint or a launch-configuration change. Some controls can be applied while the trainer is live, while others require stopping, editing, and restarting. The hard part is not merely choosing among those actions. It is preserving enough evidence and state lineage to know what changed, whether it took effect, and which observations belong to the current model.

Most automated research loops are organized as experiment-level search. One complete run produces a score, and an agent chooses the next run. This can compare broad hypotheses, but it discards the dense evidence generated inside an expensive job. A NaN at step 41, a reward-parser defect exposed at step 3, and a plateau after step 100 are reduced to one aggregate outcome. Waiting for the next experiment is also costly: the degraded job continues to consume accelerators, and the causal chain becomes harder to reconstruct.

Inserting a general coding agent directly into the training loop is not sufficient. Monitoring demands short, frequent reactions. Diagnosing a plateau demands comparative reading, code inspection, historical retrieval, and small tests. If both activities share one context, the flood of rollout logs can crowd out the reasoning needed for intervention. A single agent may also revisit stale logs, repeat an earlier check, or mistake a proposed edit for an applied edit. The system needs role boundaries, evidence versioning, and runtime receipts, not just a stronger model.

AutoRL is designed around three questions: how to detect a problem while useful state is still available, how to separate continuous monitoring from deeper investigation, and how to reject bad proposals before paying for a full GPU restart. Its answer is an organizational one. Perception, proposal, review, execution, and evidence retention become distinct responsibilities inside a shared training system.

System contract: bounded authority over a live training process

The main trainer still performs rollouts, reward calculation, policy updates, periodic evaluation, and checkpointing. AutoRL supervises that process within an authorized tool and configuration surface. It may repair a scorer, adjust supported loss controls, alter sampling behavior, change selected data, or address runtime faults. Evaluation data and the evaluation protocol remain fixed during an intervention study. This boundary matters: an increase obtained by editing the test set would not count as autonomous training progress.

At decision point $k$, the system maps current evidence and history into an action. $E_k$ denotes the observations available for this decision, $H_k$ is the retrievable record of earlier runs and interventions, and $a_k$ is the action approved for execution. The function $\mathcal{F}$ represents the full role workflow, evidence freeze, debate, and execution control rather than a single model call.

$$a_k=\mathcal{F}(E_k,H_k).$$

The four perceivers divide the observable state of training. The training perceiver watches gradients, losses, and numerical stability. The rollout perceiver examines rewards, generations, response behavior, and sample quality. The evaluation perceiver tracks the fixed benchmark suite and error categories. The efficiency perceiver monitors throughput, progress, and operational failures. Each observation is tied to a run identifier, training step, time window, and source artifact. A later agent receives the provenance of a claim, not an ungrounded summary detached from the metric that produced it.

The perceivers first use tools to calculate counts, means, standard deviations, slopes, and normalized deviations. Language-model interpretation is layered on top of those measurements. A gradient spike is considered together with loss and clipping activity; entropy decline is interpreted relative to progress, so normal convergence is not automatically labeled as collapse. Explicit triggers include verified NaN or infinite values and repeated evaluations that fail to improve the best score. When no trigger fires, training continues without an expensive debate.

This separation is a context-management mechanism as much as a division of expertise. Perceivers can use concise, task-specific windows, while proposers receive a frozen evidence packet when a real exception occurs. The architecture reduces the chance that the first agent to search the logs also defines the entire problem. It also gives the proposer context enough room to inspect implementation and history rather than spending it on raw rollout volume.

From anomaly to proposal: independent diagnosis, bounded debate, and executable checks

When an anomaly, regression, or plateau is confirmed, three proposer roles receive the same initial evidence. The Tuner modifies existing coefficients and safeguards. The Innovator searches for alternative mechanisms within the task boundary. The Reviewer traces implementation defects and constructs counterexamples. They diagnose independently before reading one another's proposals. This ordering is intended to avoid anchoring: a persuasive first hypothesis should not determine how the other roles interpret the frozen record.

The initial candidate set and the final recommendation can be expressed as a union of role outputs followed by a debate operator. Each $\mathcal{G}_j$ is the diagnostic process of proposer $j$, and $\mathcal{C}_k^0$ contains their initial candidates. Debate performs critique, revision, merging, ranking, and voting while new evidence $E_k$ may be introduced. The original frozen packet $E_k^0$ remains the common problem statement.

$$\mathcal{C}_k^0=\bigcup_{j=1}^{3}\mathcal{G}_j(E_k^0,H_k),\qquad \widehat a_k=\operatorname{Debate}(\mathcal{C}_k^0,E_k,H_k).$$

Proposals are not accepted merely because their prose sounds plausible. A proposer can copy authorized code and inputs into an isolated worktree, run CPU checks, and record the candidate revision, command, inputs, output, and result. The check receives one of three states: PASS, FAIL, or INCONCLUSIVE when time or dependencies prevent a decisive test. Inapplicable changes require an explicit reason. This creates a useful boundary between a coherent argument and an implementation that actually exercises the target behavior, while avoiding a full accelerator run for every idea.

Surviving candidates are ranked with a Borda-style score. Let $m$ be the number of live candidates and let $\rho_j(c)$ be the rank assigned to candidate $c$ by proposer $j$. Its total score is shown below. Highest-ranked candidates advance, with a stable identifier rule for ties.

$$S(c)=\sum_{j=1}^{3}\left(m-\rho_j(c)+1\right).$$

At most two candidates enter the final vote. A recommendation requires either a unanimous 3:0 vote or a 2:1 majority; when only one candidate remains, it still needs three confirmations. Dissents and unresolved issues remain in the record. Debate may end early if all three proposers select the same eligible candidate after a complete round. A failed check or unresolved blocker cannot be voted away. The protocol therefore combines collective judgment with explicit mechanical evidence.

Execution semantics: NONE, HOT, and COLD are not cosmetic labels

The approved action set contains three forms. NONE leaves the current recipe unchanged. HOT applies a supported configuration or code update at a trainer step boundary. COLD restores a compatible checkpoint and then applies changes that require a new process. $\delta_k$ is the concrete modification, and $b_k$ identifies a checkpoint in COLD mode.

$$a_k\in\{\mathrm{NONE},\mathrm{HOT}(\delta_k),\mathrm{COLD}(\delta_k,b_k)\}.$$

The state transition specifies how much progress each action preserves. Before execution, $x_k^-=(\theta_k^-,z_k^-,c_k^-)$ contains model parameters, runtime and optimizer state, and the effective configuration. HOT retains parameters and runtime state while incorporating the new controls. COLD restores model and runtime state from a compatible checkpoint, then combines the new controls with the inherited configuration. The operator $\oplus$ represents applying the modification while resolving which prior control entries remain active.

$$x_k^+=\begin{cases} x_k^-, & a_k=\mathrm{NONE},\\ (\theta_k^-,z_k^-,c_k^-\oplus\delta_k), & a_k=\mathrm{HOT}(\delta_k),\\ (\theta_{b_k},z_{b_k},c_k^-\oplus\delta_k), & a_k=\mathrm{COLD}(\delta_k,b_k). \end{cases}$$

HOT is powerful because it preserves a live trajectory, but it creates overwrite hazards. The backend's online-control file may describe a complete set of values rather than a patch. A new loss object may silently discard clipping, entropy, or KL settings that were not copied into it. AutoRL therefore checks that the modification lies within the interfaces the training backend actually consumes. Synchronous trainers load controls at training-step boundaries; asynchronous trainers apply trainer-local controls before dispatching policy updates. The perceivers attribute later changes to an intervention only after a runtime receipt confirms that the modification was consumed.

COLD is a longer transaction. The orchestrator decides which prior hot controls should be written into the next launch, replayed, or removed. It requests a save and stop, waits for shutdown, and obtains the code lock. A new job restores a compatible checkpoint with model, optimizer, trainer, sampling state, step count, and available random/data-consumption state. If the old job crashed, the system uses a complete non-corrupt checkpoint. The restored run records its parent and checkpoint lineage, so the active history follows one inherited branch while abandoned branches remain available as research evidence.

Lineage prevents a subtle class of false diagnosis. Suppose a run restores checkpoint 100. Observations through step 100 are inherited. Logs produced after step 100 by an abandoned branch may inform future decisions, but they cannot be treated as evidence about the current model. In the plateau experiment, cold restores from checkpoints 100 and 135 took effect at steps 101 and 136. Without parent-run records, perceivers could blend the abandoned and surviving trajectories into one misleading curve.

AutoRL roles, evidence flow, debate, and execution paths
Paper Figure 2: Four perceivers monitor training while three proposers diagnose frozen evidence. The orchestrator selects HOT or COLD execution, and shared records preserve evidence, interventions, runtime receipts, and checkpoint lineage.
flowchart LR
  T["RL trainer
rollout reward update"] --> P["Training rollout
evaluation efficiency"] P -->|"healthy"| T P -->|"anomaly plateau fault"| F["Freeze evidence packet"] F --> U["Tuner"] F --> I["Innovator"] F --> R["Reviewer"] U --> D["Independent diagnosis
critique merge Borda vote"] I --> D R --> D D --> C["CPU checks
PASS FAIL INCONCLUSIVE"] C --> O["Orchestrator"] O --> N["NONE"] O --> H["HOT
preserve live state"] O --> K["COLD
restore checkpoint lineage"] N --> X["Runtime receipt
shared record"] H --> X K --> X X --> P

The shared record has two layers. Local role histories preserve recent observations and reasoning so a role can resume after context compaction or a new session. Global records index runs, steps, evidence, candidates, checks, interventions, and checkpoints. Data characteristics and domain observations can also be retrieved from a provenance-bearing database. Periodic Markdown summaries link back to original evidence instead of becoming untraceable secondary claims.

Experimental protocol: fixed evaluation, variable training controls

The study covers four main training settings. Qwen3-1.7B and Qwen3-4B train on a processed DAPO-Math-17K set containing 17,390 mathematics problems. LongCat-Flash-Lite uses a private reasoning set of 9,340 mathematics problems and more than 700 repository-level software-engineering tasks. Qwen experiments use gpt-6.1-sol as the supervisory model. LongCat experiments use glm-5.2. The general-agent baseline is Codex with all built-in tools, also driven by gpt-6.1-sol.

The initial Qwen recipe uses a learning rate of $10^{-6}$, eight responses per prompt, and a maximum generation length of 8,192 tokens. LongCat reasoning keeps the same learning rate and group size but raises the answer limit to 131,072 tokens. The coding agent uses fully asynchronous updates, 16 trajectories per task, a $5\times10^{-6}$ learning rate, up to 400 interaction turns, and a 262,144-token context limit. Qwen evaluation uses temperature 0.6 with thinking mode disabled and does not change when an intervention is applied.

The mathematics suite contains AIME 2024, AIME 2025, HMMT 2025, Beyond AIME, and IMO-AnswerBench. General knowledge and coding are represented by MMLU-Pro, GPQA-Diamond, MBPP, and LiveCodeBench v6 Hard. Online reasoning monitoring uses 190 prompts: 30 each from AIME 2024, AIME 2025, and HMMT 2025, plus 100 Beyond AIME prompts. Each prompt is sampled twice, so a monitoring pass covers 380 responses. Final Qwen results sample each benchmark prompt four times and report mean accuracy or pass@1.

Optimization details differ by workload. Qwen uses group-normalized advantages and a token-averaged policy objective. LongCat uses Dr. GRPO advantage estimation. Both use AdamW and gradient clipping at 1.0. Qwen sets AdamW beta values to 0.9 and 0.999 with weight decay 0.1; LongCat uses 0.9 and 0.95 with weight decay 0.01. Qwen starts with entropy coefficient 0, and none of the three initial recipes uses KL regularization.

Hardware layout explains part of the observed failure surface. Qwen shares eight H800 GPUs among actor, training generation, and evaluation generation, using tensor parallelism 2 and data parallelism 4. LongCat reasoning begins on 64 GPUs. The coding agent trains asynchronously on 28 nodes and 224 GPUs, assigning 64 GPUs to optimization, 128 to training generation, and 32 to evaluation generation. Co-located roles are counted once in the reported GPU total.

Main benchmark result: gains across all nine suites

On Qwen3-4B, AutoRL improves every reported benchmark relative to the static GRPO recipe. AIME 2024 rises from 46.67% to 66.67%, a 20.00-point gain. AIME 2025 and HMMT 2025 each gain 11.67 points, reaching 56.67% and 31.67%. Beyond AIME and IMO-AnswerBench increase by 6.50 and 9.75 points. Gains also transfer beyond mathematics: MMLU-Pro, GPQA-Diamond, MBPP, and LiveCodeBench v6 Hard improve by 1.80, 2.52, 1.70, and 1.08 points.

Table 1: Representative Qwen3-4B results. Values are percentages.
BenchmarkStatic GRPOCodex general agentAutoRLGain vs. GRPO
AIME 202446.6750.0066.67+20.00
AIME 202545.0043.3356.67+11.67
HMMT 202520.0026.6731.67+11.67
Beyond AIME26.0025.0032.50+6.50
IMO-AnswerBench27.0016.0636.75+9.75
MMLU-Pro57.9337.7359.73+1.80
GPQA-Diamond44.7045.3347.22+2.52
MBPP63.5531.8065.25+1.70
LiveCodeBench v6 Hard5.714.506.79+1.08

The 1.7B result is more mixed and therefore informative. AutoRL reaches 30.00% on both AIME 2024 and AIME 2025, improving by 3.33 and 8.33 points over the static recipe. HMMT 2025 remains at 18.33%, Beyond AIME rises from 11.00% to 12.50%, and IMO-AnswerBench rises from 8.63% to 14.63%. AutoRL still leads the static recipe on all four general knowledge and coding suites, but trails the general agent by 1.67 points on AIME 2024 and 0.37 points on IMO-AnswerBench. Specialized supervision is beneficial across a broad range, not automatically dominant for every model scale and task.

The 4B comparison isolates a different issue. Relative to the Codex general agent, AutoRL raises AIME 2024 from 50.00% to 66.67%, IMO-AnswerBench from 16.06% to 36.75%, MMLU-Pro by 22.00 points, and MBPP by 33.45 points. The general agent is also below the static recipe on the latter two 4B suites. Knowing how to write code and knowing how to manage a continuously changing RL run are separate capabilities. Role specialization, RL-specific observations, and reversible execution matter in addition to the language model used for supervision.

AutoRL autonomous supervision loop and illustrative performance gain
Paper Figure 1: The loop on the left summarizes autonomous supervision. The right panel illustrates the performance pattern: a static recipe can stall or regress while continued intervention sustains later progress. Later figures supply the measured curves and intervention positions.

Repairing a reward recipe that works only at the beginning

The LongCat reasoning experiment starts from a deliberately fragile recipe. Its training scorer is simplistic, it has no KL regularization, and responses may reach 131,072 tokens. Training does not fail immediately. The defect becomes consequential only as the policy and response distribution move into formats that the scorer handles poorly. A control run keeps the original recipe, while AutoRL may modify the live run.

The first 64 steps are evaluated every two steps, yielding 33 measurements. AutoRL runs for about 115.5 hours and rises from 65.53% to 73.16% at step 64, a 7.63-point gain. The static recipe falls to 56.05%, leaving a 17.11-point gap. AutoRL first reaches its earlier peak at step 58 and matches it at step 64. No checkpoint restart is used in this experiment.

Six HOT interventions form two groups. The first four repair the training scorer: option binding and multi-part answer handling at step 3, reference-answer segmentation and multi-value matching at step 6, LaTeX parsing and degree notation at step 10, and whitespace plus unit handling at step 38. The final two introduce KL regularization at coefficient 0.001 on step 22 and raise it to 0.01 on step 27. The scorer repairs improve the learning signal; KL pressure constrains policy drift. They address different mechanisms of degradation.

This experiment supports a practical distinction between launch validity and run validity. A short smoke test would have declared the static recipe healthy. Only a process that treats the reward implementation as an evolving interaction can explain why a parser that is adequate at step 1 becomes inadequate after many policy updates. HOT preserves the model and optimizer trajectory, allowing the system to compare behavior before and after a repair without replaying the consumed training steps.

LongCat reasoning performance with scorer and KL repairs
Paper Figure 3: AutoRL continues improving while the static recipe regresses. Dashed lines mark six representative HOT interventions, including scorer repairs and KL adjustments. The run does not use a rollback.

Breaking a plateau through accumulated, ordered interventions

The Qwen3-4B plateau study begins from a functioning recipe. The first 100 steps remain fixed while AutoRL observes, discusses, and deliberately postpones one entropy proposal until it has more evidence. At step 101, a COLD action restores checkpoint 100 and repairs answer-option binding in the training scorer. At step 108, the entropy coefficient becomes 0.001. Step 111 adds an auxiliary penalty for a strict repetition pattern.

Later interventions raise entropy to 0.002 at step 121 and 0.003 at step 129. Accuracy is still 29.21% at step 130, then reaches 32.89% at step 135, exceeding the prior plateau maximum. The final entropy change before that rise is the step-129 update, but the paper does not attribute the gain to that coefficient alone. The more defensible interpretation is cumulative: an earlier scorer repair and subsequent exploration changes created the conditions for the later jump. At step 136, COLD restores checkpoint 135 and repairs two training reference answers. Step 153 raises entropy to 0.004, and step 171 improves option binding with proof certificates.

Table 2: Representative interventions around the Qwen3-4B plateau
StepActionExecutionImmediate objective
101Repair answer-option bindingCOLD from checkpoint 100Correct training reward mapping
108Set entropy coefficient to 0.001HOTIncrease exploration
111Penalize strict repetitionHOTSuppress looping behavior
121 / 129Raise entropy to 0.002 / 0.003HOTStrengthen exploration gradually
136Repair two reference answersCOLD from checkpoint 135Correct supervision targets
153Raise entropy to 0.004HOTAdjust exploration again
171Constrain option binding with certificatesHOTImprove reward verification

AutoRL reaches 41.58% at step 180, an 11.84-point increase over the 29.74% plateau maximum. It also exceeds the static recipe's best plotted result of 31.32% by 10.26 points. At the common comparison point of step 190, AutoRL remains at 37.89% while the static recipe is at 30.26%. The ordering is not based on one lucky maximum; it is present at the end of the compared range as well.

The ordering also explains why a single "magic hyperparameter" is a poor description of the result. The system repaired the reward signal before increasing exploration, continued examining reference answers after the first rise, and later revisited reward verification. Each action depended on the state produced by earlier actions. This path dependence is the main difference between process-level supervision and a one-time hyperparameter search.

Qwen3-4B plateau before and after cold restores and hot interventions
Paper Figure 4: Light traces are raw evaluations and dark traces are smoothed curves. Shaded regions mark initial improvement, the plateau, and renewed gains. Hollow diamonds mark cold restores from checkpoints 100 and 135, and the star marks the 41.58% raw peak.

A 352.5-hour coding run with eleven operational failures

The coding workload uses LongCat-Flash-Lite on 28 nodes and 224 GPUs with fully asynchronous updates. The model checkpoints every step and is evaluated every five steps. A trajectory can contain up to 400 tool calls and code executions. Long interactions, distributed communication, MoE numerical paths, and asynchronous policy versions create a larger operational failure surface than the mathematics runs.

The run lasts 352.5 hours, about 14.7 days, and experiences 11 failures: seven NaN events, one dtype mismatch, and three worker or communication failures. Each failure is followed by recovery from a valid checkpoint and a combination of agent repair and restart. The job reaches step 79. Across 15 evaluations through step 75, pass@1 rises from 52.29% to 65.13% at step 55 and finishes at 63.20%. The peak gain is 12.84 points; the final result remains 10.91 points above the starting value.

Repeated NaN events lead to smaller updates and a tighter numerical path. The log-ratio clipping range changes from $[-13,5]$ to $[-5,3]$, probability-ratio spikes are capped at 5, the learning rate and minimum learning rate both move from $5\times10^{-6}$ to $2\times10^{-6}$, and MoE components use FP32 accumulation. The objective is not merely to silence an error. It is to change the conditions that produce divergence and then verify from a checkpoint that learning can continue under the new configuration.

One repair introduces a new dtype mismatch after FP32 residual handling is enabled. AutoRL does not discard every numerical change. It withdraws only the incompatible residual item, retains the lower learning rate, MoE precision, and update constraints, and restores checkpoint 22. This partial reversion is more informative than a clean success story: an intervention batch can be partly useful and partly harmful, so recovery must distinguish what to remove from what to inherit.

Table 3: Failures, recovery, and progress in the long coding run
ObservationRecorded resultInterpretation
Run duration352.5 hoursAbout 14.7 days of unattended training
Failures11Seven NaN, one dtype, and three worker/communication events
Evaluation15 points through step 7552.29% start, 65.13% peak, 63.20% final
Numerical repairClipping $[-13,5]\to[-5,3]$Learning rate $5\times10^{-6}\to2\times10^{-6}$
Partial rollbackRemove FP32 residualKeep learning rate, MoE precision, and other constraints
Eleven failures and checkpoint recoveries during coding-agent training
Paper Figure 5: The curve aligns SWE-related pass@1 with failure events and recovery checkpoints. Red dashed lines mark failures; crosses and circles connect failure steps to recovery points. Repeated-step labels indicate reuse of the same valid checkpoint.

What specialization adds beyond a general coding agent

The controlled comparison gives AutoRL and native Codex the same Qwen3-4B starting model, data, evaluation protocol, and backend model. Runs are compared on cumulative completed training steps. AutoRL reaches 41.58%, while the general agent reaches 32.11%, a 9.47-point difference. The general agent peaks near 100 accumulated steps; AutoRL continues improving through step 180.

The research coverage differs substantially. AutoRL generates 131 initial candidates across eight target categories. The general agent generates 21 design proposals across five categories, concentrated on optimization and system configuration. AutoRL more often examines answer parsing, reward verification, and reference-answer correctness, which are defects in the learning signal itself. Within the 273-step budget, AutoRL applies 20 changes and reverts four, while the general agent applies six and reverts three. The reversion rates are 20% and 50%.

Budget use is even more revealing. The general agent returns to earlier checkpoints four times and reaches only 151 distinct training-step indices. Of its 273 accumulated steps, 122, or 44.7%, revisit a prior index. AutoRL removes four changes through HOT control rollback and does not restore a checkpoint in the main comparison, so all 273 accumulated steps advance to new indices. Reverting a control entry and replaying an old trajectory have very different consequences for both model progress and compute cost.

The accounting separates useful progress from repetition. The paper defines accumulated budget as retained distinct steps plus revisits, written below. The general agent has $U=151$ and $R=122$; AutoRL has $U=273$ and $R=0$. This is a research accounting device, not a direct GPU-hour measurement. Initialization, generation, evaluation, waiting, checkpoint I/O, and recovery still contribute to wall-clock cost.

$$B=U+R.$$

Table 4: Equal-budget comparison over 273 accumulated training steps
MetricCodex general agentAutoRL
Best accuracy32.11%41.58%
Initial candidates / categories21 / 5131 / 8
Applied / reverted6 / 320 / 4
Reversion rate50%20%
Distinct training steps151273
Revisited steps122, or 44.7%0
AutoRL and general-agent research trajectories under equal budget
Paper Figure 6: The left panel compares proposal categories with applied and reverted changes; the middle separates new and revisited training steps; the right compares raw, best-seen, and peak accuracy. Together they show that the gain comes from more than proposal volume: execution and budget use also differ.

The supervisory model still matters. Replacing GPT-6.1 with GLM-5.2 while keeping AutoRL reduces the peak from 41.58% to 32.89%, an 8.69-point drop. That result is only slightly above the 32.11% GPT-driven general agent. Over the shared budget, the last-five-evaluation means are 39.37% for GPT AutoRL, 30.68% for GLM AutoRL, and 29.16% for the general agent. The framework supplies a better execution path, but it does not erase differences in problem discovery and proposal quality.

The full intervention record shows another contrast. GPT AutoRL uses 17 HOT actions out of 20 interventions. GLM AutoRL uses eight HOT actions out of nine. All eight general-agent changes require restart. Within the 273-step comparison, GPT AutoRL reaches 273 distinct indices, GLM AutoRL reaches 193, and the general agent reaches 151. The GLM run narrows its research scope, yet still avoids 42 repeated steps relative to the general agent. Role structure and live control therefore contribute independently of the backend model.

AutoRL comparisons with GPT and GLM supervisory models
Paper Figure 7: The panels compare HOT intervention share, training-step coverage, peak accuracy, and last-five-evaluation means. GPT and GLM both benefit from AutoRL's execution framework, while problem selection and the achievable ceiling remain sensitive to the supervisory model.

Limits and open risks

The authors explicitly note that the current agents excel at tuning an existing recipe, repairing implementations, and making local reward improvements. The 131 initial candidates become 51 joint recommendations after debate. The orchestrator approves 28, defers 21, and rejects two; 20 new changes are actually applied. None of the three candidates that attempt to redesign the learning rule is deployed. The paper therefore does not establish that AutoRL can invent a fundamentally different reinforcement-learning algorithm.

Supervisory capability is a second limit. The 8.69-point peak gap between GPT and GLM accompanies different research behavior: GPT more often inspects parsing, reward validation, and reference answers, while GLM leans toward optimization and reward shaping. AutoRL shortens the distance from a good observation to a controlled change, but it cannot guarantee that the supervising model will form the right hypothesis. Higher-stakes deployments would require separate evaluation of the proposal distribution and capability ceiling.

Changing the training reward while holding evaluation fixed reduces one kind of leakage, but it does not eliminate reward gaming. A scorer repair can correct a real defect, or it can unintentionally accept an answer pattern that the evaluator happens to reward. Frozen benchmarks, proposal review, and recorded scorer changes are safeguards, not a formal guarantee. An automated system should continually measure divergence between training and evaluation behavior, especially when it can edit the source of the learning signal.

External validity is limited by models and domains. The evidence comes from two Qwen3 scales and LongCat-Flash-Lite, with tasks concentrated in mathematical reasoning and repository-level coding. Robot control, embodied interaction, offline RL, and multimodal environments introduce noisier state, expensive simulators, and different reward-repair problems. There is no basis for assuming that the same perceivers and proposers transfer unchanged. For Agientry's large-language-model section, the paper is relevant as general training automation, not as an embodied-policy experiment.

Resource accounting also remains incomplete. Long-lived agent sessions consume tokens while reading logs, inspecting code, checking candidates, and debating. The paper reports GPU layouts, training steps, and wall-clock duration, but not a full breakdown of supervisory token cost, human maintenance, or the marginal value of a correct intervention. Process-level automation may be economically unattractive if avoiding a short bad run requires more agent compute than the run itself.

Finally, observability has a boundary. Perceivers depend on logs, metrics, and evaluation summaries. Missing or misleading signals can produce a well-reasoned diagnosis from bad evidence. Provenance makes those failures easier to audit, but it does not automatically detect every recording bias. The 352.5-hour run demonstrates recovery from explicit faults, not immunity to silent quality decay. Better counterfactual comparisons, statistical power estimates, and independent audits are needed beyond simple NaN and plateau triggers.

Conclusion

AutoRL's central contribution is to move autonomous RL from launching an experiment toward maintaining research state during an experiment. Perceivers create role-specific observations, proposers diagnose frozen evidence and debate candidate repairs, and the orchestrator chooses NONE, HOT, or COLD execution. Shared records connect the intervention to a runtime receipt and a checkpoint lineage, making the next diagnosis depend on what actually happened rather than what an agent intended.

The experiments support three bounded conclusions. Specialized supervision improves all nine reported Qwen3-4B suites over both static GRPO and the general agent. Process-level intervention repairs a reward recipe that initially appeared healthy and restores progress after a long plateau. A coding run survives 352.5 hours and 11 operational failures while pass@1 rises from 52.29% to a 65.13% peak. These results show that the bottleneck in long unattended training is not only launch orchestration; it is the repeated interpretation of changing evidence and the preservation of useful state across interventions.

The same evidence marks the boundary. AutoRL tunes, repairs, and makes local changes, but it does not yet demonstrate automated invention of a new base learning rule. The supervisory model remains a major determinant of research quality. Reward modification creates unresolved safety and gaming concerns. Full economic cost is not reported. The most valuable next step is not another dashboard or a larger debate transcript, but interventions that actively choose informative controlled experiments and quantify uncertainty before spending more training compute.

Golden sentence: The hard part of autonomous reinforcement learning is not changing one hyperparameter; it is deciding which intervention deserves to survive while training keeps rewriting the evidence around it.

Related Papers

ServerBench: How can your coding agents design specialized LLM serving systems?

ServerBench: How can your coding agents design specialized LLM serving systems?

ServerBench evaluates whether coding agents can design complete specialized LLM serving systems through stateful coding-session replay, latency-qualified dollars-per-dollar scoring, and independent verification.

Xinrui Zhong, Zhiming Wang, Yongqi Liu · 537 views · Nov 4, 2026
LLMLLM Serving
Nov 4, 2026
Route Anything: A Study of Model and Policy Routing for General-Purpose AI Systems

Route Anything: A Study of Model and Policy Routing for General-Purpose AI Systems

Route Anything unifies model and policy routing across text, images, video, speech, code agents, and robot manipulation. Its 25-benchmark trajectory matrix shows 5.38 points of per-input oracle headroom over the best fixed candidate; learned routers gain 0.30 points overall, while priced routes raise efficiency by 8.82x on average and a certified zero-loss code route holds its quality floor in all 200 splits.

Pengfei Zhou, Yixing Ma, Hexin Wang · 311 views · Nov 3, 2026
LLMModel Routing
Nov 3, 2026
GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling

GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling

Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of the art is currently split into simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3-4 bits per parameter (bpp), and "second-generation" vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier but are notoriously hard to implement and to scale. In this paper, we ask whether this gap is fundamental, or whether a carefully optimized scalar quantizer can recover most of it. We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid. GSQ matches the cardinality of the relaxation to the small number of levels available in the target bit-width regime (e.g., 3-8 levels for ternary and 3 bpp, respectively), making optimization tractable. Practically, on the standard Llama-3.1-8B/70B-Instruct models, GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits, while using a symmetric scalar grid with group-wise quantization, and thus remains compatible with existing scalar inference kernels. We further show that the same discrete-assignment optimization can be applied to practical GGUF K-Quant checkpoints: starting from publicly released GGUF models, GSQ improves accuracy while projecting the result back into the same deployment format. Finally, GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply. The source code is publicly available at https://github.com/IST-DASLab/GSQ.

Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan · 283 views · Apr 20, 2026
LLMModel Quantization
Apr 20, 2026
Rufus-Air: An Open LLM Post-Training Recipe

Rufus-Air: An Open LLM Post-Training Recipe

Rufus-Air is an open and reproducible post-training recipe on GLM-4.5-Air-Base (106B-A12B), organized as a serial pipeline of eight stages: SFT, Reasoning RL, Coding RL, Instruction-Following RL, General Agent, Coding Agent, Search Agent, and RLHF. We document the data, reward design, infrastructure, stage order, and stagewise results needed to reproduce the recipe. Stages progress from basic to advanced capabilities and from hard, verifiable rewards to softer judge-based signals. Training builds on open-source components and public data, much of it used as released, without new human annotation or an in-house distillation teacher. Our main findings are that (i) diverse, high-quality SFT establishes a strong capability floor; (ii) difficulty filtering keeps RL prompts within a productive learning range; (iii) reward reliability provides a practical principle for ordering stages; and (iv) infrastructure and engineering choices are part of the recipe, not just an implementation detail. Rufus-Air improves over the official GLM-4.5-Air post-trained release and is competitive with similarly sized open models.

Chia-Yuan Chang, Renyuan Cheng, Rui Feng · 334 views · Sep 24, 2026
LLMPost-training
Sep 24, 2026

As an Amazon Associate, we earn from qualifying purchases.