ERNIE Evolve: Toward Automated Research for Real-World AI R&D

From one-shot code generation to sustained automated research

Source: ERNIE Evolve: Toward Automated Research for Real-World AI R&D, ERNIE Research, November 9, 2026.

Model development does not end when a useful piece of code is generated. Researchers must decide which direction deserves investment, design and run experiments, separate real gains from noise, and use the evidence to continue, backtrack, or change course. A productive research process spans many experiments, with each result changing the space of actions available in the next round.

Automated research systems are beginning to participate in that process. ERNIE Evolve is intended not to generate a single implementation for a fixed problem, but to operate under an explicit objective, repository, evaluation protocol, and modification boundary. It proposes research directions, edits code, runs experiments, examines results, and carries both successful and failed findings into later decisions. When the system being improved is itself part of the AI development pipeline, this forms an early form of recursive self-improvement (RSI).

ERNIE Evolve was evaluated on NanoChat AutoResearch, NanoGPT Speedrun, and SOL-ExecBench before being applied to real inference optimization for ERNIE-5.5-Flash. The public tasks show that it can find additional improvements in mature code. The production tasks add stronger baselines, narrower modification scopes, noisier feedback, and a much more complex execution environment.

How ERNIE Evolve organizes research

The system uses two nested loops. The outer loop decides what to investigate. It draws on prior experiments, papers, open-source implementations, and domain knowledge to propose directions, then assigns each direction to a research lane. Multiple lanes can run in parallel, and each lane maintains its own tree of candidate code versions.

The inner loop decides how to develop a chosen direction. It starts from an existing version in the lane, forms a hypothesis, changes the code, runs an experiment, and submits the candidate for independent evaluation and review. Accepted improvements remain available to later iterations. Failed candidates are not silently discarded: their code changes and failure reasons become context for future choices.

Five roles define the division of labor:

Role	Responsibility	Control boundary
Director	Selects the next research direction and starting version.	The starting point can be the original code, the best version so far, an intermediate version, or the latest upstream commit.
Selector	Chooses a candidate within a lane and specifies the next hypothesis.	It considers code changes, failure reasons, and prior decisions as well as scores.
Executor	Implements the plan in an isolated environment and creates a candidate.	Failed experiments and configuration errors do not contaminate other lanes.
Evaluator	Measures performance on real workloads and checks correctness.	It sits outside the optimized repository and is hidden from every agent.
Reviewer	Compares code changes with evaluation results and looks for direction drift or evaluator exploitation.	Drifting candidates cannot become best versions, and evaluation hacks are discarded.

The Experience Bank stores candidates, code changes, experimental results, and failure reasons. The Knowledge Base stores papers, open-source code, and domain experience. Because the Director chooses the research line while the Selector chooses a version within it, the system can both deepen a promising approach and change direction when the evidence warrants it.

This is the central difference from a one-shot coding tool. A candidate tree preserves useful local changes. Experience flow allows one experiment to change the next question. The outer loop can recognize when a direction is approaching its local ceiling. Research becomes a stateful, traceable experimental process rather than a conversation extended with more prompts.

Evaluation integrity is part of the architecture

An automated research system must prevent the optimizer from finding a higher score that does not represent a real improvement. ERNIE Evolve uses a fixed Evaluator to measure performance and correctness on real workloads. Candidates that modify code outside the permitted scope are excluded, and a gain must exceed the noise threshold before it can be accepted.

The Evaluator is kept outside the repository being optimized and is not visible to any agent. The Reviewer then compares the implementation with the measurement. Did the claimed gain follow from the stated hypothesis? Is it exploiting a measurement loophole? Did the candidate quietly change the problem definition? Only a version that passes both independent evaluation and review can be retained.

This creates three levels of verification. First, the result must be correct. Second, the performance gain must rise above noise. Third, the change must still match the intended research direction. Without the third layer, a candidate could specialize to evaluation samples, modify the workload, or bypass correctness checks. Without the first two, the Reviewer could be persuaded by plausible code that produces no real benefit.

Three public benchmarks: leadership, proximity, and limits

The public evaluation spans training algorithms, training speed, and GPU kernel optimization. All three tasks have public protocols and prior published results. ERNIE Evolve applies the same research loop across them rather than adapting the system to one metric or repository structure.

Task	Metric	Best public result	Recursive	Hyra-1.0	ERNIE Evolve
NanoChat AutoResearch	Validation BPB, lower is better	0.8878	0.9109	0.9015	0.8763
NanoGPT Speedrun Track 1	Time to target loss, lower is better	39.9 s	77.5 s	76.4 s	38.9 s
NanoGPT Speedrun Track 3	Steps to target loss, lower is better	2,690 steps	—	—	2,590 steps
SOL-ExecBench	Mean SOL score, higher is better	0.827	0.676	0.736	0.819

The table requires a precise reading. ERNIE Evolve exceeds the best public results listed by the source on NanoChat AutoResearch and NanoGPT Speedrun. On SOL-ExecBench, it beats Recursive and Hyra-1.0 but remains below the leading score of 0.827. The defensible conclusion is therefore leadership on two public training tasks and proximity to the leader on the kernel task, not across-the-board superiority.

The evidence can be checked through the open-sourced ERNIE Evolve results, NanoChat AutoResearch, NanoGPT Speedrun, and SOL-ExecBench. The public-best entries also draw on the Rekursiv report and results from Recursive and Hyra-1.0.

NanoChat: turn the training corpus into additional predictors

NanoChat AutoResearch requires a system to balance architecture, optimizer design, learning-rate scheduling, and execution efficiency within a five-minute, single-GPU training budget. The community baseline is already heavily optimized, leaving limited room for architecture changes alone.

ERNIE Evolve adds two prediction branches beside the neural network. One uses n-gram statistics from the full training corpus. The other retrieves training documents that match the current context and uses their continuations to predict the next token. A gate adjusts the mixture weights according to each branch's performance at earlier token positions. Information that the network cannot learn thoroughly in five minutes can therefore still contribute through a non-parametric path.

To constrain the added cost, the system updates the large input table with a sparse optimizer that stores no momentum state. It fuses softcap and cross-entropy into one Triton kernel and overlaps CPU-side table construction with GPU training. The final validation BPB is 0.8763, below the public best of 0.8878.

NanoGPT: reduce step time, then reduce the number of steps

NanoGPT Speedrun has accumulated more than two years and over ninety community improvements. Track 1 measures the time needed to reach a target loss. ERNIE Evolve focuses on per-step execution cost, improving communication overlap, backpropagation, and output-layer gradient computation. The changes address serial waits, hidden data movement, and expensive gradient work that remained inside a mature training stack. The final version reaches the target in 38.9 seconds, and repeated runs satisfy the loss requirement.

Track 3 measures the number of steps to the target. ERNIE Evolve combines two search branches. One decomposes matrix weights into direction and magnitude. The other applies EMA-Nesterov extrapolation based on an exponential moving average of past parameter displacements. In the original optimizer, normalization constrains persistent update directions and oscillating ones to similar step-size scales. Extrapolation lets persistent movement accumulate across steps and push farther along those directions.

Combining the mechanisms exposed an implementation conflict. The optimizer reconstructs weights from internal state, so extrapolation applied directly to the weights would be overwritten. ERNIE Evolve applies it to the direction and magnitude parameters instead, then adds a learning-rate floor and weight averaging near the end of training. The result reaches the target loss in 2,590 steps, compared with the public best of 2,690.

SOL-ExecBench: real-model kernels still leave room

SOL-ExecBench contains 235 kernel problems drawn from real models. Every candidate must be both correct and fast. More than seven months after the official leaderboard launched, over 700 developers had made nearly 50,000 submissions, and records were still improving.

ERNIE Evolve reaches a mean SOL score of 0.819, ahead of Recursive at 0.676 and Hyra-1.0 at 0.736, and close to the leading 0.827. The result shows that automated research can locate local opportunities in real workloads. It also shows that the system does not yet surpass the strongest implementation on every kernel problem.

Why production AI R&D is harder than a benchmark

A public task may begin with one training script or one kernel. A production inference service spans inference engines, kernel libraries, hardware plugins, communication paths, and service-launch scripts. Every experiment must start a complete service across multiple accelerators, then run load and correctness checks. A configuration mistake anywhere in that environment can invalidate the result.

ERNIE Evolve faced four additional constraints in production:

• Stronger baselines. Engineers had already optimized the target workloads, so further gains depended on remaining details in kernels, communication, and scheduling.
• Narrower scope. Changes had to be suitable for production code, limited to specified modules, and unable to degrade determinism, accuracy, or latency.
• Noisier feedback. Individual gains could be comparable to measurement noise, requiring end-to-end load tests to establish a real effect.
• More complex environments. A change could affect several dependent inference components, so validation had to cover the full service rather than an isolated output.

The team applied the system to three active ERNIE-5.5-Flash problems: low-latency inference on Kunlunxin P800, deterministic inference throughput on GPUs, and peak single-request decoding speed. Every proposed change had to pass the same load tests and correctness checks applied to modifications submitted by engineers.

Kunlunxin P800: from existing fast paths to DeepEP communication

ERNIE-5.5-Flash could already run on Kunlunxin P800, but it was not yet using all available optimized execution paths. Each low-latency decoding step performs little computation. Much of the time goes to Python calls, temporary tensor allocations, and host-device synchronization, repeated at every layer and every step.

The first round connected capabilities that were already available. ERNIE Evolve replaced the many small operators used for manifold-constrained hyper-connections with Kunlunxin's native fused implementation and enabled speculative decoding through the model's built-in multi-token prediction layer. Cumulative speed reached 282.9% of the baseline.

The second round addressed host scheduling and synchronization. Small matrix multiplications bypassed wrappers that were unnecessary during inference; verification stopped waiting for device-to-host synchronization; indices and intermediate data were reused across layers; and attention moved to a native implementation. Cumulative speed reached 588.8% of the baseline. Replacing specialized communication with general-purpose collectives was also tested, but it slowed the workload by 14.0%, so the direction was dropped.

The third round cached indices and work buffers for mixture-of-experts computation, but improved speed by only about 1.4% over the previous round. Instead of continuing to stack changes onto the same hypothesis, ERNIE Evolve reprofiled each stage. The fourth round found that automatic communication selection was still sending decoding requests through the standard path. Fixing dispatch logic and sizing communication buffers for the actual workload enabled DeepEP's low-latency path, bringing the total to 623.6% of baseline.

The curve demonstrates why the outer loop matters. A small third-round gain triggered renewed profiling. The fourth round found a routing problem rather than another kernel micro-optimization.

Deterministic inference: increase throughput without changing reduction order

Reinforcement-learning training imposes strict reproducibility requirements. A request's generated tokens and log probabilities must be bitwise identical regardless of which other requests share its batch. They must also match the training system's recomputation over the full sequence. SGLang's deterministic mode fixes reduction order to provide these guarantees, but that limits GPU utilization.

The task therefore ruled out many ordinary speedups. Batch-dependent kernel selection, atomic additions, and adaptive communication all change reduction order. Permitted changes were restricted to deterministic settings and implementations, with no regression in accuracy or latency.

The search increased throughput by 33.7%, reduced average time per output token by 26%, and reduced first-token latency by 12%, with determinism preserved. Profiling showed that deterministic sampling consumed about 14% of GPU time per step and was the largest bottleneck. ERNIE Evolve replaced it with a Triton kernel that processes vocabulary chunks in parallel. Sparse-attention and small-batch matrix-multiplication improvements then brought the cumulative gain to 20.6%.

The second stage examined deterministic settings. Changing a MoE communication flag alone produced no gain distinguishable from noise, but pairing the flag with corresponding kernel changes was effective. Adjusting the hand-tuned mHC partition setting from 8 to 32 brought the total to 32.6%.

The third stage tested a broader hypothesis: many deterministic kernels used fixed partition counts while each GPU handled very little data per decoding step, so perhaps too many compute units were idle. Tests did not support that claim for most examined kernels. Only a few changes were retained, including two-dimensional parallelism for mHC post-processing, bringing the final gain to 33.7%. The falsified hypothesis was still useful because it stopped the search from expanding an unproductive direction.

Peak decoding: finding 0.3% improvements when no bottleneck dominates

The third task optimized single-request GPU decoding. The service uses speculative decoding: a draft model proposes several tokens, and the main model verifies them in one pass. Cross-GPU communication and kernel gaps had already been reduced by the engineering team. Remaining time was spread across many small kernels in main-model verification and draft-model generation. Some useful changes improved speed by only 0.3% to 0.5% individually, below measurement noise.

ERNIE Evolve explored seven directions, three of which produced gains. The first stage rebalanced compute-unit allocation when the first MoE matrix multiplication ran alongside routing computation. It then changed mHC post-processing to produce the FP8-quantized input required by the next layer, removing a separate quantization kernel. Together these changes improved speed by 5.4%. Three other hypotheses in the same round, long-context attention, cross-GPU communication, and kernel gaps or host overhead, were ruled out by measurement. A cross-GPU operation was already close to the latency floor of an NVLink round trip, while kernel gaps accounted for only 2% to 4% of each step.

The second stage targeted main-model matrix multiplication and MoE scheduling. For inputs with fewer than eight rows, zero padding made up half of the operand data in the original FP8 matrix multiplication. ERNIE Evolve swapped the roles of the weight and input matrices and used native FP16 matrix-multiplication instructions. It also changed the MoE expert scheduler from a sequential scan of all experts to a two-level index. The scheduler produced the largest individual gain; several kernel fusions brought the cumulative improvement to 13.2%.

The final stage targeted the draft model, which remained on the critical path. ERNIE Evolve fused per-position sampling and cross-GPU reduction into one kernel and combined computations in draft attention. The total decoding-speed gain reached 14.9%, with accuracy unchanged.

Where the real contribution lies

The result is not explained by code generation alone. ERNIE Evolve connects directions, candidate versions, experimental conditions, evaluation results, failure reasons, and subsequent decisions into a stateful process. The outer loop chooses the question. The inner loop develops it. The Evaluator and Reviewer prevent the system from making the score better by making the problem worse.

The public tasks show that verified local opportunities can remain in mature projects. The production tasks show that automated research can operate across engines, hardware, and services. Both results are limited to research problems with clear objectives, reliable measurements, and bounded modification scopes. The system has not demonstrated that it can replace human researchers on ambiguous goals, incomplete evaluations, or long-horizon scientific strategy.

It is also important to distinguish disclosed results from independent replication. The numbers in this article come from the ERNIE team's public page and the leaderboards it cites. Metric definitions, evaluation versions, and submission dates should be checked against those sources. To become trusted infrastructure, automated research needs more than performance curves: it needs inspectable code, stable evaluation environments, and long operational histories.

The next step for RSI also requires control boundaries

The ERNIE team describes inference optimization as the first application of RSI to real-world AI R&D. The same loop could extend to training-system optimization, then use capability metrics to guide automated hypothesis generation, experiment design, and validation. A later stage could have AI identify capability gaps and generate or refine evaluation tasks, allowing the evaluation system to evolve with the research.

Greater capability makes boundaries more important. The stated design requirement is that RSI remain controllable, pausable, and terminable, operating within explicit goals, permissions, and resource limits under continuous safety evaluation and human oversight. Scope should expand only while people can still understand, supervise, and stop the system. This is not a principle appended after the engineering work; it is a precondition for sustained automated research.

In practical terms, a deployable RSI system needs at least four controls. The evaluator must be protected from arbitrary changes by the optimized system. Resource consumption and executable actions must have hard limits. Research history must be auditable. Humans must be able to pause or terminate the process at any stage. The faster the improvement loop becomes, the less these controls can depend on the model's own judgment.

ERNIE Evolve does not show that AI can already conduct R&D with complete autonomy. It supports a narrower, more concrete claim: when goals are explicit, feedback is real, and permissions are bounded, automated research can repeatedly produce verified engineering improvements.

The value is not only a faster individual experiment. Successful and unsuccessful results both become searchable experience for later research. That efficiency becomes a dependable AI-development capability only when the research infrastructure, evaluation integrity, and human control mature together.

Original source: ERNIE Evolve: Toward Automated Research for Real-World AI R&D