IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

Paper: IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation (arXiv:2610.04074, arxiv.org/abs/2610.04074, submitted 2026-10-02)
Authors: Jiarui Liu, Renjie Tao, Yiwei Liao, Chuanyang Jin, Kai Sun, Xiao Yang, Xinyuan Zhang, Xilun Chen, Zhuangqun Huang, Lechen Zhang, Yongjin Yang, Yinghui He, Weihao Xuan, Rakesh Wanga, Anuj Kumar, Mona T. Diab, Wen-tau Yih, Xin Luna Dong (Carnegie Mellon University and Meta Reality Labs; corresponding author Jiarui Liu)
Code: github.com/jiarui-liu/IdeaScientist  (CC BY-NC 4.0)  Data: Svalbard Idea Vault
Release status: the repository ships the ideation harness, the vault construction pipeline, the RL rewards, training, evaluation and serving behind six command-line entry points. The README is equally explicit about what is withheld: the ablation arms, the human-study materials and the LaTeX table builders are not released.

Figure 1: IdeaScientist at a glance. Top: the Svalbard Idea Vault decomposes every paper into five results-masked views, namely problem setting, challenge, research gap, intuition and proposal. Middle: an orchestrator owns nothing but the research plan and dispatches a gap finder, an innovator, a report writer and a reviewer, while full text is read only by a paper reader in an isolated context. Bottom: performance bars and one in-flight example showing how a mechanism transferred from another field rewrites the proposal.

In One Sentence

IdeaScientist lifts scientific ideation out of end-to-end autoresearch and turns it into a task that can be trained and scored on its own: an orchestrator directs three separately GRPO-trained sub-agents, a gap finder, an innovator and a report writer, over 2.77M decomposed research ideas, and on a Qwen3.6-27B backbone it reaches 74.6% overall, 14.0% above the strongest open autoresearch baseline on the same backbone and 5.9% and 5.1% above Claude Code SDK (Claude-4.8-Opus) and Codex SDK (GPT-5.4) respectively.

Background and Motivation

Most of the visible progress in automated research has happened downstream. Existing systems fold proposal generation together with implementation, experimentation and writing, so when a run succeeds there is no clean way to attribute it: was the underlying research idea good, or was the downstream execution strong enough to carry a mediocre idea? Frontier labs report the same asymmetry from the inside. OpenAI describes its automated research intern as targeting well-defined work under human direction, and notes that high-level planning accounts for only a small fraction of agent output. Ideation is the half that is least automated.

This paper makes ideation a standalone task with an explicit contract. The input is a problem definition, meaning the task input and the desired output, plus a challenge that the current solution fails to meet. The output is a grounded research proposal. The challenge may point at any desideratum of the output or of the process: poor quality, high inference latency, privacy risk. Two questions follow. First, how should an agent produce high-quality proposals for open research problems? Second, when many plausible solutions exist and none is uniquely correct, how do you evaluate those proposals reliably before running any experiment?

Where do research ideas come from? The answer here is grounded in a recurring view from cognitive and psychonomic science: novelty emerges by connecting existing knowledge in new ways, and a solution developed for one problem transfers to a seemingly unrelated one when the two share the same underlying structure (Mednick, 1962; Uzzi et al., 2013; Gentner, 1983; Gick and Holyoak, 1983; Loewenstein et al., 1999). IdeaScientist therefore does not look for insight in prior work from the same domain. It looks for work that solved a similar challenge somewhere else.

The running example is concrete. Web search cuts latency by retrieving a candidate set cheaply and only then applying a sophisticated ranker. Recommendation systems face the same latency challenge and answer it with the same two-stage structure: first-stage filtering, then fine-grained scoring. The mechanisms differ and the fields differ, but the structure, narrow the candidate pool cheaply and then spend compute on the shortlist, is shared, and that is what makes it transferable. The entire method is this intuition made executable, trainable and measurable.

Every existing route pays for something. End-to-end autoresearch systems usually reward experiment selection or downstream success, which entangles proposal quality with coding ability and experimental outcome. Supervising ideation with execution feedback is more direct but expensive, and it conflates ideation with implementation capability. On the evaluation side, reference-free rubrics lean on LLM judges, and judges disagree, show self-preference and reward stylistic features; learned reviewers estimate whether a paper looks acceptable rather than whether the idea is good; factuality metrics only test whether claims are supported by cited evidence. Expert studies add an uncomfortable finding: higher novelty may come at the cost of feasibility once the idea is actually executed.

Training is awkward for the same reason. A proposal has no directly verifiable reward the way code does. Rubric-based reinforcement learning converts qualitative criteria into scalar supervision, but a single proposal-level rubric gives weak credit assignment and invites reward hacking. In multi-agent settings, prior work either specializes roles under one shared terminal objective or distills several trajectories into a single policy, and in both cases it stays unclear which reasoning stage improved and whether the specialization survives deployment. IdeaScientist's answer is to split ideation into stages with defined inputs and outputs, train each stage separately against reference artifacts reconstructed from real human papers, and then recombine the stages inside the full agentic loop.

Preliminaries

Results-masked records. Every paper in the Svalbard Idea Vault is compressed into a results-masked record: the research question, the method, the model design, related work and limitations are all kept, while every measured result is removed, leaving only the evaluation the authors proposed and the falsifiable predictions their method implies. A record therefore describes the direction a human team intended to take without revealing whether that direction worked. Four prose views are derived from it, namely problem_definition, challenge, intuition and solution, and these serve both as the retrieval space and as the building blocks of the reference artifacts.

Temporal cutoff and held-out directions. The cutoff is 2026-01-01. The 14,977 methodological papers published after it are the post-cutoff targets: one part generates training data, where the agent receives only the problem definition and the challenge, produces a proposal, and is scored against the real solution in the target paper; the rest is held out for evaluation. Because retrieval is cut off at the same date, what is being measured is whether a system can infer a research direction before that direction exists in the retrievable literature.

GRPO and role-level rewards. Each of the three producers gets its own LoRA adapter, trained with asynchronous Group Relative Policy Optimization. The reward is not a single proposal-level scalar but a reference-anchored rubric: it scores the standalone quality of the generated artifact, its agreement with the corresponding human reference on substantive research decisions, and a deterministic set F1 over citation sets.

Method

The harness: five roles and one artifact chain

IdeaScientist decomposes ideation into three producer sub-agents coordinated by an orchestrator. The gap finder examines closely related prior solutions in the target domain and identifies shared methodological limitations that leave the challenge unresolved, or even amplify it. The innovator searches other problem settings for mechanisms that addressed a similar challenge and adapts them to the target setting. The report writer develops the surviving intuitions into complete proposals. The orchestrator maintains an evolving research plan and decides which role to call next, and it may return to earlier stages when more evidence or refinement is needed.

The interface between roles is files, not conversation. The gap finder writes gaps.md: one section per assigned axis, containing a mechanism-level summary of prior work and at most two gaps. Every gap must fill a contrastive template, that is, prior methods do X via Y but Y leaves methodological problem Z unresolved, and must list near-miss papers with what each of them is missing, a category label for the methodological flaw (missing mechanism, algorithmic limitation, wrong inductive bias, credit-assignment flaw, training-objective mismatch, representation limitation, scalability limit), why it matters, and an evidence route describing observable evidence that a future method closes the gap. Proposing a solution is explicitly forbidden at this stage.

The innovator appends one candidate block per call to candidates.md: a one-sentence intuition, the gap it attacks, the source inspiration with the source paper id, the borrowed mechanism and one of three novelty types (new topic, recombination of distant concepts, reinforcing a weak connection), a plain-language mapping onto the target problem where pseudocode, formulas, architectures and hyperparameters are all banned, why it could work together with the key assumption that must hold, how it differs from the nearest same-domain work, and the main risk most likely to sink it. The report writer emits a single JSON object, report.json, following the results-masked schema: title, a one-sentence thesis under 30 words containing no metric numbers, core problem, key novelty, problem definition, related work, method, model details, point-by-point differences from SOTA, a proposed evaluation that is a plan rather than a result, falsifiable predictions with their refutation conditions, and three to five limitations. Every citation is an inline database id.

Full-text reading is quarantined on a second tier. A paper reader receives the complete text of one paper in an isolated single-turn context, capped at 30,000 tokens per read, answers targeted questions, and returns a digest to whoever asked. No first-tier agent ever holds full text in its own context. The README notes that this is enforced at the level of tool availability rather than by prompt: a role whose full-text cap is zero simply never receives the tool. The reviewer is the other second-tier role and writes scores.md, where the verdict is survive, revise with the weakest dimension named, or reject, every evidence bullet must carry a paper id, and one strongest objection is required.

Retrieval exposes exactly one keyword-search interface, but two usage regimes that differ in how a role words its queries. The gap finder is assigned one challenge axis and must issue four to eight queries phrased in the exact terminology of that axis and the target problem definition, rephrasing when the returned pool is thin, with the goal of collecting same-challenge same-problem prior art completely, because its gap schema demands near-miss papers and what each one lacks. The innovator must issue six to ten queries that deliberately leave the problem's own sub-field and hunt for transferable mechanisms across at least three different sub-fields; same-domain sources are allowed only when the transfer is non-trivial rather than a rediscovery of recent work. The reviewer uses the gap finder's regime because its job is to falsify novelty claims against same-problem prior art. The report writer picks a regime per claim, since it must cite both the related work being displaced and the origin of the mechanism.

The orchestrator's action space is deliberately narrow: it maintains the plan and dispatches roles, which is precisely why the authors do not train it. If it tries to stop before a final proposal exists it receives up to eight continuation prompts. If a producer tries to return without having written at least 20 non-whitespace characters into its required artifact it receives up to three completion prompts. Each first-tier sub-agent may request at most 20 paper readings per task and is bounded by a per-role turn cap.

Training: one LoRA per role, reference-anchored rubric rewards

Why train the roles separately? The initial design trained the whole pipeline against the final proposal reward, but the base model rarely completed the pipeline, so most samples yielded no informative signal. Role-level training sidesteps that cold start: each role has defined inputs and outputs, human papers supply reconstructable reference artifacts, and a sparse proposal-level outcome becomes stage-level learning signal without prescribing a single tool-use trajectory. Training and deployment keep the same inputs, outputs, information access and action set.

The reward has three complementary components. The reference-free score $r_{\mathrm{free}}$ is an LLM-judge rating of the artifact's specificity, methodological significance, feasibility and evidential support, measuring whether it is coherent and well formed on its own terms. The reference-grounded score $r_{\mathrm{ref}}$ compares field by field against the human reference on substantive research decisions: core limitations, proposed method, evaluation design and related work, while still allowing a valid alternative to score high. The citation score $r_{\mathrm{cit}}$ is a deterministic set F1 measuring overlap between the papers cited by the artifact and those cited by the reference, capturing agreement in literature grounding and retrieval. A validity gate combines them:

$$r_{\mathrm{rubric}}=g\cdot\left(\lambda_{\mathrm{free}}\,r_{\mathrm{free}}+\lambda_{\mathrm{ref}}\,r_{\mathrm{ref}}+\lambda_{\mathrm{cit}}\,r_{\mathrm{cit}}\right),\qquad \lambda_{\mathrm{free}}=\lambda_{\mathrm{ref}}=\lambda_{\mathrm{cit}}=1/3. \tag{1}$$

The gate $g\in\{0,1\}$ is decided by structural, task-compliance, citation-grounding and tool-validity checks, plus one judged methodological criterion for the gap finder and the innovator. The final training reward mixes in a rule-based term:

$$r=0.85\,g\left(\frac{r_{\mathrm{free}}+r_{\mathrm{ref}}+r_{\mathrm{cit}}}{3}\right)+0.15\,r_{\mathrm{proc}}, \tag{2}$$

where $r_{\mathrm{proc}}$ only penalizes protocol violations such as disallowed tool calls, invalid formats and missing required fields, and never rewards the volume of searching or reading. That asymmetry is aimed directly at the most obvious hack, which is padding retrieval. The citation component is set F1 over the two citation sets:

$$r_{\mathrm{cit}}=F_{1}\!\left(C_{\mathrm{gen}},C_{\mathrm{ref}}\right)=\frac{2\,\left|C_{\mathrm{gen}}\cap C_{\mathrm{ref}}\right|}{\left|C_{\mathrm{gen}}\right|+\left|C_{\mathrm{ref}}\right|}. \tag{3}$$

The reference-grounded score is a weighted average over schema fields, uniformly weighted for the gap finder and tiered for the other two roles, with important fields carrying weight 2 and optional fields weight 1:

$$r_{\mathrm{ref}}=\frac{\sum_{f}w_{f}\,s_{f}}{\sum_{f}w_{f}},\qquad w_{f}\in\{1,2\}. \tag{4}$$

The validity gate differs per role. The gap finder requires a legal schema, adherence to the assigned axis, at most two gaps, every cited id resolvable and actually read during the episode, and a full score on the judged methodological focus. The innovator requires a legal schema, one candidate built on the given gap, the same citation checks, and a judged methodological contribution. The report writer requires all mandatory fields plus the citation checks. When the gate fails the rubric components are zeroed but $r_{\mathrm{proc}}$ remains available, so the model can still learn something from not violating protocol.

Appendix B.2 records two rejected designs, and both are informative. The first replaced full text with a metadata-only digest: that variant collapsed during training, with mean reward near zero and most rollouts truncated at initialization, against roughly 0.5 mean reward for the full-text reader. The authors kept full text and bought throughput with asynchronous GRPO instead of shortening inputs. The second was a pairwise innovator reward that asked field by field whether the output beat the reference. It drifted toward preferring longer, more elaborate answers, spiked to about 0.83 and then collapsed repeatedly, which is what a judge rewarding length independently of substance looks like. The absolute reference-anchored rubric held steady near 0.66 and was kept. Three further tweaks are listed: gap depth moved from a soft penalty to a hard cap of two gaps per axis; the report writer moved from a reference list with anchors to bare inline citation ids because about half of early episodes failed the read-before-cite check; and a brittle rule rejecting any digit string was replaced by a judged penalty that fires only when a prediction is stated as a measured result.

Optimization details from Appendix E.1: each adapter is LoRA rank 32 with scaling $\alpha=64$ and no dropout, applied to all linear modules except the output projection and the visual tower; AdamW with constant learning rate $5\times10^{-6}$, weight decay 0.01, no warmup, gradient-norm clipping at 1.0, bfloat16, tensor parallel 8, on H100. Each GRPO step samples 64 prompts with 8 generations each, 512 rollouts in total, and performs 2 gradient updates at training batch 256. Probability ratios are clipped asymmetrically to $[0.2,\,0.28]$, there is no KL penalty in the reward, and maximum policy staleness is one step with an importance-sampling correction. Sequence lengths and step budgets are calibrated per role: the gap finder runs up to 26,624 tokens for at most 2 epochs or 800 steps, the innovator 30,720 tokens for at most 1 epoch or 600 steps, and the report writer 49,152 tokens for at most 2 epochs or 600 steps. The checkpoint with the highest held-out validation reward is selected per role, giving step 380 for the gap finder, 310 for the innovator and 194 for the report writer. The judge is Qwen3.6-27B with reasoning disabled, temperature 0.7 and top-p 0.8. Each role is trained on 40 nodes of 8 H100, with 32 nodes on policy optimization and 8 on asynchronous rollout generation.

Inference budgets are fixed in the same way: the gap finder gets at most 15 turns of 8,192 tokens, the innovator 20 turns of 6,144 tokens, the report writer 20 turns of 16,384 tokens, and the orchestrator at most 100 turns overall. The three IdeaScientist-Base controls on Qwen3.6-27B, Gemma-4-31B and GPT-OSS-120B run under identical caps, while open-source autoresearch systems and general agents run their own native loops with no such caps imposed.

flowchart TD
  VAULT[Svalbard Idea Vault
4.7M papers, 2.8M with full text
2.77M decomposed idea records] --> MASK[Results masking
remove every measured result
keep problem, method, related work, limitations]
  MASK --> REF[Reference artifacts per paper
methodological gap, research intuition, complete proposal]
  REF --> SPLIT[14,977 post-cutoff targets
14,500 train, 200 validation, 277 held-out test
one partition shared by all three roles]
  SPLIT --> ROLL[Async GRPO rollouts
64 prompts x 8 generations per step
512 rollouts, 2 updates, batch 256]
  ROLL --> ART[Role artifact
gaps.md or candidates.md or report.json]
  ART --> GATE{Validity gate g
schema, assigned axis, resolvable citations,
judged methodological focus}
  GATE -->|g equals 0| ZERO[Rubric reward zeroed
compliance term still available]
  GATE -->|g equals 1| RUB[Rubric reward
mean of reference-free, reference-grounded, citation F1]
  RUB --> MIX[Training reward
0.85 g rubric plus 0.15 compliance]
  ZERO --> MIX
  MIX --> LORA[Per-role LoRA update
rank 32, alpha 64, learning rate 5e-6
asymmetric clip 0.2 to 0.28, no KL penalty]
  LORA --> CKPT[Checkpoint with best held-out validation reward
gap finder step 380, innovator step 310, report writer step 194]
  CKPT --> HARNESS[Recombined into the full harness
IdeaScientist-Trained]

Figure 2: the role-level training loop. Reference artifacts are reconstructed from real post-cutoff papers, each producer is optimized against its own artifact rather than the final proposal, and the three adapters are only recombined into the harness at the very end.

Svalbard Idea Vault: 2.77M decomposed research ideas

The corpus is built through the official arXiv API across all subject areas, so that the innovator can borrow from adjacent fields. It covers papers from 1986 to 2026-06-18, deduplicated, with full text collected wherever available; for papers indexed on or after 2026-01-01 the reference lists are walked one hop to pull in cited papers the vault would otherwise miss. The final vault holds 4.7M papers, 2.8M of which, or 60%, have full text. Each paper goes through idea decomposition: the problem definition and the challenge are reconstructed as inputs, and three chained reference artifacts are reconstructed as targets, namely the methodological gap, the research intuition and the complete research proposal, matching the outputs of the three producers exactly. The name is after the Svalbard Global Seed Vault: a seed bank for research ideas.

Training samples start from machine learning papers published after 2026-01-01. The initial pool is 104K candidates, filtered first to those with downloadable LaTeX source and more than 70% of their cited work available in full text inside the vault, then restricted to methodological papers that contribute a new method, model, algorithm, architecture, training procedure or technical framework, excluding surveys, position papers, dataset and benchmark papers, and purely empirical engineering reports. That leaves 14,977 targets, split at random into 14,500 training, 200 validation and 277 held-out test. All three roles share the same partition, so a paper can never be a training target for one role and a test target for another. Qwen3.6-27B converts each human target paper into the results-masked proposal schema, keeping the research question, proposed method, model design, related work and limitations and removing every measured result.

The provenance of the reference artifacts is worth recording. At the gap finder stage Qwen3.6-27B produced 14,970 and GPT-5.4 filled in 7; at the innovator stage 14,934 and 43; at the report writer stage 14,951 and 26. Across all three stages that is 44,855 and 76, or 44,931 artifacts in total, with GPT-5.4 accounting for under 0.2%. A leakage audit was also run: because problem definitions and challenges are reconstructed from complete target papers, method clues could in principle survive. The authors manually compared 50 random samples against their source papers and found 0 of 50 exposed the target method, including method-specific terminology, assumptions, examples or a problem decomposition that gives the solution away.

Experiments

Setup and metrics

The systems under test are IdeaScientist-Base and IdeaScientist-Trained on a Qwen3.6-27B backbone, the latter composing the three trained adapters. Comparisons fall into three families: general research agents, namely Claude Code with Claude Opus 4.8 and Codex with GPT 5.4; open-source autoresearch systems, both end-to-end (AutoResearchClaw, Dr. Claw, DeepScientist) and ideation-focused (ResearchAgent, CycleResearcher with DeepReviewer, SciAgents, AI-Researcher), spanning Qwen3.6-27B, Gemma-4-31B and GPT-OSS-120B backbones; and IdeaScientist-Base itself on matched backbones. Fairness is handled by giving every system the same problem definition and output schema, the same tools over the same offline vault, hidden target papers and no network access, and by instructing the end-to-end systems to stop once the machine learning proposal is complete rather than continuing into implementation or experiments.

Metrics come in two groups. Novelty is measured against three scopes of literature: All, a 1-10 judgment against the closest papers drawn from the complete comparison corpus including post-cutoff work; Cutoff, the same judgment restricted to pre-cutoff papers; and In-domain, a binary judgment of whether the proposal imports a mechanism from a different problem setting and adapts it through a shared underlying structure, which makes it a transfer judgment rather than a comparison against a literature set. Proposal quality uses the six dimensions familiar from conference reviewing, namely relevance, clarity, specificity, actionability, soundness and impact, each scored 1-10. The headline number averages the two group means:

$$\mathrm{Overall}=\frac{1}{2}\left(\overline{\mathrm{Novelty}}+\overline{\mathrm{Quality}}\right). \tag{5}$$

The retrieval ablation adds a fourth and stricter measure, Mechanism non-obviousness, implemented as two separate judge calls. The first receives the target problem, the challenge and the prior papers but never the proposal, and independently derives candidate mechanisms. The second receives those frozen candidates plus the proposal and checks whether any ex-ante candidate reaches the proposal's central mechanism without adding a hindsight-derived recipe. Four scored levels map to $\{0,\,1/3,\,2/3,\,1\}$ for trivial combination, routine extension, non-obvious step and surprising; an invalid candidate set is unscored and retried, and unscored proposals contribute zero over a fixed test-set denominator. The primary judge is Qwen3.6-27B with reasoning disabled, corroborated by Claude-4.8, Gemma-4-31B, GPT-OSS-120B and GPT-5.4.

Main results: novelty is where the gap opens

On the 277 held-out test papers, IdeaScientist-Base is already competitive without any training: 77.7 versus 68.7 overall against Claude Code SDK with Claude-4.8-Opus, a margin of 9.0 points, and 71.9 versus 69.5 against Codex SDK with GPT-5.4. Its lead over the strongest baseline on a matched backbone ranges from 3.3% on GPT-OSS-120B to 12.9% on Qwen3.6-27B. After training, IdeaScientist-Trained reaches 74.6% overall and is the strongest open system in the table, 14.0% above the best Qwen3.6-27B baseline. Isolated held-out role evaluations show the training is improving the producers themselves: gap finder validation reward rises from 56.6% to 66.7%, innovator from 58.8% to 69.5%, report writer from 78.6% to 89.1%. Recomposed into the harness, Trained gains 1.9% in average novelty and 0.4% in average proposal quality over Base.

System (Qwen3.6-27B group, except the two SDK rows)	Novelty All	Cutoff	In-domain	Novelty Avg	Quality Avg	Overall
Claude Code SDK (Claude-4.8-Opus)	68.3	70.4	17.0	51.9	85.5	68.7
Codex SDK (GPT-5.4)	72.4	71.8	19.9	54.7	76.1	69.5
AutoResearchClaw	52.2	50.9	18.4	40.5	74.6	57.6
Dr. Claw	47.5	47.5	18.4	37.8	73.9	55.9
ResearchAgent	46.1	44.5	11.2	33.9	75.0	54.4
CycleResearcher	49.9	48.8	10.5	36.4	75.8	56.1
SciAgents	44.7	44.3	9.0	32.7	73.9	53.3
AI-Researcher	54.6	56.7	19.5	43.6	75.3	59.4
DeepScientist	57.6	58.5	15.2	43.8	77.5	60.6
IdeaScientist-Base	66.0	67.1	67.5	66.9	80.0	73.5
IdeaScientist-Trained	67.3	68.3	70.9	68.8	80.4	74.6

Table 1: the Qwen3.6-27B group from the paper's Table 1 plus the two general SDK rows, on the 277 held-out test papers with a Qwen3.6-27B judge. The In-domain column carries the loudest signal in the whole paper: baselines sit between 9 and 20, while both IdeaScientist variants sit between 67 and 71.

The dimensional breakdown shows where the advantage comes from. On average proposal quality, Base is level with or slightly above the strongest baseline, from +1.8% on Gemma-4-31B to +2.5% on Qwen3.6-27B. On average novelty the gains are far larger, up to +23.1% on Qwen3.6-27B, decomposing into +8.4% against all literature, +8.6% against pre-cutoff literature and +48.0% against same-domain literature. Novelty relative to one's own field requires stepping outside that field's established practice, which the authors call the hardest kind of novelty, and it is exactly where this method stands firmest. Figure 2(a) shows the advantage holding across all six domain groups, with Trained's estimated profile above Base's in five of them.

Figure 3: paper Figure 2(a). Proposal novelty across six research domains on Qwen3.6-27B, with test-paper counts in parentheses. The outer profiles of IdeaScientist-Base and Trained clearly enclose all seven open-source baselines.

Human blind evaluation gives independent corroboration. The authors sampled 50 papers at random from the held-out test set and had two annotators compare IdeaScientist-Trained proposals against proposals from other systems on the same backbone, including Base, with system names hidden. Trained wins significantly against every system except DeepScientist; against Base the win rate is higher but the difference is not significant. Generation failures point the same way: across the 277 test problems, Trained and all three Base runs produced zero failures, while Claude Code SDK failed to return a scorable proposal on 5 problems and Codex SDK on 8.

Figure 4: paper Figure 2(b). Win rates for IdeaScientist-Trained against each system over 50 pairwise blind comparisons, with bootstrap 95% confidence intervals. Every interval clears half except the comparison with DeepScientist, and the advantage over Base is not significant.

The case studies in Appendix I explain the behavioural difference behind the scores. Trained typically builds a tighter connection between the stated failure and the mechanism used to address it, deriving key decision rules from the structure of the problem so that assumptions and expected effects become easier to test, whereas baselines tend to introduce plausible control rules with weaker justification. Four cases are covered, on composable safety alignment, adaptive rollout allocation, capability-grounded subagent abstention and adaptive multimodal speculative decoding, each placing the human method, Base, Trained and seven external systems under the same problem definition and model setup.

Ablations: every sub-agent and the cross-domain retrieval earn their keep

The sub-agent ablation compares three configurations: report writer only, which is the minimal setup able to produce a proposal; report writer plus gap finder; and the full pipeline with the innovator added. Each additional sub-agent raises both All and Cutoff novelty. On Qwen3.6-27B, All goes 0.595, 0.634, 0.660 and Cutoff goes 0.594, 0.640, 0.671, for gains of up to 7.7%; the weaker Gemma-4-31B benefits more, up to 13.0%. Identifying a research gap first and then innovating on that gap helps the system reach ideas that are hard to derive from the problem statement alone.

Figure 5: paper Figure 3(a). Sub-agent ablation on Qwen3.6-27B. Writer, Writer plus Gap Finder, and the full pipeline rise monotonically on both All and Cutoff novelty.

The retrieval-scope ablation tests the central intuition directly by restricting the innovator to same-domain retrieval and comparing it with cross-domain retrieval. Cross-domain retrieval wins on every novelty measure: All +5.3%, from 0.605 to 0.658; Cutoff +3.7%, from 0.623 to 0.660; In-domain +30.7%, from 0.363 to 0.670. Mechanism non-obviousness is the hardest measure for both variants, and cross-domain retrieval still lifts it from 0.335 to 0.357, a gain of 2.2%. In other words, leaving the problem's own sub-field to borrow a mechanism is where the +48% on In-domain novelty comes from.

Novelty measure (Qwen3.6-27B)	Same-domain retrieval	Cross-domain retrieval (this paper)	Delta
All	0.605	0.658	+5.3%
Cutoff	0.623	0.660	+3.7%
In-domain	0.363	0.670	+30.7%
Mechanism non-obviousness	0.335	0.357	+2.2%

Table 2: readings from the paper's Figure 3(c). The largest gain from cross-domain retrieval lands on In-domain novelty, that is, novelty relative to the established practice of the target field.

Figure 6: paper Figure 3(c). The innovator's retrieval scope varied between same-domain and cross-domain, scored on all four novelty measures. Non-obviousness stays the hardest axis for both arms.

Role	Selected checkpoint	Base validation reward	Trained validation reward	Delta
Gap finder	step 380	0.566	0.667	+0.101
Innovator	step 310	0.588	0.695	+0.107
Report writer	step 194	0.786	0.891	+0.105

Table 3: paper Table 11. All three roles improve by roughly 0.10 on isolated held-out evaluation, which is the evidence that the rubric reward actually delivers signal to the stage it is attached to.

Figure 7: paper Figure 5. GRPO training reward and held-out validation reward for the three roles, gap finder on the left, innovator in the middle, report writer on the right.

The training diagnostics also support the reward design itself. Across 27,225 gap finder episodes the median reward is 0.603 with no non-positive rewards; across 3,473 GRPO groups the mean within-group standard deviation is 0.104 and only 0.4% of groups have zero variance, so almost every sampled group supplies usable learning signal. At the selected gap-finder checkpoint, 512 training episodes have a median sequence length of 17.8K tokens, p95 of 22.4K and a maximum of 25.8K against a 26,624 cap, with a median of 6 turns reading 6 distinct papers. Reward is essentially uncorrelated with sequence length ($r=-0.024$), turns ($r=-0.031$) and papers read ($r=-0.017$), so the model is not scoring by writing longer or reading more.

Judge reliability and compute

Can the judge scores be trusted? Two human experts scored 50 proposals from nine systems on the same dimensions. Two observations follow. First, proposal scoring is inherently subjective: the two experts correlate at only $r=0.56$ on individual papers, which is the level of disagreement normal in conference reviewing. Second, the correlation between each LLM judge and the human mean is of the same magnitude as the correlation between the humans themselves, with the top three judges at $r=0.51$ to $0.59$, so an LLM judge is a reasonable proxy for expert assessment. At the system-ranking level, Qwen3.6-27B agrees with the leave-one-out consensus second only to Claude-4.8, with no significant difference between them, and its self-preference bias is negligible and lower than Claude-4.8's, which is the stated reason for choosing it as the primary judge. Report-level Pearson agreement with the consensus is 0.71 for Qwen3.6-27B, 0.77 for Claude-4.8, 0.77 for Gemma-4-31B, 0.67 for GPT-OSS-120B and 0.78 for GPT-5.4, placing all five judges in the same reliability band.

On compute, Trained spends 168,916 output tokens per delivered proposal against 189,942 for Base, 11.1% fewer, so the gain does not come from a larger generation budget. A controlled test-time compute sweep on a fixed 40-paper subset varies only the per-role turn budget: from the low setting to the high setting, report-writer reward rises from 0.290 to 0.465, the completion rate, meaning the fraction of runs that produce a proposal passing the validity gate, rises from 0.60 to 1.00, and the orchestrator turns actually consumed rise from 34 to 55, confirming that the low setting was genuinely compute-starved. With the caps relaxed, more test-time compute still improves quality, and that gain is complementary to role training.

Figure 8: paper Figure 6. A controlled sweep on a fixed 40-paper subset that varies only the per-role turn budget. Left: report-writer reward and completion rate. Right: orchestrator turns actually consumed.

Figure 9: paper Figure 4(a). Pearson correlation matrix between the two human annotators and between each judge and the human mean, with the final column averaging all seven dimensions. Judge-to-human agreement sits in the same band as human-to-human agreement.

Limitations

The first limitation is stated by the authors in Appendix E.2: residual model-memory contamination in a retrospective evaluation. The temporal split controls retrieval, not what is already in the parameters. The authors probed each of the five evaluated models with 10 post-cutoff papers, title only, no abstract and no retrieval, 50 queries in total, and no model could identify or describe any of them as known work. They are explicit that this cannot detect latent, paraphrased or method-level memory, so contamination cannot be ruled out. The results should be read as performance under a controlled retrieval cutoff, not as proof that every evaluated model is parametrically ignorant of the target papers.

The second limitation follows from the nature of the evaluation, and the paper acknowledges it in both related work and experiments: every quality score is a pre-execution proxy. Judge and human experts agree at only $r=0.56$ on individual proposals, which says proposal quality is not a stable ground truth, and the expert study cited in Section 2.2 found that higher novelty may cost feasibility once the idea is executed. IdeaScientist proposals are never executed. Grounded here means supported by literature and traceable to a source mechanism, not validated by experiment.

Two further concerns are mine. First, the reference artifacts and the training supervision are generated almost entirely by Qwen3.6-27B itself, with GPT-5.4 contributing only 76 of 44,931, so the anchors behind $r_{\mathrm{ref}}$ and $r_{\mathrm{cit}}$ inherit that model's preferences and blind spots, and training amounts in part to aligning with one's own teacher's problem-solving style. The primary judge is the same model, even though four external judges and human annotation were used for cross-validation. Second, In-domain novelty is a binary transfer judgment that depends on the judge applying a four-item checklist, namely naming the source setting, importing a concrete mechanism, identifying the shared structure and explaining the adaptation, which makes it sensitive to borderline cases. Together with 40 nodes of 8 H100 per role and the decomposition quality of a 4.7M-paper corpus, these set both the reproduction barrier and the ceiling of this route.

Conclusion and Outlook

IdeaScientist isolates research ideation as a capability that can be trained and evaluated independently of downstream implementation: identify the methodological limitations of closely related work, search other problem settings for mechanisms that may address the same challenge, and develop them into grounded research proposals. Using role-specific training with temporally held-out human papers as reference directions, it improves every stage of ideation and becomes the strongest open system in the paper's proposal-level evaluation. The authors' claim is that structured agent environments can make scientific ideation amenable to controlled learning and evaluation, rather than leaving it as an opaque byproduct of end-to-end autoresearch, and that the next step is composition with automated research-topic identification and automated experimentation.

For a reader, the most reusable part is the evaluation protocol itself: one temporal cutoff line, a set of held-out real human papers, a panel of cross-validated judges, and a task definition that separates the idea from its execution. The code and the full Svalbard Idea Vault are open, with six command-line entry points covering the harness, vault construction, rewards, training, evaluation and serving, so the main barrier to reproduction is compute rather than material. It also gives the AI-scientist direction an unusually controllable benchmark: the contest is not who ran the experiment, but who thought of the direction before the direction existed.

Golden Line

"Novelty emerges by connecting existing knowledge in new ways, and a solution developed for one problem transfers to a seemingly unrelated one when the two share the same underlying structure."