DiscoveryRSI: Recursively Learning to Discover Valuable Hypotheses

Paper Metadata

Title: DiscoveryRSI: Recursively Learning to Discover Valuable Hypotheses

Authors: Jianyuan Zhong, Chenwei Wu, Zhaochen Yu (co-first authors), Xixian Jiang, Xuewei Li, Hao Xiao, Xilin Dai, Junyang Wang, Jiaru Zou, Zhiqing Cui, Boyuan Zheng, Jing Xiong, Zhen Zhang, Qiang Gao, Michael Morley, Luis Nakayama, Qiang Xu, Liyue Shen, Tao Gui, Leo Anthony Celi, James Zou, Zhenfei Yin, Yingcheng Wu, Ling Yang (the Gen-Verse team, jointly with MIT, Harvard, Stanford and other institutions)

Links: arXiv:2610.23839 (v1, 2026-10-16) · arXiv HTML full text · model Gen-Verse/DiscoveryRSI-27B · benchmark DFM-Bench · live demo DiscoveryRSI Arena

Code status: released. github.com/Gen-Verse/DiscoveryRSI (Apache-2.0) publishes a portable agentic kernel: the candidate-hypothesis data structure, the propose-critique-select-retain loop, an append-only replay store, a research manifest, five role prompts (Lead / Generation / Evolution / Critique / Compiler), a public scoring rubric and a tool registry, and it can serve DiscoveryRSI-27B through an OpenAI-compatible endpoint started with vLLM. Worth stating plainly: the repository is a 433-line minimal runnable harness. The production multi-agent orchestration, the training pipeline for hypothesis credit optimization and the DFM-Bench scorers are not in this repository (DFM-Bench is a separate repo), so the public code reproduces the protocol skeleton rather than every number in the paper.

Background and Motivation

Scientific research agents have spent the last two years walking from "generate ideas" to "run experiments". Co-Scientist develops hypotheses through multi-agent evolutionary refinement, AI Scientist-v2 connects idea generation to experiment execution with agentic tree search, Biomni integrates biomedical tools and data, ScienceIDE builds a learning environment out of scientific codebases, and ScienceBuddy turns a researcher's interactions into executable tasks and feedback. These systems already produce respectable proposals, yet they share one question that has never been answered seriously: why does the experience generated by the research process itself never become a training signal for improving the agent? A human scientist who finishes a study remembers which control worked and which revision saved the project, and designs the next study accordingly. The experience of today's research agents dissipates the moment an episode ends.

The authors split the obstacle into two layers. The first is on the execution side: translating a proposal into an executable investigation requires parsing its data requirements and validation criteria without losing the research question itself. Many proposals read beautifully yet never say which data they use, what they compare against, which outcome would count as support and which would count as refutation. The second layer, on the learning side, is the more fundamental one: the final score of a hypothesis does not indicate which contributions to reinforce. One agent proposes an explanation, another adds a useful control, a third revises the investigation plan in light of new evidence. If only the final proposal is kept, all three contributions are flattened into one number. Assessment is made of the proposal, while the training target belongs to the history that produced it, and there is an attribution gap between the two.

That attribution gap is exactly where this paper cuts in. Closing it requires research history to be traceable: every hypothesis version keeps links to its predecessors and to its supporting evidence, every assessment keeps the records it was based on, and every execution keeps its failure and delivery events. Only when assessment, contribution and action are strung together by one shared record can an evaluation become supervision over a concrete agent action instead of a score floating in the air.

Hence DiscoveryRSI: a general framework that turns hypothesis-driven research into a recursive learning process. It grows a research tree of testable hypotheses with multi-agent evolutionary search, compiles selected computational investigations into executable Harbor tasks, and preserves the evidence and decisions supporting every proposal and revision. The accumulated research history drives three coupled improvement mechanisms - hypothesis credit optimization, harness replay and rediscovery - which respectively update model weights, update the research workflow, and put accepted updates back onto new questions to produce the next round of experience.

The companion DFM-Bench answers the question "assess what". Rather than only asking whether the final hypothesis is good, it evaluates at five levels: whether a single hypothesis is worth doing and testable, what one revision contributed, whether a development path accumulated useful evidence, how a whole research round allocated effort, and what independent scientific contributions emerge across proposals. This stratification is the first time "a good proposal" and "a productive research process" are measured separately.

Preliminaries

The lineage of recursive self-improvement (RSI). Reflexion, GEPA and ACE retain experience through verbal memory, prompt evolution and contextual playbooks; harness optimization extends adaptation to agent workflows and implementations; Darwin Godel Machine and Hyperagents study persistent program-level improvement; ScienceBuddy couples harness refinement with model learning; RSI-Anything steers interaction and system updates with a predictive world model. DiscoveryRSI moves this line onto hypothesis-centric research: the evolving claims, evidence and investigation decisions form one shared record that simultaneously feeds model learning via contribution attribution, harness replay, and subsequent rediscovery.

TrueSkill pairwise preference inference. The first step of hypothesis credit optimization needs a model that can aggregate "who beats whom, by how much, and with what confidence". TrueSkill was built to rate chess players: each entity carries a Gaussian belief $\mathcal{N}(\mu,\sigma^2)$, and one game outcome updates both beliefs through a likelihood with a draw margin. This paper replaces the entities from chess players with versioned hypotheses, and the games with pairwise preferences from blinded critique, so $\mu$ expresses relative preference strength under a unified rubric and $\sigma$ expresses how thin the evidence is - the more a hypothesis has been compared, the tighter its belief.

Harbor tasks. Harbor is an execution framework for containerized reproducible tasks. Compiling an investigation plan into a Harbor task means giving it concrete inputs, a runtime environment, required outputs and validation checks, which turns "can this investigation actually run" into a decidable contract question. The paper deliberately separates execution readiness from scientific value: a task can pass every execution check and still be scientifically worthless, and conversely a scientifically important proposal may depend on data that is not yet available and therefore cannot compile today.

Method

Definition 1: the hypothesis as a four-tuple

The foundation of the whole paper is one operational definition. Relative to a documented evidence base $\mathcal{E}$ and stated data, experiment and resource constraints $\mathcal{B}$, a research hypothesis is represented as

$$\mathcal{H}=(c,\pi,d,a) \tag{1}$$

where $c$ is an evidence-supported claim, $\pi$ is the investigation plan, $d$ is the decision criterion and $a$ is the set of related alternative explanations. "Valuable" means that resolving the uncertainty in $c$ could materially add to current understanding, or change a research decision that is grounded relative to $\mathcal{E}$; "testable" means $\pi$ is feasible under $\mathcal{B}$ and its observable outcomes, read through $d$ and $a$, can support, weaken or qualify $c$, with unresolved distinctions stated explicitly. Each of the four components has a precise job. The claim must name the system or population, the scope, the observables and the scientific uncertainty. The investigation plan must name the data or measurements, the comparisons, the controls, the analysis and the dependencies. The decision criterion must state, before results are seen, which outcomes support or weaken the claim and which remain uncertain. The alternatives must name competing explanations, confounders and measurement limits, plus the comparisons that would separate them and the boundaries of those comparisons. Note that this definition evaluates a research opportunity and does not assert that $c$ is true - a supported "no additional information" conclusion is just as valuable, because it stops wasted investment in a more complex model.

Figure 2: one instance of Definition 1. This MIMIC-IV proposal targets adults who survived at least 48 hours of a first ICU admission. (a) asks whether repeated creatinine-urine output discordance provides additional information about readmission or in-hospital death within 48 hours after ICU discharge; (b) B0-B2 compare three logistic models over clinical baseline, current discordance and repeated discordance, L1 additionally adds a hidden Markov summary constrained by data support and stability checks, and the models share a patient-level train/tune/test split; (c)(d) the decision criteria and alternative explanations bound the reading, and the paper claims no clinical conclusion anywhere.

Research trees and linked records: making experience traceable

Every node is a versioned hypothesis carrying its investigation plan, decision criteria, alternative explanations and evidence. Parent links record scientific derivation: a revision has one predecessor, a crossover can have several parents, so the genealogy is really a directed acyclic graph and "research tree" is only the intuitive name. The tree persists across episodes and is a different object from the sequence of agent actions that produced each node - the latter is the training unit, the former the assessment unit, a distinction the authors repeat throughout Appendix A.

The research tree in turn sits inside a larger network of records: hypotheses link to sources and data, to drafts and supporting artifacts, to scientific assessments, to agent contributions and to execution records. Files hold the research material while registry entries, identifiers, paths and content hashes hold the relationships between them. One shared artifact may support several hypotheses without making them scientific descendants of each other - this semantic matters, because it keeps evidence reuse from being misread as genealogical inheritance. These links are what let an assessed contribution be traced back to the history that produced it, and let a recurring failure be traced back to the decisions and execution events behind it.

Evolutionary operations and role division

Following operator-style search (as in Frontis-MA1), the system provides four operations: evolution develops a hypothesis, repair fixes an identified defect, crossover merges contributions from multiple parents, and scientific restart opens a new question without designating a parent (prior work may stay in context). The Lead picks direction, parents and operation; Generation and Evolution investigate evidence and publish proposals or revisions; Blinded Critique compares scientific value under matched data and review conditions and returns a preference, a tie or an abstention; ancestor-aware Progress judges substantive advancement, necessary repair, duplication or insufficient evidence relative to the true predecessor. Every assessment is written back into the linked records, becoming raw material for later learning.

From hypothesis to executable investigation

Selected computational plans are compiled into Harbor tasks with concrete inputs, environment, required outputs and validation checks. Compilation preserves the scientific commitments of the hypothesis: the population or biological system, the proposed comparisons, the discriminative outcomes, the interpretation criteria. Unavailable data and physical experiments are kept as explicit dependencies rather than quietly dropped. Execution checks decide whether the task meets its computational contract; scientific assessment decides what the investigation can establish. Separating the two means hypothesis development can supply learning experience before an investigation has been fully executed - and that separation is precisely what lets the recursive loop turn.

Hypothesis credit optimization: from assessment to bounded supervision

This chain has three links: scientific assessment determines hypothesis credit; contribution records allocate credit across the documented research history; supported actions in that history become training targets.

Link one, assessment. Blinded Critique returns preference, tie or abstention, and TrueSkill aggregates relative preference strength and uncertainty:

$$z_{h}\sim\mathcal{N}(\mu_{h,0},\sigma_{h,0}^{2}),\quad t_{h,e}\mid z_{h}\sim\mathcal{N}(z_{h},\beta^{2}),\quad P(a\succ b\mid z_{a},z_{b})=\Phi\!\left(\frac{z_{a}-z_{b}-\varepsilon}{\sqrt{2}\,\beta}\right) \tag{2}$$

where $z_h$ is the rubric-conditional preference strength, $t_{h,e}$ is the noisy comparative performance, $\beta$ the noise scale, $\varepsilon$ the draw margin and $\Phi$ the standard normal CDF. Abstentions on insufficient evidence produce no preference update at all - a conservative-looking rule that stops sparse comparisons from being over-read. The updated $(\mu_h,\sigma_h)$ describe a comparative assessment, not the probability that the hypothesis is true; judgments of advancement relative to the true predecessor come from the separate Progress role.

Link two, hypothesis weight. First take a conservative order statistic and a tie-aware rank within the same layer:

$$s_{h}=\mu_{h}-\kappa\sigma_{h},\qquad r_{h}=\frac{\sum_{k\in\mathcal{S}}\mathbf{1}[s_{k}<s_{h}]+\tfrac{1}{2}\sum_{k\in\mathcal{S}}\mathbf{1}[s_{k}=s_{h}]}{|\mathcal{S}|} \tag{3}$$

Under a Gaussian marginal approximation $s_h$ is the $\Phi(-\kappa)$ posterior quantile; with $\kappa=2$ roughly 2.3% of the modeled preference mass lies below it, i.e. a lower bound meaning "probably not bad even with bad luck". Two reliability factors then damp sparse or uncertain ratings:

$$\rho_{h}^{\mathrm{cov}}=\frac{n_{h}}{n_{h}+\nu},\qquad \rho_{h}^{\mathrm{unc}}=\frac{1}{1+\sigma_{h}/\sigma_{\mathrm{ref}}} \tag{4}$$

where $n_h$ is the number of effective comparisons. The final bounded outcome transform is

$$q_{h}=v_{h}\min\!\left\{q_{\max},\;q_{\min}+(q_{\max}-q_{\min})\,r_{h}\,\rho_{h}^{\mathrm{cov}}\,\rho_{h}^{\mathrm{unc}}+\lambda_{p}p_{h}\right\} \tag{5}$$

$v_h$ indicates a recorded valid assessment with at least one valid comparison and no marginal duplicate judgment; $p_h$ indicates a substantive-advancement or necessary-repair judgment. The implemented parameters are $\kappa=2$, $\nu=2$, $\sigma_{\mathrm{ref}}=8.3333$, $[q_{\min},q_{\max}]=[0.5,2]$ and $\lambda_p=0.25$. The paper is explicit that Eq. (5) is the authors' bounded transform of the preference posterior, not a quantity inferred by TrueSkill; missing ratings stay unknown, and zero loss is not a negative scientific label - a supported null-result hypothesis remains eligible for training.

Link three, contribution shares and family caps. The contribution graph only admits evidenced edges: direct publication requires a matching publisher, episode and creation-time genealogy; documented production or support can be recovered from the original proposal text, an exact draft-revision promotion record, or supporting file bytes matched by hash and path; Lead coordination requires a contributed work assignment or a publication/promotion record. Merely sharing an episode, appearing in the genealogy graph, or having completed one worker call produces no credit edge. For hypothesis $h$, let $L_h$ be the verified coordination trajectories and $P_h$ the direct-production or documented-support trajectories; under coordination ratio $\gamma$

$$\alpha_{ih}=\begin{cases}\gamma/|L_{h}|, & i\in L_{h},\\ (1-\gamma)/|P_{h}|, & i\in P_{h},\\ 0, & \text{otherwise},\end{cases}\qquad \sum_{i}\alpha_{ih}\leq 1 \tag{6}$$

With the implemented $\gamma=0.2$: one coordinating Lead, one Generation publisher and one documented Evolution trajectory receive 0.2, 0.4 and 0.4 respectively. Shares of a missing contribution pool stay unallocated and are never transferred to a "convenient recipient" - when the Lead publishes alone and authorship is unresolved, it takes only the coordination share. Local genealogy families and global scientific families then each get a budget of $C_{\mathrm{loc}}=C_{\mathrm{glob}}=2$ to suppress credit inflation from duplicate registration and repeated versions of the same family, giving trajectory weights

$$w_{i}=\min\!\left(W_{\max},\sum_{h}b_{ih}\right),\qquad W_{\max}=2 \tag{7}$$

where $b_{ih}$ is the allocation after contribution sharing, duplicate suppression and family budgeting. Proposition 1 in Appendix B.7 proves this scaling satisfies credit conservation before normalization: each hypothesis receives no more than $\widetilde{q}_h$, each global family no more than $C_{\mathrm{glob}}$, and each trajectory weight lands in $[0,W_{\max}]$. The authors are equally candid that $\alpha$ is a fixed responsibility prior - not a Bayesian posterior over actors, and not a counterfactual marginal-contribution estimate - and that Critique votes cannot independently certify the correctness of Critique's own output.

Training objective. Credit controls how strongly one history contributes to training; inside a history, the target set contains only assistant responses and tool-call arguments supported by recorded contributions, while tool observations and known failed actions are kept as context but excluded from the loss:

$$\mathcal{L}(\theta)=-\sum_{i}\frac{\omega_{i}}{|U_{i}|}\sum_{u\in U_{i}}\log\pi_{\theta}(a_{iu}\mid x_{iu}) \tag{8}$$

$U_i$ is the set of eligible assistant token positions, and $\omega_i$ combines contribution credit with domain and shared-memory-group balancing:

$$\omega_{i}=\frac{\eta_{d_{i}}}{|\mathcal{G}_{d_{i}}|}\cdot\frac{w_{i}}{\sum_{k:\,d_{k}=d_{i},\,g_{k}=g_{i}}w_{k}},\qquad \sum_{d}\eta_{d}=1 \tag{9}$$

Coefficients and denominators are frozen with the training release and are not renormalized inside a minibatch - otherwise a long transcript would set its own loss weight by virtue of being long. In practice domain balancing gives 90% to life sciences and 10% to battery research. So the weight of a history is spread over the targets it supports, not over everything it ever said, and a history with no eligible target simply leaves training.

Harness replay: frozen weights, edited workflow

Parallel to the model update runs a loop that never touches weights. Agents start from a mature Codex or Claude Code harness whose session, context, filesystem and tool-execution machinery is the substrate for complex investigations; research coordination, evidence access, review timing and work assignment are the adaptations layered on top. Short diagnostic runs (including runs of newly trained model candidates) first expose where the work breaks: joining agent contribution and decision records with execution and delivery records locates repeated exploration, failed tool calls, missing artifacts and premature convergence. For a history $\mathcal{H}_k$ collected with fixed model $\theta_k$ and harness $\psi_k$:

$$\Psi_{k}^{\mathrm{cand}}=\operatorname{Propose}(\psi_{k};\mathcal{H}_{k}),\qquad \psi_{k}^{+}=\operatorname{Select}_{\mathrm{dev}}\!\left(\{\psi_{k}\}\cup\Psi_{k}^{\mathrm{cand}}\;\middle|\;\theta_{k},B\right) \tag{10}$$

$B$ is the matched resource cap. Quick runs and supported replays handle diagnosis and first screening; matched development-set evaluation handles acceptance, weighing scientific quality, exploration breadth, reliability and cost together; and the incumbent harness is always in the candidate set, so nothing changes unless change wins. Replays use recorded observations and continuations; an unrecorded alternative must actually be executed from a compatible restored state.

Rediscovery: putting accepted updates back on new questions

Let $\theta_k^+$ be the model candidate from credit optimization and $\psi_k^+$ the harness candidate selected by replay; the next round is

$$\begin{aligned}(\theta_{k+1},\psi_{k+1})&=\operatorname{Accept}_{\mathrm{dev}}\!\left(\{\theta_{k},\theta_{k}^{+}\}\times\{\psi_{k},\psi_{k}^{+}\}\right),\\ \mathcal{H}_{k+1}&\sim\operatorname{Research}\!\left(\theta_{k+1},\psi_{k+1};\mathcal{Q}_{k+1}^{\mathrm{fresh}},B\right).\end{aligned} \tag{11}$$

The acceptance procedure evaluates eligible updates separately and jointly, so changing the model, the harness, both, or neither are all legitimate outcomes. The accepted system returns to fresh questions and collects a new history of hypotheses, evidence, assessments and contributions. The paper separates two easily confused notions: a scientific restart changes direction inside one research round, while rediscovery changes the system used for the next round. Versioned records string adjacent rounds together while keeping the development family and the final evaluation family separate.

flowchart TB
    subgraph EP["Episode k - hypothesis development"]
        L["Lead: pick direction, parents, operation"] --> G["Generation: propose hypothesis"]
        L --> E["Evolution: evolve / repair / crossover"]
        G --> C["Blinded Critique: pairwise preference"]
        E --> C
        E --> P["Progress: ancestor-aware judgment"]
        C --> S["Select hypothesis versions"]
        S --> K["Compiler: Harbor task with checks"]
    end
    K --> R["Linked records: evidence, drafts, assessments, contributions, execution"]
    R --> A["Hypothesis credit optimization: TrueSkill posterior, bounded weight, contributor shares"]
    A --> T["theta_k+ : weighted SFT on supported tokens only"]
    R --> H["Harness replay: diagnose failures, propose edits, dev-select with incumbent"]
    H --> W["psi_k+ : accepted workflow edit"]
    T --> D["Accept_dev on development set"]
    W --> D
    D --> N["Round k+1: rediscovery on fresh questions"]
    N --> EP

Figure 1: the recursive structure of DiscoveryRSI (corresponding to Figure 1 and Algorithm 1 of the paper). Within an episode the Lead coordinates Generation and Evolution; candidate versions pass blinded critique and progress judgment, then the Compiler turns them into Harbor tasks. The same batch of linked records feeds both update branches - credit optimization yields $\theta_k^+$ and harness replay yields $\psi_k^+$ - and once accepted on the development set they enter the next rediscovery round.

DFM-Bench: five levels of assessment object

DFM-Bench asks five connected questions: whether a single hypothesis is worth doing and testable (producing an eight-axis profile of importance, novelty, grounding, falsifiability, data resolvability, feasibility, alternatives and actionability, plus a weighted total); what one revision contributed relative to its true predecessor (substantive advancement / necessary repair / duplication / insufficient evidence); whether a development path ordered by true genealogy accumulated useful evidence; whether a whole research round was efficient in exploration breadth, branch synthesis and budget allocation (assessed on the whole graph rather than by averaging path scores); and what independent scientific contributions emerge across related proposals (new distinctions, informative null results, supported findings, uncertainty reduction), distinguishing "ideas proposed" from "contributions supported by completed investigations and independent checks". Coverage against existing benchmarks is compared in the table below: recording intermediate actions does not count as coverage. AgentIdeaBench, DiscoveryBench, BLADE and ScienceAgentBench are explicit or partial on at most one or two columns, OEB is explicit on evidence use and revision and partial on paths and whole rounds, and DFM-Bench is explicit on all six.

Benchmark	Hypothesis value	Contribution maturity	Evidence use	Revision	Development path	Round-level allocation
AgentIdeaBench	✓	✗	◔	✗	✗	✗
DiscoveryBench	◔	✗	◔	✗	✗	✗
BLADE	✗	✗	◔	✗	✗	✗
ScienceAgentBench	✗	✗	◔	✗	✗	✗
OEB	✗	✗	✓	✓	◔	◔
DFM-Bench (this paper)	✓	✓	✓	✓	✓	✓

Table 1: assessment coverage across scientific-discovery benchmarks (✓ explicit assessment, ◔ related or partial, ✗ no explicit assessment). The authors stress this compares explicit assessment constructs, not "the ability of an agent that happens to run on a given benchmark".

Experiments

Setup

The main experiments cover four life-science domains: clinical and population health, therapeutic target prioritization, disease mechanism, and population multi-omics. Inputs include an HCC clinical cohort, MIMIC-IV, eICU, UK Biobank, and molecular studies from GEO/ArrayExpress. A battery domain tests transferability: four sub-domains of electrical diagnostics, health prediction, degradation mechanisms, and usage history and aging, running the same Lead-Generation-Evolution-Critique-Compiler loop with only the domain-specific checks swapped (operating conditions, sensing, physical plausibility, competing explanations). On the training side, DiscoveryRSI-27B receives full-parameter fine-tuning at 256K context: of 15,792 collected research trajectories, 2,397 survive attribution, filtering and tokenization (634 Lead, 861 Generation, 902 Evolution), totaling 291.12M sequence tokens of which 6.18M (2.12%) receive direct supervision; Adam, learning rate $5\times10^{-6}$, global batch 16, BF16, 64 GPUs (TP4 x CP8 x DP2) for 150 steps, one epoch. Automated quality assessment covers 241 research questions with GPT-5.6 Sol as judge; behavior analysis runs six campaigns per model variant with 20 episodes per campaign; the human evaluation has 27 physicians, medical students and biological-science PhD students with biomedical or clinical research backgrounds reviewing 292 anonymized proposals under three senior supervisors.

Figure 3: the research process shared across domains. The four steps of the top row are "available evidence -> research questions -> joint development of hypothesis and investigation -> interpretation as support / weakening / unresolved", with the critique, recording and revision loop in the middle; the bottom two rows give the experimental settings for the four life-science domains and the four battery domains.

RQ1: does learning from research history improve hypothesis quality

Automated hypothesis quality rises from 4.86 for the base model to 5.18 after round one and 5.39 after round two. The round-two model beats the base in all four domains, with gains from 0.28 in disease mechanism to 0.94 in population multi-omics, while the marginal gain drops from 0.32 in round one to 0.21 in round two. More informative is the axis-level decomposition: falsifiability climbs from 4.00 to 6.20, alternatives from 4.44 to 5.98, data resolvability from 3.73 to 4.99 and feasibility from 4.79 to 5.87; meanwhile importance falls from 6.48 to 6.04 and novelty from 4.87 to 4.58.

Assessment axis (weight)	Base R0	Round 1 R1	Round 2 R2
Total Q	4.86	5.18*	5.39*
Importance (20)	6.48	6.21	6.04
Novelty (20)	4.87	4.58	4.58
Grounding (15)	4.24	4.97*	4.50*
Falsifiability (10)	4.00	5.24*	6.20*
Data resolvability (10)	3.73	4.33*	4.99*
Feasibility (10)	4.79	4.21	5.87*
Alternatives (10)	4.44	5.64*	5.98*
Actionability (5)	5.20	5.30	5.47*

Table 2: eight-axis hypothesis quality across self-improvement rounds (1-10 scale; * marks significantly better than R0 at FDR-corrected q<0.05). The improvement is concentrated in "how to test", not in "how important the question is".

Figure 4: automated hypothesis quality rises with rounds. (a) Top row: an example research tree per round (node color is eight-axis weighted quality); middle row: per-round means with 95% confidence intervals and interquartile shading; bottom row: per-domain means. (b) the table of means and axis-level scores. The right way to read these numbers: learning tightened the rigor of investigation design, not the ambition of topic selection.

RQ2: how does research behavior change

The share of evolve and repair operations rises from 77.5% for the base model to 90.0% in round one and 91.7% in round two, while restart drops from 21.7% to 5.8% and 5.0%, most visibly in episodes 11-20 (roughly 32% down to 3%). Among development-path scores, the largest gain is evidence and inference discipline (+1.25), followed by technical and knowledge exploration (+0.34) and scientific continuity and synthesis (+0.24); changes in cumulative advancement, stopping decisions and budget efficiency are all below 0.1, and disciplinary connection actually falls by 0.28. The behavior profile deserves a restrained reading: operation frequencies describe research strategy, and their value depends on the quality of the revisions they produce. The model learned to "deepen one hypothesis" instead of "keep switching topics", but it did not learn to allocate budget better or to make cross-disciplinary connections.

Metric	Base	Round 1	Round 2
evolve + repair share	77.5%	90.0%	91.7%
restart share	21.7%	5.8%	5.0%
L4 evidence and inference discipline, delta (R2 - base)	+1.25
L2 technical and knowledge exploration, delta	+0.34
L5 scientific continuity and synthesis, delta	+0.24
L1 disciplinary connection, delta	-0.28

Table 3: round-to-round changes in research behavior and the seven development-path scores (path weights L1-L7 are 5/30/20/20/10/10/5).

Figure 5: behavior change. (a) share of each operation per round; (b) heatmap of operations across episode positions for the six campaigns in each round; (c) change in the seven path scores from base to round two, where the +1.25 on L4 is the most striking bar.

RQ3: do human reviewers accept these improvements

Within the assessed research trees, 41% of DiscoveryRSI-27B revisions raise the human mean score, 47% leave it unchanged and 12% make it worse; for GPT-5.6 Sol the split is 31%/37%/32% and for GPT-6 Luna 25%/44%/31%. The mean score change per revision is +0.07 (DiscoveryRSI-27B), +0.03 (Sol) and -0.01 (Luna). Absolute ratings paint a mixed picture: DiscoveryRSI-27B scores above Luna on importance (3.10 vs 2.94) and on "claim credible with evidence boundaries" (3.37 vs 3.16), the two tie at 2.31 on "advancement relative to known work", while Luna is higher on data fit, alternative explanations and informativeness - stronger revision outcomes coexisting with residual weaknesses in individual proposals. One MIMIC-IV development path makes this concrete: later versions clarified the time origin and the 48-hour landmark, bound variables to available data fields and stated the acute kidney injury threshold, lifting the mean over six human criteria from 2.83 to 3.50, with data fit and informativeness each gaining two points; yet the same tree still contains fatally flawed branches (two depth-four siblings never state the AKI threshold), which is why assessment must be per proposal rather than averaged per tree.

Figure 6: the expert-supervised evaluation procedure. (a) An anonymized MIMIC-IV proposal presents the scientific question, the planned comparisons and the claim boundaries; (b) reviewers first record an overall judgment and recommendation, then score six anchored criteria and classify design defects (none / fixable / fatal), with optional defect lists and comments. Model identity, execution genealogy and automatic scores are all hidden.

RQ4: do the hypotheses inspire subsequent research

"Inspiration" is defined as a concrete connection between a generated hypothesis and later literature: work published after the cutoff provides an experimental design, control, measurement or testable evidence gap relevant to developing or testing that hypothesis. Across two merged production runs, 74 of 1,176 hypothesis versions qualify, an inspiration rate of 6.3%; the comparison groups are DeepSeek at 5.4% (14/257), GLM at 2.8% (1/36), GPT at 2.5% (59/2,341) and Claude at 2.3% (3/131). By domain DiscoveryRSI reaches 16.9% on therapeutic targets, 7.4% on population multi-omics, 6.8% on disease mechanism and 2.3% on clinical and population health. Formally,

$$R_{\mathrm{insp}}=\frac{\sum_{h}a_{h}}{|\mathcal{H}|} \tag{12}$$

where $a_h$ indicates a positive judgment and $\mathcal{H}$ is all generated versions in the group. The reference corpus is built from 420 backward and forward citation traversals, yielding 17,790 citation edges and 16,440 resolved publications, filtered by each group's recorded cutoff. The authors give an honest attribution analysis of Claude's low score: only 15/131 (11.5%) of Claude proposals pass the retrieval gate versus 355/1,176 (30.2%) for DiscoveryRSI, yet among proposals that do pass retrieval the positive-judgment proportions are similar (20.0% vs 20.8%). Claude proposals have a median length of 41,134 characters while retrieval uses only the first 6,000 and the semantic audit only the first 2,500, so genealogy and revision notes can push the scientific content out of the assessed fragment; a diagnostic that queries by title lifts Claude's retrieved count from 15 to 95. Part of the measured gap therefore comes from representation and retrieval coverage rather than scientific quality alone.

Figure 7: (a) inspiration rates ranked by model group, with DiscoveryRSI-27B leading at 6.3%; (b) the per-domain pattern, where therapeutic target prioritization is the high-hit domain shared by DiscoveryRSI and DeepSeek.

The battery domain and the training ledger

The battery domain applies the five-level DFM-Bench assessment to 200 episodes and 351 generated hypotheses (three AI reviews each): mean hypothesis quality 5.61 over 165 complete scores, 214 of 366 transitions (58.47%) judged substantive advancement, mean full-trajectory score 81.52 over 152 paths, research-round score 71.25, and 20 contribution families identified of which 7 carry diagnostic or research evidence. The training ledger is equally worth reading: Critique (4,988 records), Compiler (2,056) and Progress (513) are kept as context or provenance but receive no direct supervision, because the attribution policy holds that they cannot certify their own correctness.

Role	Collected trajectories	Entering training	Sequence tokens	Supervised tokens
Lead	3,423	634	98,357,399	977,931
Generation	2,502	861	96,414,270	2,437,985
Evolution	2,310	902	96,349,331	2,761,689
Critique	4,988	0	-	0
Compiler	2,056	0	-	0
Progress	513	0	-	0
Total	15,792	2,397	291,121,000	6,177,605

Table 4: trajectory and token ledger for hypothesis-credit training. A 2.12% supervision ratio says what this method really is: inside an enormous research context, learn only the few sentences that evidence supports.

Limitations

Stated by the authors, part one: lopsided gains. Quality improvements concentrate in falsifiability, alternatives, data resolvability and feasibility, while importance and novelty decline rather than rise (6.48 to 6.04, 4.87 to 4.58), disciplinary connection falls by 0.28, and cumulative advancement, stopping decisions and budget efficiency move by less than 0.1. The authors accordingly limit the claim to "learning improves investigation design" and do not claim better topic selection or better resource allocation.

Stated by the authors, part two: the graders are not independent experts. The automated profile is produced by three prompted roles from the same model family (rubric judge, domain scientist, methods reviewer) scoring independently, which measures correlated AI perspectives rather than three independent expert opinions; human calibration (independent expert judgment, arbitration, unseen acceptance cases) is still pending. The human evaluation itself is also only 27 reviewers over 292 proposals, and shared genealogy makes revision observations interdependent, so the paper reports descriptive percentages without significance testing.

Stated by the authors, part three: inspiration rate is retrospective correspondence, not adoption. The 6.3% measures concrete connections between post-cutoff literature and the hypotheses; it does not mean the publishing authors saw, adopted, or were causally influenced by a given hypothesis. Retrieval gates and proposal formatting systematically affect the measured rate, as the Claude case demonstrates. Prospective evaluation would require freezing hypotheses before researchers see them and tracking subsequent research plans.

My own judgment: there is a reproducibility distance between the public code and the paper's numbers. The repository is a 433-line portable harness: the Candidate dataclass carries claim/plan/alternatives/evidence/parents, DiscoveryLoop implements propose-critique-select-retain and writes append-only JSONL, ResearchHarness realizes Lead and Critique through prompts, and ToolRegistry provides the three tools record_evidence/compare/falsify. It faithfully reflects the protocol skeleton of Definition 1 and Section 3.1, but the TrueSkill credit pipeline, the family budgets, the 256K full-parameter fine-tune and the DFM-Bench scorers are all absent (DFM-Bench being a separate repository), so the paper's headline numbers cannot be reproduced from the public repository alone. In addition, Candidate does not expose the decision criterion $d$ as its own field - it is carried by the critique prompt and rubric text instead - leaving a small field-level gap between the public harness and Definition 1.

Takeaways and Outlook

The contribution of DiscoveryRSI compresses into one sentence: treat the research process, not the research result, as the training data. Definition 1 makes a hypothesis a revisable four-tuple object; research trees and linked records let assessment be traced back to contributions; hypothesis credit optimization turns that tracing into a bounded, conserving training signal that supervises only supported tokens; harness replay and rediscovery bring the workflow and the system itself into the learning loop; and DFM-Bench measures "a good proposal" separately from "a good process". The experimental evidence is internally consistent: automated quality rises 0.53 points over two rounds with the gains landing on testability axes, behavior shifts from topic-switching to deepening, human reviewers see a higher net gain per revision, and the literature correspondence rate is more than twice the comparison groups. The authors state the next steps clearly: execute and refine selected proposals with a solver such as RSI-Anything, add wet-lab validation, turn supporting, refuting and uncertain results alike into better research decisions, and extend hypothesis discovery and DFM-Bench to more scientific domains. For teams building research agents, the genuinely reusable part of this paper is not any single score but the record semantics: parent links, contribution edges, execution events and assessment versions. Miss one of them and recursive learning has nothing to stand on.