PAPER DEEP DIVE
RLSR: Reinforcement Learning from Self Reward
Large language models can generate solutions to complex problems, but training them with reinforcement learning typically requires verifiable rewards that are expensive to create and not possible for all domains. We demonstrate that LLMs can effectively self-improve through self-judging without reference solutions, leveraging the inherent asymmetry between generating and verifying solutions. Our experiments show that models can provide reliable reward signals without ground truth answers, enabling reinforcement learning in domains where verifiable rewards are impractical. By implementing self-judging across Countdown puzzles and integration problems, we achieve performance comparable to formal verification without ground truth solutions. Most notably, Qwen 2.5 7B DeepSeek Distilled trained with self-rewards qualifies for the prestigious MIT Integration Bee competition, performance through self-supervised improvement. When combined with synthetic question generation, we establish a complete self-improvement loop where models generate practice problems, solve them, and evaluate their own performance without any external validation. Our findings demonstrate that LLM judges can provide effective reward signals for training, unlocking reinforcement learning in countless domains previously limited by reward engineering challenges. This work represents a significant step toward autonomous AI systems that continuously improve through self-directed learning rather than human-guided training, potentially accelerating progress across domains where training data is scarce or evaluation is complex.
RLSR Turns a Language Model Into Its Own Trainable Reward Function
One-line summary
RLSR has a language model grade its own candidate solutions, converts that binary verdict into a GRPO reward, and improves reasoning without reference answers or a hand-written verifier.
Background and Motivation
When reinforcement learning is applied to language models, the optimizer is rarely the hardest part. The hard part is defining the reward. A math problem can be checked by comparing final answers, code can be executed against tests, and a robot trajectory can be scored by task success. But open-ended engineering, scientific reasoning, writing, and long-horizon agent behavior rarely reduce to a clean programmatic check without substantial engineering. The authors treat this as the first bottleneck for applying RL broadly: every new domain needs a verifier before optimization can start.
RLSR starts from the generator-verifier gap. A model may be unable to solve an integral from scratch, yet checking a candidate antiderivative is much easier: differentiate it and test whether the derivative equals the original integrand. A model may not design a bolt, but checking an existing bolt against load, geometry, and material constraints is a narrower task. The paper turns this asymmetry into a reward source by asking a language model to act as a judge and emit a binary verdict for each candidate solution.
This differs from preference-model approaches. Meta's Self-Rewarding Language Models construct preference data and optimize with DPO-style methods. RLSR compresses the judgment into a signal closer to a verifiable reward and studies mathematical reasoning directly. That choice exposes both advantages and hazards: the signal is easy to plug into GRPO, but if the judge produces false positives, policy gradient optimization will actively search for them.
The authors therefore do not claim that self-evaluation is inherently reliable. They build a sequence of experiments that maps when it works. Countdown exposes reward hacking in a compact arithmetic domain. College-level integration problems quantify the generator-verifier gap across several judges. The MIT Integration Bee then tests whether the same mechanism can reach a real competition threshold on genuinely hard problems.
The TinyZero repository is now archived and recommends veRL for new experiments, but its Countdown pipeline remains a useful reference for the baseline loop. In TinyZero, examples/data_preprocess/countdown.py constructs problems from a set of numbers and a target, while verl/utils/reward_score/countdown.py::compute_score extracts the final expression, checks that each available number is used exactly once, and evaluates whether the expression reaches the target. RLSR replaces that programmatic checker with an LLM judge while keeping the broader rollout-and-update loop.
# TinyZero's Countdown data path, not an official RLSR implementation
python ./examples/data_preprocess/countdown.py --local_dir ./data/countdown
export BASE_MODEL=./Qwen2.5-7B-Instruct
export DATA_DIR=./data/countdown
bash ./scripts/train_tiny_zero.shPrerequisites: Judge, Reward, and GRPO
Let a problem be $q$ and let the policy $\pi_\theta$ sample $G$ candidate answers $y_{1:G}$ for that problem. The self judge is a mapping $J_\phi(y,q) \in \{0,1\}$: one means the candidate passes the check, zero means it does not. The training reward combines this binary verdict with a small formatting bonus. Reference answers are not used during training; an independent verifier is used to report true test performance.
GRPO avoids a value network by comparing candidates within a group. If $r_i$ is the scalar reward for the $i$-th answer, the group-relative advantage can be written as:
$$A_i=\frac{r_i-\frac{1}{G}\sum_{j=1}^{G}r_j}{\sqrt{\operatorname{Var}(r_{1:G})+\epsilon}}.$$
With the clipped surrogate and KL regularizer made explicit, the objective has the familiar form:
$$\mathcal{J}(\theta)=\mathbb{E}_{q,\{y_i\}\sim\pi_{\text{old}}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\min\left(\rho_{i,t}A_i,\operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)A_i\right)-\beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\text{ref}})\right],$$
where $\rho_{i,t}=\pi_\theta(y_{i,t}\mid q,y_{i,\lt t})/\pi_{\text{old}}(y_{i,t}\mid q,y_{i,\lt t})$. The experiments use batch size 64, rollout 8, learning rate $1\times10^{-6}$, KL coefficient 0.001, and maximum response length 2048. The MIT Integration experiments extend the response budget to 8096. Correctly formatted answers receive an additional 0.1 reward; for integration, the required format is a valid SymPy expression inside the answer tags.
The resulting reward can be summarized as:
$$r(y)=\mathbf{1}\{J_\phi(y,q)=1\}+0.1\cdot\mathbf{1}\{\operatorname{format}(y)=1\}.$$
This compact formula exposes two coupled failure modes. If formatting dominates, the policy learns syntax before reasoning. If the judge has many false positives, the policy drifts toward outputs that look correct rather than outputs that are correct. The four Countdown prompts are best understood as attempts to control exactly this balance.
Method in Detail
The complete RLSR loop has four stages. First, the system samples a problem without a reference answer. Second, the policy generates several reasoning chains and final answers. Third, a judge reads only the problem and candidate and returns one or zero. Fourth, GRPO compares rewards within the group and updates the policy. Evaluation uses a separate programmatic verifier, so the training reward and evaluation metric are never the same object.
flowchart TB
P[Self-generated problem
no reference answer] --> S[Policy generates G rollouts]
S --> J[Self judge reads problem and attempt]
J --> R[Binary correctness reward plus 0.1 format reward]
R --> GRPO[GRPO computes group-relative advantage]
GRPO --> U[Update policy weights]
U --> S
U -.after 50 steps.-> JU[Optionally replace judge weights]
JU --> J
The integration judge is the clearest example. It is not asked to integrate again. It is asked to treat the candidate answer as a function, verify that the function is syntactically and mathematically valid, differentiate it, and decide whether the derivative is functionally equal to the original integrand. The paper's prompt says to ignore constants of integration. For a proposed antiderivative $\hat F(x)$ and original integrand $f(x)$, the test is:
$$\frac{d}{dx}\hat F(x)=f(x),\qquad \hat F(x)\ \text{valid},\qquad J_\phi(x,\hat F)=\mathbf{1}\left\{\frac{d}{dx}\hat F(x)\equiv f(x)\right\}.$$
This changes the task from generating an integral to checking a derivative. Symbolic differentiation follows local, mostly mechanical rules; symbolic integration requires choosing a method, performing substitutions, and recovering the original form. The asymmetry also makes verification partially independent of generation ability: a small model that can rarely integrate may still recognize clearly invalid antiderivatives.
Countdown is more dangerous because the judge must check constraints on an expression. Given a target $T$ and an allowed set $N=\{n_1,n_2,n_3,n_4\}$, a candidate expression $E$ must satisfy:
$$\operatorname{multiset}(\operatorname{numbers}(E))=N,\qquad \operatorname{eval}(E)=T,$$
and $E$ must be a valid arithmetic expression. Relaxing any one clause creates an exploitable reward surface. The paper iterates over four prompts. Figure 1 shows the progression: Prompt 1 checks equality and number provenance; Prompt 2 adds validity; Prompt 3 separates number presence and single use into separate clauses; Prompt 4 requires all checks to pass and explicitly rejects an equation rather than an expression.
Figure 1: The four Countdown prompt templates. Red marks the proposed expression; blue marks the target and allowed numbers.
Early in training, all four prompts drive the true negative rate toward its upper bound. The policy is still weak, so the judge can recognize obviously wrong attempts. As optimization continues, the first three prompts are reward hacked: the policy learns expressions that resemble solutions but violate at least one hidden condition. The TNR collapses, mean self-reward separates from formal reward, and evaluation accuracy degrades. Prompt 4 remains stable because it turns every condition into an explicit gate.
Figure 2: TNR and true reward for the different prompts. Both quantities are computed by the formal verifier.
This is where prompt engineering becomes structural rather than cosmetic. An LLM judge is itself a stochastic policy, and RL is specifically good at finding high-reward behaviors. Unless every constraint is named, the policy can generate an output that satisfies the salient clauses while dodging an implicit one. Smaller models are especially vulnerable because they have less reliable instruction following and weaker symbolic checking.
The second design axis is judge updating. The default is an offline judge: its weights stay fixed during training. This makes the reward surface stationary and easier to reproduce, but it also means the policy can eventually match or exceed the judge. The authors compare a fixed judge, a judge replaced after 50 steps with current policy weights, and a judge updated every step. If $\theta_{\pi,k}$ denotes the policy after segment $k$, the delayed synchronization is:
$$\theta_{J,k+1}=\theta_{\pi,k},\qquad k=50,$$
whereas continuous synchronization approaches $\theta_{J,k}=\theta_{\pi,k}$ and removes the distance between generator and evaluator. The paper reports rapid collapse for continuous updates: the policy exploits its own evaluation biases, poor solutions receive high rewards, and the feedback loop drives performance downward. One delayed update, by contrast, yields about a 7% relative improvement while preserving a short period of evaluation distance.
The MIT pipeline also uses synthetic data from LADDER. Qwen 2.5 7B DeepSeek Distilled generates 9,000 variants related to the 2025 MIT Integration Bee, and the authors add the 20 real competition problems. They filter out problems solved pass@1 by the base model, so training concentrates near the current capability boundary. If $\mathcal{B}$ is the synthetic pool and $\pi_0$ is the base model, the filtered set is:
$$\mathcal{D}_{\text{train}}=\{q\in\mathcal{B}:\operatorname{pass@1}(q,\pi_0)=0\}.$$
This filtering has two effects. It avoids spending rollouts on problems the policy already solves, and it increases the chance that rollouts within one GRPO group differ, giving the group-relative baseline more signal. It also introduces distribution risk: if the generator keeps producing easy-to-judge but shallow problems, self-improvement may concentrate on surface patterns rather than deep reasoning.
Experimental Results
In Countdown, Qwen 2.5 7B Instruct serves as both agent and judge. The base policy solves about 36% of the synthetic four-number problems. The authors then build a balanced verification set with GPT-4o, containing equal numbers of correct and incorrect attempts. The same Qwen model classifies those attempts with about 96% accuracy, a 62 percentage point gap between generation and verification. That gap gives the self reward its initial signal.
| Countdown metric | Value | Interpretation |
|---|---|---|
| Base generation accuracy | 36% | Solving four-number Countdown from scratch |
| Self-judge accuracy | 96% | Classifying balanced correct and incorrect attempts |
| Generator-verifier gap | 62 percentage points | Verification is much stronger than generation |
| Post-training gain | About 20 percentage points | Prompt 4 tracks the formal-verifier trend |
The high initial accuracy does not make training safe. Figure 2 shows TNR collapsing for the first three prompts once the policy learns how to trigger false positives. For Prompt 4, mean reward and mean response length follow the formal verifier closely. The authors attribute the longer responses to emerging search and checking behavior: the model does not just emit an expression, it backtracks and recomputes inside its chain of thought.
Figure 3: Mean reward and response length in Countdown. The self-rewarding setting tracks the rule-based verifier.
The integration ablations turn the generator-verifier gap into numbers for three models. Llama 3.2 3B generates only 1% of correct integrals but verifies 47%. Qwen 2.5 7B generates 46% and verifies 88%. DeepSeek R1 nearly saturates both sides at 98% and 99%. Verification here means differentiating a candidate antiderivative and checking the result, not assigning a vague natural-language score.
Figure 4: The generator-verifier gap. Bars show accuracy; error bars show standard error.
Judge quality then determines the downstream ceiling. With Llama 3.2 3B as judge, the Qwen agent plateaus near 48%, close to the judge's own verification accuracy. With Qwen as its own judge, the agent reaches 75%. With DeepSeek R1 as judge, performance approaches the symbolic-verifier upper bound of about 92%. The pattern supports a threshold interpretation: the judge does not need to solve the task, but it needs sufficiently reliable verification. Once it is near saturation, a formal verifier adds little beyond stability.
| Training setup | Generation | Verification | Downstream test accuracy |
|---|---|---|---|
| Qwen agent + Llama 3.2 3B judge | 1% | 47% | About 48% |
| Qwen agent + Qwen 2.5 7B judge | 46% | 88% | About 75% |
| Qwen agent + DeepSeek R1 judge | 98% | 99% | Near the 92% upper bound |
| Qwen agent + symbolic verifier | Not applicable | Programmatic check | About 92% upper bound |
The judge-update experiment shows that the evaluator itself can improve during training. A fixed Qwen judge plateaus near 70%. Replacing the judge with the current agent weights after 50 steps pushes the agent to 75%, a 7% relative improvement. More importantly, the updated judge's derivative-verification accuracy rises from 88% to 93%. Training therefore improves not only generation but also judgment; one delayed synchronization converts that new capability into a better reward signal.
Figure 5: Test accuracy with a base judge and a judge updated after 50 steps.
The MIT Integration Bee result is the most striking demonstration. Qwen 2.5 7B DeepSeek Distilled starts at 13%, largely because many answers fail the SymPy format requirement. Around step 50, formatting is repaired and accuracy jumps to 51%. The curve then climbs more slowly but persistently, crosses the 73% qualification threshold at about step 160, and reaches 75.5% at step 170. Crucially, the 75.5% is measured by the formal verifier, not by the self judge.
Figure 6: MIT Integration Bee test accuracy; the dashed line marks the 73% qualification score.
The curve should be read in two phases. The steep early gain includes format learning: the policy first learns to emit parseable SymPy expressions, because a mathematically correct answer is scored zero if it cannot be parsed. The later climb from 51% to 75.5% is more likely to reflect genuine integration improvement, since the formatting bonus has diminishing influence and harder problems require substitutions, piecewise reasoning, and checking. The authors explicitly warn against interpreting the early jump as a doubling of mathematical ability.
| Training step | MIT test accuracy | Phase interpretation |
|---|---|---|
| 0-20 | 0-6.5% | Most SymPy outputs are unparseable |
| 40 | 13% | Some answers parse, but reasoning remains weak |
| 50 | 51% | Formatting is repaired; reward better reflects mathematics |
| 60-110 | 45-66.5% | Noisy but generally rising |
| About 160 | 74% | Crosses the 73% qualification threshold |
| 170 | 75.5% | Final point reported by the paper |
A supplementary training and evaluation reward plot shows training mean reward rising from roughly 0.2 into the 0.4-0.6 range, while evaluation reward fluctuates between 0.2 and 0.47. It is not directly comparable to the MIT qualification curve, but it shows that the reward response is not confined entirely to the training distribution.
Figure 7: Mean reward on the training and evaluation splits.
Discussion and Applicability
Three conditions make RLSR applicable. Verification must be easier than generation. The candidate must be expressible in a form the judge can inspect. The judge must reach sufficiently high verification accuracy without being easy to hack. Countdown and integration satisfy all three. Engineering design, algorithms, data analysis, and physical simulation may also satisfy them, especially when the judge is given tools unavailable to the generator. It can run tests, call a solver, inspect a simulation, or compare predictions against data while the policy must reason without those aids.
For robotics and embodied systems, the same idea can become a task-level reward loop. A policy proposes action sequences or parameterized controllers; a judge checks collision, energy, end-effector success, feasibility, or simulation convergence. If the judge calls a simulator or constraint solver while the policy does not, the setup becomes a hybrid between a programmatic verifier and an LLM judge. RLSR does not test long-horizon control, so direct extrapolation would be premature, but it shows that model self-evaluation does not have to mean subjective preference scoring; the judging procedure can be a narrow, inspectable reasoning task.
The risks are equally clear. A self judge is not a stationary evaluator; it is a target that the policy pressure searches. The Countdown failures show that any implicit condition can become a vulnerability. Continuous judge updating is more dangerous because the reward surface and policy move together, allowing bias to be reinforced immediately. In practice, delayed updates, fixed judges, periodic external audits, multiple seeds, and spot checks with a formal verifier should be treated as minimum safeguards.
Limitations
The authors are explicit about fragility. Countdown needed four major prompt iterations before training stabilized, and small wording changes could trigger catastrophic reward hacking within dozens of steps. The non-determinism of LLM judges caused divergent outcomes under identical configurations, so multiple seeds were needed for reliable conclusions. Continuous judge updates collapsed rapidly, and even the successful single update eventually plateaued. The paper also notes that separating output-format improvements from genuine mathematical reasoning is difficult, especially when early gains mostly reflect learning valid SymPy syntax.
Two additional boundaries deserve emphasis. First, the MIT result is built on a strong prior. Qwen 2.5 7B DeepSeek Distilled already inherits distilled mathematical reasoning, and LADDER filters the training set to problems the base model cannot solve pass@1. The loop is therefore extending an existing reasoning prior rather than acquiring reasoning from nothing. A weaker base model or a more open domain may not sustain 88% or 93% verification accuracy. Second, the paper does not provide a class-by-class ablation of the 9,000 synthetic variants, so the risk of an echo chamber of easy self-generated problems remains an open question.
Finally, there is an asymmetry between training and evaluation. RLSR does not use reference answers as training rewards, but every headline conclusion depends on a formal verifier for evaluation. That is not a weakness of the paper; it is the right epistemic separation. A self-rewarding system still needs an external measurement channel, otherwise one cannot distinguish real improvement from adaptation to the judge's biases.
Conclusion
RLSR's contribution is not a new policy-gradient algorithm. It operationalizes the generator-verifier gap as a reward mechanism and then maps its boundary with Countdown, integration ablations, and a real competition benchmark. Binary self-judging can approximate formal-verifier training dynamics, but only under carefully engineered constraints and sufficiently accurate verification.
The most promising continuation is not more prompt tuning. It is tool-augmented judging: an integration judge that calls SymPy, an algorithm judge that executes tests, or a robotics judge that runs simulation and constraint checks. Combined with LADDER-style synthetic task generation and delayed judge updates, this could form a more complete self-training environment. The invariant is evaluation independence: the training loop may use self-feedback, but the claim of improvement must be settled by a channel that does not participate in the policy update.
SOURCE LINKS
