Post-Training Large Language Models via Reinforcement Learning from Self-Feedback

Authors: Carel van Niekerk, Renato Vukovic, Benjamin Ruppik, Hsien-chin Lin, Shutong Feng, and Milica Gašić. Paper: https://arxiv.org/abs/2507.21931. Official implementation: https://github.com/carelvniekerk/ConfidentLLM, released under Apache 2.0. This analysis follows the second arXiv version revised on 15 September 2026.

Executive summary

The paper turns a model's own answer-token confidence into training preferences. A frozen language model generates several chain-of-thought traces, the final answer span of each trace is scored by the gap between the top and second-best token probabilities, and the resulting ranking is used for PPO or DPO post-training. The method reduces the need for human preference labels and removes multi-path search from deployment. Its strongest evidence concerns calibration and inference cost, not uniform gains in exact accuracy.

Why calibration needs its own training objective

A language model can write a polished arithmetic solution and still return the wrong number. Accuracy records whether the final answer is correct; calibration asks whether the probability attached to that answer means what it appears to mean. A model claiming ninety percent confidence should be correct in roughly ninety percent of comparable cases. Human-feedback alignment may make the wording more agreeable while leaving this probability poorly matched to correctness.

The paper argues that ordinary alignment optimizes what people prefer to read. It does not necessarily teach the model how uncertainty should behave when a reasoning chain becomes unreliable. This creates a practical deployment problem: downstream systems can observe a confident sequence of tokens, but they cannot infer whether the model has solved the problem or merely found a familiar-looking wrong answer.

Chain-of-thought decoding offers one partial answer. Instead of producing a single response, the model branches over several first tokens and continues each candidate greedily. Earlier work observed that traces containing coherent intermediate steps tend to put more probability on their final answers. That correlation suggests that internal confidence can act as a signal of reasoning quality, not only as a stylistic feature.

Inference-time search, however, is expensive. A ten-way search can require approximately ten generations for one question. The authors therefore ask whether those useful confident traces can be absorbed into the model during training. If so, deployment could return to a single greedy pass while retaining part of the benefit previously obtained through repeated decoding.

Language generation as a decision process

The method treats a causal language model as a policy. The initial state is the question, an action is the next token, and the state grows as generated text is appended. If the current state is $s_t$ and the chosen token is $a_t$, the transition is written as:

Here $\odot$ denotes sequence concatenation in the paper's notation. The formulation matches ordinary autoregressive generation and makes standard reinforcement-learning objects such as trajectories, rewards, values, and advantages applicable to text.

The paper distinguishes chain-of-thought thinking from chain-of-thought decoding. Prompting asks the model to explain its steps in one response. Decoding changes the search process: it keeps multiple first-token candidates and continues each one separately. The distinction matters because RLSF does not require a handcrafted instruction to reason step by step. It uses the model's existing decoding alternatives to expose multiple possible reasoning paths.

Confidence is calculated at answer-span level. For every token in the final answer, the implementation subtracts the probability of the second-most-likely vocabulary item from the probability of the generated item. These differences are averaged. A generated token at 0.90 with a runner-up at 0.05 is treated as much more decisive than one at 0.40 with a runner-up at 0.35.

How RLSF constructs its own supervision

Generate diverse starts, then continue deterministically

For a prompt $q$, the frozen model keeps the $K$ most probable first tokens. Each token seeds a separate response:

Each path is then completed with greedy decoding. The experiments use $K=10$, citing prior evidence that performance tends to plateau at this number. No policy gradient is applied during this generation step. The frozen model is being used as a sampler of its own latent reasoning tendencies.

Figure 1: The RLSF pipeline from the paper. Multiple reasoning traces are generated, ranked by final-answer confidence, and converted into preference data.

This generation design is neither fully deterministic nor fully random. The first-token branch introduces variety, while greedy continuation reduces variance after that branch point. It is an efficient way to collect plausible responses, but the candidate set is still constrained by the frozen model and by the top ten first tokens. The procedure cannot discover a useful reasoning route whose opening token lies outside that set.

Identify the answer span before scoring it

A trace may end with a sentence, a number, or a mixture of explanation and answer. To isolate the relevant tokens, the authors append the phrase "So the answer is" and ask the model to generate a short answer. A string-matching step then locates this generated answer inside the original response.

The implementation removes special and stop tokens, searches for the largest overlap between the generated answer and the response, and assigns confidence zero when no overlap exists. This defensive choice avoids treating an extraction failure as strong evidence. It also makes answer extraction an explicit source of engineering risk: a valid reasoning trace with an unusual answer format can be discarded before the training signal is formed.

Measure confidence disparity over the answer tokens

For an answer span of length $M$ beginning at position $m$, the score is:

$s_{m+i}$ contains the prompt and the prefix generated before the token being scored. The first term is the probability of the most likely token, while the second is the probability of the next-best alternative. Unlike raw maximum probability, this difference captures how isolated the leading token is from its strongest competitor.

The code mirrors the equation. ProbabilityDisparity in confidentllm/confidence_metrics/probability.py:58-81 extracts the probability assigned to the generated token, obtains the second-largest probability with torch.topk, and subtracts the latter.

next_token_probs = super().__call__(next_token_ids, scores)
second_highest_probs = torch.topk(scores, k=2, dim=-1)
disparity = next_token_probs - second_highest_probs.values[:, :, 1]

The expression is simple, but its meaning is important. RLSF is not training against a calibrated probability of correctness. It is using the model's local token ranking as a proxy for the coherence of the path that produced the answer. The paper's empirical claim is that this proxy is useful enough to create preferences; it is not a theorem that the proxy is always correct.

Turn confidence rankings into preferred and rejected pairs

Once ten candidates have scores, higher-confidence responses become preferred and lower-confidence responses become rejected. The dataset builder creates pairs from each question. It also stores an exponential confidence margin. The relevant logic appears in confidentllm/data/datasets/cot_preference_data.py:163-204.

if response_1["confidence"] <= response_2["confidence"]:
    continue
ranked_data["preferred_response"].append(preferred_response)
ranked_data["rejected_response"].append(rejected_response)
ranked_data["margin"].append(exp(confidence_1 - confidence_2))

No gold answer enters this ranking step. That is the source of the method's scalability and also its central weakness. If the base model consistently gives its highest confidence to a mistaken pattern, RLSF will construct a clean preference dataset around that mistake. The procedure optimizes internal agreement, not correspondence with an external world.

PPO route: learn a Bradley-Terry reward model

RLSF with PPO first trains a scalar reward model on the synthetic pairs. When a generated response reaches its final token, the reward model supplies the terminal reward:

The reward model is trained under the Bradley-Terry assumption. For prompt $q$, preferred trace $h^1$, and rejected trace $h^2$, its loss is:

$\sigma$ is the logistic sigmoid and $\phi$ represents reward-model parameters. The loss raises the margin between the preferred and rejected complete sequences. Only the final state is scored, which matches the idea that confidence is attached to the complete answer.

The policy objective is the expected discounted return:

Its policy gradient uses advantage values:

The advantage estimates whether a token action produced a better outcome than expected from that state. The paper uses generalized advantage estimation:

PPO then clips the policy ratio around one and adds a KL penalty to limit movement from the reference policy. The implementation loads a policy, a frozen reference, a reward model, and a value model. They must share a tokenizer, and the reward and reference components are set to evaluation mode with gradients disabled. These details appear in confidentllm/scripts/run_reinforcement_learning.py:72-103.

This architecture shows why removing human labels does not make RLSF lightweight. It removes annotation, but the PPO route still carries the memory cost of several language models and the instability of online policy optimization.

DPO route: optimize preferences without a separate reward model

The same preference pairs can update the policy directly with DPO:

$\beta$ controls how far the trained policy may move from the reference. DPO removes the separate reward model and online sampling loop. It is operationally simpler, but the paper's experiments do not make it the default winner. On several comparisons, DPO either improves calibration at a painful accuracy cost or worsens both relative to PPO.

Return to one-pass inference

flowchart TD
    A["Prompt q"] --> B["Frozen model: top K first tokens"]
    B --> C["K greedy continuations"]
    C --> D["Append So the answer is"]
    D --> E["Extract answer spans"]
    E --> F["Average top1 minus top2 confidence"]
    F --> G["Rank traces by confidence"]
    G --> H["Synthetic preferred and rejected pairs"]
    H --> I["Bradley-Terry reward model"]
    I --> J["PPO update"]
    H --> K["DPO update"]
    J --> L["Greedy inference at 1x cost"]
    K --> L

The diagram captures the central economic argument. Ten-way candidate generation appears only during training-data construction. After PPO or DPO, the model uses ordinary greedy decoding. RLSF therefore trades one-time training computation for lower recurring inference latency, while accepting the risk that the confidence source is internal and potentially wrong.

Results: calibration gains are real, but not uniform

The experiments cover MultiArith and GSM8K for arithmetic, CommonsenseQA and ARC Easy for multiple choice, and the mathematical subset of RewardBench for reward-model evaluation. Baselines include greedy decoding, chain-of-thought prompting, ten-way chain-of-thought decoding, URM-based RLHF, RLSF-DPO, and RLSF-PPO. Exact-match accuracy and expected calibration error are reported separately.

Qwen 2.5 7B is the clearest poorly calibrated case. Its GSM8K greedy accuracy is 53.67 with an ECE of 42.86. RLSF-PPO reaches 58.62 accuracy and lowers ECE to 41.92. On MultiArith, accuracy moves from 79.44 to 78.06 while ECE improves from 20.75 to 18.35. This pattern supports the paper's calibration thesis, but it directly contradicts any blanket claim that accuracy is preserved in every setting.

DPO is especially revealing. Its GSM8K ECE of 38.15 is lower than PPO's 41.92, but its accuracy falls to 50.73. Calibration and correctness cannot be collapsed into a single ranking. A model can become less overconfident while also losing useful correct answers. The appropriate choice therefore depends on whether a deployment values abstention-like uncertainty or maximum exact accuracy.

Gemma 2B begins from a much stronger operating point. Ten-way CoT decoding reaches 99.01 on MultiArith and 89.18 on GSM8K. RLSF-PPO reaches 98.83 and 88.14. It also improves ECE over greedy decoding, but does not match the best inference-time search. The real advantage is that RLSF obtains a close operating point with one greedy pass instead of keeping ten candidates alive.

Multiple-choice experiments follow the same conditional pattern. Phi-2 is poorly calibrated on CommonsenseQA and improves substantially under RLSF-PPO, moving from 54.46 to 61.13 accuracy and from 25.12 to 19.64 ECE. Gemma 2 on ARC Easy is already strong; RLSF-PPO moves accuracy from 96.96 to 97.04 and ECE from 16.12 to 5.12. It does not beat the 3.03 ECE of ten-way decoding, but it avoids the tenfold deployment cost.

The authors also test whether self-generated preferences produce useful reward models. On the RewardBench math subset, RLSF reaches 76.13 with Qwen and 81.43 with Gemma. URM and QRM reach about 97, but they use human preferences. AfD ranges from 73.12 to 89.29 while using demonstration answers. RLSF sees prompts only in this comparison, so a sixteen-to-twenty-one point deficit is a meaningful limitation rather than a footnote.

What the figures reveal

Figure 2 visualizes token-level rewards from the learned model. Correct numerical expressions often receive high reward, but the model can also reward a plausible intermediate step that later supports a wrong answer. The color map is therefore evidence of learned preference structure, not a formal proof checker. A high-reward token sequence may still contain invalid arithmetic or post-hoc rationalization.

Figure 2: Token-level rewards from the paper. Color indicates reward-model preference, not verified logical correctness.

Figure 3 is a more focused calibration example. Both the before and after models choose option A when the correct answer is E. Before RLSF, the model assigns A a probability of 0.993 and E only 0.004. After RLSF, A falls to 0.712 and E rises to 0.138. The prediction remains wrong, but the probability distribution becomes less dangerously certain and gives the correct answer a larger share.

Figure 3: A render of the paper's inline vector figure. The answer remains incorrect, but RLSF lowers confidence in the wrong choice and raises probability for the correct choice.

Limitations and independent assessment

The authors state that RLSF handles single-step reasoning better than long-horizon interaction. This is a serious boundary for embodied systems. A robot may make an apparently reasonable decision, receive delayed evidence of failure, and need to revise a plan over many actions. Final-answer confidence does not identify which intermediate observation or action caused the failure.

The method also assumes that useful latent reasoning paths already exist. For a severely undertrained model, the top ten branches may all be incoherent, and the confidence gap may mostly measure familiarity with vocabulary rather than quality. The experiments stop around nine-billion-parameter models. Scaling to frontier models and substantially harder reasoning domains remains untested.

Most importantly, self-feedback does not create external knowledge. If the pretrained model lacks a fact, RLSF can reinforce a confidently wrong internal pattern. Figure 3 makes the distinction concrete: confidence becomes less extreme, but the answer is still wrong. The method should be understood as an internal-consistency and calibration stage, not as a replacement for retrieval, tool use, world-model verification, or safety constraints.

There is also a reproducibility ambiguity. The model section names Gemma 2 2B, while Table 2 and the public README refer to a Gemma 9B configuration for some question-answering and reward-model work. The paper may be reporting different configurations in different experiments, but it should identify the exact model behind each table more explicitly. Without that mapping, readers cannot fully separate method effects from model-size effects.

Finally, the abstract and conclusion imply that calibration improves without harming accuracy. The detailed tables are more nuanced. RLSF-PPO lowers Qwen MultiArith accuracy from 79.44 to 78.06, and its Gemma GSM8K result of 88.14 remains below the 89.18 obtained by ten-way decoding. A defensible summary is that RLSF offers substantial benefits for poorly calibrated bases and a useful cost trade-off for strong bases, not an unconditional accuracy guarantee.

Conclusion

The central idea is economical: a model already exposes alternatives during decoding, and those alternatives can be ranked without asking a human to label them. RLSF converts expensive multi-path reasoning at inference into synthetic preferences at training time, then distills the preferred behavior through PPO or DPO. It is most persuasive when the base model is poorly calibrated and a single-pass model is needed after training.

The unresolved question is whether confidence is measuring reasoning quality or merely internal familiarity. Combining self-feedback with external execution, retrieval, or formal verification would address that question more directly than another round of self-training. For robotics and other safety-sensitive systems, such external grounding should remain a separate layer rather than being inferred from a lower answer-token probability alone.

Self-feedback can make uncertainty more useful, but it cannot turn internal confidence into external evidence.