#GRPO (10)

unsloth: the 2x-faster, 70%-less-VRAM fine-tuning library, now a desktop runtime that wires local weights into Claude Code and Codex

Self-described as the first desktop app to run and train models (Windows/macOS/Linux; multi-GPU across NVIDIA, AMD, Intel, CPU and Vulkan). Fine-tuning is claimed at 2x faster with 70% less VRAM, the post-training menu covers LoRA, QLoRA, full fine-tuning, pretraining, GRPO, DPO and FP8, and export reaches GGUF, NVFP4 and FP8. Unsloth Start connects Claude Code or Codex to local weights in one command; the model list includes Qwen3.8, GLM-5.3-Flash, Kimi K3, DeepSeek-V4, MiniMax-H3 and Gemma 4, with private web search, deep research, auto-compaction and RAG built in.

Fine-tuningQLoRA

Python77k7.1k385

A curriculum for post-training agents

Hugging Face maps agent post-training from SFT and distillation through GRPO, stateful environments and asynchronous production harnesses.

BLOG

Post-trainingSFT

Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning

We explore a method for improving the performance of large language models through self-reflection and reinforcement learning. By incentivizing the model to generate better self-reflections when it answers incorrectly, we demonstrate that a model's ability to solve complex, verifiable tasks can be enhanced even when generating synthetic data is infeasible and only binary feedback is available. Our framework operates in two stages: first, upon failing a given task, the model generates a self-reflective commentary analyzing its previous attempt; second, the model is given another attempt at the task with the self-reflection in context. If the subsequent attempt succeeds, the tokens generated during the self-reflection phase are rewarded. Our experimental results show substantial performance gains across a variety of model architectures, as high as 34.7% improvement at math equation writing and 18.1% improvement at function calling. Notably, smaller fine-tuned models (1.5 billion to 7 billion parameters) outperform models in the same family that are 10 times larger. Our novel paradigm is thus an exciting pathway to more useful and reliable language models that can self-improve on challenging tasks with limited external feedback.

Shelly Bensal, Umar Jamil, Christopher Bryant · 204 views · May 30, 2025

Large Language ModelsReinforcement Learning

May 30, 2025

RLSR: Reinforcement Learning from Self Reward

Large language models can generate solutions to complex problems, but training them with reinforcement learning typically requires verifiable rewards that are expensive to create and not possible for all domains. We demonstrate that LLMs can effectively self-improve through self-judging without reference solutions, leveraging the inherent asymmetry between generating and verifying solutions. Our experiments show that models can provide reliable reward signals without ground truth answers, enabling reinforcement learning in domains where verifiable rewards are impractical. By implementing self-judging across Countdown puzzles and integration problems, we achieve performance comparable to formal verification without ground truth solutions. Most notably, Qwen 2.5 7B DeepSeek Distilled trained with self-rewards qualifies for the prestigious MIT Integration Bee competition, performance through self-supervised improvement. When combined with synthetic question generation, we establish a complete self-improvement loop where models generate practice problems, solve them, and evaluate their own performance without any external validation. Our findings demonstrate that LLM judges can provide effective reward signals for training, unlocking reinforcement learning in countless domains previously limited by reward engineering challenges. This work represents a significant step toward autonomous AI systems that continuously improve through self-directed learning rather than human-guided training, potentially accelerating progress across domains where training data is scarce or evaluation is complex.

Toby Simonds, Kevin Lopez, Akira Yoshiyama · 191 views · May 12, 2025

Reinforcement LearningLarge Language Models

May 12, 2025

Agent World Model in the Wild: Computer-Use Agent Environment Synthesis

Agent learning is shifting from learning from human data toward learning through interaction with the world. However, scaling such learning directly in real-world computer systems is challenging and can introduce substantial security risks. We therefore propose Agent World Model in the Wild (WildAWM), a scalable pipeline for synthesizing controllable environments and tasks for training general computer-use agents (CUAs). Whereas existing approaches typically specialize in a single agent interaction interface, we construct executable environments that jointly support graphical user interfaces, model context protocol services, and command-line interfaces, allowing agents to flexibly combine visual interaction, tool use, and code execution. WildAWM is entirely powered by open-source models, demonstrating that high-quality computer-use environments can be generated without proprietary frontier models. Beyond environments, we address three recurring challenges in task synthesis: ensuring solvability, maintaining sufficient difficulty, and producing reliable verifiers, by introducing execution-based quality gates that combine heterogeneous agent rollouts, cross-judge consistency checks, and counterfactual probes of judgment robustness. The resulting pipeline synthesizes 341 long-horizon environment-task pairs that can require over 100 agent steps to solve and remain challenging even for frontier models, with GPT-6-Astra solving only 60% of them. We train two Qwen models of different sizes exclusively in these synthetic worlds. Both models achieve substantial gains across 8 out-of-distribution benchmarks spanning web, claw, and computer-use agents, demonstrating that learning in WildAWM's synthesized worlds generalizes broadly to unseen computer-use tasks. The code is publicly available at https://github.com/MSR-Orchard/WildAWM.

Zhaoyang Wang, Qianhui Wu, Haorui Wang · 350 views · Oct 19, 2026

Computer-Use AgentWorld Models

Oct 19, 2026

IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

Most autoresearch systems couple idea generation with implementation, experiments, and writing, so it is unclear whether success comes from the idea or from downstream execution. IdeaScientist isolates research ideation as a standalone task: given a problem definition and a challenge that current solutions fail to meet, produce a grounded research proposal. An orchestrator coordinates three sub-agents - a gap finder that locates the shared methodological limitation in same-problem prior work, an innovator that retrieves mechanisms which solved an analogous challenge in other fields and adapts them, and a report writer that develops the surviving intuition into a complete proposal. Each role is trained separately with asynchronous GRPO and reference-grounded rubric rewards. The Svalbard Idea Vault decomposes 4.7M arXiv papers into 2.77M (problem, challenge, gap, intuition, proposal) tuples for retrieval, training, and temporally controlled evaluation with a January 1, 2026 cutoff and 277 held-out post-cutoff methodology papers. On Qwen3.6-27B, IdeaScientist reaches 74.6% overall, 14.0% above the strongest open-source autoresearch baseline, driven mainly by novelty, and beats Claude Code SDK with Claude-4.8-Opus by 5.9% and Codex SDK with GPT-5.4 by 5.1% on the same open backbone.

Jiarui Liu, Renjie Tao, Yiwei Liao · 638 views · Oct 2, 2026

AI ScientistLLM Agent

Oct 2, 2026

RIWANav: Recursive World-Action Models with Self-Improvement for Urban Navigation

Long-horizon urban navigation compounds single-step errors, imitation learning only supervises expert successes, and a frozen action-conditioned world model drifts out of alignment as the policy evolves. RIWANav casts the coupled adaptation of a world model and a navigation policy as task-specific recursive self-improvement: Cosmos imagines 17 future frames for four sampled waypoint chunks per state, the resulting scores drive GRPO with a KL penalty and an expert anchor, and the improved policy then proposes actions whose expert-consistent action-video pairs (weighted ADE/MADE/AOE/MAOE consistency Q >= 0.35) enter a policy-induced replay pool prioritized by behavioral novelty and current prediction error to refine the world model over K=19 rounds. On UrbanNav it reaches 89.92% SR / 86.87% SPL on test-seen and 88.25% / 82.59% on test-unseen, beating frozen-world-model GRPO-WM by 7.94 / 6.50 points and cutting navigation error from 3.3314 to 2.0429, with world-model L1/LPIPS on policy-induced actions down 3.2% / 7.1%. Real-world trials on a custom mobile robot confirm practical transfer.

Jing Xie, Shouwei Ruan, Yubin Wang · 168 views · Oct 6, 2026

UrbanNavigationWorldModel

Oct 6, 2026

The ultimate guide to multi-harness RL: train models inside Claude Code and Codex — Hugging Face's four-harness experiment with a 2.6B model

The same model behaves like two different models depending on the agent harness around it: GLM-5.2 scores 23% in one harness and 52% in another on SWE-bench Pro; in a controlled experiment swapping the harness moved scores 13 points while swapping the model moved only 2.5–5. A Hugging Face team (Adithya S Kolavi, Joel Niklaus, Lewis Tunstall, Leandro von Werra and colleagues, with Liquid AI) publishes an open fix: run agentic RL inside the real harnesses — Claude Code, Codex, OpenCode, Mini-SWE-Agent — with zero harness code changes. The stack: OpenEnv as the shared interface, a capture proxy that mints a session id per rollout and records exact token ids, per-token behavior logprobs and loss masks at the model endpoint (full-distribution sampling, top_p=1.0), Harbor for 40+ harness adapters and 26 sandbox backends, and TRL's Async GRPO consuming the masked training sequences. The numbers: LFM2.5-2.6B trained across four harnesses rises from 42.2% to 54.2% average pass@1 with gains under all four, and uses 31% fewer tool calls on tasks both it and the base model solved, while the OpenCode-only arm's gains stay mostly at home. The article also reports the SFT comparison (RL 54.6% vs SFT 47.5%), the full attribution of the earlier Qwen rise-and-decline (output-budget exhaustion, working without submitting, and a reward term that collapsed held-out accuracy from 0.740 to 0.178), and engineering boundaries like the proxy's concurrency ceiling.

BLOG

Reinforcement LearningRL

ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search

GRPO is the representative RL method for improving LLM reasoning, yet it only ever sees one sparse reward at the end of a whole trajectory: when the rewards inside a sampled group land close together, the group-relative advantage collapses into noise and the policy learns almost nothing. ReST-RL reconnects policy optimization and value-guided search into a single self-training pipeline. Stage one, ReST-GRPO, first filters out low-information prompts by reward standard deviation, then draws prefixes from each prompt's highest-reward trajectory under a discrete exponential distribution and uses them as fresh online-GRPO starting contexts. A selected prefix is context only; its suffix is re-sampled and optimized rather than imitated. Stage two, VM-MCTS, runs MCTS under the now-static policy to self-collect value targets and trains a value model that predicts expected terminal reward. At inference the same model both allocates the tree search through UCT and ranks completed candidates in a Best-of-N fashion, so search and verification share one state-value scale. On coding benchmarks including APPS, BigCodeBench and HumanEval, Qwen3-8B moves from 0.503 to 0.689 average. In matched policy-value controls, ReST-GRPO + VM-MCTS reaches 0.642 on APPS-500 while GRPO + VM-MCTS reaches only 0.538, so the stage-one distributional shift survives value learning. End-to-end accounting puts ReST-GRPO at 1,752 GPU-hours against 2,080 for GRPO, hitting a 9% gain in 71 hours instead of 207. A value model trained only on code trajectories also transfers to MATH, Omni-MATH and GPQA-Diamond without target-domain tuning.

Sining Zhoubian, Dan Zhang, Jie Tang · 752 views · Aug 27, 2025

Reinforcement LearningGRPO

Aug 27, 2025

ABot-N1: Toward a General Visual Language Navigation Foundation Model

ABot-N1 is a general visual-language navigation foundation model built on a slow-fast dual-system architecture: a 4B slow VLM reasoner emits explicit chain-of-thought plus dual pixel goals (Target Pixel + Affordance Pixel), while a 2B fast action expert decodes continuous SE(2) waypoints via QFormer action queries. The unified pixel-goal interface covers five tasks — point-goal, object-goal, POI-goal, instruction-following and person-following — in a single 30M-sample multi-task checkpoint, further aligned by GRPO post-training with format/target/safety rewards. Two closed-loop benchmarks are released (ABotN-PointBench and ABotN-POIBench). ABot-N1 sets new SOTA on all five benchmarks, boosting POI entrance arrival to 77.3% (+35.0 pp) and reaching 92.9%/95.4% outdoor/indoor point-goal SR, with full deployment on the TuTu quadruped running on a Jetson AGX Orin.

Ruiyan Gong, Yingnan Guo, Junjun Hu · 865 views · Jul 11, 2026

VLNNavigation Foundation Model

Jul 11, 2026

As an Amazon Associate, we earn from qualifying purchases.