#Reinforcement Learning (206)

unsloth: the 2x-faster, 70%-less-VRAM fine-tuning library, now a desktop runtime that wires local weights into Claude Code and Codex

Self-described as the first desktop app to run and train models (Windows/macOS/Linux; multi-GPU across NVIDIA, AMD, Intel, CPU and Vulkan). Fine-tuning is claimed at 2x faster with 70% less VRAM, the post-training menu covers LoRA, QLoRA, full fine-tuning, pretraining, GRPO, DPO and FP8, and export reaches GGUF, NVFP4 and FP8. Unsloth Start connects Claude Code or Codex to local weights in one command; the model list includes Qwen3.8, GLM-5.3-Flash, Kimi K3, DeepSeek-V4, MiniMax-H3 and Gemma 4, with private web search, deep research, auto-compaction and RAG built in.

Fine-tuningQLoRA

Python77k7.1k385

Genesis-Embodied-AI/genesis-worldApache-2.0

Genesis World: The Unified Multi-Physics Simulation Platform for Physical AI

Genesis AI's open simulation platform for physical AI: one Pythonic API over a multi-physics engine (rigid, FEM, MPM, PBD, SPH, Stable Fluid, IPC), the Nyx path tracer, and Quadrants, a cross-platform Python-to-GPU compiler. It is built to be the evaluation engine for robot foundation models rather than a data factory: 30,000 parallel environments at 43M FPS, and under a zero-shot real-to-sim protocol, sim-vs-real evaluation agreement of Pearson 0.8996 with mean maximum rank violation 0.0166. Apache 2.0, 29.9k stars.

SimulationPhysics Engine

Python30k2.9k224

dexmal/dexboticMIT

Dexbotic: A One-Stop VLA Toolbox for Pretraining, Fine-Tuning, Inference, and Evaluation

A one-stop VLA toolbox from Dexmal: the unified DexData format spans pretraining, fine-tuning (full/LoRA/RL), inference, and evaluation, supporting mainstream models (π0, CogACT, OFT, MemVLA, GR00T N1) plus the in-house dual-expert DM0; DB-pretraining brings consistent gains across five simulation benchmarks, with 62% average success on RoboChallenge Table30 real-robot evaluation.

VLAManipulation

Python1.4k1814

Speech & AudioMediaTopC

Fun-CosyVoice3-0.5B: open TTS that turns controllability into an interface

The third generation LLM based speech synthesis system from the FunAudioLLM group at Alibaba, 0.5B parameters, released as Fun-CosyVoice3-0.5B-2512 with base and RL weight sets plus training and inference scripts; the code repository has moved to QwenAudio/CosyVoice (Apache-2.0, 23,794 stars, verified through the GitHub API on 2026-09-29). It sits at the open source top row for speech in the audio domain, and the reason is not that it sounds most human but that it gives each of the four failure modes of production TTS an explicit interface: misread polyphonic characters go through pronunciation correction with Chinese pinyin or English CMU phonemes written straight into the input, wrong readings of numbers and symbols go through the built in text normalisation, collapsed long sentences go through RAS repetition aware sampling, and low latency goes through bidirectional streaming with an official first packet at 150ms, which is the right order of magnitude to sit inside a realtime conversational agent. Three readings matter on the metric sheet. The test-zh speaker similarity of 78.0 is the highest among 0.5B open models and above the human baseline of 75.5, yet still below the closed source Seed-TTS at 79.6, so open source first holds while world first does not. The test-hard column is the real strength, base 6.71 and RL 5.44 being the lowest in the table and better than the closed source 7.59, and hard is exactly long sentences and tongue twisters, so that lead maps to usable real scripts. English needs a discount: base test-en similarity 71.8 sits below the human baseline 73.4 and only the RL tier WER of 1.68 recovers it. Base and RL are two post training weights of one architecture, with all three error rates falling (zh CER down 33 percent, hard CER down 19 percent) and all three similarities slipping slightly, which is a good trade for broadcast and support workloads where one misread character is an incident, while voice fidelity work for a specific IP should audition the RL tier first. The repository also ships GRPO training scripts and a triton plus TensorRT-LLM runtime claimed at four times the speed of HF transformers, so post training is a path others can continue rather than a one off delivery, and the language surface covers nine languages with more than eighteen Chinese dialect accents, a tier the closed APIs still barely offer. Boundaries: the licence is two layered, Apache-2.0 for code while the weight terms live on the model pages and must be checked separately before commercial use; every reading comes from the evaluation of the authors themselves with no third party blind listening table to cross check and not recomputed here; zero shot cloning similarity comes from standard sets while real deployments depend on the noise and style of the reference recording. Graded C (vendor claim).

78.0test-zh speaker similarity (0.5B open-weight tier, human baseline 75.5)Vendor Claim · 2025-12

ProductAlibaba Tongyi FunAudioLLM (QwenAudio)Site Repo

AI CodingDevelopmentTopC

Cursor: one of the few vendors holding all three layers - in-house coding model, own harness, cloud-parallel execution

Cursor began as a VS Code fork from Anysphere and by 2026 is one of very few vendors holding three layers at once: an in-house agentic coding model (Composer), its own harness, and cloud-parallel execution - layers that feed each other, since the real long tasks running through the product every day become RL environments and the resulting Composer is tuned against the tool surface of Cursor itself. The Agent is officially decomposed into instructions (system prompt plus rules), tools (file editing, codebase search, terminal execution) and model, with instructions and tools tuned per frontier model and no cap on tool calls. Composer 2 (2026-03-19) reports CursorBench 61.3, Terminal-Bench 2.0 61.7 and SWE-bench Multilingual 73.7 at $0.50 input / $2.50 output per million tokens, with the same-intelligence fast tier as the product default. Composer 2.5 (2026-05-18) stops adding benchmark tables and targets long-horizon persistence, instruction following and collaboration feel via targeted RL with text feedback (hint-augmented policy as teacher, original-context policy as student, an on-policy distillation KL term) plus 25x the synthetic tasks of Composer 2, including dynamic feature-removal problems. Eval footnotes disclose harness differences: Anthropic models on Claude Code, OpenAI models on Simple Codex, Cursor scores on the Harbor framework averaged over 5 runs per pairing. Boundaries: Composer is closed source and is not the same thing as the open-weight base, CursorBench is in-house, and the forked IDE lags the upstream extension ecosystem. Confidence C (vendor-claim).

61.7Terminal-Bench 2.0 (Composer 2, Harbor / mean of 5 runs)Vendor Claim · 2026-03

ProductAnysphereSite

MathResearchTopC

Kimina-Prover: reaching 92.2% on miniF2F with whole-proof RL and no search tree

The Lean 4 theorem proving line from Moonshot AI with Numina, deliberately contrarian: no prover feedback in training or inference, no Monte Carlo tree search, value function or process reward model (arXiv:2504.11354), so search is internalised into a single whole-proof generation, and a custom Formal Reasoning Pattern writes a natural-language draft before Lean code, recasting a sparse binary reward as a language problem. The base is Qwen2.5-72B with up to 32K context. Keep two generations apart: Preview (2025-04) reached 80.7% on miniF2F-test at pass@8192, the first public result above 80%; the 72B model (2025-07-10) reached 92.2% after adding TTRL search and error repair, the highest public reading then and the number on the card, while pass@1 under the same protocol is only 63.9%. Equal-budget pass@1/32/1024 of 63.9/84.0/87.7 leads the 671B DeepSeek-Prover-V2 (61.9/82.4/86.6) at all three budgets. Discounts: that table is Kimina's own and its DeepSeek rows are their reruns, the repo declares no licence, and miniF2F holds 244 problems now past 92%, so its power to separate frontier models is decaying. No third-party board exists here, so read the numbers as direction. Graded C (vendor-stated): nothing recomputed on a pinned commit.

92.2%miniF2F (after TTRL search)Vendor Claim · 2025-07

ResearchMoonshot AI (Numina & Kimi)Site Repo

mekion/the-bimo-projectApache-2.0

Bimo: a 45 cm Open-Source Hip-Head Biped Trained in Isaac Lab and Distilled onto an RP2040

Bimo is a 45 cm, roughly 1.6 kg hip-head biped kit: eight STS-3215 bus servos, a BNO08x 9-DoF IMU, four VL53L0X rangefinders, two 180-degree cameras, and a custom RP2040 board closing a 20 Hz control loop. The repository publishes the Python control API (1023 lines), three MCU firmware builds (1349 lines), a ROS2 wrapper (1431 lines) and an Isaac Lab training environment (874 lines: six reward terms, dense domain randomization, and a system-identified STS3215Actuator with 3.113 Hz bandwidth, directional gear backlash and a 5 ms bus delay), plus a 26MB Bimo.usd model. The CPG gait's 104 Fourier coefficients and per-joint amplitude gains are hardcoded in both Python and C, and a [256,128,64] PPO teacher distills into a [64,32] student that goes through onnx2c into the firmware, so the robot walks untethered on the MCU. All code is Apache-2.0, but the CAD and electronics the README promises are still marked coming soon: no STL, STEP, schematic, gerbers or BOM exist in the working tree or in git history, and neither release ships pretrained weights (zero assets) or a DIY assembly manual. Currently v1.1.0 with 198 stars, in pre-order status.

Bipedal robotsHumanoid

Python19887

0:33

D&A Turns Any VLM Into a Robot Reward Model

@memmelma · 16 views · 2026-11-03

Robot RewardsVLM

Rufus-Air: An Open LLM Post-Training Recipe

Rufus-Air is an open and reproducible post-training recipe on GLM-4.5-Air-Base (106B-A12B), organized as a serial pipeline of eight stages: SFT, Reasoning RL, Coding RL, Instruction-Following RL, General Agent, Coding Agent, Search Agent, and RLHF. We document the data, reward design, infrastructure, stage order, and stagewise results needed to reproduce the recipe. Stages progress from basic to advanced capabilities and from hard, verifiable rewards to softer judge-based signals. Training builds on open-source components and public data, much of it used as released, without new human annotation or an in-house distillation teacher. Our main findings are that (i) diverse, high-quality SFT establishes a strong capability floor; (ii) difficulty filtering keeps RL prompts within a productive learning range; (iii) reward reliability provides a practical principle for ordering stages; and (iv) infrastructure and engineering choices are part of the recipe, not just an implementation detail. Rufus-Air improves over the official GLM-4.5-Air post-trained release and is competitive with similarly sized open models.

Chia-Yuan Chang, Renyuan Cheng, Rui Feng · 94 views · Sep 24, 2026

LLMPost-training

Sep 24, 2026

0:13

Gym-Anything Builds Robot Training Worlds

@wellecks · 66 views · 2026-10-30

Robot simulationAgentic Coding

0:24

Gym-Anything: 200 Robot RL Environments

@PranjalAggarw16 · 65 views · 2026-10-30

Robotics SimulationReinforcement Learning

BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion

BeyondMimic first masters diverse agile humanoid skills with a compact motion-tracking formulation, then composes and guides them through a unified latent diffusion model for unseen tasks and zero-shot deployment.

Qiayuan Liao, Takara E. Truong, Xiaoyu Huang · 57 views · Aug 11, 2025

HumanoidMotion Tracking

Aug 11, 2025

Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning

We explore a method for improving the performance of large language models through self-reflection and reinforcement learning. By incentivizing the model to generate better self-reflections when it answers incorrectly, we demonstrate that a model's ability to solve complex, verifiable tasks can be enhanced even when generating synthetic data is infeasible and only binary feedback is available. Our framework operates in two stages: first, upon failing a given task, the model generates a self-reflective commentary analyzing its previous attempt; second, the model is given another attempt at the task with the self-reflection in context. If the subsequent attempt succeeds, the tokens generated during the self-reflection phase are rewarded. Our experimental results show substantial performance gains across a variety of model architectures, as high as 34.7% improvement at math equation writing and 18.1% improvement at function calling. Notably, smaller fine-tuned models (1.5 billion to 7 billion parameters) outperform models in the same family that are 10 times larger. Our novel paradigm is thus an exciting pathway to more useful and reliable language models that can self-improve on challenging tasks with limited external feedback.

Shelly Bensal, Umar Jamil, Christopher Bryant · 174 views · May 30, 2025

Large Language ModelsReinforcement Learning

May 30, 2025

RLSR: Reinforcement Learning from Self Reward

Large language models can generate solutions to complex problems, but training them with reinforcement learning typically requires verifiable rewards that are expensive to create and not possible for all domains. We demonstrate that LLMs can effectively self-improve through self-judging without reference solutions, leveraging the inherent asymmetry between generating and verifying solutions. Our experiments show that models can provide reliable reward signals without ground truth answers, enabling reinforcement learning in domains where verifiable rewards are impractical. By implementing self-judging across Countdown puzzles and integration problems, we achieve performance comparable to formal verification without ground truth solutions. Most notably, Qwen 2.5 7B DeepSeek Distilled trained with self-rewards qualifies for the prestigious MIT Integration Bee competition, performance through self-supervised improvement. When combined with synthetic question generation, we establish a complete self-improvement loop where models generate practice problems, solve them, and evaluate their own performance without any external validation. Our findings demonstrate that LLM judges can provide effective reward signals for training, unlocking reinforcement learning in countless domains previously limited by reward engineering challenges. This work represents a significant step toward autonomous AI systems that continuously improve through self-directed learning rather than human-guided training, potentially accelerating progress across domains where training data is scarce or evaluation is complex.

Toby Simonds, Kevin Lopez, Akira Yoshiyama · 167 views · May 12, 2025

Reinforcement LearningLarge Language Models

May 12, 2025

Post-Training Large Language Models via Reinforcement Learning from Self-Feedback

Large Language Models (LLMs) often produce plausible but poorly-calibrated answers, limiting their reliability on reasoning-intensive tasks. Recent research suggests that Chain-of-Thought (CoT) reasoning paths are inherent in pre-trained LLMs and can be elicited by simply altering the decoding process, where the presence of a CoT path correlates with higher answer confidence. Building on these insights, we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward. By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences. These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards. RLSF simultaneously (i) refines the model's probability estimates–restoring well-behaved calibration–and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering. By converting a model's own uncertainty into structured self-feedback, RLSF affirms reinforcement learning on intrinsic model behaviour as a principled and data-efficient component of the LLM post-training pipeline. Our results demonstrate that leveraging these inherent reasoning capabilities provides a robust path for enhancing model reliability without manual prompt engineering or external supervision.

Carel van Niekerk, Renato Vukovic, Benjamin Ruppik · 320 views · Jul 29, 2025

Large Language ModelsReinforcement Learning

Jul 29, 2025

NanoDreamer: A 750k-Parameter World Model That Learns to Act Through Imagination

Built from a custom autograd engine to a 750k-parameter Dreamer, NanoDreamer trains its actor entirely in imagined rollouts and beats random play across all five MinAtar games. The write-up keeps the failed first attempt, reward-timing diagnostics, critic optimism, CNN/multi-step rebuild, and ablations.

BLOG

World ModelsReinforcement Learning

Learning Multi-Phase Robot Loco-Manipulation via Phase-Conditioned Policy Optimization

Complex whole-body loco-manipulation requires executing diverse maneuvers, such as crouching, grasping, and locomotion, within a single long-horizon task. While performing simple pick-and-place behaviors is well-established, scaling to complex, long-horizon tasks remains difficult. Currently, this challenge is addressed by relying on human motion data and correspondingly humanoid platforms, which limits platform independence and introduces substantial data processing overhead. Instead, we extend proximal policy optimization (PPO) to train a single policy for multi-phase tasks across diverse robotic hardware, leveraging prior task structure to shape behavior. Our method adapts PPO using a phase-conditioned critic, phase-gated reward functions, and a multi-phase reset curriculum to guide exploration. To support varied hardware, we employ a decoupled hierarchical control architecture that separates lower-level balancing dynamics from high-level phase execution. We demonstrate multi-stage loco-manipulation skills on hardware on both the Unitree G1 and a Boston Dynamics Spot equipped with a manipulator arm, and validate our design choices through ablations. Our approach enables the acquisition of complex loco-manipulation behaviors purely through environment interaction while directly incorporating high-level task structure into policy learning.

Simone Manni, Annika Wollschläger, Dario Bellicoso · 250 views · Oct 26, 2026

Whole-Body Loco-ManipulationPhase-Conditioned PPO

Oct 26, 2026

Why is AI-generated language so hard to understand?

The authors compare psycholinguistic processing-cost features across 14 language models in chat, coding, and essay settings. Relative to round-trip paraphrases, frontier models overuse low-frequency n-grams, long dependencies, stacked noun compounds, denominal verbs, reduced object relatives, and atypical animacy patterns. They hypothesize that post-training for math, code, and verifiable rewards rewards information density rather than human communication. A Qwen2.5-1.5B Instruct experiment with GRPO shows that a simple length penalty can create a near-categorical preference for shorter variants and transfer to some untrained constructions. Task success therefore does not guarantee human-comprehensible language.

Thomas Hikaru Clark, Matias B. Martin, Kyle Mahowald · 277 views · Oct 21, 2026

Large Language ModelsPsycholinguistics

Oct 21, 2026

FLORA: An Open Model Suite for Scaling Robotics Data Engines

The life-cycle of robotics foundation models crucially depends on data labels. Outcome reward labels are needed for monitoring real-world deployments. Process reward labels are needed during RL post-training. Timestamped captions are needed for filtering and balancing pretraining data. We present Flora, an industrial-quality data engine used for training GR00T models, that simultaneously covers these capabilities. Flora achieves SOTA scores on the captioning benchmark WGO-Bench and reward benchmarks RoboReward and Robometer. On internal benchmarks it is competitive with GPT-6 Astra while beating other frontier models. The model is substantially cheaper than API-based alternatives, roughly 125 times cheaper than GPT-6 Astra. Flora is battle tested at scale, labeling 200,000 hours of data per day across 512 GB300s. By open-sourcing our data engine, we enable researchers to scale custom robotics datasets and explore emerging applications such as physical auto-research and human-to-robot transfer.

Sicong Leng, Yunze Man, Zhiqi Li · 136 views · Oct 22, 2026

Robotics Data EngineTemporal Action Segmentation

Oct 22, 2026

MorphMimic: Morphology-Conditioned World Model and Policy Sharing Across Humanoids

MorphMimic conditions a shared differentiable world model and tracking policy on a 125-value morphology descriptor, enabling three humanoids to learn from 128,786 common motion clips and share one controller.

Lizhi Yang, Fengze Xie, Yuyang Wang · 356 views · Oct 25, 2026

HumanoidCross-Embodiment

Oct 25, 2026

1:27

NanoLoong vs Microduck: Two Open-Source Bipeds at Very Different Scales

@techniahqrobot · 155 views · 2026-10-26

HumanoidLegged Locomotion

Compliant GMT: Making General Motion Tracking Compliant via Residual Reference Adaptation

We propose Compliant GMT (C-GMT), a modular framework for full-body compliant responses in general motion tracking. As humanoid robots move into real world deployment, they not only need to perform any motion but also respond safely and robustly to interaction forces. Existing general motion tracking policies can imitate diverse motions but are trained to be stiff, leading to unexpected and unsafe behaviors under disturbance. Prior work in compliant humanoid control trades off compliance and generality. There exist examples of full-body compliant responses, but these are heavily curated to single-reference policies. On the other hand, general motion trackers (GMT) are trained to rigidly imitate massive motion capture datasets. Our key insight is to leverage the generality of pretrained GMTs for motion tracking while introducing compliance from a residual adapter that conditions on a history of proprioception. This enables our approach, C-GMT, to achieve both general motion tracking and whole-body compliant responses. Compared to state-of-the-art general motion trackers and single-motion compliant policies, C-GMT demonstrates competitive performance in both motion tracking accuracy and compliance adherence. We further demonstrate improved safety, robustness, and error tolerance in simulation and real-world G1 experiments.

Venki Parsuram, Michelle Wang, Jin-Hyuk Hong · 259 views · Oct 15, 2026

HumanoidMotion Tracking

Oct 15, 2026

AgenticRoboRL: Bringing Coding-Agent Post-Training to Agentic Robotics

Agentic robot systems pair a language-model planner with a robot harness, but recent gains have largely come from stronger closed-source planners, better harnesses, and external memory rather than from improving the planner itself. AgenticRoboRL investigates whether coding-agent post-training can improve robot planners across both VLA skill invocation and control-code generation. Its recipe combines supervised fine-tuning on successful teacher trajectories with multi-turn reinforcement learning using task success as the terminal reward. A recording proxy preserves each sampled response and its actual context, while harness-specific adapters associate model calls with task outcomes and exclude infrastructure-invalid episodes. Only the planner is trained; perception and execution components remain fixed. With a 9B planner, success rises from 18.1% to 71.0% on LIBERO-PRO and from 19.2% to 72.2% on LIBERO-Plus. With the CaP-X control-code harness, the same post-training pipeline raises success from 2.7% to 28.5%.

Xinxin Liu, Bingxin Xu, Wenrui Bao · 173 views · Oct 20, 2026

Agentic RoboticsRobot Manipulation

Oct 20, 2026

From Video to Vision: Learning Generalizable Humanoid Carrying from Human Motion

Most video-driven humanoid imitation policies still need an upper-level module to generate reference trajectories at deployment, and the resulting controllers often depend on global position in a motion-capture studio. MOVER replaces that dependency with a three-stage teacher-assistant-student pipeline. It first recovers human motion from 100 videos with GENMO, retargets it to a 29-DoF Tiangong 3.0 humanoid through GMR, and augments object geometry, scale, and pose to obtain 4,000 reference trajectories. A teacher policy learns strict motion and object tracking; an assistant relaxes global body-position tracking to improve placement generalization; and a DAgger-PPO student removes proprioceptive position and reference-joint-position inputs. At deployment, a head-mounted RGB-D camera and an object tag provide relative object pose, while QuatNet predicts the reference anchor orientation from object pose, joint state, and target location. The paper evaluates four object placements and two shape-size configurations in simulation, then demonstrates carrying on the physical robot without motion capture. Remaining limitations include limited demonstration diversity, imperfect target localization without global proprioception, and a small real-world trial count.

Ziluo Ding · 251 views · Oct 10, 2026

HumanoidObject Carrying

Oct 10, 2026

As an Amazon Associate, we earn from qualifying purchases.