mlsanityData to 7 Oct 2026

Rufus-Air: An Open LLM Post-Training Recipe

arXiv 2609.29421 ↗First posted 24 Sep 2026cs.CL cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Rufus-Air is an open and reproducible post-training recipe on GLM-4.5-Air-Base (106B-A12B), organized as a serial pipeline of eight stages: SFT, Reasoning RL, Coding RL, Instruction-Following RL, General Agent, Coding Agent, Search Agent, and RLHF. We document the data, reward design, infrastructure, stage order, and stagewise results needed to reproduce the recipe. Stages progress from basic to advanced capabilities and from hard, verifiable rewards to softer judge-based signals. Training builds on open-source components and public data, much of it used as released, without new human annotation or an in-house distillation teacher. Our main findings are that (i) diverse, high-quality SFT establishes a strong capability floor; (ii) difficulty filtering keeps RL prompts within a productive learning range; (iii) reward reliability provides a practical principle for ordering stages; and (iv) infrastructure and engineering choices are part of the recipe, not just an implementation detail. Rufus-Air improves over the official GLM-4.5-Air post-trained release and is competitive with similarly sized open models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 79 of its 121 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

79 of 121 references.

References · 79

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  6. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  13. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  14. 2021Measuring Coding Challenge Competence With APPScited by 879
  15. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  16. 2025Group Sequence Policy Optimizationcited by 788
  17. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  18. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  19. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  20. 2025Kimi K2: Open Agentic Intelligencecited by 523
  21. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  22. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  23. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  24. 2025Humanity's Last Examcited by 467
  25. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  26. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  27. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  28. 2025Generalizing Verifiable Instruction Followingcited by 178
  29. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 171
  30. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 151
  31. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  32. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  33. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 106
  34. 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scalingcited by 72
  35. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  36. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
  37. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 65
  38. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 61

50 most cited of 79 All 79 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?
  2. 2026Performance, Efficiency and Collapse -- Advantages and Challenges in Offline Post-training of Code LLMs
  3. 2026ExpRL: Exploratory RL for LLM Mid-Training
  4. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
  5. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning
  6. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
  7. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  8. 2026What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA