mlsanityData to 7 Oct 2026

Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models

arXiv 2406.13542 ↗First posted 19 Jun 2024cs.CL cs.AI cs.LGcited by 73

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

One core capability of large language models (LLMs) is to follow natural language instructions. However, the issue of automatically constructing high-quality training data to enhance the complex instruction-following abilities of LLMs without manual annotation remains unresolved. In this paper, we introduce AutoIF, the first scalable and reliable method for automatically generating instruction-following training data. AutoIF transforms the validation of instruction-following data quality into code verification, requiring LLMs to generate instructions, the corresponding code to check the correctness of the instruction responses, and unit test samples to verify the code's correctness. Then, execution feedback-based rejection sampling can generate data for Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) training. AutoIF achieves significant improvements across three training algorithms, SFT, Offline DPO, and Online DPO, when applied to the top open-source LLMs, Qwen2 and LLaMA3, in self-alignment and strong-to-weak distillation settings. Our code is publicly available at https://github.com/QwenLM/AutoIF.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 42 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

30 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024CodecLM: Aligning Language Models with Tailored Synthetic Data8 Apr
  2. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment10 Aug
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2023Mistral 7Bcited by 3,065
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2024Mixtral of Expertscited by 1,785
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  12. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
15 more
  1. 2023Large Language Modelscited by 774
  2. 2023Instruction Tuning with GPT-4cited by 676
  3. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  4. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  5. 2023Scaling Data-Constrained Language Modelscited by 391
  6. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  7. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  8. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  9. 2024Long-form factuality in large language modelscited by 146
  10. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 112
  11. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 79
  12. 2023Making Language Models Better Tool Learners with Execution Feedbackcited by 69
  13. 2024Towards Scalable Automated Alignment of LLMs: A Surveycited by 33
  14. 2023Toward Understanding Generative Data Augmentationcited by 25
  15. 2024RefuteBench: Evaluating Refuting Instruction-Following for Large Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 38; 2026: 20.

73 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2025Kimi K2: Open Agentic Intelligencecited by 523
  4. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  5. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  6. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  7. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  8. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
  9. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  10. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  11. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  12. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
38 more
  1. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generationcited by 34
  2. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Modelscited by 29
  3. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generationcited by 27
  4. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  5. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  6. 2025Leanabell-Prover-V2: Verifier-integrated Reasoning for Formal Theorem Proving via Reinforcement Learningcited by 26
  7. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  8. 2024Language Models can Self-Lengthen to Generate Long Textscited by 24
  9. 2024SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Modelscited by 23
  10. 2024RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generationcited by 20
  11. 2024DotaMath: Decomposition of Thought with Code Assistance and Self-correction for Mathematical Reasoningcited by 19
  12. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  13. 2024Montessori-Instruct: Generate Influential Training Data Tailored for Student Learningcited by 12
  14. 2024Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Modelscited by 12
  15. 2025What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Tracescited by 11
  16. 2025RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Datacited by 11
  17. 2025Propose, Solve, Verify: Self-Play Through Formal Verificationcited by 11
  18. 2024MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmarkcited by 8
  19. 2025Towards Understanding Self-play for LLM Reasoningcited by 8
  20. 2025Beyond Single-Task: Robust Multi-Task Length Generalization for LLMscited by 7
  21. 2025A Course Correction in Steerability Evaluation: Revealing Miscalibration and Side Effects in LLMscited by 7
  22. 2025Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Followingcited by 7
  23. 2024Baichuan Alignment Technical Reportcited by 6
  24. 2025Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Followingcited by 6
  25. 2025Quantum Verifiable Rewards for Post-Training Qiskit Code Assistantcited by 6
  26. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 6
  27. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
  28. 2025SHARP: Synthesizing High-quality Aligned Reasoning Problems for Large Reasoning Models Reinforcement Learningcited by 5
  29. 2025Reasoning Up the Instruction Ladder for Controllable Language Modelscited by 5
  30. 2026Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learningcited by 5
  31. 2025Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generationcited by 4
  32. 2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
  33. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluationcited by 4
  34. 2025Self-Improving Model Steeringcited by 3
  35. 2025Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Followingcited by 3
  36. 2025Adversarial Agent Collaboration for Correctness Improvements of C to Safe Rust Translationcited by 3
  37. 2025ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Modelscited by 3
  38. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 3

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025REFINE-AF: A Task-Agnostic Framework to Align Language Models via Self-Generated Instructions using Reinforcement Learning from Automated Feedback
  2. 2026SEIF: Self-Evolving Reinforcement Learning for Instruction Following
  3. 2024Self-Judge: Selective Instruction Following with Alignment Self-Evaluation
  4. 2026AutoTrainess: Teaching Language Models to Improve Language Models Autonomously
  5. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  6. 2025Aligning Instruction Tuning with Pre-training
  7. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
  8. 2023Tuna: Instruction Tuning using Feedback from Large Language Models