mlsanityData to 14 Oct 2026

Generalizing Verifiable Instruction Following

arXiv 2507.02833 ↗First posted 3 Jul 2025cs.CLcited by 194

Not on the map. Why?

Cited by 39 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A crucial factor for successful human and AI interaction is the ability of language models or chatbots to follow human instructions precisely. A common feature of instructions are output constraints like ``only answer with yes or no" or ``mention the word `abrakadabra' at least 3 times" that the user adds to craft a more useful answer. Even today's strongest models struggle with fulfilling such constraints. We find that most models strongly overfit on a small set of verifiable constraints from the benchmarks that test these abilities, a skill called precise instruction following, and are not able to generalize well to unseen output constraints. We introduce a new benchmark, IFBench, to evaluate precise instruction following generalization on 58 new, diverse, and challenging verifiable out-of-domain constraints. In addition, we perform an extensive analysis of how and on what data models can be trained to improve precise instruction following generalization. Specifically, we carefully design constraint verification modules and show that reinforcement learning with verifiable rewards (RLVR) significantly improves instruction following. In addition to IFBench, we release 29 additional new hand-annotated training constraints and verification functions, RLVR training prompts, and code.

Key facts. Cited by 194 papers in the map (data through 14 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

31 of 40 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2025Gemma 3 Technical Reportcited by 1,824
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
19 more
  1. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  2. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  3. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  4. 20242 OLMo 2 Furiouscited by 340
  5. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  6. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  7. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  8. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  9. 2024Nemotron-4 340B Technical Reportcited by 144
  10. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  11. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
  12. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  13. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  14. 2023Training Chain-of-Thought via Latent-Variable Inferencecited by 47
  15. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 34
  16. 2025Verifiable Format Control for Large Language Model Generationscited by 17
  17. 2025WildIFEval: Instruction Following in the Wildcited by 11
  18. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Followingcited by 8
  19. 2024Models That Prove Their Own Correctnesscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 30; 2026: 164.

194 papers.

Sort:

By area:

  1. 2026Gemma 4 Technical Reportcited by 259
  2. 2026Qwen3.5-Omni Technical Reportcited by 174
  3. 2025Towards a Science of Scaling Agent Systemscited by 138
  4. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  5. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  6. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 96
  7. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  8. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
  9. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  10. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 44
  11. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  12. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
38 more
  1. 2026Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scalecited by 29
  2. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 24
  3. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 24
  4. 2025Training LLMs for Honesty via Confessionscited by 22
  5. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  6. 2026Olmo Hybrid: From Theory to Practice and Backcited by 22
  7. 2026Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agentcited by 22
  8. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 20
  9. 2026TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectoriescited by 16
  10. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 16
  11. 2025Game-Time: Evaluating Temporal Dynamics in Spoken Language Modelscited by 13
  12. 2025LiveTradeBench: Seeking Real-World Alpha with Large Language Modelscited by 13
  13. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  14. 2026Shepherd: Enabling Programmable Meta-Agents via Reversible Agentic Execution Tracescited by 12
  15. 2026Computer Environments Elicit General Agentic Intelligence in LLMscited by 11
  16. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Modelscited by 11
  17. 2026HEARTS: Benchmarking LLM Reasoning on Health Time Seriescited by 11
  18. 2026Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Modelcited by 11
  19. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 10
  20. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 10
  21. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 10
  22. 2025Maestro: Joint Graph & Config Optimization for Reliable AI Agentscited by 9
  23. 2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9
  24. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  25. 2025Hermes 4 Technical Reportcited by 8
  26. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewardscited by 8
  27. 2025Safety Alignment of LMs via Non-cooperative Gamescited by 8
  28. 2025Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Followingcited by 8
  29. 2026KAT-Coder-V2 Technical Reportcited by 8
  30. 2026Weak-to-Strong On-Policy Distillationcited by 8
  31. 2025IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewardscited by 7
  32. 2025Revisiting the Reliability of Language Models in Instruction-Followingcited by 7
  33. 2026AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputscited by 7
  34. 2025Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Followingcited by 6
  35. 2025Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Followingcited by 6
  36. 2026Many-Tier Instruction Hierarchy in LLM Agentscited by 6
  37. 2026CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systemscited by 6
  38. 2026SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Followingcited by 6

50 most cited of 194 All 194 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Following
  2. 2025VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains
  3. 2024Do LLMs "know" internally when they follow instructions?
  4. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
  5. 2025Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Models
  6. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
  7. 2025Agentic Policy Optimization via Instruction-Policy Co-Evolution
  8. 2026ImpRIF: Stronger Implicit Reasoning Leads to Better Complex Instruction Following