mlsanityData to 14 Oct 2026

WildIFEval: Instruction Following in the Wild

arXiv 2503.06573 ↗First posted 9 Mar 2025cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent LLMs have shown remarkable success in following user instructions, yet handling instructions with multiple constraints remains a significant challenge. In this work, we introduce WildIFEval - a large-scale dataset of 7K real user instructions with diverse, multi-constraint conditions. Unlike prior datasets, our collection spans a broad lexical and topical spectrum of constraints, extracted from natural user instructions. We categorize these constraints into eight high-level classes to capture their distribution and dynamics in real-world scenarios. Leveraging WildIFEval, we conduct extensive experiments to benchmark the instruction-following capabilities of leading LLMs. WildIFEval clearly differentiates between small and large models, and demonstrates that all models have a large room for improvement on such tasks. We analyze the effects of the number and type of constraints on performance, revealing interesting patterns of model constraint-following behavior. We release our dataset to promote further research on instruction-following under complex, realistic conditions.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 38 of its 82 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

38 of 82 references.

References · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
26 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  4. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  5. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  6. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  7. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  8. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  9. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  10. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  11. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  12. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
  13. 2020CTRLsum: Towards Generic Controllable Text Summarizationcited by 98
  14. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  15. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 70
  16. 2024CFBench: A Comprehensive Constraints-Following Benchmark for LLMscited by 67
  17. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 51
  18. 2023Learning to Rewrite Prompts for Personalized Text Generationcited by 40
  19. 2024Why Do Large Language Models (LLMs) Struggle to Count Letters?cited by 38
  20. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 35
  21. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 29
  22. 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 27
  23. 2022Why is constrained neural language generation particularly challenging?cited by 19
  24. 2024A User-Centric Multi-Intent Benchmark for Evaluating Large Language Modelscited by 15
  25. 2022AutoTemplate: A Simple Recipe for Lexically Constrained Text Generationcited by 6
  26. 2024Conversational Prompt Engineeringcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025Generalizing Verifiable Instruction Followingcited by 194
  2. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  3. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 20
  4. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  5. 2025Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Followingcited by 8
  6. 2026Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Codingcited by 3
  7. 2025Reasoning-Intensive Regressioncited by 2
  8. 2026CCR-Bench: A Comprehensive Benchmark for Evaluating LLMs on Complex Constraints, Control Flows, and Real-World Casescited by 1
  9. 2026UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Followingcited by 0
  10. 2026Prompts in the Wild: A Large Analyzed Collection of Transactional Prompts in Codecited by 0
  11. 2026Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarioscited by 0