The Instruction Gap: LLMs get lost in Following Instruction
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding and generation, yet their deployment in enterprise environments reveals a critical limitation: inconsistent adherence to custom instructions. This study presents a comprehensive evaluation of 13 leading LLMs across instruction compliance, response accuracy, and performance metrics in realworld RAG (Retrieval-Augmented Generation) scenarios. Through systematic testing with samples and enterprise-grade evaluation protocols, we demonstrate that instruction following varies dramatically across models, with Claude-Sonnet-4 and GPT-5 achieving the highest results. Our findings reveal the "instruction gap" - a fundamental challenge where models excel at general tasks but struggle with precise instruction adherence required for enterprise deployment. This work provides practical insights for organizations deploying LLM-powered solutions and establishes benchmarks for instruction-following capabilities across major model families.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 19 Dec 2025.
Builds on
22 of 32 references.
References · 22
- 2017Attention Is All You Needcited by 59,932
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2024GPT-4o System Cardcited by 4,467
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
10 more
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2023Active Retrieval Augmented Generationcited by 506
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
- 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
- 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
- 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
- 2024Corrective Retrieval Augmented Generationcited by 187
- 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
- 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
- 2024Enterprise Benchmarks for Large Language Model Evaluationcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications
- 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
- 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models
- 2025The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives
- 2026Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs
- 2025How Many Instructions Can LLMs Follow at Once?
- 2025A Survey of LLM Alignment: Instruction Understanding, Intention Reasoning, and Reliable Generation
- 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants