# Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities

arXiv: https://arxiv.org/abs/2601.18554 · HTML: https://mlsanity.com/paper/2601.18554/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 8 of 8)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [Can Large Language Models Understand Real-World Complex Instructions?](https://mlsanity.com/paper/2309.09150/) (2023)
- [IHEval: Evaluating Language Models on Following the Instruction Hierarchy](https://mlsanity.com/paper/2502.08745/) (2025)
- [LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios](https://mlsanity.com/paper/2411.07037/) (2024)
- [StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following](https://mlsanity.com/paper/2502.14494/) (2025)
- [Why is constrained neural language generation particularly challenging?](https://mlsanity.com/paper/2206.05395/) (2022)
- [MaXIFE: Multilingual and Cross-lingual Instruction Following Evaluation](https://mlsanity.com/paper/2506.01776/) (2025)

## Built on by (most cited first, 2 of 2)

- [Prompt Complexity Dilutes Structured Reasoning: A Follow-Up Study on the Car Wash Problem](https://mlsanity.com/paper/2603.13351/) (2026)
- [When Verifiable Counts Depend on Wording: Auditing Wording Robustness in Instruction Following](https://mlsanity.com/paper/2610.05278/) (2026)

## Abstract (arXiv)

Reliably ensuring Large Language Models (LLMs) follow complex instructions is a critical challenge, as existing benchmarks often fail to reflect real-world use or isolate compliance from task success. We introduce MOSAIC (MOdular Synthetic Assessment of Instruction Compliance), a modular framework that uses a dynamically generated dataset with up to 20 application-oriented generation constraints to enable a granular and independent analysis of this capability. Our evaluation of five LLMs from different families based on this new benchmark demonstrates that compliance is not a monolithic capability but varies significantly with constraint type, quantity, and position. The analysis reveals model-specific weaknesses, uncovers synergistic and conflicting interactions between instructions, and identifies distinct positional biases such as primacy and recency effects. These granular insights are critical for diagnosing model failures and developing more reliable LLMs for systems that demand strict adherence to complex instructions.