When Verifiable Counts Depend on Wording: Auditing Wording Robustness in Instruction Following
Not on the map. Why?
Abstract (arXiv)
Verifiable instruction-following benchmarks often express each constraint through one fixed template. We test whether scores remain stable when the operational requirement is unchanged but its wording varies. We introduce WISE, a matched evaluation suite and reporting protocol instantiated on exact word count, keyword inclusion exactly once, and an inclusive 8--12 word range. Across 100 matched tasks, up to thirteen models from seven providers, and repeated generations scored over the complete visible output, wording alone produces substantial compliance shifts. In an avoidance-family panel, five avoidance and exclusion forms fall below the positive baseline, while constructional controls also shift compliance substantially: in the nine-model control panel, compliance is 54.9% for the original positive form, 48.2% for a longer positive form, 36.7% when the target appears later, and 33.8% for AVOID1. A strict JSON-structure probe shows wording sensitivity beyond counting, with a different direction of effect. Effect sizes, failure directions, weakest forms, and model rankings vary across realizations. Under the most disruptive exclusion form, the top-ranked model changes and 24.1% of strictly ordered model pairs reverse. Human validation further shows that unanimous agreement on an exact-count interpretation can coexist with substantially different model behavior. WISE supplements conventional scores with mean and worst-form compliance, wording gaps, failure profiles, and ranking stability.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 4 Oct 2026.
Builds on
22 of 31 references.
References · 22
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
- 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
- 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
- 2025Generalizing Verifiable Instruction Followingcited by 194
- 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 191
- 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
- 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
- 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
- 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 74
- 2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 66
10 more
- 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
- 2024Following Length Constraints in Instructionscited by 46
- 2024On the Worst Prompt Performance of Large Language Modelscited by 38
- 2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMscited by 30
- 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructionscited by 27
- 2022Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negationcited by 25
- 2025MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Modelscited by 15
- 2024Paraphrasing in Affirmative Terms Improves Negation Understandingcited by 8
- 2025Revisiting the Reliability of Language Models in Instruction-Followingcited by 7
- 2026Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilitiescited by 2
Built on by
Not cited yet.