Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation
Not on the map. Why?
Abstract (arXiv)
When instructed to underperform on multiple-choice evaluations, do language models engage with question content or fall back on positional shortcuts? We map the boundary between these regimes using a six-condition adversarial instruction-specificity gradient administered to two instruction-tuned LLMs (Llama-3-8B and Llama-3.1-8B) on 2,000 MMLU-Pro items. Distributional screening (response-position entropy) and an independent content-engagement criterion (difficulty-accuracy correlation) jointly characterise each condition. The gradient reveals three regimes rather than a monotonic transition. Vague adversarial instructions produce moderate accuracy reduction with preserved content engagement. Standard sandbagging and capability-imitation instructions produce positional entropy collapse with partial content engagement. A two-step answer-aware avoidance instruction produces extreme positional collapse, with near-total concentration on a single response position (99.9% and 87.4%) and no measurable content sensitivity. This was the only multi-step instruction tested, and it produced the most extreme shortcut. The attractor position matches each model's content-absent null-prompt default. The effect replicates across both models and four academic domains. Distributional collapse and content engagement can co-occur (50% concordance between screening criteria), indicating that entropy-based screening and difficulty-based content assessment capture partially independent dimensions of response validity. Results suggest that instruction complexity can determine whether adversarial compliance uses content-aware or content-blind mechanisms in small instruction-tuned LLMs under greedy decoding.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 9 of its 16 references that are in the map. First posted on arXiv 29 Apr 2026.
Builds on
9 of 16 references.
References · 9
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
- 2024Frontier Models are Capable of In-context Schemingcited by 252
- 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
- 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
- 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
- 2025LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoringcited by 17
- 2025Separating Constraint Compliance from Semantic Accuracy: A Novel Benchmark for Evaluating Instruction-Following Under Compressioncited by 1
Built on by
Not cited yet.