mlsanityData to 7 Oct 2026

Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging

arXiv 2604.26206 ↗First posted 29 Apr 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

A predecessor pilot (Cacioli, 2026) found that Llama-3-8B implements prompted sandbagging as positional collapse rather than answer avoidance. However, fixed option ordering in MMLU-Pro left open whether this reflected a model-level position-dominant policy or dataset-level distractor structure. This pre-registered follow-up (3 models, 2,000 MMLU-Pro items, 4 conditions, 24,000 primary trials) added cyclic option-order randomisation as the critical control. The pre-registered item-level same-letter diagnostic did not confirm deterministic position-tracking (same-letter rate 37.3%, below the 50% threshold). However, pre-specified supporting analyses revealed that the response-position distribution under sandbagging was highly stable under complete content rotation (Pearson r = 0.9994; Jensen-Shannon divergence = 0.027, compared to 0.386 between honest and sandbagging conditions). Accuracy spiked to 72.1% when the correct answer coincidentally occupied the preferred position E, and fell to 4.3% at position A. The data provide strong evidence for a soft distributional attractor: under sandbagging instruction, the model enters a low-entropy response-position basin centred on E/F/G that is highly stable and largely content-invariant at the aggregate level. Qwen-2.5-7B served as a negative control (non-compliant, no distributional shift). These results provide evidence, at the 7-9 billion parameter scale, that response-position entropy is a promising black-box behavioural signature of this sandbagging mode.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 16 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

13 of 16 references.

References · 13

  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  3. 2024Frontier Models are Capable of In-context Schemingcited by 252
  4. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  5. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  6. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 31
  7. 2024Noise Injection Reveals Hidden Capabilities of Sandbagging Language Modelscited by 22
  8. 2025LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoringcited by 17
  9. 2025Set-LLM: A Permutation-Invariant LLMcited by 10
  10. 2025Auditing Games for Sandbaggingcited by 10
  11. 2026ABCD: All Biases Come Disguisedcited by 2
  12. 2025Sandbagging in a Simple Survival Bandit Problemcited by 1
1 more
  1. 2026Below-Chance Blindness: Prompted Underperformance in Small LLMs Produces Positional Bias Rather than Answer Avoidancecited by 1

Built on by

Not cited yet.