mlsanityData to 7 Oct 2026

Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models

arXiv 2412.01784 ↗First posted 2 Dec 2024cs.AI cs.CRcited by 22

Not on the map. Why?

Abstract (arXiv)

Capability evaluations play a crucial role in assessing and regulating frontier AI systems. The effectiveness of these evaluations faces a significant challenge: strategic underperformance, or ``sandbagging'', where models deliberately underperform during evaluation. Sandbagging can manifest either through explicit developer intervention or through unintended model behavior, presenting a fundamental obstacle to accurate capability assessment. We introduce a novel sandbagging detection method based on injecting noise of varying magnitudes into model weights. While non-sandbagging models show predictable performance degradation with increasing noise, we demonstrate that sandbagging models exhibit anomalous performance improvements, likely due to disruption of underperformance mechanisms while core capabilities remain partially intact. Through experiments across various model architectures, sizes, and sandbagging techniques, we establish this distinctive response pattern as a reliable, model-agnostic signal for detecting sandbagging behavior. Importantly, we find noise-injection is capable of eliciting the full performance of Mistral Large 120B in a setting where the model underperforms without being instructed to do so. Our findings provide a practical tool for AI evaluation and oversight, addressing a challenge in ensuring accurate capability assessment of frontier AI systems.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 30 of its 56 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

30 of 56 references.

References · 30

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  10. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  11. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  12. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
18 more
  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  2. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  3. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  4. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  5. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  6. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  7. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  8. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  9. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 90
  10. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  11. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  12. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 77
  13. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  14. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 46
  15. 2024Sabotage Evaluations for Frontier Modelscited by 45
  16. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 42
  17. 2024Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internalscited by 13
  18. 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agentscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 17.

22 papers.

Sort:

By area:

  1. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 31
  2. 2026Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weightscited by 25
  3. 2025Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awarenesscited by 7
  4. 2025Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Frameworkcited by 4
  5. 2026Language models recognize dropout and Gaussian noise applied to their activationscited by 3
  6. 2026Removing Sandbagging in LLMs by Training with Weak Supervisioncited by 3
  7. 2025Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Modelscited by 1
  8. 2025Sandbagging in a Simple Survival Bandit Problemcited by 1
  9. 2026Mechanistic Anomaly Detection via Functional Attributioncited by 1
  10. 2026Below-Chance Blindness: Prompted Underperformance in Small LLMs Produces Positional Bias Rather than Answer Avoidancecited by 1
  11. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimizationcited by 1
  12. 2026Overthinking: Amplifying Reasoning Weights to Extract Learned Secretscited by 1
10 more
  1. 2026Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspectivecited by 0
  2. 2026Disentangling Deception and Hallucination Failures in LLMscited by 0
  3. 2026Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solvercited by 0
  4. 2026Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbaggingcited by 0
  5. 2026Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneitycited by 0
  6. 2026Fuzzing Large Language Models to Elicit Hidden Behaviourscited by 0
  7. 2026Detecting Hidden Behaviors in LLMs via Activation-matched Finetuningcited by 0
  8. 2026One Perturbation Is Not Enough: Identifiability and Blind Baselines for Behavioral AI Evaluationcited by 0
  9. 2026A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Revealcited by 0
  10. 2026Activation Flow: Manufacturing Activations for Steeringcited by 0