# Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models

arXiv: https://arxiv.org/abs/2412.01784 · HTML: https://mlsanity.com/paper/2412.01784/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Builds on (most cited first, 30 of 30)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Foundational Challenges in Assuring Alignment and Safety of Large Language Models](https://mlsanity.com/paper/2404.09932/) (2024)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity](https://mlsanity.com/paper/2401.01967/) (2024)
- [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://mlsanity.com/paper/2402.14811/) (2024)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Scheming AIs: Will AIs fake alignment during training in order to get power?](https://mlsanity.com/paper/2311.08379/) (2023)
- [How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions](https://mlsanity.com/paper/2309.15840/) (2023)
- [Overthinking the Truth: Understanding how Language Models Process False Demonstrations](https://mlsanity.com/paper/2307.09476/) (2023)
- [Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks](https://mlsanity.com/paper/2307.00175/) (2023)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [Evaluating Superhuman Models with Consistency Checks](https://mlsanity.com/paper/2306.09983/) (2023)
- [Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Models](https://mlsanity.com/paper/2308.01404/) (2023)
- [Stress-Testing Capability Elicitation With Password-Locked Models](https://mlsanity.com/paper/2405.19550/) (2024)
- [Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agents](https://mlsanity.com/paper/2408.02205/) (2024)
- [Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internals](https://mlsanity.com/paper/2405.05466/) (2024)

## Built on by (most cited first, 24 of 24)

- [Probing and Steering Evaluation Awareness of Language Models](https://mlsanity.com/paper/2507.01786/) (2025)
- [Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights](https://mlsanity.com/paper/2603.12228/) (2026)
- [Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness](https://mlsanity.com/paper/2509.00591/) (2025)
- [Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework](https://mlsanity.com/paper/2507.12872/) (2025)
- [Language models recognize dropout and Gaussian noise applied to their activations](https://mlsanity.com/paper/2604.17465/) (2026)
- [Removing Sandbagging in LLMs by Training with Weak Supervision](https://mlsanity.com/paper/2604.22082/) (2026)
- [Mechanism Design for Alignment and Control](https://mlsanity.com/paper/2609.01595/) (2026)
- [Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Models](https://mlsanity.com/paper/2504.03302/) (2025)
- [Sandbagging in a Simple Survival Bandit Problem](https://mlsanity.com/paper/2509.26239/) (2025)
- [Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspective](https://mlsanity.com/paper/2602.07259/) (2026)
- [Mechanistic Anomaly Detection via Functional Attribution](https://mlsanity.com/paper/2604.18970/) (2026)
- [Below-Chance Blindness: Prompted Underperformance in Small LLMs Produces Positional Bias Rather than Answer Avoidance](https://mlsanity.com/paper/2604.25249/) (2026)
- [Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization](https://mlsanity.com/paper/2605.29396/) (2026)
- [Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets](https://mlsanity.com/paper/2607.08173/) (2026)
- [Disentangling Deception and Hallucination Failures in LLMs](https://mlsanity.com/paper/2602.14529/) (2026)
- [Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver](https://mlsanity.com/paper/2604.25067/) (2026)
- [Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging](https://mlsanity.com/paper/2604.26206/) (2026)
- [Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity](https://mlsanity.com/paper/2605.06327/) (2026)
- [Fuzzing Large Language Models to Elicit Hidden Behaviours](https://mlsanity.com/paper/2606.29646/) (2026)
- [Detecting Hidden Behaviors in LLMs via Activation-matched Finetuning](https://mlsanity.com/paper/2609.00351/) (2026)
- [One Perturbation Is Not Enough: Identifiability and Blind Baselines for Behavioral AI Evaluation](https://mlsanity.com/paper/2609.06190/) (2026)
- [A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Reveal](https://mlsanity.com/paper/2609.21996/) (2026)
- [Activation Flow: Manufacturing Activations for Steering](https://mlsanity.com/paper/2609.32530/) (2026)
- [AI Models Can Provably Hide Arbitrary Capabilities](https://mlsanity.com/paper/2610.13928/) (2026)

## Abstract (arXiv)

Capability evaluations play a crucial role in assessing and regulating frontier AI systems. The effectiveness of these evaluations faces a significant challenge: strategic underperformance, or ``sandbagging'', where models deliberately underperform during evaluation. Sandbagging can manifest either through explicit developer intervention or through unintended model behavior, presenting a fundamental obstacle to accurate capability assessment. We introduce a novel sandbagging detection method based on injecting noise of varying magnitudes into model weights. While non-sandbagging models show predictable performance degradation with increasing noise, we demonstrate that sandbagging models exhibit anomalous performance improvements, likely due to disruption of underperformance mechanisms while core capabilities remain partially intact. Through experiments across various model architectures, sizes, and sandbagging techniques, we establish this distinctive response pattern as a reliable, model-agnostic signal for detecting sandbagging behavior. Importantly, we find noise-injection is capable of eliciting the full performance of Mistral Large 120B in a setting where the model underperforms without being instructed to do so. Our findings provide a practical tool for AI evaluation and oversight, addressing a challenge in ensuring accurate capability assessment of frontier AI systems.