# Stress-Testing Capability Elicitation With Password-Locked Models

arXiv: https://arxiv.org/abs/2405.19550 · HTML: https://mlsanity.com/paper/2405.19550/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 64 references that are in the map.
- First posted on arXiv 29 May 2024.

## Builds on (most cited first, 30 of 53)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Spectral Signatures in Backdoor Attacks](https://mlsanity.com/paper/1811.00636/) (2018)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020)

## Built on by (most cited first, 30 of 46)

- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs](https://mlsanity.com/paper/2407.15549/) (2024)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities](https://mlsanity.com/paper/2502.05209/) (2025)
- [Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space](https://mlsanity.com/paper/2406.19370/) (2024)
- [Safety Guardrails for LLM-Enabled Robots](https://mlsanity.com/paper/2503.07885/) (2025)
- [Towards evaluations-based safety cases for AI scheming](https://mlsanity.com/paper/2411.03336/) (2024)
- [Adversarial Attacks on Robotic Vision Language Action Models](https://mlsanity.com/paper/2506.03350/) (2025)
- [Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models](https://mlsanity.com/paper/2412.01784/) (2024)
- [Async Control: Stress-testing Asynchronous Control Measures for LLM Agents](https://mlsanity.com/paper/2512.13526/) (2025)
- [Forecasting Rare Language Model Behaviors](https://mlsanity.com/paper/2502.16797/) (2025)
- [The Elicitation Game: Evaluating Capability Elicitation Techniques](https://mlsanity.com/paper/2502.02180/) (2025)
- [The Singapore Consensus on Global AI Safety Research Priorities](https://mlsanity.com/paper/2506.20702/) (2025)
- [Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors](https://mlsanity.com/paper/2512.11949/) (2025)
- [A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoring](https://mlsanity.com/paper/2602.23163/) (2026)
- [The Shape of Adversarial Influence: Characterizing LLM Latent Spaces with Persistent Homology](https://mlsanity.com/paper/2505.20435/) (2025)
- [Prompt Attacks Reveal Superficial Knowledge Removal in Unlearning Methods](https://mlsanity.com/paper/2506.10236/) (2025)
- [What AI evaluations for preventing catastrophic risks can and cannot do](https://mlsanity.com/paper/2412.08653/) (2024)
- [Adaptively profiling models with task elicitation](https://mlsanity.com/paper/2503.01986/) (2025)
- [Removing Sandbagging in LLMs by Training with Weak Supervision](https://mlsanity.com/paper/2604.22082/) (2026)
- [Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting](https://mlsanity.com/paper/2606.09809/) (2026)
- ["Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms](https://mlsanity.com/paper/2606.12618/) (2026)
- [Evaluating Language Model Reasoning about Confidential Information](https://mlsanity.com/paper/2508.19980/) (2025)
- [Fundamental Limits of Black-Box Safety Evaluation: Information-Theoretic and Computational Barriers from Latent Context Conditioning](https://mlsanity.com/paper/2602.16984/) (2026)
- [Routing Subspaces: Auditing Evaluation-to-Deployment Mismatch in Fine-Tuned Language Models](https://mlsanity.com/paper/2607.20436/) (2026)
- [Layered Unlearning for Adversarial Relearning](https://mlsanity.com/paper/2505.09500/) (2025)
- [Locket: Robust Feature-Locking Technique for Language Models](https://mlsanity.com/paper/2510.12117/) (2025)
- [Mechanistic Anomaly Detection via Functional Attribution](https://mlsanity.com/paper/2604.18970/) (2026)
- [LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs](https://mlsanity.com/paper/2606.06286/) (2026)

## Abstract (arXiv)

To determine the safety of large language models (LLMs), AI developers must be able to assess their dangerous capabilities. But simple prompting strategies often fail to elicit an LLM's full capabilities. One way to elicit capabilities more robustly is to fine-tune the LLM to complete the task. In this paper, we investigate the conditions under which fine-tuning-based elicitation suffices to elicit capabilities. To do this, we introduce password-locked models, LLMs fine-tuned such that some of their capabilities are deliberately hidden. Specifically, these LLMs are trained to exhibit these capabilities only when a password is present in the prompt, and to imitate a much weaker LLM otherwise. Password-locked models enable a novel method of evaluating capabilities elicitation methods, by testing whether these password-locked capabilities can be elicited without using the password. We find that a few high-quality demonstrations are often sufficient to fully elicit password-locked capabilities. More surprisingly, fine-tuning can elicit other capabilities that have been locked using the same password, or even different passwords. Furthermore, when only evaluations, and not demonstrations, are available, approaches like reinforcement learning are still often able to elicit capabilities. Overall, our findings suggest that fine-tuning is an effective method of eliciting hidden capabilities of current models, but may be unreliable when high-quality demonstrations are not available, e.g. as may be the case when models' (hidden) capabilities exceed those of human demonstrators.