mlsanityData to 7 Oct 2026

Evaluating Language Model Reasoning about Confidential Information

arXiv 2508.19980 ↗First posted 27 Aug 2025cs.LGcited by 3

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

As language models are increasingly deployed as autonomous agents in high-stakes settings, ensuring that they reliably follow user-defined rules has become a critical safety concern. To this end, we study whether language models exhibit contextual robustness, or the capability to adhere to context-dependent safety specifications. For this analysis, we develop a benchmark (PasswordEval) that measures whether language models can correctly determine when a user request is authorized (i.e., with a correct password). We find that current open- and closed-source models struggle with this seemingly simple task, and that, perhaps surprisingly, reasoning capabilities do not generally improve performance. In fact, we find that reasoning traces frequently leak confidential information, which calls into question whether reasoning traces should be exposed to users in such applications. We also scale the difficulty of our evaluation along multiple axes: (i) by adding adversarial user pressure through various jailbreaking strategies, and (ii) through longer multi-turn conversations where password verification is more challenging. Overall, our results suggest that current frontier models are not well-suited to handling confidential information, and that reasoning capabilities may need to be trained in a different manner to make them safer for release in high-stakes settings.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 47 of its 51 references that are in the map. First posted on arXiv 27 Aug 2025.

Builds on

47 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Trading Inference-Time Compute for Adversarial Robustness31 Jan
  2. 2024Deliberative Alignment: Reasoning Enables Safer Language Models20 Dec

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
33 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  8. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  10. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  12. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  13. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  14. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  15. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  16. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  17. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  18. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  19. 2023Pretraining Language Models with Human Preferencescited by 245
  20. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
  21. 2023Effective Prompt Extraction from Language Modelscited by 83
  22. 2023Large Language Models can Learn Rulescited by 66
  23. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 48
  24. 2023Can LLMs Follow Simple Rules?cited by 45
  25. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 44
  26. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 42
  27. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  28. 2024Backtracking Improves Generation Safetycited by 31
  29. 2025A Closer Look at System Prompt Robustnesscited by 31
  30. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  31. 2025Antidistillation Samplingcited by 24
  32. 2024Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Modelscited by 23
  33. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigationcited by 11
  2. 2026When Should We Introduce Safety Interventions During Pretraining?cited by 5
  3. 2026From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselvescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Content Safety: Real-Time Monitoring for Reasoning Vulnerabilities in Large Language Models
  2. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
  3. 2025When Intelligence Fails: An Empirical Study on Why LLMs Struggle with Password Cracking
  4. 2025Weakest Link in the Chain: Security Vulnerabilities in Advanced Reasoning Models
  5. 2025SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Models
  6. 2025Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMs
  7. 2026Control-Token Injection Suppresses Chain-of-Thought and Defeats Reasoning-Based Oversight in Tool-Using Agents
  8. 2025MetaSC: Test-Time Safety Specification Optimization for Language Models