# Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs

arXiv: https://arxiv.org/abs/2407.04694 · HTML: https://mlsanity.com/paper/2407.04694/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2024.

## Built on by (most cited first, 30 of 84)

- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Natural Emergent Misalignment from Reward Hacking in Production RL](https://mlsanity.com/paper/2511.18397/) (2025)
- [Model Organisms for Emergent Misalignment](https://mlsanity.com/paper/2506.11613/) (2025)
- [Convergent Linear Representations of Emergent Misalignment](https://mlsanity.com/paper/2506.11618/) (2025)
- [Aligning LLMs with Individual Preferences via Interaction](https://mlsanity.com/paper/2410.03642/) (2024)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [A sketch of an AI control safety case](https://mlsanity.com/paper/2501.17315/) (2025)
- [Evaluating Frontier Models for Stealth and Situational Awareness](https://mlsanity.com/paper/2505.01420/) (2025)
- [Probing and Steering Evaluation Awareness of Language Models](https://mlsanity.com/paper/2507.01786/) (2025)
- [Towards evaluations-based safety cases for AI scheming](https://mlsanity.com/paper/2411.03336/) (2024)
- [Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions](https://mlsanity.com/paper/2505.17120/) (2025)
- [Does It Make Sense to Speak of Introspection in Large Language Models?](https://mlsanity.com/paper/2506.05068/) (2025)
- [How to evaluate control measures for LLM agents? A trajectory from today to superintelligence](https://mlsanity.com/paper/2504.05259/) (2025)
- [Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs](https://mlsanity.com/paper/2601.11369/) (2026)
- [Search-Time Data Contamination](https://mlsanity.com/paper/2508.13180/) (2025)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Compromising Honesty and Harmlessness in Language Models via Deception Attacks](https://mlsanity.com/paper/2502.08301/) (2025)
- [Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols?](https://mlsanity.com/paper/2412.12480/) (2024)
- [From Imitation to Introspection: Probing Self-Consciousness in Language Models](https://mlsanity.com/paper/2410.18819/) (2024)
- [Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems](https://mlsanity.com/paper/2505.17815/) (2025)
- [Evidence for Limited Metacognition in LLMs](https://mlsanity.com/paper/2509.21545/) (2025)
- [EgoSocialArena: Benchmarking the Social Intelligence of Large Language Models from a First-person Perspective](https://mlsanity.com/paper/2410.06195/) (2024)
- [Do Large Language Models Know What They Are Capable Of?](https://mlsanity.com/paper/2512.24661/) (2025)
- [Agentic Knowledgeable Self-awareness](https://mlsanity.com/paper/2504.03553/) (2025)
- [Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases](https://mlsanity.com/paper/2509.04373/) (2025)
- [AI Awareness](https://mlsanity.com/paper/2504.20084/) (2025)
- [Noticing the Watcher: LLM Agents Can Infer CoT Monitoring from Blocking Feedback](https://mlsanity.com/paper/2603.16928/) (2026)
- [Evaluation Awareness in Language Models Has Limited Effect on Behaviour](https://mlsanity.com/paper/2605.05835/) (2026)

## Abstract (arXiv)

AI assistants such as ChatGPT are trained to respond to users by saying, "I am a large language model". This raises questions. Do such models know that they are LLMs and reliably act on this knowledge? Are they aware of their current circumstances, such as being deployed to the public? We refer to a model's knowledge of itself and its circumstances as situational awareness. To quantify situational awareness in LLMs, we introduce a range of behavioral tests, based on question answering and instruction following. These tests form the Situational Awareness Dataset (SAD), a benchmark comprising 7 task categories and over 13,000 questions. The benchmark tests numerous abilities, including the capacity of LLMs to (i) recognize their own generated text, (ii) predict their own behavior, (iii) determine whether a prompt is from internal evaluation or real-world deployment, and (iv) follow instructions that depend on self-knowledge. We evaluate 16 LLMs on SAD, including both base (pretrained) and chat models. While all models perform better than chance, even the highest-scoring model (Claude 3 Opus) is far from a human baseline on certain tasks. We also observe that performance on SAD is only partially predicted by metrics of general knowledge (e.g. MMLU). Chat models, which are finetuned to serve as AI assistants, outperform their corresponding base models on SAD but not on general knowledge tasks. The purpose of SAD is to facilitate scientific understanding of situational awareness in LLMs by breaking it down into quantitative abilities. Situational awareness is important because it enhances a model's capacity for autonomous planning and action. While this has potential benefits for automation, it also introduces novel risks related to AI safety and control. Code and latest results available at https://situational-awareness-dataset.org .