mlsanityData to 14 Oct 2026

Taken out of context: On measuring situational awareness in LLMs

arXiv 2309.00667 ↗First posted 1 Sep 2023cs.CL cs.LGcited by 143

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We aim to better understand the emergence of `situational awareness' in large language models (LLMs). A model is situationally aware if it's aware that it's a model and can recognize whether it's currently in testing or deployment. Today's LLMs are tested for safety and alignment before they are deployed. An LLM could exploit situational awareness to achieve a high score on safety tests, while taking harmful actions after deployment. Situational awareness may emerge unexpectedly as a byproduct of model scaling. One way to better foresee this emergence is to run scaling experiments on abilities necessary for situational awareness. As such an ability, we propose `out-of-context reasoning' (in contrast to in-context learning). We study out-of-context reasoning experimentally. First, we finetune an LLM on a description of a test while providing no examples or demonstrations. At test time, we assess whether the model can pass the test. To our surprise, we find that LLMs succeed on this out-of-context reasoning task. Their success is sensitive to the training setup and only works when we apply data augmentation. For both GPT-3 and LLaMA-1, performance improves with model size. These findings offer a foundation for further empirical study, towards predicting and potentially controlling the emergence of situational awareness in LLMs. Code is available at: https://github.com/AsaCooperStickland/situational-awareness-evals.

Key facts. Cited by 143 papers in the map (data through 14 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 1 Sep 2023.

Builds on

42 of 63 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
30 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2019Language Models as Knowledge Bases?cited by 1,936
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  5. 2022Red Teaming Language Models with Language Modelscited by 1,026
  6. 2022Mass-Editing Memory in a Transformercited by 1,001
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  8. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  9. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  10. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  12. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  13. 2021Editing Factual Knowledge in Language Modelscited by 553
  14. 2021Fast Model Editing at Scalecited by 498
  15. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  16. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  17. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  18. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  19. 2020Modifying Memories in Transformer Modelscited by 224
  20. 2023An Overview of Catastrophic AI Riskscited by 222
  21. 2022Predictability and Surprise in Large Generative Modelscited by 210
  22. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  23. 2023Model evaluation for extreme riskscited by 177
  24. 2021Truthful AI: Developing and governing AI that does not liecited by 113
  25. 2022Is Power-Seeking AI an Existential Risk?cited by 105
  26. 2023Eight Things to Know about Large Language Modelscited by 73
  27. 2022Kformer: Knowledge Injection in Transformer Feed-Forward Layerscited by 37
  28. 2020Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweightcited by 34
  29. 2023TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AIcited by 26
  30. 2019Further advantages of data augmentation on convolutional neural networkscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 27; 2025: 41; 2026: 60.

143 papers.

Sort:

By area:

  1. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  2. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  3. 2024Alignment faking in large language modelscited by 360
  4. 2024Frontier Models are Capable of In-context Schemingcited by 287
  5. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  6. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  7. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  8. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  9. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 158
  10. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  11. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  12. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 113
38 more
  1. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  2. 2025Persona Features Control Emergent Misalignmentcited by 86
  3. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 84
  4. 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
  5. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 72
  6. 2024Large Language Model Safety: A Holistic Surveycited by 59
  7. 2024Reverse Training to Nurse the Reversal Cursecited by 49
  8. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 47
  9. 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 45
  10. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  11. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  12. 2024Moral Alignment for LLM Agentscited by 39
  13. 2025A sketch of an AI control safety casecited by 39
  14. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  15. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 35
  16. 2024Towards evaluations-based safety cases for AI schemingcited by 33
  17. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 28
  18. 2023Compositional preference models for aligning LMscited by 26
  19. 2026Reasoning Models Struggle to Control their Chains of Thoughtcited by 25
  20. 2024LLMs Are Prone to Fallacies in Causal Inferencecited by 23
  21. 2026Emergent Misalignment is Easy, Narrow Misalignment is Hardcited by 23
  22. 2023Towards Evaluating AI Systems for Moral Status Using Self-Reportscited by 22
  23. 2024I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBenchcited by 21
  24. 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligencecited by 20
  25. 2024HonestLLM: Toward an Honest and Helpful Large Language Modelcited by 18
  26. 2024Lessons from Studying Two-Hop Latent Reasoningcited by 17
  27. 2025Position: Privacy Is Not Just Memorization!cited by 17
  28. 2024Understanding the Learning Dynamics of Alignment with Human Feedbackcited by 15
  29. 2024Self-Cognition in Large Language Models: An Exploratory Studycited by 15
  30. 2024Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMscited by 15
  31. 2025Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiencescited by 15
  32. 2024From Imitation to Introspection: Probing Self-Consciousness in Language Modelscited by 14
  33. 2024Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Factscited by 14
  34. 2023In-Context Learning Dynamics with Random Binary Sequencescited by 13
  35. 2026Shaping capabilities with token-level data filteringcited by 13
  36. 2026Subliminal Learning Is Steering Vector Distillationcited by 13
  37. 2025Do Large Language Models Know How Much They Know?cited by 11
  38. 2023Implicit meta-learning may lead language models to trust more reliable sourcescited by 10

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking the Unsolvable: When In-Context Search Meets Test-Time Scaling
  2. 2026Reasoning Shift: How Context Silently Shortens LLM Reasoning
  3. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  4. 2026Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
  5. 2024Do Large Language Models Understand Logic or Just Mimick Context?
  6. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Models
  7. 2025Aware First, Think Less: Dynamic Boundary Self-Awareness Drives Extreme Reasoning Efficiency in Large Language Models
  8. 2025Out-of-Context Reasoning in Large Language Models