mlsanityData to 7 Oct 2026

I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench

arXiv 2401.17882 ↗First posted 31 Jan 2024cs.CLcited by 20

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Do large language models (LLMs) exhibit any forms of awareness similar to humans? In this paper, we introduce AwareBench, a benchmark designed to evaluate awareness in LLMs. Drawing from theories in psychology and philosophy, we define awareness in LLMs as the ability to understand themselves as AI models and to exhibit social intelligence. Subsequently, we categorize awareness in LLMs into five dimensions, including capability, mission, emotion, culture, and perspective. Based on this taxonomy, we create a dataset called AwareEval, which contains binary, multiple-choice, and open-ended questions to assess LLMs' understandings of specific awareness dimensions. Our experiments, conducted on 13 LLMs, reveal that the majority of them struggle to fully recognize their capabilities and missions while demonstrating decent social intelligence. We conclude by connecting awareness of LLMs with AI alignment and safety, emphasizing its significance to the trustworthy and ethical development of LLMs. Our dataset and code are available at https://github.com/HowieHwong/Awareness-in-LLM.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 31 of its 73 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

31 of 73 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Better Zero-Shot Reasoning with Role-Play Prompting15 Aug
  2. 2023Dissociating language and thought in large language models16 Jan

Other references · 29

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Mistral 7Bcited by 3,065
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2022Red Teaming Language Models with Language Modelscited by 983
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  7. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  8. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  9. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  10. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  11. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  12. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
17 more
  1. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  2. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 217
  3. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  4. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  5. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  6. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 143
  7. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  8. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  9. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94
  10. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  11. 2023InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrievalcited by 73
  12. 2023Alignment for Honestycited by 72
  13. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  14. 2018Evaluating Theory of Mind in Question Answeringcited by 64
  15. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  16. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 49
  17. 2018Towards Empathetic Open-domain Conversation Models: a New Benchmark and Datasetcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  3. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  4. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  5. 2024From Imitation to Introspection: Probing Self-Consciousness in Language Modelscited by 14
  6. 2024Imagining and building wise machines: The centrality of AI metacognitioncited by 12
  7. 2025Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awarenesscited by 7
  8. 2025Know Thyself? On the Incapability and Implications of AI Self-Recognitioncited by 7
  9. 2025AI Awarenesscited by 6
  10. 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbationscited by 5
  11. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 3
  12. 2024Self-controller: Controlling LLMs with Multi-round Step-by-step Self-awarenesscited by 2
8 more
  1. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  2. 2026Self-Reference in Large Language Models: The Introspection Threshold for Recursive Self-Improvementcited by 2
  3. 2024The Epistemic Asymmetry of Consciousness Self-Reports: A Formal Analysis of AI Consciousness Denialcited by 1
  4. 2025Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20Bcited by 1
  5. 2025Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Modelscited by 1
  6. 2026Just aware enough: Evaluating awareness across artificial systemscited by 1
  7. 2025Self-Critique-Guided Curiosity Refinement: Enhancing Honesty and Helpfulness in Large Language Models via In-Context Learningcited by 0
  8. 2026Decomposing and Steering Functional Metacognition in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
  2. 2025AIPsychoBench: Understanding the Psychometric Differences between LLMs and Humans
  3. 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications
  4. 2026Understanding Large Language Models
  5. 2023Understanding Social Reasoning in Language Models with Language Models
  6. 2024EgoSocialArena: Benchmarking the Social Intelligence of Large Language Models from a First-person Perspective
  7. 2024Evaluating Cultural and Social Awareness of LLM Web Agents
  8. 2025SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations