mlsanityData to 14 Oct 2026

AwarenessBench: Assessing Cognitive Capabilities of Language Models

arXiv 2609.35409 ↗First posted 28 Sep 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

As language models (LMs) exhibit increasingly consciousness-like behaviors, evaluating their cognitive abilities becomes essential. We introduce AwarenessBench, the first comprehensive benchmark for assessing the cognitive abilities of LMs in four dimensions: metacognition, self-awareness, social awareness, and situational awareness, covering 15 cognitive functions and 14,381 samples. Evaluating 18 state-of-the-art LMs, we find that all consistently surpass random baselines, with more advanced models performing better. We further compare LMs with human performance across three demographic groups, where the best-performing model surpasses human averages overall, but most still fall markedly short in metacognition and self-awareness. Finally, we show that awareness is a distinct capability: progress in language modeling or reasoning does not necessarily translate into improved cognition.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 138 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

50 of 138 references.

References · 50

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  6. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  7. 2024DeepSeek-V3 Technical Reportcited by 811
  8. 2025Humanity's Last Examcited by 491
  9. 2024Alignment faking in large language modelscited by 360
  10. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  11. 2024Frontier Models are Capable of In-context Schemingcited by 287
  12. 2023Do Large Language Models Know What They Don't Know?cited by 243
38 more
  1. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  2. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  3. 2023Theory of Mind for Multi-Agent Collaboration via Large Language Modelscited by 135
  4. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  5. 2023Consciousness in Artificial Intelligence: Insights from the Science of Consciousnesscited by 104
  6. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
  7. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  8. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 72
  9. 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Modelscited by 63
  10. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Modelscited by 62
  11. 2023Dynamic Planning with a LLMcited by 61
  12. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 59
  13. 2024Evaluating Cultural and Social Awareness of LLM Web Agentscited by 45
  14. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 42
  15. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  16. 2019SocialIQA: Commonsense Reasoning about Social Interactionscited by 37
  17. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  18. 2024Centaur: a foundation model of human cognitioncited by 36
  19. 2024Causal Inference with Large Language Model: A Surveycited by 32
  20. 2024Taking AI Welfare Seriouslycited by 31
  21. 2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmarkcited by 28
  22. 2024Self-rationalization improves LLM as a fine-grained judgecited by 27
  23. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  24. 2024LLMs Simulate Big Five Personality Traits: Further Evidencecited by 22
  25. 2024I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBenchcited by 21
  26. 2024LLM Theory of Mind and Alignment: Opportunities and Riskscited by 18
  27. 2024Self-Recognition in Language Modelscited by 18
  28. 2022On the link between conscious function and general intelligence in humans and machinescited by 16
  29. 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 16
  30. 2023Evaluating statistical language models as pragmatic reasonerscited by 15
  31. 2024The Call for Socially Aware Language Technologiescited by 13
  32. 2024GPT-4 is judged more human than humans in displaced and inverted Turing testscited by 8
  33. 2025Pragmatic Reasoning improves LLM Code Generationcited by 7
  34. 2025AI Awarenesscited by 7
  35. 2024"A Woman is More Culturally Knowledgeable than A Man?": The Effect of Personas on Cultural Norm Interpretation in LLMscited by 5
  36. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 5
  37. 2025Large language model-powered AI systems achieve self-replication with no human interventioncited by 4
  38. 2025AICrypto: Evaluating Cryptography Capabilities of Large Language Modelscited by 4

Built on by

Not cited yet.