# Beneath the Surface: Investigating LLMs' Capabilities for Communicating with Subtext

arXiv: https://arxiv.org/abs/2604.05273 · HTML: https://mlsanity.com/paper/2604.05273/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Builds on (most cited first, 30 of 34)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Large Language Models can Strategically Deceive their Users when Put Under Pressure](https://mlsanity.com/paper/2311.07590/) (2023)
- [SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents](https://mlsanity.com/paper/2403.08715/) (2024)
- [EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models](https://mlsanity.com/paper/2312.06281/) (2023)
- [Are Large Language Models Capable of Generating Human-Level Narratives?](https://mlsanity.com/paper/2407.13248/) (2024)
- [LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Models](https://mlsanity.com/paper/2310.03903/) (2023)
- [MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration](https://mlsanity.com/paper/2311.08562/) (2023)
- [Agents' Room: Narrative Generation through Multi-step Collaboration](https://mlsanity.com/paper/2410.02603/) (2024)
- [SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs](https://mlsanity.com/paper/2410.13648/) (2024)
- [AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text](https://mlsanity.com/paper/2410.04265/) (2024)
- [Position: Theory of Mind Benchmarks are Broken for Large Language Models](https://mlsanity.com/paper/2412.19726/) (2024)
- [Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games](https://mlsanity.com/paper/2312.00746/) (2023)
- [Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf](https://mlsanity.com/paper/2405.19946/) (2024)
- [Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection](https://mlsanity.com/paper/2504.11900/) (2025)
- [Re-evaluating Theory of Mind evaluation in large language models](https://mlsanity.com/paper/2502.21098/) (2025)
- [Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers](https://mlsanity.com/paper/2403.01061/) (2024)
- [Generative Text Steganography with Large Language Model](https://mlsanity.com/paper/2404.10229/) (2024)
- [SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agents](https://mlsanity.com/paper/2502.15538/) (2025)
- [from Benign import Toxic: Jailbreaking the Language Model via Adversarial Metaphors](https://mlsanity.com/paper/2503.00038/) (2025)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [Measuring Psychological Depth in Language Models](https://mlsanity.com/paper/2406.12680/) (2024)
- [On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts](https://mlsanity.com/paper/2509.06952/) (2025)

## Abstract (arXiv)

Human communication is fundamentally creative, and often makes use of subtext -- implied meaning that goes beyond the literal content of the text. Here, we systematically study whether language models can use subtext in communicative settings, and introduce four new evaluation suites to assess these capabilities. Our evaluation settings range from writing & interpreting allegories to playing multi-agent and multi-modal games inspired by the rules of board games like Dixit. We find that frontier models generally exhibit a strong bias towards overly literal, explicit communication, and thereby fail to account for nuanced constraints -- even the best performing models generate literal clues 60% of times in one of our environments -- Visual Allusions. However, we find that some models can sometimes make use of common ground with another party to help them communicate with subtext, achieving 30%-50% reduction in overly literal clues; but they struggle at inferring presence of a common ground when not explicitly stated. For allegory understanding, we find paratextual and persona conditions to significantly shift the interpretation of subtext. Overall, our work provides quantifiable measures for an inherently complex and subjective phenomenon like subtext and reveals many weaknesses and idiosyncrasies of current LLMs. We hope this research to inspire future work towards socially grounded creative communication and reasoning.