# ExpressivityBench: Can LLMs Communicate Implicitly?

arXiv: https://arxiv.org/abs/2411.08010 · HTML: https://mlsanity.com/paper/2411.08010/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 62 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://mlsanity.com/paper/2306.04181/) (2023)
- [A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations](https://mlsanity.com/paper/2407.04069/) (2024)
- [FLUTE: Figurative Language Understanding through Textual Explanations](https://mlsanity.com/paper/2205.12404/) (2022)
- [COLLIE: Systematic Construction of Constrained Text Generation Tasks](https://mlsanity.com/paper/2307.08689/) (2023)
- [Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark](https://mlsanity.com/paper/2305.14938/) (2023)
- [Do LLMs write like humans? Variation in grammatical and rhetorical styles](https://mlsanity.com/paper/2410.16107/) (2024)
- [PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities](https://mlsanity.com/paper/2401.07078/) (2024)
- ["We Need Structured Output": Towards User-centered Constraints on Large Language Model Output](https://mlsanity.com/paper/2404.07362/) (2024)
- [Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate](https://mlsanity.com/paper/2401.16788/) (2024)
- [Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions](https://mlsanity.com/paper/2403.01222/) (2024)
- [Can Large Language Models Understand Context?](https://mlsanity.com/paper/2402.00858/) (2024)
- [Why is constrained neural language generation particularly challenging?](https://mlsanity.com/paper/2206.05395/) (2022)
- [Ten Social Dimensions of Conversations and Relationships](https://mlsanity.com/paper/2001.09954/) (2020)
- [The Call for Socially Aware Language Technologies](https://mlsanity.com/paper/2405.02411/) (2024)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE](https://mlsanity.com/paper/2210.16407/) (2022)
- [Qbias -- A Dataset on Media Bias in Search Queries and Query Suggestions](https://mlsanity.com/paper/2311.17780/) (2023)
- [Competence-Based Analysis of Language Models](https://mlsanity.com/paper/2303.00333/) (2023)

## Built on by (most cited first, 2 of 2)

- [From Guessing to Asking: An Approach to Resolving the Persona Knowledge Gap in LLMs during Multi-Turn Conversations](https://mlsanity.com/paper/2503.12556/) (2025)
- [Explicit vs. Implicit Biographies: Evaluating and Adapting LLM Information Extraction on Wikidata-Derived Texts](https://mlsanity.com/paper/2509.14943/) (2025)

## Abstract (arXiv)

Human communication is often implicit, conveying tone, identity, and intent beyond literal meanings. While large language models have achieved strong performance on explicit tasks such as summarization and reasoning, their capacity for expressivity, or implicit communication, remains underexplored. We introduce ExpressivityBench, a framework for evaluating the expressivity of LLMs using information-theoretic communication models. Our approach quantifies how well LLM-generated text communicates target properties without explicit mention, across nine tasks spanning emotion, identity, and tone. To enable scalable and reproducible evaluation, we employ LLM-based graders validated against human judgments. Our results reveal that while models are adept at expressing affective content, they struggle with sociolinguistic signals, lagging behind human baselines. This study provides a necessary step to evaluate human-like implicit communication, with implications for applications such as education, mental health support, and socially-aware dialogue systems. We provide code and data for our benchmark alongside our paper.