# Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth

arXiv: https://arxiv.org/abs/2509.03867 · HTML: https://mlsanity.com/paper/2509.03867/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 57 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 30 of 30)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Evaluating Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2302.02083/) (2023)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores](https://mlsanity.com/paper/2311.09766/) (2023)
- [Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning](https://mlsanity.com/paper/2312.05230/) (2023)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [Sarcasm Detection using Hybrid Neural Network](https://mlsanity.com/paper/1908.07414/) (2019)
- [Enhancing Dialogue Generation via Dynamic Graph Knowledge Aggregation](https://mlsanity.com/paper/2306.16195/) (2023)
- [The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation](https://mlsanity.com/paper/2311.05552/) (2023)
- [Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions](https://mlsanity.com/paper/2405.19088/) (2024)
- [From Facts to Insights: A Study on the Generation and Evaluation of Analytical Reports for Deciphering Earnings Calls](https://mlsanity.com/paper/2410.01039/) (2024)
- [Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes](https://mlsanity.com/paper/2507.13335/) (2025)
- [FanChuan: A Multilingual and Graph-Structured Benchmark For Parody Detection and Analysis](https://mlsanity.com/paper/2502.16503/) (2025)
- [Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts](https://mlsanity.com/paper/2504.21117/) (2025)
- [Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks](https://mlsanity.com/paper/2501.02654/) (2025)
- [Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal](https://mlsanity.com/paper/2507.21750/) (2025)

## Built on by (most cited first, 4 of 4)

- [Who's Laughing Now? An Overview of Computational Humour Generation and Explanation](https://mlsanity.com/paper/2509.21175/) (2025)
- [Finding Sense in Nonsense with Generated Contexts: Perspectives from Humans and Language Models](https://mlsanity.com/paper/2602.11699/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [Reading Between the Frames of Multimodal Drivelology](https://mlsanity.com/paper/2608.04939/) (2026)

## Abstract (arXiv)

We introduce Drivelology, a unique linguistic phenomenon characterised as "nonsense with depth" - utterances that are syntactically coherent yet pragmatically paradoxical, emotionally loaded, or rhetorically subversive. While such expressions may resemble surface-level nonsense, they encode implicit meaning requiring contextual inference, moral reasoning, or emotional interpretation. We find that current large language models (LLMs), despite excelling at many natural language processing (NLP) tasks, consistently fail to grasp the layered semantics of Drivelological text. To investigate this, we construct a benchmark dataset of over 1,200+ meticulously curated and diverse examples across English, Mandarin, Spanish, French, Japanese, and Korean. Each example underwent careful expert review to verify its Drivelological characteristics, involving multiple rounds of discussion and adjudication to address disagreements. Using this dataset, we evaluate a range of LLMs on classification, generation, and reasoning tasks. Our results reveal clear limitations of LLMs: models often confuse Drivelology with shallow nonsense, produce incoherent justifications, or miss implied rhetorical functions altogether. These findings highlight a deep representational gap in LLMs' pragmatic understanding and challenge the assumption that statistical fluency implies cognitive comprehension. We release our dataset and code to facilitate further research in modelling linguistic depth beyond surface-level coherence.