mlsanityData to 7 Oct 2026

Post-Hoc Understanding of Metaphor Processing in Decoder-Only Language Models via Conditional Scale Entropy

arXiv 2605.21391 ↗First posted 20 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Metaphor requires a language model to resolve a token whose contextual meaning diverges from its basic literal sense. Understanding how transformer models organize this reinterpretation across depth remains an open problem in mechanistic interpretability. We introduce conditional scale entropy (CSE), a wavelet-derived measure of how broadly transformer computation engages across frequency scales at each layer position. Two theorems establish that CSE is invariant to update magnitude, isolating the structural pattern of updates from their intensity. Using CSE, we find that metaphorical tokens produce significantly higher spectral breadth than literal tokens at contiguous layer positions on every decoder-only architecture tested, from 124M to 20B parameters (GPT-2 family, LLaMA-2 7B, GPT-oss 20B). The effect survives cluster-based permutation correction, recurs in the early-to-mid relative depth range across models, and converges with an independent analysis of 200 naturalistic VUA pairs. Specificity controls further show that the effect is not explained by semantic complexity or by matched propositional content. These results identify multi-scale coordination as a consistent signature of metaphorical language processing in the decoder-only architectures examined, and establish CSE as a principled tool for characterizing cross-depth structure in transformers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 29 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

15 of 29 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018On the Spectral Bias of Neural Networkscited by 1,321
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  6. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  7. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  8. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  9. 2023A mathematical perspective on Transformerscited by 160
  10. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  11. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 58
  12. 2024Transformer Layers as Painterscited by 50
3 more
  1. 2021MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theoriescited by 46
  2. 2018Neural Metaphor Detection in Contextcited by 34
  3. 2022Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languagescited by 30

Built on by

Not cited yet.