# Simulating Meaning, Nevermore! Introducing ICR: A Semiotic-Hermeneutic Metric for Evaluating Meaning in LLM Text Summaries

arXiv: https://arxiv.org/abs/2603.04413 · HTML: https://mlsanity.com/paper/2603.04413/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 66 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language](https://mlsanity.com/paper/2405.12856/) (2024)
- [Generative Artificial Intelligence: A Systematic Review and Applications](https://mlsanity.com/paper/2405.11029/) (2024)
- [Large Language Models are Biased Because They Are Large Language Models](https://mlsanity.com/paper/2406.13138/) (2024)
- [On Robustness and Reliability of Benchmark-Based Evaluation of LLMs](https://mlsanity.com/paper/2509.04013/) (2025)
- [Evaluating LLM Metrics Through Real-World Capabilities](https://mlsanity.com/paper/2505.08253/) (2025)
- [Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence](https://mlsanity.com/paper/2309.14379/) (2023)
- [Potential and Perils of Large Language Models as Judges of Unstructured Textual Data](https://mlsanity.com/paper/2501.08167/) (2025)
- [From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models](https://mlsanity.com/paper/2411.05036/) (2024)
- [Theoretical foundations and limits of word embeddings: what types of meaning can they capture?](https://mlsanity.com/paper/2107.10413/) (2021)
- [Transformers, Contextualism, and Polysemy](https://mlsanity.com/paper/2404.09577/) (2024)
- [Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap](https://mlsanity.com/paper/2512.04680/) (2025)
- [AI and Human Oversight: A Risk-Based Framework for Alignment](https://mlsanity.com/paper/2510.09090/) (2025)
- [The Golden Rule as a Heuristic to Measure the Fairness of Texts Using Machine Learning](https://mlsanity.com/paper/2111.00107/) (2021)
- [What Generative Artificial Intelligence Means for Terminological Definitions](https://mlsanity.com/paper/2402.16139/) (2024)
- [Unbiasing on the Fly: Explanation-Guided Human Oversight of Machine Learning System Decisions](https://mlsanity.com/paper/2406.17906/) (2024)

## Abstract (arXiv)

Meaning in human language is relational, context dependent, and emergent, arising from dynamic systems of signs rather than fixed word-concept mappings. In computational settings, this semiotic and interpretive complexity complicates the generation and evaluation of meaning. This article proposes an interdisciplinary framework for studying meaning in large language model (LLM) generated language by integrating semiotics and hermeneutics with qualitative research methods. We review prior scholarship on meaning and machines, examining how linguistic signs are transformed into vectorized representations in static and contextualized embedding models, and identify gaps between statistical approximation and human interpretive meaning. We then introduce the Inductive Conceptual Rating (ICR) metric, a qualitative evaluation approach grounded in inductive content analysis and reflexive thematic analysis, designed to assess semantic accuracy and meaning alignment in LLM-outputs beyond lexical similarity metrics. We apply ICR in an empirical comparison of LLM generated and human generated thematic summaries across five datasets (N = 50 to 800). While LLMs achieve high linguistic similarity, they underperform on semantic accuracy, particularly in capturing contextually grounded meanings. Performance improves with larger datasets but remains variable across models, potentially reflecting differences in the frequency and coherence of recurring concepts and meanings. We conclude by arguing for evaluation frameworks that leverage systematic qualitative interpretation practices when assessing meaning in LLM-generated outputs from reference texts.