# Topic Identification in LLM Input-Output Pairs through the Lens of Information Bottleneck

arXiv: https://arxiv.org/abs/2509.03533 · HTML: https://mlsanity.com/paper/2509.03533/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 14 references that are in the map.
- First posted on arXiv 26 Aug 2025.

## Builds on (most cited first, 7 of 7)

- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [The deterministic information bottleneck](https://mlsanity.com/paper/1604.00268/) (2016)
- [Revisiting LLM Reasoning via Information Bottleneck](https://mlsanity.com/paper/2507.18391/) (2025)
- [A comprehensive taxonomy of hallucinations in Large Language Models](https://mlsanity.com/paper/2508.01781/) (2025)
- [Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory](https://mlsanity.com/paper/2501.00999/) (2025)
- [The information bottleneck and geometric clustering](https://mlsanity.com/paper/1712.09657/) (2017)

## Built on by (most cited first, 1 of 1)

- [Semantic Faithfulness and Entropy Production Measures to Tame Your LLM Demons and Manage Hallucinations](https://mlsanity.com/paper/2512.05156/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are prone to critical failure modes, including textitintrinsic faithfulness hallucinations (also known as confabulations), where a response deviates semantically from the provided context. Frameworks designed to detect this, such as Semantic Divergence Metrics (SDM), rely on identifying latent topics shared between prompts and responses, typically by applying geometric clustering to their sentence embeddings. This creates a disconnect, as the topics are optimized for spatial proximity, not for the downstream information-theoretic analysis. In this paper, we bridge this gap by developing a principled topic identification method grounded in the Deterministic Information Bottleneck (DIB) for geometric clustering. Our key contribution is to transform the DIB method into a practical algorithm for high-dimensional data by substituting its intractable KL divergence term with a computationally efficient upper bound. The resulting method, which we dub UDIB, can be interpreted as an entropy-regularized and robustified version of K-means that inherently favors a parsimonious number of informative clusters. By applying UDIB to the joint clustering of LLM prompt and response embeddings, we generate a shared topic representation that is not merely spatially coherent but is fundamentally structured to be maximally informative about the prompt-response relationship. This provides a superior foundation for the SDM framework and offers a novel, more sensitive tool for detecting confabulations.