# Behavioral Analysis of Information Salience in Large Language Models

arXiv: https://arxiv.org/abs/2502.14613 · HTML: https://mlsanity.com/paper/2502.14613/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 64 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [WiCE: Real-World Entailment for Claims in Wikipedia](https://mlsanity.com/paper/2303.01432/) (2023)
- [Controllable Abstractive Summarization](https://mlsanity.com/paper/1711.05217/) (2017)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM](https://mlsanity.com/paper/2404.12259/) (2024)
- [FABLES: Evaluating faithfulness and content selection in book-length summarization](https://mlsanity.com/paper/2404.01261/) (2024)
- [Goal-Driven Explainable Clustering via Language Descriptions](https://mlsanity.com/paper/2305.13749/) (2023)
- [Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles](https://mlsanity.com/paper/2309.09369/) (2023)
- [Training Dynamics for Text Summarization Models](https://mlsanity.com/paper/2110.08370/) (2021)
- [Explaining Datasets in Words: Statistical Models with Natural Language Parameters](https://mlsanity.com/paper/2409.08466/) (2024)
- [QUDEVAL: The Evaluation of Questions Under Discussion Discourse Parsing](https://mlsanity.com/paper/2310.14520/) (2023)
- [Atomic Inference for NLI with Generated Facts as Atoms](https://mlsanity.com/paper/2305.13214/) (2023)
- [A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents](https://mlsanity.com/paper/2305.14772/) (2023)
- [Elaborative Simplification as Implicit Questions Under Discussion](https://mlsanity.com/paper/2305.10387/) (2023)
- [InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification](https://mlsanity.com/paper/2401.16475/) (2024)

## Built on by (most cited first, 12 of 12)

- [Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectory](https://mlsanity.com/paper/2601.22984/) (2026)
- [Selection, Not Salience: The Shape and Limits of Personalization in Social Highlighting](https://mlsanity.com/paper/2606.10398/) (2026)
- [ARC: Argument Representation and Coverage Analysis for Zero-Shot Long Document Summarization with Instruction Following LLMs](https://mlsanity.com/paper/2505.23654/) (2025)
- [The Long Tail, Not the Front Page: Cold-Start Prediction of Crowd Highlight Salience](https://mlsanity.com/paper/2606.11654/) (2026)
- [When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysis](https://mlsanity.com/paper/2606.29251/) (2026)
- [Measuring Alignment With Reader Highlights Net of Position and Length](https://mlsanity.com/paper/2607.27739/) (2026)
- [Language Models Agree With Each Other, Not With Readers](https://mlsanity.com/paper/2607.29274/) (2026)
- [What makes an entity salient in discourse?](https://mlsanity.com/paper/2508.16464/) (2025)
- [What Matters to an LLM? Behavioral and Computational Evidences from Summarization](https://mlsanity.com/paper/2602.00459/) (2026)
- [Which course? Discourse! Teaching Discourse and Generation in the Era of LLMs](https://mlsanity.com/paper/2602.02878/) (2026)
- [Improving Argument Saliency Coverage in Small LLMs for Long Legal Opinion Summarization via Sequence-Level Distillation](https://mlsanity.com/paper/2608.29884/) (2026)
- [LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs](https://mlsanity.com/paper/2610.02902/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) excel at text summarization, a task that requires models to select content based on its importance. However, the exact notion of salience that LLMs have internalized remains unclear. To bridge this gap, we introduce an explainable framework to systematically derive and investigate information salience in LLMs through their summarization behavior. Using length-controlled summarization as a behavioral probe into the content selection process, and tracing the answerability of Questions Under Discussion throughout, we derive a proxy for how models prioritize information. Our experiments on 13 models across four datasets reveal that LLMs have a nuanced, hierarchical notion of salience, generally consistent across model families and sizes. While models show highly consistent behavior and hence salience patterns, this notion of salience cannot be accessed through introspection, and only weakly correlates with human perceptions of information salience.