# Measuring Alignment With Reader Highlights Net of Position and Length

arXiv: https://arxiv.org/abs/2607.27739 · HTML: https://mlsanity.com/paper/2607.27739/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 30 Jul 2026.

## Builds on (most cited first, 13 of 13)

- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [Content Selection in Deep Learning Models of Summarization](https://mlsanity.com/paper/1810.12343/) (2018)
- [Searching for Effective Neural Extractive Summarization: What Works and What's Next](https://mlsanity.com/paper/1907.03491/) (2019)
- [Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses](https://mlsanity.com/paper/1909.04028/) (2019)
- [Relative Importance in Sentence Processing](https://mlsanity.com/paper/2106.03471/) (2021)
- [HighRES: Highlight-based Reference-less Evaluation of Summarization](https://mlsanity.com/paper/1906.01361/) (2019)
- [Behavioral Analysis of Information Salience in Large Language Models](https://mlsanity.com/paper/2502.14613/) (2025)
- [LAST at CMCL 2021 Shared Task: Predicting Gaze Data During Reading with a Gradient Boosting Decision Tree Approach](https://mlsanity.com/paper/2104.13043/) (2021)
- [Selection, Not Salience: The Shape and Limits of Personalization in Social Highlighting](https://mlsanity.com/paper/2606.10398/) (2026)
- ["You are no Jack Kennedy": On Media Selection of Highlights from Presidential Debates](https://mlsanity.com/paper/1802.08690/) (2018)

## Built on by (most cited first, 2 of 2)

- [Language Models Agree With Each Other, Not With Readers](https://mlsanity.com/paper/2607.29274/) (2026)
- [More Reader Data or a Better Teacher? Updating a Model of Crowd Highlights in Late 2026](https://mlsanity.com/paper/2610.17768/) (2026)

## Abstract (arXiv)

Context compression discards most of a document before a language model reads it, and is normally evaluated by downstream task accuracy - which makes another model the judge of what mattered. Naturalistic social highlighting offers a non-circular reference: many people independently marking passages on the same page. But the obvious metric, the fraction of crowd-marked sentences a compressor keeps, is confounded twice: crowd marks are front-loaded and crowd-marked sentences are longer, so any method favouring early or long sentences scores well regardless of readers. We remove both by matching each marked sentence against unmarked sentences of the same document at equal relative depth and equal within-document length rank, and we calibrate every estimator on synthetic nulls built from position and length alone - a step that matters, since depth-only stratification returns a false positive on 20-36% of nulls containing no effect. On 120 web documents (at least 12 independent readers each), a language-model importance ranking keeps 38.4% of crowd-marked sentences against 19.9% of their matched neighbours: an enrichment of +0.196 [+0.148, +0.239], at p = 0.0005 under an exact randomization test that assumes nothing about clustering, and replicated cross-vendor. Naive truncation, whose keep rule is position, correctly falls to +0.003. To give the number a scale: scored identically, on the same budget, against a crowd label recomputed to exclude them, a single human reader reaches +0.182 - indistinguishable from GPT-5.4 (+0.002 [-0.081, +0.088]) and below Claude Opus 5. Classical methods are not null - Luhn's 1958 heuristic reaches +0.088 - so reader selection is partly recoverable by counting words; conditioning additionally on lexical centrality removes only 0.010, so the agreement is not centrality. We also report that a claim in our own prior work does not reproduce on this corpus.