Floor, Ceiling, and the Fusion Gap: How Much of Crowd Reading Attention Can Machines Predict?
Not on the map. Why?
Abstract (arXiv)
A benchmark score means nothing without knowing what a trivial method achieves and what the best possible method could achieve. We construct both bounds for a task with a rare kind of ground truth: predicting which sentences a crowd of readers -- highlighting for their own purposes, unpaid, uninstructed, and blind to each other -- marked in 120 web documents. The floor is naive truncation (lead); the ceiling is a split-half oracle: half the crowd predicting the other half. The gap between them is +0.2028 AP [+0.1698, +0.2342, domain-clustered], and three findings structure it. First, the gap is semantic: position and length features recover 5% of it. Second, frontier language models reach 35-53% of it zero-shot -- far above classical baselines, far below the crowd; a state-of-the-art prompt compressor (LLMLingua-2) lands below the floor, indistinguishable from random selection. Third, an unweighted cross-vendor fusion of five frontier rankings plus a position prior reaches 60%, beating the best single model by +0.0159 [+0.0044, +0.0269; Holm p=0.019] -- a gain that survives ablation of its best member, split-half arm selection, prompt paraphrase, and label, gate, and seed perturbations, and was CONFIRMED by a pre-registered replication on 217 independent documents (+0.0179, Holm p=0.042). Finally, the bracket compresses: distilling the fusion into one open-weight 8B student that reads the whole document retains 90% of the fusion's edge and reaches statistical parity with the strongest single frontier model (+0.0070 [-0.0068, +0.0200]), where a local-context student retains only 63% -- the crowd's signal lives in document-level structure, and the cheapest known improvement is to ask several different models and average.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 12 references that are in the map. First posted on arXiv 3 Aug 2026.
Builds on
11 of 12 references.
References · 11
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
- 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
- 2025Great Models Think Alike and this Undermines AI Oversightcited by 53
- 2026Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulnesscited by 8
- 2025Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowdscited by 6
- 2026Personal Salience: Highlighting Is Social, but Individuality Lives in Selectioncited by 5
- 2025The Law of Multi-Model Collaboration: Scaling Limits of Model Ensembling for Large Language Modelscited by 3
- 2026The Long Tail, Not the Front Page: Cold-Start Prediction of Crowd Highlight Saliencecited by 2
- 2026Language Models Agree With Each Other, Not With Readerscited by 1
Built on by
Not cited yet.