mlsanityData to 7 Oct 2026

Floor, Ceiling, and the Fusion Gap: How Much of Crowd Reading Attention Can Machines Predict?

arXiv 2608.01704 ↗First posted 3 Aug 2026cs.IR cs.CL cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

A benchmark score means nothing without knowing what a trivial method achieves and what the best possible method could achieve. We construct both bounds for a task with a rare kind of ground truth: predicting which sentences a crowd of readers -- highlighting for their own purposes, unpaid, uninstructed, and blind to each other -- marked in 120 web documents. The floor is naive truncation (lead); the ceiling is a split-half oracle: half the crowd predicting the other half. The gap between them is +0.2028 AP [+0.1698, +0.2342, domain-clustered], and three findings structure it. First, the gap is semantic: position and length features recover 5% of it. Second, frontier language models reach 35-53% of it zero-shot -- far above classical baselines, far below the crowd; a state-of-the-art prompt compressor (LLMLingua-2) lands below the floor, indistinguishable from random selection. Third, an unweighted cross-vendor fusion of five frontier rankings plus a position prior reaches 60%, beating the best single model by +0.0159 [+0.0044, +0.0269; Holm p=0.019] -- a gain that survives ablation of its best member, split-half arm selection, prompt paraphrase, and label, gate, and seed perturbations, and was CONFIRMED by a pre-registered replication on 217 independent documents (+0.0179, Holm p=0.042). Finally, the bracket compresses: distilling the fusion into one open-weight 8B student that reads the whole document retains 90% of the fusion's edge and reaches statistical parity with the strongest single frontier model (+0.0070 [-0.0068, +0.0200]), where a local-context student retains only 63% -- the crowd's signal lives in document-level structure, and the cheapest known improvement is to ask several different models and average.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 12 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

11 of 12 references.

References · 11

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  4. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  5. 2025Great Models Think Alike and this Undermines AI Oversightcited by 53
  6. 2026Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulnesscited by 8
  7. 2025Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowdscited by 6
  8. 2026Personal Salience: Highlighting Is Social, but Individuality Lives in Selectioncited by 5
  9. 2025The Law of Multi-Model Collaboration: Scaling Limits of Model Ensembling for Large Language Modelscited by 3
  10. 2026The Long Tail, Not the Front Page: Cold-Start Prediction of Crowd Highlight Saliencecited by 2
  11. 2026Language Models Agree With Each Other, Not With Readerscited by 1

Built on by

Not cited yet.