LLMs learn scientific taste from institutional traces across the social sciences
Not on the map. Why?
Abstract (arXiv)
Reinforcement-learned reasoning has powered recent AI leaps on verifiable tasks, including mathematics, code, and structure prediction. The harder bottleneck is evaluative judgment in low-verifiability domains, where no oracle anchors reward and the core question is which untested ideas deserve attention. We test whether institutional traces, the record of what fields published, where, and at which tier, can serve as a training signal for AI evaluators. Across eight social science disciplines (psychology, economics, communication, sociology, political science, management, business and finance, public administration), we built held-out four-tier research-pitch benchmarks and supervised-fine-tuned (SFT) LLMs on field-specific publication outcomes. The fine-tuned models cleared the 25 percent chance baseline and exceeded frontier-model performance by wide margins, with best single-model accuracy ranging from 55.0 percent in public administration to 85.5 percent in psychology. In management, evaluated against 48 expert gatekeepers, 174 junior researchers, and 11 frontier reasoning models, the best single fine-tuned model (Qwen3-4B) reached 59.2 percent, 17.6 percentage points above expert majority vote (41.6 percent, non-tied) and 28.1 percentage points above the frontier mean (31.1 percent). The fine-tuned models also showed calibrated confidence: confidence rose when predictions were correct and fell when wrong, mirroring how a skilled reviewer can say "I'm sure" versus "I'm guessing." Selective triage on this signal reached very high accuracy on the highest-confidence subsets in every field. Institutional traces, we conclude, encode a scalable training signal for the low-verifiability judgment on which science depends.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Mar 2026.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation
- 2026Contemporary AI lacks the imagination to diverge or negate in science
- 2025Can AI Validate Science? Benchmarking LLMs for Accurate Scientific Claim rightarrow Evidence Reasoning
- 2025The simulation of judgment in LLMs
- 2025Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMs
- 2026Diagnostic Foundation for Evaluating LLMs' Research Integrity as Co-Scientists
- 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
- 2026LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability