Split the Labor: Separating Evidence Interpretation from Decision Aggregation
Not on the map. Why?
Abstract (arXiv)
Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt. This conflates two operations with different requirements. Interpreting a source rewards capacity and context. Combining interpretations rewards fixed arithmetic, comparability across instances, and the option to return nothing. Once separated, the design problem becomes the interface between them. We propose a four-field evidence tuple (hypothesis, reliability bucket, rationale, provenance) and show that fixing it determines both halves. The separation also reveals a failure mode in how such systems combine, which we call count-scale drift. Thresholding a sum of unnormalized weights is exactly posterior thresholding, but at an operating point that slides with the number of sources consulted. The slide grows with reader reliability. When source reliabilities differ, the vote rule and the posterior order instances differently, and no threshold reconciles them. Pooling calibrated log-likelihood ratios addresses both problems. The fix is arithmetic rather than architectural, and applies to a class of rules beyond language models: score-summing triage engines, diagnostic panels scored by counting positives, and additive multi-signal detectors. We then instantiate the principle twice on one longitudinal corpus, once after outcomes resolve and once before. The same partition helps in both, at different granularities: over reading in the first, over learning capacity in the second. There, a small sequence encoder on an easy auxiliary objective plus a tree ensemble carrying the censored survival loss reaches 0.921 AUPRC against 0.805 for a hand-crafted baseline. We separate what transfers from what must be re-estimated per domain, and state five predictions that would falsify the framework, three negative results, and which comparisons remain confounded.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 14 Aug 2026.
Builds on
19 of 40 references.
References · 19
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
7 more
- 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
- 2023Enabling Large Language Models to Generate Text with Citationscited by 595
- 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
- 2021Tabular Data: Deep Learning is Not All You Needcited by 535
- 2016DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Networkcited by 358
- 2021Measuring Attribution in Natural Language Generation Modelscited by 256
- 2017Machine Learning for Survival Analysis: A Surveycited by 112
Built on by
Citing papers per year: 2026: 2.
2 papers.