# When benchmark inferences do not compose: Projectibility in AI evaluation

arXiv: https://arxiv.org/abs/2607.26159 · HTML: https://mlsanity.com/paper/2607.26159/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 51 references that are in the map.
- First posted on arXiv 28 Jul 2026.

## Builds on (most cited first, 18 of 18)

- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [External Validity: From Do-Calculus to Transportability Across Populations](https://mlsanity.com/paper/1503.01603/) (2015)
- [Measurement to Meaning: A Validity-Centered Framework for AI Evaluation](https://mlsanity.com/paper/2505.10573/) (2025)
- [ECBD: Evidence-Centered Benchmark Design for NLP](https://mlsanity.com/paper/2406.08723/) (2024)
- [Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?](https://mlsanity.com/paper/2310.11616/) (2023)
- [A Definition of AGI](https://mlsanity.com/paper/2510.18212/) (2025)
- [The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models](https://mlsanity.com/paper/2510.23191/) (2025)
- [Towards a Benchmark for Scientific Understanding in Humans and Machines](https://mlsanity.com/paper/2304.10327/) (2023)
- [Do Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Morality](https://mlsanity.com/paper/2510.11254/) (2025)
- [Improving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Framework](https://mlsanity.com/paper/2406.10366/) (2024)
- [Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networks](https://mlsanity.com/paper/2603.15121/) (2026)
- [The Rise and Fall of G in AGI](https://mlsanity.com/paper/2604.09911/) (2026)
- [The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context](https://mlsanity.com/paper/2607.12963/) (2026)
- [Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control](https://mlsanity.com/paper/2607.01153/) (2026)

## Abstract (arXiv)

An AI benchmark result rarely reaches a consequential claim in one step. Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site, and combine it with assumptions about human review and downstream consequences. Validity-centred approaches require evidence for each claim. This paper makes explicit and operationalizes a problem those approaches leave to the analyst: warranted links don't automatically make a warranted chain. The target of one study may not be the source of the next; system, population, outcome, or conditions may change at the interface; and shared data or model lineage may make apparently independent support dependent. Projectibility concerns whether a bounded extension from observed to unobserved cases is warranted. Goodman supplies the problem of rival extensions; argument-based validity supplies an architecture for testing them. The contribution is an interface audit for distributed AI evidence: typed source and target descriptions, and a procedure separating endpoints that never meet from endpoints that meet while warrant fails to cross. A legal-research case shows how benchmark evidence and a deployment study can each be sound while remaining parallel. A known-truth demonstration shows why aggregate stability can erase distinctions a later projection requires. The resulting projectibility audit diagnoses unsupported joins in benchmark-to-use arguments.