Position: AI for Science Should Treat Measurement-to-Dataset Pipelines as Inference Components
Not on the map. Why?
Abstract (arXiv)
AI for Science (AI4Science) workflows often treat the released dataset as a fixed interface to the underlying system. However, in domains relying on indirect observation, the learner observes a derivative representation produced by multi-stage measurement, reconstruction, and preprocessing pipelines. We argue that these measurement-to-dataset pipelines are inference components: treating their outputs as ``given data'' freezes an observation model and obscures uncertainty over feasible pipeline choices. We identify three failure modes arising from this ``frozen lens'': (C1) hidden hypothesis space, where the released dataset does not specify the pipeline configuration or its validity conditions; (C2) uncertified transportability, where a pipeline may be documented but its regime of validity is untested, so failures under distribution shift cannot be adjudicated; (C3) ungoverned multiplicity, where many defensible pipelines exist and dispersion is real but not propagated into uncertainty-aware evidence. We stress-test these claims with a large-scale neuroscience empirical audit, finding a survival rate of ≈ 0.0004% under a cross-dataset stability criterion. We call on the AI4Science community to make pipelines computable inference objects via domain-specific Computable Observation Frameworks. This shift enables quantifying pipeline adequacy and stability, converting implicit implementation choices into auditable, reproducible, and cumulative scientific evidence.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 126 references that are in the map. First posted on arXiv 23 May 2026.
Builds on
36 of 126 references.
References · 36
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
- 2019AutoML: A Survey of the State-of-the-Artcited by 388
- 2017MoDL: Model Based Deep Learning Architecture for Inverse Problemscited by 384
- 2023ClimaX: A foundation model for weather and climatecited by 325
- 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
- 2019On instabilities of deep learning in image reconstruction - Does AI come at a cost?cited by 211
- 2023Data-centric Artificial Intelligence: A Surveycited by 202
- 2020DDSP: Differentiable Digital Signal Processingcited by 177
- 2024Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applicationscited by 139
- 2014Preserving Statistical Validity in Adaptive Data Analysiscited by 102
- 2018Why rankings of biomedical image analysis competitions should be interpreted with carecited by 90
24 more
- 2015The Ladder: A Reliable Leaderboard for Machine Learning Competitionscited by 67
- 2021Scientific Machine Learning Benchmarkscited by 60
- 2023GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restorationcited by 58
- 2021Disentangling Syntax and Semantics in the Brain with Deep Networkscited by 39
- 2020MODMA dataset: a Multi-modal Open Dataset for Mental-disorder Analysiscited by 38
- 2020Sample Complexity Bounds for 1-bit Compressive Sensing and Binary Stable Embeddings with Generative Priorscited by 21
- 2024The 2024 Foundation Model Transparency Indexcited by 18
- 2024Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Reviewcited by 17
- 2024Automating the Practice of Science -- Opportunities, Challenges, and Implicationscited by 17
- 2025Position: The Most Expensive Part of an LLM should be its Training Datacited by 15
- 2021End-to-end learnable EEG channel selection for deep neural networks with Gumbel-softmaxcited by 12
- 2022Modeling the Machine Learning Multiversecited by 11
- 2023Drug Discovery under Covariate Shift with Domain-Informed Prior Distributions over Functionscited by 10
- 2020Probabilistic Programs with Stochastic Conditioningcited by 8
- 2022Unifying Pairwise Interactions in Complex Dynamicscited by 8
- 2024How EEG preprocessing shapes decoding performancecited by 7
- 2024M3LEO: A Multi-Modal, Multi-Label Earth Observation Dataset Integrating Interferometric SAR and Multispectral Datacited by 6
- 2022Uncertain Evidence in Probabilistic Models and Stochastic Simulatorscited by 5
- 2024Is machine learning good or bad for the natural sciences?cited by 4
- 2022Accelerated functional brain aging in major depressive disorder: evidence from a large scale fMRI analysis of Chinese participantscited by 3
- 2024Sex Differences in Hierarchical and Modular Organization of Functional Brain Networks: Insights from Hierarchical Entropy and Modularity Analysiscited by 2
- 2025Beyond Pairwise Connections: Extracting High-Order Functional Brain Network Structures under Global Constraintscited by 1
- 2026Accelerating Benchmarking of Functional Connectivity Modeling via Structure-aware Core-set Selectioncited by 1
- 2026PDSL: Propagation Dynamics Aware Framework for Source Localizationcited by 1
Built on by
Not cited yet.