# Paths Explored, Paths Omitted, Paths Obscured: Decision Points & Selective Reporting in End-to-End Data Analysis

arXiv: https://arxiv.org/abs/1910.13602 · HTML: https://mlsanity.com/paper/1910.13602/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2019.

## Built on by (most cited first, 7 of 7)

- [Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition](https://mlsanity.com/paper/2407.02651/) (2024)
- [Sensitivity-Aware Amortized Bayesian Inference](https://mlsanity.com/paper/2310.11122/) (2023)
- [LightVA: Lightweight Visual Analytics with LLM Agent-Based Task Planning and Execution](https://mlsanity.com/paper/2411.05651/) (2024)
- [Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships](https://mlsanity.com/paper/2201.02705/) (2022)
- [rTisane: Externalizing conceptual models for data analysis increases engagement with domain knowledge and improves statistical model quality](https://mlsanity.com/paper/2310.16262/) (2023)
- [Survey for Categorising Explainable AI Studies Using Data Analysis Task Frameworks](https://mlsanity.com/paper/2507.10208/) (2025)
- [ForkSCOPE: Charting the Agentic Garden of Forking Paths](https://mlsanity.com/paper/2609.12438/) (2026)

## Abstract (arXiv)

Drawing reliable inferences from data involves many, sometimes arbitrary, decisions across phases of data collection, wrangling, and modeling. As different choices can lead to diverging conclusions, understanding how researchers make analytic decisions is important for supporting robust and replicable analysis. In this study, we pore over nine published research studies and conduct semi-structured interviews with their authors. We observe that researchers often base their decisions on methodological or theoretical concerns, but subject to constraints arising from the data, expertise, or perceived interpretability. We confirm that researchers may experiment with choices in search of desirable results, but also identify other reasons why researchers explore alternatives yet omit findings. In concert with our interviews, we also contribute visualizations for communicating decision processes throughout an analysis. Based on our results, we identify design opportunities for strengthening end-to-end analysis, for instance via tracking and meta-analysis of multiple decision paths.