# Hypothesis Formalization: Empirical Findings, Software Limitations, and Design Implications

arXiv: https://arxiv.org/abs/2104.02712 · HTML: https://mlsanity.com/paper/2104.02712/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2021.

## Built on by (most cited first, 7 of 7)

- [Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM](https://mlsanity.com/paper/2404.12259/) (2024)
- [Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design](https://mlsanity.com/paper/2303.02884/) (2023)
- [Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships](https://mlsanity.com/paper/2201.02705/) (2022)
- ["The Diagram is like Guardrails": Structuring GenAI-assisted Hypotheses Exploration with an Interactive Shared Representation](https://mlsanity.com/paper/2503.16791/) (2025)
- [Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks](https://mlsanity.com/paper/2507.02819/) (2025)
- [rTisane: Externalizing conceptual models for data analysis increases engagement with domain knowledge and improves statistical model quality](https://mlsanity.com/paper/2310.16262/) (2023)
- [How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats](https://mlsanity.com/paper/2609.35815/) (2026)

## Abstract (arXiv)

Data analysis requires translating higher level questions and hypotheses into computable statistical models. We present a mixed-methods study aimed at identifying the steps, considerations, and challenges involved in operationalizing hypotheses into statistical models, a process we refer to as hypothesis formalization. In a formative content analysis of research papers, we find that researchers highlight decomposing a hypothesis into sub-hypotheses, selecting proxy variables, and formulating statistical models based on data collection design as key steps. In a lab study, we find that analysts fixated on implementation and shaped their analysis to fit familiar approaches, even if sub-optimal. In an analysis of software tools, we find that tools provide inconsistent, low-level abstractions that may limit the statistical models analysts use to formalize hypotheses. Based on these observations, we characterize hypothesis formalization as a dual-search process balancing conceptual and statistical considerations constrained by data and computation, and discuss implications for future tools.