# The Prediction-Measurement Gap: Toward Meaning Representations as Scientific Instruments

arXiv: https://arxiv.org/abs/2603.10130 · HTML: https://mlsanity.com/paper/2603.10130/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 19 of 19)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Improving and generalizing flow-based generative models with minibatch optimal transport](https://mlsanity.com/paper/2302.00482/) (2023)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [Text Embeddings Reveal (Almost) As Much As Text](https://mlsanity.com/paper/2310.06816/) (2023)
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://mlsanity.com/paper/1808.08949/) (2018)
- [The Geometry of Culture: Analyzing Meaning through Word Embeddings](https://mlsanity.com/paper/1803.09288/) (2018)
- [Language Models are Injective and Hence Invertible](https://mlsanity.com/paper/2510.15511/) (2025)
- [Factors Influencing the Surprising Instability of Word Embeddings](https://mlsanity.com/paper/1804.09692/) (2018)
- [Substitution-based Semantic Change Detection using Contextual Embeddings](https://mlsanity.com/paper/2309.02403/) (2023)
- [Machine learning as a model for cultural learning: Teaching an algorithm what it means to be fat](https://mlsanity.com/paper/2003.12133/) (2020)
- [Analyzing the Surprising Variability in Word Embedding Stability Across Languages](https://mlsanity.com/paper/2004.14876/) (2020)
- [Measuring the Instability of Fine-Tuning](https://mlsanity.com/paper/2302.07778/) (2023)
- [Theoretical foundations and limits of word embeddings: what types of meaning can they capture?](https://mlsanity.com/paper/2107.10413/) (2021)

## Built on by (most cited first, 1 of 1)

- [Psychological Constructs in Shared Semantic Space](https://mlsanity.com/paper/2605.26801/) (2026)

## Abstract (arXiv)

Text embeddings have become central to computational social science and psychology, enabling scalable measurement of meaning and mixed-method inference. Yet most representation learning is optimized and evaluated for prediction and retrieval, yielding a prediction-measurement gap: representations that perform well as features may be poorly suited as scientific instruments. The paper argues that scientific meaning analysis motivates a distinct family of objectives - scientific usability - emphasizing geometric legibility, interpretability and traceability to linguistic evidence, robustness to non-semantic confounds, and compatibility with regression-style inference over semantic directions. Grounded in cognitive and neuro-psychological views of meaning, the paper assesses static word embeddings and contextual transformer representations against these requirements: static spaces remain attractive for transparent measurement, whereas contextual spaces offer richer semantics but entangle meaning with other signals and exhibit geometric and interpretability issues that complicate inference. The paper then outlines a course-setting agenda around (i) geometry-first design for gradients and abstraction, including hierarchy-aware spaces constrained by psychologically privileged levels; (ii) invertible post-hoc transformations that recondition embedding geometry and reduce nuisance influence; and (iii) meaning atlases and measurement-oriented evaluation protocols for reliable and traceable semantic inference. As the field debates the limits of scale-first progress, measurement-ready representations offer a principled new frontier.