Making Political Text Scaling Comparable: Infrastructure and Hyperparameter Sensitivity for 17 Algorithms
Not on the map. Why?
Abstract (arXiv)
Computational text-based ideal point estimation (CT-IPE) methods are usually compared as named algorithms, yet applying them involves numerous researcher choices that configure how political text is turned into position estimates. This paper argues that CT-IPE methods are better understood as configurable measurement pipelines than as fixed estimators. Building on a large-scale comparative experiment spanning 17 CT-IPE algorithms, 5,537 experimental runs, and approximately 4.25 million left-right position estimates, I describe the shared infrastructure that makes these heterogeneous methods jointly executable and quantify how sensitive their estimates are to alternative hyperparameter choices. Variance-partitioning and SHAP-based sensitivity analyses show that, for most algorithms, hyperparameter profiles explain little residual variance through a shared shift: 13 of the 17 algorithms exhibit ICC values below .10. Where this profile-level sensitivity is present, it is concentrated in a small number of consequential researcher choices, most notably the selection of the underlying language or embedding model, the seed keyword lists that anchor the construct, and the number of topics.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 46 references that are in the map. First posted on arXiv 13 Sep 2026.
Builds on
12 of 46 references.
References · 12
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
- 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
- 2020On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practicecited by 182
- 2023Large Language Models Can Be Used to Estimate the Latent Positions of Politicianscited by 33
- 2024Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Conceptscited by 30
- 2020Text-Based Ideal Pointscited by 22
- 2019Political Text Scaling Meets Computational Semanticscited by 10
- 2024A Public Dataset Tracking Social Media Discourse about the 2024 U.S. Presidential Election on Twitter/Xcited by 7
- 2024Bridging Dictionary: AI-Generated Dictionary of Partisan Language Usecited by 3
- 2024SpeakGer: A meta-data enriched speech corpus of German state and federal parliamentscited by 2
- 2025Wordkrill: Extending Wordfish into the multidimensional political spacecited by 1
- 2025Computational Measurement of Political Positions: A Review of Text-Based Ideal Point Estimation Algorithmscited by 1
Built on by
Not cited yet.