# Measurement in the Age of LLMs: An Application to Ideological Scaling

arXiv: https://arxiv.org/abs/2312.09203 · HTML: https://mlsanity.com/paper/2312.09203/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 60 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 16 of 16)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [Improving Language Models via Plug-and-Play Retrieval Feedback](https://mlsanity.com/paper/2305.14002/) (2023)
- [Cognitive Mirage: A Review of Hallucinations in Large Language Models](https://mlsanity.com/paper/2309.06794/) (2023)
- [Are Large Language Models Really Robust to Word-Level Perturbations?](https://mlsanity.com/paper/2309.11166/) (2023)
- [Sorting through the noise: Testing robustness of information processing in pre-trained language models](https://mlsanity.com/paper/2109.12393/) (2021)
- [From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models](https://mlsanity.com/paper/2305.17174/) (2023)
- [Text-Based Ideal Points](https://mlsanity.com/paper/2005.04232/) (2020)
- [Unsupervised Contrast-Consistent Ranking with Language Models](https://mlsanity.com/paper/2309.06991/) (2023)
- [Concept-Guided Chain-of-Thought Prompting for Pairwise Comparison Scoring of Texts with Large Language Models](https://mlsanity.com/paper/2310.12049/) (2023)

## Built on by (most cited first, 11 of 11)

- [Linear Representations of Political Perspective Emerge in Large Language Models](https://mlsanity.com/paper/2503.02080/) (2025)
- [Context versus Prior Knowledge in Language Models](https://mlsanity.com/paper/2404.04633/) (2024)
- [Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Models](https://mlsanity.com/paper/2412.16746/) (2024)
- [Measuring Scalar Constructs in Social Science with LLMs](https://mlsanity.com/paper/2509.03116/) (2025)
- [LABOR-LLM: Language-Based Occupational Representations with Large Language Models](https://mlsanity.com/paper/2406.17972/) (2024)
- [Adaptive Uncertainty Quantification for Generative AI](https://mlsanity.com/paper/2408.08990/) (2024)
- [AI Propaganda factories with language models](https://mlsanity.com/paper/2508.20186/) (2025)
- [LLM-POTUS Score: A Framework of Analyzing Presidential Debates with Large Language Models](https://mlsanity.com/paper/2409.08147/) (2024)
- [Identifying the sources of ideological bias in GPT models through linguistic variation in output](https://mlsanity.com/paper/2409.06043/) (2024)
- [Large Language Models Can Be a Viable Substitute for Expert Political Surveys When a Shock Disrupts Traditional Measurement Approaches](https://mlsanity.com/paper/2506.06540/) (2025)
- [Validating LLMs in social science: Epistemic threats and emerging norms](https://mlsanity.com/paper/2607.07915/) (2026)

## Abstract (arXiv)

Much of social science is centered around terms like ``ideology'' or ``power'', which generally elude precise definition, and whose contextual meanings are trapped in surrounding language. This paper explores the use of large language models (LLMs) to flexibly navigate the conceptual clutter inherent to social scientific measurement tasks. We rely on LLMs' remarkable linguistic fluency to elicit ideological scales of both legislators and text, which accord closely to established methods and our own judgement. A key aspect of our approach is that we elicit such scores directly, instructing the LLM to furnish numeric scores itself. This approach affords a great deal of flexibility, which we showcase through a variety of different case studies. Our results suggest that LLMs can be used to characterize highly subtle and diffuse manifestations of political ideology in text.