# Position: Insights from Survey Methodology can Improve Training Data

arXiv: https://arxiv.org/abs/2403.01208 · HTML: https://mlsanity.com/paper/2403.01208/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2024.

## Built on by (most cited first, 13 of 13)

- [LLM Generated Persona is a Promise with a Catch](https://mlsanity.com/paper/2503.16527/) (2025)
- [The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models](https://mlsanity.com/paper/2406.11096/) (2024)
- [Mitigating Selection Bias with Node Pruning and Auxiliary Options](https://mlsanity.com/paper/2409.18857/) (2024)
- [SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questions](https://mlsanity.com/paper/2506.00643/) (2025)
- [HESEIA: A community-based dataset for evaluating social biases in large language models, co-designed in real school settings in Latin America](https://mlsanity.com/paper/2505.24712/) (2025)
- [Bias in the Loop: How Humans Evaluate AI-Generated Suggestions](https://mlsanity.com/paper/2509.08514/) (2025)
- [From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLP](https://mlsanity.com/paper/2510.12817/) (2025)
- [From Ground Truth to Measurement: A Statistical Framework for Human Labeling](https://mlsanity.com/paper/2604.07591/) (2026)
- [Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication](https://mlsanity.com/paper/2501.06826/) (2025)
- [Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation](https://mlsanity.com/paper/2510.13884/) (2025)
- [Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models](https://mlsanity.com/paper/2607.26173/) (2026)
- [An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration](https://mlsanity.com/paper/2605.18648/) (2026)
- [Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation](https://mlsanity.com/paper/2609.35824/) (2026)

## Abstract (arXiv)

Whether future AI models are fair, trustworthy, and aligned with the public's interests rests in part on our ability to collect accurate data about what we want the models to do. However, collecting high-quality data is difficult, and few AI/ML researchers are trained in data collection methods. Recent research in data-centric AI has show that higher quality training data leads to better performing models, making this the right moment to introduce AI/ML researchers to the field of survey methodology, the science of data collection. We summarize insights from the survey methodology literature and discuss how they can improve the quality of training and feedback data. We also suggest collaborative research ideas into how biases in data collection can be mitigated, making models more accurate and human-centric.