QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models
Not on the map. Why?
Abstract (arXiv)
We introduce QSTN, an open-source Python framework for systematically generating responses from questionnaire-style prompts to support in-silico surveys and annotation tasks with large language models (LLMs). QSTN enables robust evaluation of questionnaire presentation, prompt perturbations, and response generation methods. Our extensive evaluation (>40 million survey responses) shows that question structure and response generation methods have a significant impact on the alignment of generated survey responses with human answers. We also find that answers can be obtained for a fraction of the compute cost, by changing the presentation method. In addition, we offer a no-code user interface that allows researchers to set up robust experiments with LLMs without coding knowledge. We hope that QSTN will support the reproducibility and reliability of LLM-based research in the future.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 9 Dec 2025.
Builds on
27 of 44 references.
References · 27
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2023Whose Opinions Do Language Models Reflect?cited by 773
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
- 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
- 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
- 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
- 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
- 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
- 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 105
- 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 90
15 more
- 2024The Political Preferences of LLMscited by 84
- 2022Stop Measuring Calibration When Humans Disagreecited by 71
- 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 70
- 2024Benchmarking Distributional Alignment of Large Language Modelscited by 61
- 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 59
- 2023You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instrumentscited by 48
- 2025Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancementcited by 45
- 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 30
- 2024Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchangecited by 22
- 2024Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AIcited by 18
- 2025The threat of analytic flexibility in using large language models to simulate human datacited by 15
- 2025Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Modelscited by 8
- 2025MoVa: Towards Generalizable Classification of Human Morals and Valuescited by 7
- 2025PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generationcited by 3
- 2025PAFT: Prompt-Agnostic Fine-Tuningcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.