# Next Reply Prediction X Dataset: Linguistic Discrepancies in Naively Generated Content

arXiv: https://arxiv.org/abs/2602.19177 · HTML: https://mlsanity.com/paper/2602.19177/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 45 references that are in the map.
- First posted on arXiv 22 Feb 2026.

## Builds on (most cited first, 14 of 14)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [TweetNLP: Cutting-Edge Natural Language Processing for Social Media](https://mlsanity.com/paper/2206.14774/) (2022)
- [Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations](https://mlsanity.com/paper/2504.03274/) (2025)
- [Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks](https://mlsanity.com/paper/2408.01346/) (2024)
- [Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realism](https://mlsanity.com/paper/2506.21974/) (2025)
- [Political Bias in LLMs: Unaligned Moral Values in Agent-centric Simulations](https://mlsanity.com/paper/2408.11415/) (2024)
- [Designing Domain-Specific Large Language Models: The Critical Role of Fine-Tuning in Public Opinion Simulation](https://mlsanity.com/paper/2409.19308/) (2024)

## Abstract (arXiv)

The increasing use of Large Language Models (LLMs) as proxies for human participants in social science research presents a promising, yet methodologically risky, paradigm shift. While LLMs offer scalability and cost-efficiency, their "naive" application, where they are prompted to generate content without explicit behavioral constraints, introduces significant linguistic discrepancies that challenge the validity of research findings. This paper addresses these limitations by introducing a novel, history-conditioned reply prediction task on authentic X (formerly Twitter) data, to create a dataset designed to evaluate the linguistic output of LLMs against human-generated content. We analyze these discrepancies using stylistic and content-based metrics, providing a quantitative framework for researchers to assess the quality and authenticity of synthetic data. Our findings highlight the need for more sophisticated prompting techniques and specialized datasets to ensure that LLM-generated content accurately reflects the complex linguistic patterns of human communication, thereby improving the validity of computational social science studies.