# LLMs Among Us: Generative AI Participating in Digital Discourse

arXiv: https://arxiv.org/abs/2402.07940 · HTML: https://mlsanity.com/paper/2402.07940/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 48 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 7 of 7)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning](https://mlsanity.com/paper/2304.06588/) (2023)
- [Online Human-Bot Interactions: Detection, Estimation, and Characterization](https://mlsanity.com/paper/1703.03107/) (2017)
- [BotOrNot: A System to Evaluate Social Bots](https://mlsanity.com/paper/1602.00975/) (2016)
- [The DARPA Twitter Bot Challenge](https://mlsanity.com/paper/1601.05140/) (2016)

## Built on by (most cited first, 9 of 9)

- [Decoding Hate: Exploring Language Models' Reactions to Hate Speech](https://mlsanity.com/paper/2410.00775/) (2024)
- [AI Agent Behavioral Science](https://mlsanity.com/paper/2506.06366/) (2025)
- [Seeing Through AI's Lens: Enhancing Human Skepticism Towards LLM-Generated Fake News](https://mlsanity.com/paper/2406.14012/) (2024)
- [AI Propaganda factories with language models](https://mlsanity.com/paper/2508.20186/) (2025)
- [What is a Social Media Bot? A Global Comparison of Bot and Human Characteristics](https://mlsanity.com/paper/2501.00855/) (2025)
- [Public Discourse Sandbox: Facilitating Human and AI Digital Communication Research](https://mlsanity.com/paper/2505.21604/) (2025)
- [Pragmatics beyond humans: meaning, communication, and LLMs](https://mlsanity.com/paper/2508.06167/) (2025)
- [Large language models for spreading dynamics in complex systems](https://mlsanity.com/paper/2602.08085/) (2026)
- [Telescope: Improving Zero Shot Detection of LLM Generated Content By Measuring Token Repetition Probability](https://mlsanity.com/paper/2607.04061/) (2026)

## Abstract (arXiv)

The emergence of Large Language Models (LLMs) has great potential to reshape the landscape of many social media platforms. While this can bring promising opportunities, it also raises many threats, such as biases and privacy concerns, and may contribute to the spread of propaganda by malicious actors. We developed the "LLMs Among Us" experimental framework on top of the Mastodon social media platform for bot and human participants to communicate without knowing the ratio or nature of bot and human participants. We built 10 personas with three different LLMs, GPT-4, LLama 2 Chat, and Claude. We conducted three rounds of the experiment and surveyed participants after each round to measure the ability of LLMs to pose as human participants without human detection. We found that participants correctly identified the nature of other users in the experiment only 42% of the time despite knowing the presence of both bots and humans. We also found that the choice of persona had substantially more impact on human perception than the choice of mainstream LLMs.