# Who Gets Which Message? Auditing Demographic Bias in LLM-Generated Targeted Text

arXiv: https://arxiv.org/abs/2601.17172 · HTML: https://mlsanity.com/paper/2601.17172/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 55 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [Towards Controllable Biases in Language Generation](https://mlsanity.com/paper/2005.00268/) (2020)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- ["I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation](https://mlsanity.com/paper/2305.09941/) (2023)
- [Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception](https://mlsanity.com/paper/2403.14896/) (2024)
- [Men Are Elected, Women Are Married: Events Gender Bias on Wikipedia](https://mlsanity.com/paper/2106.01601/) (2021)
- [Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model](https://mlsanity.com/paper/2310.11648/) (2023)
- [The Authors Matter: Understanding and Mitigating Implicit Bias in Deep Text Classification](https://mlsanity.com/paper/2105.02778/) (2021)
- [Uncovering Latent Arguments in Social Media Messaging by Employing LLMs-in-the-Loop Strategy](https://mlsanity.com/paper/2404.10259/) (2024)
- [LLMs for Customized Marketing Content Generation and Evaluation at Scale](https://mlsanity.com/paper/2506.17863/) (2025)
- [Weakly Supervised Learning for Analyzing Political Campaigns on Facebook](https://mlsanity.com/paper/2210.10669/) (2022)
- [Post-hoc Study of Climate Microtargeting on Social Media Ads with LLMs: Thematic Insights and Fairness Evaluation](https://mlsanity.com/paper/2410.05401/) (2024)

## Built on by (most cited first, 4 of 4)

- [Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA](https://mlsanity.com/paper/2603.24580/) (2026)
- [Reasoning-Based Refinement of Unsupervised Text Clusters with LLMs](https://mlsanity.com/paper/2604.07562/) (2026)
- [Pareto-Guided Teacher Alignment for Fair Personalized Text Generation](https://mlsanity.com/paper/2606.10126/) (2026)
- [RelLedger: Relational Information Governance for LLM Agent Organizations](https://mlsanity.com/paper/2610.16281/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly capable of generating personalized, persuasive text at scale, raising new questions about bias and fairness in automated communication. This paper presents the first systematic analysis of how LLMs behave when tasked with demographic-conditioned targeted messaging. We introduce a controlled evaluation framework using three leading models: GPT-4o, Llama-3.3, and Mistral-Large-2.1, across two generation settings: Standalone Generation, which isolates intrinsic demographic effects, and Context-Rich Generation, which incorporates thematic and regional context to emulate realistic targeting. We evaluate generated messages along three dimensions: lexical content, language style, and persuasive framing. We instantiate this framework on climate communication and find consistent age- and gender-based asymmetries across models: male- and youth-targeted messages tend to emphasize more assertive and progressive framing, while female- and senior-targeted messages more often reflect warmth, care, and traditional themes. Contextual prompts systematically amplify these disparities, with persuasion scores being higher for male-targeted messages, while age-related differences vary across models. Our findings demonstrate how demographic stereotypes can surface and intensify in LLM-generated targeted communication, underscoring the need for bias-aware generation pipelines and transparent auditing frameworks that explicitly account for demographic conditioning in socially sensitive applications.