# CommunityLM: Probing Partisan Worldviews from Language Models

arXiv: https://arxiv.org/abs/2209.07065 · HTML: https://mlsanity.com/paper/2209.07065/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [SemEval-2017 Task 4: Sentiment Analysis in Twitter](https://mlsanity.com/paper/1912.00741/) (2019)
- [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [TimeLMs: Diachronic Language Models from Twitter](https://mlsanity.com/paper/2202.03829/) (2022)
- [Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings](https://mlsanity.com/paper/1904.01596/) (2019)
- [We Don't Speak the Same Language: Interpreting Polarization through Machine Translation](https://mlsanity.com/paper/2010.02339/) (2020)
- [Analyzing COVID-19 Tweets with Transformer-based Language Models](https://mlsanity.com/paper/2104.10259/) (2021)
- [Quantifying Polarization on Twitter: the Kavanaugh Nomination](https://mlsanity.com/paper/2001.02125/) (2020)

## Built on by (most cited first, 30 of 36)

- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Generative Language Models Exhibit Social Identity Biases](https://mlsanity.com/paper/2310.15819/) (2023)
- [Training Socially Aligned Language Models on Simulated Social Interactions](https://mlsanity.com/paper/2305.16960/) (2023)
- [Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity](https://mlsanity.com/paper/2209.12106/) (2022)
- [Questioning the Survey Responses of Large Language Models](https://mlsanity.com/paper/2306.07951/) (2023)
- [On the steerability of large language models toward data-driven personas](https://mlsanity.com/paper/2311.04978/) (2023)
- [Virtual Personas for Language Models via an Anthology of Backstories](https://mlsanity.com/paper/2407.06576/) (2024)
- [Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information](https://mlsanity.com/paper/2402.18144/) (2024)
- [Stereotype or Personalization? User Identity Biases Chatbot Recommendations](https://mlsanity.com/paper/2410.05613/) (2024)
- [Linear Representations of Political Perspective Emerge in Large Language Models](https://mlsanity.com/paper/2503.02080/) (2025)
- [AI for social science and social science of AI: A Survey](https://mlsanity.com/paper/2401.11839/) (2024)
- [How Susceptible are Large Language Models to Ideological Manipulation?](https://mlsanity.com/paper/2402.11725/) (2024)
- [Language Agents as Digital Representatives in Collective Decision-Making](https://mlsanity.com/paper/2502.09369/) (2025)
- [Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges](https://mlsanity.com/paper/2401.08664/) (2023)
- [Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Models](https://mlsanity.com/paper/2412.16746/) (2024)
- [Persona Setting Pitfall: Persistent Outgroup Biases in Large Language Models Arising from Social Identity Adoption](https://mlsanity.com/paper/2409.03843/) (2024)
- [Skin-in-the-Game: Decision Making via Multi-Stakeholder Alignment in LLMs](https://mlsanity.com/paper/2405.12933/) (2024)
- [What Is The Political Content in LLMs' Pre- and Post-Training Data?](https://mlsanity.com/paper/2509.22367/) (2025)
- [textttBluePrint: A Social Media User Dataset for LLM Persona Evaluation and Training](https://mlsanity.com/paper/2510.02343/) (2025)
- [Exposing Bias in Online Communities through Large-Scale Language Models](https://mlsanity.com/paper/2306.02294/) (2023)
- [Political Ideology Shifts in Large Language Models](https://mlsanity.com/paper/2508.16013/) (2025)
- [Identity, Cooperation and Framing Effects within Groups of Real and Simulated Humans](https://mlsanity.com/paper/2601.16355/) (2026)
- [Leveraging Machine Learning to Identify Gendered Stereotypes and Body Image Concerns on Diet and Fitness Online Forums](https://mlsanity.com/paper/2407.03551/) (2024)
- [Bridging Dictionary: AI-Generated Dictionary of Partisan Language Use](https://mlsanity.com/paper/2407.09661/) (2024)
- [Examining Alignment of Large Language Models through Representative Heuristics: The Case of Political Stereotypes](https://mlsanity.com/paper/2501.14294/) (2025)
- [Is the U.S. Legal System Ready for AI's Challenges to Human Values?](https://mlsanity.com/paper/2308.15906/) (2023)
- [HugAgent: A Human Simulation Benchmark for Individual-Level Reasoning](https://mlsanity.com/paper/2510.15144/) (2025)
- [Hedging and Non-Affirmation: Quantifying LLM Alignment on Questions of Human Rights](https://mlsanity.com/paper/2502.19463/) (2025)

## Abstract (arXiv)

As political attitudes have diverged ideologically in the United States, political speech has diverged lingusitically. The ever-widening polarization between the US political parties is accelerated by an erosion of mutual understanding between them. We aim to make these communities more comprehensible to each other with a framework that probes community-specific responses to the same survey questions using community language models CommunityLM. In our framework we identify committed partisan members for each community on Twitter and fine-tune LMs on the tweets authored by them. We then assess the worldviews of the two groups using prompt-based probing of their corresponding LMs, with prompts that elicit opinions about public figures and groups surveyed by the American National Election Studies (ANES) 2020 Exploratory Testing Survey. We compare the responses generated by the LMs to the ANES survey results, and find a level of alignment that greatly exceeds several baseline methods. Our work aims to show that we can use community LMs to query the worldview of any group of people given a sufficiently large sample of their social media discussions or media diet.