# Identifying and Measuring Token-Level Sentiment Bias in Pre-trained Language Models with Prompts

arXiv: https://arxiv.org/abs/2204.07289 · HTML: https://mlsanity.com/paper/2204.07289/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 15 Apr 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://mlsanity.com/paper/2005.00813/) (2020)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [The Radicalization Risks of GPT-3 and Advanced Neural Language Models](https://mlsanity.com/paper/2009.06807/) (2020)
- [Investigating African-American Vernacular English in Transformer-Based Text Generation](https://mlsanity.com/paper/2010.02510/) (2020)
- [SqueezeBERT: What can computer vision teach NLP about efficient neural networks?](https://mlsanity.com/paper/2006.11316/) (2020)
- [Investigating Gender Bias in BERT](https://mlsanity.com/paper/2009.05021/) (2020)
- ["You are grounded!": Latent Name Artifacts in Pre-trained Language Models](https://mlsanity.com/paper/2004.03012/) (2020)

## Built on by (most cited first, 4 of 4)

- [Evaluation of African American Language Bias in Natural Language Generation](https://mlsanity.com/paper/2305.14291/) (2023)
- [Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs](https://mlsanity.com/paper/2402.11442/) (2024)
- [Can LLMs facilitate interpretation of pre-trained language models?](https://mlsanity.com/paper/2305.13386/) (2023)
- [Structured Reasoning for Fairness: A Multi-Agent Approach to Bias Detection in Textual Data](https://mlsanity.com/paper/2503.00355/) (2025)

## Abstract (arXiv)

Due to the superior performance, large-scale pre-trained language models (PLMs) have been widely adopted in many aspects of human society. However, we still lack effective tools to understand the potential bias embedded in the black-box models. Recent advances in prompt tuning show the possibility to explore the internal mechanism of the PLMs. In this work, we propose two token-level sentiment tests: Sentiment Association Test (SAT) and Sentiment Shift Test (SST) which utilize the prompt as a probe to detect the latent bias in the PLMs. Our experiments on the collection of sentiment datasets show that both SAT and SST can identify sentiment bias in PLMs and SST is able to quantify the bias. The results also suggest that fine-tuning can possibly augment the existing bias in PLMs.