# Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica

arXiv: https://arxiv.org/abs/2109.02738 · HTML: https://mlsanity.com/paper/2109.02738/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [A Computational Approach to Politeness with Application to Social Factors](https://mlsanity.com/paper/1306.6078/) (2013)
- [Did the Model Understand the Question?](https://mlsanity.com/paper/1805.05492/) (2018)
- [SelfExplain: A Self-Explaining Architecture for Neural Text Classifiers](https://mlsanity.com/paper/2103.12279/) (2021)

## Built on by (most cited first, 23 of 23)

- [Benchmarking Cognitive Biases in Large Language Models as Evaluators](https://mlsanity.com/paper/2309.17012/) (2023)
- [Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark](https://mlsanity.com/paper/2305.14938/) (2023)
- [The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks](https://mlsanity.com/paper/2304.13861/) (2023)
- [The Homogenizing Effect of Large Language Models on Human Expression and Thought](https://mlsanity.com/paper/2508.01491/) (2025)
- [Under the Surface: Tracking the Artifactuality of LLM-Generated Data](https://mlsanity.com/paper/2401.14698/) (2024)
- [How Far Can We Extract Diverse Perspectives from Large Language Models?](https://mlsanity.com/paper/2311.09799/) (2023)
- [A Fine-grained Interpretability Evaluation Benchmark for Neural NLP](https://mlsanity.com/paper/2205.11097/) (2022)
- [Through the Lens of Core Competency: Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2308.07902/) (2023)
- [SoMeLVLM: A Large Vision Language Model for Social Media Processing](https://mlsanity.com/paper/2402.13022/) (2024)
- [Building Knowledge-Guided Lexica to Model Cultural Variation](https://mlsanity.com/paper/2406.11622/) (2024)
- [On Evaluating Explanation Utility for Human-AI Decision Making in NLP](https://mlsanity.com/paper/2407.03545/) (2024)
- [SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Tasks](https://mlsanity.com/paper/2402.01980/) (2024)
- [An Attention Matrix for Every Decision: Faithfulness-based Arbitration Among Multiple Attention-Based Interpretations of Transformers in Text Classification](https://mlsanity.com/paper/2209.10876/) (2022)
- [StyLEx: Explaining Style Using Human Lexical Annotations](https://mlsanity.com/paper/2210.07469/) (2022)
- [Investigating the Impact of Model Instability on Explanations and Uncertainty](https://mlsanity.com/paper/2402.13006/) (2024)
- [A Comparative Study on Textual Saliency of Styles from Eye Tracking, Annotations, and Language Models](https://mlsanity.com/paper/2212.09873/) (2022)
- [Comparing Styles across Languages: A Cross-Cultural Exploration of Politeness](https://mlsanity.com/paper/2310.07135/) (2023)
- [Learning Explainable Dense Reward Shapes via Bayesian Optimization](https://mlsanity.com/paper/2504.16272/) (2025)
- [Main Predicate and Their Arguments as Explanation Signals For Intent Classification](https://mlsanity.com/paper/2502.01270/) (2025)
- [Capturing Human Cognitive Styles with Language: Towards an Experimental Evaluation Paradigm](https://mlsanity.com/paper/2502.13326/) (2025)
- [RedPen: Region- and Reason-Annotated Dataset of Unnatural Speech](https://mlsanity.com/paper/2210.14406/) (2022)
- [Splits! Flexible Sociocultural Linguistic Investigation at Scale](https://mlsanity.com/paper/2504.04640/) (2025)
- [Reasoning Beyond Literal: Cross-style Multimodal Reasoning for Figurative Language Understanding](https://mlsanity.com/paper/2601.17197/) (2026)

## Abstract (arXiv)

People convey their intention and attitude through linguistic styles of the text that they write. In this study, we investigate lexicon usages across styles throughout two lenses: human perception and machine word importance, since words differ in the strength of the stylistic cues that they provide. To collect labels of human perception, we curate a new dataset, Hummingbird, on top of benchmarking style datasets. We have crowd workers highlight the representative words in the text that makes them think the text has the following styles: politeness, sentiment, offensiveness, and five emotion types. We then compare these human word labels with word importance derived from a popular fine-tuned style classifier like BERT. Our results show that the BERT often finds content words not relevant to the target style as important words used in style prediction, but humans do not perceive the same way even though for some styles (e.g., positive sentiment and joy) human- and machine-identified words share significant overlap for some styles.