# Understanding Pre-trained BERT for Aspect-based Sentiment Analysis

arXiv: https://arxiv.org/abs/2011.00169 · HTML: https://mlsanity.com/paper/2011.00169/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [DiaASQ : A Benchmark of Conversational Aspect-based Sentiment Quadruple Analysis](https://mlsanity.com/paper/2211.05705/) (2022)
- [Prompt Tuning Large Language Models on Personalized Aspect Extraction for Recommendations](https://mlsanity.com/paper/2306.01475/) (2023)
- [Incorporating Dynamic Semantics into Pre-Trained Language Model for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2203.16369/) (2022)
- [Aspect-Based Sentiment Analysis with Explicit Sentiment Augmentations](https://mlsanity.com/paper/2312.10961/) (2023)
- [Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language](https://mlsanity.com/paper/2201.10262/) (2022)
- [Survey of Generative Methods for Social Media Analysis](https://mlsanity.com/paper/2112.07041/) (2021)
- [Fine-tuning BERT with Bidirectional LSTM for Fine-grained Movie Reviews Sentiment Analysis](https://mlsanity.com/paper/2502.20682/) (2025)
- [Learning from Bootstrapping and Stepwise Reinforcement Reward: A Semi-Supervised Framework for Text Style Transfer](https://mlsanity.com/paper/2205.09324/) (2022)
- [Generative Sentiment Transfer via Adaptive Masking](https://mlsanity.com/paper/2302.12045/) (2023)

## Abstract (arXiv)

This paper analyzes the pre-trained hidden representations learned from reviews on BERT for tasks in aspect-based sentiment analysis (ABSA). Our work is motivated by the recent progress in BERT-based language models for ABSA. However, it is not clear how the general proxy task of (masked) language model trained on unlabeled corpus without annotations of aspects or opinions can provide important features for downstream tasks in ABSA. By leveraging the annotated datasets in ABSA, we investigate both the attentions and the learned representations of BERT pre-trained on reviews. We found that BERT uses very few self-attention heads to encode context words (such as prepositions or pronouns that indicating an aspect) and opinion words for an aspect. Most features in the representation of an aspect are dedicated to the fine-grained semantics of the domain (or product category) and the aspect itself, instead of carrying summarized opinions from its context. We hope this investigation can help future research in improving self-supervised learning, unsupervised learning and fine-tuning for ABSA. The pre-trained model and code can be found at https://github.com/howardhsu/BERT-for-RRC-ABSA.