# Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference

arXiv: https://arxiv.org/abs/2002.04815 · HTML: https://mlsanity.com/paper/2002.04815/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 14 of 14)

- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses](https://mlsanity.com/paper/2106.05426/) (2021)
- [Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews](https://mlsanity.com/paper/2104.05861/) (2021)
- [SILT: Efficient transformer training for inter-lingual inference](https://mlsanity.com/paper/2103.09635/) (2021)
- [Transfer learning for conflict and duplicate detection in software requirement pairs](https://mlsanity.com/paper/2301.03709/) (2023)
- [Visualizing and Explaining Language Models](https://mlsanity.com/paper/2205.10238/) (2022)
- [Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documents](https://mlsanity.com/paper/2403.06872/) (2024)
- [BERT-Deep CNN: State-of-the-Art for Sentiment Analysis of COVID-19 Tweets](https://mlsanity.com/paper/2211.09733/) (2022)
- [Depression detection in social media posts using transformer-based models and auxiliary features](https://mlsanity.com/paper/2409.20048/) (2024)
- [A multi-model-based deep learning framework for short text multiclass classification with the imbalanced and extremely small data set](https://mlsanity.com/paper/2206.12027/) (2022)
- [Aspect-specific Context Modeling for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2207.08099/) (2022)
- [Adaptive Large Language Models By Layerwise Attention Shortcuts](https://mlsanity.com/paper/2409.10870/) (2024)
- [T-BERT -- Model for Sentiment Analysis of Micro-blogs Integrating Topic Model and BERT](https://mlsanity.com/paper/2106.01097/) (2021)
- [The Dream Within Huang Long Cave: AI-Driven Interactive Narrative for Family Storytelling and Emotional Reflection](https://mlsanity.com/paper/2504.04968/) (2025)

## Abstract (arXiv)

Aspect based sentiment analysis aims to identify the sentimental tendency towards a given aspect in text. Fine-tuning of pretrained BERT performs excellent on this task and achieves state-of-the-art performances. Existing BERT-based works only utilize the last output layer of BERT and ignore the semantic knowledge in the intermediate layers. This paper explores the potential of utilizing BERT intermediate layers to enhance the performance of fine-tuning of BERT. To the best of our knowledge, no existing work has been done on this research. To show the generality, we also apply this approach to a natural language inference task. Experimental results demonstrate the effectiveness and generality of the proposed approach.