# SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

arXiv: https://arxiv.org/abs/2005.05635 · HTML: https://mlsanity.com/paper/2005.05635/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 30 of 54)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey](https://mlsanity.com/paper/2311.07914/) (2023)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2112.12731/) (2021)
- [Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification](https://mlsanity.com/paper/2109.05427/) (2021)
- [Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa](https://mlsanity.com/paper/2104.04986/) (2021)
- [Learning Implicit Sentiment in Aspect-based Sentiment Analysis with Supervised Contrastive Pre-Training](https://mlsanity.com/paper/2111.02194/) (2021)
- [Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey](https://mlsanity.com/paper/2110.08455/) (2021)
- [Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing](https://mlsanity.com/paper/2106.02559/) (2021)
- [DiaASQ : A Benchmark of Conversational Aspect-based Sentiment Quadruple Analysis](https://mlsanity.com/paper/2211.05705/) (2022)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Preference-Aware Memory Update for Long-Term LLM Agents](https://mlsanity.com/paper/2510.09720/) (2025)
- [Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognition](https://mlsanity.com/paper/2505.19694/) (2025)
- [Think Twice: A Human-like Two-stage Conversational Agent for Emotional Response Generation](https://mlsanity.com/paper/2301.04907/) (2023)
- [Billion-scale Pre-trained E-commerce Product Knowledge Graph Model](https://mlsanity.com/paper/2105.00388/) (2021)
- [K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce](https://mlsanity.com/paper/2104.06960/) (2021)
- [Cross-lingual Text Classification with Heterogeneous Graph Neural Network](https://mlsanity.com/paper/2105.11246/) (2021)
- [Astock: A New Dataset and Automated Stock Trading based on Stock-specific News Analyzing Model](https://mlsanity.com/paper/2206.06606/) (2022)
- [Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis](https://mlsanity.com/paper/2402.09151/) (2024)
- [DILBERT: Customized Pre-Training for Domain Adaptation withCategory Shift, with an Application to Aspect Extraction](https://mlsanity.com/paper/2109.00571/) (2021)
- [FinReport: Explainable Stock Earnings Forecasting via News Factor Analyzing Model](https://mlsanity.com/paper/2403.02647/) (2024)
- [Using Knowledge-Embedded Attention to Augment Pre-trained Language Models for Fine-Grained Emotion Recognition](https://mlsanity.com/paper/2108.00194/) (2021)
- [Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting](https://mlsanity.com/paper/2404.12782/) (2024)
- [Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis](https://mlsanity.com/paper/2210.09803/) (2022)
- [Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning](https://mlsanity.com/paper/2108.00356/) (2021)
- [DuTrust: A Sentiment Analysis Dataset for Trustworthiness Evaluation](https://mlsanity.com/paper/2108.13140/) (2021)
- [FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networks](https://mlsanity.com/paper/2110.11869/) (2021)
- [Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis](https://mlsanity.com/paper/2201.02026/) (2022)
- [ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2306.15796/) (2023)

## Abstract (arXiv)

Recently, sentiment analysis has seen remarkable advance with the help of pre-training approaches. However, sentiment knowledge, such as sentiment words and aspect-sentiment pairs, is ignored in the process of pre-training, despite the fact that they are widely used in traditional sentiment analysis approaches. In this paper, we introduce Sentiment Knowledge Enhanced Pre-training (SKEP) in order to learn a unified sentiment representation for multiple sentiment analysis tasks. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair. Experiments on three kinds of sentiment tasks show that SKEP significantly outperforms strong pre-training baseline, and achieves new state-of-the-art results on most of the test datasets. We release our code at https://github.com/baidu/Senta.