# Unified Instance and Knowledge Alignment Pretraining for Aspect-based Sentiment Analysis

arXiv: https://arxiv.org/abs/2110.13398 · HTML: https://mlsanity.com/paper/2110.13398/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 86 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1904.02232/) (2019)
- [Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting](https://mlsanity.com/paper/2004.12651/) (2020)
- [Relational Graph Attention Network for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2004.12362/) (2020)
- [A Unified Generative Framework for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2106.04300/) (2021)
- [Interactive Attention Networks for Aspect-Level Sentiment Classification](https://mlsanity.com/paper/1709.00893/) (2017)
- [Effective LSTMs for Target-Dependent Sentiment Classification](https://mlsanity.com/paper/1512.01100/) (2015)
- [Aspect Level Sentiment Classification with Deep Memory Network](https://mlsanity.com/paper/1605.08900/) (2016)
- [Aspect Based Sentiment Analysis with Gated Convolutional Networks](https://mlsanity.com/paper/1805.07043/) (2018)
- [Aspect-based Sentiment Classification with Aspect-specific Graph Convolutional Networks](https://mlsanity.com/paper/1909.03477/) (2019)
- [A Joint Training Dual-MRC Framework for Aspect Based Sentiment Analysis](https://mlsanity.com/paper/2101.00816/) (2021)
- [Self-Knowledge Distillation in Natural Language Processing](https://mlsanity.com/paper/1908.01851/) (2019)
- [Understanding and Improving Lexical Choice in Non-Autoregressive Translation](https://mlsanity.com/paper/2012.14583/) (2020)
- [Transformation Networks for Target-Oriented Sentiment Classification](https://mlsanity.com/paper/1805.01086/) (2018)
- [Aspect Level Sentiment Classification with Attention-over-Attention Neural Networks](https://mlsanity.com/paper/1804.06536/) (2018)
- [Parameterized Convolutional Neural Networks for Aspect Level Sentiment Classification](https://mlsanity.com/paper/1909.06276/) (2019)
- [Exploiting Document Knowledge for Aspect-level Sentiment Classification](https://mlsanity.com/paper/1806.04346/) (2018)
- [Exploiting Coarse-to-Fine Task Transfer for Aspect-level Sentiment Classification](https://mlsanity.com/paper/1811.10999/) (2018)
- [Progressive Self-Supervised Attention Learning for Aspect-Level Sentiment Analysis](https://mlsanity.com/paper/1906.01213/) (2019)
- [Graph Ensemble Learning over Multiple Dependency Trees for Aspect-level Sentiment Classification](https://mlsanity.com/paper/2103.11794/) (2021)
- [An Iterative Multi-Knowledge Transfer Network for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2004.01935/) (2020)
- [Deep Context- and Relation-Aware Learning for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2106.03806/) (2021)
- [Selective Attention Based Graph Convolutional Networks for Aspect-Level Sentiment Classification](https://mlsanity.com/paper/1910.10857/) (2019)

## Built on by (most cited first, 11 of 11)

- [PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation](https://mlsanity.com/paper/2208.10160/) (2022)
- [Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE](https://mlsanity.com/paper/2212.01853/) (2022)
- [Knowledge Graph Augmented Network Towards Multiview Representation Learning for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2201.04831/) (2022)
- [A Contrastive Cross-Channel Data Augmentation Framework for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2204.07832/) (2022)
- [Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE](https://mlsanity.com/paper/2302.09268/) (2023)
- [Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understanding](https://mlsanity.com/paper/2204.07834/) (2022)
- [Iterative Data Generation with Large Language Models for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2407.00341/) (2024)
- [Linguistics-Vision Monotonic Consistent Network for Sign Language Production](https://mlsanity.com/paper/2412.16944/) (2024)
- [Instruct-DeBERTa: A Hybrid Approach for Aspect-based Sentiment Analysis on Textual Reviews](https://mlsanity.com/paper/2408.13202/) (2024)
- [Enhancing Aspect-based Sentiment Analysis in Tourism Using Large Language Models and Positional Information](https://mlsanity.com/paper/2409.14997/) (2024)
- [HILGEN: Hierarchically-Informed Data Generation for Biomedical NER Using Knowledgebases and Large Language Models](https://mlsanity.com/paper/2503.04930/) (2025)

## Abstract (arXiv)

Aspect-based Sentiment Analysis (ABSA) aims to determine the sentiment polarity towards an aspect. Because of the expensive and limited labelled data, the pretraining strategy has become the de-facto standard for ABSA. However, there always exists severe domain shift between the pretraining and downstream ABSA datasets, hindering the effective knowledge transfer when directly finetuning and making the downstream task performs sub-optimal. To mitigate such domain shift, we introduce a unified alignment pretraining framework into the vanilla pretrain-finetune pipeline with both instance- and knowledge-level alignments. Specifically, we first devise a novel coarse-to-fine retrieval sampling approach to select target domain-related instances from the large-scale pretraining dataset, thus aligning the instances between pretraining and target domains (First Stage). Then, we introduce a knowledge guidance-based strategy to further bridge the domain gap at the knowledge level. In practice, we formulate the model pretrained on the sampled instances into a knowledge guidance model and a learner model, respectively. On the target dataset, we design an on-the-fly teacher-student joint fine-tuning approach to progressively transfer the knowledge from the knowledge guidance model to the learner model (Second Stage). Thereby, the learner model can maintain more domain-invariant knowledge when learning new knowledge from the target dataset. In the Third Stage, the learner model is finetuned to better adapt its learned knowledge to the target dataset. Extensive experiments and analyses on several ABSA benchmarks demonstrate the effectiveness and universality of our proposed pretraining framework. Our source code and models are publicly available at https://github.com/WHU-ZQH/UIKA.