# Removing Non-Stationary Knowledge From Pre-Trained Language Models for Entity-Level Sentiment Classification in Finance

arXiv: https://arxiv.org/abs/2301.03136 · HTML: https://mlsanity.com/paper/2301.03136/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 36 references that are in the map.
- First posted on arXiv 9 Jan 2023.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [Perturbation Sensitivity Analysis to Detect Unintended Model Biases](https://mlsanity.com/paper/1910.04210/) (2019)
- [CrossAug: A Contrastive Data Augmentation Method for Debiasing Fact Verification Models](https://mlsanity.com/paper/2109.15107/) (2021)

## Built on by (most cited first, 7 of 7)

- [HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models](https://mlsanity.com/paper/2309.02706/) (2023)
- [Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap](https://mlsanity.com/paper/2501.02448/) (2025)
- [Beyond Classification: Financial Reasoning in State-of-the-Art Language Models](https://mlsanity.com/paper/2305.01505/) (2023)
- [TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Model Bring? -- A Case Study on Korea Financial Texts](https://mlsanity.com/paper/2502.07131/) (2025)
- [KPC-cF: Aspect-Based Sentiment Analysis via Implicit-Feature Alignment with Corpus Filtering](https://mlsanity.com/paper/2407.00342/) (2024)
- [Can ChatGPT Overcome Behavioral Biases in the Financial Sector? Classify-and-Rethink: Multi-Step Zero-Shot Reasoning in the Gold Investment](https://mlsanity.com/paper/2411.13599/) (2024)
- [AI PB: A Grounded Generative Agent for Personalized Investment Insights](https://mlsanity.com/paper/2510.20099/) (2025)

## Abstract (arXiv)

Extraction of sentiment signals from news text, stock message boards, and business reports, for stock movement prediction, has been a rising field of interest in finance. Building upon past literature, the most recent works attempt to better capture sentiment from sentences with complex syntactic structures by introducing aspect-level sentiment classification (ASC). Despite the growing interest, however, fine-grained sentiment analysis has not been fully explored in non-English literature due to the shortage of annotated finance-specific data. Accordingly, it is necessary for non-English languages to leverage datasets and pre-trained language models (PLM) of different domains, languages, and tasks to best their performance. To facilitate finance-specific ASC research in the Korean language, we build KorFinASC, a Korean aspect-level sentiment classification dataset for finance consisting of 12,613 human-annotated samples, and explore methods of intermediate transfer learning. Our experiments indicate that past research has been ignorant towards the potentially wrong knowledge of financial entities encoded during the training phase, which has overestimated the predictive power of PLMs. In our work, we use the term "non-stationary knowledge'' to refer to information that was previously correct but is likely to change, and present "TGT-Masking'', a novel masking pattern to restrict PLMs from speculating knowledge of the kind. Finally, through a series of transfer learning with TGT-Masking applied we improve 22.63% of classification accuracy compared to standalone models on KorFinASC.