# Exploring Robustness of Prefix Tuning in Noisy Data: A Case Study in Financial Sentiment Analysis

arXiv: https://arxiv.org/abs/2211.05584 · HTML: https://mlsanity.com/paper/2211.05584/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts](https://mlsanity.com/paper/1307.5336/) (2013)
- [Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT](https://mlsanity.com/paper/2003.04985/) (2020)
- [A BERT based Sentiment Analysis and Key Entity Detection Approach for Online Financial Texts](https://mlsanity.com/paper/2001.05326/) (2020)
- [Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks](https://mlsanity.com/paper/2002.06261/) (2020)
- [On Robust Prefix-Tuning for Text Classification](https://mlsanity.com/paper/2203.10378/) (2022)
- [A Financial Service Chatbot based on Deep Bidirectional Transformers](https://mlsanity.com/paper/2003.04987/) (2020)

## Built on by (most cited first, 2 of 2)

- [Language Modeling for the Future of Finance: A Survey into Metrics, Tasks, and Data Opportunities](https://mlsanity.com/paper/2504.07274/) (2025)
- [Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning](https://mlsanity.com/paper/2408.17070/) (2024)

## Abstract (arXiv)

The invention of transformer-based models such as BERT, GPT, and RoBERTa has enabled researchers and financial companies to finetune these powerful models and use them in different downstream tasks to achieve state-of-the-art performance. Recently, a lightweight alternative (approximately 0.1% - 3% of the original model parameters) to fine-tuning, known as prefix tuning has been introduced. This method freezes the model parameters and only updates the prefix to achieve performance comparable to full fine-tuning. Prefix tuning enables researchers and financial practitioners to achieve similar results with much fewer parameters. In this paper, we explore the robustness of prefix tuning when facing noisy data. Our experiments demonstrate that fine-tuning is more robust to noise than prefix tuning -- the latter method faces a significant decrease in performance on most corrupted data sets with increasing noise levels. Furthermore, prefix tuning has high variances in the F1 scores compared to fine-tuning in many corruption methods. We strongly advocate that caution should be carefully taken when applying the state-of-the-art prefix tuning method to noisy data.