# Sensitivity Analysis on Transferred Neural Architectures of BERT and GPT-2 for Financial Sentiment Analysis

arXiv: https://arxiv.org/abs/2207.03037 · HTML: https://mlsanity.com/paper/2207.03037/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 8 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Learning Autocompletion from Real-World Datasets](https://mlsanity.com/paper/2011.04542/) (2020)
- [On the Role of Conceptualization in Commonsense Knowledge Graph Construction](https://mlsanity.com/paper/2003.03239/) (2020)

## Abstract (arXiv)

The explosion in novel NLP word embedding and deep learning techniques has induced significant endeavors into potential applications. One of these directions is in the financial sector. Although there is a lot of work done in state-of-the-art models like GPT and BERT, there are relatively few works on how well these methods perform through fine-tuning after being pre-trained, as well as info on how sensitive their parameters are. We investigate the performance and sensitivity of transferred neural architectures from pre-trained GPT-2 and BERT models. We test the fine-tuning performance based on freezing transformer layers, batch size, and learning rate. We find the parameters of BERT are hypersensitive to stochasticity in fine-tuning and that GPT-2 is more stable in such practice. It is also clear that the earlier layers of GPT-2 and BERT contain essential word pattern information that should be maintained.