# Few Dimensions are Enough: Fine-tuning BERT with Selected Dimensions Revealed Its Redundant Nature

arXiv: https://arxiv.org/abs/2504.04966 · HTML: https://mlsanity.com/paper/2504.04966/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Positive-Unlabeled Learning with Non-Negative Risk Estimator](https://mlsanity.com/paper/1703.00593/) (2017)
- [Learning from positive and unlabeled data: a survey](https://mlsanity.com/paper/1811.04820/) (2018)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [Automated Concatenation of Embeddings for Structured Prediction](https://mlsanity.com/paper/2010.05006/) (2020)
- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning](https://mlsanity.com/paper/1906.01378/) (2019)
- [Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection](https://mlsanity.com/paper/1707.02483/) (2017)
- [PromptNER: Prompting For Named Entity Recognition](https://mlsanity.com/paper/2305.15444/) (2023)
- [Classification from Positive, Unlabeled and Biased Negative Data](https://mlsanity.com/paper/1810.00846/) (2018)
- [Low-Resource Name Tagging Learned with Weakly Labeled Data](https://mlsanity.com/paper/1908.09659/) (2019)
- [Noisy-Labeled NER with Confidence Estimation](https://mlsanity.com/paper/2104.04318/) (2021)
- [Named Entity Recognition with Partially Annotated Training Data](https://mlsanity.com/paper/1909.09270/) (2019)
- [Scientific Information Extraction with Semi-supervised Neural Tagging](https://mlsanity.com/paper/1708.06075/) (2017)
- [De-biasing Distantly Supervised Named Entity Recognition via Causal Intervention](https://mlsanity.com/paper/2106.09233/) (2021)
- [Distantly Supervised Named Entity Recognition via Confidence-Based Multi-Class Positive and Unlabeled Learning](https://mlsanity.com/paper/2204.09589/) (2022)
- [Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models](https://mlsanity.com/paper/2311.08921/) (2023)
- [End-to-end Named Entity Recognition and Relation Extraction using Pre-trained Language Models](https://mlsanity.com/paper/1912.13415/) (2019)
- [Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss](https://mlsanity.com/paper/2108.07216/) (2021)
- [SANTA: Separate Strategies for Inaccurate and Incomplete Annotation Noise in Distantly-Supervised Named Entity Recognition](https://mlsanity.com/paper/2305.04076/) (2023)
- [Positive-unlabeled learning for binary and multi-class cell detection in histopathology images with incomplete annotations](https://mlsanity.com/paper/2302.08050/) (2023)

## Built on by (most cited first, 1 of 1)

- [Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation](https://mlsanity.com/paper/2605.07302/) (2026)

## Abstract (arXiv)

When fine-tuning BERT models for specific tasks, it is common to select part of the final layer's output and input it into a newly created fully connected layer. However, it remains unclear which part of the final layer should be selected and what information each dimension of the layers holds. In this study, we comprehensively investigated the effectiveness and redundancy of token vectors, layers, and dimensions through BERT fine-tuning on GLUE tasks. The results showed that outputs other than the CLS vector in the final layer contain equivalent information, most tasks require only 2-3 dimensions, and while the contribution of lower layers decreases, there is little difference among higher layers. We also evaluated the impact of freezing pre-trained layers and conducted cross-fine-tuning, where fine-tuning is applied sequentially to different tasks. The findings suggest that hidden layers may change significantly during fine-tuning, BERT has considerable redundancy, enabling it to handle multiple tasks simultaneously, and its number of dimensions may be excessive.