mlsanityData to 14 Oct 2026

Representation Projection Invariance Mitigates Representation Collapse

arXiv 2205.11603 ↗First posted 23 May 2022cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Fine-tuning contextualized representations learned by pre-trained language models remains a prevalent practice in NLP. However, fine-tuning can lead to representation degradation (also known as representation collapse), which may result in instability, sub-optimal performance, and weak generalization. In this paper, we propose Representation Projection Invariance (REPINA), a novel regularization method to maintain the information content of representation and reduce representation collapse during fine-tuning by discouraging undesirable changes in the representations. We study the empirical behavior of the proposed regularization in comparison to 5 comparable baselines across 13 language understanding tasks (GLUE benchmark and six additional datasets). When evaluating in-domain performance, REPINA consistently outperforms other baselines on most tasks (10 out of 13). We also demonstrate its effectiveness in few-shot settings and robustness to label perturbation. As a by-product, we extend previous studies of representation collapse and propose several metrics to quantify it. Our empirical findings show that our approach is significantly more effective at mitigating representation collapse.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 45 of its 64 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

45 of 64 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
33 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  4. 2017A Survey on Multi-Task Learningcited by 840
  5. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  6. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  9. 2017Learning multiple visual domains with residual adapterscited by 557
  10. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  11. 2021A Survey of Data Augmentation Approaches for NLPcited by 409
  12. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  13. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  14. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  15. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  16. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 276
  17. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
  18. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  19. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  20. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  21. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  22. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  23. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  24. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  25. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  26. 2019Structural Scaffolds for Citation Intent Classification in Scientific Publicationscited by 130
  27. 2017Dataset Augmentation in Feature Spacecited by 124
  28. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  29. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 89
  30. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  31. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  32. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  33. 2020PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Modelscited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 3.

10 papers.

  1. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
  2. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  3. 2025Exploring Representation Invariance in Finetuningcited by 2
  4. 2025Prompt and Parameter Co-Optimization for Large Language Modelscited by 2
  5. 2025Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagationcited by 1
  6. 2026Model soups need only one ingredientcited by 1
  7. 2026Continual Learning Mechanisms Compose for Long-Horizon Memorizationcited by 1
  8. 2024HateDebias: On the Diversity and Variability of Hate Speech Debiasingcited by 0
  9. 2025TRepLiNa: Layer-wise CKA+REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8Bcited by 0
  10. 2026Representation Collapse in Machine Translation Through the Lens of Angular Dispersioncited by 0