mlsanityData to 7 Oct 2026

Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations

arXiv 2211.08794 ↗First posted 16 Nov 2022cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Due to the huge amount of parameters, fine-tuning of pretrained language models (PLMs) is prone to overfitting in the low resource scenarios. In this work, we present a novel method that operates on the hidden representations of a PLM to reduce overfitting. During fine-tuning, our method inserts random autoencoders between the hidden layers of a PLM, which transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. The autoencoders are plugged out after fine-tuning, so our method does not add extra parameters or increase computation cost during inference. Our method demonstrates promising performance improvement across a wide range of sequence- and token-level low-resource NLP tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 43 of its 73 references that are in the map. First posted on arXiv 16 Nov 2022.

Builds on

43 of 73 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
31 more
  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  7. 2016Deep Variational Information Bottleneckcited by 915
  8. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  10. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  11. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  12. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  13. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  14. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  15. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  16. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  17. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  18. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  19. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  20. 2021Aspect Sentiment Quad Prediction as Paraphrase Generationcited by 99
  21. 2021AEDA: An Easier Data Augmentation Technique for Text Classificationcited by 67
  22. 2022RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extractioncited by 60
  23. 2022Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networkscited by 55
  24. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  25. 2021MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NERcited by 42
  26. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  27. 2022Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillationcited by 39
  28. 2019Overfitting Mechanism and Avoidance in Deep Neural Networkscited by 30
  29. 2020Unsupervised Cross-lingual Adaptation for Sequence Tagging and Beyondcited by 22
  30. 2020UXLA: A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLPcited by 14
  31. 2022PeerDA: Data Augmentation via Modeling Peer Relation for Span Identification Taskscited by 7

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Methodcited by 3
  2. 2025Mining Useful General Data for Low-Resource Domain Adaptationcited by 3
  3. 2025Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learningcited by 0