mlsanityData to 7 Oct 2026

Embedding Hallucination for Few-Shot Language Fine-tuning

arXiv 2205.01307 ↗First posted 3 May 2022cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Few-shot language learners adapt knowledge from a pre-trained model to recognize novel classes from a few-labeled sentences. In such settings, fine-tuning a pre-trained language model can cause severe over-fitting. In this paper, we propose an Embedding Hallucination (EmbedHalluc) method, which generates auxiliary embedding-label pairs to expand the fine-tuning dataset. The hallucinator is trained by playing an adversarial game with the discriminator, such that the hallucinated embedding is indiscriminative to the real ones in the fine-tuning dataset. By training with the extended dataset, the language learner effectively learns from the diverse hallucinated embeddings to overcome the over-fitting issue. Experiments demonstrate that our proposed method is effective in a wide range of language tasks, outperforming current fine-tuning methods. Further, we show that EmbedHalluc outperforms other methods that address this over-fitting problem, such as common data augmentation, semi-supervised pseudo-labeling, and regularization. The code will be made available at: https://github.com/yiren-jian/EmbedHalluc.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 3 May 2022.

Builds on

38 of 53 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  8. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  9. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  10. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  11. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  12. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
26 more
  1. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  2. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  3. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  4. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  5. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  6. 2018Low-Shot Learning from Imaginary Datacited by 181
  7. 2021Few-NERD: A Few-Shot Named Entity Recognition Datasetcited by 150
  8. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  9. 2020Adversarial Feature Hallucination Networks for Few-Shot Learningcited by 110
  10. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  11. 2021AEDA: An Easier Data Augmentation Technique for Text Classificationcited by 67
  12. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  13. 2020Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentationcited by 53
  14. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
  15. 2019A Closer Look At Feature Space Data Augmentation For Few-Shot Intent Classificationcited by 45
  16. 2021Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learningcited by 35
  17. 2019Enhanced generative adversarial network for 3D brain MRI super-resolutioncited by 27
  18. 2021Learning from Miscellaneous Other-Class Words for Few-shot Named Entity Recognitioncited by 21
  19. 2020Meta-Learning for Few-Shot NMT Adaptationcited by 20
  20. 2020Few-Shot Complex Knowledge Base Question Answering via Meta Reinforcement Learningcited by 16
  21. 2021Label Hallucination for Few-Shot Classificationcited by 11
  22. 2021Adversarial Semantic Hallucination for Domain Generalized Semantic Segmentationcited by 9
  23. 2021Few-shot learning through contextual data augmentationcited by 8
  24. 2021Tensor feature hallucination for few-shot learningcited by 6
  25. 2021Semi-Supervised Few-Shot Intent Classification and Slot Fillingcited by 3
  26. 2021MetaPix: Domain Transfer for Semantic Segmentation by Meta Pixel Weightingcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 1.

5 papers.

  1. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 55
  2. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  3. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17
  4. 2025Hybrid Retrieval for Hallucination Mitigation in Large Language Models: A Comparative Analysiscited by 5
  5. 2024Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion for Efficient Inference Intervention in Large Language Modelcited by 0