mlsanityData to 14 Oct 2026

Few-shot Mining of Naturally Occurring Inputs and Outputs

arXiv 2205.04050 ↗First posted 9 May 2022cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Creating labeled natural language training data is expensive and requires significant human effort. We mine input output examples from large corpora using a supervised mining function trained using a small seed set of only 100 examples. The mining consists of two stages -- (1) a biencoder-based recall-oriented dense search which pairs inputs with potential outputs, and (2) a crossencoder-based filter which re-ranks the output of the biencoder stage for better precision. Unlike model-generated data augmentation, our method mines naturally occurring high-quality input output pairs to mimic the style of the seed set for multiple tasks. On SQuAD-style reading comprehension, augmenting the seed set with the mined data results in an improvement of 13 F1 over a BART-large baseline fine-tuned only on the seed set. Likewise, we see improvements of 1.46 ROUGE-L on Xsum abstractive summarization.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 39 of its 44 references that are in the map. First posted on arXiv 9 May 2022.

Builds on

39 of 44 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2017Billion-scale similarity search with GPUscited by 2,497
  5. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  11. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  12. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
27 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  2. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  3. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  4. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  5. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  6. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  7. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  8. 2021Generating Datasets with Pretrained Language Modelscited by 193
  9. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  10. 2019Not Enough Data? Deep Learning to the Rescue!cited by 175
  11. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  12. 2019Revisiting Self-Training for Neural Sequence Generationcited by 155
  13. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  14. 2020Exploring and Predicting Transferability across NLP Taskscited by 128
  15. 2021Entailment as Few-Shot Learnercited by 123
  16. 2016Data Recombination for Neural Semantic Parsingcited by 111
  17. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  18. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  19. 2020Training Question Answering Models From Synthetic Datacited by 95
  20. 2020Pre-training via Paraphrasingcited by 92
  21. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
  22. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  23. 2020Cross-lingual Retrieval for Iterative Self-Supervised Trainingcited by 54
  24. 2021Neural Data Augmentation via Example Extrapolationcited by 54
  25. 2020Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentationcited by 53
  26. 2021UDALM: Unsupervised Domain Adaptation through Language Modelingcited by 33
  27. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompting-based Synthetic Data Generation for Few-Shot Question Answering
  2. 2019Mixture Content Selection for Diverse Sequence Generation
  3. 2023DisCo: Distilled Student Models Co-training for Semi-supervised Text Mining
  4. 2024Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance
  5. 2025Paired by the Teacher: Turning Unpaired Data into High-Fidelity Pairs for Low-Resource Text Generation
  6. 2025Mining Useful General Data for Low-Resource Domain Adaptation
  7. 2025Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
  8. 2024Scalable and Domain-General Abstractive Proposition Segmentation