mlsanityData to 7 Oct 2026

Unsupervised Cross-Task Generalization via Retrieval Augmentation

arXiv 2204.07937 ↗First posted 17 Apr 2022cs.CL cs.AI cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Humans can perform unseen tasks by recalling relevant skills acquired previously and then generalizing them to the target tasks, even if there is no supervision at all. In this paper, we aim to improve this kind of cross-task generalization ability of massive multi-task language models, such as T0 and FLAN, in an unsupervised setting. We propose a retrieval-augmentation method named ReCross that takes a few unlabelled examples as queries to retrieve a small subset of upstream data and uses them to update the multi-task model for better generalization. ReCross is a straightforward yet effective retrieval method that combines both efficient dense retrieval and effective pair-wise reranking. Our results and analysis show that it significantly outperforms both non-retrieval methods and other baseline methods.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 17 Apr 2022.

Builds on

22 of 28 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
10 more
  1. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  2. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  3. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  4. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  5. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  6. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  7. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  8. 2022Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Datacited by 61
  9. 2020Differentiable Open-Ended Commonsense Reasoningcited by 29
  10. 2021To Share or not to Share: Predicting Sets of Sources for Model Transfer Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 16; 2024: 10; 2025: 2; 2026: 1.

42 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  3. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  4. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  5. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  6. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  7. 2023Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-Incited by 71
  8. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  9. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  10. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  11. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  12. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samplescited by 25
30 more
  1. 2022Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learnerscited by 22
  2. 2023Instruction Tuned Models are Quick Learnerscited by 21
  3. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  4. 2022Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalizationcited by 15
  5. 2023Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Datacited by 13
  6. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  7. 2023TaskWeb: Selecting Better Source Tasks for Multi-task NLPcited by 11
  8. 2023Automated Few-shot Classification with Instruction-Finetuned Language Modelscited by 9
  9. 2024What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinementcited by 9
  10. 2024Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Modelscited by 9
  11. 2022Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Taskscited by 8
  12. 2023QA-NatVer: Question Answering for Natural Logic-based Fact Verificationcited by 8
  13. 2024Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inferencecited by 7
  14. 2024Retrieval Augmented Deep Anomaly Detection for Tabular Datacited by 6
  15. 2024Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Taskscited by 6
  16. 2023Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorercited by 5
  17. 2024Zero-Shot Fact Verification via Natural Logic and Large Language Modelscited by 5
  18. 2022Z-BERT-A: a zero-shot Pipeline for Unknown Intent detectioncited by 4
  19. 2023Improving Few-shot Generalization of Safety Classifiers via Data Augmented Parameter-Efficient Fine-Tuningcited by 4
  20. 2024Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learningcited by 4
  21. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 4
  22. 2023CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogscited by 3
  23. 2024TabVer: Tabular Fact Verification with Natural Logiccited by 3
  24. 2022Leveraging Natural Supervision for Language Representation Learning and Generationcited by 2
  25. 2022Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Promptcited by 2
  26. 2022Data-Efficiency with a Single GPU: An Exploration of Transfer Methods for Small Language Modelscited by 2
  27. 2023Toward Zero-Shot Instruction Followingcited by 2
  28. 2025Data-Efficient Hate Speech Detection via Cross-Lingual Nearest Neighbor Retrieval with Limited Labeled Datacited by 2
  29. 2022Neural Retriever and Go Beyond: A Thesis Proposalcited by 1
  30. 2026Multimodal UNcommonsense: From Odd to Ordinary and Ordinary to Oddcited by 0