# Unsupervised Cross-Task Generalization via Retrieval Augmentation

arXiv: https://arxiv.org/abs/2204.07937 · HTML: https://mlsanity.com/paper/2204.07937/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 17 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data](https://mlsanity.com/paper/2203.08773/) (2022)
- [Differentiable Open-Ended Commonsense Reasoning](https://mlsanity.com/paper/2010.14439/) (2020)
- [To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning](https://mlsanity.com/paper/2104.08078/) (2021)

## Built on by (most cited first, 30 of 42)

- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [Exploring the Benefits of Training Expert Language Models over Instruction Tuning](https://mlsanity.com/paper/2302.03202/) (2023)
- [Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In](https://mlsanity.com/paper/2305.17331/) (2023)
- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [Improving In-Context Few-Shot Learning via Self-Supervised Training](https://mlsanity.com/paper/2205.01703/) (2022)
- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [Data-Efficient Finetuning Using Cross-Task Nearest Neighbors](https://mlsanity.com/paper/2212.00196/) (2022)
- [Human-Instruction-Free LLM Self-Alignment with Limited Samples](https://mlsanity.com/paper/2401.06785/) (2024)
- [Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners](https://mlsanity.com/paper/2210.02969/) (2022)
- [Instruction Tuned Models are Quick Learners](https://mlsanity.com/paper/2306.05539/) (2023)
- [ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization](https://mlsanity.com/paper/2311.13171/) (2023)
- [Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization](https://mlsanity.com/paper/2210.09175/) (2022)
- [Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Data](https://mlsanity.com/paper/2302.00674/) (2023)
- [Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts](https://mlsanity.com/paper/2205.12701/) (2022)
- [TaskWeb: Selecting Better Source Tasks for Multi-task NLP](https://mlsanity.com/paper/2305.13256/) (2023)
- [Automated Few-shot Classification with Instruction-Finetuned Language Models](https://mlsanity.com/paper/2305.12576/) (2023)
- [What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement](https://mlsanity.com/paper/2402.01865/) (2024)
- [Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2404.03514/) (2024)
- [Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks](https://mlsanity.com/paper/2210.00185/) (2022)
- [QA-NatVer: Question Answering for Natural Logic-based Fact Verification](https://mlsanity.com/paper/2310.14198/) (2023)
- [Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference](https://mlsanity.com/paper/2406.07007/) (2024)
- [Retrieval Augmented Deep Anomaly Detection for Tabular Data](https://mlsanity.com/paper/2401.17052/) (2024)
- [Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Tasks](https://mlsanity.com/paper/2404.16418/) (2024)
- [From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems](https://mlsanity.com/paper/2506.04565/) (2025)
- [Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorer](https://mlsanity.com/paper/2311.06720/) (2023)
- [Zero-Shot Fact Verification via Natural Logic and Large Language Models](https://mlsanity.com/paper/2410.03341/) (2024)

## Abstract (arXiv)

Humans can perform unseen tasks by recalling relevant skills acquired previously and then generalizing them to the target tasks, even if there is no supervision at all. In this paper, we aim to improve this kind of cross-task generalization ability of massive multi-task language models, such as T0 and FLAN, in an unsupervised setting. We propose a retrieval-augmentation method named ReCross that takes a few unlabelled examples as queries to retrieve a small subset of upstream data and uses them to update the multi-task model for better generalization. ReCross is a straightforward yet effective retrieval method that combines both efficient dense retrieval and effective pair-wise reranking. Our results and analysis show that it significantly outperforms both non-retrieval methods and other baseline methods.