mlsanityData to 7 Oct 2026

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

arXiv 2104.08835 ↗First posted 18 Apr 2021cs.CL cs.LGcited by 142

Not on the map. Why?

Abstract (arXiv)

Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit, a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CrossFit and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from open-access NLP datasets and converted to a unified text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Builds on 115 of its 183 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

115 of 183 references.

References · 115

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
38 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  10. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  11. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  12. 2018Neural Network Acceptability Judgmentscited by 952
  13. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  14. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  15. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  16. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  17. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  18. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  19. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  20. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  21. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  22. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  23. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  24. 2019ELI5: Long Form Question Answeringcited by 567
  25. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  26. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  27. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  28. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  29. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  30. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  31. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  32. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  33. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
  34. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  35. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  36. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  37. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  38. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 361

50 most cited of 115 All 115 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 37; 2023: 50; 2024: 28; 2025: 8; 2026: 5.

142 papers.

Sort:

By area:

  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  6. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  7. 2023A Comprehensive Overview of Large Language Modelscited by 639
  8. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  9. 2021MetaICL: Learning to Learn In Contextcited by 450
  10. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  11. 2021True Few-Shot Learning with Language Modelscited by 343
  12. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
38 more
  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  3. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  4. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  5. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  6. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  7. 2021Meta-learning via Language Model In-context Tuningcited by 163
  8. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 155
  9. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  10. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  11. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  12. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  13. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  14. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  15. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  16. 2023Revisiting Relation Extraction in the era of Large Language Modelscited by 100
  17. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  18. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 85
  19. 2023Modular Deep Learningcited by 82
  20. 2023Symbol tuning improves in-context learning in language modelscited by 72
  21. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  22. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  23. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  24. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  25. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  26. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  27. 2022ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalizationcited by 54
  28. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  29. 2023Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuningcited by 48
  30. 2022CSL: A Large-scale Chinese Scientific Literature Datasetcited by 45
  31. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  32. 2021RAFT: A Real-World Few-Shot Text Classification Benchmarkcited by 44
  33. 2022Unsupervised Cross-Task Generalization via Retrieval Augmentationcited by 42
  34. 2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
  35. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  36. 2023Chinese Open Instruction Generalist: A Preliminary Releasecited by 35
  37. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  38. 2024Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statementscited by 33

50 most cited of 142 All 142 (JSON)