mlsanityData to 7 Oct 2026

RAFT: A Real-World Few-Shot Text Classification Benchmark

arXiv 2109.14076 ↗First posted 28 Sep 2021cs.CL cs.AI cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Large pre-trained language models have shown promise for few-shot learning, completing text-based tasks given only a few task-specific examples. Will models soon solve classification tasks that have so far been reserved for human research assistants? Existing benchmarks are not designed to measure progress in applied settings, and so don't directly answer this question. The RAFT benchmark (Real-world Annotated Few-shot Tasks) focuses on naturally occurring tasks and uses an evaluation setup that mirrors deployment. Baseline evaluations on RAFT reveal areas current techniques struggle with: reasoning over long texts and tasks with many classes. Human baselines show that some classification tasks are difficult for non-expert humans, reflecting that real-world value sometimes depends on domain expertise. Yet even non-expert human baseline F1 scores exceed GPT-3 by an average of 0.11. The RAFT datasets and leaderboard will track which model improvements translate into real-world benefits at https://raft.elicit.org .

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 28 Sep 2021.

Builds on

22 of 46 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  11. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  12. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
10 more
  1. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  2. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 361
  3. 2021True Few-Shot Learning with Language Modelscited by 343
  4. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 299
  5. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  6. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  7. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  8. 2018CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Servicecited by 72
  9. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  10. 2019Automatically Identifying Complaints in Social Mediacited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 18; 2024: 10; 2025: 2; 2026: 5.

44 papers.

Sort:

By area:

  1. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  4. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  5. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  6. 2022Efficient Few-Shot Learning Without Promptscited by 135
  7. 2022Parallel Context Windows for Large Language Modelscited by 81
  8. 2023Symbol tuning improves in-context learning in language modelscited by 72
  9. 2023Efficient Benchmarking of Language Modelscited by 71
  10. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  11. 2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
  12. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
32 more
  1. 2023PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillationcited by 25
  2. 2025RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problemscited by 21
  3. 2024Learning to Generate Answers with Citations via Factual Consistency Modelscited by 13
  4. 2022TAPE: Assessing Few-shot Russian Language Understandingcited by 11
  5. 2022Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representationscited by 11
  6. 2023TART: A plug-and-play Transformer module for task-agnostic reasoningcited by 10
  7. 2023Automated Few-shot Classification with Instruction-Finetuned Language Modelscited by 9
  8. 2023Weaker Than You Think: A Critical Look at Weakly Supervised Learningcited by 8
  9. 2023QA-NatVer: Question Answering for Natural Logic-based Fact Verificationcited by 8
  10. 2023Framing the News:From Human Perception to Large Language Model Inferencescited by 7
  11. 2023Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluationcited by 7
  12. 2024Stronger Random Baselines for In-Context Learningcited by 4
  13. 2023Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIscited by 3
  14. 2024TabVer: Tabular Fact Verification with Natural Logiccited by 3
  15. 2022Empowering Sentence Encoders with Prompting and Label Retrieval for Zero-shot Text Classificationcited by 2
  16. 2023Weakly-Supervised Questions for Zero-Shot Relation Extractioncited by 2
  17. 2023Catwalk: A Unified Language Model Evaluation Framework for Many Datasetscited by 2
  18. 2024Concept-aware Data Construction Improves In-context Learning of Language Modelscited by 2
  19. 2024SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learningcited by 2
  20. 2023Towards Zero-Shot Functional Compositionality of Language Modelscited by 1
  21. 2023PEFTT: Parameter-Efficient Fine-Tuning for low-resource Tibetan pre-trained language modelscited by 1
  22. 2024Evaluation of Few-Shot Learning for Classification Tasks in the Polish Languagecited by 1
  23. 2024Can Language Models Explain Their Own Classification Behavior?cited by 1
  24. 2024Evaluating the fairness of task-adaptive pretraining on unlabeled test data before few-shot text classificationcited by 1
  25. 2026CAOS: Conformal Aggregation of One-Shot Predictorscited by 1
  26. 2026Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleadingcited by 1
  27. 2022Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Taskscited by 0
  28. 2024PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuningcited by 0
  29. 2025FineRef: Fine-Grained Error Reflection and Correction for Long-Form Generation with Citationscited by 0
  30. 2026Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interactioncited by 0
  31. 2026Generative AI-assisted Participatory Modeling in Socio-Environmental Planning under Deep Uncertaintycited by 0
  32. 2026BatteryPass-12K: The First Dataset for the Novel Digital Battery Passport Conformance Taskcited by 0