mlsanityData to 14 Oct 2026

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

arXiv 1910.10683 ↗First posted 23 Oct 2019cs.LG cs.CL stat.MLcited by 14,599

Not on the map. Why?

Cited by 754 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

Key facts. Cited by 14,599 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 6; 2021: 1189; 2022: 2035; 2023: 3456; 2024: 3609; 2025: 2699; 2026: 1605.

14,599 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  13. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  14. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  15. 2023Qwen Technical Reportcited by 3,344
  16. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  17. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  18. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  19. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  20. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  21. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  22. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  23. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  24. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  25. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  26. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  27. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  28. 2020Learning to summarize from human feedbackcited by 2,588
  29. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  30. 2023A Survey of Large Language Modelscited by 2,576
  31. 2022Emergent Abilities of Large Language Modelscited by 2,512
  32. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  33. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  34. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  35. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  36. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  37. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  38. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212

50 most cited of 14,599 All 14,599 (JSON)