mlsanityData to 13 Oct 2026

UL2: Unifying Language Learning Paradigms

arXiv 2205.05131 ↗First posted 10 May 2022cs.CLcited by 294

Not on the map. Why?

Cited by 30 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized & unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 & GPT-like models across multiple diverse setups. By scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised finetuning based NLP tasks. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization. On 0-shot MMLU, UL2 20B outperforms T0 and T5 models. UL2 20B also works well with chain-of-thought prompting and reasoning, making it an appealing choice for research into reasoning at a small to medium scale of 20B parameters. Finally, we apply FLAN instruction tuning to the UL2 20B model, achieving MMLU and Big-Bench scores competitive to FLAN-PaLM 62B. We release Flax-based T5X checkpoints for the UL2 20B & Flan-UL2 20B.

Key facts. Cited by 294 papers in the map (data through 13 Oct 2026). Builds on 106 of its 144 references that are in the map. First posted on arXiv 10 May 2022.

Builds on

106 of 144 references.

References · 106

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  9. 2020Scaling Laws for Neural Language Modelscited by 6,494
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
38 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,467
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,319
  10. 2018Deep contextualized word representationscited by 2,286
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,171
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,133
  13. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  14. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,815
  15. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,805
  16. 2020GLU Variants Improve Transformercited by 1,759
  17. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  18. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
  19. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  20. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,312
  21. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219
  22. 2015Teaching Machines to Read and Comprehendcited by 1,045
  23. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 900
  24. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 896
  25. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 865
  26. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 854
  27. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 833
  28. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 804
  29. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 738
  30. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  31. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  32. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 646
  33. 2017The NarrativeQA Reading Comprehension Challengecited by 592
  34. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 533
  35. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 513
  36. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 507
  37. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 478
  38. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 18; 2023: 117; 2024: 100; 2025: 43; 2026: 16.

294 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,412
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,022
  3. 2023A Survey of Large Language Modelscited by 2,559
  4. 2023PaLM 2 Technical Reportcited by 1,162
  5. 2023BloombergGPT: A Large Language Model for Financecited by 932
  6. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 913
  7. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 808
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 677
  9. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 666
  10. 2023A Comprehensive Overview of Large Language Modelscited by 644
  11. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 642
  12. 2023Instruction Tuning for Large Language Models: A Surveycited by 622
38 more
  1. 2023The Falcon Series of Open Language Modelscited by 533
  2. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  3. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 482
  4. 2023Learning Interactive Real-World Simulatorscited by 445
  5. 2023Scaling Data-Constrained Language Modelscited by 398
  6. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 378
  7. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 362
  8. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 351
  9. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 350
  10. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  11. 2022Teaching Small Language Models to Reasoncited by 282
  12. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  13. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 277
  14. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 264
  15. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  16. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  17. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  18. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 228
  19. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 205
  20. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 203
  21. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 193
  22. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 186
  23. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 177
  24. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  25. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 163
  26. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 162
  27. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  28. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  29. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 158
  30. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  31. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 141
  32. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 135
  33. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 126
  34. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  35. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  36. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  37. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  38. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 121

50 most cited of 294 All 294 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Model Merging in Pre-training of Large Language Models
  2. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
  3. 2024Tuning Language Models by Mixture-of-Depths Ensemble
  4. 2026Reducing Pretraining-Generation Mismatch in Diffusion Language Models
  5. 2026Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
  6. 2025Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
  7. 2024Pruning Large Language Models with Semi-Structural Adaptive Sparse Training
  8. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning