mlsanityData to 7 Oct 2026

Scaling Up Models and Data with textttt5x and textttseqio

arXiv 2203.17189 ↗First posted 31 Mar 2022cs.LG cs.CLcited by 187

Not on the map. Why?

Abstract (arXiv)

Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can be complicated due to various factors including the need to distribute computation on supercomputer clusters (e.g., TPUs), prevent bottlenecks when infeeding data, and ensure reproducible results. In this work, we present two software libraries that ease these issues: textttt5x simplifies the process of building and training large language models at scale while maintaining ease of use, and textttseqio provides a task-based API for simple creation of fast and reproducible training data and evaluation pipelines. These open-source libraries have been used to train models with hundreds of billions of parameters on datasets with multiple terabytes of training data. Along with the libraries, we release configurations and instructions for T5-like encoder-decoder models as well as GPT-like decoder-only architectures. textttt5x and textttseqio are open source and available at https://github.com/google-research/t5x and https://github.com/google/seqio, respectively.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Builds on 14 of its 25 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

14 of 25 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  9. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  10. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  11. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  12. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
2 more
  1. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  2. 2018Tensor2Tensor for Neural Machine Translationcited by 85

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 43; 2023: 67; 2024: 54; 2025: 16; 2026: 7.

187 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2025Gemma 3 Technical Reportcited by 1,686
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  8. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  10. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  12. 2022Efficiently Scaling Transformer Inferencecited by 529
38 more
  1. 2023The Falcon Series of Open Language Modelscited by 526
  2. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  4. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 401
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  6. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  7. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  8. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  9. 2022Confident Adaptive Language Modelingcited by 297
  10. 2022UL2: Unifying Language Learning Paradigmscited by 292
  11. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  12. 2023Nash Learning from Human Feedbackcited by 220
  13. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  14. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  15. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  16. 2022TALM: Tool Augmented Language Modelscited by 197
  17. 2023Recommender Systems with Generative Retrievalcited by 191
  18. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  19. 2026Gemma 4 Technical Reportcited by 187
  20. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  21. 2022Translation between Molecules and Natural Languagecited by 173
  22. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  23. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  24. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  25. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  26. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  27. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  28. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  29. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  30. 2023Rich Human Feedback for Text-to-Image Generationcited by 139
  31. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  32. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  33. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  34. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  35. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  36. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
  37. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  38. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90

50 most cited of 187 All 187 (JSON)