mlsanityData to 14 Oct 2026

A Generalist Agent

arXiv 2205.06175 ↗First posted 12 May 2022cs.AI cs.CL cs.LGcited by 808

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.

Key facts. Cited by 808 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 86 of its 105 references that are in the map. First posted on arXiv 12 May 2022.

Builds on

86 of 105 references.

References · 86

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2016Layer Normalizationcited by 5,669
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2020GLU Variants Improve Transformercited by 1,804
  13. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  14. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  15. 2016Concrete Problems in AI Safetycited by 1,642
  16. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  17. 2018Group Normalizationcited by 1,524
  18. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  19. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  20. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  21. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  22. 2016Progressive Neural Networkscited by 1,361
  23. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  24. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  25. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  26. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  27. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  28. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  29. 2018Model Cards for Model Reportingcited by 1,080
  30. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  31. 2016Deep Networks with Stochastic Depthcited by 979
  32. 2021Ethical and social risks of harm from Language Modelscited by 966
  33. 2018DeepMind Control Suitecited by 956
  34. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  35. 2020Quantifying Attention Flow in Transformerscited by 775
  36. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  37. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  38. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 100; 2023: 232; 2024: 192; 2025: 158; 2026: 126.

808 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,344
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2023Mastering Diverse Domains through World Modelscited by 1,253
  9. 2022Planning-oriented Autonomous Drivingcited by 1,087
  10. 2024Genie: Generative Interactive Environmentscited by 747
  11. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
38 more
  1. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  2. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  3. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  4. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  5. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  7. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  8. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  9. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  10. 2023Language Models can Solve Computer Taskscited by 372
  11. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  12. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  13. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  14. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  15. 2023ClimaX: A foundation model for weather and climatecited by 337
  16. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 304
  17. 2023Tool Learning with Foundation Modelscited by 302
  18. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  19. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  20. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  21. 2023Towards Generalist Biomedical AIcited by 289
  22. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 282
  23. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  24. 2023Imitating Human Behaviour with Diffusion Modelscited by 263
  25. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  26. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  27. 2024Generative Representational Instruction Tuningcited by 233
  28. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  29. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 225
  30. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 220
  31. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  32. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  33. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  34. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 207
  35. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  36. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192
  37. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  38. 2022Pure Transformers are Powerful Graph Learnerscited by 188

50 most cited of 808 All 808 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Survey
  2. 2025Small Language Models are the Future of Agentic AI
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Survey
  4. 2023Agents: An Open-source Framework for Autonomous Language Agents
  5. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
  6. 2025Graph-Augmented Large Language Model Agents: Current Progress and Future Prospects
  7. 2025GTM: Simulating the World of Tools for AI Agents
  8. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers