mlsanityData to 7 Oct 2026

Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent

arXiv 2402.09844 ↗First posted 15 Feb 2024cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

The search for a general model that can operate seamlessly across multiple domains remains a key goal in machine learning research. The prevailing methodology in Reinforcement Learning (RL) typically limits models to a single task within a unimodal framework, a limitation that contrasts with the broader vision of a versatile, multi-domain model. In this paper, we present Jack of All Trades (JAT), a transformer-based model with a unique design optimized for handling sequential decision-making tasks and multi-modal data types. The JAT model demonstrates its robust capabilities and versatility by achieving strong performance on very different RL benchmarks, along with promising results on Computer Vision (CV) and Natural Language Processing (NLP) tasks, all using a single set of weights. The JAT model marks a significant step towards more general, cross-domain AI model design, and notably, it is the first model of its kind to be fully open-sourced at https://huggingface.co/jat-project/jat, including a pioneering general-purpose dataset.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 41 of its 57 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

41 of 57 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
29 more
  1. 2016OpenAI Gymcited by 1,684
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2018DeepMind Control Suitecited by 935
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2022A Generalist Agentcited by 802
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  12. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  13. 2015Policy Distillationcited by 414
  14. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  15. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  16. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  17. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  18. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  19. 2022Multi-Game Decision Transformerscited by 212
  20. 2022Online Decision Transformercited by 194
  21. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  22. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  23. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  24. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  25. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  26. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  27. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 84
  28. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  29. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 6; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  2. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  3. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Taskscited by 23
  4. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  5. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Modelscited by 19
  6. 2024Goals as Reward-Producing Programscited by 14
  7. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  8. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 13
  9. 2024REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environmentscited by 12
  10. 2024LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformercited by 8
  11. 2024ClevrSkills: Compositional Language and Visual Reasoning in Roboticscited by 7
  12. 2025Focusing on What Matters: Object-Agent-centric Tokenization for Vision Language Action modelscited by 6
8 more
  1. 2025Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformerscited by 4
  2. 2025One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planningcited by 3
  3. 2026Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Taskscited by 3
  4. 2025Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Modelcited by 2
  5. 2025Geometric Analysis of Neural Regression Collapse via Intrinsic Dimensioncited by 1
  6. 2026WestWorld: A Knowledge-Encoded Scalable Trajectory World Model for Diverse Robotic Systemscited by 1
  7. 2026Test-Time Perturbation Learning with Delayed Feedback for Vision-Language-Action Modelscited by 0
  8. 2026Towards Scalable Multi-Task Reinforcement Learning with Large Decision Modelscited by 0