mlsanityData to 7 Oct 2026

REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environments

arXiv 2412.04759 ↗First posted 6 Dec 2024cs.AIcited by 12

On the map: Retrieval-augmented generation → Agentic RAG, agents and memory

Abstract (arXiv)

Building generalist agents that can rapidly adapt to new environments is a key challenge for deploying AI in the digital and real worlds. Is scaling current agent architectures the most effective way to build generalist agents? We propose a novel approach to pre-train relatively small policies on relatively small datasets and adapt them to unseen environments via in-context learning, without any finetuning. Our key idea is that retrieval offers a powerful bias for fast adaptation. Indeed, we demonstrate that even a simple retrieval-based 1-nearest neighbor agent offers a surprisingly strong baseline for today's state-of-the-art generalist agents. From this starting point, we construct a semi-parametric agent, REGENT, that trains a transformer-based policy on sequences of queries and retrieved neighbors. REGENT can generalize to unseen robotics and game-playing environments via retrieval augmentation and in-context learning, achieving this with up to 3x fewer parameters and up to an order-of-magnitude fewer pre-training datapoints, significantly outperforming today's state-of-the-art generalist agents. Website: https://kaustubhsridhar.github.io/regent-research

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation. Builds on 52 of its 63 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

52 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RL9 Oct
  2. 2024RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents6 Feb

Other references · 50

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2016Matching Networks for One Shot Learningcited by 2,246
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  12. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
38 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  5. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  6. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  7. 2022A Generalist Agentcited by 802
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2019A Closer Look at Few-shot Classificationcited by 679
  10. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  11. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  12. 2024The Faiss librarycited by 370
  13. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  14. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  15. 2022Multi-Game Decision Transformerscited by 212
  16. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  17. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  18. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  19. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  20. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  21. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  22. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  23. 2024BAKU: An Efficient Transformer for Multi-Task Policy Learningcited by 80
  24. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
  25. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
  26. 2024Pushing the Limits of Cross-Embodiment Learning for Manipulation and Navigationcited by 60
  27. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  28. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 41
  29. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  30. 2024Retrieval-Augmented Embodied Agentscited by 30
  31. 2023Polybot: Training One Policy Across Robots While Embracing Variabilitycited by 28
  32. 2024Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agentcited by 20
  33. 2021Improving Neural Network Robustness via Persistency of Excitationcited by 14
  34. 2023Memory-Consistent Neural Networks for Imitation Learningcited by 14
  35. 2021Nearest Neighbour Few-Shot Learning for Cross-lingual Classificationcited by 8
  36. 2022Guaranteed Conformance of Neurosymbolic Models to Natural Constraintscited by 6
  37. 2022Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templatescited by 2
  38. 2024Automating Weak Label Generation for Data Programming with Clinicians in the Loopcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  2. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Modelscited by 19
  3. 2025Vintix: Action Model via In-Context Reinforcement Learningcited by 8
  4. 2026AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agentscited by 5
  5. 2025Universal Retrieval for Multimodal Trajectory Modelingcited by 3
  6. 2025Policy Compatible Skill Incremental Learning via Lazy Learning Interfacecited by 2
  7. 2026SAIL: Test-Time Scaling for In-Context Imitation Learning with VLMcited by 2
  8. 2026ARMove: Learning to Predict Human Mobility through Agentic Reasoningcited by 1
  9. 2026Difference-Aware Retrieval Policies for Imitation Learningcited by 0
  10. 2026Towards Scalable Multi-Task Reinforcement Learning with Large Decision Modelscited by 0
  11. 2026Vector Search As Nearest Neighbor Matching: RAG-based Policy Learning in Causal Inferencecited by 0
  12. 2026ICI-VLA: In-Context Imitation with Spatiotemporally Aligned Demonstrations for Vision-Language-Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Retrieval-Augmented LLM Agents: Learning to Learn from Experience
  2. 2022Large-Scale Retrieval for Reinforcement Learning
  3. 2026Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization
  4. 2025MaskSearch: A Universal Pre-Training Framework to Enhance Agentic Search Capability
  5. 2026Experience as a Compass: Multi-agent RAG with Evolving Orchestration and Agent Prompts
  6. 2026Comparative Approaches to Agent Retrieval over Large Skill Libraries
  7. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
  8. 2026Can Generalist Agents Automate Data Curation?