# Bootstrapping Cognitive Agents with a Large Language Model

arXiv: https://arxiv.org/abs/2403.00810 · HTML: https://mlsanity.com/paper/2403.00810/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 25 Feb 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 21 of 21)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [Language Models of Code are Few-Shot Commonsense Learners](https://mlsanity.com/paper/2210.07128/) (2022)
- [Language Models Meet World Models: Embodied Experiences Enhance Language Models](https://mlsanity.com/paper/2305.10626/) (2023)
- [Towards A Unified Agent with Foundation Models](https://mlsanity.com/paper/2307.09668/) (2023)
- [GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?](https://mlsanity.com/paper/2311.15732/) (2023)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [AmadeusGPT: a natural language interface for interactive animal behavioral analysis](https://mlsanity.com/paper/2307.04858/) (2023)
- [Towards Modeling and Influencing the Dynamics of Human Learning](https://mlsanity.com/paper/2301.00901/) (2023)
- [Integrating Diverse Knowledge Sources for Online One-shot Learning of Novel Tasks](https://mlsanity.com/paper/2208.09554/) (2022)
- [Embodied Executable Policy Learning with Language-based Scene Summarization](https://mlsanity.com/paper/2306.05696/) (2023)

## Built on by (most cited first, 9 of 9)

- [Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges](https://mlsanity.com/paper/2409.02387/) (2024)
- [From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users](https://mlsanity.com/paper/2508.17281/) (2025)
- [PoE-World: Compositional World Modeling with Products of Programmatic Experts](https://mlsanity.com/paper/2505.10819/) (2025)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [Modeling Others' Minds as Code](https://mlsanity.com/paper/2510.01272/) (2025)
- [Games Agents Play: Towards Transactional Analysis in LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2507.21354/) (2025)
- [Sample-Efficient Behavior Cloning Using General Domain Knowledge](https://mlsanity.com/paper/2501.16546/) (2025)
- [Applying Cognitive Design Patterns to General LLM Agents](https://mlsanity.com/paper/2505.07087/) (2025)
- [AutoContext: Instance-Level Context Learning for LLM Agents](https://mlsanity.com/paper/2510.02369/) (2025)

## Abstract (arXiv)

Large language models contain noisy general knowledge of the world, yet are hard to train or fine-tune. On the other hand cognitive architectures have excellent interpretability and are flexible to update but require a lot of manual work to instantiate. In this work, we combine the best of both worlds: bootstrapping a cognitive-based model with the noisy knowledge encoded in large language models. Through an embodied agent doing kitchen tasks, we show that our proposed framework yields better efficiency compared to an agent based entirely on large language models. Our experiments indicate that large language models are a good source of information for cognitive architectures, and the cognitive architecture in turn can verify and update the knowledge of large language models to a specific domain.