mlsanityData to 7 Oct 2026

BuilderBench: The Building Blocks of Intelligent Agents

arXiv 2510.06288 ↗First posted 7 Oct 2025cs.AI cs.LGcited by 4

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Today's AI models learn primarily through mimicry and refining, so it is not surprising that they struggle to solve problems beyond the limits set by existing data. To solve novel problems, agents should acquire skills by exploring and learning through experience. Finding a scalable learning mechanism for developing agents that learn through interaction remains a major open problem. In this work, we introduce BuilderBench, a benchmark to accelerate research into agent training that centers open-ended exploration. BuilderBench requires agents to learn how to build any structure using blocks. BuilderBench is equipped with (1) a simulator of a robot interacting with various physical blocks, and (2) a task-suite with over 50 diverse target structures that are carefully curated to test an understanding of physics, mathematics, and long-horizon planning. Agents are provided with a target structure at the start, and can interact with the environment for multiple episodes to experiment and learn various skills for building the structure. Solving these tasks requires embodied reasoning in a way that is not reflected in words but rather in actions, experimenting with different strategies and piecing them together. Our experiments with multiple state-of-the-art frontier language model based agents and tabula rasa reinforcement learning algorithms show that these agents cannot solve any of the non-trivial tasks in the BuilderBench. Our analysis throws light on the lack of exploration abilities in these models.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 68 of its 103 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

68 of 103 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023See and Think: Embodied Agent in Virtual Environment26 Nov
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 66

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  11. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
38 more
  1. 2016Matching Networks for One Shot Learningcited by 2,246
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2023Mastering Diverse Domains through World Modelscited by 1,193
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2018Exploration by Random Network Distillationcited by 966
  9. 2018DeepMind Control Suitecited by 935
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  11. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  12. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  13. 2016Deep Exploration via Bootstrapped DQNcited by 518
  14. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  15. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  16. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  17. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  18. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  19. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  20. 2019A Baseline for Few-Shot Image Classificationcited by 298
  21. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  22. 2016Variational Intrinsic Controlcited by 262
  23. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  24. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  25. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  26. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  27. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  28. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  29. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  30. 2020The NetHack Learning Environmentcited by 196
  31. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  32. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  33. 2019Efficient Exploration via State Marginal Matchingcited by 186
  34. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  35. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  36. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  37. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  38. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
  2. 2026VIA: Visual Interface Agent for Robot Controlcited by 5
  3. 2025Benchmarking the Generality of Vision-Language-Action Modelscited by 0
  4. 2026LeAct: Learning to Reason from Expert Actionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Learning to Guide and to Be Guided in the Architect-Builder Problem
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
  3. 2025Generative AI in Embodied Systems: System-Level Analysis of Performance, Efficiency and Scalability
  4. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Models
  5. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning
  6. 2025Think, Act, Learn: A Framework for Autonomous Robotic Agents using Closed-Loop Large Language Models
  7. 2026Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards
  8. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents