mlsanityData to 7 Oct 2026

RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Models

arXiv 2601.11801 ↗First posted 16 Jan 2026cs.RO cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Robot design is a nontrivial process that involves careful consideration of multiple criteria, including user specifications, kinematic structures, and visual appearance. Therefore, the design process often relies heavily on domain expertise and significant human effort. The majority of current methods are rule-based, requiring the specification of a grammar or a set of primitive components and modules that can be composed to create a design. We propose a novel automated robot design framework, RobotDesignGPT, that leverages the general knowledge and reasoning capabilities of large pre-trained vision-language models to automate the robot design synthesis process. Our framework synthesizes an initial robot design from a simple user prompt and a reference image. Our novel visual feedback approach allows us to greatly improve the design quality and reduce unnecessary manual feedback. We demonstrate that our framework can design visually appealing and kinematically valid robots inspired by nature, ranging from legged animals to flying creatures. We justify the proposed framework by conducting an ablation study and a user study.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 59 references that are in the map. First posted on arXiv 16 Jan 2026.

Builds on

45 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model3 Oct
  2. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Models27 Jun
  3. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  4. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation6 Dec
  5. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar

Other references · 40

  1. 2024GPT-4o System Cardcited by 4,467
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  6. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  7. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  8. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  9. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  10. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  11. 2024Reinforcement learningcited by 236
  12. 2021Embodied Intelligence via Learning and Evolutioncited by 154
28 more
  1. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  2. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  3. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  4. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  5. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  6. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 62
  7. 2023Prompt a Robot to Walk with Large Language Modelscited by 44
  8. 2023Creative Robot Tool Use with Large Language Modelscited by 42
  9. 2023How Can Large Language Models Help Humans in Design and Manufacturing?cited by 41
  10. 2023SayTap: Language to Quadrupedal Locomotioncited by 40
  11. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  12. 2023DiffuseBot: Breeding Soft Robots With Physics-Augmented Generative Diffusion Modelscited by 30
  13. 2024Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptionscited by 22
  14. 20243D-PreMise: Can Large Language Models Generate 3D Shapes with Sharp Features and Parametric Control?cited by 21
  15. 2024CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Modelscited by 19
  16. 2022GLSO: Grammar-guided Latent Space Optimization for Sample-efficient Robot Design Automationcited by 16
  17. 2020Emergent Hand Morphology and Control from Optimizing Robust Grasps of Diverse Objectscited by 11
  18. 2022Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfercited by 10
  19. 2022Meta Reinforcement Learning for Optimal Design of Legged Robotscited by 9
  20. 2022Multi-embodiment Legged Robot Control as a Sequence Modeling Problemcited by 9
  21. 2023MORPH: Design Co-optimization with Reinforcement Learning via a Differentiable Hardware Model Proxycited by 8
  22. 2024Text2Robot: Evolutionary Robot Design from Text Descriptionscited by 8
  23. 2024RoboMorph: Evolving Robot Morphology using Large Language Modelscited by 8
  24. 2024Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Modelscited by 8
  25. 2023Learning-based Design and Control for Quadrupedal Robots with Parallel-Elastic Actuatorscited by 7
  26. 2023Reinforcement learning for freeform robot designcited by 7
  27. 2022EvoRobogami: Co-designing with Humans in Evolutionary Robotics Experimentscited by 1
  28. 2024On the Exploration of LM-Based Soft Modular Robot Designcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset
  2. 2024What Matters in Building Vision-Language-Action Models for Generalist Robots
  3. 2025VLMgineer: Vision Language Models as Robotic Toolsmiths
  4. 2026PDDL-ART: Autonomous Symbolic Abstraction From Demonstration For Long-Horizon Robotic Manipulation Using Vision-Language Models
  5. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
  6. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks
  7. 2025Foundation Model Driven Robotics: A Comprehensive Review
  8. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges