mlsanityData to 7 Oct 2026

Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset

arXiv 2409.17126 ↗First posted 25 Sep 2024cs.RO cs.AI cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Generative AI systems have shown impressive capabilities in creating text, code, and images. Inspired by the rich history of research in industrial ''Design for Assembly'', we introduce a novel problem: Generative Design-for-Robot-Assembly (GDfRA). The task is to generate an assembly based on a natural language prompt (e.g., ''giraffe'') and an image of available physical components, such as 3D-printed blocks. The output is an assembly, a spatial arrangement of these components, and instructions for a robot to build this assembly. The output must 1) resemble the requested object and 2) be reliably assembled by a 6 DoF robot arm with a suction gripper. We then present Blox-Net, a GDfRA system that combines generative vision language models with well-established methods in computer vision, simulation, perturbation analysis, motion planning, and physical robot experimentation to solve a class of GDfRA problems with minimal human supervision. Blox-Net achieved a Top-1 accuracy of 63.5% in the ''recognizability'' of its designed assemblies (eg, resembling giraffe as judged by a VLM). These designs, after automated perturbation redesign, were reliably assembled by a robot, achieving near-perfect success across 10 consecutive assembly iterations with human intervention only during reset prior to assembly. Surprisingly, this entire design process from textual word (''giraffe'') to reliable physical assembly is performed with zero human intervention.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 69 of its 103 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

69 of 103 references.

References · 69

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  9. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  10. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  11. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  12. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  13. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  14. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  15. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  16. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  17. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  18. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  19. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  20. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  21. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  22. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  23. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  24. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  25. 2021RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matchingcited by 348
  26. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  27. 2023Large Language Models as General Pattern Machinescited by 196
  28. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  29. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  30. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  31. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  32. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  33. 2019Meta-Sim: Learning to Generate Synthetic Datasetscited by 128
  34. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  35. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  36. 2022Generating Long Videos of Dynamic Scenescited by 113
  37. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  38. 2019Improved Conditional VRNNs for Video Predictioncited by 104

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 72
  2. 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 37
  3. 2025Generating Physically Stable and Buildable Brick Structures from Textcited by 23
  4. 2025Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Modelscited by 13
  5. 2025VLMgineer: Vision Language Models as Robotic Toolsmithscited by 9
  6. 2025Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Modelscited by 5
  7. 2025Hierarchical DLO Routing with Reinforcement Learning and In-Context Vision-language Modelscited by 2
  8. 2025From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoningcited by 1
  9. 2025Prompt-to-Product: Generative Assembly via Bimanual Manipulationcited by 1
  10. 2025Prompt2Craft: Generating Functional Craft Assemblies with LLMscited by 0
  11. 2026Robot-Assisted Social Dining as a White Glove Servicecited by 0
  12. 2026EUPHORIA: Efficient Universal Planning via Hybrid Optimization for Robust Industrial Robotic Assemblycited by 0
2 more
  1. 2026RoboVista: Evaluating Vision Language Models for Diverse Robot Applicationscited by 0
  2. 2026ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models
  2. 2026RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Models
  3. 2024AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries
  4. 2026GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learning
  5. 2025VLM-driven Skill Selection for Robotic Assembly Tasks
  6. 2026Brick-Composer: Using MLLMs for Assembly with Diverse Bricks
  7. 2025Evolution 6.0: Robot Evolution through Generative Design ↗
  8. 2024Towards Natural Language-Driven Assembly Using Foundation Models