mlsanityData to 7 Oct 2026

RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis

arXiv 2402.16117 ↗First posted 25 Feb 2024cs.RO cs.AI cs.CVcited by 55

Not on the map. Why?

Cited by 20 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Robotic behavior synthesis, the problem of understanding multimodal inputs and generating precise physical control for robots, is an important part of Embodied AI. Despite successes in applying multimodal large language models for high-level understanding, it remains challenging to translate these conceptual understandings into detailed robotic actions while achieving generalization across various scenarios. In this paper, we propose a tree-structured multimodal code generation framework for generalized robotic behavior synthesis, termed RoboCodeX. RoboCodeX decomposes high-level human instructions into multiple object-centric manipulation units consisting of physical preferences such as affordance and safety constraints, and applies code generation to introduce generalization ability across various robotics platforms. To further enhance the capability to map conceptual and perceptual understanding into control commands, a specialized multimodal reasoning dataset is collected for pre-training and an iterative self-updating methodology is introduced for supervised fine-tuning. Extensive experiments demonstrate that RoboCodeX achieves state-of-the-art performance in both simulators and real robots on four different kinds of manipulation tasks and one navigation task.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 81 of its 97 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

81 of 97 references.

References · 81

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
38 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  15. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  17. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  18. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  19. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  20. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  21. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  22. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  23. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  24. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  25. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  26. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  27. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  28. 2018On Evaluation of Embodied Navigation Agentscited by 526
  29. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  30. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  31. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  32. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  33. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  34. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  35. 2022Visual Language Maps for Robot Navigationcited by 362
  36. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  37. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  38. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 19; 2026: 19.

55 papers.

Sort:

By area:

  1. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  2. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  3. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 106
  4. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  5. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  6. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 72
  7. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 68
  8. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  9. 2026ASPIRE: Agentic /Skills Discovery for Roboticscited by 44
  10. 2026Code as Agent Harnesscited by 40
  11. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  12. 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 37
38 more
  1. 2025ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editingcited by 28
  2. 2025Text2World: Benchmarking Large Language Models for Symbolic World Model Generationcited by 25
  3. 2024EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agentscited by 24
  4. 2026Playful Agentic Robot Learningcited by 18
  5. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  6. 2024DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulationcited by 15
  7. 2024DAG-Plan: Generating Directed Acyclic Dependency Graphs for Dual-Arm Cooperative Planningcited by 14
  8. 2025ReCode: Unify Plan and Action for Universal Granularity Controlcited by 14
  9. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  10. 2025Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulationcited by 10
  11. 2025Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planningcited by 10
  12. 2024VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actionscited by 8
  13. 2026Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulationcited by 8
  14. 2024RoboCAS: A Benchmark for Robotic Manipulation in Complex Object Arrangement Scenarioscited by 7
  15. 2024Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Datacited by 6
  16. 2025Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skillscited by 6
  17. 2026ETA: A New Agentic Paradigm for Embodied Taskscited by 6
  18. 2026Agent as Policy for Robotic Manipulationcited by 6
  19. 2024Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMscited by 4
  20. 2024Accelerating Pre-training of Multimodal LLMs via Chain-of-Sightcited by 4
  21. 2025RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skillscited by 4
  22. 2024E2Map: Experience-and-Emotion Map for Self-Reflective Robot Navigation with Language Modelscited by 3
  23. 2024Integrating Large Language Models for UAV Control in Simulated Environments: A Modular Interaction Approachcited by 3
  24. 2025Robotic Long-Horizon Manipulation with Progressive In-Context Code Generation and Episodic Feedbackcited by 3
  25. 2025VER: Vision Expert Transformer for Robot Learning via Foundation Distillation and Dynamic Routingcited by 3
  26. 2026SmoothVLA: Aligning Vision-Language-Action Models with Physical Constraints via Intrinsic Smoothness Optimizationcited by 3
  27. 2024Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programmingcited by 2
  28. 2025Multilingual Multimodal Software Developer for Code Generationcited by 2
  29. 2025HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agentscited by 2
  30. 2025EmbodiedCoder: Parameterized Embodied Mobile Manipulation via Modern Coding Modelcited by 2
  31. 2026EmbodiedSWE: Coding Agents for Long Horizon Dexterous Roboticscited by 2
  32. 2025Camera Control at the Edge with Language Models for Scene Understandingcited by 1
  33. 2025RoboPARA: Dual-Arm Robot Planning with Parallel Allocation and Recomposition Across Taskscited by 1
  34. 2026Evolve Vision-Language-Action Model into an Agent with On-the-fly Tool-usecited by 1
  35. 2026World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsalcited by 1
  36. 2025Rethinking Intermediate Representation for VLM-based Robot Manipulationcited by 0
  37. 2026CABTO: Context-Aware Behavior Tree Grounding for Robot Manipulationcited by 0
  38. 2026Efficient Skill Grounding via Code Refactoring with Small Language Modelscited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation
  2. 2026Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation
  3. 2024RoboMP2: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models
  4. 2026CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
  5. 2025RoboBERT: An End-to-end Multimodal Robotic Manipulation Model
  6. 2022VIMA: General Robot Manipulation with Multimodal Prompts
  7. 2025RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulation
  8. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation