mlsanityData to 7 Oct 2026

Language-Conditioned Path Planning

arXiv 2308.16893 ↗First posted 31 Aug 2023cs.RO cs.AI cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Contact is at the core of robotic manipulation. At times, it is desired (e.g. manipulation and grasping), and at times, it is harmful (e.g. when avoiding obstacles). However, traditional path planning algorithms focus solely on collision-free paths, limiting their applicability in contact-rich tasks. To address this limitation, we propose the domain of Language-Conditioned Path Planning, where contact-awareness is incorporated into the path planning problem. As a first step in this domain, we propose Language-Conditioned Collision Functions (LACO) a novel approach that learns a collision function using only a single-view image, language prompt, and robot configuration. LACO predicts collisions between the robot and the environment, enabling flexible, conditional path planning without the need for manual object annotations, point cloud data, or ground-truth object meshes. In both simulation and the real world, we demonstrate that LACO can facilitate complex, nuanced path plans that allow for interaction with objects that are safe to collide, rather than prohibiting any collision.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

26 of 40 references.

References · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  5. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  8. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  9. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  10. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  11. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  12. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
14 more
  1. 2018Motion Planning Networkscited by 78
  2. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  3. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  4. 2019Motion Planning Networks: Bridging the Gap Between Learning-based and Classical Motion Plannerscited by 62
  5. 2019PyRep: Bringing V-REP to Deep Robot Learningcited by 43
  6. 2020Object Rearrangement Using Learned Implicit Collision Functionscited by 43
  7. 2020SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigationcited by 43
  8. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 42
  9. 2020Learning a Decentralized Multi-arm Motion Plannercited by 17
  10. 2023CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generationcited by 17
  11. 2022Instruction-Following Agents with Multimodal Transformercited by 13
  12. 2020Perception-Based Temporal Logic Planning in Uncertain Semantic Mapscited by 12
  13. 2022Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Datacited by 9
  14. 2023Local object crop collision network for efficient simulation of non-convex objects in GPU-based simulatorscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 6; 2025: 1.

9 papers.

  1. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  2. 2023LLM A*: Human in the Loop Large Language Models Enabled A* Search for Roboticscited by 26
  3. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  4. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teacherscited by 14
  5. 2024MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?cited by 14
  6. 2024GenCHiP: Generating Robot Policy Code for High-Precision and Contact-Rich Manipulation Taskscited by 9
  7. 2024Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMscited by 8
  8. 2025IMPACT: Intelligent Motion Planning with Acceptable Contact Trajectories via Vision-Language Modelscited by 4
  9. 2024LASMP: Language Aided Subset Sampling Based Motion Plannercited by 0