mlsanityData to 14 Oct 2026

Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution

arXiv 2511.19430 ↗First posted 24 Nov 2025cs.CVcited by 6

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Task scheduling is critical for embodied AI, enabling agents to follow natural language instructions and execute actions efficiently in 3D physical worlds. However, existing datasets often simplify task planning by ignoring operations research (OR) knowledge and 3D spatial grounding. In this work, we propose Operations Research knowledge-based 3D Grounded Task Scheduling (ORS3D), a new task that requires the synergy of language understanding, 3D grounding, and efficiency optimization. Unlike prior settings, ORS3D demands that agents minimize total completion time by leveraging parallelizable subtasks, e.g., cleaning the sink while the microwave operates. To facilitate research on ORS3D, we construct ORS3D-60K, a large-scale dataset comprising 60K composite tasks across 4K real-world scenes. Furthermore, we propose GRANT, an embodied multi-modal large language model equipped with a simple yet effective scheduling token mechanism to generate efficient task schedules and grounded actions. Extensive experiments on ORS3D-60K validate the effectiveness of GRANT across language understanding, 3D grounding, and scheduling efficiency. The code is available at https://github.com/H-EmbodVis/GRANT

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 29 of its 42 references that are in the map. First posted on arXiv 24 Nov 2025.

Builds on

29 of 42 references.

References · 29

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  3. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  4. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  5. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  6. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  7. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  8. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  9. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  10. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  11. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
  12. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
17 more
  1. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 178
  2. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  3. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  4. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  5. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  6. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  7. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 69
  8. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 68
  9. 2022Accurate and Efficient Stereo Matching via Attention Concatenation Volumecited by 59
  10. 2024IGEV++: Iterative Multi-range Geometry Encoding Volumes for Stereo Matchingcited by 59
  11. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  12. 2024Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learningcited by 43
  13. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
  14. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  15. 2024Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intentioncited by 16
  16. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  17. 2025Integrating Chain-of-Thought for Multimodal Alignment: A Study on 3D Vision-Language Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 44
  2. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 22
  3. 2026Towards Generalizable Robotic Manipulation in Dynamic Environmentscited by 20
  4. 2026TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAMcited by 10
  5. 2026PointTPA: Dynamic Network Parameter Adaptation for 3D Scene Understandingcited by 2
  6. 2026Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025P3: Toward Versatile Embodied Agents
  2. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
  3. 2023An Embodied Generalist Agent in 3D World
  4. 2025OmniEVA: Embodied Versatile Planner via Task-Adaptive 3D-Grounded and Embodiment-aware Reasoning
  5. 2025Generative AI in Embodied Systems: System-Level Analysis of Performance, Efficiency and Scalability
  6. 2024LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents
  7. 2019Habitat: A Platform for Embodied AI Research
  8. 2026HoloAgent-0: A Unified Embodied Agent Framework with 3D Spatial Memory