# Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution

arXiv: https://arxiv.org/abs/2511.19430 · HTML: https://mlsanity.com/paper/2511.19430/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 29 of 29)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023)
- [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019)
- [LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness](https://mlsanity.com/paper/2409.18125/) (2024)
- [ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation](https://mlsanity.com/paper/2503.19755/) (2025)
- [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023)
- [SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding](https://mlsanity.com/paper/2401.09340/) (2024)
- [Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes](https://mlsanity.com/paper/2308.08769/) (2023)
- [Grounded 3D-LLM with Referent Tokens](https://mlsanity.com/paper/2405.10370/) (2024)
- [Unifying 3D Vision-Language Understanding via Promptable Queries](https://mlsanity.com/paper/2405.11442/) (2024)
- [EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning](https://mlsanity.com/paper/2312.06722/) (2023)
- [3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer](https://mlsanity.com/paper/2501.01163/) (2025)
- [Accurate and Efficient Stereo Matching via Attention Concatenation Volume](https://mlsanity.com/paper/2209.12699/) (2022)
- [IGEV++: Iterative Multi-range Geometry Encoding Volumes for Stereo Matching](https://mlsanity.com/paper/2409.00638/) (2024)
- [ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities](https://mlsanity.com/paper/2407.01525/) (2024)
- [Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning](https://mlsanity.com/paper/2410.08114/) (2024)
- [On Grounded Planning for Embodied Tasks with Language Models](https://mlsanity.com/paper/2209.00465/) (2022)
- [Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning](https://mlsanity.com/paper/2410.00255/) (2024)
- [Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention](https://mlsanity.com/paper/2405.18295/) (2024)
- [Multimodal 3D Reasoning Segmentation with Complex Scenes](https://mlsanity.com/paper/2411.13927/) (2024)
- [Integrating Chain-of-Thought for Multimodal Alignment: A Study on 3D Vision-Language Learning](https://mlsanity.com/paper/2503.06232/) (2025)

## Built on by (most cited first, 6 of 6)

- [Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments](https://mlsanity.com/paper/2605.30280/) (2026)
- [Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously](https://mlsanity.com/paper/2603.12262/) (2026)
- [Towards Generalizable Robotic Manipulation in Dynamic Environments](https://mlsanity.com/paper/2603.15620/) (2026)
- [TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM](https://mlsanity.com/paper/2607.27205/) (2026)
- [PointTPA: Dynamic Network Parameter Adaptation for 3D Scene Understanding](https://mlsanity.com/paper/2604.04933/) (2026)
- [Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agents](https://mlsanity.com/paper/2609.22120/) (2026)

## Abstract (arXiv)

Task scheduling is critical for embodied AI, enabling agents to follow natural language instructions and execute actions efficiently in 3D physical worlds. However, existing datasets often simplify task planning by ignoring operations research (OR) knowledge and 3D spatial grounding. In this work, we propose Operations Research knowledge-based 3D Grounded Task Scheduling (ORS3D), a new task that requires the synergy of language understanding, 3D grounding, and efficiency optimization. Unlike prior settings, ORS3D demands that agents minimize total completion time by leveraging parallelizable subtasks, e.g., cleaning the sink while the microwave operates. To facilitate research on ORS3D, we construct ORS3D-60K, a large-scale dataset comprising 60K composite tasks across 4K real-world scenes. Furthermore, we propose GRANT, an embodied multi-modal large language model equipped with a simple yet effective scheduling token mechanism to generate efficient task schedules and grounded actions. Extensive experiments on ORS3D-60K validate the effectiveness of GRANT across language understanding, 3D grounding, and scheduling efficiency. The code is available at https://github.com/H-EmbodVis/GRANT