# M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes

arXiv: https://arxiv.org/abs/2410.11402 · HTML: https://mlsanity.com/paper/2410.11402/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 90 of its 108 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 90)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration](https://mlsanity.com/paper/2310.07896/) (2023)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Diffusion-based Generation, Optimization, and Planning in 3D Scenes](https://mlsanity.com/paper/2301.06015/) (2023)
- [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [Foundation Models in Robotics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2312.07843/) (2023)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [On the Importance of Noise Scheduling for Diffusion Models](https://mlsanity.com/paper/2301.10972/) (2023)
- [Foundation Models for Decision Making: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2303.04129/) (2023)
- [Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models](https://mlsanity.com/paper/2308.01557/) (2023)

## Built on by (most cited first, 27 of 27)

- [DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2411.15139/) (2024)
- [ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.08052/) (2025)
- [MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulation](https://mlsanity.com/paper/2503.13446/) (2025)
- [AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation](https://mlsanity.com/paper/2507.01961/) (2025)
- [Dexterous Manipulation through Imitation Learning: A Survey](https://mlsanity.com/paper/2504.03515/) (2025)
- [UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving](https://mlsanity.com/paper/2604.02190/) (2026)
- [SafeBimanual: Diffusion-based Trajectory Optimization for Safe Bimanual Manipulation](https://mlsanity.com/paper/2508.18268/) (2025)
- [λ: A Benchmark for Data-Efficiency in Long-Horizon Indoor Mobile Manipulation Robotics](https://mlsanity.com/paper/2412.05313/) (2024)
- [HGDiffuser: Efficient Task-Oriented Grasp Generation via Human-Guided Grasp Diffusion Models](https://mlsanity.com/paper/2503.00508/) (2025)
- [BoxFusion: Reconstruction-Free Open-Vocabulary 3D Object Detection via Real-Time Multi-View Box Fusion](https://mlsanity.com/paper/2506.15610/) (2025)
- [Drift is a Sampling Error: SNR-Aware Power Distributions for Long-Horizon Robotic Planning](https://mlsanity.com/paper/2605.09537/) (2026)
- [IKDiffuser: a Diffusion-based Generative Inverse Kinematics Solver for Kinematic Trees](https://mlsanity.com/paper/2506.13087/) (2025)
- [M4Diffuser: Multi-View Diffusion Policy with Manipulability-Aware Control for Robust Mobile Manipulation](https://mlsanity.com/paper/2509.14980/) (2025)
- [MoDeSuite: Robot Learning Task Suite for Benchmarking Mobile Manipulation with Deformable Objects](https://mlsanity.com/paper/2507.21796/) (2025)
- [Human Centric General Physical Intelligence for Agile Manufacturing Automation](https://mlsanity.com/paper/2508.11960/) (2025)
- [Diffusing Trajectory Optimization Problems for Recovery During Multi-Finger Manipulation](https://mlsanity.com/paper/2510.07030/) (2025)
- [Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunities](https://mlsanity.com/paper/2603.24318/) (2026)
- [Flow Motion Policy: Manipulator Motion Planning with Flow Matching Models](https://mlsanity.com/paper/2604.07084/) (2026)
- [Towards Safe Imitation Learning via Potential Field-Guided Flow Matching](https://mlsanity.com/paper/2508.08707/) (2025)
- [ISS Policy : Scalable Diffusion Policy with Implicit Scene Supervision](https://mlsanity.com/paper/2512.15020/) (2025)
- [EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation](https://mlsanity.com/paper/2604.01421/) (2026)
- [Primitive-based Truncated Diffusion for Efficient Trajectory Generation of Differential Drive Mobile Manipulators](https://mlsanity.com/paper/2604.04166/) (2026)
- [Safer Trajectory Planning with CBF-guided Diffusion Model for Unmanned Aerial Vehicles](https://mlsanity.com/paper/2604.17527/) (2026)
- [Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2605.02777/) (2026)
- [Guided Streaming Stochastic Interpolant Policy](https://mlsanity.com/paper/2605.10051/) (2026)
- [RLMM-Flow: A Flow-based Mobile Manipulation Framework with Latent-Space Reinforcement Learning](https://mlsanity.com/paper/2607.26460/) (2026)
- [Bridge3D: Enabling Vision-Language-Action Models to See and Act in 3D](https://mlsanity.com/paper/2609.24525/) (2026)

## Abstract (arXiv)

Recent advances in diffusion models have opened new avenues for research into embodied AI agents and robotics. Despite significant achievements in complex robotic locomotion and skills, mobile manipulation-a capability that requires the coordination of navigation and manipulation-remains a challenge for generative AI techniques. This is primarily due to the high-dimensional action space, extended motion trajectories, and interactions with the surrounding environment. In this paper, we introduce M2Diffuser, a diffusion-based, scene-conditioned generative model that directly generates coordinated and efficient whole-body motion trajectories for mobile manipulation based on robot-centric 3D scans. M2Diffuser first learns trajectory-level distributions from mobile manipulation trajectories provided by an expert planner. Crucially, it incorporates an optimization module that can flexibly accommodate physical constraints and task objectives, modeled as cost and energy functions, during the inference process. This enables the reduction of physical violations and execution errors at each denoising step in a fully differentiable manner. Through benchmarking on three types of mobile manipulation tasks across over 20 scenes, we demonstrate that M2Diffuser outperforms state-of-the-art neural planners and successfully transfers the generated trajectories to a real-world robot. Our evaluations underscore the potential of generative AI to enhance the generalization of traditional planning and learning-based robotic methods, while also highlighting the critical role of enforcing physical constraints for safe and robust execution.