# Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation

arXiv: https://arxiv.org/abs/2409.14411 · HTML: https://mlsanity.com/paper/2409.14411/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 57 of its 71 references that are in the map.
- First posted on arXiv 22 Sep 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12514/) (2024)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2311.17600/) (2023)
- [Imitating Human Behaviour with Diffusion Models](https://mlsanity.com/paper/2301.10677/) (2023)
- [Goal-Conditioned Imitation Learning using Score-based Diffusion Policies](https://mlsanity.com/paper/2304.02532/) (2023)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling](https://mlsanity.com/paper/2209.14548/) (2022)
- [Masked World Models for Visual Control](https://mlsanity.com/paper/2206.14244/) (2022)
- [Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation](https://mlsanity.com/paper/2405.07503/) (2024)
- [Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition](https://mlsanity.com/paper/2307.14535/) (2023)

## Built on by (most cited first, 30 of 38)

- [DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control](https://mlsanity.com/paper/2502.05855/) (2025)
- [ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model](https://mlsanity.com/paper/2502.14420/) (2025)
- [PointVLA: Injecting the 3D World into Vision-Language-Action Models](https://mlsanity.com/paper/2503.07511/) (2025)
- [OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulation](https://mlsanity.com/paper/2505.03912/) (2025)
- [Large Video Planner Enables Generalizable Robot Control](https://mlsanity.com/paper/2512.15840/) (2025)
- [WorldEval: World Model as Real-World Robot Policies Evaluator](https://mlsanity.com/paper/2505.19017/) (2025)
- [ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledge](https://mlsanity.com/paper/2505.21906/) (2025)
- [Pure Vision Language Action (VLA) Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.19012/) (2025)
- [FP3: A 3D Foundation Policy for Robotic Manipulation](https://mlsanity.com/paper/2503.08950/) (2025)
- [CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance](https://mlsanity.com/paper/2412.20451/) (2024)
- [Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2503.10743/) (2025)
- [Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2409.18707/) (2024)
- [Bidirectional Sparse Attention for Faster Video Diffusion Training](https://mlsanity.com/paper/2509.01085/) (2025)
- [Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy](https://mlsanity.com/paper/2512.11218/) (2025)
- [Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robots](https://mlsanity.com/paper/2507.17049/) (2025)
- [LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment](https://mlsanity.com/paper/2410.21352/) (2024)
- [FreqPolicy: Efficient Flow-based Visuomotor Policy via Frequency Consistency](https://mlsanity.com/paper/2506.08822/) (2025)
- [Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition](https://mlsanity.com/paper/2510.01068/) (2025)
- [Modality-Composable Diffusion Policy via Inference-Time Distribution-level Composition](https://mlsanity.com/paper/2503.12466/) (2025)
- [Asynchronous Fast-Slow Vision-Language-Action Policies for Whole-Body Robotic Manipulation](https://mlsanity.com/paper/2512.20188/) (2025)
- [ShowUI-π: Flow-based Generative Models as GUI Dexterous Hands](https://mlsanity.com/paper/2512.24965/) (2025)
- [EDT: An Efficient Diffusion Transformer Framework Inspired by Human-like Sketching](https://mlsanity.com/paper/2410.23788/) (2024)
- [InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphs](https://mlsanity.com/paper/2512.07410/) (2025)
- [APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies](https://mlsanity.com/paper/2606.12366/) (2026)
- [ADPro: a Test-time Adaptive Diffusion Policy via Manifold-constrained Denoising and Task-aware Initialization for Robotic Manipulation](https://mlsanity.com/paper/2508.06266/) (2025)
- [Ask-to-Clarify: Resolving Instruction Ambiguity through Multi-turn Dialogue](https://mlsanity.com/paper/2509.15061/) (2025)
- [FORGE-Tree: Diffusion-Forcing Tree Search for Long-Horizon Robot Manipulation](https://mlsanity.com/paper/2510.21744/) (2025)
- [Fourier Features Let Agents Learn High Precision Policies with Imitation Learning](https://mlsanity.com/paper/2606.12334/) (2026)
- [U-DiT Policy: U-shaped Diffusion Transformers for Robotic Manipulation](https://mlsanity.com/paper/2509.24579/) (2025)
- [Embodiment Transfer Learning for Vision-Language-Action Models](https://mlsanity.com/paper/2511.01224/) (2025)

## Abstract (arXiv)

Diffusion Policy is a powerful technique tool for learning end-to-end visuomotor robot control. It is expected that Diffusion Policy possesses scalability, a key attribute for deep neural networks, typically suggesting that increasing model size would lead to enhanced performance. However, our observations indicate that Diffusion Policy in transformer architecture (DP) struggles to scale effectively; even minor additions of layers can deteriorate training outcomes. To address this issue, we introduce Scalable Diffusion Transformer Policy for visuomotor learning. Our proposed method, namely methodname, introduces two modules that improve the training dynamic of Diffusion Policy and allow the network to better handle multimodal action distribution. First, we identify that DP~suffers from large gradient issues, making the optimization of Diffusion Policy unstable. To resolve this issue, we factorize the feature embedding of observation into multiple affine layers, and integrate it into the transformer blocks. Additionally, our utilize non-causal attention which allows the policy network to enquotesee future actions during prediction, helping to reduce compounding errors. We demonstrate that our proposed method successfully scales the Diffusion Policy from 10 million to 1 billion parameters. This new model, named methodname, can effectively scale up the model size with improved performance and generalization. We benchmark methodname~across 50 different tasks from MetaWorld and find that our largest methodname~outperforms DP~with an average improvement of 21.6%. Across 7 real-world robot tasks, our ScaleDP demonstrates an average improvement of 36.25% over DP-T on four single-arm tasks and 75% on three bimanual tasks. We believe our work paves the way for scaling up models for visuomotor learning. The project page is available at scaling-diffusion-policy.github.io.