mlsanityData to 7 Oct 2026

Distributed Dexterous Manipulation with Spatially Conditioned Multi-Agent Transformers

arXiv 2609.06930 ↗First posted 7 Sep 2026cs.RO cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Distributed Dexterous Manipulation (DDM) is a novel paradigm that presents significant control challenges due to high action-space redundancy, inter-robot cooperation, and dynamic object-robot interactions. This paper introduces a framework based on spatially conditioned Multi-Agent Transformers (MATs) to efficiently learn robust control policies for a DDM system grounded in an array of 64 soft delta robots arranged in an 8x8 grid. Our three core contributions are: (i) an MAT with adaptive layer norm for compute efficiency, (ii) spatial contrastive embeddings to ground transformer embeddings in the spatial configuration of the robots, and (iii) an MAT-based behavior cloning method fine-tuned using Soft Actor Critic. We also propose an action selection formulation to analyze the trade-off between task performance and the number of robots utilized. Our experiments show that MATs iteratively refine their actions through the stacked attention blocks. This further informs the benefit of spatial conditioning in transformers to learn DDM policies. We demonstrate long-horizon planar manipulation tasks with objects of various geometries in simulation and real-world. Finally, we show how action selection mitigates robot maintenance by reducing wear and tear due to inter-robot collisions while maintaining the ability to manipulate objects along various trajectories in the real-world, achieving an average error of ~1.5 cm, while using ~65% fewer robots.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

37 of 53 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2023Segment Anythingcited by 8,158
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
25 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  2. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  3. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  4. 2022A Generalist Agentcited by 802
  5. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  9. 2018Learning Dexterous In-Hand Manipulationcited by 643
  10. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  11. 2021Implicit Behavioral Cloningcited by 442
  12. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  13. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  14. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  15. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  16. 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 61
  17. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  18. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  19. 2019Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Realcited by 39
  20. 2022RB2: Robotic Manipulation Benchmarking with a Twistcited by 23
  21. 2022Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidancecited by 20
  22. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimizationcited by 20
  23. 2023ArrayBot: Reinforcement Learning for Generalizable Distributed Manipulation through Touchcited by 9
  24. 2023Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policiescited by 5
  25. 2022Linear Delta Arrays for Compliant Dexterous Distributed Manipulationcited by 2

Built on by

Not cited yet.