# Body Transformer: Leveraging Robot Embodiment for Policy Learning

arXiv: https://arxiv.org/abs/2408.06316 · HTML: https://mlsanity.com/paper/2408.06316/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [On the Bottleneck of Graph Neural Networks and its Practical Implications](https://mlsanity.com/paper/2006.05205/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control](https://mlsanity.com/paper/2007.04976/) (2020)
- [HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation](https://mlsanity.com/paper/2403.10506/) (2024)
- [General In-Hand Object Rotation with Vision and Touch](https://mlsanity.com/paper/2309.09979/) (2023)
- [MetaMorph: Learning Universal Controllers with Transformers](https://mlsanity.com/paper/2203.11931/) (2022)
- [My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control](https://mlsanity.com/paper/2010.01856/) (2020)
- [GRPE: Relative Positional Encoding for Graph Transformer](https://mlsanity.com/paper/2201.12787/) (2022)
- [Humanoid Locomotion as Next Token Prediction](https://mlsanity.com/paper/2402.19469/) (2024)

## Built on by (most cited first, 30 of 32)

- [Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks](https://mlsanity.com/paper/2410.23208/) (2024)
- [Instant Policy: In-Context Imitation Learning via Graph Diffusion](https://mlsanity.com/paper/2411.12633/) (2024)
- [LocoFormer: Generalist Locomotion via Long-context Adaptation](https://mlsanity.com/paper/2509.23745/) (2025)
- [UMI-on-Air: Embodiment-Aware Guidance for Embodiment-Agnostic Visuomotor Policies](https://mlsanity.com/paper/2510.02614/) (2025)
- [Tool-as-Interface: Learning Robot Policies from Observing Human Tool Use](https://mlsanity.com/paper/2504.04612/) (2025)
- [AnyBody: A Benchmark Suite for Cross-Embodiment Manipulation](https://mlsanity.com/paper/2505.14986/) (2025)
- [Beyond Topology: A Morphological Symmetry Graph Representation for Locomotion Policy Learning](https://mlsanity.com/paper/2512.00727/) (2025)
- [Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges](https://mlsanity.com/paper/2512.04770/) (2025)
- [Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotion](https://mlsanity.com/paper/2604.08780/) (2026)
- [Multi-Embodiment Robotic Retargeting via Guided Diffusion Model](https://mlsanity.com/paper/2505.20857/) (2025)
- [Shared Modular Recurrence in Contextual MDPs for Universal Morphology Control](https://mlsanity.com/paper/2506.08630/) (2025)
- [Embedding Morphology into Transformers for Cross-Robot Policy Learning](https://mlsanity.com/paper/2603.00182/) (2026)
- [Rodrigues Network for Learning Robot Actions](https://mlsanity.com/paper/2506.02618/) (2025)
- [Arnold: A multi-task, multi-embodiment muscle transformer policy](https://mlsanity.com/paper/2508.18066/) (2025)
- [Toward Humanoid Brain-Body Co-design: Joint Optimization of Control and Morphology for Fall Recovery](https://mlsanity.com/paper/2510.22336/) (2025)
- [Self-evolving Embodied AI](https://mlsanity.com/paper/2602.04411/) (2026)
- [Shape Your Body: Value Gradients for Multi-Embodiment Robot Design](https://mlsanity.com/paper/2606.00702/) (2026)
- [Unified Locomotion Transformer with Simultaneous Sim-to-Real Transfer for Quadrupeds](https://mlsanity.com/paper/2503.08997/) (2025)
- [McARL:Morphology-Control-Aware Reinforcement Learning for Generalizable Quadrupedal Locomotion](https://mlsanity.com/paper/2505.18418/) (2025)
- [Articulated-Body Dynamics Network: Dynamics-Grounded Prior for Robot Learning](https://mlsanity.com/paper/2603.19078/) (2026)
- [MVAdapt: Zero-Shot Multi-Vehicle Adaptation for End-to-End Autonomous Driving](https://mlsanity.com/paper/2604.11854/) (2026)
- [KITE: Decoupling Kinematics and Interaction for Zero-Shot Cross-Embodiment Manipulation](https://mlsanity.com/paper/2606.22113/) (2026)
- [Rapid Embodiment Adaptation for Quadrupedal Locomotion](https://mlsanity.com/paper/2608.01506/) (2026)
- [The Embodiment Gap in Robot Foundation Models](https://mlsanity.com/paper/2608.18433/) (2026)
- [Efficient Morphology-Aware Policy Transfer to New Embodiments](https://mlsanity.com/paper/2508.03660/) (2025)
- [EGAM: Extended Graph Attention Model for Solving Routing Problems](https://mlsanity.com/paper/2601.21281/) (2026)
- [Embodiment-conditioned Generalist Control for Multirotor Aerial Robots](https://mlsanity.com/paper/2606.10857/) (2026)
- [VENOM: Versatile Embodied Network for Omni-bodied Motion tracking](https://mlsanity.com/paper/2606.16696/) (2026)
- [RecMorph: Topology-Guided Spatial Recurrence for Generalized Morphology Control](https://mlsanity.com/paper/2609.18359/) (2026)
- [Screw Attention: Rigid-Body Algebra Inside a Transformer](https://mlsanity.com/paper/2610.00904/) (2026)

## Abstract (arXiv)

In recent years, the transformer architecture has become the de facto standard for machine learning algorithms applied to natural language processing and computer vision. Despite notable evidence of successful deployment of this architecture in the context of robot learning, we claim that vanilla transformers do not fully exploit the structure of the robot learning problem. Therefore, we propose Body Transformer (BoT), an architecture that leverages the robot embodiment by providing an inductive bias that guides the learning process. We represent the robot body as a graph of sensors and actuators, and rely on masked attention to pool information throughout the architecture. The resulting architecture outperforms the vanilla transformer, as well as the classical multilayer perceptron, in terms of task completion, scaling properties, and computational efficiency when representing either imitation or reinforcement learning policies. Additional material including the open-source code is available at https://sferrazza.cc/bot_site.