# T6D-Direct: Transformers for Multi-Object 6D Pose Direct Regression

arXiv: https://arxiv.org/abs/2109.10948 · HTML: https://mlsanity.com/paper/2109.10948/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 37 references that are in the map.
- First posted on arXiv 22 Sep 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [BOP Challenge 2020 on 6D Object Localization](https://mlsanity.com/paper/2009.07378/) (2020)
- [Implicit 3D Orientation Learning for 6D Object Detection from RGB Images](https://mlsanity.com/paper/1902.01275/) (2019)
- [GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimator](https://mlsanity.com/paper/2102.12145/) (2021)
- [Learning non-maximum suppression](https://mlsanity.com/paper/1705.02950/) (2017)
- [Segmentation-driven 6D Object Pose Estimation](https://mlsanity.com/paper/1812.02541/) (2018)
- [Single-Stage 6D Object Pose Estimation](https://mlsanity.com/paper/1911.08324/) (2019)
- [Self6D: Self-Supervised Monocular 6D Object Pose Estimation](https://mlsanity.com/paper/2004.06468/) (2020)
- [Making Deep Heatmaps Robust to Partial Occlusions for 3D Object Pose Estimation](https://mlsanity.com/paper/1804.03959/) (2018)
- [Deep Model-Based 6D Pose Refinement in RGB](https://mlsanity.com/paper/1810.03065/) (2018)
- [PFRL: Pose-Free Reinforcement Learning for 6D Pose Estimation](https://mlsanity.com/paper/2102.12096/) (2021)
- [Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing](https://mlsanity.com/paper/1810.02977/) (2018)

## Built on by (most cited first, 16 of 16)

- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training](https://mlsanity.com/paper/2211.10801/) (2022)
- [LPViT: Low-Power Semi-structured Pruning for Vision Transformers](https://mlsanity.com/paper/2407.02068/) (2024)
- [PoET: Pose Estimation Transformer for Single-View, Multi-Object 6D Pose Estimation](https://mlsanity.com/paper/2211.14125/) (2022)
- [Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes](https://mlsanity.com/paper/2406.17438/) (2024)
- [A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships](https://mlsanity.com/paper/2408.15178/) (2024)
- [CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs](https://mlsanity.com/paper/2309.15755/) (2023)
- [TransPose: 6D Object Pose Estimation with Geometry-Aware Transformer](https://mlsanity.com/paper/2310.16279/) (2023)
- [Video based Object 6D Pose Estimation using Transformers](https://mlsanity.com/paper/2210.13540/) (2022)
- [RDPN6D: Residual-based Dense Point-wise Network for 6Dof Object Pose Estimation Based on RGB-D Images](https://mlsanity.com/paper/2405.08483/) (2024)
- [Active 6D Pose Estimation for Textureless Objects using Multi-View RGB Frames](https://mlsanity.com/paper/2503.03726/) (2025)
- [Review on 6D Object Pose Estimation with the focus on Indoor Scene Understanding](https://mlsanity.com/paper/2212.01920/) (2022)
- [TransPose: A Transformer-based 6D Object Pose Estimation Network with Depth Refinement](https://mlsanity.com/paper/2307.05561/) (2023)
- [MVTOP: Multi-View Transformer-based Object Pose-Estimation](https://mlsanity.com/paper/2508.03243/) (2025)
- [PoseStreamer: A Multi-modal Framework for 3D Tracking of Unseen Moving Objects](https://mlsanity.com/paper/2512.22979/) (2025)
- [TinyDETR-Pose: Towards End-to-End Real-Time Single-Stage 6DoF Object Pose Estimation with Lightweight Transformers](https://mlsanity.com/paper/2608.15297/) (2026)

## Abstract (arXiv)

6D pose estimation is the task of predicting the translation and orientation of objects in a given input image, which is a crucial prerequisite for many robotics and augmented reality applications. Lately, the Transformer Network architecture, equipped with a multi-head self-attention mechanism, is emerging to achieve state-of-the-art results in many computer vision tasks. DETR, a Transformer-based model, formulated object detection as a set prediction problem and achieved impressive results without standard components like region of interest pooling, non-maximal suppression, and bounding box proposals. In this work, we propose T6D-Direct, a real-time single-stage direct method with a transformer-based architecture built on DETR to perform 6D multi-object pose direct estimation. We evaluate the performance of our method on the YCB-Video dataset. Our method achieves the fastest inference time, and the pose estimation accuracy is comparable to state-of-the-art methods.