mlsanityData to 7 Oct 2026

End-to-End Trainable Multi-Instance Pose Estimation with Transformers

arXiv 2103.12115 ↗First posted 22 Mar 2021cs.CVcited by 30

Not on the map. Why?

Abstract (arXiv)

We propose an end-to-end trainable approach for multi-instance pose estimation, called POET (POse Estimation Transformer). Combining a convolutional neural network with a transformer encoder-decoder architecture, we formulate multiinstance pose estimation from images as a direct set prediction problem. Our model is able to directly regress the pose of all individuals, utilizing a bipartite matching scheme. POET is trained using a novel set-based global loss that consists of a keypoint loss, a visibility loss and a class loss. POET reasons about the relations between multiple detected individuals and the full image context to directly predict their poses in parallel. We show that POET achieves high accuracy on the COCO keypoint detection task while having less parameters and higher inference speed than other bottom-up and top-down approaches. Moreover, we show successful transfer learning when applying POET to animal pose estimation. To the best of our knowledge, this model is the first end-to-end trainable multi-instance pose estimation method and we hope it will serve as a simple and promising alternative.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 22 Mar 2021.

Builds on

39 of 54 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
27 more
  1. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  2. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  3. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  4. 2021Transformers in Vision: A Surveycited by 1,009
  5. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  8. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  9. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  10. 2017Cascaded Pyramid Network for Multi-Person Pose Estimationcited by 341
  11. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 271
  12. 2019HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimationcited by 247
  13. 2018CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmarkcited by 214
  14. 2016Associative Embedding: End-to-End Learning for Joint Detection and Groupingcited by 211
  15. 2018Markerless tracking of user-defined features with deep learningcited by 203
  16. 2019Distribution-Aware Coordinate Representation for Human Pose Estimationcited by 165
  17. 2020Rethinking Transformer-based Set Prediction for Object Detectioncited by 156
  18. 2018PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Modelcited by 133
  19. 2017Towards Accurate Multi-person Pose Estimation in the Wildcited by 131
  20. 2019PifPaf: Composite Fields for Human Pose Estimationcited by 120
  21. 2016DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Modelcited by 96
  22. 2019Single-Stage Multi-Person Pose Machinescited by 95
  23. 2020Monocular Human Pose Estimation: A Survey of Deep Learning-based Methodscited by 59
  24. 2021OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Associationcited by 55
  25. 2016ArtTrack: Articulated Multi-person Tracking in the Wildcited by 27
  26. 2016Multi-Person Pose Estimation with Local Joint-to-Person Associationscited by 18
  27. 2020A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectivescited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 8; 2023: 5; 2024: 4; 2025: 2; 2026: 1.

30 papers.

Sort:

  1. 2022DETRs with Hybrid Matchingcited by 149
  2. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  3. 2021TokenPose: Learning Keypoint Tokens for Human Pose Estimationcited by 106
  4. 2021Multi-Person 3D Motion Prediction with Multi-Range Transformerscited by 63
  5. 2021Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Predictioncited by 57
  6. 2023Rank-DETR for High Quality Object Detectioncited by 49
  7. 2023Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimationcited by 48
  8. 2021TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Networkcited by 45
  9. 2023Lite DETR : An Interleaved Multi-Scale Encoder for Efficient DETRcited by 36
  10. 2021Test-Time Personalization with a Transformer for Human Pose Estimationcited by 33
  11. 2021Generative Transformer for Accurate and Reliable Salient Object Detectioncited by 18
  12. 2022Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Taskscited by 12
18 more
  1. 2021Boosting Salient Object Detection with Transformer-based Asymmetric Bilateral U-Netcited by 9
  2. 2024ProbPose: A Probabilistic Approach to 2D Human Pose Estimationcited by 9
  3. 2021PE-former: Pose Estimation Transformercited by 7
  4. 2023FeatAug-DETR: Enriching One-to-Many Matching for DETRs with Feature Augmentationcited by 7
  5. 2024Referring Human Pose and Mask Estimation in the Wildcited by 6
  6. 2022LwPosr: Lightweight Efficient Fine-Grained Head Pose Estimationcited by 4
  7. 2022An Energy-Based Prior for Generative Saliencycited by 3
  8. 2022VTP: Volumetric Transformer for Multi-view Multi-person 3D Pose Estimationcited by 3
  9. 2022DPIT: Dual-Pipeline Integrated Transformer for Human Pose Estimationcited by 3
  10. 2021Attend to Who You Are: Supervising Self-Attention for Keypoint Detection and Instance-Aware Associationcited by 2
  11. 2023Joint Coordinate Regression and Association For Multi-Person Pose Estimation, A Pure Neural Network Approachcited by 2
  12. 2024Detection, Pose Estimation and Segmentation for Multiple Bodies: Closing the Virtuous Circlecited by 2
  13. 2022HeadPosr: End-to-end Trainable Head Pose Estimation using Transformer Encoderscited by 1
  14. 2022Person Re-Identificationcited by 0
  15. 2024Map-Free Visual Relocalization Enhanced by Instance Knowledge and Depth Knowledgecited by 0
  16. 2025Improvement of human health lifespan with hybrid group pose estimation methodscited by 0
  17. 2025LGM-Pose: A Lightweight Global Modeling Network for Real-time Human Pose Estimationcited by 0
  18. 2026BBoxMaskPose v2: Expanding Mutual Conditioning to 3Dcited by 0