mlsanityData to 14 Oct 2026

QORT-Former: Query-optimized Real-time Transformer for Understanding Two Hands Manipulating Objects

arXiv 2502.19769 ↗First posted 27 Feb 2025cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Significant advancements have been achieved in the realm of understanding poses and interactions of two hands manipulating an object. The emergence of augmented reality (AR) and virtual reality (VR) technologies has heightened the demand for real-time performance in these applications. However, current state-of-the-art models often exhibit promising results at the expense of substantial computational overhead. In this paper, we present a query-optimized real-time Transformer (QORT-Former), the first Transformer-based real-time framework for 3D pose estimation of two hands and an object. We first limit the number of queries and decoders to meet the efficiency requirement. Given limited number of queries and decoders, we propose to optimize queries which are taken as input to the Transformer decoder, to secure better accuracy: (1) we propose to divide queries into three types (a left hand query, a right hand query and an object query) and enhance query features (2) by using the contact information between hands and an object and (3) by using three-step update of enhanced image and query features with respect to one another. With proposed methods, we achieved real-time pose estimation performance using just 108 queries and 1 decoder (53.5 FPS on an RTX 3090TI GPU). Surpassing state-of-the-art results on the H2O dataset by 17.6% (left hand), 22.8% (right hand), and 27.2% (object), as well as on the FPHA dataset by 5.3% (right hand) and 10.4% (object), our method excels in accuracy. Additionally, it sets the state-of-the-art in interaction recognition, maintaining real-time efficiency with an off-the-shelf action recognition module.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

39 of 54 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  6. 2016Pyramid Scene Parsing Networkcited by 2,479
  7. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  8. 2017Non-local Neural Networkscited by 2,293
  9. 2018SlowFast Networks for Video Recognitioncited by 1,867
  10. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  11. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 573
  12. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 488
27 more
  1. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  2. 2023Reconstructing Hands in 3D with Transformerscited by 306
  3. 2021Mesh Graphormercited by 265
  4. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  5. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  6. 2017SSD-6D: Making RGB-based 3D detection and 6D pose estimation great againcited by 243
  7. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  8. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
  9. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  10. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 141
  11. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  12. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  13. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 128
  14. 2021Efficient DETR: Improving End-to-End Object Detector with Dense Priorcited by 128
  15. 2020Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimationcited by 124
  16. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  17. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 78
  18. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  19. 2022Sparse Instance Activation for Real-Time Instance Segmentationcited by 63
  20. 2019A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Imagecited by 56
  21. 2022RNNPose: Recurrent 6-DoF Object Pose Refinement with Robust Correspondence Field Estimation and Pose Optimizationcited by 38
  22. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  23. 2023FastInst: A Simple Query-Based Model for Real-Time Instance Segmentationcited by 32
  24. 2022Interacting Hand-Object Pose Estimation via Dense Mutual Attentioncited by 22
  25. 2018Augmented Skeleton Space Transfer for Depth-based Hand Pose Estimationcited by 15
  26. 2022D^2ETR: Decoder-Only DETR with Computationally Efficient Cross-Scale Attentioncited by 12
  27. 2020Temporal-Aware Self-Supervised Learning for 3D Hand Pose and Mesh Estimation in Videoscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splattingcited by 10
  2. 2026PAD-Hand: Physics-Aware Diffusion for Hand Motion Recoverycited by 4
  3. 2025Reconstructing Objects along Hand Interaction Timelines in Egocentric Videocited by 3