mlsanityData to 7 Oct 2026

Towards Balanced Multi-Modal Learning in 3D Human Pose Estimation

arXiv 2501.05264 ↗First posted 9 Jan 2025cs.CV cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

3D human pose estimation (3D HPE) has emerged as a prominent research topic, particularly in the realm of RGB-based methods. However, the use of RGB images is often limited by issues such as occlusion and privacy constraints. Consequently, multi-modal sensing, which leverages non-intrusive sensors, is gaining increasing attention. Nevertheless, multi-modal 3D HPE still faces challenges, including modality imbalance. In this work, we introduce a novel balanced multi-modal learning method for 3D HPE, which harnesses the power of RGB, LiDAR, mmWave, and WiFi. Specifically, we propose a Shapley value-based contribution algorithm to assess the contribution of each modality and detect modality imbalance. To address this imbalance, we design a modality learning regulation strategy that decelerates the learning process during the early stages of training. We conduct extensive experiments on the widely adopted multi-modal dataset, MM-Fi, demonstrating the superiority of our approach in enhancing 3D pose estimation under complex conditions. Our source code is available at https://github.com/MICLAB-BUPT/AWC.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 38 of its 64 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

38 of 64 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Point Transformercited by 1,462
  3. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
  4. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  5. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  6. 2016Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Posecited by 220
  7. 2019Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Imagecited by 131
  8. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 114
  9. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  10. 2021Probabilistic Monocular 3D Human Pose Estimation with Normalizing Flowscited by 84
  11. 2023MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensingcited by 77
  12. 2018Attentive Relational Networks for Mapping Images to Scene Graphscited by 72
26 more
  1. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
  2. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 71
  3. 2024ReconBoost: Boosting Can Achieve Modality Reconcilementcited by 56
  4. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 43
  5. 2024Diagnosing and Re-learning for Balanced Multimodal Learningcited by 40
  6. 2021Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Drivingcited by 39
  7. 2022STCrowd: A Multimodal Dataset for Pedestrian Perception in Crowded Scenescited by 35
  8. 2024T2SG: Traffic Topology Scene Graph for Topology Reasoning in Autonomous Drivingcited by 35
  9. 2024Classifier-guided Gradient Modulation for Enhanced Multimodal Learningcited by 34
  10. 20183D Human Pose Estimation in RGBD Images for Robotic Task Learningcited by 33
  11. 2024KTPFormer: Kinematics and Trajectory Prior Knowledge-Enhanced Transformer for 3D Human Pose Estimationcited by 33
  12. 2023SGFormer: Semantic Graph Transformer for Point Cloud-based 3D Scene Graph Generationcited by 27
  13. 2015Direct Prediction of 3D Body Poses from Motion Compensated Sequencescited by 26
  14. 2024FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Modelscited by 23
  15. 2022ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditionscited by 21
  16. 2025Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regressioncited by 21
  17. 2022Fast and Scalable Human Pose Estimation using mmWave Point Cloudcited by 20
  18. 2023Unsupervised Self-Driving Attention Prediction via Uncertainty Mining and Knowledge Embeddingcited by 19
  19. 2020HPERL: 3D Human Pose Estimation from RGB and LiDARcited by 18
  20. 2023Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimationcited by 15
  21. 2023FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibrationcited by 13
  22. 2024X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensingcited by 12
  23. 2022SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalitiescited by 11
  24. 2024Video-Based Human Pose Regression via Decoupled Space-Time Aggregationcited by 10
  25. 2024PITN: Physics-Informed Temporal Networks for Cuffless Blood Pressure Estimationcited by 8
  26. 2024ProbRadarM3F: mmWave Radar based Human Skeletal Pose Estimation with Probability Map Guided Multi-Format Feature Fusioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2026CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Drivingcited by 2
  2. 2025Chain-of-Evidence Multimodal Reasoning for Few-shot Temporal Action Localizationcited by 1
  3. 2026Reason-Then-Retrieve for CoVR-R with Structured Edit Prompts and Dense-Sparse Fusioncited by 1
  4. 2025Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answeringcited by 0
  5. 2026Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulationcited by 0
  6. 2026Question-Aware Evidence Ledgers for Video Relational Reasoningcited by 0
  7. 2026Claude Code-Driving Scenario Mining for the Argoverse 2 Challengecited by 0
  8. 2026A VideoMAE-v2 Approach to Zero-Shot Traffic Accident Anticipationcited by 0
  9. 2026Leveraging Metric Depth for Relative Depth Predictioncited by 0
  10. 2026A DVDrive Approach for doScenes Instructed Driving Challengecited by 0
  11. 2026Boosting Robustness for All-Weather Self-Supervised Depth Estimation in Autonomous Drivingcited by 0