mlsanityData to 14 Oct 2026

Efficient Egocentric Action Recognition with Multimodal Data

arXiv 2506.01757 ↗First posted 2 Jun 2025cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The increasing availability of wearable XR devices opens new perspectives for Egocentric Action Recognition (EAR) systems, which can provide deeper human understanding and situation awareness. However, deploying real-time algorithms on these devices can be challenging due to the inherent trade-offs between portability, battery life, and computational resources. In this work, we systematically analyze the impact of sampling frequency across different input modalities - RGB video and 3D hand pose - on egocentric action recognition performance and CPU usage. By exploring a range of configurations, we provide a comprehensive characterization of the trade-offs between accuracy and computational efficiency. Our findings reveal that reducing the sampling rate of RGB frames, when complemented with higher-frequency 3D hand pose input, can preserve high accuracy while significantly lowering CPU demands. Notably, we observe up to a 3x reduction in CPU usage with minimal to no loss in recognition performance. This highlights the potential of multimodal input strategies as a viable approach to achieving efficient, real-time EAR on XR devices.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 18 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

13 of 18 references.

References · 13

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  3. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  4. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  5. 2020Designing Network Design Spacescited by 960
  6. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  7. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  8. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 166
  9. 2023Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Modelcited by 107
  10. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 105
  11. 2018Attention is All We Need: Nailing Down Object-centric Attention for Egocentric Activity Recognitioncited by 28
  12. 2024On the Utility of 3D Hand Poses for Action Recognitioncited by 17
1 more
  1. 2017Convolutional Long Short-Term Memory Networks for Recognizing First Person Interactionscited by 8

Built on by

Not cited yet.