mlsanityData to 7 Oct 2026

Human-AI Divergence in Ego-centric Action Recognition under Spatial and Spatiotemporal Manipulations

arXiv 2603.08317 ↗First posted 9 Mar 2026cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Humans consistently outperform state-of-the-art AI models in action recognition, particularly in challenging real-world conditions involving low resolution, occlusion, and visual clutter. Understanding the sources of this performance gap is essential for developing more robust and human-aligned models. In this paper, we present a large-scale human-AI comparative study of egocentric action recognition using Minimal Identifiable Recognition Crops (MIRCs), defined as the smallest spatial or spatiotemporal regions sufficient for reliable human recognition. We used our previously introduced, Epic ReduAct, a systematically spatially reduced and temporally scrambled dataset derived from 36 EPIC KITCHENS videos, spanning multiple spatial reduction levels and temporal conditions. Recognition performance is evaluated using over 3,000 human participants and the Side4Video model. Our analysis combines quantitative metrics, Average Reduction Rate and Recognition Gap, with qualitative analyses of spatial (high-, mid-, and low-level visual features) and spatiotemporal factors, including a categorisation of actions into Low Temporal Actions (LTA) and High Temporal Actions (HTA). Results show that human performance exhibits sharp declines when transitioning from MIRCs to subMIRCs, reflecting a strong reliance on sparse, semantically critical cues such as hand-object interactions. In contrast, the model degrades more gradually and often relies on contextual and mid- to low-level features, sometimes even exhibiting increased confidence under spatial reduction. Temporally, humans remain robust to scrambling when key spatial cues are preserved, whereas the model often shows insensitivity to temporal disruption, revealing class-dependent temporal sensitivities.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 89 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

34 of 89 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  11. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  12. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
22 more
  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2021Partial success in closing the gap between human and machine visioncited by 185
  3. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
  4. 2015Semantic Amodal Segmentationcited by 102
  5. 2014Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathwaycited by 93
  6. 2020Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistencycited by 77
  7. 2022Harmonizing the object recognition strategies of deep neural networks with humanscited by 59
  8. 2022ViewFool: Evaluating the Robustness of Visual Recognition to Adversarial Viewpointscited by 48
  9. 2023Neither hype nor gloom do DNNs justicecited by 40
  10. 2015Bio-inspired Unsupervised Learning of Visual Features Leads to Robust Invariant Object Recognitioncited by 11
  11. 2022If a Human Can See It, So Should Your System: Reliability Requirements for Machine Vision Componentscited by 11
  12. 2023Side4Video: Spatial-Temporal Side Network for Memory-Efficient Image-to-Video Transfer Learningcited by 11
  13. 2019Minimal Images in Deep Neural Networks: Fragile Object Recognition in Natural Imagescited by 5
  14. 2023Do humans and Convolutional Neural Networks attend to similar areas during scene classification: Effects of task and image typecited by 4
  15. 2015Feature-based Attention in Convolutional Neural Networkscited by 3
  16. 2020Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulationcited by 3
  17. 2021What can human minimal videos tell us about dynamic recognition models?cited by 3
  18. 2023MOFO: MOtion FOcused Self-Supervision for Video Understandingcited by 3
  19. 2020What takes the brain so long: Object recognition at the level of minimal images develops for up to seconds of presentation timecited by 2
  20. 2020On the Minimal Recognizable Image Patchcited by 1
  21. 2024FILS: Self-Supervised Video Feature Prediction In Semantic Language Spacecited by 1
  22. 2024DEAR: Depth-Enhanced Action Recognitioncited by 1

Built on by

Not cited yet.