Free-Form Composition Networks for Egocentric Action Recognition
Not on the map. Why?
Abstract (arXiv)
Egocentric action recognition is gaining significant attention in the field of human action recognition. In this paper, we address data scarcity issue in egocentric action recognition from a compositional generalization perspective. To tackle this problem, we propose a free-form composition network (FFCN) that can simultaneously learn disentangled verb, preposition, and noun representations, and then use them to compose new samples in the feature space for rare classes of action videos. First, we use a graph to capture the spatial-temporal relations among different hand/object instances in each action video. We thus decompose each action into a set of verb and preposition spatial-temporal representations using the edge features in the graph. The temporal decomposition extracts verb and preposition representations from different video frames, while the spatial decomposition adaptively learns verb and preposition representations from action-related instances in each frame. With these spatial-temporal representations of verbs and prepositions, we can compose new samples for those rare classes in a free-form manner, which is not restricted to a rigid form of a verb and a noun. The proposed FFCN can directly generate new training data samples for rare classes, hence significantly improve action recognition performance. We evaluated our method on three popular egocentric action recognition datasets, Something-Something V2, H2O, and EPIC-KITCHENS-100, and the experimental results demonstrate the effectiveness of the proposed method for handling data scarcity problems, including long-tailed and few-shot egocentric action recognition.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 58 of its 92 references that are in the map. First posted on arXiv 13 Jul 2023.
Builds on
58 of 92 references.
References · 58
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
- 2018SlowFast Networks for Video Recognitioncited by 1,867
- 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
- 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
- 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
- 2020Rescaling Egocentric Visioncited by 540
- 2015Neural Module Networkscited by 376
- 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 373
38 more
- 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
- 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
- 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
- 2018Videos as Space-Time Region Graphscited by 221
- 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
- 2020TDN: Temporal Difference Networks for Efficient Action Recognitioncited by 183
- 2016Learning Shape Abstractions by Assembling Volumetric Primitivescited by 181
- 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 141
- 2020Feature Space Augmentation for Long-Tailed Datacited by 136
- 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 130
- 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
- 2020Deep Representation Learning on Long-tailed Data: A Learnable Embedding Augmentation Perspectivecited by 120
- 2019Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networkscited by 114
- 2019Graphical Contrastive Losses for Scene Graph Parsingcited by 100
- 2021Space-time Mixing Attention for Video Transformercited by 84
- 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
- 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
- 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
- 2021A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentationcited by 79
- 2021FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentationcited by 76
- 2021Object-Region Video Transformerscited by 73
- 2018LSTA: Long Short-Term Attention for Egocentric Action Recognitioncited by 57
- 2020Robust Object Detection under Occlusion with Context-Aware CompositionalNetscited by 54
- 2016Going Deeper into First-Person Activity Recognitioncited by 48
- 2020Compositional Convolutional Neural Networks: A Deep Architecture with Innate Robustness to Partial Occlusioncited by 47
- 2019LaSO: Label-Set Operations networks for multi-label few-shot learningcited by 38
- 2021Learning Dynamics via Graph Neural Networks for Human Pose Estimation and Trackingcited by 37
- 2022Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videoscited by 35
- 2022Video Test-Time Adaptation for Action Recognitioncited by 35
- 2022MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detectioncited by 34
- 2022DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognitioncited by 31
- 2015Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognitioncited by 27
- 2020Symbiotic Attention with Privileged Information for Egocentric Action Recognitioncited by 27
- 2022Stand-Alone Inter-Frame Attention in Video Modelscited by 19
- 2022Group Contextualization for Video Recognitioncited by 18
- 2022End-to-end video instance segmentation via spatial-temporal graph neural networkscited by 16
- 2020Generative Compositional Augmentations for Scene Graph Predictioncited by 14
- 2014Pooled Motion Features for First-Person Videoscited by 10
50 most cited of 58 All 58 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.