# M3Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition

arXiv: https://arxiv.org/abs/2308.03063 · HTML: https://mlsanity.com/paper/2308.03063/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 67 references that are in the map.
- First posted on arXiv 6 Aug 2023.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [Task-Agnostic Meta-Learning for Few-shot Learning](https://mlsanity.com/paper/1805.07722/) (2018)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [Delta-encoder: an effective sample synthesis method for few-shot object recognition](https://mlsanity.com/paper/1806.04734/) (2018)
- [Temporal-Relational CrossTransformers for Few-Shot Action Recognition](https://mlsanity.com/paper/2101.06184/) (2021)
- [TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition](https://mlsanity.com/paper/1907.09021/) (2019)
- [Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusion](https://mlsanity.com/paper/2006.08159/) (2020)
- [Spatio-temporal Relation Modeling for Few-shot Action Recognition](https://mlsanity.com/paper/2112.05132/) (2021)
- [Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition](https://mlsanity.com/paper/2010.09982/) (2020)
- [Hybrid Relation Guided Set Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2204.13423/) (2022)
- [Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning](https://mlsanity.com/paper/2206.06122/) (2022)
- [GiT: Graph Interactive Transformer for Vehicle Re-identification](https://mlsanity.com/paper/2107.05475/) (2021)
- [Label, Verify, Correct: A Simple Few Shot Object Detection Method](https://mlsanity.com/paper/2112.05749/) (2021)
- [Social Adaptive Module for Weakly-supervised Group Activity Recognition](https://mlsanity.com/paper/2007.09470/) (2020)
- [Image-Specific Information Suppression and Implicit Local Alignment for Text-based Person Search](https://mlsanity.com/paper/2208.14365/) (2022)
- [Exploring Spatial Significance via Hybrid Pyramidal Graph Network for Vehicle Re-identification](https://mlsanity.com/paper/2005.14684/) (2020)
- [An Interactively Reinforced Paradigm for Joint Infrared-Visible Image Fusion and Saliency Object Detection](https://mlsanity.com/paper/2305.09999/) (2023)

## Built on by (most cited first, 30 of 34)

- [MVP-Shot: Multi-Velocity Progressive-Alignment Framework for Few-Shot Action Recognition](https://mlsanity.com/paper/2405.02077/) (2024)
- [EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answering](https://mlsanity.com/paper/2508.10729/) (2025)
- [EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmark](https://mlsanity.com/paper/2510.06218/) (2025)
- [Delving into Multimodal Prompting for Fine-grained Visual Classification](https://mlsanity.com/paper/2309.08912/) (2023)
- [A Comprehensive Review of Few-shot Action Recognition](https://mlsanity.com/paper/2407.14744/) (2024)
- [Shape-centered Representation Learning for Visible-Infrared Person Re-identification](https://mlsanity.com/paper/2310.17952/) (2023)
- [Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning](https://mlsanity.com/paper/2510.11115/) (2025)
- [Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization](https://mlsanity.com/paper/2605.26501/) (2026)
- [DVF: Advancing Robust and Accurate Fine-Grained Image Retrieval with Retrieval Guidelines](https://mlsanity.com/paper/2404.15771/) (2024)
- [SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2412.11026/) (2024)
- [Enhancing Zero-Shot Brain Tumor Subtype Classification via Fine-Grained Patch-Text Alignment](https://mlsanity.com/paper/2508.01602/) (2025)
- [Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition](https://mlsanity.com/paper/2602.18043/) (2026)
- [Diverse Semantics-Guided Feature Alignment and Decoupling for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2505.00619/) (2025)
- [ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment](https://mlsanity.com/paper/2506.22967/) (2025)
- [Fine-grained Image Retrieval via Dual-Vision Adaptation](https://mlsanity.com/paper/2506.16273/) (2025)
- [FIND: A Simple yet Effective Baseline for Diffusion-Generated Image Detection](https://mlsanity.com/paper/2603.14220/) (2026)
- [MVRD-Bench: Multi-View Learning and Benchmarking for Dynamic Remote Photoplethysmography under Occlusion](https://mlsanity.com/paper/2603.22826/) (2026)
- [Harnessing the Power of Local Representations for Few-Shot Classification](https://mlsanity.com/paper/2407.01967/) (2024)
- [OT-DETECTOR: Delving into Optimal Transport for Zero-shot Out-of-Distribution Detection](https://mlsanity.com/paper/2503.06442/) (2025)
- [FourierPET: Deep Fourier-based Unrolled Network for Low-count PET Reconstruction](https://mlsanity.com/paper/2601.11680/) (2026)
- [Multi-scale Activation, Refinement, and Aggregation: Exploring Diverse Cues for Fine-Grained Bird Recognition](https://mlsanity.com/paper/2504.09215/) (2025)
- [Hierarchical Relation-augmented Representation Generalization for Few-shot Action Recognition](https://mlsanity.com/paper/2504.10079/) (2025)
- [ShapeSpeak: Body Shape-Aware Textual Alignment for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2504.18025/) (2025)
- [A Multi-Scale Spatial Attention-Based Zero-Shot Learning Framework for Low-Light Image Enhancement](https://mlsanity.com/paper/2506.18323/) (2025)
- [FADE: Adversarial Concept Erasure in Flow Models](https://mlsanity.com/paper/2507.12283/) (2025)
- [Robust Concept Erasure in Diffusion Models: A Theoretical Perspective on Security and Robustness](https://mlsanity.com/paper/2509.12024/) (2025)
- [Domain Adaptive SAR Wake Detection: Leveraging Similarity Filtering and Memory Guidance](https://mlsanity.com/paper/2509.12279/) (2025)
- [A2M2-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognition](https://mlsanity.com/paper/2509.17638/) (2025)
- [H3Former: Hypergraph-based Semantic-Aware Aggregation via Hyperbolic Hierarchical Contrastive Loss for Fine-Grained Visual Classification](https://mlsanity.com/paper/2511.10260/) (2025)
- [Cross-modal Context-aware Learning for Visual Prompt Guided Multimodal Image Understanding in Remote Sensing](https://mlsanity.com/paper/2512.11680/) (2025)

## Abstract (arXiv)

Due to the scarcity of manually annotated data required for fine-grained video understanding, few-shot fine-grained (FS-FG) action recognition has gained significant attention, with the aim of classifying novel fine-grained action categories with only a few labeled instances. Despite the progress made in FS coarse-grained action recognition, current approaches encounter two challenges when dealing with the fine-grained action categories: the inability to capture subtle action details and the insufficiency of learning from limited data that exhibit high intra-class variance and inter-class similarity. To address these limitations, we propose M3Net, a matching-based framework for FS-FG action recognition, which incorporates textitmulti-view encoding, textitmulti-view matching, and textitmulti-view fusion to facilitate embedding encoding, similarity matching, and decision making across multiple viewpoints. textitMulti-view encoding captures rich contextual details from the intra-frame, intra-video, and intra-episode perspectives, generating customized higher-order embeddings for fine-grained data. textitMulti-view matching integrates various matching functions enabling flexible relation modeling within limited samples to handle multi-scale spatio-temporal variations by leveraging the instance-specific, category-specific, and task-specific perspectives. textitMulti-view fusion consists of matching-predictions fusion and matching-losses fusion over the above views, where the former promotes mutual complementarity and the latter enhances embedding generalizability by employing multi-task collaborative learning. Explainable visualizations and experimental results on three challenging benchmarks demonstrate the superiority of M3Net in capturing fine-grained action details and achieving state-of-the-art performance for FS-FG action recognition.