# Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion

arXiv: https://arxiv.org/abs/2410.01506 · HTML: https://mlsanity.com/paper/2410.01506/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning](https://mlsanity.com/paper/2101.10030/) (2021)
- [SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning](https://mlsanity.com/paper/2111.13196/) (2021)
- [MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection](https://mlsanity.com/paper/2104.01633/) (2021)
- [VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2308.11681/) (2023)
- [Attentional Feature Fusion](https://mlsanity.com/paper/2009.14082/) (2020)
- [Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2302.05160/) (2023)
- [MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2211.15098/) (2022)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Location-aware Graph Convolutional Networks for Video Question Answering](https://mlsanity.com/paper/2008.09105/) (2020)
- [Advancing Video Anomaly Detection: A Concise Review and a New Dataset](https://mlsanity.com/paper/2402.04857/) (2024)
- [Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion](https://mlsanity.com/paper/1911.07848/) (2019)
- [On the Benefits of Early Fusion in Multimodal Representation Learning](https://mlsanity.com/paper/2011.07191/) (2020)
- [Self-supervising Action Recognition by Statistical Moment and Subspace Descriptors](https://mlsanity.com/paper/2001.04627/) (2020)
- [Temporal-Viewpoint Transportation Plan for Skeletal Few-shot Action Recognition](https://mlsanity.com/paper/2210.16820/) (2022)
- [Uncertainty-DTW for Time Series and Sequences](https://mlsanity.com/paper/2211.00005/) (2022)
- [Loss Switching Fusion with Similarity Search for Video Classification](https://mlsanity.com/paper/1906.11465/) (2019)
- [3D Skeleton-based Few-shot Action Recognition with JEANIE is not so Na"ive](https://mlsanity.com/paper/2112.12668/) (2021)
- [Meet JEANIE: a Similarity Measure for 3D Skeleton Sequences via Temporal-Viewpoint Alignment](https://mlsanity.com/paper/2402.04599/) (2024)
- [Semantic2Graph: Graph-based Multi-modal Feature Fusion for Action Segmentation in Videos](https://mlsanity.com/paper/2209.05653/) (2022)

## Built on by (most cited first, 23 of 23)

- [VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.23504/) (2025)
- [Do Language Models Understand Time?](https://mlsanity.com/paper/2412.13845/) (2024)
- [A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysis](https://mlsanity.com/paper/2511.00962/) (2025)
- [Quo Vadis, Anomaly Detection? LLMs and VLMs in the Spotlight](https://mlsanity.com/paper/2412.18298/) (2024)
- [AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection](https://mlsanity.com/paper/2504.04495/) (2025)
- [Feature Hallucination for Self-supervised Action Recognition](https://mlsanity.com/paper/2506.20342/) (2025)
- [Learning Time in Static Classifiers](https://mlsanity.com/paper/2511.12321/) (2025)
- [Evolving Skeletons: Motion Dynamics in Action Recognition](https://mlsanity.com/paper/2501.02593/) (2025)
- [Graph Self-Supervised Learning with Learnable Structural and Positional Encodings](https://mlsanity.com/paper/2502.16233/) (2025)
- [Uncertainty-Weighted Fusion of Image and Synthetic Event for Video Anomaly Detection](https://mlsanity.com/paper/2505.02393/) (2025)
- [Multimodal Industrial Anomaly Detection via Geometric Prior](https://mlsanity.com/paper/2603.22757/) (2026)
- [Probe-VAD: Ordinal Likelihood Probing for Training-Free Video Anomaly Detection](https://mlsanity.com/paper/2609.17211/) (2026)
- [Video Understanding by Design: How Datasets Shape Video Models](https://mlsanity.com/paper/2509.09151/) (2025)
- [SRVAU-R1: Enhancing Video Anomaly Understanding via Reflection-Aware Learning](https://mlsanity.com/paper/2602.01004/) (2026)
- [Learning to Fuse and Reconstruct Multi-View Graphs for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2602.21944/) (2026)
- [Subspace Kernel Learning on Tensor Sequences](https://mlsanity.com/paper/2603.19546/) (2026)
- [Privacy-Aware Video Anomaly Detection through Orthogonal Subspace Projection](https://mlsanity.com/paper/2605.08651/) (2026)
- [CAST: Collapse-Aware multi-Scale Topology Fusion for Multimodal Coreset Selection](https://mlsanity.com/paper/2605.11705/) (2026)
- [Trust-Aware Joint Feature-Prediction Discrepancy for Robust Domain Adaptation](https://mlsanity.com/paper/2605.25119/) (2026)
- [Test-Time Self-Adaptive Conditioning for Stable Audio-Driven Talking-Head Generation](https://mlsanity.com/paper/2605.25488/) (2026)
- [HyperAgent4POI: Dynamic Semantic Message Passing on Multi-Agent Hypergraphs for Missing-Modality Recommendation](https://mlsanity.com/paper/2608.01846/) (2026)
- [Can We Anticipate Violence? Multimodal Learning from Pre-Incident Behavioral Cues](https://mlsanity.com/paper/2609.40014/) (2026)
- [Anomaly-Guided Observation and Two-Level Semantic Feedback for Training-Free Video Anomaly Analysis](https://mlsanity.com/paper/2610.16324/) (2026)

## Abstract (arXiv)

In computer vision tasks, features often come from diverse representations, domains (e.g., indoor and outdoor), and modalities (e.g., text, images, and videos). Effectively fusing these features is essential for robust performance, especially with the availability of powerful pre-trained models like vision-language models. However, common fusion methods, such as concatenation, element-wise operations, and non-linear techniques, often fail to capture structural relationships, deep feature interactions, and suffer from inefficiency or misalignment of features across domains or modalities. In this paper, we shift from high-dimensional feature space to a lower-dimensional, interpretable graph space by constructing relationship graphs that encode feature relationships at different levels, e.g., clip, frame, patch, token, etc. To capture deeper interactions, we expand graphs through iterative graph relationship updates and introduce a learnable graph fusion operator to integrate these expanded relationships for more effective fusion. Our approach is relationship-centric, operates in a homogeneous space, and is mathematically principled, resembling element-wise relationship score aggregation via multilinear polynomials. We demonstrate the effectiveness of our graph-based fusion method on video anomaly detection, showing strong performance across multi-representational, multi-modal, and multi-domain feature fusion tasks.