# Generative-based Fusion Mechanism for Multi-Modal Tracking

arXiv: https://arxiv.org/abs/2309.01728 · HTML: https://mlsanity.com/paper/2309.01728/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 4 Sep 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [All are Worth Words: A ViT Backbone for Diffusion Models](https://mlsanity.com/paper/2209.12152/) (2022)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [Visual Prompt Multi-Modal Tracking](https://mlsanity.com/paper/2303.10826/) (2023)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [Wavelet Diffusion Models are fast and scalable Image Generators](https://mlsanity.com/paper/2211.16152/) (2022)
- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2204.04120/) (2022)
- [DepthTrack : Unveiling the Power of RGBD Tracking](https://mlsanity.com/paper/2108.13962/) (2021)
- [VITAL: VIsual Tracking via Adversarial Learning](https://mlsanity.com/paper/1804.04273/) (2018)
- [Multi-Modal Fusion for End-to-End RGB-T Tracking](https://mlsanity.com/paper/1908.11714/) (2019)
- [Prompting for Multi-Modal Tracking](https://mlsanity.com/paper/2207.14571/) (2022)
- [Challenge-Aware RGBT Tracking](https://mlsanity.com/paper/2007.13143/) (2020)
- [Jointly Modeling Motion and Appearance Cues for Robust RGB-T Tracking](https://mlsanity.com/paper/2007.02041/) (2020)
- [RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking](https://mlsanity.com/paper/2208.09787/) (2022)
- [Dense Feature Aggregation and Pruning for RGBT Tracking](https://mlsanity.com/paper/1907.10451/) (2019)
- [Multi-Adapter RGBT Tracking](https://mlsanity.com/paper/1907.07485/) (2019)
- [Duality-Gated Mutual Condition Network for RGBT Tracking](https://mlsanity.com/paper/2011.07188/) (2020)

## Built on by (most cited first, 30 of 33)

- [One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion](https://mlsanity.com/paper/2502.19854/) (2025)
- [RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba](https://mlsanity.com/paper/2408.08827/) (2024)
- [MambaVT: Spatio-Temporal Contextual Modeling for robust RGB-T Tracking](https://mlsanity.com/paper/2408.07889/) (2024)
- [Adaptive Hyper-Graph Convolution Network for Skeleton-based Human Action Recognition with Virtual Connections](https://mlsanity.com/paper/2411.14796/) (2024)
- [Adaptive Perception for Unified Visual Multi-modal Object Tracking](https://mlsanity.com/paper/2502.06583/) (2025)
- [CADTrack: Learning Contextual Aggregation with Deformable Alignment for Robust RGBT Tracking](https://mlsanity.com/paper/2511.17967/) (2025)
- [TENet: Targetness Entanglement Incorporating with Multi-Scale Pooling and Mutually-Guided Fusion for RGB-E Object Tracking](https://mlsanity.com/paper/2405.05004/) (2024)
- [Cross-Modal Object Tracking via Modality-Aware Fusion Network and A Large-Scale Dataset](https://mlsanity.com/paper/2312.14446/) (2023)
- [AFter: Attention-based Fusion Router for RGBT Tracking](https://mlsanity.com/paper/2405.02717/) (2024)
- [Transformer-based RGB-T Tracking with Channel and Spatial Feature Fusion](https://mlsanity.com/paper/2405.03177/) (2024)
- [Explicit Context Reasoning with Supervision for Visual Tracking](https://mlsanity.com/paper/2507.16191/) (2025)
- [Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm](https://mlsanity.com/paper/2509.24741/) (2025)
- [Dynamic Subframe Splitting and Spatio-Temporal Motion Entangled Sparse Attention for RGB-E Tracking](https://mlsanity.com/paper/2409.17560/) (2024)
- [Modality-Guided Dynamic Graph Fusion and Temporal Diffusion for Self-Supervised RGB-T Tracking](https://mlsanity.com/paper/2505.03507/) (2025)
- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)
- [RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.03617/) (2026)
- [EvaNet: Towards More Efficient and Consistent Infrared and Visible Image Fusion Assessment](https://mlsanity.com/paper/2604.02896/) (2026)
- [BTMTrack: Robust RGB-T Tracking via Dual-template Bridging and Temporal-Modal Candidate Elimination](https://mlsanity.com/paper/2501.03616/) (2025)
- [UASTrack: A Unified Adaptive Selection Framework with Modality-Customization in Single Object Tracking](https://mlsanity.com/paper/2502.18220/) (2025)
- [Towards General Multimodal Visual Tracking](https://mlsanity.com/paper/2503.11218/) (2025)
- [Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking](https://mlsanity.com/paper/2506.23972/) (2025)
- [DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter](https://mlsanity.com/paper/2508.01592/) (2025)
- [SUIT: Spatial-Spectral Union-Intersection Interaction Network for Hyperspectral Object Tracking](https://mlsanity.com/paper/2508.07250/) (2025)
- [SwiTrack: Tri-State Switch for Cross-Modal Object Tracking](https://mlsanity.com/paper/2511.16227/) (2025)
- [Tracking and Segmenting Anything in Any Modality](https://mlsanity.com/paper/2511.19475/) (2025)
- [Spatio-Temporal Conditional Denoising Transformer for Modality-Missing RGBT Tracking](https://mlsanity.com/paper/2607.24701/) (2026)
- [Revisiting RGBT Tracking Benchmarks from the Perspective of Modality Validity: A New Benchmark, Problem, and Solution](https://mlsanity.com/paper/2405.00168/) (2024)
- [Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalities](https://mlsanity.com/paper/2408.14585/) (2024)
- [Open-RGBT: Open-vocabulary RGB-T Zero-shot Semantic Segmentation in Open-world Environments](https://mlsanity.com/paper/2410.06626/) (2024)
- [Serial Over Parallel: Learning Continual Unification for Multi-Modal Visual Object Tracking and Benchmarking](https://mlsanity.com/paper/2508.10655/) (2025)

## Abstract (arXiv)

Generative models (GMs) have received increasing research interest for their remarkable capacity to achieve comprehensive understanding. However, their potential application in the domain of multi-modal tracking has remained relatively unexplored. In this context, we seek to uncover the potential of harnessing generative techniques to address the critical challenge, information fusion, in multi-modal tracking. In this paper, we delve into two prominent GM techniques, namely, Conditional Generative Adversarial Networks (CGANs) and Diffusion Models (DMs). Different from the standard fusion process where the features from each modality are directly fed into the fusion block, we condition these multi-modal features with random noise in the GM framework, effectively transforming the original training samples into harder instances. This design excels at extracting discriminative clues from the features, enhancing the ultimate tracking performance. To quantitatively gauge the effectiveness of our approach, we conduct extensive experiments across two multi-modal tracking tasks, three baseline methods, and three challenging benchmarks. The experimental results demonstrate that the proposed generative-based fusion mechanism achieves state-of-the-art performance, setting new records on LasHeR and RGBD1K.