# M5L: Multi-Modal Multi-Margin Metric Learning for RGBT Tracking

arXiv: https://arxiv.org/abs/2003.07650 · HTML: https://mlsanity.com/paper/2003.07650/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2020.

## Built on by (most cited first, 9 of 9)

- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [Multi-spectral Vehicle Re-identification with Cross-directional Consistency Network and a High-quality Benchmark](https://mlsanity.com/paper/2208.00632/) (2022)
- [Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens](https://mlsanity.com/paper/2401.01674/) (2024)
- [Unified Single-Stage Transformer Network for Efficient RGB-T Tracking](https://mlsanity.com/paper/2308.13764/) (2023)
- [RGBT Tracking via Progressive Fusion Transformer with Dynamically Guided Learning](https://mlsanity.com/paper/2303.14778/) (2023)
- [X Modality Assisting RGBT Object Tracking](https://mlsanity.com/paper/2312.17273/) (2023)
- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)
- [Middle Fusion and Multi-Stage, Multi-Form Prompts for Robust RGB-T Tracking](https://mlsanity.com/paper/2403.18193/) (2024)
- [EANet: Enhanced Attribute-based RGBT Tracker Network](https://mlsanity.com/paper/2307.01893/) (2023)

## Abstract (arXiv)

Classifying the confusing samples in the course of RGBT tracking is a quite challenging problem, which hasn't got satisfied solution. Existing methods only focus on enlarging the boundary between positive and negative samples, however, the structured information of samples might be harmed, e.g., confusing positive samples are closer to the anchor than normal positive samples.To handle this problem, we propose a novel Multi-Modal Multi-Margin Metric Learning framework, named M5L for RGBT tracking in this paper. In particular, we design a multi-margin structured loss to distinguish the confusing samples which play a most critical role in tracking performance boosting. To alleviate this problem, we additionally enlarge the boundaries between confusing positive samples and normal ones, between confusing negative samples and normal ones with predefined margins, by exploiting the structured information of all samples in each modality.Moreover, a cross-modality constraint is employed to reduce the difference between modalities and push positive samples closer to the anchor than negative ones from two modalities.In addition, to achieve quality-aware RGB and thermal feature fusion, we introduce the modality attentions and learn them using a feature fusion module in our network. Extensive experiments on large-scale datasets testify that our framework clearly improves the tracking performance and outperforms the state-of-the-art RGBT trackers.