# Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking

arXiv: https://arxiv.org/abs/2610.05707 · HTML: https://mlsanity.com/paper/2610.05707/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 20 of 20)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking](https://mlsanity.com/paper/1911.07241/) (2019)
- [A Distractor-Aware Memory for Visual Object Tracking with SAM2](https://mlsanity.com/paper/2411.17576/) (2024)
- [MixFormerV2: Efficient Fully Transformer Tracking](https://mlsanity.com/paper/2305.15896/) (2023)
- [Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning](https://mlsanity.com/paper/2303.14771/) (2023)
- [SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Tracking](https://mlsanity.com/paper/2503.18338/) (2025)
- [Similarity-Guided Layer-Adaptive Vision Transformer for UAV Tracking](https://mlsanity.com/paper/2503.06625/) (2025)
- [Dynamic Updates for Language Adaptation in Visual-Language Tracking](https://mlsanity.com/paper/2503.06621/) (2025)
- [Dual Relation Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2302.05637/) (2023)
- [VRM: Knowledge Distillation via Virtual Relation Matching](https://mlsanity.com/paper/2502.20760/) (2025)

## Abstract (arXiv)

In autonomous driving perception, visual object tracking systems must satisfy stringent latency and power constraints while remaining robust in complex and dynamic environments. Although transformer-based trackers achieve state-of-the-art accuracy, their substantial computational and memory overheads hinder deployment on real-time, resource-constrained platforms. To move toward this goal, we propose Difference Feature Map Knowledge Distillation (DFM-KD), a novel relational distillation framework tailored for transformer-based visual object tracking. Unlike conventional feature distillation methods that minimize point-wise discrepancies (e.g., mean squared error) between teacher and student feature representations, DFM-KD transfers knowledge through inter-sample feature differences, explicitly aligning the relational structure of the feature space. By distilling how the teacher models appearance variation and consistency across samples, rather than enforcing similarity in absolute activations, DFM-KD enables the student to better capture the structural dynamics of visual changes within a batch. As a result, the distilled model exhibits enhanced feature robustness and improved tracking performance. Extensive experiments demonstrate that DFM-KD consistently outperforms conventional feature-level distillation methods in both tracking precision and success rates.