# All in One Framework for Multimodal Re-identification in the Wild

arXiv: https://arxiv.org/abs/2405.04741 · HTML: https://mlsanity.com/paper/2405.04741/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 84 references that are in the map.
- First posted on arXiv 8 May 2024.

## Builds on (most cited first, 30 of 53)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Deep Learning for Person Re-identification: A Survey and Outlook](https://mlsanity.com/paper/2001.04193/) (2020)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://mlsanity.com/paper/1903.07071/) (2019)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [Omni-Scale Feature Learning for Person Re-Identification](https://mlsanity.com/paper/1905.00953/) (2019)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labels](https://mlsanity.com/paper/2211.13977/) (2022)
- [Sketch-Guided Text-to-Image Diffusion Models](https://mlsanity.com/paper/2211.13752/) (2022)
- [BEVT: BERT Pretraining of Video Transformers](https://mlsanity.com/paper/2112.01529/) (2021)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Person Search with Natural Language Description](https://mlsanity.com/paper/1702.05729/) (2017)
- [Meta-Transformer: A Unified Framework for Multimodal Learning](https://mlsanity.com/paper/2307.10802/) (2023)
- [End-to-End Referring Video Object Segmentation with Multimodal Transformers](https://mlsanity.com/paper/2111.14821/) (2021)
- [Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval](https://mlsanity.com/paper/2303.12501/) (2023)

## Built on by (most cited first, 18 of 18)

- [IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identification](https://mlsanity.com/paper/2503.10324/) (2025)
- [Human-Centric Foundation Models: Perception, Generation and Agentic Modeling](https://mlsanity.com/paper/2502.08556/) (2025)
- [Cross-video Identity Correlating for Person Re-identification Pre-training](https://mlsanity.com/paper/2409.18569/) (2024)
- [ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model](https://mlsanity.com/paper/2506.09385/) (2025)
- [Cross-modal Ship Re-Identification via Optical and SAR Imagery: A Novel Dataset and Method](https://mlsanity.com/paper/2506.22027/) (2025)
- [From Laboratory to Real World: A New Benchmark Towards Privacy-Preserved Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2503.12232/) (2025)
- [Towards Anytime Retrieval: A Benchmark for Anytime Person Re-Identification](https://mlsanity.com/paper/2509.16635/) (2025)
- [A Theory-Inspired Framework for Few-Shot Cross-Modal Sketch Person Re-Identification](https://mlsanity.com/paper/2511.18677/) (2025)
- [Multi-Domain Biometric Recognition using Body Embeddings](https://mlsanity.com/paper/2503.10931/) (2025)
- [ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology](https://mlsanity.com/paper/2511.02946/) (2025)
- [Towards Context-Aware Image Anonymization with Multi-Agent Reasoning](https://mlsanity.com/paper/2603.27817/) (2026)
- [GRAP-MOT: Unsupervised Graph-based Position Weighted Person Multi-camera Multi-object Tracking in a Highly Congested Space](https://mlsanity.com/paper/2510.21482/) (2025)
- [Skeletons Speak Louder than Text: A Motion-Aware Pretraining Paradigm for Video-Based Person Re-Identification](https://mlsanity.com/paper/2511.13150/) (2025)
- [Adaptive Multimodal Person Recognition: A Robust Framework for Handling Missing Modalities](https://mlsanity.com/paper/2512.14961/) (2025)
- [Unleashing MLLMs on the Edge: A Unified Framework for Cross-Modal ReID via Adaptive SVD Distillation](https://mlsanity.com/paper/2602.12936/) (2026)
- [Beyond Visual Cues: Semantic-Driven Token Filtering and Expert Routing for Anytime Person ReID](https://mlsanity.com/paper/2604.15090/) (2026)
- [MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Experts](https://mlsanity.com/paper/2607.19826/) (2026)
- [MMGait: Benchmarking and Unifying Gait Recognition across Heterogeneous Modalities](https://mlsanity.com/paper/2609.11601/) (2026)

## Abstract (arXiv)

In Re-identification (ReID), recent advancements yield noteworthy progress in both unimodal and cross-modal retrieval tasks. However, the challenge persists in developing a unified framework that could effectively handle varying multimodal data, including RGB, infrared, sketches, and textual information. Additionally, the emergence of large-scale models shows promising performance in various vision tasks but the foundation model in ReID is still blank. In response to these challenges, a novel multimodal learning paradigm for ReID is introduced, referred to as All-in-One (AIO), which harnesses a frozen pre-trained big model as an encoder, enabling effective multimodal retrieval without additional fine-tuning. The diverse multimodal data in AIO are seamlessly tokenized into a unified space, allowing the modality-shared frozen encoder to extract identity-consistent features comprehensively across all modalities. Furthermore, a meticulously crafted ensemble of cross-modality heads is designed to guide the learning trajectory. AIO is the first framework to perform all-in-one ReID, encompassing four commonly used modalities. Experiments on cross-modal and multimodal ReID reveal that AIO not only adeptly handles various modal data but also excels in challenging contexts, showcasing exceptional performance in zero-shot and domain generalization scenarios.