# Connecting Language and Vision for Natural Language-Based Vehicle Retrieval

arXiv: https://arxiv.org/abs/2105.14897 · HTML: https://mlsanity.com/paper/2105.14897/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 71 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline)](https://mlsanity.com/paper/1711.09349/) (2017)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2002.06353/) (2020)
- [Joint Discriminative and Generative Learning for Person Re-identification](https://mlsanity.com/paper/1904.07223/) (2019)

## Built on by (most cited first, 12 of 12)

- [ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cues](https://mlsanity.com/paper/2201.06696/) (2022)
- [The 6th AI City Challenge](https://mlsanity.com/paper/2204.10380/) (2022)
- [FindVehicle and VehicleFinder: A NER dataset for natural language-based vehicle retrieval and a keyword-based cross-modal vehicle retrieval system](https://mlsanity.com/paper/2304.10893/) (2023)
- [Weakly-Supervised HOI Detection from Interaction Labels Only and Language/Vision-Language Priors](https://mlsanity.com/paper/2303.05546/) (2023)
- [Deep Clustering with a Constraint for Topological Invariance based on Symmetric InfoNCE](https://mlsanity.com/paper/2303.03036/) (2023)
- [OMG: Observe Multiple Granularities for Natural Language-Based Vehicle Retrieval](https://mlsanity.com/paper/2204.08209/) (2022)
- [Symmetric Network with Spatial Relationship Modeling for Natural Language-based Vehicle Retrieval](https://mlsanity.com/paper/2206.10879/) (2022)
- [SNEAK: Synonymous Sentences-Aware Adversarial Attack on Natural Language Video Localization](https://mlsanity.com/paper/2112.04154/) (2021)
- [ACTrack: Adding Spatio-Temporal Condition for Visual Object Tracking](https://mlsanity.com/paper/2403.07914/) (2024)
- [Veri-Car: Towards Open-world Vehicle Information Retrieval](https://mlsanity.com/paper/2411.06864/) (2024)
- [Copy and Paste method based on Pose for Re-identification](https://mlsanity.com/paper/2107.10479/) (2021)
- [T2I-VeRW: Part-level Fine-grained Perception for Text-to-Image Vehicle Retrieval](https://mlsanity.com/paper/2605.06012/) (2026)

## Abstract (arXiv)

Vehicle search is one basic task for the efficient traffic management in terms of the AI City. Most existing practices focus on the image-based vehicle matching, including vehicle re-identification and vehicle tracking. In this paper, we apply one new modality, i.e., the language description, to search the vehicle of interest and explore the potential of this task in the real-world scenario. The natural language-based vehicle search poses one new challenge of fine-grained understanding of both vision and language modalities. To connect language and vision, we propose to jointly train the state-of-the-art vision models with the transformer-based language model in an end-to-end manner. Except for the network structure design and the training strategy, several optimization objectives are also re-visited in this work. The qualitative and quantitative experiments verify the effectiveness of the proposed method. Our proposed method has achieved the 1st place on the 5th AI City Challenge, yielding competitive performance 18.69% MRR accuracy on the private test set. We hope this work can pave the way for the future study on using language description effectively and efficiently for real-world vehicle retrieval systems. The code will be available at https://github.com/ShuaiBai623/AIC2021-T5-CLV.