# OMG: Observe Multiple Granularities for Natural Language-Based Vehicle Retrieval

arXiv: https://arxiv.org/abs/2204.08209 · HTML: https://mlsanity.com/paper/2204.08209/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 18 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [A Strong Baseline and Batch Normalization Neck for Deep Person Re-identification](https://mlsanity.com/paper/1906.08332/) (2019)
- [Align and Prompt: Video-and-Language Pre-training with Entity Prompts](https://mlsanity.com/paper/2112.09583/) (2021)
- [Dual-Path Convolutional Image-Text Embeddings with Instance Loss](https://mlsanity.com/paper/1711.05535/) (2017)
- [CityFlow: A City-Scale Benchmark for Multi-Target Multi-Camera Vehicle Tracking and Re-Identification](https://mlsanity.com/paper/1903.09254/) (2019)
- [TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment](https://mlsanity.com/paper/2108.09980/) (2021)
- [COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning](https://mlsanity.com/paper/2011.00597/) (2020)
- [HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval](https://mlsanity.com/paper/2103.15049/) (2021)
- [HANet: Hierarchical Alignment Networks for Video-Text Retrieval](https://mlsanity.com/paper/2107.12059/) (2021)
- [CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptions](https://mlsanity.com/paper/2101.04741/) (2021)
- [Connecting Language and Vision for Natural Language-Based Vehicle Retrieval](https://mlsanity.com/paper/2105.14897/) (2021)
- [SBNet: Segmentation-based Network for Natural Language-based Vehicle Search](https://mlsanity.com/paper/2104.11589/) (2021)
- [All You Can Embed: Natural Language based Vehicle Retrieval with Spatio-Temporal Transformers](https://mlsanity.com/paper/2106.10153/) (2021)

## Built on by (most cited first, 3 of 3)

- [The 6th AI City Challenge](https://mlsanity.com/paper/2204.10380/) (2022)
- [Boosting Medical Visual Understanding From Multi-Granular Language Learning](https://mlsanity.com/paper/2511.15943/) (2025)
- [YYDS: Visible-Infrared Person Re-Identification with Coarse Descriptions](https://mlsanity.com/paper/2403.04183/) (2024)

## Abstract (arXiv)

Retrieving tracked-vehicles by natural language descriptions plays a critical role in smart city construction. It aims to find the best match for the given texts from a set of tracked vehicles in surveillance videos. Existing works generally solve it by a dual-stream framework, which consists of a text encoder, a visual encoder and a cross-modal loss function. Although some progress has been made, they failed to fully exploit the information at various levels of granularity. To tackle this issue, we propose a novel framework for the natural language-based vehicle retrieval task, OMG, which Observes Multiple Granularities with respect to visual representation, textual representation and objective functions. For the visual representation, target features, context features and motion features are encoded separately. For the textual representation, one global embedding, three local embeddings and a color-type prompt embedding are extracted to represent various granularities of semantic features. Finally, the overall framework is optimized by a cross-modal multi-granularity contrastive loss function. Experiments demonstrate the effectiveness of our method. Our OMG significantly outperforms all previous methods and ranks the 9th on the 6th AI City Challenge Track2. The codes are available at https://github.com/dyhBUPT/OMG.