mlsanityData to 7 Oct 2026

All You Can Embed: Natural Language based Vehicle Retrieval with Spatio-Temporal Transformers

arXiv 2106.10153 ↗First posted 18 Jun 2021cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Combining Natural Language with Vision represents a unique and interesting challenge in the domain of Artificial Intelligence. The AI City Challenge Track 5 for Natural Language-Based Vehicle Retrieval focuses on the problem of combining visual and textual information, applied to a smart-city use case. In this paper, we present All You Can Embed (AYCE), a modular solution to correlate single-vehicle tracking sequences with natural language. The main building blocks of the proposed architecture are (i) BERT to provide an embedding of the textual descriptions, (ii) a convolutional backbone along with a Transformer model to embed the visual information. For the training of the retrieval model, a variation of the Triplet Margin Loss is proposed to learn a distance measure between the visual and language embeddings. The code is publicly available at https://github.com/cscribano/AYCE_2021.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

26 of 40 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  10. 2015Fast R-CNNcited by 2,930
  11. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  12. 2018Deep contextualized word representationscited by 2,285
14 more
  1. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  4. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  5. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  6. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  7. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  8. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  9. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  10. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  11. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  12. 2019Image Captioning: Transforming Objects into Wordscited by 171
  13. 2019Improved Embeddings with Easy Positive Triplet Miningcited by 66
  14. 2021CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptionscited by 26

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2.

3 papers.

  1. 2023FindVehicle and VehicleFinder: A NER dataset for natural language-based vehicle retrieval and a keyword-based cross-modal vehicle retrieval systemcited by 15
  2. 2023Structural Information Guided Multimodal Pre-training for Vehicle-centric Perceptioncited by 7
  3. 2022OMG: Observe Multiple Granularities for Natural Language-Based Vehicle Retrievalcited by 3