mlsanityData to 7 Oct 2026

You Only Speak Once to See

arXiv 2409.18372 ↗First posted 27 Sep 2024cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Grounding objects in images using visual cues is a well-established approach in computer vision, yet the potential of audio as a modality for object recognition and grounding remains underexplored. We introduce YOSS, "You Only Speak Once to See," to leverage audio for grounding objects in visual scenes, termed Audio Grounding. By integrating pre-trained audio models with visual models using contrastive learning and multi-modal alignment, our approach captures speech commands or descriptions and maps them directly to corresponding objects within images. Experimental results indicate that audio guidance can be effectively applied to object grounding, suggesting that incorporating audio guidance may enhance the precision and robustness of current object grounding methods and improve the performance of robotic systems and computer vision applications. This finding opens new possibilities for advanced object recognition, scene understanding, and the development of more intuitive and capable robotic systems.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 40 of its 50 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

40 of 50 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  10. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  12. 2021Grounded Language-Image Pre-trainingcited by 1,147
28 more
  1. 2016Modeling Context in Referring Expressionscited by 1,054
  2. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  3. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  6. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  7. 2020Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detectioncited by 408
  8. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  9. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  10. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  11. 2019NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detectioncited by 289
  12. 2017Objects that Soundcited by 254
  13. 2016Lip Reading Sentences in the Wildcited by 226
  14. 2016UnitBox: An Advanced Object Detection Networkcited by 205
  15. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  16. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  17. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  18. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  19. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  20. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  21. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  22. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  23. 2023CDUL: CLIP-Driven Unsupervised Learning for Multi-Label Image Classificationcited by 39
  24. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  25. 2023CPSeg: Finer-grained Image Semantic Segmentation via Chain-of-Thought Language Promptingcited by 28
  26. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  27. 2022Mask-FPAN: Semi-Supervised Face Parsing in the Wild With De-Occlusion and UV GANcited by 16
  28. 2024Open 3D World in Autonomous Drivingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025CountLLM: Towards Generalizable Repetitive Action Counting via Large Language Modelcited by 13
  2. 2026RAM: Recover Any 3D Human Motion in-the-Wildcited by 11
  3. 2025Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animationcited by 8
  4. 2025CNSocialDepress: A Chinese Social Media Dataset for Depression Risk Detection and Structured Analysiscited by 1