mlsanityData to 14 Oct 2026

Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping

arXiv 2309.10667 ↗First posted 19 Sep 2023cs.CV cs.MM cs.SDcited by 14

Not on the map. Why?

Abstract (arXiv)

We focus on the task of soundscape mapping, which involves predicting the most probable sounds that could be perceived at a particular geographic location. We utilise recent state-of-the-art models to encode geotagged audio, a textual description of the audio, and an overhead image of its capture location using contrastive pre-training. The end result is a shared embedding space for the three modalities, which enables the construction of soundscape maps for any geographic region from textual or audio queries. Using the SoundingEarth dataset, we find that our approach significantly outperforms the existing SOTA, with an improvement of image-to-audio Recall@100 from 0.256 to 0.450. Our code is available at https://github.com/mvrl/geoclap.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 20 of its 40 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

20 of 40 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  8. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 408
  9. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  10. 2022Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learningcited by 308
  11. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  12. 2020Geography-Aware Self-Supervised Learningcited by 190
8 more
  1. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  2. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  3. 2016Ambient Sound Provides Supervision for Visual Learningcited by 78
  4. 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
  5. 2021Self-supervised Audiovisual Representation Learning for Remote Sensing Datacited by 24
  6. 2021Semantics-Consistent Representation Learning for Remote Sensing Image-Voice Retrievalcited by 8
  7. 2016Chatty Maps: Constructing sound maps of urban areas from social media datacited by 3
  8. 2022ARAUS: A Large-Scale Dataset and Baseline Models of Affective Responses to Augmented Urban Soundscapescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 4; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Mission Critical -- Satellite Data is a Distinct Modality in Machine Learningcited by 77
  2. 2024TaxaBind: A Unified Embedding Space for Ecological Applicationscited by 35
  3. 2024GeoSynth: Contextually-Aware High-Resolution Satellite Image Synthesiscited by 32
  4. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 19
  5. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignmentcited by 15
  6. 2024WildSAT: Learning Satellite Image Representations from Wildlife Observationscited by 13
  7. 2023BirdSAT: Cross-View Contrastive Masked Autoencoders for Bird Species Classification and Mappingcited by 12
  8. 2024GEOBIND: Binding Text, Image, and Audio through Satellite Imagescited by 7
  9. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  10. 2024ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Modelscited by 6
  11. 2025ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecologycited by 1
  12. 2026Interpretable Perception and Reasoning for Audiovisual Geolocationcited by 1
2 more
  1. 2025Sat2Sound: A Unified Framework for Zero-Shot Soundscape Mappingcited by 0
  2. 2026Multi-Modal Contrastive Learning for Implicit Earth Embeddings via Location Tyingcited by 0