# Cross-Modal Urban Sensing: Evaluating Sound-Vision Alignment Across Street-Level and Aerial Imagery

arXiv: https://arxiv.org/abs/2506.03388 · HTML: https://mlsanity.com/paper/2506.03388/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2110.08733/) (2021)
- [Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges](https://mlsanity.com/paper/2102.12219/) (2021)
- [PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation](https://mlsanity.com/paper/2102.01243/) (2021)
- [LSKNet: A Foundation Lightweight Backbone for Remote Sensing](https://mlsanity.com/paper/2403.11735/) (2024)
- [SegEarth-R1: Geospatial Pixel Reasoning via Large Language Model](https://mlsanity.com/paper/2504.09644/) (2025)
- [Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Sounds](https://mlsanity.com/paper/2003.04210/) (2020)
- [SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2410.01768/) (2024)
- [Self-supervised Audiovisual Representation Learning for Remote Sensing Data](https://mlsanity.com/paper/2108.00688/) (2021)
- [MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detection](https://mlsanity.com/paper/2309.14976/) (2023)
- [A Novel CNN-LSTM-based Approach to Predict Urban Expansion](https://mlsanity.com/paper/2103.01695/) (2021)
- [FLAIR #1: semantic segmentation and domain adaptation dataset](https://mlsanity.com/paper/2211.12979/) (2022)

## Abstract (arXiv)

Environmental soundscapes convey substantial ecological and social information regarding urban environments; however, their potential remains largely untapped in large-scale geographic analysis. In this study, we investigate the extent to which urban sounds correspond with visual scenes by comparing various visual representation strategies in capturing acoustic semantics. We employ a multimodal approach that integrates geo-referenced sound recordings with both street-level and remote sensing imagery across three major global cities: London, New York, and Tokyo. Utilizing the AST model for audio, along with CLIP and RemoteCLIP for imagery, as well as CLIPSeg and Seg-Earth OV for semantic segmentation, we extract embeddings and class-level features to evaluate cross-modal similarity. The results indicate that street view embeddings demonstrate stronger alignment with environmental sounds compared to segmentation outputs, whereas remote sensing segmentation is more effective in interpreting ecological categories through a Biophony--Geophony--Anthrophony (BGA) framework. These findings imply that embedding-based models offer superior semantic alignment, while segmentation-based methods provide interpretable links between visual structure and acoustic ecology. This work advances the burgeoning field of multimodal urban sensing by offering novel perspectives for incorporating sound into geospatial analysis.