mlsanityData to 14 Oct 2026

What Do I Hear? Generating Sounds for Visuals with ChatGPT

arXiv 2311.05609 ↗First posted 9 Nov 2023cs.SD cs.CV cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

This short paper introduces a workflow for generating realistic soundscapes for visual media. In contrast to prior work, which primarily focus on matching sounds for on-screen visuals, our approach extends to suggesting sounds that may not be immediately visible but are essential to crafting a convincing and immersive auditory environment. Our key insight is leveraging the reasoning capabilities of language models, such as ChatGPT. In this paper, we describe our workflow, which includes creating a scene context, brainstorming sounds, and generating the sounds.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 17 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

11 of 17 references.

References · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  6. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  7. 2022AudioGen: Textually Guided Audio Generationcited by 295
  8. 2018The Sound of Pixelscited by 259
  9. 2016Grad-CAM: Why did you say that?cited by 161
  10. 2020AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learningcited by 19
  11. 2021Soundify: Matching Sound Effects to Videocited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026BackgroundMellow: A Multi-Modal Cohesive Framework for Narrative-Driven Rich Cinematic Soundscape Generation
  2. 2026Audible World Models: Spatially Aware Sound Generation for 3D Worlds
  3. 2026Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds
  4. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing
  5. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
  6. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation
  7. 2025Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation
  8. 2024SonicVisionLM: Playing Sound with Vision Language Models