Exploring Efficient-Tuned Learning Audio Representation Method from BriVL
Not on the map. Why?
Abstract (arXiv)
Recently, researchers have gradually realized that in some cases, the self-supervised pre-training on large-scale Internet data is better than that of high-quality/manually labeled data sets, and multimodal/large models are better than single or bimodal/small models. In this paper, we propose a robust audio representation learning method WavBriVL based on Bridging-Vision-and-Language (BriVL). WavBriVL projects audio, image and text into a shared embedded space, so that multi-modal applications can be realized. We demonstrate the qualitative evaluation of the image generated from WavBriVL as a shared embedded space, with the main purposes of this paper:(1) Learning the correlation between audio and image;(2) Explore a new way of image generation, that is, use audio to generate pictures. Experimental results show that this method can effectively generate appropriate images from audio.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 8 Mar 2023.
Builds on
30 of 42 references.
References · 30
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
18 more
- 2019Clotho: An Audio Captioning Datasetcited by 354
- 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
- 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
- 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
- 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
- 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
- 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
- 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
- 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
- 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
- 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
- 2020Deep Audio-Visual Learning: A Surveycited by 68
- 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
- 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
- 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
- 2019Large-scale representation learning from visually grounded untranscribed speechcited by 33
- 2018Towards Audio to Scene Image Synthesis using Generative Adversarial Networkcited by 23
- 2022Estimating Visual Information From Audio Through Manifold Learningcited by 7
Built on by
Citing papers per year: 2024: 1.
1 paper.