# Exploring Efficient-Tuned Learning Audio Representation Method from BriVL

arXiv: https://arxiv.org/abs/2303.04585 · HTML: https://mlsanity.com/paper/2303.04585/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [Towards artificial general intelligence via a multimodal foundation model](https://mlsanity.com/paper/2110.14378/) (2021)
- [Visual to Sound: Generating Natural Sound for Videos in the Wild](https://mlsanity.com/paper/1712.01393/) (2017)
- [Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose](https://mlsanity.com/paper/2002.10137/) (2020)
- [FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning](https://mlsanity.com/paper/2108.07938/) (2021)
- [Everybody's Talkin': Let Me Talk as You Want](https://mlsanity.com/paper/2001.05201/) (2020)
- [Deep Audio-Visual Learning: A Survey](https://mlsanity.com/paper/2001.04758/) (2020)
- [LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalization](https://mlsanity.com/paper/2106.04185/) (2021)
- [Slow-Fast Auditory Streams For Audio Recognition](https://mlsanity.com/paper/2103.03516/) (2021)
- [Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesis](https://mlsanity.com/paper/2111.00203/) (2021)
- [Large-scale representation learning from visually grounded untranscribed speech](https://mlsanity.com/paper/1909.08782/) (2019)
- [Towards Audio to Scene Image Synthesis using Generative Adversarial Network](https://mlsanity.com/paper/1808.04108/) (2018)
- [Estimating Visual Information From Audio Through Manifold Learning](https://mlsanity.com/paper/2208.02337/) (2022)

## Built on by (most cited first, 1 of 1)

- [Bridging the Gap between Text, Audio, Image, and Any Sequence: A Novel Approach using Gloss-based Annotation](https://mlsanity.com/paper/2410.03146/) (2024)

## Abstract (arXiv)

Recently, researchers have gradually realized that in some cases, the self-supervised pre-training on large-scale Internet data is better than that of high-quality/manually labeled data sets, and multimodal/large models are better than single or bimodal/small models. In this paper, we propose a robust audio representation learning method WavBriVL based on Bridging-Vision-and-Language (BriVL). WavBriVL projects audio, image and text into a shared embedded space, so that multi-modal applications can be realized. We demonstrate the qualitative evaluation of the image generated from WavBriVL as a shared embedded space, with the main purposes of this paper:(1) Learning the correlation between audio and image;(2) Explore a new way of image generation, that is, use audio to generate pictures. Experimental results show that this method can effectively generate appropriate images from audio.