UniBriVL: Robust Universal Representation and Generation of Audio Driven Diffusion Models
On the map: Diffusion models → Image generation, text-to-image and editing
Abstract (arXiv)
Multimodal large models have been recognized for their advantages in various performance and downstream tasks. The development of these models is crucial towards achieving general artificial intelligence in the future. In this paper, we propose a novel universal language representation learning method called UniBriVL, which is based on Bridging-Vision-and-Language (BriVL). Universal BriVL embeds audio, image, and text into a shared space, enabling the realization of various multimodal applications. Our approach addresses major challenges in robust language (both text and audio) representation learning and effectively captures the correlation between audio and image. Additionally, we demonstrate the qualitative evaluation of the generated images from UniBriVL, which serves to highlight the potential of our approach in creating images from audio. Overall, our experimental results demonstrate the efficacy of UniBriVL in downstream tasks and its ability to choose appropriate images from audio. The proposed approach has the potential for various applications such as speech recognition, music signal processing, and captioning systems.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 40 of its 59 references that are in the map. First posted on arXiv 29 Jul 2023.
Builds on
40 of 59 references.
Closest ancestors · 1
Same direction, newest first
Other references · 39
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
27 more
- 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
- 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
- 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
- 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
- 2019Clotho: An Audio Captioning Datasetcited by 354
- 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
- 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
- 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
- 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
- 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
- 2020Talking-head Generation with Rhythmic Head Motioncited by 112
- 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
- 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
- 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
- 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
- 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
- 2020Deep Audio-Visual Learning: A Surveycited by 68
- 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
- 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
- 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
- 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
- 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
- 2019Large-scale representation learning from visually grounded untranscribed speechcited by 33
- 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
- 2018Towards Audio to Scene Image Synthesis using Generative Adversarial Networkcited by 23
- 2022Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognitioncited by 12
- 2022Estimating Visual Information From Audio Through Manifold Learningcited by 7
Built on by
Citing papers per year: 2024: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs
- 2024An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment
- 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
- 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All
- 2024E5-V: Universal Embeddings with Multimodal Large Language Models
- 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning
- 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark
- 2023OneLLM: One Framework to Align All Modalities with Language