Identity as Presence: Towards Appearance and Voice Personalized Joint Audio-Video Generation
Not on the map. Why?
Abstract (arXiv)
Recent advances in video synthesis have enabled realistic integration of real individuals, driving demand for identity-aware generation. While emerging methods support joint appearance and voice injection in audio-visual models, they primarily focus on single-subject settings. Multimodal identity integration across multiple subjects remains limited, and precise alignment between visual and vocal identities in multi-subject scenarios remains underexplored. We present Identity-as-Presence, a unified framework for joint personalized audio-video generation. An automated data curation pipeline constructs identity-labeled audio-visual pairs for single- and multi-subject scenes. A unified identity injection mechanism then binds paired appearance and voice through shared cross-modal identity binding and subject-anchored captions. A multi-stage training strategy further leverages large-scale unimodal data alongside scarce paired clips to mitigate modality imbalance. Experiments show superior audio quality, video fidelity, and audio-visual consistency, with stronger multi-subject binding than the compared methods. For more details and qualitative results, please refer to our webpage: hrefhttps://chen-yingjie.github.io/projects/Identity-as-PresenceIdentity-as-Presence.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 53 of its 69 references that are in the map. First posted on arXiv 18 Mar 2026.
Builds on
53 of 69 references.
References · 53
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
- 2025Qwen-Image Technical Reportcited by 973
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
38 more
- 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
- 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
- 2025VACE: All-in-One Video Creation and Editingcited by 467
- 2024YOLOv11: An Overview of the Key Architectural Enhancementscited by 414
- 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
- 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
- 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
- 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
- 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
- 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
- 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
- 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
- 2022MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectorscited by 148
- 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
- 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
- 2025Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Soundcited by 129
- 2025Phantom: Subject-consistent video generation via cross-modal alignmentcited by 126
- 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 122
- 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
- 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
- 2025HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generationcited by 84
- 2024SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesiscited by 81
- 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generationcited by 77
- 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 76
- 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 72
- 2025ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuningcited by 67
- 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 58
- 2025ACE-Step: A Step Towards Music Generation Foundation Modelcited by 54
- 2019Demucs: Deep Extractor for Music Sources with extra unlabeled data remixedcited by 53
- 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
- 2025Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Promptscited by 28
- 2025Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generationcited by 27
- 2025MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglementcited by 22
- 2025InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio Conditionscited by 20
- 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 19
- 2024ERes2NetV2: Boosting Short-Duration Speaker Verification Performance with Computational Efficiencycited by 18
- 20243D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarizationcited by 15
- 2025DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matchingcited by 14
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.