mlsanityData to 7 Oct 2026

MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesis

arXiv 2409.07129 ↗First posted 11 Sep 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper introduces MVLLaVA, an intelligent agent designed for novel view synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a large multimodal model, LLaVA, enabling it to handle a wide range of tasks efficiently. MVLLaVA represents a versatile and unified platform that adapts to diverse input types, including a single image, a descriptive caption, or a specific change in viewing azimuth, guided by language instructions for viewpoint generation. We carefully craft task-specific instruction templates, which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires the capability to generate novel view images based on user instructions, demonstrating its flexibility across diverse tasks. Experiments are conducted to validate the effectiveness of MVLLaVA, demonstrating its robust performance and versatility in tackling diverse novel view synthesis challenges.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

43 of 53 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
31 more
  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  7. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  8. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  9. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  10. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  11. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  12. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  13. 2023Instruction Tuning with GPT-4cited by 676
  14. 2023Mip-Splatting: Alias-free 3D Gaussian Splattingcited by 661
  15. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  16. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  17. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 516
  18. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  19. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  20. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  21. 2022Novel View Synthesis with Diffusion Modelscited by 299
  22. 2023Generating Images with Multimodal Language Modelscited by 259
  23. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generationcited by 256
  24. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  25. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
  26. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  27. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  28. 2023HoloDiffusion: Training a 3D Diffusion Model using 2D Imagescited by 106
  29. 2024Advances in 3D Generation: A Surveycited by 62
  30. 2024LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Modelcited by 17
  31. 2024ExpLLM: Towards Chain of Thought for Facial Expression Recognitioncited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Thinking with Novel Views: A Systematic Analysis of Generative-Augmented Spatial Intelligence
  2. 2026M3: High-fidelity Text-to-Image Generation via Multi-Modal, Multi-Agent and Multi-Round Visual Reasoning
  3. 2025UniView: Enhancing Novel View Synthesis From A Single Image By Unifying Reference Features
  4. 2025LLM-I: LLMs are Naturally Interleaved Multimodal Creators
  5. 2024DiffusionAgent: Navigating Expert Models for Agentic Image Generation
  6. 2025CRAFT: Continuous Reasoning and Agentic Feedback Tuning for Multimodal Text-to-Image Generation
  7. 2026LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents
  8. 2025Sphinx: Efficiently Serving Novel View Synthesis using Regression-Guided Selective Refinement