mlsanityData to 14 Oct 2026

LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing

arXiv 2311.00571 ↗First posted 1 Nov 2023cs.CV cs.AI cs.CLcited by 47

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

LLaVA-Interactive is a research prototype for multimodal human-AI interaction. The system can have multi-turn dialogues with human users by taking multimodal user inputs and generating multimodal responses. Importantly, LLaVA-Interactive goes beyond language prompt, where visual prompt is enabled to align human intents in the interaction. The development of LLaVA-Interactive is extremely cost-efficient as the system combines three multimodal skills of pre-built AI models without additional model training: visual chat of LLaVA, image segmentation from SEEM, as well as image generation and editing from GLIGEN. A diverse set of application scenarios is presented to demonstrate the promises of LLaVA-Interactive and to inspire future research in multimodal interactive systems.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

27 of 36 references.

References · 27

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  7. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
  8. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  9. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  10. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  11. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  12. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
15 more
  1. 2023Segment Everything Everywhere All at Oncecited by 548
  2. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  3. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  4. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  5. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  6. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  7. 2023Generating Images with Multimodal Language Modelscited by 260
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2023SegGPT: Segmenting Everything In Contextcited by 194
  10. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 170
  11. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  12. 2022A Unified Sequence Interface for Vision Taskscited by 152
  13. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  14. 2022Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Taskscited by 57
  15. 2021Towards General Purpose Vision Systemscited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 21; 2025: 15; 2026: 6.

47 papers.

Sort:

By area:

  1. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  2. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  3. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  4. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 109
  5. 2024Large Multimodal Agents: A Surveycited by 105
  6. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  7. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 49
  8. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  9. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  10. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  11. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  12. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 29
35 more
  1. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  2. 2024Holistic Evaluation for Interleaved Text-and-Image Generationcited by 28
  3. 2024DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generationcited by 26
  4. 2025Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibrationcited by 23
  5. 2025MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generationcited by 23
  6. 2025CROP: Contextual Region-Oriented Visual Token Pruningcited by 14
  7. 2024UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Modelcited by 13
  8. 2023Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Casescited by 12
  9. 2024SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editingcited by 11
  10. 2024DeiSAM: Segment Anything with Deictic Promptingcited by 9
  11. 2024Empowering Segmentation Ability to Multi-modal Large Language Modelscited by 8
  12. 2024Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effortcited by 7
  13. 2024ReGround: Improving Textual and Spatial Grounding at No Costcited by 6
  14. 2024NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Visioncited by 5
  15. 2024Generative Timelines for Instructed Visual Assemblycited by 5
  16. 2025Building LLM Agents by Incorporating Insights from Computer Systemscited by 5
  17. 2024Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaborationcited by 4
  18. 2025Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encodingcited by 4
  19. 2025The Power of One: A Single Example is All it Takes for Segmentation in VLMscited by 4
  20. 2025QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQAcited by 4
  21. 2025ZeroVO: Visual Odometry with Minimal Assumptionscited by 4
  22. 2025FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentationcited by 4
  23. 2025Ego-centric Predictive Model Conditioned on Hand Trajectoriescited by 4
  24. 2026Anticipatory Planning for Multimodal AI Agentscited by 4
  25. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Modelcited by 3
  26. 2024Temporal Grounding of Activities using Multimodal Large Language Modelscited by 3
  27. 2025Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Groundercited by 2
  28. 2025Training-Free Multimodal Large Language Model Orchestrationcited by 2
  29. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  30. 2026Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learningcited by 1
  31. 2026Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarificationcited by 1
  32. 2025Foundation Model-Driven Framework for Human-Object Interaction Prediction with Segmentation Mask Integrationcited by 0
  33. 2026Topology-Aware Layer Pruning for Large Vision-Language Modelscited by 0
  34. 2026A Comprehensive Study of Implementation Bugs in Multi-modal Agentscited by 0
  35. 2026Reason Before You Retrieve: Agentic Planning for Multi-modal RAGcited by 0