mlsanityData to 7 Oct 2026

CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update

arXiv 2312.10908 ↗First posted 18 Dec 2023cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

Utilizing large language models (LLMs) to compose off-the-shelf visual tools represents a promising avenue of research for developing robust visual assistants capable of addressing diverse visual tasks. However, these methods often overlook the potential for continual learning, typically by freezing the utilized tools, thus limiting their adaptation to environments requiring new knowledge. To tackle this challenge, we propose CLOVA, a Closed-Loop Visual Assistant, which operates within a framework encompassing inference, reflection, and learning phases. During the inference phase, LLMs generate programs and execute corresponding tools to complete assigned tasks. In the reflection phase, a multimodal global-local reflection scheme analyzes human feedback to determine which tools require updating. Lastly, the learning phase employs three flexible approaches to automatically gather training data and introduces a novel prompt tuning scheme to update the tools, allowing CLOVA to efficiently acquire new knowledge. Experimental findings demonstrate that CLOVA surpasses existing tool-usage methods by 5% in visual question answering and multiple-image reasoning, by 10% in knowledge tagging, and by 20% in image editing. These results underscore the significance of the continual learning capability in general visual assistants.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 87 of its 97 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

87 of 97 references.

References · 87

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  9. 2022Visual Prompt Tuningcited by 1,674
  10. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  12. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  13. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  14. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  16. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  17. 2023Teaching Large Language Models to Self-Debugcited by 963
  18. 2019How Can We Know What Language Models Know?cited by 953
  19. 2021Learning to Prompt for Continual Learningcited by 938
  20. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  21. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  22. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  23. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  24. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  25. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  26. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  27. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  28. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  29. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  30. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  31. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  32. 2022Self-critiquing models for assisting human evaluatorscited by 372
  33. 2023Language Models can Solve Computer Taskscited by 371
  34. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  35. 2023Tool Learning with Foundation Modelscited by 302
  36. 2023Large Language Models as Tool Makerscited by 293
  37. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  38. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 19; 2026: 6.

42 papers.

Sort:

By area:

  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  2. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  3. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  4. 2024Tool Learning with Large Language Models: A Surveycited by 221
  5. 2024Large Multimodal Agents: A Surveycited by 99
  6. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  7. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  8. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  9. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  10. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  11. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 24
  12. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
30 more
  1. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledgecited by 20
  2. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  3. 2025Benchmarking Multimodal Knowledge Conflict for Large Multimodal Modelscited by 18
  4. 2024CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planningcited by 14
  5. 2024ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Modelcited by 12
  6. 2024Budget-Constrained Tool Learning with Planningcited by 11
  7. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Modelscited by 11
  8. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  9. 2024SoAy: A Solution-based LLM API-using Methodology for Academic Information Seekingcited by 7
  10. 2024FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Modelscited by 6
  11. 2024Collaborative Evolving Strategy for Automatic Data-Centric Developmentcited by 6
  12. 2025CoSTAast: Cost-Sensitive Toolpath Agent for Multi-turn Image Editingcited by 5
  13. 2025Building LLM Agents by Incorporating Insights from Computer Systemscited by 4
  14. 2025MV-Debate: Multi-view Agent Debate with Dynamic Reflection Gating for Multimodal Harmful Content Detection in Social Mediacited by 4
  15. 2024ViUniT: Visual Unit Tests for More Robust Visual Programmingcited by 3
  16. 2025FaSTA^*: Fast-Slow Toolpath Agent with Subroutine Mining for Efficient Multi-turn Image Editingcited by 3
  17. 2026Omni-MMSI: Toward Identity-attributed Social Interaction Understandingcited by 3
  18. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  19. 2025AgriDoctor: A Multimodal Intelligent Assistant for Agriculturecited by 2
  20. 2026HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenescited by 2
  21. 2024Large Language Models, and LLM-Based Agents, Should Be Used to Enhance the Digital Public Spherecited by 1
  22. 2025Multimodal Agricultural Agent Architecture (MA3): A New Paradigm for Intelligent Agricultural Decision-Makingcited by 1
  23. 2025Efficiently Enhancing General Agents With Hierarchical-categorical Memorycited by 1
  24. 2025Long-Horizon Visual Imitation Learning via Plan and Code Reflectioncited by 1
  25. 2025MATRIX: Multimodal Agent Tuning for Robust Tool-Use Reasoningcited by 1
  26. 2025Enhancing Visual Programming for Visual Reasoning via Probabilistic Graphscited by 1
  27. 2026PerfGuard: A Performance-Aware Agent for Visual Content Generationcited by 1
  28. 2026Evolve Vision-Language-Action Model into an Agent with On-the-fly Tool-usecited by 1
  29. 2026ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generationcited by 0
  30. 2026ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agentscited by 0