mlsanityData to 7 Oct 2026

DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)

arXiv 2401.08392 ↗First posted 16 Jan 2024cs.CV cs.CLcited by 56

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Recent LLM-driven visual agents mainly focus on solving image-based tasks, which limits their ability to understand dynamic scenes, making it far from real-life applications like guiding students in laboratory experiments and identifying their mistakes. Hence, this paper explores DoraemonGPT, a comprehensive and conceptually elegant system driven by LLMs to understand dynamic scenes. Considering the video modality better reflects the ever-changing nature of real-world scenarios, we exemplify DoraemonGPT as a video agent. Given a video with a question/task, DoraemonGPT begins by converting the input video into a symbolic memory that stores task-related attributes. This structured representation allows for spatial-temporal querying and reasoning by well-designed sub-task tools, resulting in concise intermediate results. Recognizing that LLMs have limited internal knowledge when it comes to specialized domains (e.g., analyzing the scientific principles underlying experiments), we incorporate plug-and-play tools to assess external knowledge and address tasks across different domains. Moreover, a novel LLM-driven planner based on Monte Carlo Tree Search is introduced to explore the large planning space for scheduling various tools. The planner iteratively finds feasible solutions by backpropagating the result's reward, and multiple solutions can be summarized into an improved final answer. We extensively evaluate DoraemonGPT's effectiveness on three benchmarks and several in-the-wild scenarios. The code will be released at https://github.com/z-x-yang/DoraemonGPT.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 103 of its 131 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

103 of 131 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 102

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Segment Anythingcited by 8,158
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
38 more
  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  13. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  14. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  15. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  16. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  17. 2021ViViT: A Video Vision Transformercited by 1,610
  18. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  19. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  20. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  21. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  22. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  23. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  24. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  25. 2023PaLM 2 Technical Reportcited by 1,154
  26. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  27. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  28. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  29. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  30. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  31. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  32. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  33. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  34. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  35. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  36. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  37. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  38. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511

50 most cited of 102 All 102 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 16; 2026: 13.

56 papers.

Sort:

By area:

  1. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  2. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  3. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  4. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  5. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  6. 2024Large Multimodal Agents: A Surveycited by 99
  7. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  8. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  9. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  10. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  11. 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35
  12. 2024HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splattingcited by 33
38 more
  1. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  2. 2024CapHuman: Capture Your Moments in Parallel Universescited by 22
  3. 2024Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activitycited by 22
  4. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  5. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  6. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  7. 2024Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoningcited by 16
  8. 2024VividDreamer: Invariant Score Distillation For Hyper-Realistic Text-to-3D Generationcited by 15
  9. 2025Agentic Keyframe Search for Video Question Answeringcited by 13
  10. 2025ToolVQA: A Dataset for Multi-step Reasoning VQA with External Toolscited by 13
  11. 2025AdsQA: Towards Advertisement Video Understandingcited by 12
  12. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  13. 2024Towards Hierarchical Multi-Agent Workflows for Zero-Shot Prompt Optimizationcited by 10
  14. 2024Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generationcited by 10
  15. 2025DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generationcited by 10
  16. 2026VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoningcited by 7
  17. 2025FieldWorkArena: Agentic AI Benchmark for Real Field Work Taskscited by 6
  18. 2025AViLA: Asynchronous Vision-Language Agent for Streaming Multimodal Data Interactioncited by 6
  19. 2024FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filteringcited by 5
  20. 2024Agent-based Video Trimmingcited by 4
  21. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  22. 2025Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectioncited by 4
  23. 2025T^2Agent A Tool-augmented Multimodal Misinformation Detection Agent with Monte Carlo Tree Searchcited by 4
  24. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  25. 2025SciEducator: Scientific Video Understanding and Educating via Deming-Cycle Multi-Agent Systemcited by 4
  26. 2024Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Modelscited by 3
  27. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  28. 2024Balancing Rigor and Utility: Mitigating Cognitive Biases in Large Language Models for Multiple-Choice Questionscited by 2
  29. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  30. 2025COLT: Enhancing Video Large Language Models with Continual Tool Usagecited by 2
  31. 2026TRACE: High-Fidelity 3D Scene Editing via Tangible Reconstruction and Geometry-Aligned Contextual Video Maskingcited by 2
  32. 2024PiPa++: Towards Unification of Domain Adaptive Semantic Segmentation via Self-supervised Learningcited by 1
  33. 2026SinkTrack: Attention Sink based Context Anchoring for Large Language Modelscited by 1
  34. 2026HiCrew: Hierarchical Reasoning for Long-Form Video Understanding via Question-Aware Multi-Agent Collaborationcited by 1
  35. 2026ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Groundingcited by 1
  36. 2026Pedestrian-Aware LLM-Driven Behavioral Planning for Autonomous Vehiclescited by 1
  37. 2024Mutual Learning for Acoustic Matching and Dereverberation via Visual Scene-driven Diffusioncited by 0
  38. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation
  2. 2026Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoning
  3. 2026VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis
  4. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning
  5. 2026V-CORE: Temporally Consistent Video Understanding for Video-LLM
  6. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMs
  7. 2025Video-QTR: Query-Driven Temporal Reasoning Framework for Lightweight Video Understanding
  8. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data