OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Data incompleteness severely impedes the reliability of multimodal systems. Existing reconstruction methods face distinct bottlenecks: conventional parametric/generative models are prone to hallucinations due to over-reliance on internal memory, while retrieval-augmented frameworks struggle with retrieval rigidity. Critically, these end-to-end architectures are fundamentally constrained by Semantic-Detail Entanglement -- a structural conflict between logical reasoning and signal synthesis that compromises fidelity. In this paper, we present textbfunderlineOmni-textbfunderlineModality textbfunderlineGeneration Agent (OMG-Agent), a novel framework that shifts the paradigm from static mapping to a dynamic coarse-to-fine Agentic Workflow. By mimicking a textitdeliberate-then-act cognitive process, OMG-Agent explicitly decouples the task into three synergistic stages: (1) an MLLM-driven Semantic Planner that resolves input ambiguity via Progressive Contextual Reasoning, creating a deterministic structured semantic plan; (2) a non-parametric Evidence Retriever that grounds abstract semantics in external knowledge; and (3) a Retrieval-Injected Executor that utilizes retrieved evidence as flexible feature prompts to overcome rigidity and synthesize high-fidelity details. Extensive experiments on multiple benchmarks demonstrate that OMG-Agent consistently surpasses state-of-the-art methods, maintaining robustness under extreme missingness, e.g., a 2.6-point gain on CMU-MOSI at 70% missing rates.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 42 references that are in the map. First posted on arXiv 4 Feb 2026.
Builds on
27 of 42 references.
Closest ancestors · 1
Same direction, newest first
Other references · 26
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Visual Instruction Tuningcited by 7,792
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
- 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
- 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
- 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
- 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
- 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
- 2016On Deep Multi-View Representation Learning: Objectives and Optimizationcited by 162
14 more
- 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
- 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
- 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
- 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
- 2023Unified Multi-Modal Image Synthesis for Missing Modality Imputationcited by 39
- 2024Efficient Tool Use with Chain-of-Abstraction Reasoningcited by 33
- 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
- 2022Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalitiescited by 28
- 2024Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognitioncited by 26
- 2024OG-RAG: Ontology-Grounded Retrieval-Augmented Generation For Large Language Modelscited by 23
- 2025Pre-Act: Multi-Step Planning and Reasoning Improves Acting in LLM Agentscited by 20
- 2024Unity by Diversity: Improved Representation Learning in Multimodal VAEscited by 13
- 2025InsTALL: Context-aware Instructional Task Assistance with Multi-modal Large Language Modelscited by 5
- 2025Plan-X: Instruct Video Generation via Semantic Planningcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything
- 2026Qwen3.8-Omni: Towards Native Omni-Modal Agents
- 2026Reason Before You Retrieve: Agentic Planning for Multi-modal RAG
- 2026XR: Cross-Modal Agents for Composed Image Retrieval
- 2026Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory
- 2026Unify-Agent: A Unified Multimodal Agent for World-Grounded Image Synthesis
- 2026E-mem: Multi-agent based Episodic Context Reconstruction for LLM Agent Memory
- 2025A Unified Multi-Agent Framework for Universal Multimodal Understanding and Generation