# OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows

arXiv: https://arxiv.org/abs/2602.04144 · HTML: https://mlsanity.com/paper/2602.04144/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [On Deep Multi-View Representation Learning: Objectives and Optimization](https://mlsanity.com/paper/1602.01024/) (2016)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Contextual Object Detection with Multimodal Large Language Models](https://mlsanity.com/paper/2305.18279/) (2023)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Unified Multi-Modal Image Synthesis for Missing Modality Imputation](https://mlsanity.com/paper/2304.05340/) (2023)
- [Efficient Tool Use with Chain-of-Abstraction Reasoning](https://mlsanity.com/paper/2401.17464/) (2024)
- [Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities](https://mlsanity.com/paper/2404.16456/) (2024)
- [Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities](https://mlsanity.com/paper/2204.13707/) (2022)
- [Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2407.05374/) (2024)
- [Pre-Act: Multi-Step Planning and Reasoning Improves Acting in LLM Agents](https://mlsanity.com/paper/2505.09970/) (2025)
- [OG-RAG: Ontology-Grounded Retrieval-Augmented Generation For Large Language Models](https://mlsanity.com/paper/2412.15235/) (2024)
- [Unity by Diversity: Improved Representation Learning in Multimodal VAEs](https://mlsanity.com/paper/2403.05300/) (2024)
- [InsTALL: Context-aware Instructional Task Assistance with Multi-modal Large Language Models](https://mlsanity.com/paper/2501.12231/) (2025)
- [Plan-X: Instruct Video Generation via Semantic Planning](https://mlsanity.com/paper/2511.17986/) (2025)
- [Reasoning-Driven Amodal Completion: Collaborative Agents and Perceptual Evaluation](https://mlsanity.com/paper/2512.20936/) (2025)

## Abstract (arXiv)

Data incompleteness severely impedes the reliability of multimodal systems. Existing reconstruction methods face distinct bottlenecks: conventional parametric/generative models are prone to hallucinations due to over-reliance on internal memory, while retrieval-augmented frameworks struggle with retrieval rigidity. Critically, these end-to-end architectures are fundamentally constrained by Semantic-Detail Entanglement -- a structural conflict between logical reasoning and signal synthesis that compromises fidelity. In this paper, we present textbfunderlineOmni-textbfunderlineModality textbfunderlineGeneration Agent (OMG-Agent), a novel framework that shifts the paradigm from static mapping to a dynamic coarse-to-fine Agentic Workflow. By mimicking a textitdeliberate-then-act cognitive process, OMG-Agent explicitly decouples the task into three synergistic stages: (1) an MLLM-driven Semantic Planner that resolves input ambiguity via Progressive Contextual Reasoning, creating a deterministic structured semantic plan; (2) a non-parametric Evidence Retriever that grounds abstract semantics in external knowledge; and (3) a Retrieval-Injected Executor that utilizes retrieved evidence as flexible feature prompts to overcome rigidity and synthesize high-fidelity details. Extensive experiments on multiple benchmarks demonstrate that OMG-Agent consistently surpasses state-of-the-art methods, maintaining robustness under extreme missingness, e.g., a 2.6-point gain on CMU-MOSI at 70% missing rates.