mlsanityData to 14 Oct 2026

Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignment

arXiv 2503.09081 ↗First posted 12 Mar 2025cs.CV cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

While multi-modal learning has advanced significantly, current approaches often create inconsistencies in representation and reasoning of different modalities. We propose UMaT, a theoretically-grounded framework that unifies visual and auditory inputs as structured text for large language models, addressing semantic alignment, temporal synchronization, and efficient sparse information retrieval. It significantly improves state-of-the-art Long Video Question Answering accuracy (up to 13.7%, and 16.9% on long videos) via redundancy minimization and structured textual representation for unified multi-modal reasoning

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 41 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

27 of 41 references.

References · 27

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  6. 2024Long Context Transfer from Language to Visioncited by 515
  7. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  8. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  9. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  10. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  11. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  12. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
15 more
  1. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  2. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  3. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  4. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  5. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 146
  6. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 138
  7. 2020Memory Enhanced Global-Local Aggregation for Video Object Detectioncited by 128
  8. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  9. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  10. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
  11. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94
  12. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  13. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  14. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  15. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questionscited by 14

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  2. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
  3. 2025F4-ITS: Fine-grained Feature Fusion for Food Image-Text Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MANTA: Cross-Modal Semantic Alignment and Information-Theoretic Optimization for Long-form Multimodal Understanding
  2. 2026HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering
  3. 2021Cross-Modal Retrieval Augmentation for Multi-Modal Classification
  4. 2025RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Language
  5. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities
  6. 2025A Video Is Not Worth a Thousand Words
  7. 2022Clover: Towards A Unified Video-Language Alignment and Fusion Model
  8. 2025From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignment