mlsanityData to 14 Oct 2026

From Efficient Multimodal Models to World Models: A Survey

arXiv 2407.00118 ↗First posted 27 Jun 2024cs.LG cs.AIcited by 13

On the map: World models → World models for language agents

Abstract (arXiv)

Multimodal Large Models (MLMs) are becoming a significant research focus, combining powerful large language models with multimodal learning to perform complex tasks across different data modalities. This review explores the latest developments and challenges in MLMs, emphasizing their potential in achieving artificial general intelligence and as a pathway to world models. We provide an overview of key techniques such as Multimodal Chain of Thought (M-COT), Multimodal Instruction Tuning (M-IT), and Multimodal In-Context Learning (M-ICL). Additionally, we discuss both the fundamental and specific technologies of multimodal models, highlighting their applications, input/output modalities, and design characteristics. Despite significant advancements, the development of a unified multimodal model remains elusive. We discuss the integration of 3D generation and embodied intelligence to enhance world simulation capabilities and propose incorporating external rule systems for improved reasoning and decision-making. Finally, we outline future research directions to address these challenges and advance the field.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 97 of its 121 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

97 of 121 references.

References · 97

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2022Scalable Diffusion Models with Transformerscited by 6,054
  3. 2022Generative Adversarial Networkscited by 5,627
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  13. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  14. 2020Reformer: The Efficient Transformercited by 1,668
  15. 2020Knowledge Distillation: A Surveycited by 1,603
  16. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  17. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  18. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  19. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  20. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  21. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  22. 2021Variational Diffusion Modelscited by 1,153
  23. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  24. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  25. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  26. 2022Diffusion Models in Vision: A Surveycited by 1,015
  27. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 938
  28. 2023A Survey on Multimodal Large Language Modelscited by 909
  29. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  30. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  31. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  32. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  33. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  34. 2016Language Modeling with Gated Convolutional Networkscited by 702
  35. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  36. 2018Rethinking the Value of Network Pruningcited by 599
  37. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  38. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  2. 20253D and 4D World Modeling: A Surveycited by 67
  3. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  4. 2025MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Modelscited by 30
  5. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  6. 2024All rivers run into the sea: Unified Modality Brain-like Emotional Central Mechanismcited by 5
  7. 2025From 2D to 3D Cognition: A Brief Survey of General World Modelscited by 4
  8. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  9. 2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applicationscited by 3
  10. 2026Three-in-One World Model: Energy-Based Consistency, Prediction, and Counterfactual Inference for Marketing Interventioncited by 1
  11. 2025Lightweight Structured Multimodal Reasoning for Clinical Scene Understanding in Roboticscited by 0
  12. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Surveycited by 0
1 more
  1. 2026Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  2. 2026Toward Native Multimodal Modeling: A Roadmap
  3. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  6. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
  7. 2026The Trinity of Consistency as a Defining Principle for General World Models
  8. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models