mlsanityData to 14 Oct 2026

Visual AI and Linguistic Intelligence Through Steerability and Composability

arXiv 2312.12383 ↗First posted 18 Nov 2023cs.AI cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This study explores the capabilities of multimodal large language models (LLMs) in handling challenging multistep tasks that integrate language and vision, focusing on model steerability, composability, and the application of long-term memory and context understanding. The problem addressed is the LLM's ability (Nov 2023 GPT-4 Vision Preview) to manage tasks that require synthesizing visual and textual information, especially where stepwise instructions and sequential logic are paramount. The research presents a series of 14 creatively and constructively diverse tasks, ranging from AI Lego Designing to AI Satellite Image Analysis, designed to test the limits of current LLMs in contexts that previously proved difficult without extensive memory and contextual understanding. Key findings from evaluating 800 guided dialogs include notable disparities in task completion difficulty. For instance, 'Image to Ingredient AI Bartender' (Low difficulty) contrasted sharply with 'AI Game Self-Player' (High difficulty), highlighting the LLM's varying proficiency in processing complex visual data and generating coherent instructions. Tasks such as 'AI Genetic Programmer' and 'AI Negotiator' showed high completion difficulty, emphasizing challenges in maintaining context over multiple steps. The results underscore the importance of developing LLMs that combine long-term memory and contextual awareness to mimic human-like thought processes in complex problem-solving scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 42 references that are in the map. First posted on arXiv 18 Nov 2023.

Builds on

14 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  2. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep

Other references · 12

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  4. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  5. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  6. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  7. 2019MUREL: Multimodal Relational Reasoning for Visual Question Answeringcited by 68
  8. 2023Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generationcited by 38
  9. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  10. 2021Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neuronscited by 29
  11. 2023The Multimodal And Modular Ai Chef: Complex Recipe Generation From Imagerycited by 5
  12. 2023Professional Certification Benchmark Dataset: The First 500 Jobs For Large Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
  2. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
  3. 2023Visual Instruction Tuning
  4. 2025Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models
  5. 2026From Instructions to Assistance: a Dataset Aligning Instruction Manuals with Assembly Videos for Evaluating Multimodal LLMs
  6. 2025Evaluating Large Language Models on the Frame and Symbol Grounding Problems: A Zero-shot Benchmark
  7. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches
  8. 2025What do vision-language models see in the context? Investigating multimodal in-context learning