# Semantic Alignment for Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2408.12867 · HTML: https://mlsanity.com/paper/2408.12867/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 54 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning](https://mlsanity.com/paper/2309.07915/) (2023)
- [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018)
- [Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions](https://mlsanity.com/paper/2308.04152/) (2023)
- [DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalization](https://mlsanity.com/paper/2209.05227/) (2022)
- [RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training](https://mlsanity.com/paper/2303.00534/) (2023)
- [Image Retrieval from Contextual Descriptions](https://mlsanity.com/paper/2203.15867/) (2022)
- [iEdit: Localised Text-guided Image Editing with Weak Supervision](https://mlsanity.com/paper/2305.05947/) (2023)
- [Context-Aware Group Captioning via Self-Attention and Contrastive Features](https://mlsanity.com/paper/2004.03708/) (2020)
- [Exploring Stochastic Autoregressive Image Modeling for Visual Representation](https://mlsanity.com/paper/2212.01610/) (2022)
- [ModelGPT: Unleashing LLM's Capabilities for Tailored Model Generation](https://mlsanity.com/paper/2402.12408/) (2024)
- [Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Model](https://mlsanity.com/paper/2308.01126/) (2023)
- [Unsupervised Discovery of Multimodal Links in Multi-image, Multi-sentence Documents](https://mlsanity.com/paper/1904.07826/) (2019)
- [Ordered Attention for Coherent Visual Storytelling](https://mlsanity.com/paper/2108.02180/) (2021)
- [Detecting and Grounding Important Characters in Visual Stories](https://mlsanity.com/paper/2303.17647/) (2023)
- [COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignment](https://mlsanity.com/paper/2308.03475/) (2023)
- [Improving Reference-based Distinctive Image Captioning with Contrastive Rewards](https://mlsanity.com/paper/2306.14259/) (2023)
- [Multi-Image Summarization: Textual Summary from a Set of Cohesive Images](https://mlsanity.com/paper/2006.08686/) (2020)
- [Text-Only Training for Visual Storytelling](https://mlsanity.com/paper/2308.08881/) (2023)
- [Transform, Contrast and Tell: Coherent Entity-Aware Multi-Image Captioning](https://mlsanity.com/paper/2302.02124/) (2023)

## Built on by (most cited first, 12 of 12)

- [AutoGeo: Automating Geometric Image Dataset Creation for Enhanced Geometry Understanding](https://mlsanity.com/paper/2409.09039/) (2024)
- [Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning](https://mlsanity.com/paper/2503.12972/) (2025)
- [Show and Polish: Reference-Guided Identity Preservation in Face Video Restoration](https://mlsanity.com/paper/2507.10293/) (2025)
- [Position: Foundation Models Need Digital Twin Representations](https://mlsanity.com/paper/2505.03798/) (2025)
- [Multi-speaker Attention Alignment for Multimodal Social Interaction](https://mlsanity.com/paper/2511.17952/) (2025)
- [Mnemis: Dual-Route Retrieval on Hierarchical Graphs for Long-Term LLM Memory](https://mlsanity.com/paper/2602.15313/) (2026)
- [Contrastive Cross-Course Knowledge Tracing via Concept Graph Guided Knowledge Transfer](https://mlsanity.com/paper/2505.13489/) (2025)
- [UrbanFeel: A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective](https://mlsanity.com/paper/2509.22228/) (2025)
- [WorldEdit: Towards Open-World Image Editing with a Knowledge-Informed Benchmark](https://mlsanity.com/paper/2602.07095/) (2026)
- [TPA: Temporal Prompt Alignment for Fetal Congenital Heart Defect Classification](https://mlsanity.com/paper/2508.15298/) (2025)
- [DeCoT: Decomposing Complex Instructions for Enhanced Text-to-Image Generation with Large Language Models](https://mlsanity.com/paper/2508.12396/) (2025)
- [Towards Accurate UAV Image Perception: Guiding Vision-Language Models with Stronger Task Prompts](https://mlsanity.com/paper/2512.07302/) (2025)

## Abstract (arXiv)

Research on Multi-modal Large Language Models (MLLMs) towards the multi-image cross-modal instruction has received increasing attention and made significant progress, particularly in scenarios involving closely resembling images (e.g., change captioning). Existing MLLMs typically follow a two-step process in their pipelines: first, extracting visual tokens independently for each input image, and then aligning these visual tokens from different images with the Large Language Model (LLM) in its textual feature space. However, the independent extraction of visual tokens for each image may result in different semantics being prioritized for different images in the first step, leading to a lack of preservation of linking information among images for subsequent LLM analysis. This issue becomes more serious in scenarios where significant variations exist among the images (e.g., visual storytelling). To address this challenge, we introduce Semantic Alignment for Multi-modal large language models (SAM). By involving the bidirectional semantic guidance between different images in the visual-token extraction process, SAM aims to enhance the preservation of linking information for coherent analysis and align the semantics of different images before feeding them into LLM. As the test bed, we propose a large-scale dataset named MmLINK consisting of 69K samples. Different from most existing datasets for MLLMs fine-tuning, our MmLINK dataset comprises multi-modal instructions with significantly diverse images. Extensive experiments on the group captioning task and the storytelling task prove the effectiveness of our SAM model, surpassing the state-of-the-art methods by a large margin (+37% for group captioning and +22% for storytelling on CIDEr score). Project page: https://mccartney01.github.io/SAM.