# Expressing Visual Relationships via Language

arXiv: https://arxiv.org/abs/1906.07689 · HTML: https://mlsanity.com/paper/1906.07689/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2019.

## Built on by (most cited first, 30 of 49)

- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Guiding Instruction-based Image Editing via Multimodal Large Language Models](https://mlsanity.com/paper/2309.17102/) (2023)
- [MileBench: Benchmarking MLLMs in Long Context](https://mlsanity.com/paper/2404.18532/) (2024)
- [Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions](https://mlsanity.com/paper/2308.04152/) (2023)
- [Towards Semantic Equivalence of Tokenization in Multimodal LLM](https://mlsanity.com/paper/2406.05127/) (2024)
- [Image Difference Captioning with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2202.04298/) (2022)
- [Visual Instruction Tuning with Polite Flamingo](https://mlsanity.com/paper/2307.01003/) (2023)
- [Learning Action and Reasoning-Centric Image Editing from Videos and Simulations](https://mlsanity.com/paper/2407.03471/) (2024)
- [Auto-Encoding Morph-Tokens for Multimodal LLM](https://mlsanity.com/paper/2405.01926/) (2024)
- [Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens](https://mlsanity.com/paper/2504.14666/) (2025)
- [CLIP4IDC: CLIP for Image Difference Captioning](https://mlsanity.com/paper/2206.00629/) (2022)
- [Towards Generic Image Manipulation Detection with Weakly-Supervised Self-Consistency Learning](https://mlsanity.com/paper/2309.01246/) (2023)
- [Self-supervised Cross-view Representation Reconstruction for Change Captioning](https://mlsanity.com/paper/2309.16283/) (2023)
- [Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models](https://mlsanity.com/paper/2408.04594/) (2024)
- [Neighborhood Contrastive Transformer for Change Captioning](https://mlsanity.com/paper/2303.03171/) (2023)
- [OneDiff: A Generalist Model for Image Difference Captioning](https://mlsanity.com/paper/2407.05645/) (2024)
- [REALEDIT: Reddit Edits As a Large-scale Empirical Dataset for Image Transformations](https://mlsanity.com/paper/2502.03629/) (2025)
- [TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contexts](https://mlsanity.com/paper/2512.22748/) (2025)
- [CCExpert: Advancing MLLM Capability in Remote Sensing Change Captioning with Difference-Aware Integration and a Foundational Dataset](https://mlsanity.com/paper/2411.11360/) (2024)
- [FixMyPose: Pose Correctional Captioning and Retrieval](https://mlsanity.com/paper/2104.01703/) (2021)
- [R3Net:Relation-embedded Representation Reconstruction Network for Change Captioning](https://mlsanity.com/paper/2110.10328/) (2021)
- [VIXEN: Visual Text Comparison Network for Image Difference Captioning](https://mlsanity.com/paper/2402.19119/) (2024)
- [OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning](https://mlsanity.com/paper/2503.11093/) (2025)
- [MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference](https://mlsanity.com/paper/2506.15724/) (2025)
- [L2C: Describing Visual Differences Needs Semantic Understanding of Individuals](https://mlsanity.com/paper/2102.01860/) (2021)
- [Rethinking the Reference-based Distinctive Image Captioning](https://mlsanity.com/paper/2207.11118/) (2022)
- [Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLM](https://mlsanity.com/paper/2505.17726/) (2025)
- [Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning](https://mlsanity.com/paper/2407.11683/) (2024)
- [FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inference](https://mlsanity.com/paper/2511.05534/) (2025)
- [Understanding Generative AI Capabilities in Everyday Image Editing Tasks](https://mlsanity.com/paper/2505.16181/) (2025)

## Abstract (arXiv)

Describing images with text is a fundamental problem in vision-language research. Current studies in this domain mostly focus on single image captioning. However, in various real applications (e.g., image editing, difference interpretation, and retrieval), generating relational captions for two images, can also be very useful. This important problem has not been explored mostly due to lack of datasets and effective models. To push forward the research in this direction, we first introduce a new language-guided image editing dataset that contains a large number of real image pairs with corresponding editing instructions. We then propose a new relational speaker model based on an encoder-decoder architecture with static relational attention and sequential multi-head attention. We also extend the model with dynamic relational attention, which calculates visual alignment while decoding. Our models are evaluated on our newly collected and two public datasets consisting of image pairs annotated with relationship sentences. Experimental results, based on both automatic and human evaluation, demonstrate that our model outperforms all baselines and existing methods on all the datasets.