# RotBench: Evaluating Multimodal Large Language Models on Identifying Image Rotation

arXiv: https://arxiv.org/abs/2508.13968 · HTML: https://mlsanity.com/paper/2508.13968/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 24 of its 61 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models](https://mlsanity.com/paper/2411.06048/) (2024)
- [CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Counting](https://mlsanity.com/paper/2504.15485/) (2025)
- [CTRL-C: Camera calibration TRansformer with Line-Classification](https://mlsanity.com/paper/2109.02259/) (2021)
- [UprightNet: Geometry-Aware Camera Orientation Estimation from Single Images](https://mlsanity.com/paper/1908.07070/) (2019)
- [Learning Rotation-Equivariant Features for Visual Correspondence](https://mlsanity.com/paper/2303.15472/) (2023)
- [Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2501.04670/) (2025)
- [Analysing the Robustness of Vision-Language-Models to Common Corruptions](https://mlsanity.com/paper/2504.13690/) (2025)
- [Visual Chirality](https://mlsanity.com/paper/2006.09512/) (2020)
- [Neural Geometric Parser for Single Image Camera Calibration](https://mlsanity.com/paper/2007.11855/) (2020)
- [On the Limitations of Vision-Language Models in Understanding Image Transforms](https://mlsanity.com/paper/2503.09837/) (2025)
- [Substance or Style: What Does Your Image Embedding Know?](https://mlsanity.com/paper/2307.05610/) (2023)
- [Automatic Photo Orientation Detection with Convolutional Neural Networks](https://mlsanity.com/paper/2305.10319/) (2023)

## Built on by (most cited first, 4 of 4)

- [Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance](https://mlsanity.com/paper/2604.01848/) (2026)
- [Why MLLMs Struggle to Determine Object Orientations](https://mlsanity.com/paper/2604.13321/) (2026)
- [When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise](https://mlsanity.com/paper/2605.05045/) (2026)
- [Readable Yet Unpredictable: Rotated-Outcome Prediction in Vision-Language Models](https://mlsanity.com/paper/2606.07641/) (2026)

## Abstract (arXiv)

We investigate to what extent Multimodal Large Language Models (MLLMs) can accurately identify the orientation of input images rotated 0deg, 90deg, 180deg, and 270deg. This task demands robust visual reasoning capabilities to detect rotational cues and contextualize spatial relationships within images, regardless of their orientation. To evaluate MLLMs on these abilities, we introduce RotBench, a 350-image manually-filtered benchmark comprising lifestyle, portrait, and landscape images. Despite the relatively simple nature of this task, we show that several state-of-the-art open and proprietary MLLMs, including GPT-5, o3, and Gemini-2.5-Pro, do not reliably identify rotation in input images. Providing models with auxiliary information -- including captions, depth maps, and more -- or using chain-of-thought prompting offers only small and inconsistent improvements. Our results indicate that most models are able to reliably identify right-side-up (0deg) images, while certain models are able to identify upside-down (180deg) images. None can reliably distinguish between 90deg and 270deg rotated images. Simultaneously showing the image rotated in different orientations leads to moderate performance gains for reasoning models, while a modified setup using voting improves the performance of weaker models. We further show that fine-tuning does not improve models' ability to distinguish 90deg and 270deg rotations, despite substantially improving the identification of 180deg images. Together, these results reveal a significant gap between MLLMs' spatial reasoning capabilities and human perception in identifying rotation.