mlsanityData to 14 Oct 2026

Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuning

arXiv 2411.16761 ↗First posted 24 Nov 2024cs.CV cs.AIcited by 8

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Multimodal large language models (MLLMs) act as essential interfaces, connecting humans with AI technologies in multimodal applications. However, current MLLMs face challenges in accurately interpreting object orientation in images due to inconsistent orientation annotations in training data, hindering the development of a coherent orientation understanding. To overcome this, we propose egocentric instruction tuning, which aligns MLLMs' orientation understanding with the user's perspective, based on a consistent annotation standard derived from the user's egocentric viewpoint. We first generate egocentric instruction data that leverages MLLMs' ability to recognize object details and applies prior knowledge for orientation understanding. Using this data, we perform instruction tuning to enhance the model's capability for accurate orientation interpretation. In addition, we introduce EgoOrientBench, a benchmark that evaluates MLLMs' orientation understanding across three tasks using images collected from diverse domains. Experimental results on this benchmark show that egocentric instruction tuning significantly improves orientation understanding without compromising overall MLLM performance. The instruction data and benchmark dataset are available on our project page at https://github.com/jhCOR/EgoOrientBench.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 58 references that are in the map. First posted on arXiv 24 Nov 2024.

Builds on

38 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interaction27 Feb
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
  3. 2023Physically Grounded Vision-Language Models for Robotic Manipulation5 Sep
  4. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May
  5. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
21 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  3. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  5. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  8. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  10. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  11. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 326
  12. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  13. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 166
  14. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 95
  15. 2022GPV-Pose: Category-level Object Pose Estimation via Geometry-guided Point-wise Votingcited by 94
  16. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  17. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  18. 2024LLaVA-Docent: Instruction Tuning with Multimodal Large Language Model to Support Art Appreciation Educationcited by 8
  19. 2024LOC-ZSON: Language-driven Object-Centric Zero-Shot Object Retrieval and Navigationcited by 8
  20. 2024Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Realitycited by 8
  21. 2024Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Modelscited by 30
  2. 2026Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoningcited by 7
  3. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5
  4. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  5. 2026Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Textcited by 2
  6. 2026Readable Yet Unpredictable: Rotated-Outcome Prediction in Vision-Language Modelscited by 0
  7. 2026EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenescited by 0
  8. 2026In-Context Learning Can Help Vision-Language Models Overcome Training Priorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Why MLLMs Struggle to Determine Object Orientations
  2. 2026Do MLLMs Understand Pointing? Benchmarking and Enhancing Referential Reasoning in Egocentric Vision
  3. 2025Seeing Isn't Orienting: A Cognitively Informed Hierarchical Benchmark for Object Orientation in MLLMs
  4. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?
  5. 2026OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignment
  6. 2026Empowering Reliable Visual-Centric Instruction Following in MLLMs
  7. 2025Visual Intention Grounding for Egocentric Assistants
  8. 2025RotBench: Evaluating Multimodal Large Language Models on Identifying Image Rotation