# ByDeWay: Boost Your multimodal LLM with DEpth prompting in a Training-Free Way

arXiv: https://arxiv.org/abs/2507.08679 · HTML: https://mlsanity.com/paper/2507.08679/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 11 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 20 of 20)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](https://mlsanity.com/paper/2305.11175/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020)
- [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024)
- [X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning](https://mlsanity.com/paper/2203.00843/) (2022)
- [Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds](https://mlsanity.com/paper/2204.10688/) (2022)
- [List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs](https://mlsanity.com/paper/2404.16375/) (2024)
- [M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning](https://mlsanity.com/paper/2409.15657/) (2024)
- [Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images](https://mlsanity.com/paper/1912.11691/) (2019)
- [BannerAgency: Advertising Banner Design with Multimodal LLM Agents](https://mlsanity.com/paper/2503.11060/) (2025)
- [Hybridnet for depth estimation and semantic segmentation](https://mlsanity.com/paper/2402.06539/) (2024)
- [DEVICE: Depth and Visual Concepts Aware Transformer for OCR-based Image Captioning](https://mlsanity.com/paper/2302.01540/) (2023)
- [Enhancing image captioning with depth information using a Transformer-based framework](https://mlsanity.com/paper/2308.03767/) (2023)

## Built on by (most cited first, 3 of 3)

- [Think3D: Thinking with Space for Spatial Reasoning](https://mlsanity.com/paper/2601.13029/) (2026)
- [When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoning](https://mlsanity.com/paper/2607.11173/) (2026)
- [Explainable and Resource-Efficient Spatial Reasoning in Multimodal LLMs for Decision-Critical Applications](https://mlsanity.com/paper/2607.27145/) (2026)

## Abstract (arXiv)

We introduce ByDeWay, a training-free framework designed to enhance the performance of Multimodal Large Language Models (MLLMs). ByDeWay uses a novel prompting strategy called Layered-Depth-Based Prompting (LDP), which improves spatial reasoning and grounding without modifying any model parameters. It segments the scene into closest, mid-range, and farthest layers using monocular depth estimation, then generates region-specific captions with a grounded vision-language model. These structured, depth-aware captions are appended to the image-question prompt, enriching it with spatial context. This guides MLLMs to produce more grounded and less hallucinated responses. Our method is lightweight, modular, and compatible with black-box MLLMs. Experiments on hallucination-sensitive (POPE) and reasoning-intensive (GQA) benchmarks show consistent improvements across multiple MLLMs, validating the effectiveness of depth-aware prompting in a zero-training setting.