# Composition Vision-Language Understanding via Segment and Depth Anything Model

arXiv: https://arxiv.org/abs/2406.18591 · HTML: https://mlsanity.com/paper/2406.18591/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 9 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 9 of 9)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [Human-oriented Representation Learning for Robotic Manipulation](https://mlsanity.com/paper/2310.03023/) (2023)
- [Joint Pedestrian Trajectory Prediction through Posterior Sampling](https://mlsanity.com/paper/2404.00237/) (2024)

## Built on by (most cited first, 2 of 2)

- [Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving](https://mlsanity.com/paper/2501.06680/) (2025)
- [First-place Solution for Streetscape Shop Sign Recognition Competition](https://mlsanity.com/paper/2501.02811/) (2025)

## Abstract (arXiv)

We introduce a pioneering unified library that leverages depth anything, segment anything models to augment neural comprehension in language-vision model zero-shot understanding. This library synergizes the capabilities of the Depth Anything Model (DAM), Segment Anything Model (SAM), and GPT-4V, enhancing multimodal tasks such as vision-question-answering (VQA) and composition reasoning. Through the fusion of segmentation and depth analysis at the symbolic instance level, our library provides nuanced inputs for language models, significantly advancing image interpretation. Validated across a spectrum of in-the-wild real-world images, our findings showcase progress in vision-language models through neural-symbolic integration. This novel approach melds visual and language analysis in an unprecedented manner. Overall, our library opens new directions for future research aimed at decoding the complexities of the real world through advanced multimodal technologies and our code is available at urlhttps://github.com/AnthonyHuo/SAM-DAM-for-Compositional-Reasoning.