# PhotoBot: Reference-Guided Interactive Photography via Natural Language

arXiv: https://arxiv.org/abs/2401.11061 · HTML: https://mlsanity.com/paper/2401.11061/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 19 Jan 2024.

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [MAGSAC++, a fast, reliable and accurate robust estimator](https://mlsanity.com/paper/1912.05909/) (2019)
- [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022)
- [Best-Buddies Similarity - Robust Template Matching using Mutual Nearest Neighbors](https://mlsanity.com/paper/1609.01571/) (2016)
- [AutoPhoto: Aesthetic Photo Capture using Reinforcement Learning](https://mlsanity.com/paper/2109.09923/) (2021)
- [ANSEL Photobot: A Robot Event Photographer with Semantic Intelligence](https://mlsanity.com/paper/2302.07931/) (2023)
- [Learning to Take Good Pictures of People with a Robot Photographer](https://mlsanity.com/paper/1904.05688/) (2019)
- [Stargazer: An Interactive Camera Robot for Capturing How-To Videos Based on Subtle Instructor Cues](https://mlsanity.com/paper/2303.03221/) (2023)

## Built on by (most cited first, 3 of 3)

- [Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories](https://mlsanity.com/paper/2509.16176/) (2025)
- [How to Take a Memorable Picture? Empowering Users with Actionable Feedback](https://mlsanity.com/paper/2602.21877/) (2026)
- [PhotoAgent: A Robotic Photographer with Spatial and Aesthetic Understanding](https://mlsanity.com/paper/2603.22796/) (2026)

## Abstract (arXiv)

We introduce PhotoBot, a framework for fully automated photo acquisition based on an interplay between high-level human language guidance and a robot photographer. We propose to communicate photography suggestions to the user via reference images that are selected from a curated gallery. We leverage a visual language model (VLM) and an object detector to characterize the reference images via textual descriptions and then use a large language model (LLM) to retrieve relevant reference images based on a user's language query through text-based reasoning. To correspond the reference image and the observed scene, we exploit pre-trained features from a vision transformer capable of capturing semantic similarity across marked appearance variations. Using these features, we compute suggested pose adjustments for an RGB-D camera by solving a perspective-n-point (PnP) problem. We demonstrate our approach using a manipulator equipped with a wrist camera. Our user studies show that photos taken by PhotoBot are often more aesthetically pleasing than those taken by users themselves, as measured by human feedback. We also show that PhotoBot can generalize to other reference sources such as paintings.