mlsanityData to 7 Oct 2026

OCTO+: A Suite for Automatic Open-Vocabulary Object Placement in Mixed Reality

arXiv 2401.08973 ↗First posted 17 Jan 2024cs.CV cs.AI cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

One key challenge in Augmented Reality is the placement of virtual content in natural locations. Most existing automated techniques can only work with a closed-vocabulary, fixed set of objects. In this paper, we introduce and evaluate several methods for automatic object placement using recent advances in open-vocabulary vision-language models. Through a multifaceted evaluation, we identify a new state-of-the-art method, OCTO+. We also introduce a benchmark for automatically evaluating the placement of virtual objects in augmented reality, alleviating the need for costly user studies. Through this, in addition to human evaluations, we find that OCTO+ places objects in a valid region over 70% of the time, outperforming other methods on a range of metrics.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 15 of its 31 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

15 of 31 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2021Image Segmentation Using Text and Image Promptscited by 509
  12. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
3 more
  1. 2022Can language models learn from explanations in context?cited by 248
  2. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  3. 2023OCTOPUS: Open-vocabulary Content Tracking and Object Placement Using Semantic Understanding in Mixed Realitycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2025PlaceIt3D: Language-Guided Object Placement in Real 3D Scenescited by 8
  2. 2025Detecting Visual Information Manipulation Attacks in Augmented Reality: A Multimodal Semantic Reasoning Approachcited by 6
  3. 2024Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AIcited by 4
  4. 2025Slot-Level Robotic Placement via Visual Imitation from Single Human Videocited by 2
  5. 2026Benchmarking Vision-Language Models under Contradictory Virtual Content Attacks in Augmented Realitycited by 1
  6. 2025Words into World: A Task-Adaptive Agent for Language-Guided Spatial Retrieval in ARcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumble
  2. 2026Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality
  3. 2026presto: Efficient, Training-free, and Open-world Object Placement via Imaginary Search
  4. 2024Imagining the Unseen: Generative Location Modeling for Object Placement
  5. 2025OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding
  6. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation
  7. 2026Semantic Reality: Interactive Context-Aware Visualization of Inter-Object Relationships in Augmented Reality ↗
  8. 2024DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenes