mlsanityData to 14 Oct 2026

MISAR: A Multimodal Instructional System with Augmented Reality

arXiv 2310.11699 ↗First posted 18 Oct 2023cs.CL cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Augmented reality (AR) requires the seamless integration of visual, auditory, and linguistic channels for optimized human-computer interaction. While auditory and visual inputs facilitate real-time and contextual user guidance, the potential of large language models (LLMs) in this landscape remains largely untapped. Our study introduces an innovative method harnessing LLMs to assimilate information from visual, auditory, and contextual modalities. Focusing on the unique challenge of task performance quantification in AR, we utilize egocentric video, speech, and context analysis. The integration of LLMs facilitates enhanced state estimation, marking a step towards more adaptive AR systems. Code, dataset, and demo will be available at https://github.com/nguyennm1024/misar.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

12 of 17 references.

References · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  4. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  7. 2022Learning Video Representations from Large Language Modelscited by 205
  8. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  9. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  10. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 73
  11. 2013Learning to answer questionscited by 2
  12. 2023Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQAcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 4; 2026: 1.

10 papers.

  1. 2023Video Understanding with Large Language Models: A Surveycited by 206
  2. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 35
  3. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  4. 2024Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understandingcited by 18
  5. 2024EAGLE: Egocentric AGgregated Language-video Enginecited by 16
  6. 2025Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Promptingcited by 10
  7. 2025Llamarine: Open-source Maritime Industry-specific Large Language Modelcited by 4
  8. 2025"I Can See Forever!": Evaluating Real-time VideoLLMs for Assisting Individuals with Visual Impairmentscited by 3
  9. 2025From Voices to Worlds: Developing an AI-Powered Framework for 3D Object Generation in Augmented Realitycited by 2
  10. 2026How Much Does It Cost to Answer My Question? Benchmarking Cloud VLM-based VQA Systemscited by 0