mlsanityData to 7 Oct 2026

Lumos : Empowering Multimodal LLMs with Scene Text Recognition

arXiv 2402.08017 ↗First posted 12 Feb 2024cs.CV cs.CL cs.LGcited by 8

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 42 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

25 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding19 Aug

Other references · 24

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  4. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2018Path Aggregation Network for Instance Segmentationcited by 796
  7. 2021A White Paper on Neural Network Quantizationcited by 453
  8. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  9. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  10. 2020VarifocalNet: An IoU-aware Dense Object Detectorcited by 230
  11. 2017Arbitrary-Oriented Scene Text Detection via Rotation Proposalscited by 176
  12. 2019Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapescited by 171
12 more
  1. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 147
  2. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  3. 2014Reading Text in the Wild with Convolutional Neural Networkscited by 122
  4. 2016Robust Scene Text Recognition with Automatic Rectificationcited by 121
  5. 2022Towards End-to-End Unified Scene Text Detection and Layout Analysiscited by 96
  6. 2019Rosetta: Large scale system for text detection and recognition in imagescited by 69
  7. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  8. 2021Adapting Long Context NLM for ASR Rescoring in Conversational Agentscited by 12
  9. 2021Remember the context! ASR slot error correction through memorizationcited by 9
  10. 2021ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modelingcited by 7
  11. 2021STRIDE : Scene Text Recognition In-Devicecited by 4
  12. 2023DISGO: Automatic End-to-End Evaluation for Scene Text OCRcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 3.

8 papers.

  1. 2024An Introduction to Vision-Language Modelingcited by 175
  2. 2025CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmarkcited by 9
  3. 2025Reading in the Dark with Foveated Event Visioncited by 2
  4. 2026VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integrationcited by 2
  5. 2025Beyond CNNs: Efficient Fine-Tuning of Multi-Modal LLMs for Object Detection on Low-Data Regimescited by 1
  6. 2024EgoQR: Efficient QR Code Reading in Egocentric Settingscited by 0
  7. 2026GLIMPSE : Real-Time Text Recognition and Contextual Understanding for VQA in Wearablescited by 0
  8. 2026Embedding Rotation Invariance for Provable Multi-Oriented Scene Text Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
  2. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  3. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding
  4. 2024Efficient Multimodal Large Language Models: A Survey
  5. 2026All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts
  6. 2024Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering
  7. 2023Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models
  8. 2024MMR: Evaluating Reading Ability of Large Multimodal Models