mlsanityData to 14 Oct 2026

TalkMosaic: Interactive PhotoMosaic with Multi-modal LLM Q&A Interactions

arXiv 2409.13941 ↗First posted 20 Sep 2024cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

We use images of cars of a wide range of varieties to compose an image of an animal such as a bird or a lion for the theme of environmental protection to maximize the information about cars in a single composed image and to raise the awareness about environmental challenges. We present a novel way of image interaction with an artistically-composed photomosaic image, in which a simple operation of "click and display" is used to demonstrate the interactive switch between a tile image in a photomosaic image and the corresponding original car image, which will be automatically saved on the Desktop. We build a multimodal custom GPT named TalkMosaic by incorporating car images information and the related knowledge to ChatGPT. By uploading the original car image to TalkMosaic, we can ask questions about the given car image and get the corresponding answers efficiently and effectively such as where to buy the tire in the car image that satisfies high environmental standards. We give an in-depth analysis on how to speed up the inference of multimodal LLM using sparse attention and quantization techniques with presented probabilistic FlashAttention (PrFlashAttention) and Staircase Adaptive Quantization (SAQ) methods. The implemented prototype demonstrates the feasibility and effectiveness of the presented approach.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 22 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

13 of 22 references.

References · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023A Survey on Multimodal Large Language Modelscited by 909
  6. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  8. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  9. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  10. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  11. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  12. 2023Faster Causal Attention Over Large Sequences Through Sparse Flash Attentioncited by 25
1 more
  1. 2018Composing photomosaic images using clustering based evolutionary programmingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
  2. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
  3. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  4. 2026Less Is More: Graph-free Multimodal RAG via Multi-signal Late Fusion
  5. 2025Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks
  6. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark
  7. 2025QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models
  8. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs