mlsanityData to 7 Oct 2026

Towards More Unified In-context Visual Understanding

arXiv 2312.02520 ↗First posted 5 Dec 2023cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

The rapid advancement of large language models (LLMs) has accelerated the emergence of in-context learning (ICL) as a cutting-edge approach in the natural language processing domain. Recently, ICL has been employed in visual understanding tasks, such as semantic segmentation and image captioning, yielding promising results. However, existing visual ICL framework can not enable producing content across multiple modalities, which limits their potential usage scenarios. To address this issue, we present a new ICL framework for visual understanding with multi-modal output enabled. First, we quantize and embed both text and visual prompt into a unified representational space, structured as interleaved in-context sequences. Then a decoder-only sparse transformer architecture is employed to perform generative modeling on them, facilitating in-context learning. Thanks to this design, the model is capable of handling in-context vision understanding tasks with multimodal output in a unified pipeline.Experimental results demonstrate that our model achieves competitive performance compared with specialized models and previous ICL baselines. Overall, our research takes a further step toward unified multimodal in-context learning.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 51 of its 60 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

51 of 60 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  8. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  11. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  12. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  13. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  14. 2021MetaICL: Learning to Learn In Contextcited by 450
  15. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  16. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  17. 2023Generating Images with Multimodal Language Modelscited by 259
  18. 2022Visual Prompting via Image Inpaintingcited by 248
  19. 2017Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detectioncited by 209
  20. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  21. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  22. 2023SegGPT: Segmenting Everything In Contextcited by 193
  23. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  24. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  25. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  26. 2022A Unified Sequence Interface for Vision Taskscited by 152
  27. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  28. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  29. 2022Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentationcited by 114
  30. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  31. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  32. 2022Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentationcited by 65
  33. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  34. 2022Feature-Proxy Transformer for Few-Shot Segmentationcited by 58
  35. 2022Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Taskscited by 57
  36. 2023Towards In-context Scene Understandingcited by 45
  37. 2013Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networkscited by 44
  38. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 2.

10 papers.

  1. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  2. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  3. 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learningcited by 39
  4. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  5. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  6. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
  7. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 9
  8. 2025Context-Aware Weakly Supervised Image Manipulation Localization with SAM Refinementcited by 2
  9. 2026Learning to Focus and Precise Cropping: A Reinforcement Learning Framework with Information Gaps and Grounding Loss for MLLMscited by 1
  10. 2026Personalized Object Identification and Localization via In-Context Inference with Vision-Language Modelscited by 0