UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy
Not on the map. Why?
Abstract (arXiv)
In-context learning (ICL) enables fast task adaptation from demonstrations without per-task parameter updates but remains highly sensitive to example selection and formatting. In unified multimodal models spanning understanding and generation, this sensitivity is exacerbated by cross-modal interference and varying cognitive demands. Consequently, in-context learning efficacy is often non-monotonic and highly task-dependent. To diagnose these behaviors, we introduce a six-level Capability-Oriented Taxonomy that categorizes the functional role of demonstrations from basic perception to high-order discernment. Guided by this cognitive framework, we construct UniICL-760K, a large-scale corpus featuring curated 8-shot in-context learning episodes across 15 subtasks, alongside UniICL-Bench for rigorous, controlled evaluation. We show that this data-driven assembly is the primary source of our gains. As a complementary, lightweight stabilizer, we additionally propose the Context-Adaptive Prototype Modulator, a plug-and-play module that further improves few-shot stability. Evaluations on UniICL-Bench show that our approach yields highly competitive unified results, outperforming larger-parameter multimodal large language model baselines on most understanding in-context learning tasks. Data and code are available at https://github.com/xuyicheng-zju/UniICL.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 55 of its 66 references that are in the map. First posted on arXiv 25 Mar 2026.
Builds on
55 of 66 references.
References · 55
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2025DINOv3cited by 1,414
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
- 2022In-context Learning and Induction Headscited by 964
- 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
- 2022Transformers learn in-context by gradient descentcited by 732
- 2023VILA: On Pre-training for Visual Language Modelscited by 692
- 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
- 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
- 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
- 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
- 2023Generative Multimodal Models are In-Context Learnerscited by 446
- 2024What matters when building vision-language models?cited by 326
- 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
- 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
- 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
- 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
- 2023Generating Images with Multimodal Language Modelscited by 260
- 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 211
- 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
- 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 153
- 2023Planting a SEED of Vision in Large Language Modelcited by 140
- 2025Detect Anything via Next Point Predictioncited by 73
- 2025Ovis-U1 Technical Reportcited by 67
- 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 66
- 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
- 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learningcited by 39
- 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 36
- 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 34
50 most cited of 55 All 55 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
- 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
- 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering
- 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
- 2025What do vision-language models see in the context? Investigating multimodal in-context learning
- 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks
- 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations
- 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning