mlsanityData to 14 Oct 2026

Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models

arXiv 2312.10104 ↗First posted 15 Dec 2023cs.CV cs.CL cs.LGcited by 7

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

As Archimedes famously said, ``Give me a lever long enough and a fulcrum on which to place it, and I shall move the world'', in this study, we propose to use a tiny Language Model (LM), eg, a Transformer with 67M parameters, to lever much larger Vision-Language Models (LVLMs) with 9B parameters. Specifically, we use this tiny Lever-LM to configure effective in-context demonstration (ICD) sequences to improve the In-Context Learinng (ICL) performance of LVLMs. Previous studies show that diverse ICD configurations like the selection and ordering of the demonstrations heavily affect the ICL performance, highlighting the significance of configuring effective ICD sequences. Motivated by this and by re-considering the the process of configuring ICD sequence, we find this is a mirror process of human sentence composition and further assume that effective ICD configurations may contain internal statistical patterns that can be captured by Lever-LM. Then a dataset with effective ICD sequences is constructed to train Lever-LM. After training, given novel queries, new ICD sequences are configured by the trained Lever-LM to solve vision-language tasks through ICL. Experiments show that these ICD sequences can improve the ICL performance of two LVLMs compared with some strong baselines in Visual Question Answering and Image Captioning, validating that Lever-LM can really capture the statistical patterns for levering LVLMs. The code is available at urlhttps://github.com/ForJadeForest/Lever-LM.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 66 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

50 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023A Survey on Multimodal Large Language Models23 Jun
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
36 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  7. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  12. 2023Larger language models do in-context learning differentlycited by 394
  13. 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 294
  14. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  15. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  16. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  17. 2022Active Example Selection for In-Context Learningcited by 196
  18. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  19. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  20. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
  21. 2022Forward Compatible Few-Shot Class-Incremental Learningcited by 127
  22. 2014Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognitioncited by 111
  23. 2023Finding Support Examples for In-Context Learningcited by 104
  24. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  25. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  26. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  27. 2023In-context Example Selection with Influencescited by 61
  28. 2023Learning to Retrieve In-Context Examples for Large Language Modelscited by 60
  29. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 57
  30. 2021Reordering Examples Helps during Priming-based Few-Shot Learningcited by 50
  31. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 33
  32. 2022Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibrationcited by 24
  33. 2023Learngene: Inheriting Condensed Knowledge from the Ancestry Model to Descendant Modelscited by 24
  34. 2023In-Context Demonstration Selection with Cross Entropy Differencecited by 17
  35. 2023Improving Probability-based Prompt Selection Through Unified Evaluation and Analysiscited by 13
  36. 2022Zero-Label Prompt Selectioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimizationcited by 10
  2. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 7
  3. 2025EcoAlign: An Economically Rational Framework for Efficient LVLM Alignmentcited by 5
  4. 2026Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agentscited by 2
  5. 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Taskscited by 1
  6. 2026Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Modelscited by 0
  7. 2026ActICL: Post-Training for Agentic Multimodal In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Online In-Context Distillation for Low-Resource Vision Language Models
  2. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations
  3. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  4. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  5. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
  6. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  7. 2024In-Context Language Learning: Architectures and Algorithms
  8. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning