mlsanityData to 7 Oct 2026

Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations

arXiv 2310.01651 ↗First posted 2 Oct 2023cs.LGcited by 24

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large language and vision-language models are rapidly being deployed in practice thanks to their impressive capabilities in instruction following, in-context learning, and so on. This raises an urgent need to carefully analyse their robustness so that stakeholders can understand if and when such models are trustworthy enough to be relied upon in any given application. In this paper, we highlight a specific vulnerability in popular models, namely permutation sensitivity in multiple-choice question answering (MCQA). Specifically, we show empirically that popular models are vulnerable to adversarial permutation in answer sets for multiple-choice prompting, which is surprising as models should ideally be as invariant to prompt permutation as humans are. These vulnerabilities persist across various model sizes, and exist in very recent language and vision-language models. Code is available at https://github.com/ys-zong/FoolyourVLLMs.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 52 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

39 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
25 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2023A Survey on Multimodal Large Language Modelscited by 898
  13. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  14. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  15. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  16. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  17. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  18. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  19. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  20. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  21. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  22. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  23. 2022Linearly Mapping from Image to Text Spacecited by 139
  24. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  25. 2023Self-Supervised Multimodal Learning: A Surveycited by 58

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 6; 2026: 4.

24 papers.

Sort:

By area:

  1. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  2. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  3. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  4. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  5. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  6. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 50
  7. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  8. 2024Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQAcited by 19
  9. 2024Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Modelscited by 15
  10. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  11. 2024Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Modelscited by 11
  12. 2025Set-LLM: A Permutation-Invariant LLMcited by 10
12 more
  1. 2025MONICA: Real-Time Monitoring and Calibration of Chain-of-Thought Sycophancy in Large Reasoning Modelscited by 10
  2. 2025Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasetscited by 5
  3. 2025Protecting Your Video Content: Disrupting Automated Video-based LLM Annotationscited by 5
  4. 2025Diagnosing Corruption-Induced Reliability Failures in Vision-Language Modelscited by 3
  5. 2024Too Big to Fool: Resisting Deception in Language Modelscited by 2
  6. 2025SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Modelscited by 2
  7. 2023What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Modelscited by 1
  8. 2024Variational Bayesian Bow tie Neural Networks with Shrinkagecited by 0
  9. 2026Models Know Models Best: Evaluation via Model-Preferred Formatscited by 0
  10. 2026The Power of Order: Fooling LLMs with Adversarial Table Permutationscited by 0
  11. 2026Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problemscited by 0
  12. 2026Candidate-Expanding Routing with Permutation-Stabilized Experts for Mixed-Format Medical VQAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think
  2. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models
  3. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?
  4. 2024Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?
  5. 2023Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets
  6. 2026When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraints
  7. 2023Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis
  8. 2022Understanding Prior Bias and Choice Paralysis in Transformer-based Language Representation Models through Four Experimental Probes