mlsanityData to 14 Oct 2026

Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models

arXiv 2408.06518 ↗First posted 12 Aug 2024cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. In this paper, we identify and characterize a phenomenon never discussed before, which we call semantic leakage, where models leak irrelevant information from the prompt into the generation in unexpected ways. We propose an evaluation setting to detect semantic leakage both by humans and automatically, curate a diverse test suite for diagnosing this behavior, and measure significant semantic leakage in 13 flagship models. We also show that models exhibit semantic leakage in languages besides English and across different settings and generation scenarios. This discovery highlights yet another type of bias in language models that affects their generation patterns and behavior.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 12 Aug 2024.

Builds on

33 of 50 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  11. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  12. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
21 more
  1. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  2. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  3. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  4. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  5. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  6. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  7. 2019Evaluating Gender Bias in Machine Translationcited by 249
  8. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  9. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  10. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 207
  11. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignmentcited by 171
  12. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  13. 2022Thinking Fast and Slow in Large Language Modelscited by 137
  14. 2019It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitutioncited by 103
  15. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 98
  16. 2022Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writerscited by 69
  17. 2022Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Listscited by 55
  18. 2022DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Modelscited by 51
  19. 2020Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Primingcited by 41
  20. 2022DALL-E 2 Fails to Reliably Capture Common Syntactic Processescited by 31
  21. 2020Automatically Identifying Gender Issues in Machine Translation using Perturbationscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 7.

15 papers.

Sort:

By area:

  1. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  2. 2025Do language models accommodate their users? A study of linguistic convergencecited by 7
  3. 2026Covert Influence Between Language Modelscited by 6
  4. 2025Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Modelscited by 4
  5. 2026Are Language Models Sensitive to Morally Irrelevant Distractors?cited by 3
  6. 2025Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Modelscited by 2
  7. 2025DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Modelscited by 2
  8. 2026How Open Must Language Models be to Enable Reliable Scientific Inference?cited by 2
  9. 2025Generative AI and the future of scientometrics: current topics and future questionscited by 1
  10. 2025Am I Blue or Is My Hobby Counting Teardrops? Expression Leakage in Large Language Models as a Symptom of Irrelevancy Disruptioncited by 1
  11. 2025On the Brittleness of LLMs: A Journey around Set Membershipcited by 1
  12. 2026Can You Keep a Secret? Involuntary Information Leakage in Language Model Writingcited by 1
3 more
  1. 2026Reading Too Much into Context: Passive Exposure Can Steer LLM Decisionscited by 0
  2. 2026It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Themcited by 0
  3. 2026Misinformation Without Triggers: From Factual Answers to Downstream Decisionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evil twins are not that evil: Qualitative insights into machine-generated prompts
  2. 2024Undesirable Memorization in Large Language Models: A Survey
  3. 2026Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it
  4. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judge
  5. 2024Benchmarking Benchmark Leakage in Large Language Models
  6. 2026Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity
  7. 2025Existing Large Language Model Unlearning Evaluations Are Inconclusive
  8. 2026ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models