# Writing Style Matters: An Examination of Bias and Fairness in Information Retrieval Systems

arXiv: https://arxiv.org/abs/2411.13173 · HTML: https://mlsanity.com/paper/2411.13173/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 20 Nov 2024.

## Builds on (most cited first, 19 of 19)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering](https://mlsanity.com/paper/2307.16877/) (2023)
- [TopiOCQA: Open-domain Conversational Question Answering with Topic Switching](https://mlsanity.com/paper/2110.00768/) (2021)
- [LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores](https://mlsanity.com/paper/2311.09766/) (2023)
- [Information Retrieval: Recent Advances and Beyond](https://mlsanity.com/paper/2301.08801/) (2023)
- [Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark](https://mlsanity.com/paper/2406.01607/) (2024)
- [Inclusive normalization of face images to passport format](https://mlsanity.com/paper/2312.14544/) (2023)

## Built on by (most cited first, 15 of 15)

- [Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation](https://mlsanity.com/paper/2603.03116/) (2026)
- [Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks](https://mlsanity.com/paper/2506.21182/) (2025)
- [Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications](https://mlsanity.com/paper/2504.02867/) (2025)
- [Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs](https://mlsanity.com/paper/2505.15524/) (2025)
- [Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias](https://mlsanity.com/paper/2601.16934/) (2026)
- [When LLMs Imagine People: A Human-Centered Persona Brainstorm Audit for Bias and Fairness in Creative Applications](https://mlsanity.com/paper/2602.00044/) (2026)
- [Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality](https://mlsanity.com/paper/2504.00584/) (2025)
- [Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysis](https://mlsanity.com/paper/2506.12189/) (2025)
- [The Other Side of the Coin: Exploring Fairness in Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.12323/) (2025)
- [MetaSynth: Multi-Agent Metadata Generation from Implicit Feedback in Black-Box Systems](https://mlsanity.com/paper/2510.01523/) (2025)
- [Training-Induced Bias Toward LLM-Generated Content in Dense Retrieval](https://mlsanity.com/paper/2602.10833/) (2026)
- [Trust Stack for Mental Health AI: A Survey of Calibration across Human, Interaction, and AI Layers](https://mlsanity.com/paper/2604.20166/) (2026)
- [Building Persona-Based Agents On Demand: Tailoring Multi-Agent Workflows to User Needs](https://mlsanity.com/paper/2604.27882/) (2026)
- [Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors](https://mlsanity.com/paper/2606.02814/) (2026)
- [The Significance of Style Diversity in Annotation-Free Synthetic Data Generation](https://mlsanity.com/paper/2606.20400/) (2026)

## Abstract (arXiv)

The rapid advancement of Language Model technologies has opened new opportunities, but also introduced new challenges related to bias and fairness. This paper explores the uncharted territory of potential biases in state-of-the-art universal text embedding models towards specific document and query writing styles within Information Retrieval (IR) systems. Our investigation reveals that different embedding models exhibit different preferences of document writing style, while more informal and emotive styles are less favored by most embedding models. In terms of query writing styles, many embedding models tend to match the style of the query with the style of the retrieved documents, but some show a consistent preference for specific styles. Text embedding models fine-tuned on synthetic data generated by LLMs display a consistent preference for certain style of generated data. These biases in text embedding based IR systems can inadvertently silence or marginalize certain communication styles, thereby posing a significant threat to fairness in information retrieval. Finally, we also compare the answer styles of Retrieval Augmented Generation (RAG) systems based on different LLMs and find out that most text embedding models are biased towards LLM's answer styles when used as evaluation metrics for answer correctness. This study sheds light on the critical issue of writing style based bias in IR systems, offering valuable insights for the development of more fair and robust models.