# Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs

arXiv: https://arxiv.org/abs/2505.16134 · HTML: https://mlsanity.com/paper/2505.16134/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 30 of 33)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://mlsanity.com/paper/2307.01379/) (2023)
- [Why do LLMs attend to the first token?](https://mlsanity.com/paper/2504.02732/) (2025)
- [Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG](https://mlsanity.com/paper/2410.05983/) (2024)
- [Attention Sorting Combats Recency Bias In Long Context Language Models](https://mlsanity.com/paper/2310.01427/) (2023)
- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding](https://mlsanity.com/paper/2403.04797/) (2024)
- [Eliminating Position Bias of Language Models: A Mechanistic Approach](https://mlsanity.com/paper/2407.01100/) (2024)
- [Why do language models perform worse for morphologically complex languages?](https://mlsanity.com/paper/2411.14198/) (2024)
- [On the Emergence of Position Bias in Transformers](https://mlsanity.com/paper/2502.01951/) (2025)
- [A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice](https://mlsanity.com/paper/2410.15326/) (2024)
- [In Defense of RAG in the Era of Long-Context Language Models](https://mlsanity.com/paper/2409.01666/) (2024)
- [Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?](https://mlsanity.com/paper/2408.10811/) (2024)
- [SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysis](https://mlsanity.com/paper/1912.09723/) (2019)
- [Serial Position Effects of Large Language Models](https://mlsanity.com/paper/2406.15981/) (2024)
- [REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering](https://mlsanity.com/paper/2402.17497/) (2024)
- [Evaluating Morphological Compositional Generalization in Large Language Models](https://mlsanity.com/paper/2410.12656/) (2024)
- [Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs](https://mlsanity.com/paper/2410.15956/) (2024)
- [Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyond](https://mlsanity.com/paper/2402.14259/) (2024)

## Built on by (most cited first, 5 of 5)

- [Mitigating Conversational Inertia in Multi-Turn Agents](https://mlsanity.com/paper/2602.03664/) (2026)
- [A Structural Theory of Position Bias in Transformers](https://mlsanity.com/paper/2602.16837/) (2026)
- [Layer-wise Positional Bias in Short-Context Language Modeling](https://mlsanity.com/paper/2601.04098/) (2026)
- [Token Homogenization under Positional Bias](https://mlsanity.com/paper/2508.17126/) (2025)
- [Dynamic Multi-Byte Prediction With Hierarchical Language Models](https://mlsanity.com/paper/2608.15454/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) exhibit position bias systematically underweighting information based on its location in the context but how this bias varies across languages and models remains unclear. We conduct a multilingual study across five typologically diverse languages (English, Russian, German, Hindi, Vietnamese) and five model architectures, analyzing how position bias interacts with prompting strategies and affects output entropy. Our key findings are: (1) Position bias is primarily model-driven but shows language-specific nuances. Notably, Qwen2.5-7B-Instruct, DeepSeek 7B Chat and Mistral 7B consistently favor late positions challenging the common assumption of universal early-token preference. (2) Explicitly instructing the model, in the presence of irrelevant distractors, that "the most relevant context to the query is marked as 1" unexpectedly reduces accuracy across all languages, questioning standard prompt-engineering practices. (3) Accuracy consistently drops most when relevant information appears in the middle of the context, yet this is not reflected in a corresponding increase in output entropy, suggesting the model remains confident even when it fails to use mid-context cues.