mlsanityData to 14 Oct 2026

Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs

arXiv 2505.16134 ↗First posted 22 May 2025cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) exhibit position bias systematically underweighting information based on its location in the context but how this bias varies across languages and models remains unclear. We conduct a multilingual study across five typologically diverse languages (English, Russian, German, Hindi, Vietnamese) and five model architectures, analyzing how position bias interacts with prompting strategies and affects output entropy. Our key findings are: (1) Position bias is primarily model-driven but shows language-specific nuances. Notably, Qwen2.5-7B-Instruct, DeepSeek 7B Chat and Mistral 7B consistently favor late positions challenging the common assumption of universal early-token preference. (2) Explicitly instructing the model, in the presence of irrelevant distractors, that "the most relevant context to the query is marked as 1" unexpectedly reduces accuracy across all languages, questioning standard prompt-engineering practices. (3) Accuracy consistently drops most when relevant information appears in the middle of the context, yet this is not reflected in a corresponding increase in output entropy, suggesting the model remains confident even when it fails to use mid-context cues.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

33 of 51 references.

References · 33

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  8. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  9. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  11. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  12. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
21 more
  1. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 150
  2. 2025Why do LLMs attend to the first token?cited by 127
  3. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  4. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
  5. 2025Do Multilingual LLMs Think In English?cited by 78
  6. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  7. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  8. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  9. 2025On the Emergence of Position Bias in Transformerscited by 59
  10. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 55
  11. 2024In Defense of RAG in the Era of Long-Context Language Modelscited by 53
  12. 2024Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?cited by 41
  13. 2019SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysiscited by 38
  14. 2024Serial Position Effects of Large Language Modelscited by 30
  15. 2024REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answeringcited by 28
  16. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 19
  17. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  18. 2024Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyondcited by 18
  19. 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QAcited by 13
  20. 2024A Morphology-Based Investigation of Positional Encodingscited by 4
  21. 2025When an LLM is apprehensive about its answers -- and when its uncertainty is justifiedcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026Mitigating Conversational Inertia in Multi-Turn Agentscited by 7
  2. 2026A Structural Theory of Position Bias in Transformerscited by 5
  3. 2026Layer-wise Positional Bias in Short-Context Language Modelingcited by 1
  4. 2025Token Homogenization under Positional Biascited by 0
  5. 2026Dynamic Multi-Byte Prediction With Hierarchical Language Modelscited by 0