mlsanityData to 14 Oct 2026

Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering

arXiv 2508.18093 ↗First posted 25 Aug 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a case study evaluating large language models (LLMs) with 128K-token context windows on a technical question answering (QA) task. Our benchmark is built on a user manual for an agricultural machine, available in English, French, and German. It simulates a cross-lingual information retrieval scenario where questions are posed in English against all three language versions of the manual. The evaluation focuses on realistic "needle-in-a-haystack" challenges and includes unanswerable questions to test for hallucinations. We compare nine long-context LLMs using direct prompting against three Retrieval-Augmented Generation (RAG) strategies (keyword, semantic, hybrid), with an LLM-as-a-judge for evaluation. Our findings for this specific manual show that Hybrid RAG consistently outperforms direct long-context prompting. Models like Gemini 2.5 Flash and the smaller Qwen 2.5 7B achieve high accuracy (over 85%) across all languages with RAG. This paper contributes a detailed analysis of LLM performance in a specialized industrial domain and an open framework for similar evaluations, highlighting practical trade-offs and challenges.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 17 references that are in the map. First posted on arXiv 25 Aug 2025.

Builds on

8 of 17 references.

References · 8

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  3. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  4. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  5. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 119
  6. 2024In Defense of RAG in the Era of Long-Context Language Modelscited by 53
  7. 2024Docling Technical Reportcited by 34
  8. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generation
  2. 2026Analyzing performance of Retrieval Augmented Generation models for Question Answering Tasks
  3. 2024Long Context vs. RAG for LLMs: An Evaluation and Revisits
  4. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering
  5. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems
  6. 2023ToolQA: A Dataset for LLM Question Answering with External Tools
  7. 2025Multilingual Retrieval-Augmented Generation for Knowledge-Intensive Task
  8. 2024QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling