mlsanityData to 14 Oct 2026

When Do Agents Help? Embedding, LLM and Agentic Alignment of Classical Texts and Their Translations

arXiv 2609.33691 ↗First posted 27 Sep 2026cs.CLcited by 0

On the map: LLM agents and tool use → Web, search, and research agents also fits: General LLM agent frameworks and benchmarks

Abstract (arXiv)

Classical texts aligned with their translations support machine translation, retrieval and computational research, but evidence comparing alignment workflows is scattered. This study compares seven systems on 452 texts in Pali, Sanskrit, Mishnaic Hebrew and Tibetan, comprising 9,833 human-aligned units: four embedding pipelines, a direct LLM call, an autonomous agent, and the agent revised by an independent auditor. Generative workflows recover 93-94% of reference correspondences, against at most 77% for embeddings. A ceiling analysis shows that sentence boundaries make some references unrepresentable by the embedding pipelines. Reference recovery is similar across generative workflows: the agent's advantage is 0.5 percentage points (95% CI -0.02 to 1.17), and auditing adds no established benefit. Agents nevertheless produce structurally valid output for all 452 texts, against 437 for direct calls. A blinded three-LLM panel assesses every generative mismatch against the source and human reference. Most mismatches are labelled defensible editorial variation; consensus major-error labels cover only 0.06-0.14% of units. The panel labels significantly fewer residual defects for agents than direct calls (0.7% versus 1.4%), suggesting that reference recovery alone understates alignment quality. On ten long Pali discourses taken as published online, agents and audited agents raise recovery from the direct call's 71% to 84% and 92%. Identical reference-located chunks bring all three to 93%. Agents thus improve structural reliability and reduce judged defects on short passages, while their large recovery advantage on long documents disappears after chunking. In this setting, independent auditing offers little measurable additional benefit on prepared passages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 38 of its 63 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

38 of 63 references.

References · 38

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  5. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  6. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  7. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  8. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  9. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  10. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  11. 2024AI Agents That Mattercited by 219
  12. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
26 more
  1. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 139
  2. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 125
  3. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 96
  4. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 88
  5. 2026DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compressioncited by 76
  6. 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
  7. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  8. 2021Itihasa: A large-scale corpus for Sanskrit to English translationcited by 13
  9. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual Worldcited by 12
  10. 2024Investigating Length Issues in Document-level Machine Translationcited by 10
  11. 2023SentAlign: Accurate and Scalable Sentence Alignmentcited by 9
  12. 2023Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillationcited by 6
  13. 2025Extending Automatic Machine Translation Evaluation to Book-Length Documentscited by 6
  14. 2026MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for P=ali, Sanskrit, Buddhist Chinese, and Tibetancited by 5
  15. 2025Align-then-Slide: A complete evaluation framework for Ultra-Long Document-Level Machine Translationcited by 2
  16. 2025Building from Scratch: A Multi-Agent Framework with Human-in-the-Loop for Multilingual Legal Terminology Mappingcited by 2
  17. 2026PaliBench: A Multi-Reference Blueprint for Classical Language Translation Benchmarkscited by 2
  18. 2024Automatic Translation Alignment Pipeline for Multilingual Digital Editions of Literary Workscited by 1
  19. 2025AlignAR: Generative Sentence Alignment for Arabic-English Parallel Corpora of Legal and Literary Textscited by 1
  20. 2026LATA: A Tool for LLM-Assisted Translation Annotationcited by 1
  21. 2026From Outliers to Errors: Auditing Pali-to-English LLM Translations with Multi-Reference Adjudicationcited by 1
  22. 2026Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languagescited by 1
  23. 2026OmniAlign: A Unified Multilingual Aligner for Word and Sentence Alignmentcited by 1
  24. 2026Scaling Unsupervised Word Alignment to Documents via Structural Constraintscited by 1
  25. 2026STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translationcited by 1
  26. 2026Can We Triage LLM Translation Errors in Classical Texts Without Human References? Source Novelty, GEMBA Scoring, and Budgeted Review through Pali-to-English Translationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Methodological reflections for AI alignment research using human feedback
  2. 2023Large Language Model Alignment: A Survey
  3. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits
  4. 2025Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments
  5. 2026Latent Communication Between Language Model Agents: Channels, Alignment, and the Limits of Text
  6. 2026Agentic AI Translate: An Agentic Translator Prototype for Translation as Communication Design
  7. 2024Your Weak LLM is Secretly a Strong Teacher for Alignment
  8. 2025Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment