mlsanityData to 7 Oct 2026

Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models

arXiv 2604.08970 ↗First posted 10 Apr 2026cs.CL cs.AI cs.HCcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We study predictive multilingual evaluation: estimating how well a model will perform on a task in a target language when direct benchmark results are missing. This problem is common in multilingual deployment, where evaluation coverage is sparse and published evidence is uneven across languages, tasks, and model families. We introduce a controlled benchmark of 1,500 questions spanning six tasks and five evidence scenarios. The benchmark separates accessible evidence from ground truth, enabling evaluation of systems that must infer missing results from incomplete literature evidence. We also present Litmus (Re)Agent, a DAG-orchestrated agentic system that decomposes queries into hypotheses, retrieves evidence, and synthesises predictions through feature-aware aggregation. Across six systems, Litmus (Re)Agent achieves the best overall performance, with the largest gains in transfer-heavy scenarios where direct evidence is weak or absent. These results show that structured agentic reasoning is a promising approach to multilingual performance estimation under incomplete evidence.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 18 of its 36 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

18 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks7 Nov

Other references · 17

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  7. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  8. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  9. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  10. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 182
  11. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  12. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
5 more
  1. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
  2. 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languagescited by 38
  3. 2021Analysing The Impact Of Linguistic Features On Cross-Lingual Transfercited by 16
  4. 2022Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languagescited by 10
  5. 2023DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfercited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs
  2. 2026GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation
  3. 2025Ticket-Bench: A Kickoff for Multilingual and Regionalized Agent Evaluation
  4. 2026BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents
  5. 2026OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents
  6. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
  7. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  8. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluation