mlsanityData to 14 Oct 2026

On the missing benchmarks layer and a potential solution

arXiv 2608.02996 ↗First posted 4 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Latin America is missing a foundational layer for native AI development: the benchmark layer. The benchmark layer does two things no other layer can - it audits AI systems against regional social requirements and it directs AI optimization in economically relevant environments. Without it, public institutions cannot independently evaluate foreign AI systems, and companies cannot optimize AI systems to solve local problems with SOTA performance. The cost of the missing layer is dual: a loss of auditability and a loss of optimization direction over a technology that is increasingly critical infrastructure. We propose an EvalsHub, with LatamBoard as its first regional instance - an open, task-first benchmark infrastructure where universities, public institutions, professional communities, and companies can publish, execute, compare, and maintain evaluations across models, workflows, and agents. Built once, measured forever - re-run by institutions as new AI systems ship and by industry teams after every system change. Open by design and incentive-driven by construction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 27 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

13 of 27 references.

References · 13

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  4. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  5. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
  6. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  7. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  8. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  9. 2023Auditing large language models: a three-layered approachcited by 137
  10. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  11. 2024Large Language Models are Geographically Biasedcited by 93
  12. 2022Evaluation Gaps in Machine Learning Practicecited by 49
1 more
  1. 2024Epistemic Injustice in Generative AIcited by 16

Built on by

Not cited yet.