mlsanityData to 14 Oct 2026

Trusting sovereign language models as scientific instruments: evidence from Portugal's AMALIA

arXiv 2607.08731 ↗First posted 9 Jul 2026cs.CL cs.AI cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

National language models are becoming publicly funded epistemic infrastructure. Public ownership, linguistic specialization, and open weights create a presumption of trustworthiness. Such an instrument, built by and for a language community, looks like the natural choice for measuring what that community says and values. Whether such a model validly measures anything is untested at release. The evaluation of LLMs as measurement instruments is typically task-specific and stops at agreement with human coders. Agreement cannot distinguish an LLM instrument that measures a construct from one that reaches matching codes through surface correlates. We audit the presumption on a favourable case: AMALIA, Portugal's publicly funded 9B model, coding the moral foundation of authority in European Portuguese. The textitrecovery gap operationalizes the audit: decompose the codebook into its theory-defined clauses, recombine them through the theory's explicit rule, and measure how much of the original prompt's performance the stated theory reproduces. In a pre-registered, out-of-sample study on a transcreated (English to European Portuguese) corpus, AMALIA agrees with trained coders within six points of open models eight to thirteen times its size. Yet, the recovery gap shows that only about half of coding performance on authority can be attributed to the theory. A larger multilingual LLM closes the recovery gap on the same corpus, suggesting the shortfall lies in the annotator model, not the corpus or its translation. Sovereignty earns operational and performance trust; epistemic trust requires calibration -- and the audit method is inexpensive, and portable across models, languages and tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 21 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

8 of 21 references.

References · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  3. 2024NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Modelscited by 48
  4. 2022The Moral Foundations Reddit Corpuscited by 21
  5. 2025EuroLLM-9B: Technical Reportcited by 17
  6. 2024On Limitations of LLM as Annotator for Low Resource Languagescited by 12
  7. 2026Typhoon-S: Minimal Open Post-Training for Sovereign Large Language Modelscited by 2
  8. 2026Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructscited by 2

Built on by

Not cited yet.