mlsanityData to 14 Oct 2026

Toten: A Knowledge-Based System For Structure-Preserving Representation Of Physical Quantities And Technical Notation In Brazilian Portuguese

arXiv 2606.19626 ↗First posted 17 Jun 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

AI pipelines that reason quantitatively over technical text depend on input where physical quantities, numbers, units, and symbolic expressions arrive intact; when these entities fragment at tokenization, errors propagate downstream. Byte-Pair Encoding, optimized for vocabulary compression, is blind to such entities and fragments them into arbitrary subwords -- a problem aggravated in technical Brazilian Portuguese. We present TOTEN, a knowledge-based system whose input representation preserves each technical entity as a whole, typed unit: vocabulary is not derived statistically but classified declaratively under a formal ontology of engineering entities (OEE). The core is the triple <O, classify, {inst_tau}>: types, principles, and invariants; a classifier mapping raw text into typed regions; and instantiators yielding a self-descriptive representation. Integrity rests on deterministic coupling to three external authorities: Pint (dimensional), Unicode Character Database (typographic), and RSLP (Portuguese morphology). We evaluate four properties verifiable by construction -- atomicity, dimensional equivalence, typographic robustness, numerical reconstruction -- on an internal benchmark (EngQuant, N=800) and four Brazilian Portuguese external corpora (N=1771 eligible cases), and report detection recall. Against eight state-of-the-art baselines, TOTEN reaches unit atomicity in all contrasts and reconstruction of 0.775-0.904 externally vs. 0.627-0.703 for the best (Quantulum3); on EngQuant, 0.780 vs. 0.340. Differences are significant (McNemar, Holm-corrected). Spearman correlation between internal and external rankings confirms concurrent validity of the control benchmark. TOTEN shows statistical parity with Pint in dimensional equivalence. The result is a structurally faithful, auditable, low-cost input layer for intelligent systems on technical knowledge, without generative models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 54 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

22 of 54 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  7. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  8. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  9. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 159
  10. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  11. 2024Tokenization Is More Than Compressioncited by 89
  12. 2023GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformercited by 88
10 more
  1. 2020Inductive logic programming at 30: a new introductioncited by 85
  2. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  3. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  4. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 46
  5. 2018Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numberscited by 42
  6. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 42
  7. 2023Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Examscited by 25
  8. 2023Sabi'a: Portuguese Large Language Modelscited by 19
  9. 2025Tokenization is Sensitive to Language Variationcited by 13
  10. 2023CQE: A Comprehensive Quantity Extractorcited by 4

Built on by

Not cited yet.