# Beyond Tokens: Concept-Level Training Objectives for LLMs

arXiv: https://arxiv.org/abs/2601.11791 · HTML: https://mlsanity.com/paper/2601.11791/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 16 Jan 2026.

## Builds on (most cited first, 9 of 9)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations](https://mlsanity.com/paper/1805.06201/) (2018)
- [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019)
- [From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning](https://mlsanity.com/paper/2505.17117/) (2025)

## Built on by (most cited first, 3 of 3)

- [Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models](https://mlsanity.com/paper/2603.29123/) (2026)
- [Mimir: Large-scale Multilingual Concept Modeling](https://mlsanity.com/paper/2605.25263/) (2026)
- [From Found to Designed: Concepts as a Design Axis for Large Language Models](https://mlsanity.com/paper/2607.26825/) (2026)

## Abstract (arXiv)

The next-token prediction (NTP) objective has been foundational in the development of modern large language models (LLMs), driving advances in fluency and generalization. However, NTP operates at the textittoken level, treating deviations from a single reference continuation as errors even when alternative continuations are equally plausible or semantically equivalent (e.g., ``mom'' vs. ``mother''). As a result, token-level loss can penalize valid abstractions, paraphrases, or conceptually correct reasoning paths, biasing models toward surface form rather than underlying meaning. This mismatch between the training signal and semantic correctness motivates learning objectives that operate over higher-level representations. We propose a shift from token-level to concept-level prediction, where concepts group multiple surface forms of the same idea (e.g., ``mom,'' ``mommy,'' ``mother'' rightarrow textitMOTHER). We introduce various methods for integrating conceptual supervision into LLM training and show that concept-aware models achieve lower perplexity, improved robustness under domain shift, and stronger performance than NTP-based models on diverse NLP benchmarks. This suggests textitconcept-level supervision as an improved training signal that better aligns LLMs with human semantic abstractions.