mlsanityData to 7 Oct 2026

Autolearn: Learn by Surprise, Commit by Proof

arXiv 2604.01951 ↗First posted 2 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose Autolearn, a framework that enables language models to learn from documents they read, with no external supervision. Passages that produce anomalously high per-token loss are flagged, verified through a self-generated Q&A chain, and trained on with conviction-proportional β2 adjustment. We introduce the perturbation gap (paraphrase-to-original perplexity ratio) as a metric that distinguishes memorization from understanding. The key mechanism is the training data format: Q&A-format training drives the perturbation gap below the pre-trained baseline (2.098 vs. 2.204, Delta = -0.106, > 10σ), suppressing token-sequence memorization, while standard fine-tuning's best attempt remains within noise (Delta = -0.010, < 1σ). Across four models spanning Qwen3 and Phi-4 families, Autolearn is the only method that enters this regime. Stochastic evaluation reveals passage-specific knowledge acquisition: the probability of generating a correct novel fact rises from 6% to 54% after training (p < 10-4), and Q&A format outperforms standard fine-tuning on genuinely novel facts. The system is self-extinguishing: learned content reduces surprisal below threshold and is skipped on re-encounter.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

25 of 38 references.

References · 25

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
13 more
  1. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  3. 2023Do Large Language Models Know What They Don't Know?cited by 238
  4. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  5. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  6. 2024Distinguishing the Knowable from the Unknowable with Language Modelscited by 45
  7. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  8. 2025Pre-training Limited Memory Language Models with Internal and External Knowledgecited by 11
  9. 2025Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probingcited by 7
  10. 2025ALAS: Autonomous Learning Agent for Self-Updating Language Modelscited by 6
  11. 2025Rote Learning Considered Useful: Generalizing over Memorized Data in LLMscited by 5
  12. 2025Self-Evolving LLMs via Continual Instruction Tuningcited by 4
  13. 2025STABLE: Gated Continual Learning for Large Language Modelscited by 2

Built on by

Not cited yet.