mlsanityData to 14 Oct 2026

StoicLLM: Preference Optimization for Philosophical Alignment in Small Language Models

arXiv 2605.11483 ↗First posted 12 May 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

While large language models excel at factual adaptation, their ability to internalize nuanced philosophical frameworks under severe data constraints remains underexplored. We investigate this by specializing small LLMs on micro-datasets of foundational Stoic texts using preference optimization (ORPO, AlphaPO). Evaluated via a multi-model critic bank, our results show that just 300 high-fidelity examples can induce strong alignment with inward-facing Stoic virtues, closely approaching few-shot prompting while freeing the context window. Critically, however, all models, including few-shot baselines, exhibit a persistent failure on Stoicism's outward-facing cosmopolitan duties, pointing to a representational limitation of small models that micro-dataset adaptation alone cannot overcome.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 7 of its 15 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

7 of 15 references.

References · 7

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024A Survey on LLM-as-a-Judgecited by 1,423
  3. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  4. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 120
  5. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
  6. 2024LAB: Large-Scale Alignment for ChatBotscited by 39
  7. 2026Learning to Judge: LLMs Designing and Applying Evaluation Rubricscited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts
  2. 2026Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models
  3. 2026BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences
  4. 2026The Collapse of Heterogeneity in Silicon Philosophers
  5. 2026Mechanistic Origin of Moral Indifference in Language Models
  6. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
  7. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  8. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment