mlsanityData to 14 Oct 2026

TD-DPO: Difference-Aware Preference Optimization for Mitigating Sycophancy in Clinical Autism Intervention Dialogue

arXiv 2607.18304 ↗First posted 17 Jul 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Medical LLM alignment

Abstract (arXiv)

The sycophancy of large language models can increase the safety risk in intervention dialogue for autistic children. Supervised fine-tuning can somewhat reduce sycophancy, but relying solely on positive examples is often insufficient to identify and correct failure patterns. We observe that sycophancy behaviors can often be localized to a limited span within the model response. In this regime, sequence-level preference optimization can over-update preference-irrelevant tokens and degrade intervention ability. To address this, we propose the Minimal Edit Data Augmentation (MEDA) strategy to construct controlled, stable, minimal edit preference pairs and Token-level Difference Direct Preference Optimization (TD-DPO), which upweights difference tokens between chosen and rejected responses while downweighting shared tokens to suppress background drift. Extensive experiments across multiple backbones and evaluators show that TD-DPO achieves a better trade-off between sycophancy mitigation and intervention ability retention in our offline settings, highlighting its potential as a practical alignment approach for autism intervention.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 55 references that are in the map. First posted on arXiv 17 Jul 2026.

Builds on

30 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialogue7 Aug

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  6. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  10. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  11. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  12. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
17 more
  1. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  2. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
  3. 2024CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counselingcited by 89
  4. 2023When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviourcited by 62
  5. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  6. 2024CBT-LLM: A Chinese Large Language Model for Cognitive Behavioral Therapy-based Mental Health Question Answeringcited by 45
  7. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  8. 2025Be Friendly, Not Friends: How LLM Sycophancy Shapes User Trustcited by 34
  9. 2025TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Modelscited by 33
  10. 2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
  11. 2025Sycophancy under Pressure: Evaluating and Mitigating Sycophantic Bias via Adversarial Dialogues in Scientific QAcited by 18
  12. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
  13. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimizationcited by 10
  14. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimizationcited by 7
  15. 2024ASD-Chat: An Innovative Dialogue Intervention System for Children with Autism based on LLM and VB-MAPPcited by 3
  16. 2025Echo-Teddy: Preliminary Design and Development of Large Language Model-based Social Robot for Autistic Studentscited by 2
  17. 2026From Synthesis to Clinical Assistance: A Strategy-Aware Agent Framework for Autism Intervention based on Real Clinical Datasetcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Proactive Multi-Agent Dialogue Framework for Assessing Social Language Disorder Traits in Autism
  2. 2026LLMdoctor: Token-Level Flow-Guided Preference Optimization for Efficient Test-Time Alignment of Large Language Models
  3. 2025Conversation Forests: The Key to Fine Tuning Large Language Models for Multi-Turn Medical Conversations is Branching
  4. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models
  5. 2025Empathy by Design: Aligning Large Language Models for Healthcare Dialogue
  6. 2025Evaluating the Effectiveness of Direct Preference Optimization for Personalizing German Automatic Text Simplifications for Persons with Intellectual Disabilities
  7. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
  8. 2025DecoupledESC: Enhancing Emotional Support Generation via Strategy-Response Decoupled Preference Optimization