mlsanityData to 7 Oct 2026

Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Intervention

arXiv 2411.10156 ↗First posted 15 Nov 2024cs.AIcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

To address the sycophancy problem caused by reinforcement learning from human feedback in large language models, this research applies synthetic data intervention technology to the decoder-only transformer architecture. Based on the research gaps in the existing literature, the researcher designed an experimental process to reduce the tendency of models to cater by generating diversified data, and used GPT4o as an experimental tool for verification. The experiment used 100 true and false questions, and compared the performance of the model trained with synthetic data intervention and the original untrained model on multiple indicators. The results show that the SDI training model supports the technology in terms of accuracy rate and sycophancy rate and has significant effectiveness in reducing sycophancy phenomena.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 35 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

31 of 35 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation22 Jul
  2. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF29 Apr
  3. 2024Self-Rewarding Language Models18 Jan
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul

Other references · 27

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  4. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  5. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  6. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  7. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  8. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  9. 2023Machine Learning for Synthetic Data Generation: A Reviewcited by 118
  10. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  11. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 96
  12. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
15 more
  1. 2023When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviourcited by 61
  2. 2024Comprehensive Exploration of Synthetic Data Generation: A Surveycited by 59
  3. 2023A Study on ReLU and Softmax in Transformercited by 53
  4. 2023Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decodercited by 44
  5. 2023Learning Interpretable Style Embeddings via Prompting LLMscited by 29
  6. 2023How Powerful are Decoder-Only Transformer Neural Models?cited by 20
  7. 2023One Wide Feedforward is All You Needcited by 15
  8. 2024Decoder-only Architecture for Streaming End-to-end Speech Recognitioncited by 12
  9. 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMscited by 10
  10. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitationscited by 10
  11. 2021Generating Synthetic Text Data to Evaluate Causal Inference Methodscited by 9
  12. 2024Multimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedbackcited by 9
  13. 2024Mining Causality: AI-Assisted Search for Instrumental Variablescited by 7
  14. 2024Reducing Reasoning Costs: The Path of Optimization for Chain of Thought via Sparse Attention Mechanismcited by 4
  15. 2024Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2025Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactionscited by 12
  2. 2025PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Modelscited by 4
  3. 2026Consistency of Large Reasoning Models Under Multi-Turn Attackscited by 4
  4. 2026Acting Flatterers via LLMs Sycophancy: Combating Clickbait with LLMs Opposing-Stance Reasoningcited by 2
  5. 2024"Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
  2. 2026Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Models
  3. 2025Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformers
  4. 2026LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
  5. 2022TransDreamer: Reinforcement Learning with Transformer World Models
  6. 2025Exploring Diffusion Transformer Designs via Grafting
  7. 2022On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning
  8. 2026Decoupling Return-to-Go for Efficient Decision Transformer