mlsanityData to 7 Oct 2026

Can Large Language Models Change User Preference Adversarially?

arXiv 2302.10291 ↗First posted 5 Jan 2023cs.CL cs.LGcited by 5

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Pretrained large language models (LLMs) are becoming increasingly powerful and ubiquitous in mainstream applications such as being a personal assistant, a dialogue model, etc. As these models become proficient in deducing user preferences and offering tailored assistance, there is an increasing concern about the ability of these models to influence, modify and in the extreme case manipulate user preference adversarially. The issue of lack of interpretability in these models in adversarial settings remains largely unsolved. This work tries to study adversarial behavior in user preferences from the lens of attention probing, red teaming and white-box analysis. Specifically, it provides a bird's eye view of existing literature, offers red teaming samples for dialogue models like ChatGPT and GODEL and probes the attention mechanism in the latter for non-adversarial and adversarial settings.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 43 of its 62 references that are in the map. First posted on arXiv 5 Jan 2023.

Builds on

43 of 62 references.

References · 43

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
31 more
  1. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  3. 2019Designing and Interpreting Probes with Control Taskscited by 608
  4. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  5. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  6. 2021Causal Abstractions of Neural Networkscited by 401
  7. 2019Revealing the Dark Secrets of BERTcited by 300
  8. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  9. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  10. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  11. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  12. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  13. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  14. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  15. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  16. 2020Compositional Explanations of Neuronscited by 172
  17. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  18. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 150
  19. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  20. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  21. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  22. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77
  23. 2021Tailor: Generating and Perturbing Text with Semantic Controlscited by 67
  24. 2021Improved OOD Generalization via Adversarial Training and Pre-trainingcited by 66
  25. 2019Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QAcited by 63
  26. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  27. 2022GODEL: Large-Scale Pre-Training for Goal-Directed Dialogcited by 50
  28. 2020Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Modelscited by 49
  29. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  30. 2021Analyzing Dynamic Adversarial Training Data in the Limitcited by 24
  31. 2022Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AIcited by 12

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1.

5 papers.

  1. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  2. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  3. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  4. 2024Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methodscited by 26
  5. 2023RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Modelscited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2025WorldPM: Scaling Human Preference Modeling
  2. 2025Matching Game Preferences Through Dialogical Large Language Models: A Perspective
  3. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game
  4. 2024A Survey on Human Preference Learning for Large Language Models
  5. 2024Dissecting Human and LLM Preferences
  6. 2024Self-Preference Bias in LLM-as-a-Judge
  7. 2025Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?
  8. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs