mlsanityData to 7 Oct 2026

Align on the Fly: Adapting Chatbot Behavior to Established Norms

arXiv 2312.15907 ↗First posted 26 Dec 2023cs.CLcited by 22

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

In this paper, we aim to align large language models with the ever-changing, complex, and diverse human values (e.g., social norms) across time and locations. This presents a challenge to existing alignment techniques, such as supervised fine-tuning, which internalize values within model parameters. To overcome this, we propose an On-the-fly Preference Optimization (OPO) method, which is a real-time alignment that works in a streaming way. It employs an external memory to store established rules for alignment, which can constrain LLMs' behaviors without further training, allowing for convenient updates and customization of human values. We also introduce a scalable evaluation to assess the proposed method more effectively. Experimental results on both human-annotated and auto-generated questions from legal and moral domains indicate the effectiveness of the proposed OPO method. Our code and data are released at https://github.com/GAIR-NLP/OPO.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 63 references that are in the map. First posted on arXiv 26 Dec 2023.

Builds on

42 of 63 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Training7 Nov
  2. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
  3. 2023LIMA: Less Is More for Alignment18 May
  4. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  5. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  7. 2022Training language models to follow instructions with human feedback4 Mar
  8. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
22 more
  1. 2019Language Models as Knowledge Bases?cited by 1,922
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  13. 2022Large Language Models Can Self-Improvecited by 656
  14. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  15. 2020Artificial Intelligence, Values and Alignmentcited by 443
  16. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  17. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  18. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  19. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  20. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  21. 2023XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameterscited by 92
  22. 2023Learning to Filter Context for Retrieval-Augmented Generationcited by 82

Built on by

Cited by year

Citing papers per year: 2024: 15; 2025: 7.

22 papers.

Sort:

By area:

  1. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 144
  2. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  3. 2024Towards Human-centered Proactive Conversational Agentscited by 48
  4. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  5. 2024Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Samplingcited by 33
  6. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  7. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
  8. 2024Assistive Large Language Model Agents for Socially-Aware Negotiation Dialoguescited by 18
  9. 2024Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reductioncited by 18
  10. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  11. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 11
  12. 2024LLMCRIT: Teaching Large Language Models to Use Criteriacited by 9
10 more
  1. 2024ProgressGym: Alignment with a Millennium of Moral Progresscited by 9
  2. 2024Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairnesscited by 6
  3. 2025LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimizationcited by 6
  4. 2024Evaluating Usability and Engagement of Large Language Models in Virtual Reality for Traditional Scottish Curlingcited by 4
  5. 2025AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Differencecited by 4
  6. 2024SAFETY-J: Evaluating Safety with Critiquecited by 3
  7. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 2
  8. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
  9. 2024Examining the Role of Relationship Alignment in Large Language Modelscited by 1
  10. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026HAL: Inducing Human-likeness in LLMs with Alignment
  2. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment
  4. 2024LAB: Large-Scale Alignment for ChatBots
  5. 2025On-the-fly Preference Alignment via Principle-Guided Decoding
  6. 2023Preference Ranking Optimization for Human Alignment
  7. 2025Human Preferences for Constructive Interactions in Language Model Alignment
  8. 2024Understanding the Learning Dynamics of Alignment with Human Feedback