mlsanityData to 7 Oct 2026

ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignment

arXiv 2601.06157 ↗First posted 6 Jan 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment in large language models (LLMs) is still largely static: after training, the policy is frozen. DPO, GRPO methods typically imprint one behavior into the weights, leaving little runtime control beyond prompt hacks or expensive re-alignment. We introduce ECLIPTICA, which treats alignment as instruction-driven and runtime-controllable: natural-language alignment instructions act as an explicit behavioral contract (stance, refusal boundary, verbosity) that modulates behavior on the fly under evolving safety requirements, user roles, and governance constraints. We introduce CITA (Contrastive Instruction-Tuned Alignment), combining SFT with contrastive preference optimization under an explicit geometric anchor to a reference model. This yields a stable Riemannian chart and keeps instruction updates within a shared neighborhood, so regimes stay nearby and traversable for reliable switching. To isolate policy switching from ordinary instruction following, we release the ECLIPTICA benchmark: 3000 controlled cases (300 prompts x 10 instruction types) where the user request is fixed and only the alignment instruction changes. On Llama-3.1-8B across five suites (ECLIPTICA, TruthfulQA, Conditional Safety, Length Control, LITMUS), CITA reaches 86.7% instruction-alignment efficiency, beating DPO (56.1%), GRPO (36.1%), and PPO (20.4%).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 60 of its 76 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

60 of 76 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  6. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2023LIMA: Less Is More for Alignment18 May
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2023Mistral 7Bcited by 3,065
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  13. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  14. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  15. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  16. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  17. 2022Red Teaming Language Models with Language Modelscited by 983
  18. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  19. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  20. 2020Aligning AI With Shared Human Valuescited by 706
  21. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  22. 2023Instruction Tuning with GPT-4cited by 676
  23. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  24. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  25. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  26. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  27. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  28. 2023Are aligned neural networks adversarially aligned?cited by 305
  29. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  30. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  31. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  32. 2022Holistic Evaluation of Language Modelscited by 115
  33. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 93
  34. 2024Reinforcement Learning for LLM Post-Training: A Surveycited by 91
  35. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  36. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Modelscited by 27
  37. 2025Multimodal Benchmarking and Recommendation of Text-to-Image Generation Modelscited by 3
  38. 2025A Comprehensive Dataset for Human vs. AI Generated Text Detectioncited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models
  2. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
  3. 2025MAIN: Mutual Alignment Is Necessary for instruction tuning
  4. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  5. 2026Ready2Blend: From Natural-Language Instructions to Composable Alignment Prompts
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
  7. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning
  8. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs