mlsanityData to 7 Oct 2026

Reformatted Alignment

arXiv 2402.12219 ↗First posted 19 Feb 2024cs.CL cs.AI cs.LGcited by 15

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The quality of finetuning data is crucial for aligning large language models (LLMs) with human values. Current methods to improve data quality are either labor-intensive or prone to factual errors caused by LLM hallucinations. This paper explores elevating the quality of existing instruction data to better align with human values, introducing a simple and effective approach named ReAlign, which reformats the responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence. This approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques. Experimentally, ReAlign significantly boosts the general alignment ability, math reasoning, factuality, and readability of the LLMs. Encouragingly, without introducing any additional data or advanced training techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy. Additionally, a mere 5% of ReAlign data yields a 67% boost in general alignment ability measured by the Alpaca dataset. This work highlights the need for further research into the science and mechanistic interpretability of LLMs. We have made the associated code and data publicly accessible to support future studies at https://github.com/GAIR-NLP/ReAlign.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 63 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

28 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  2. 2023LIMA: Less Is More for Alignment18 May
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  12. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
11 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  2. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  3. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  4. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  5. 2023The False Promise of Imitating Proprietary LLMscited by 254
  6. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  7. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  8. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  9. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  10. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  11. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
  2. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  3. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 56
  4. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  5. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  6. 2024Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?cited by 23
  7. 2024Are Large Language Models Good at Utility Judgments?cited by 22
  8. 2024Learn Beyond The Answer: Training Language Models with Reflection for Mathematical Reasoningcited by 15
  9. 2024Alignment at Pre-training! Towards Native Alignment for Arabic LLMscited by 7
  10. 2025MetaLadder: Ascending Mathematical Solution Quality via Analogical-Problem Reasoning Transfercited by 6
  11. 2025Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scalecited by 5
  12. 2025SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoningcited by 4
3 more
  1. 2025Toward Cybersecurity-Expert Small Language Modelscited by 3
  2. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 1
  3. 2026UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  2. 2024Aligners: Decoupling LLMs and Alignment
  3. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samples
  4. 2026Are Aligned Large Language Models Still Misaligned?
  5. 2025Scaling Item-to-Standard Alignment with Large Language Models: Accuracy, Limits, and Solutions
  6. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models
  7. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
  8. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis