mlsanityData to 7 Oct 2026

DimPO: Dimensionality Reduction for Attention using Preference Optimization

arXiv 2609.32579 ↗First posted 26 Sep 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods also fits: Preference alignment for recommenders and RAG

Abstract (arXiv)

A linear projection can reduce the dimension of query and key vectors without updating the pretrained model, but it remains unclear which training objective best preserves model behavior. We ask whether preferences over keys and attention mass on the highest-weighted keys provide a better signal than matching the full attention distribution with KL divergence, especially in long-context settings. We introduce DimPO, which combines listwise preference optimization with a lightweight top-k cross-entropy term for head-fidelity. DimPO is trained offline from the attention patterns of a frozen language model, with one map per layer, shared by the query and the keys and trained separately from the other layers. Across LLaMA3.2-3B, LLaMA3.1-8B, Qwen2.5-7B, and Qwen3-4B Instruct models, pairwise preference objectives outperform the triplet baseline and retain 98% of the original score on short-context tasks when projecting to half the dimension on the last 40% of the layers. With more projected layers or on long-context RULER, they degrade rapidly. In contrast, KL and DimPO, which use every key during training, retain about 95% of the original RULER 4k score on the 8B model when projecting up to 50% of the layers. KL-based projections remain closer to the original attention distribution and attention output, yet DimPO achieves better downstream performance. Beyond 50% of projected layers, DimPO increasingly outperforms KL on tasks including SQuAD, common-word extraction, frequent-word extraction, and variable tracking. These results suggest that under dimensionality reduction, preserving the ordering and concentration of task-relevant attention can matter more than reproducing the full attention distribution.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 55 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

44 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  3. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  4. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan

Other references · 40

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
28 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  7. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  8. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  9. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  10. 2019Axial Attention in Multidimensional Transformerscited by 350
  11. 2023Adapting Language Models to Compress Contextscited by 350
  12. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  13. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  14. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  15. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  16. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  17. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  18. 2024Long Context Compression with Activation Beaconcited by 101
  19. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  20. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  21. 2023KDEformer: Accelerating Transformers via Kernel Density Estimationcited by 58
  22. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  23. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
  24. 2017DIMAL: Deep Isometric Manifold Learning Using Sparse Geodesic Samplingcited by 16
  25. 2024LLoCO: Learning Long Contexts Offlinecited by 16
  26. 2025ZETA: Leveraging Z-order Curves for Efficient Top-k Attentioncited by 9
  27. 2024IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUscited by 8
  28. 2026DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings
  2. 2026Token-weighted Direct Preference Optimization with Attention
  3. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
  4. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment
  5. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  6. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  7. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  8. 2026Latent Adversarial Regularization for Offline Preference Optimization