mlsanityData to 14 Oct 2026

Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs

arXiv 2405.14862 ↗First posted 23 May 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Decoder-only large language models typically rely solely on masked causal attention, which limits their expressiveness by restricting information flow to one direction. We propose Bitune, a method that enhances pretrained decoder-only LLMs by incorporating bidirectional attention into prompt processing. We evaluate Bitune in instruction-tuning and question-answering settings, showing significant improvements in performance on commonsense reasoning, arithmetic, and language understanding tasks. Furthermore, extensive ablation studies validate the role of each component of the method, and demonstrate that Bitune is compatible with various parameter-efficient finetuning techniques and full model finetuning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

25 of 41 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
13 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  3. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  4. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  6. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  7. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  8. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  9. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  10. 2022UL2: Unifying Language Learning Paradigmscited by 295
  11. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  12. 2023AnglE-optimized Text Embeddingscited by 99
  13. 2024Repetition Improves Language Model Embeddingscited by 72

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
  2. 2023Re-Reading Improves Reasoning in Large Language Models
  3. 2026PARTREP: Learning What to Repeat for Decoder-only LLMs
  4. 2024Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs
  5. 2026Sliding-window beats linear attention
  6. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation
  7. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
  8. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention