mlsanityData to 13 Oct 2026

Enhancing Instruction-Following Capabilities in Seq2Seq Models: DoLA Adaptations for T5

arXiv 2512.03803 ↗First posted 3 Dec 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Encoder-decoder models such as FLAN-T5 are finetuned to follow instructions, but often fail when the instructions conflict with memorized continuations ingrained during training. To understand this behavior, we adapt DoLa to FLAN-T5 and examine how representations evolve in the decoder. Our findings show that T5's intermediate layers undergo rapid shifts driven by cross-attention to the encoder. When projected through the language modeling head, each depth presents highly volatile token preferences, leading to unreliable behavior with contrastive decoding. Motivated by this, we introduce a gradient-based activation-steering method that injects an "instruction-compliance" direction into mid-decoder layers, where the representation is both meaningful and still malleable. This intervention dramatically improves MemoTrap performance (52% to 99.7%), demonstrating that mechanistic steering can succeed where contrastive decoding fails in Seq2Seq architectures.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 40 of its 44 references that are in the map. First posted on arXiv 3 Dec 2025.

Builds on

40 of 44 references.

References · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,412
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,981
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  11. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,068
28 more
  1. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,323
  2. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,191
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,049
  4. 2023Steering Language Models With Activation Engineeringcited by 873
  5. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 757
  6. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 731
  7. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  8. 2023Instruction Tuning with GPT-4cited by 681
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 600
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 597
  11. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 437
  12. 2021A Survey of Transformerscited by 400
  13. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 352
  14. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 324
  15. 2023Function Vectors in Large Language Modelscited by 299
  16. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  17. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  18. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  19. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  20. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  21. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 92
  22. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 72
  23. 2018A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognitioncited by 66
  24. 2021Coherence boosting: When your pretrained language model is not paying enough attentioncited by 38
  25. 2023The Benefits of Bad Advice: Autocontrastive Decoding across Model Layerscited by 29
  26. 2024StableMask: Refining Causal Masking in Decoder-only Transformercited by 20
  27. 2023A Closer Look into Automatic Evaluation Using Large Language Modelscited by 17
  28. 2023Becoming self-instruct: introducing early stopping criteria for minimal instruct tuningcited by 13

Built on by

Not cited yet.