# DIANet: Dense-and-Implicit Attention Network

arXiv: https://arxiv.org/abs/1905.10671 · HTML: https://mlsanity.com/paper/1905.10671/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2019.

## Built on by (most cited first, 30 of 37)

- [RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs](https://mlsanity.com/paper/2503.10657/) (2025)
- [SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models](https://mlsanity.com/paper/2305.05189/) (2023)
- [ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection](https://mlsanity.com/paper/2310.13545/) (2023)
- [Cross-Layer Retrospective Retrieving via Layer Attention](https://mlsanity.com/paper/2302.03985/) (2023)
- [Blending Pruning Criteria for Convolutional Neural Networks](https://mlsanity.com/paper/2107.05033/) (2021)
- [MoExtend: Tuning New Experts for Modality and Task Extension](https://mlsanity.com/paper/2408.03511/) (2024)
- [Strengthening Layer Interaction via Dynamic Layer Attention](https://mlsanity.com/paper/2406.13392/) (2024)
- [BA-Net: Bridge Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2112.04150/) (2021)
- [On Fast Simulation of Dynamical System with Neural Vector Enhanced Numerical Solver](https://mlsanity.com/paper/2208.03680/) (2022)
- [Understanding Self-attention Mechanism via Dynamical System Perspective](https://mlsanity.com/paper/2308.09939/) (2023)
- [The Lottery Ticket Hypothesis for Self-attention in Convolutional Neural Network](https://mlsanity.com/paper/2207.07858/) (2022)
- [Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models](https://mlsanity.com/paper/2505.12343/) (2025)
- [AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training](https://mlsanity.com/paper/2107.05804/) (2021)
- [Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minima](https://mlsanity.com/paper/2402.11262/) (2024)
- [A Generic Shared Attention Mechanism for Various Backbone Neural Networks](https://mlsanity.com/paper/2210.16101/) (2022)
- [TDAM: Top-Down Attention Module for Contextually Guided Feature Selection in CNNs](https://mlsanity.com/paper/2111.13470/) (2021)
- [Switchable Self-attention Module](https://mlsanity.com/paper/2209.05680/) (2022)
- [AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios](https://mlsanity.com/paper/2302.10184/) (2023)
- [LSAS: Lightweight Sub-attention Strategy for Alleviating Attention Bias Problem](https://mlsanity.com/paper/2305.05200/) (2023)
- [HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2506.17587/) (2025)
- [Mix-Pooling Strategy for Attention Mechanism](https://mlsanity.com/paper/2208.10322/) (2022)
- [Enhancing Layer Attention Efficiency through Pruning Redundant Retrievals](https://mlsanity.com/paper/2503.06473/) (2025)
- [ASR: Attention-alike Structural Re-parameterization](https://mlsanity.com/paper/2304.06345/) (2023)
- [Cross-layer Navigation Convolutional Neural Network for Fine-grained Visual Classification](https://mlsanity.com/paper/2106.10920/) (2021)
- [Signature-Graph Networks](https://mlsanity.com/paper/2110.11551/) (2021)
- [GCCN: Global Context Convolutional Network](https://mlsanity.com/paper/2110.11664/) (2021)
- [A novel adversarial learning strategy for medical image classification](https://mlsanity.com/paper/2206.11501/) (2022)
- [Efficient Climate Simulation via Machine Learning Method](https://mlsanity.com/paper/2209.08151/) (2022)
- [CSA-Net: Channel-wise Spatially Autocorrelated Attention Networks](https://mlsanity.com/paper/2405.05755/) (2024)
- [Causal Inference for Chatting Handoff](https://mlsanity.com/paper/2210.02862/) (2022)

## Abstract (arXiv)

Attention networks have successfully boosted the performance in various vision problems. Previous works lay emphasis on designing a new attention module and individually plug them into the networks. Our paper proposes a novel-and-simple framework that shares an attention module throughout different network layers to encourage the integration of layer-wise information and this parameter-sharing module is referred as Dense-and-Implicit-Attention (DIA) unit. Many choices of modules can be used in the DIA unit. Since Long Short Term Memory (LSTM) has a capacity of capturing long-distance dependency, we focus on the case when the DIA unit is the modified LSTM (refer as DIA-LSTM). Experiments on benchmark datasets show that the DIA-LSTM unit is capable of emphasizing layer-wise feature interrelation and leads to significant improvement of image classification accuracy. We further empirically show that the DIA-LSTM has a strong regularization ability on stabilizing the training of deep networks by the experiments with the removal of skip connections or Batch Normalization in the whole residual network. The code is released at https://github.com/gbup-group/DIANet.