# Time-aware Large Kernel Convolutions

arXiv: https://arxiv.org/abs/2002.03184 · HTML: https://mlsanity.com/paper/2002.03184/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2020.

## Built on by (most cited first, 8 of 8)

- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Fast Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.14528/) (2021)
- [TorchDEQ: A Library for Deep Equilibrium Models](https://mlsanity.com/paper/2310.18605/) (2023)
- [Faster Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2112.08152/) (2021)
- [A Survey on Mamba Architecture for Vision Applications](https://mlsanity.com/paper/2502.07161/) (2025)
- [Modeling Temporal Concept Receptive Field Dynamically for Untrimmed Video Analysis](https://mlsanity.com/paper/2111.11653/) (2021)
- [Surveying the Landscape of Text Summarization with Deep Learning: A Comprehensive Review](https://mlsanity.com/paper/2310.09411/) (2023)
- [A Quantitative Review on Language Model Efficiency Research](https://mlsanity.com/paper/2306.01768/) (2023)

## Abstract (arXiv)

To date, most state-of-the-art sequence modeling architectures use attention to build generative models for language based tasks. Some of these models use all the available sequence tokens to generate an attention distribution which results in time complexity of O(n2). Alternatively, they utilize depthwise convolutions with softmax normalized kernels of size k acting as a limited-window self-attention, resulting in time complexity of O(k·n). In this paper, we introduce Time-aware Large Kernel (TaLK) Convolutions, a novel adaptive convolution operation that learns to predict the size of a summation kernel instead of using a fixed-sized kernel matrix. This method yields a time complexity of O(n), effectively making the sequence encoding process linear to the number of tokens. We evaluate the proposed method on large-scale standard machine translation, abstractive summarization and language modeling datasets and show that TaLK Convolutions constitute an efficient improvement over other attention/convolution based approaches.