mlsanityData to 14 Oct 2026

Iterated Piecewise Affine (IPA) Approximation for Language Modeling

arXiv 2306.12317 ↗First posted 21 Jun 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this work, we demonstrate the application of a first-order Taylor expansion to approximate a generic function F: Rn × m → Rn × m and utilize it in language modeling. To enhance the basic Taylor expansion, we introduce iteration and piecewise modeling, leading us to name the algorithm the Iterative Piecewise Affine (IPA) approximation. The final algorithm exhibits interesting resemblances to the Transformers decoder architecture. By comparing parameter arrangements in IPA and Transformers, we observe a strikingly similar performance, with IPA outperforming Transformers by 1.5% in the next token prediction task with cross-entropy loss for smaller sequence lengths.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 29 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

24 of 29 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2020Rethinking Attention with Performerscited by 1,601
12 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  3. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  4. 2019Attention Augmented Convolutional Networkscited by 363
  5. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  6. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  7. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  8. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  9. 2020RealFormer: Transformer Likes Residual Attentioncited by 62
  10. 2017Learning Deep ResNet Blocks Sequentially using Boosting Theorycited by 32
  11. 2021Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognitioncited by 24
  12. 2017Efficient Attention using a Fixed-Size Memory Representationcited by 4

Built on by

Not cited yet.