mlsanityData to 7 Oct 2026

Spline-based Transformers

arXiv 2504.02797 ↗First posted 3 Apr 2025cs.LG cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

We introduce Spline-based Transformers, a novel class of Transformer models that eliminate the need for positional encoding. Inspired by workflows using splines in computer animation, our Spline-based Transformers embed an input sequence of elements as a smooth trajectory in latent space. Overcoming drawbacks of positional encoding such as sequence length extrapolation, Spline-based Transformers also provide a novel way for users to interact with transformer latent spaces by directly manipulating the latent control points to create new latent trajectories and sequences. We demonstrate the superior performance of our approach in comparison to conventional positional encoding on a variety of datasets, ranging from synthetic 2D to large-scale real-world datasets of images, 3D shapes, and animations.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

32 of 46 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
20 more
  1. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  2. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  3. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  4. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  6. 2022Human Motion Diffusion Modelcited by 1,062
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  8. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  9. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  10. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  11. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  12. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  13. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  14. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  15. 2021Single-Shot Motion Completion with Transformercited by 59
  16. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  17. 2022Neural Strands: Learning Hair Geometry and Appearance from Multi-View Imagescited by 39
  18. 2021Augmenting Convolutional networks with attention-based aggregationcited by 29
  19. 2023GroomGen: A High-Quality Generative Hair Model Using Hierarchical Latent Representationscited by 28
  20. 2023Position Interpolation Improves ALiBi Extrapolationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026DesigNet: Learning to Draw Vector Graphics as Designers Docited by 0
  2. 2026Representing 3D Faces with Learnable B-Spline Volumescited by 0