# View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrums

arXiv: https://arxiv.org/abs/2110.07288 · HTML: https://mlsanity.com/paper/2110.07288/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 63 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2103.07719/) (2021)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction](https://mlsanity.com/paper/1903.02793/) (2019)
- [The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction](https://mlsanity.com/paper/1912.06445/) (2019)
- [MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction](https://mlsanity.com/paper/2006.03340/) (2020)
- [Recursive Social Behavior Graph for Trajectory Prediction](https://mlsanity.com/paper/2004.10402/) (2020)
- [Trajectory Prediction with Latent Belief Energy-Based Model](https://mlsanity.com/paper/2104.03086/) (2021)
- [LOKI: Long Term and Key Intentions for Trajectory Prediction](https://mlsanity.com/paper/2108.08236/) (2021)
- [Probabilistic Vehicle Trajectory Prediction over Occupancy Grid Map via Recurrent Neural Network](https://mlsanity.com/paper/1704.07049/) (2017)
- [Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection](https://mlsanity.com/paper/1702.05552/) (2017)

## Built on by (most cited first, 30 of 45)

- [EigenTrajectory: Low-Rank Descriptors for Multi-Modal Trajectory Forecasting](https://mlsanity.com/paper/2307.09306/) (2023)
- [SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model](https://mlsanity.com/paper/2403.18452/) (2024)
- [Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction](https://mlsanity.com/paper/2403.18447/) (2024)
- [SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2310.05370/) (2023)
- [LG-Traj: LLM Guided Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2403.08032/) (2024)
- [End-to-End Trajectory Distribution Prediction Based on Occupancy Grid Maps](https://mlsanity.com/paper/2203.16910/) (2022)
- [Domain Re-Modulation for Few-Shot Generative Domain Adaptation](https://mlsanity.com/paper/2302.02550/) (2023)
- [Unsupervised Sampling Promoting for Stochastic Human Trajectory Prediction](https://mlsanity.com/paper/2304.04298/) (2023)
- [Joint Metrics Matter: A Better Standard for Trajectory Forecasting](https://mlsanity.com/paper/2305.06292/) (2023)
- [Progressive Pretext Task Learning for Human Trajectory Prediction](https://mlsanity.com/paper/2407.11588/) (2024)
- [Another Vertical View: A Hierarchical Network for Heterogeneous Trajectory Prediction via Spectrums](https://mlsanity.com/paper/2304.05106/) (2023)
- [Joint Pedestrian Trajectory Prediction through Posterior Sampling](https://mlsanity.com/paper/2404.00237/) (2024)
- [Pishgu: Universal Path Prediction Network Architecture for Real-time Cyber-physical Edge Systems](https://mlsanity.com/paper/2210.08057/) (2022)
- [BP-SGCN: Behavioral Pseudo-Label Informed Sparse Graph Convolution Network for Pedestrian and Heterogeneous Trajectory Prediction](https://mlsanity.com/paper/2502.14676/) (2025)
- [Pedestrian Trajectory Prediction Based on Social Interactions Learning With Random Weights](https://mlsanity.com/paper/2501.07711/) (2025)
- [Efficient Backdoor Attacks for Deep Neural Networks in Real-world Scenarios](https://mlsanity.com/paper/2306.08386/) (2023)
- [Human Trajectory Forecasting with Explainable Behavioral Uncertainty](https://mlsanity.com/paper/2307.01817/) (2023)
- [SocialCircle+: Learning the Angle-based Conditioned Interaction Representation for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2409.14984/) (2024)
- [Continuous Locomotive Crowd Behavior Generation](https://mlsanity.com/paper/2504.04756/) (2025)
- [SocialMOIF: Multi-Order Intention Fusion for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2504.15616/) (2025)
- [T2FPV: Dataset and Method for Correcting First-Person View Errors in Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2209.11294/) (2022)
- [Resonance: Learning to Predict Social-Aware Pedestrian Trajectories as Co-Vibrations](https://mlsanity.com/paper/2412.02447/) (2024)
- [AMD: Adaptive Momentum and Decoupled Contrastive Learning Framework for Robust Long-Tail Trajectory Prediction](https://mlsanity.com/paper/2507.01801/) (2025)
- [Pedestrian Trajectory Forecasting Using Deep Ensembles Under Sensing Uncertainty](https://mlsanity.com/paper/2305.16620/) (2023)
- [Sparse Pedestrian Character Learning for Trajectory Prediction](https://mlsanity.com/paper/2311.15512/) (2023)
- [Risk-aware Trajectory Prediction by Incorporating Spatio-temporal Traffic Interaction Analysis](https://mlsanity.com/paper/2407.10639/) (2024)
- [Agile Decision-Making and Safety-Critical Motion Planning for Emergency Autonomous Vehicles](https://mlsanity.com/paper/2409.08665/) (2024)
- [CCF: Cross Correcting Framework for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2406.00749/) (2024)
- [G-PECNet: Towards a Generalizable Pedestrian Trajectory Prediction System](https://mlsanity.com/paper/2210.09846/) (2022)
- [GoalNet: Goal Areas Oriented Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2402.19002/) (2024)

## Abstract (arXiv)

Understanding and forecasting future trajectories of agents are critical for behavior analysis, robot navigation, autonomous cars, and other related applications. Previous methods mostly treat trajectory prediction as time sequence generation. Different from them, this work studies agents' trajectories in a "vertical" view, i.e., modeling and forecasting trajectories from the spectral domain. Different frequency bands in the trajectory spectrums could hierarchically reflect agents' motion preferences at different scales. The low-frequency and high-frequency portions could represent their coarse motion trends and fine motion variations, respectively. Accordingly, we propose a hierarchical network V2-Net, which contains two sub-networks, to hierarchically model and predict agents' trajectories with trajectory spectrums. The coarse-level keypoints estimation sub-network first predicts the "minimal" spectrums of agents' trajectories on several "key" frequency portions. Then the fine-level spectrum interpolation sub-network interpolates the spectrums to reconstruct the final predictions. Experimental results display the competitiveness and superiority of V2-Net on both ETH-UCY benchmark and the Stanford Drone Dataset.