# 3D-LFM: Lifting Foundation Model

arXiv: https://arxiv.org/abs/2312.11894 · HTML: https://mlsanity.com/paper/2312.11894/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [CanonPose: Self-Supervised Monocular 3D Human Pose Estimation in the Wild](https://mlsanity.com/paper/2011.14679/) (2020)
- [C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion](https://mlsanity.com/paper/1909.02533/) (2019)
- [Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shape](https://mlsanity.com/paper/2308.11737/) (2023)
- [Deep Non-Rigid Structure from Motion](https://mlsanity.com/paper/1908.00052/) (2019)
- [AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild](https://mlsanity.com/paper/2103.13282/) (2021)
- [H3WB: Human3.6M 3D WholeBody Dataset and Benchmark](https://mlsanity.com/paper/2211.15692/) (2022)
- [Jointformer: Single-Frame Lifting Transformer with Error Prediction and Refinement for 3D Human Pose Estimation](https://mlsanity.com/paper/2208.03704/) (2022)
- [MHR-Net: Multiple-Hypothesis Reconstruction of Non-Rigid Shapes from 2D Views](https://mlsanity.com/paper/2207.09086/) (2022)
- [Unsupervised 3D Pose Estimation with Non-Rigid Structure-from-Motion Modeling](https://mlsanity.com/paper/2308.10705/) (2023)
- [Robust Point Cloud Processing through Positional Embedding](https://mlsanity.com/paper/2309.00339/) (2023)
- [High Fidelity 3D Reconstructions with Limited Physical Views](https://mlsanity.com/paper/2110.11599/) (2021)
- [MBW: Multi-view Bootstrapping in the Wild](https://mlsanity.com/paper/2210.01721/) (2022)

## Built on by (most cited first, 8 of 8)

- [EFM3D: A Benchmark for Measuring Progress Towards 3D Egocentric Foundation Models](https://mlsanity.com/paper/2406.10224/) (2024)
- [AniMer: Animal Pose and Shape Estimation Using Family Aware Transformer](https://mlsanity.com/paper/2412.00837/) (2024)
- [AniMer+: Unified Pose and Shape Estimation Across Mammalia and Aves via Family-Aware Transformer](https://mlsanity.com/paper/2508.00298/) (2025)
- [3D WholeBody Pose Estimation based on Semantic Graph Attention Network and Distance Information](https://mlsanity.com/paper/2406.01196/) (2024)
- [Object Agnostic 3D Lifting in Space and Time](https://mlsanity.com/paper/2412.01166/) (2024)
- [Skel3D: Skeleton Guided Novel View Synthesis](https://mlsanity.com/paper/2412.03407/) (2024)
- [PoseMoE: Mixture-of-Experts Network for Monocular 3D Human Pose Estimation](https://mlsanity.com/paper/2512.16494/) (2025)
- [Prosthesis-Aware 3D Human Pose Estimation: A Dataset and Benchmark for RSP Users](https://mlsanity.com/paper/2609.18406/) (2026)

## Abstract (arXiv)

The lifting of 3D structure and camera from 2D landmarks is at the cornerstone of the entire discipline of computer vision. Traditional methods have been confined to specific rigid objects, such as those in Perspective-n-Point (PnP) problems, but deep learning has expanded our capability to reconstruct a wide range of object classes (e.g. C3DPO and PAUL) with resilience to noise, occlusions, and perspective distortions. All these techniques, however, have been limited by the fundamental need to establish correspondences across the 3D training data -- significantly limiting their utility to applications where one has an abundance of "in-correspondence" 3D data. Our approach harnesses the inherent permutation equivariance of transformers to manage varying number of points per 3D data instance, withstands occlusions, and generalizes to unseen categories. We demonstrate state of the art performance across 2D-3D lifting task benchmarks. Since our approach can be trained across such a broad class of structures we refer to it simply as a 3D Lifting Foundation Model (3D-LFM) -- the first of its kind.