# Y^2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences

arXiv: https://arxiv.org/abs/1811.02745 · HTML: https://mlsanity.com/paper/1811.02745/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2018.

## Built on by (most cited first, 27 of 27)

- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://mlsanity.com/paper/2110.02624/) (2021)
- [SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer](https://mlsanity.com/paper/2108.04444/) (2021)
- [CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training](https://mlsanity.com/paper/2210.01055/) (2022)
- [Towards Implicit Text-Guided 3D Shape Generation](https://mlsanity.com/paper/2203.14622/) (2022)
- [Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priors](https://mlsanity.com/paper/2204.10603/) (2022)
- [Surface Reconstruction from Point Clouds by Learning Predictive Context Priors](https://mlsanity.com/paper/2204.11015/) (2022)
- [Snowflake Point Deconvolution for Point Cloud Completion and Generation with Skip-Transformer](https://mlsanity.com/paper/2202.09367/) (2022)
- [Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching](https://mlsanity.com/paper/2108.03746/) (2021)
- [Latent Partition Implicit with Surface Codes for 3D Representation](https://mlsanity.com/paper/2207.08631/) (2022)
- [MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Mesh](https://mlsanity.com/paper/2508.01242/) (2025)
- [TriCoLo: Trimodal Contrastive Loss for Text to Shape Retrieval](https://mlsanity.com/paper/2201.07366/) (2022)
- [Parts2Words: Learning Joint Embedding of Point Clouds and Texts by Bidirectional Matching between Parts and Words](https://mlsanity.com/paper/2107.01872/) (2021)
- [T2TD: Text-3D Generation Model based on Prior Knowledge Guidance](https://mlsanity.com/paper/2305.15753/) (2023)
- [Duoduo CLIP: Efficient 3D Understanding with Multi-View Images](https://mlsanity.com/paper/2406.11579/) (2024)
- [3DAlign-DAER: Dynamic Attention Policy and Efficient Retrieval Strategy for Fine-grained 3D-Text Alignment at Scale](https://mlsanity.com/paper/2511.13211/) (2025)
- [Uni3DL: Unified Model for 3D and Language Understanding](https://mlsanity.com/paper/2312.03026/) (2023)
- [SCA3D: Enhancing Cross-modal 3D Retrieval via 3D Shape and Caption Paired Data Augmentation](https://mlsanity.com/paper/2502.19128/) (2025)
- [Hierarchical View Predictor: Unsupervised 3D Global Feature Learning through Hierarchical Prediction among Unordered Views](https://mlsanity.com/paper/2108.03743/) (2021)
- [COM3D: Leveraging Cross-View Correspondence and Cross-Modal Mining for 3D Retrieval](https://mlsanity.com/paper/2405.04103/) (2024)
- [Enhanced Cross-modal 3D Retrieval via Tri-modal Reconstruction](https://mlsanity.com/paper/2504.01476/) (2025)
- [MXM-CLR: A Unified Framework for Contrastive Learning of Multifold Cross-Modal Representations](https://mlsanity.com/paper/2303.10839/) (2023)
- [What's in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models through User-Provided Names in CAD Files](https://mlsanity.com/paper/2304.14275/) (2023)
- [Text2shape Deep Retrieval Model: Generating Initial Cases for Mechanical Part Redesign under the Context of Case-Based Reasoning](https://mlsanity.com/paper/2302.06341/) (2023)
- [Diff-3DCap: Shape Captioning with Diffusion Models](https://mlsanity.com/paper/2509.23718/) (2025)
- [RI-Mamba: Rotation-Invariant Mamba for Robust Text-to-Shape Retrieval](https://mlsanity.com/paper/2602.11673/) (2026)
- [Text-to-CAD Retrieval: a Strong Baseline](https://mlsanity.com/paper/2605.05572/) (2026)
- [Octrees as an Explicit 3D Language](https://mlsanity.com/paper/2610.02388/) (2026)

## Abstract (arXiv)

A recent method employs 3D voxels to represent 3D shapes, but this limits the approach to low resolutions due to the computational cost caused by the cubic complexity of 3D voxels. Hence the method suffers from a lack of detailed geometry. To resolve this issue, we propose Y^2Seq2Seq, a view-based model, to learn cross-modal representations by joint reconstruction and prediction of view and word sequences. Specifically, the network architecture of Y^2Seq2Seq bridges the semantic meaning embedded in the two modalities by two coupled `Y' like sequence-to-sequence (Seq2Seq) structures. In addition, our novel hierarchical constraints further increase the discriminability of the cross-modal representations by employing more detailed discriminative information. Experimental results on cross-modal retrieval and 3D shape captioning show that Y^2Seq2Seq outperforms the state-of-the-art methods.