OptimusMesh: Compact Autoregressive Mesh Generation from Point Clouds via Sparse Latent Pivots

Generating compact and geometrically faithful 3D meshes directly from point clouds remains a fundamental challenge. Point clouds are unordered and sparse, whereas meshes exhibit irregular structure and varying topology. As a result, many existing approaches rely on implicit representations followed by surface extraction or reconstruction. Although effective, these pipelines can produce dense or over-smoothed meshes, often requiring computationally expensive post-processing and simplification. We present OptimusMesh, a framework for direct compact triangle mesh generation from point clouds using sparse latent pivot conditioning. Our key idea is to compress 2,048 oriented input points into only 16 sparse latent pivots, reducing the geometric conditioning set by 128×. These pivots provide a compact structural representation shared across a two-stage autoregressive framework that first generates mesh vertices and then predicts triangular faces conditioned on the generated vertices and the same pivots. Compared with the evaluated recent point-cloud-conditioned autoregressive methods, which use 257 decoder-conditioning tokens, OptimusMesh uses only 16, yielding a 16.1× shorter conditioning sequence. Experiments show that OptimusMesh produces the most compact outputs among the compared recent autoregressive methods, using 25.7%--94.1% fewer faces while maintaining competitive geometric fidelity and distributional quality.