Repurposing 2D Diffusion Models for 3D Shape Completion

Yao He, Youngjoong Kwon, Tiange Xiang, Wenxiao Cai, Ehsan Adeli
Stanford University

PDF Code ArXiv

Abstract

We present a framework that adapts 2D diffusion models for 3D shape completion from incomplete point clouds. While text-to-image diffusion models have achieved remarkable success with abundant 2D data, 3D diffusion models lag due to the scarcity of high-quality 3D datasets and a persistent modality gap between 3D inputs and 2D latent spaces. To overcome these limitations, we introduce the Shape Atlas, a compact 2D representation of 3D geometry that (1) en- ables full utilization of the generative power of pretrained 2D diffusion models, and (2) aligns the modalities between the conditional input and output spaces, allowing more ef- fective conditioning. This unified 2D formulation facilitates learning from limited 3D data and produces high-quality, detail-preserving shape completions. We validate the effec- tiveness of our results on the PCN and ShapeNet-55 datasets. Additionally, we show the downstream application of creat- ing artist-created meshes from our completed point clouds, further demonstrating the practicality of our method.

Reformulating 3D as 2D Shape Atlas

The input point cloud is mapped onto the surface of a standard sphere S through spherical offsetting, where points sharing the same color indicate the 1-to-1 correspondence. The spherical points are then flattened onto the 2D plane through plane offsetting, forming a dense √N x √N 2D atlas. Our formulation supports both complete and incomplete inputs. Each offsetting is performing an optimal transport with KNN-based acceleration.




Diffusion-based Shape Completion

We train a conditional diffusion model in which 3D reconstruction losses (i.e., L_CD, L_InfoCD, L_mesh) complement the standard 2D diffusion objective (i.e., L_denoise). The incomplete atlas is processed by a conditioning U-Net U_cond to provide control signals to the denoiser U_denoiser. The denoiser then generates a complete atlas, which is subsequently mapped back to a full 3D point cloud via the inverse of plane offsetting.



Checkout our demos and more results in the paper!












BibTeX

@inproceedings{yao2026repurpose,
    title={Repurposing 2D Diffusion Models for 3D Shape Completion},
    author={He, Yao and Kwon, Youngjoong and Xiang, Tiange and Cai, Wenxiao and Adeli, Ehsan},
    booktitle={},
    year={2025}
}