CVPR 2023 - Magic3D: High-Resolution Text-to-3D Content Creation

In this episode we discuss Magic3D: High-Resolution Text-to-3D Content Creation by Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin. The paper introduces a two-stage optimization framework called Magic3D to address the slow optimization and low-resolution image space supervision limitations of the pre-trained text-to-image diffusion model called DreamFusion. The first stage involves obtaining a coarse model using a low-resolution diffusion prior and accelerating it with a sparse 3D hash grid structure. In the second stage, a textured 3D mesh model is optimized using an efficient differentiable renderer interacting with a high-resolution latent diffusion model. Magic3D can create high-quality 3D mesh models in 40 minutes, 2x faster than DreamFusion, while achieving higher resolution. User studies show that 61.7% of raters prefer Magic3D over DreamFusion.