Arxiv paper - Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control

In this episode, we discuss Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control by Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, Yuan Liu. The paper introduces "Diffusion as Shader" (DaS), a novel approach that supports various video control tasks within a unified framework by utilizing 3D control signals, overcoming the limitations of existing methods which are typically restricted to 2D signals. DaS achieves precise video manipulation, such as camera control and content editing, by employing 3D tracking videos, resulting in enhanced temporal consistency. The approach was fine-tuned within three days using 8 H800 GPUs and demonstrates strong performance in tasks like mesh-to-video generation and motion transfer, with further resources available online.