ICCV 2023 - DreamTeacher: Pretraining Image Backbones with Deep Generative Models

In this episode we discuss DreamTeacher: Pretraining Image Backbones with Deep Generative Models by Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, Sanja Fidler. This paper presents DreamTeacher, a self-supervised feature representation learning framework that utilizes generative networks to pre-train image backbones. The authors propose two methods of knowledge distillation: transferring generative features to target backbones and transferring labels from generative networks to target backbones. Through extensive analysis and experiments, they demonstrate that DreamTeacher outperforms existing self-supervised learning approaches and that pre-training with DreamTeacher enhances performance on downstream datasets, showcasing the potential of generative models for representation learning without manual labeling.