ICLR 2023 - On the duality between contrastive and non-contrastive self-supervised learning

In this episode we discuss On the duality between contrastive and non-contrastive self-supervised learning by Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, Yann Lecun. This paper discusses the duality between contrastive and non-contrastive self-supervised learning methods for image representations. It highlights the theoretical similarities between these approaches and introduces algebraically related contrastive and covariance-based non-contrastive criteria. The authors demonstrate through analysis and experiments that the performance gaps between the two methods can be closed with improved network design and hyperparameter tuning, challenging the assumption that non-contrastive methods require large output dimensions.