ICML 2023 - Generalization on the Unseen, Logic Reasoning and Degree Curriculum

In this episode we discuss Generalization on the Unseen, Logic Reasoning and Degree Curriculum by Emmanuel Abbe, Samy Bengio, Aryo Lotfi, Kevin Rizk. This paper examines the performance of different network architectures trained by stochastic gradient descent (SGD) in the generalization on the unseen (GOTU) setting. The authors find that certain network models, such as Transformers, random features models, and diagonal linear networks, can learn a min-degree-interpolator on unseen data. They also introduce a curriculum learning algorithm called Degree-Curriculum to address the challenges of learning in combinatorial reasoning tasks.