arxiv preprint - To Code, or Not To Code? Exploring Impact of Code in Pre-training

In this episode, we discuss To Code, or Not To Code? Exploring Impact of Code in Pre-training by Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, Sara Hooker. In this study, the impact of incorporating code data during pre-training on various downstream tasks was systematically investigated. The findings indicate that including code enhances performance in natural language reasoning, world knowledge, and code-specific tasks, suggesting that code data is essential for generalization beyond just coding tasks. Specifically, code inclusion resulted in significant performance improvements, highlighting the importance of maintaining high-quality code data in pre-training LLMs.