arxiv preprint - When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

In this episode, we discuss When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method by Biao Zhang, Zhongtao Liu, Colin Cherry, Orhan Firat. The paper investigates how various scaling factors impact the effectiveness of finetuning large language models (LLMs), focusing on full-model tuning (FMT) and parameter-efficient tuning (PET). Through experiments with bilingual LLMs and tasks like machine translation and summarization, the authors find that finetuning follows a joint scaling law where increasing model size is more beneficial than increasing the size of the pretraining data, and that PET's additional parameters typically don't improve performance. They conclude that the best finetuning approach depends on the specific task and the amount of finetuning data available, providing insights for selecting and improving LLM finetuning methods.