arxiv Preprint - Language Modeling Is Compression

In this episode we discuss Language Modeling Is Compression by Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, Joel Veness. The authors argue that large language models can be seen as powerful compressors due to their predictive capabilities. They demonstrate that these models outperform specific compressors like PNG and FLAC. The paper explores the implications of the prediction-compression equivalence and discusses the use of any compressor to build a conditional generative model.