The Markovian Thinker

In this episode, we discuss The Markovian Thinker by Milad Aghajohari, Kamran Chitsaz, Amirhossein Kazemnejad, Sarath Chandar, Alessandro Sordoni, Aaron Courville, Siva Reddy. The paper proposes Markovian Thinking, a reinforcement learning paradigm that limits reasoning context to a constant-size state, enabling linear compute with constant memory rather than quadratic overhead. They implement this approach in Delethink, an environment that segments reasoning into fixed-size chunks with learned textual states to seamlessly continue reasoning after resets. Experiments show Delethink-trained models achieve longer reasoning chains more efficiently and scale better than standard methods, significantly reducing computational costs.