AI Breakdown

arxiv Preprint - Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading

In this episode we discuss Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading
by Howard Chen, Ramakanth Pasunuru, Jason Weston, Asli Celikyilmaz. The paper introduces MEMWALKER, an approach to address the limitations of the self-attention mechanism in large language models (LLMs) when processing long sequences. MEMWALKER treats the LLM as an interactive agent that iteratively reads the text, processing the long context into a tree of summary nodes. The model is then able to navigate this tree to gather relevant information and respond to queries. The paper demonstrates that MEMWALKER outperforms existing methods for long-text question answering tasks and enhances explainability by highlighting reasoning steps and relevant text segments.

arxiv Preprint - HyperAttention: Long-context Attention in Near-Linear Time

In this episode we discuss HyperAttention: Long-context Attention in Near-Linear Time
by Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P. Woodruff, Amir Zandieh. The paper introduces "HyperAttention," an approximate attention mechanism for handling long contexts in Large Language Models (LLMs). It proposes two parameters to measure problem difficulty and presents a linear time sampling algorithm for attention. Empirical results demonstrate that HyperAttention outperforms existing methods, significantly speeding up inference time while maintaining comparable perplexity. The paper concludes by highlighting the scalability limitations of exact computation in attention layers and discussing the potential of HyperAttention to overcome these limitations.

arxiv Preprint - InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists

In this episode we discuss InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists
by Yulu Gan, Sungwoo Park, Alexander Schubert, Anthony Philippakis, Ahmed M. Alaa. The paper proposes a unified language interface for computer vision tasks that allows for task execution through natural language instructions. The approach involves training a text-to-image diffusion model using a multi-modal and multi-task training dataset created through paraphrasing prompt templates. Experimental results show that the model, called InstructCV, performs competitively compared to other vision models and exhibits strong generalization capabilities.

arxiv Preprint - Large Language Models Cannot Self-Correct Reasoning Yet

In this episode we discuss Large Language Models Cannot Self-Correct Reasoning Yet
by Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou. The paper explores the effectiveness of self-correction in Large Language Models (LLMs) for improving the accuracy and appropriateness of generated content. It specifically focuses on the role of self-correction in reasoning tasks. The study reveals that LLMs struggle to self-correct without external feedback and, in some cases, their performance declines after self-correction. Possible areas for further research and practical applications in this domain are also discussed.

arxiv Preprint - Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution

In this episode we discuss Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
by Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, Tim Rocktäschel. The paper presents PROMPTBREEDER, a method for evolving and adapting prompts for Large Language Models (LLMs) in order to enhance their reasoning abilities. It uses an LLM to mutate a population of task-prompts and evaluates their fitness on a training set. The mutation of task-prompts is guided by self-improved mutation-prompts generated by the LLM, leading to improved performance in tasks such as arithmetic, commonsense reasoning, and hate speech classification.

arxiv Preprint - Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation

In this episode we discuss Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
by Eric Zelikman, Eliana Lorch, Lester Mackey, Adam Tauman Kalai. The paper presents a method called Self-Taught Optimizer (STOP) that utilizes a language model to enhance a scaffolding program for solving optimization problems. The language model suggests self-improvement strategies like beam search, genetic algorithms, and simulated annealing. The study demonstrates the success of STOP by comparing the improved program to its original version in various downstream tasks and analyzes the potential risks associated with bypassing a sandbox in the generated code.

arxiv Preprint - Improved Baselines with Visual Instruction Tuning

In this episode we discuss Improved Baselines with Visual Instruction Tuning
by Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee. The authors propose enhancements to the LLaVA framework for large multimodal models (LMMs) with visual instruction tuning. By incorporating CLIP-ViT-L-336px with MLP projection and academic-task-oriented VQA data, they achieve superior performance on multiple benchmarks. These improvements are independent of the LLaVA framework and enable enhanced multimodal understanding with state-of-the-art results using a smaller dataset and shorter training time.

arxiv Preprint - Tree of Thoughts: Deliberate Problem Solving with Large Language Models

In this episode we discuss Tree of Thoughts: Deliberate Problem Solving with Large Language Models
by Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan. The authors of this paper introduce a framework called "Tree of Thoughts" (ToT) to enhance language model inference. The ToT framework allows language models to make deliberate decisions by considering multiple reasoning paths and self-evaluating choices. The authors demonstrate the effectiveness of ToT on three tasks, showing significant improvement in problem-solving abilities compared to traditional prompting methods.

Neurips 2023 - Evaluating Cognitive Maps and Planning in Large Language Models with CogEval

In this episode we discuss Evaluating Cognitive Maps and Planning in Large Language Models with CogEval
by Ida Momennejad, Hosein Hasanbeig, Felipe Vieira, Hiteshi Sharma, Robert Osazuwa Ness, Nebojsa Jojic, Hamid Palangi, Jonathan Larson. The paper presents CogEval, a protocol designed to evaluate the cognitive abilities of Large Language Models (LLMs). The authors note the lack of rigorous evaluation in previous studies claiming human-level cognitive abilities in LLMs and propose CogEval as a framework for systematic evaluation. They apply CogEval to assess the cognitive maps and planning skills of eight different LLMs, finding that while they perform well in simpler planning tasks, there are significant failure modes such as hallucinations and being trapped in loops, indicating a lack of understanding of underlying cognitive structures.

ICCV 2023 - Diffusion Models as Masked Autoencoders

In this episode we discuss Diffusion Models as Masked Autoencoders
by Chen Wei, Karttikeya Mangalam, Po-Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, Christoph Feichtenhofer. The authors present a method called Diffusion Models as Masked Autoencoders (DiffMAE) that combines generative pre-training with diffusion models for visual data. They show that DiffMAE can be a strong initialization for recognition tasks, perform high-quality image inpainting, and achieve state-of-the-art classification accuracy for video. The paper emphasizes the need to consider the specific challenges and requirements of downstream tasks when using generative pre-training.