AI Breakdown

arxiv preprint - SliceGPT: Compress Large Language Models by Deleting Rows and Columns

In this episode, we discuss SliceGPT: Compress Large Language Models by Deleting Rows and Columns by Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, James Hensman. The paper introduces SliceGPT, a new method for post-training sparsification of large language models that reduces their size and computational requirements by replacing weight matrices with smaller ones and thus cutting down the embedding dimension. This approach can eliminate up to 25% of parameters in certain models with minimal loss in task performance. The authors highlight computational invariance in transformer networks, which SliceGPT utilizes, and demonstrate that models can run faster and on fewer GPUs, all without additional optimization, providing code for their method at a provided GitHub repository.

arxiv preprint - Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video

In this episode, we discuss Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video by Shashanka Venkataramanan, Mamshad Nayeem Rizve, João Carreira, Yuki M. Asano, Yannis Avrithis. The paper presents two innovations in self-supervised learning: a new dataset called "Walking Tours," which features high-resolution, long duration, first-person videos ideal for self-supervision, and a novel pretraining method called DORA which uses transformer cross-attention to track and learn object recognition in videos. This method diverges from adapting image-based pretraining to videos by instead focusing on tracking objects over time. The researchers found that their approach, combining the Walking Tours dataset with DORA, performed comparably to ImageNet on various image and video recognition tasks, showcasing the efficiency of their method.

arxiv preprint - MambaByte: Token-free Selective State Space Model

In this episode, we discuss MambaByte: Token-free Selective State Space Model by Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, Alexander M Rush. "MambaByte, a token-free language model, removes the bias associated with subword tokenization by learning from raw bytes. It capitalizes on the Mamba state space model's adaptability to byte sequences, offering computational efficiency and often outperforming traditional subword Transformers despite the increased sequence length. With linear scaling, MambaByte also achieves faster inference, demonstrating its potential for efficient token-free language modeling."

arxiv preprint - Lumiere: A Space-Time Diffusion Model for Video Generation

In this episode, we discuss Lumiere: A Space-Time Diffusion Model for Video Generation by Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, Inbar Mosseri. The paper presents Lumiere, a novel text-to-video diffusion model capable of generating realistic and coherently moving videos by producing the full temporal sequence in a single pass, using a Space-Time U-Net architecture. Unlike other methods that create videos by interpolating between keyframes, Lumiere ensures global temporal consistency by using spatial and temporal down- and up-sampling. The model shows superior performance in text-to-video generation and is versatile, allowing for content creation tasks such as image-to-video conversion, video inpainting, and stylized video generation.

arxiv preprint - Self-Rewarding Language Models

In this episode, we discuss Self-Rewarding Language Models by Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, Jason Weston. The paper introduces self-rewarding language models (SR-LMs) which generate their own rewards for self-improvement beyond human performance levels. Using a method called Iterative Direct Preference Optimization, SR-LMs can enhance their ability to follow instructions and improve the quality of self-generated rewards through iteration. The authors demonstrate that their approach, when applied to Llama 2 70B, exceeds the performance of other systems on the AlpacaEval 2.0 leaderboard, suggesting potential for models to self-improve continuously.

arxiv preprint - Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data

In this episode, we discuss Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data by Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao. "Depth Anything" is an approach to improve monocular depth estimation by exploiting a massive collection of about 62 million unlabeled images, aiming to extend dataset size and lessen generalization errors without the need for novel technical developments. The model's performance is heightened through strategic data augmentation and the incorporation of semantic knowledge from pre-trained encoders, leading to exceptional zero-shot generalization demonstrated on various public datasets and random images. By additionally fine-tuning with metric depth data, the model sets new benchmarks on NYUv2 and KITTI datasets and enhances the efficacy of a depth-conditioned ControlNet, with all models released for public use.

arxiv preprint - MoVQA: A Benchmark of Versatile Question-Answering for Long-Form Movie Understanding

In this episode, we discuss MoVQA: A Benchmark of Versatile Question-Answering for Long-Form Movie Understanding by Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, Yu Qiao. The newly introduced dataset MoVQA aims to enhance the evaluation of AI systems' understanding of long-form video content, such as movies, addressing the limitations of previous datasets that did not fully capture the complexity and lengthy nature of such content. It challenges AI models with a more realistic range of temporal lengths and multimodal questions to mimic human-level comprehension from a moviegoer's perspective. Initial experiments with MoVQA show that current methods struggle as video and clue lengths increase, indicating substantial room for improvement in long-form video understanding AI research.

arxiv preprint - Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

In this episode, we discuss Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model by Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, Xinggang Wang. The paper introduces a new vision backbone called Vim, which leverages bidirectional Mamba blocks for efficient and effective visual representation learning, sidestepping the need for self-attention mechanisms. Vim incorporates position embeddings for handling the position-sensitivity of visual data and uses state space models to handle global context, leading to better performance on various tasks such as ImageNet classification and COCO object detection, while being more computationally and memory efficient than existing models like DeiT. Tests show that Vim is significantly faster and more memory-efficient, making it a promising candidate for advanced vision backbone algorithms, especially for high-resolution image processing.

arxiv preprint - Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models

In this episode, we discuss Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models by Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, Mor Geva. The paper presents a novel framework named Patchscopes designed to improve understanding of the hidden representations in large language models (LLMs) by using the models themselves to articulate these representations in natural language. Patchscopes integrates and extends existing interpretability techniques, overcoming limitations like the inability to inspect early layers and enhancing expressivity. Beyond reconciling former methods, Patchscopes also enables innovative applications, including having more advanced LLMs explain the workings of simpler ones and facilitating self-correction in complex reasoning tasks.

arxiv preprint - Time Travel in LLMs: Tracing Data Contamination in Large Language Models

In this episode, we discuss Time Travel in LLMs: Tracing Data Contamination in Large Language Models by Shahriar Golchin, Mihai Surdeanu. The paper presents a method to detect test data contamination in large language models by checking if the model's output closely matches specific segments of reference data. This process involves guided instructions using dataset names and partition types, comparing the model's output to reference instances, and assessing partitions based on statistical overlap measures or classification by GPT-4's few-shot in-context learning. The results show high accuracy in identifying contamination, revealing that GPT-4 has been contaminated with certain datasets such as AG News, WNLI, and XSum.