AI Breakdown

arxiv Preprint - eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models

In this episode we discuss eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models
by Minsik Cho, Keivan A. Vahid, Qichen Fu, Saurabh Adya, Carlo C Del Mundo, Mohammad Rastegari, Devang Naik, Peter Zatloukal. The paper proposes eDKM, a memory-efficient implementation of weight clustering for large language models (LLMs). LLMs have high performance but require compression for storage-limited devices. The eDKM technique reduces the memory footprint of Differentiable KMeans Clustering (DKM) by orders of magnitudes, allowing for efficient LLM compression with good accuracy.

arxiv Preprint - Link-Context Learning for Multimodal LLMs

In this episode we discuss Link-Context Learning for Multimodal LLMs
by Yan Tai, Weichen Fan, Zhao Zhang, Feng Zhu, Rui Zhao, Ziwei Liu. The paper presents a method called link-context learning (LCL) that enhances the learning abilities of Multimodal Large Language Models (MLLMs). LCL aims to enable MLLMs to recognize new images and understand unfamiliar concepts without the need for training. It focuses on strengthening the causal relationship between the support set and the query set to help MLLMs discern analogies and causal associations between data points. Experimental results demonstrate that the proposed LCL-MLLM performs better in link-context learning compared to traditional MLLMs.

arxiv Preprint - ProPainter: Improving Propagation and Transformer for Video Inpainting

In this episode we discuss ProPainter: Improving Propagation and Transformer for Video Inpainting
by Shangchen Zhou, Chongyi Li, Kelvin C. K. Chan, Chen Change Loy. The paper discusses the limitations of existing approaches to video inpainting, specifically flow-based propagation and spatiotemporal Transformer methods, due to spatial misalignment and limited temporal range. To address these challenges, the authors propose ProPainter, a framework that combines dual-domain propagation with image and feature warping for reliable global correspondences. They also introduce a mask-guided sparse video Transformer to enhance efficiency. ProPainter achieves superior results with a 1.46 dB improvement in PSNR while maintaining efficiency, making it a valuable tool for video inpainting applications.

arxiv Preprint - Large Language Models as Optimizers

In this episode we discuss Large Language Models as Optimizers
by Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen. The paper introduces Optimization by PROmpting (OPRO), a method that uses large language models as optimizers in the absence of gradients. OPRO utilizes natural language descriptions of the optimization task to generate new solutions in each step, which are evaluated and added to the prompt for subsequent steps. Experimental results demonstrate that prompts optimized by OPRO outperform human-designed prompts on various tasks, with performance improvements of up to 8% on GSM8K and up to 50% on Big-Bench Hard tasks.

arxiv Preprint - Active Retrieval Augmented Generation

In this episode we discuss Active Retrieval Augmented Generation
by Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, Graham Neubig. The paper presents FLARE, a method that improves the performance of language models by incorporating retrieval of information from external knowledge resources. Unlike existing retrieval-augmented models, FLARE actively decides when and what to retrieve throughout the generation process, anticipating future content using sentence predictions. The authors demonstrate the effectiveness of FLARE in four knowledge-intensive generation tasks, showing its superiority or comparability to baseline models and its ability to improve the accuracy and reliability of generated text.

arxiv Preprint - Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation

In this episode we discuss Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
by Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, Qifeng Chen. This paper presents Animate-A-Story, a framework for generating storytelling videos by customizing existing video clips. The framework includes two modules: Motion Structure Retrieval and Structure-Guided Text-to-Video Synthesis. The first module retrieves relevant video clips based on query texts, while the second module generates coherent videos guided by motion structure and text prompts. The approach proposed in the paper surpasses existing baselines in terms of visual consistency and performance.

arxiv Preprint - FACET: Fairness in Computer Vision Evaluation Benchmark

In this episode we discuss FACET: Fairness in Computer Vision Evaluation Benchmark
by Laura Gustafson, Chloe Rolland, Nikhila Ravi, Quentin Duval, Aaron Adcock, Cheng-Yang Fu, Melissa Hall, Candace Ross. The paper introduces a new benchmark called FACET, which measures performance disparities in computer vision models across different attributes such as gender and skin tone. It consists of a large evaluation set of 32k images and expert reviewers manually annotated person-related attributes. The benchmark revealed performance disparities across demographic attributes and aims to contribute to the development of fairer and more robust vision models.

arxiv Preprint - Baseline Defenses for Adversarial Attacks Against Aligned Language Models

In this episode we discuss Baseline Defenses for Adversarial Attacks Against Aligned Language Models
by Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, Tom Goldstein. The paper discusses the security vulnerabilities of Large Language Models (LLMs) and explores defense strategies against adversarial attacks. Three types of defenses are considered: detection, input preprocessing, and adversarial training. The study emphasizes the effectiveness of filtering and preprocessing in LLM defenses and highlights the need for further understanding of LLM security as these models become more prevalent.

ICCV 2023 - Verbs in Action: Improving verb understanding in video-language models

In this episode we discuss Verbs in Action: Improving verb understanding in video-language models
by Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, Cordelia Schmid. The paper proposes a Verb-Focused Contrastive (VFC) framework to address the limited understanding of verbs in video-language models. The framework utilizes pre-trained large language models (LLMs) to generate hard negative captions by changing only the verb while keeping the context intact. The method achieves state-of-the-art results in zero-shot performance on three downstream tasks: video-text matching, video question-answering, and video classification.

arxiv Preprint - RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

In this episode we discuss RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
by Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, Abhinav Rastogi. The paper introduces a new technique called RL from AI Feedback (RLAIF) as a solution to the scalability limitations of reinforcement learning from human feedback (RLHF). RLAIF involves using a large language model (LLM) to label preferences instead of relying on humans. The study compared RLAIF and RLHF on the task of summarization and found that both techniques resulted in similar improvements over a baseline model. Human evaluators preferred both RLAIF and RLHF summaries over the baseline model, suggesting that RLAIF can achieve human-level performance while overcoming the scalability limitations of RLHF.