AI Breakdown

arxiv Preprint - Active Retrieval Augmented Generation

In this episode we discuss Active Retrieval Augmented Generation
by Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, Graham Neubig. The paper presents FLARE, a method that improves the performance of language models by incorporating retrieval of information from external knowledge resources. Unlike existing retrieval-augmented models, FLARE actively decides when and what to retrieve throughout the generation process, anticipating future content using sentence predictions. The authors demonstrate the effectiveness of FLARE in four knowledge-intensive generation tasks, showing its superiority or comparability to baseline models and its ability to improve the accuracy and reliability of generated text.

arxiv Preprint - Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation

In this episode we discuss Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
by Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, Qifeng Chen. This paper presents Animate-A-Story, a framework for generating storytelling videos by customizing existing video clips. The framework includes two modules: Motion Structure Retrieval and Structure-Guided Text-to-Video Synthesis. The first module retrieves relevant video clips based on query texts, while the second module generates coherent videos guided by motion structure and text prompts. The approach proposed in the paper surpasses existing baselines in terms of visual consistency and performance.

arxiv Preprint - FACET: Fairness in Computer Vision Evaluation Benchmark

In this episode we discuss FACET: Fairness in Computer Vision Evaluation Benchmark
by Laura Gustafson, Chloe Rolland, Nikhila Ravi, Quentin Duval, Aaron Adcock, Cheng-Yang Fu, Melissa Hall, Candace Ross. The paper introduces a new benchmark called FACET, which measures performance disparities in computer vision models across different attributes such as gender and skin tone. It consists of a large evaluation set of 32k images and expert reviewers manually annotated person-related attributes. The benchmark revealed performance disparities across demographic attributes and aims to contribute to the development of fairer and more robust vision models.

arxiv Preprint - Baseline Defenses for Adversarial Attacks Against Aligned Language Models

In this episode we discuss Baseline Defenses for Adversarial Attacks Against Aligned Language Models
by Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, Tom Goldstein. The paper discusses the security vulnerabilities of Large Language Models (LLMs) and explores defense strategies against adversarial attacks. Three types of defenses are considered: detection, input preprocessing, and adversarial training. The study emphasizes the effectiveness of filtering and preprocessing in LLM defenses and highlights the need for further understanding of LLM security as these models become more prevalent.

ICCV 2023 - Verbs in Action: Improving verb understanding in video-language models

In this episode we discuss Verbs in Action: Improving verb understanding in video-language models
by Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, Cordelia Schmid. The paper proposes a Verb-Focused Contrastive (VFC) framework to address the limited understanding of verbs in video-language models. The framework utilizes pre-trained large language models (LLMs) to generate hard negative captions by changing only the verb while keeping the context intact. The method achieves state-of-the-art results in zero-shot performance on three downstream tasks: video-text matching, video question-answering, and video classification.

arxiv Preprint - RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

In this episode we discuss RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
by Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, Abhinav Rastogi. The paper introduces a new technique called RL from AI Feedback (RLAIF) as a solution to the scalability limitations of reinforcement learning from human feedback (RLHF). RLAIF involves using a large language model (LLM) to label preferences instead of relying on humans. The study compared RLAIF and RLHF on the task of summarization and found that both techniques resulted in similar improvements over a baseline model. Human evaluators preferred both RLAIF and RLHF summaries over the baseline model, suggesting that RLAIF can achieve human-level performance while overcoming the scalability limitations of RLHF.

arxiv Preprint - LLM-Rec: Personalized Recommendation via Prompting Large Language Models

In this episode we discuss LLM-Rec: Personalized Recommendation via Prompting Large Language Models
by Hanjia Lyu, Song Jiang, Hanqing Zeng, Qifan Wang, Si Zhang, Ren Chen, Chris Leung, Jiajie Tang, Yinglong Xia, Jiebo Luo. The paper examines different prompting strategies for improving personalized recommendation performance using large language models (LLMs) through input augmentation. The proposed approach, LLM-Rec, incorporates four prompting strategies, demonstrating that using LLM-generated augmented input text enhances recommendation performance. The recommendation-driven and engagement-guided prompting strategies highlight the LLM's comprehension of global and local item characteristics, underscoring the necessity of diverse prompts and input augmentation techniques to enhance recommendation capabilities with LLMs.

ICCV 2023 - Robust Monocular Depth Estimation under Challenging Conditions

In this episode we discuss Robust Monocular Depth Estimation under Challenging Conditions
by Stefano Gasperini, Nils Morbitzer, HyunJun Jung, Nassir Navab, Federico Tombari. The paper addresses the limitations of existing monocular depth estimation methods in challenging lighting and weather conditions. The authors propose md4all, a simple and reliable solution that can handle diverse conditions without modification at inference time. The approach involves generating complex training samples, training the model using self- or full-supervision, and computing standard losses on the original images. Extensive experiments on public datasets demonstrate the effectiveness of the approach, surpassing previous works in both standard and challenging conditions.

arxiv Preprint - LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models

In this episode we discuss LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models
by Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang. The paper introduces LM-Infinite as a solution to the challenge of length generalization in Large Language Models (LLMs). Existing methods for handling longer sequences are resource-intensive and time-consuming, prompting the need for a simpler approach. LM-Infinite employs a Λ-shaped attention mask and a distance limit, requiring no parameter updates or learning. The technique consistently generates fluent texts and performs well on downstream tasks even with input sequences exceeding training lengths.

arxiv Preprint - Llama 2: Open Foundation and Fine-Tuned Chat Models

In this episode we discuss Llama 2: Open Foundation and Fine-Tuned Chat Models
by Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom. The paper introduces Llama 2, a collection of pretrained and fine-tuned large language models optimized for dialogue purposes. Ranging from 7 billion to 70 billion parameters, the Llama 2-Chat models surpass existing open-source chat models in performance across different benchmarks. The authors also conduct human evaluations, indicating that their models could be viable alternatives to closed-source models, and provide detailed insights into their fine-tuning process and safety enhancements.