AI Breakdown

arxiv Preprint - Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition

In this episode we discuss Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
by Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen, Prashanth G. Shivakumar, Yile Gu, Sungho Ryu, Roger Ren, Qi Luo, Aditya Gourav, I-Fan Chen, Yi-Chieh Liu, Tuan Dinh, Ankur Gandhe, Denis Filimonov, Shalini Ghosh, Andreas Stolcke, Ariya Rastow, Ivan Bulyko. The paper presents a low-rank adaptation method called LoRB for training neural language models. LoRB uses low-rank decomposition to adapt a pretrained model to new domains with fewer parameters. The experimental results demonstrate that LoRB achieves faster training times while maintaining performance on the target domain.

arxiv Preprint - DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

In this episode we discuss DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
by Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Rajbhandari, Yuxiong He. DeepSpeed-Ulysses is a methodology for efficient and scalable training of large language models with long sequence lengths. It addresses the limitations of existing sequence parallelism approaches by partitioning input data and using efficient all-to-all collective communication for attention computation. Experimental evaluations show that DeepSpeed-Ulysses trains 2.5 times faster with sequence lengths four times longer than existing methods, highlighting its importance for generative AI and AI for science.

arxiv Preprint - VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning

In this episode we discuss VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning
by Han Lin, Abhay Zala, Jaemin Cho, Mohit Bansal. The paper presents VIDEODIRECTORGPT, a framework for generating multi-scene videos with consistency using large language models. It consists of a video planner LLM (GPT-4) that expands a text prompt into a "video plan" and a video generator called Layout2Vid that creates the videos while maintaining spatial and temporal consistency. The framework achieves competitive performance in single-scene video generation and allows for dynamic control of layout guidance strength and user-provided images.

arxiv Preprint - PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

In this episode we discuss PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
by Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li. The paper presents a training method called PoSE for adapting large language models to longer context windows. It addresses the challenge of extending the context window of pre-trained models without disrupting performance. The method simulates long inputs using a fixed context window with manipulated position indices, reducing memory and time overhead while maintaining performance.

arxiv Preprint - Summarization is (Almost) Dead

In this episode we discuss Summarization is (Almost) Dead
by Xiao Pu, Mingqi Gao, Xiaojun Wan. The paper investigates the capabilities of large language models (LLMs) in summary generation. Through new datasets and human evaluation experiments, the authors find that LLM-generated summaries are preferred by evaluators compared to human-written summaries and fine-tuned model summaries. LLM-generated summaries exhibit improved factual consistency and fewer instances of extrinsic hallucinations, leading the authors to suggest that traditional text summarization methods may no longer be necessary. However, the authors emphasize the need for further exploration in areas such as dataset creation and evaluation methods.

arxiv Preprint - LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent

In this episode we discuss LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent
by Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F. Fouhey, Joyce Chai. The paper introduces a new approach called LLM-Grounder for grounding 3D visual scenes using natural language queries. It utilizes a Large Language Model (LLM) to break down complex queries and a visual grounding tool to identify objects in the scene. The method does not require labeled training data and achieves state-of-the-art accuracy on the ScanRefer benchmark.

Neurips 2023 spotlight - Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systems

In this episode we discuss Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systems
by Benjamin Coleman, Wang-Cheng Kang, Matthew Fahrbach, Ruoxi Wang, Lichan Hong, Ed H. Chi, Derek Zhiyuan Cheng. The paper introduces a framework called Feature Multiplexing, which allows for the use of a single representation space across multiple categorical features in web-scale machine learning systems. This framework addresses the high parameter count issue that arises from representing each feature value as a d-dimensional embedding. The paper also proposes a practical approach called Unified Embedding, which simplifies feature configuration, adapts to dynamic data distributions, and is compatible with modern hardware. The effectiveness of Unified Embedding is demonstrated in improving offline and online metrics across various web-scale systems.

arxiv Preprint - Chain-of-Verification Reduces Hallucination in Large Language Models

In this episode we discuss Chain-of-Verification Reduces Hallucination in Large Language Models
by Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, Jason Weston. The paper proposes the Chain-of-Verification (COVE) method to address the issue of factual hallucination in large language models. COVE involves generating an initial response, planning independent fact-checking questions, and generating a final verified response. The experiments demonstrate that COVE reduces hallucinations in various tasks and variations of the method further improve performance.

arxiv Preprint - Language Modeling Is Compression

In this episode we discuss Language Modeling Is Compression
by Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, Joel Veness. The authors argue that large language models can be seen as powerful compressors due to their predictive capabilities. They demonstrate that these models outperform specific compressors like PNG and FLAC. The paper explores the implications of the prediction-compression equivalence and discusses the use of any compressor to build a conditional generative model.

arxiv Preprint - From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting

In this episode we discuss From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting
by Griffin Adams, Alexander Fabbri, Faisal Ladhak, Eric Lehman, Noémie Elhadad. The paper introduces a method called "Chain of Density" (CoD) for generating summaries with varying levels of information density. Using GPT-4, the authors generate entity-sparse summaries and then iteratively add missing salient entities without increasing the length. CoD summaries are found to be more abstractive, exhibit more fusion, and have less lead bias compared to GPT-4 summaries generated by a vanilla prompt, with human preference favoring denser GPT-4 summaries.