AI Breakdown

arxiv Preprint - Conditional Diffusion Distillation

In this episode we discuss Conditional Diffusion Distillation
by Kangfu Mei, Mauricio Delbracio, Hossein Talebi, Zhengzhong Tu, Vishal M. Patel, Peyman Milanfar. The authors of this paper propose a new method called conditional distillation to speed up the sampling time of diffusion models in text-to-image generation. The method incorporates image conditions to enhance the diffusion priors and enable conditional sampling with fewer steps. The proposed method simplifies the distillation process by directly distilling the unconditional pre-training in a single stage through joint-learning, and it outperforms existing distillation techniques in terms of sampling time.

arxiv Preprint - Enable Language Models to Implicitly Learn Self-Improvement From Data

In this episode we discuss Enable Language Models to Implicitly Learn Self-Improvement From Data
by Ziqi Wang, Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, Heng Ji. The paper introduces a framework called ImPlicit Self-ImprovemenT (PIT) that allows large language models (LLMs) to learn self-improvement from data. PIT learns the improvement goal from human preference data without requiring explicit rubrics, making it more efficient and effective compared to previous approaches that rely on explicit inputs. Experimental results show that PIT outperforms prompting-based methods in enhancing LLM performance.

arxiv Preprint - Efficient Streaming Language Models with Attention Sinks

In this episode we discuss Efficient Streaming Language Models with Attention Sinks
by Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis. The paper proposes StreamingLLM, a framework that allows Large Language Models (LLMs) to generalize to infinite sequence length without fine-tuning. By observing the phenomenon of attention sink, where initial tokens have a significant impact on performance, the authors show that caching the Key and Value states of these tokens enhances the efficiency and stability of window attention. The authors demonstrate that StreamingLLM outperforms the sliding window recomputation baseline in streaming applications with a speedup of up to 22.2x.

Neurips 2023 - PuzzleFusion: Unleashing the Power of Diffusion Models for Spatial Puzzle Solving

In this episode we discuss PuzzleFusion: Unleashing the Power of Diffusion Models for Spatial Puzzle Solving
by Sepidehsadat Hosseini, Mohammad Amin Shabani, Saghar Irandoust, Yasutaka Furukawa. The paper introduces PuzzleFusion, a neural architecture based on Diffusion Models for spatial puzzle solving. It focuses on jigsaw puzzle solving and room arrangement tasks, using new datasets including synthetic ones generated by Voronoi diagrams and a real dataset from MagicPlan. The paper shows that PuzzleFusion outperforms other methods in both qualitative and quantitative evaluations.

arxiv Preprint - Vision Transformers Need Registers

In this episode we discuss Vision Transformers Need Registers
by Timothée Darcet, Maxime Oquab, Julien Mairal, Piotr Bojanowski. The paper discusses a solution to artifacts found in the feature maps of Vision Transformers (ViT) in low-informative background areas of images. By adding additional tokens called "registers" to the input sequence, the feature maps and attention maps are improved, leading to better visual processing. This solution is effective for both supervised and self-supervised ViT models and achieves state-of-the-art performance on self-supervised visual models. Additionally, the use of registers enables object discovery methods with larger models.

arxiv Preprint - VPA: Fully Test-Time Visual Prompt Adaptation

In this episode we discuss VPA: Fully Test-Time Visual Prompt Adaptation
by Jiachen Sun, Mark Ibrahim, Melissa Hall, Ivan Evtimov, Z. Morley Mao, Cristian Canton Ferrer, Caner Hazirbas. The paper presents Visual Prompt Adaptation (VPA), a framework that extends prompt tuning to visual recognition tasks. VPA allows for test-time adaptation without source-domain information and improves out-of-distribution generalization, corruption robustness, domain adaptation, and zero-shot recognition. Experimental results show improvements of 3.3% in OOD generalization, 6.5% in corruption robustness, and 5.2% in domain adaptation.

arxiv Preprint - Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition

In this episode we discuss Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
by Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen, Prashanth G. Shivakumar, Yile Gu, Sungho Ryu, Roger Ren, Qi Luo, Aditya Gourav, I-Fan Chen, Yi-Chieh Liu, Tuan Dinh, Ankur Gandhe, Denis Filimonov, Shalini Ghosh, Andreas Stolcke, Ariya Rastow, Ivan Bulyko. The paper presents a low-rank adaptation method called LoRB for training neural language models. LoRB uses low-rank decomposition to adapt a pretrained model to new domains with fewer parameters. The experimental results demonstrate that LoRB achieves faster training times while maintaining performance on the target domain.

arxiv Preprint - DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

In this episode we discuss DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
by Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Rajbhandari, Yuxiong He. DeepSpeed-Ulysses is a methodology for efficient and scalable training of large language models with long sequence lengths. It addresses the limitations of existing sequence parallelism approaches by partitioning input data and using efficient all-to-all collective communication for attention computation. Experimental evaluations show that DeepSpeed-Ulysses trains 2.5 times faster with sequence lengths four times longer than existing methods, highlighting its importance for generative AI and AI for science.

arxiv Preprint - VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning

In this episode we discuss VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning
by Han Lin, Abhay Zala, Jaemin Cho, Mohit Bansal. The paper presents VIDEODIRECTORGPT, a framework for generating multi-scene videos with consistency using large language models. It consists of a video planner LLM (GPT-4) that expands a text prompt into a "video plan" and a video generator called Layout2Vid that creates the videos while maintaining spatial and temporal consistency. The framework achieves competitive performance in single-scene video generation and allows for dynamic control of layout guidance strength and user-provided images.

arxiv Preprint - PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

In this episode we discuss PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
by Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li. The paper presents a training method called PoSE for adapting large language models to longer context windows. It addresses the challenge of extending the context window of pre-trained models without disrupting performance. The method simulates long inputs using a fixed context window with manipulated position indices, reducing memory and time overhead while maintaining performance.