AI Breakdown

ICCV 2023 - Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editing

In this episode we discuss Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editing
by Alberto Baldrati, Davide Morelli, Giuseppe Cartella, Marcella Cornia, Marco Bertini, Rita Cucchiara. This paper introduces a framework for generating fashion images using multimodal prompts such as text, body poses, and garment sketches. The proposed architecture utilizes latent diffusion models, a novel approach in the fashion domain. The effectiveness of the framework is demonstrated through experiments using extended fashion datasets, showing realistic and coherent results.

arxiv Preprint - GPT Can Solve Mathematical Problems Without a Calculator

In this episode we discuss GPT Can Solve Mathematical Problems Without a Calculator
by Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Guo, Jinfeng Bai, Jie Tang. The paper challenges the belief that large language models cannot perform arithmetic operations accurately without calculator tools. The researchers present MathGLM, a 2 billion-parameter language model, which achieves nearly 100% accuracy in multi-digit arithmetic operations, surpassing GPT-4. They demonstrate the model's capability by training it on a dataset containing multi-step arithmetic operations and math problems described in text, and it performs similarly to GPT-4 on a Chinese math problem test set. The results suggest that language models can excel in mathematical problem-solving without the need for calculators, given sufficient training data.

ICCV 2023 - Adding Conditional Control to Text-to-Image Diffusion Models

In this episode we discuss Adding Conditional Control to Text-to-Image Diffusion Models
by Lvmin Zhang, Anyi Rao, Maneesh Agrawala. The paper introduces ControlNet, a neural network architecture that enhances spatial control in text-to-image diffusion models. It incorporates additional images, such as edge maps and human pose skeletons, as conditioning factors to specify desired image composition. ControlNet utilizes pretrained encoding layers and gradually adjusts parameters for improved spatial control. Experimental results with different conditioning controls and datasets demonstrate the effectiveness of ControlNet, suggesting its potential to broaden the applications of image diffusion models.

arxiv Preprint - Neurons in Large Language Models: Dead, N-gram, Positional

In this episode we discuss Neurons in Large Language Models: Dead, N-gram, Positional
by Elena Voita, Javier Ferrando, Christoforos Nalmpantis. In this paper, the authors analyze a family of language models called OPT models and focus on the activation of neurons in the feedforward blocks. They find that there are many inactive "dead" neurons in the early part of the network and that active neurons in this region primarily act as token and n-gram detectors. The authors also identify positional neurons that are activated based on position rather than textual data.

arxiv Preprint - eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models

In this episode we discuss eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models
by Minsik Cho, Keivan A. Vahid, Qichen Fu, Saurabh Adya, Carlo C Del Mundo, Mohammad Rastegari, Devang Naik, Peter Zatloukal. The paper proposes eDKM, a memory-efficient implementation of weight clustering for large language models (LLMs). LLMs have high performance but require compression for storage-limited devices. The eDKM technique reduces the memory footprint of Differentiable KMeans Clustering (DKM) by orders of magnitudes, allowing for efficient LLM compression with good accuracy.

arxiv Preprint - Link-Context Learning for Multimodal LLMs

In this episode we discuss Link-Context Learning for Multimodal LLMs
by Yan Tai, Weichen Fan, Zhao Zhang, Feng Zhu, Rui Zhao, Ziwei Liu. The paper presents a method called link-context learning (LCL) that enhances the learning abilities of Multimodal Large Language Models (MLLMs). LCL aims to enable MLLMs to recognize new images and understand unfamiliar concepts without the need for training. It focuses on strengthening the causal relationship between the support set and the query set to help MLLMs discern analogies and causal associations between data points. Experimental results demonstrate that the proposed LCL-MLLM performs better in link-context learning compared to traditional MLLMs.

arxiv Preprint - ProPainter: Improving Propagation and Transformer for Video Inpainting

In this episode we discuss ProPainter: Improving Propagation and Transformer for Video Inpainting
by Shangchen Zhou, Chongyi Li, Kelvin C. K. Chan, Chen Change Loy. The paper discusses the limitations of existing approaches to video inpainting, specifically flow-based propagation and spatiotemporal Transformer methods, due to spatial misalignment and limited temporal range. To address these challenges, the authors propose ProPainter, a framework that combines dual-domain propagation with image and feature warping for reliable global correspondences. They also introduce a mask-guided sparse video Transformer to enhance efficiency. ProPainter achieves superior results with a 1.46 dB improvement in PSNR while maintaining efficiency, making it a valuable tool for video inpainting applications.

arxiv Preprint - Large Language Models as Optimizers

In this episode we discuss Large Language Models as Optimizers
by Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen. The paper introduces Optimization by PROmpting (OPRO), a method that uses large language models as optimizers in the absence of gradients. OPRO utilizes natural language descriptions of the optimization task to generate new solutions in each step, which are evaluated and added to the prompt for subsequent steps. Experimental results demonstrate that prompts optimized by OPRO outperform human-designed prompts on various tasks, with performance improvements of up to 8% on GSM8K and up to 50% on Big-Bench Hard tasks.

arxiv Preprint - Active Retrieval Augmented Generation

In this episode we discuss Active Retrieval Augmented Generation
by Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, Graham Neubig. The paper presents FLARE, a method that improves the performance of language models by incorporating retrieval of information from external knowledge resources. Unlike existing retrieval-augmented models, FLARE actively decides when and what to retrieve throughout the generation process, anticipating future content using sentence predictions. The authors demonstrate the effectiveness of FLARE in four knowledge-intensive generation tasks, showing its superiority or comparability to baseline models and its ability to improve the accuracy and reliability of generated text.

arxiv Preprint - Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation

In this episode we discuss Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
by Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, Qifeng Chen. This paper presents Animate-A-Story, a framework for generating storytelling videos by customizing existing video clips. The framework includes two modules: Motion Structure Retrieval and Structure-Guided Text-to-Video Synthesis. The first module retrieves relevant video clips based on query texts, while the second module generates coherent videos guided by motion structure and text prompts. The approach proposed in the paper surpasses existing baselines in terms of visual consistency and performance.