AI Breakdown

ArXiv Preprint - S-LoRA: Serving Thousands of Concurrent LoRA Adapters

In this episode we discuss S-LoRA: Serving Thousands of Concurrent LoRA Adapters
by Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica. The paper introduces S-LoRA, a system for efficiently serving a large number of Low-Rank Adaptation (LoRA) language model adapters by storing them in memory and using optimized memory management and computation strategies. S-LoRA utilizes Unified Paging for managing memory and custom CUDA kernels for improved tensor parallelism, resulting in up to 4 times higher throughput and serving capacity for thousands of adapters on a single or multiple GPUs compared to current state-of-the-art libraries. The system allows for scalable and customized fine-tuning services, and the authors have made their code publicly available.

ArXiv Preprint - Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities

In this episode we discuss Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities
by AJ Piergiovanni, Isaac Noble, Dahun Kim, Michael S. Ryoo, Victor Gomes, Anelia Angelova. The paper presents Mirasol3B, a multimodal model that handles the disparate natures of video, audio, and text modalities through separate autoregressive components, dividing the process according to the modalities' distinct characteristics. It introduces a Combiner mechanism to manage large volumes of audio and video data by partitioning input sequences into snippets and learning compact representations that capture temporal dependencies. This innovative approach achieves superior performance on multimodal benchmarks while maintaining computational efficiency compared to larger models.

Arxiv Preprint - LCM-LoRA: A Universal Stable-Diffusion Acceleration Module

In this episode we discuss LCM-LoRA: A Universal Stable-Diffusion Acceleration Module by Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, Hang Zhao (Project page). The paper discusses the advancements in Latent Consistency Models (LCMs), which have shown great efficiency in text-to-image generation by being distilled from larger latent diffusion models, requiring only about 32 training hours on A100 GPUs. The research has successfully extended LCMs to work with larger models like Stable-Diffusion, resulting in higher-quality images and reduced memory usage through LoRA distillation. Additionally, the paper introduces LCM-LoRA, a universal acceleration module that can enhance various Stable-Diffusion models without additional training, outperforming traditional numerical solvers with its strong generalization capabilities.

ArXiv Preprint - Fine-tuning Language Models for Factuality

In this episode we discuss Fine-tuning Language Models for Factuality
by Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, Chelsea Finn. The paper presents a method to improve the factual accuracy of large pre-trained language models (LLMs) without human fact-checking. By utilizing recent advancements in natural language processing (NLP), such as judging the factuality of generated text and optimizing model responses through preference rankings, the authors fine-tuned models to reduce errors in open-ended text generation. Their approach, tested on the Llama-2 model, achieved significant reductions in factual error rates when generating biographies and answering medical questions, highlighting the potential for more reliable automated content generation.

arxiv preprint - Language Models can be Logical Solvers

In this episode we discuss Language Models can be Logical Solvers
by Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, Weizhu Chen. The paper presents LOGIPT, a new language model designed to tackle complex logical reasoning by directly mimicking the reasoning process of logical solvers, which avoids errors caused by parsing natural language into symbolic representations. LOGIPT is fine-tuned using a dataset that captures the hidden reasoning steps of deductive solvers, ensuring strict adherence to solver syntax and grammar. The model's performance surpasses that of existing solver-augmented language models and few-shot prompting techniques on benchmark deductive reasoning datasets.

ArXiv Preprint - Prompt Engineering a Prompt Engineer

In this episode we discuss Prompt Engineering a Prompt Engineer
by Qinyuan Ye, Maxamed Axmed, Reid Pryzant, Fereshte Khani. The paper presents PE2, an advanced method for automatically engineering prompts for large language models (LLMs), enabling them to perform better at complex tasks. By incorporating elements like a step-by-step reasoning template and verbalized optimization concepts (akin to batch size and momentum), PE2 significantly improves LLMs' task performance, surpassing previous methods on various datasets. The versatility and effectiveness of PE2 are demonstrated through successful applications across different benchmarks, including the Instruction Induction benchmark and real-world industrial prompts, with the method showing a strong ability to refine and correct existing prompts.

arxiv preprint - CogVLM: Visual Expert for Pretrained Language Models

In this episode we discuss CogVLM: Visual Expert for Pretrained Language Models
by Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, Jie Tang. CogVLM is an open-source visual language foundation model that significantly improves the integration of vision and language by incorporating a trainable visual expert module within a pre-trained language model's attention and feed-forward layers. Unlike other models, CogVLM deeply fuses visual and language features without losing any natural language processing capabilities. It delivers state-of-the-art results on several cross-modal benchmarks and is competitive on others, with resources and code accessible publicly.

ArXiv Preprint - De-Diffusion Makes Text a Strong Cross-Modal Interface

In this episode we discuss De-Diffusion Makes Text a Strong Cross-Modal Interface
by Chen Wei, Chenxi Liu, Siyuan Qiao, Zhishuai Zhang, Alan Yuille, Jiahui Yu. The paper introduces De-Diffusion, a new approach that uses text to represent images. An autoencoder is used to transform an image into text, which can be reconstructed back into the original image using a pre-trained text-to-image diffusion model. The De-Diffusion text representation of images is shown to be accurate and comprehensive, making it compatible with various multi-modal tasks and achieving state-of-the-art performance on vision-language tasks.

ArXiv Preprint - E3 TTS: Easy End-to-End Diffusion-based Text to Speech

In this episode we discuss E3 TTS: Easy End-to-End Diffusion-based Text to Speech
by Yuan Gao, Nobuyuki Morioka, Yu Zhang, Nanxin Chen. The paper introduces Easy End-to-End Diffusion-based Text to Speech (E3 TTS), an innovative text-to-speech model that converts text to audio using a diffusion process without the need for intermediate representations or alignment information. E3 TTS functions through iterative refinement directly from plain text to audio waveform, supporting flexible latent structures that enable zero-shot tasks like editing. The model has been tested and offers high-fidelity audio generation, comparable to the performance of advanced neural TTS systems, with samples available online for evaluation.

ArXiv Preprint - Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges

In this episode we discuss Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges
by Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, Huaxiu Yao. The study introduces the Bingo benchmark to analyze hallucination behavior in GPT-4V(ision), a model processing both visual and textual data. Hallucinations, categorized as either bias or interference, reveal that GPT-4V(ision) prefers Western-centric images and is sensitive to how questions and images are presented, with established mitigation strategies proving ineffective. The findings expose similar issues in other leading visual-language models, suggesting an industry-wide challenge that necessitates novel solutions.