AI Breakdown

arxiv preprint - Can Large Language Models Understand Context?

In this episode, we discuss Can Large Language Models Understand Context? by Yilun Zhu, Joel Ruben Antony Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, Bo-Hsiang Tseng. The paper introduces a novel benchmark consisting of four tasks and nine datasets aimed at rigorously evaluating Large Language Models' (LLMs) ability to understand context. The authors find that while pre-trained dense models show some competency, they are less adept at grasping nuanced contextual information compared to fine-tuned state-of-the-art models. Additionally, the research reveals that applying 3-bit post-training quantization to these models results in decreased performance on the benchmark, with an in-depth analysis provided to explain the findings.

arxiv preprint - Long Story Short: a Summarize-then-Search Method for Long Video Question Answering

In this episode, we discuss Long Story Short: a Summarize-then-Search Method for Long Video Question Answering by Jiwan Chung, Youngjae Yu. The paper presents "Long Story Short," a new framework for video question-answering (QA) tasks that involves summarizing long multimodal narratives (like movies or dramas) into brief plots. This summary is then used to find video segments pertinent to specific questions. The paper also introduces an enhancement called CLIPCheck for improved visual matching, and their model significantly surpasses existing supervised models in performance, demonstrating the effectiveness of zero-shot QA for lengthy video content.

arxiv preprint - System 2 Attention (is something you might need too)

In this episode, we discuss System 2 Attention (is something you might need too) by Jason Weston, Sainbayar Sukhbaatar. The paper introduces System 2 Attention (S2A), an approach that improves Transformer-based Large Language Models by regenerating input contexts to focus on relevant information before processing, thereby enhancing the generation of the next token. S2A was created to address the problem of standard soft attention mechanisms that often integrate distracting information into outputs. In testing, S2A demonstrated superior performance by producing more factual, objective, and less biased responses on tasks such as question answering, math word problems, and longform content generation.

arxiv preprint - DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

In this episode, we discuss DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models by Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo. The paper presents DeepSeekMath 7B, an advanced language model trained on 120 billion math-related tokens to improve mathematical reasoning. The model scores 51.7% on the MATH benchmark, and by using an approach called self-consistency, it reaches 60.9%, approaching the results of state-of-the-art models like Gemini-Ultra and GPT-4 without external aids. The success of DeepSeekMath is attributed to the use of an extensive web data collection and a novel optimization algorithm called Group Relative Policy Optimization (GRPO) that improves math reasoning while being memory-efficient.

arxiv preprint - KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

In this episode, we discuss KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization by Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami. The paper introduces KVQuant, a novel method for reducing memory usage in Large Language Models (LLMs) by efficiently quantizing key-value (KV) cache activations to sub-4-bit precision. KVQuant improves the accuracy of ultra-low precision representations through techniques such as per-channel and pre-rotary positional embedding quantization, non-uniform datatypes, per-vector dense-and-sparse quantization, and normalization of quantization centroids. The application of KVQuant results in negligible performance loss, increased maximum context lengths on GPUs, and a speedup in computation, with the code made available for public use.

arxiv preprint - Language Model Inversion

In this episode, we discuss Language Model Inversion by John X. Morris, Wenting Zhao, Justin T. Chiu, Vitaly Shmatikov, Alexander M. Rush. The paper explores language model inversion, revealing that the probabilities given by language models for the next token can reveal significant details about the preceding text. The authors introduce a technique to reconstruct hidden prompts solely based on the model's probability outputs, even without full access to all token predictions. They demonstrate the effectiveness of this method on Llama-2 7b, achieving 59 BLEU score, 78 token-level F1, and an exact recovery of 27% of the prompts.

arxiv preprint - Tree Prompting: Efficient Task Adaptation without Fine-Tuning

In this episode, we discuss Tree Prompting: Efficient Task Adaptation without Fine-Tuning by John X. Morris, Chandan Singh, Alexander M. Rush, Jianfeng Gao, Yuntian Deng. Tree Prompting is a novel method for interacting with smaller language models (LMs) that creates a decision tree of prompts to guide the model's responses. This technique significantly enhances accuracy on tasks compared to traditional prompting methods and rivals the performance of gradient-based fine-tuning. Additionally, some versions of Tree Prompting provide insights into the LM's decision-making process.

arxiv preprint - Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens

In this episode, we discuss Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens by Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, Hannaneh Hajishirzi. The paper introduces an improved n-gram language model named "Infini-gram," which scales to 1.4 trillion tokens and has the capacity to use n-grams of arbitrary length. The authors develop a suffix array-powered engine called infini-gram that calculates probabilities for these extended n-grams quickly, without the need for pre-computing count tables. This new framework demonstrated its utility by enhancing the performance of neural large language models and revealing limitations in machine-generated text, and the authors have made the engine available as an open-source tool for further research.

arxiv preprint - Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

In this episode, we discuss Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning by Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, Lijuan Wang. This paper introduces LRV-Instruction, a diverse dataset designed for visual instruction tuning with a focus on mitigating hallucination in large multi-modal models (LMMs). The dataset contains 400k visual instructions generated by GPT4 and includes negative as well as positive instructions to increase robustness, structured at different semantic levels of complexity. The authors propose GAVIE, an evaluation method that mimics human expert assessment without needing annotated ground truth, and demonstrate that training on the LRV-Instruction dataset, with an appropriate mix of positive and negative samples, reduces LMM hallucinations and improves performance across several tasks.

arxiv preprint - RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture

In this episode, we discuss RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture by Angels Balaguer, Vinamra Benara, Renato Luiz de Freitas Cunha, Roberto de M. Estevão Filho, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O. Nunes, Rafael Padilha, Morris Sharp, Bruno Silva, Swati Sharma, Vijay Aski, Ranveer Chandra. The paper explores two methods of integrating specialized data into Large Language Models (LLMs): Retrieval-Augmented Generation (RAG), which adds external data to the input, and Fine-Tuning, which embeds the data into the model itself. A multi-stage pipeline for these methods is tested on an agricultural dataset to evaluate their effectiveness in providing geographically tailored insights to farmers. Results indicate substantial improvements in accuracy (over 6 percentage points with Fine-Tuning and an additional 5 with RAG), with fine-tuned models effectively using cross-regional information, showcasing the potential for LLMs to be customized for industry-specific applications.