AI Breakdown

Arxiv paper - Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate

In this episode, we discuss Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate by Yubo Wang, Xiang Yue, Wenhu Chen. The paper introduces Critique Fine-Tuning (CFT), a novel approach where language models are trained to critique noisy responses instead of simply imitating correct ones, inspired by human critical thinking. Using a 50K-sample dataset generated by GPT-4o, CFT demonstrated consistent improvements of 4–10% over traditional supervised fine-tuning across various math benchmarks and datasets. The results show that CFT is both efficient and competitive, matching or outperforming models trained with much larger datasets and more compute, thereby effectively enhancing the reasoning capabilities of language models.

Arxiv paper - Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs

In this episode, we discuss Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs by Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, Dong Yu. The paper identifies "underthinking" in large language models like OpenAI’s GPT-4, where models frequently switch reasoning paths without fully exploring promising solutions, leading to errors on complex tasks such as challenging mathematical problems. Through experiments on multiple test sets and models, the authors demonstrate that frequent thought switching is linked to incorrect responses and introduce a metric to measure this underthinking based on token efficiency. To address the issue, they propose a thought switching penalty (TIP) decoding strategy that encourages deeper exploration of each reasoning path, resulting in improved accuracy without requiring model fine-tuning.

Arxiv paper - MetaMorph: Multimodal Understanding and Generation via Instruction Tuning

In this episode, we discuss MetaMorph: Multimodal Understanding and Generation via Instruction Tuning by Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, Zhuang Liu. The paper introduces Visual-Predictive Instruction Tuning (VPiT), which enhances pretrained large language models to generate both text and visual tokens by training on mixed image and text data. The study finds that visual generation naturally arises from improved visual understanding and that understanding data is more effective than generation data for enhancing both capabilities. Using VPiT, the authors develop the MetaMorph model, which achieves strong performance in visual understanding and generation by leveraging the inherent vision capabilities of language models through simple instruction tuning.

Arxiv paper - Improving Video Generation with Human Feedback

In this episode, we discuss Improving Video Generation with Human Feedback by Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, Xintao Wang, Xiaohong Liu, Fei Yang, Pengfei Wan, Di Zhang, Kun Gai, Yujiu Yang, Wanli Ouyang. The paper introduces a pipeline that utilizes human feedback to enhance video generation, addressing issues like unsmooth motion and prompt-video misalignment. It presents **VideoReward**, a multi-dimensional reward model trained on a large-scale human preference dataset, and develops three alignment algorithms—Flow-DPO, Flow-RWR, and Flow-NRG—to optimize flow-based video models. Experimental results show that VideoReward outperforms existing models, Flow-DPO achieves superior performance over other methods, and Flow-NRG allows for personalized video quality adjustments during inference.

Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

In this episode, we discuss Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling by The authors of the paper are: - Xiaokang Chen - Zhiyu Wu - Xingchao Liu - Zizheng Pan - Wen Liu - Zhenda Xie - Xingkai Yu - Chong Ruan. The paper introduces Janus-Pro, an enhanced version of the original Janus model that features an optimized training strategy, expanded training data, and a larger model size. These improvements lead to significant advancements in multimodal understanding, text-to-image instruction-following capabilities, and the stability of text-to-image generation. Additionally, the authors have made the code and models publicly available to encourage further research and exploration in the field.

Arxiv paper - DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

In this episode, we discuss DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning by DeepSeek-AI. The paper introduces DeepSeek-R1-Zero, a reasoning model trained solely with large-scale reinforcement learning, which exhibits strong reasoning abilities but struggles with readability and language mixing. To overcome these limitations, the authors developed DeepSeek-R1 by adding multi-stage training and cold-start data, achieving performance on par with OpenAI’s models. Additionally, they open-sourced DeepSeek-R1-Zero, DeepSeek-R1, and six distilled dense models to support the research community.

Arxiv paper - Can We Generate Images with CoT? Let’s Verify and Reinforce Image Generation Step by Step

In this episode, we discuss Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step by Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, Pheng-Ann Heng. The paper investigates the use of Chain-of-Thought (CoT) reasoning to improve autoregressive image generation through techniques like test-time computation scaling, Direct Preference Optimization (DPO), and their integration. The authors introduce the Potential Assessment Reward Model (PARM) and an enhanced version, PARM++, which evaluate and refine image generation for better performance, showing significant improvements over baseline models in benchmarks. The study offers insights into applying CoT reasoning to image generation, achieving notable advancements and releasing code and models for further research.

Arxiv paper - Improving Factuality with Explicit Working Memory

In this episode, we discuss Improving Factuality with Explicit Working Memory by Mingda Chen, Yang Li, Karthik Padthe, Rulin Shao, Alicia Sun, Luke Zettlemoyer, Gargi Gosh, Wen-tau Yih. The paper presents Ewe, a novel method that incorporates explicit working memory into large language models to improve factuality in long-form text generation by updating memory in real-time based on feedback from external resources. Ewe demonstrates superior performance over existing approaches across four datasets, boosting the VeriScore metric without compromising response helpfulness. The study highlights the significance of memory update rules, configuration, and retrieval datastore quality in enhancing the model's accuracy.

Arxiv paper - Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control

In this episode, we discuss Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control by Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, Yuan Liu. The paper introduces "Diffusion as Shader" (DaS), a novel approach that supports various video control tasks within a unified framework by utilizing 3D control signals, overcoming the limitations of existing methods which are typically restricted to 2D signals. DaS achieves precise video manipulation, such as camera control and content editing, by employing 3D tracking videos, resulting in enhanced temporal consistency. The approach was fine-tuned within three days using 8 H800 GPUs and demonstrates strong performance in tasks like mesh-to-video generation and motion transfer, with further resources available online.

Arxiv paper - FaceLift: Single Image to 3D Head with View Generation and GS-LRM

In this episode, we discuss FaceLift: Single Image to 3D Head with View Generation and GS-LRM by Weijie Lyu, Yi Zhou, Ming-Hsuan Yang, Zhixin Shu. FaceLift is a feed-forward approach for rapid and high-quality 360-degree head reconstruction using a single image, utilizing a multi-view latent diffusion model followed by a GS-LRM reconstructor to create 3D representations from generated views. It is trained primarily on synthetic datasets, showing strong real-world generalization, and outperforms existing 3D head reconstruction methods. Additionally, FaceLift enables 4D novel view synthesis for video inputs and can be integrated with 2D reanimation techniques for 3D facial animation.