AI Breakdown

Arxiv paper - VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection

In this episode, we discuss VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection by Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, Si Liu. The paper introduces VideoEspresso, a high-quality, large-scale VideoQA dataset that maintains essential spatial and temporal details and includes multimodal annotations for intermediate reasoning steps. Utilizing a semantic-aware construction pipeline and GPT-4 for generating QA pairs and Chain-of-Thought annotations, the dataset enhances scalability and reasoning complexity. Additionally, the authors propose a Hybrid LVLMs Collaboration framework that outperforms existing models on 14 tasks, demonstrating superior video reasoning capabilities.

Arxiv paper - VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models

In this episode, we discuss VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models by Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, Shelly Sheynin. Generative video models typically prioritize appearance accuracy over motion coherence, limiting their ability to capture realistic dynamics. The paper presents VideoJAM, a framework that integrates a joint appearance-motion representation and uses an Inner-Guidance mechanism to enhance motion consistency during generation. VideoJAM achieves state-of-the-art motion realism and visual quality while being easily adaptable to existing video models without major changes.

Arxiv paper - HunyuanVideo: A Systematic Framework For Large Video Generative Models

In this episode, we discuss HunyuanVideo: A Systematic Framework For Large Video Generative Models by Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhentao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zunnan Xu, Yangyu Tao, Qinglin Lu, Songtao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, Caesar Zhong. HunyuanVideo is an innovative open-source video generation model that matches or exceeds the performance of leading closed-source alternatives. It leverages a comprehensive framework encompassing data curation, advanced architecture, progressive scaling, and efficient infrastructure to train a 13-billion-parameter model, the largest of its kind in the open-source domain. Extensive evaluations reveal that HunyuanVideo delivers superior visual quality, motion dynamics, and text-video alignment, and its publicly available code aims to bridge the gap between closed and open-source communities, fostering a more dynamic video generation ecosystem.

Arxiv paper - s1: Simple test-time scaling

In this episode, we discuss s1: Simple test-time scaling by Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, Tatsunori Hashimoto. The paper introduces a straightforward method for test-time scaling in language models to enhance reasoning performance by utilizing additional computational resources during inference. The authors develop a curated dataset of 1,000 high-quality, diverse, and challenging questions with reasoning traces and implement a "budget forcing" technique that controls the model's computation by either terminating its reasoning process or extending it to encourage double-checking answers. Using this approach, their fine-tuned Qwen2.5-32B-Instruct model outperforms OpenAI’s o1 model on competitive math benchmarks by up to 27% and the resources are made available as open-source.

Arxiv paper - Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation

In this episode, we discuss Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation by The authors of the paper are the **Hunyuan3D Team**. Specific contributor names are indicated to be listed at the end of the full report.. Hunyuan3D 2.0 is a large-scale 3D synthesis system featuring Hunyuan3D-DiT for generating detailed geometry and Hunyuan3D-Paint for producing high-resolution textures. It includes Hunyuan3D-Studio, a user-friendly platform that allows both professionals and amateurs to efficiently create and manipulate 3D assets. The system outperforms previous models in geometry detail, texture quality, and condition alignment, and it is publicly released to support the open-source 3D community.

Arxiv paper - MatAnyone: Stable Video Matting with Consistent Memory Propagation

In this episode, we discuss MatAnyone: Stable Video Matting with Consistent Memory Propagation by Peiqing Yang, Shangchen Zhou, Jixin Zhao, Qingyi Tao, Chen Change Loy. The paper introduces **MatAnyone**, a robust framework for target-assigned video matting that overcomes challenges posed by complex or ambiguous backgrounds without relying on auxiliary inputs. It employs a memory-based approach with a consistent memory propagation module and region-adaptive memory fusion to maintain semantic stability and preserve detailed object boundaries across frames. Additionally, the authors present a large, high-quality dataset and a novel training strategy leveraging extensive segmentation data to enhance matting stability and performance.

Arxiv paper - Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate

In this episode, we discuss Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate by Yubo Wang, Xiang Yue, Wenhu Chen. The paper introduces Critique Fine-Tuning (CFT), a novel approach where language models are trained to critique noisy responses instead of simply imitating correct ones, inspired by human critical thinking. Using a 50K-sample dataset generated by GPT-4o, CFT demonstrated consistent improvements of 4–10% over traditional supervised fine-tuning across various math benchmarks and datasets. The results show that CFT is both efficient and competitive, matching or outperforming models trained with much larger datasets and more compute, thereby effectively enhancing the reasoning capabilities of language models.

Arxiv paper - Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs

In this episode, we discuss Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs by Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, Dong Yu. The paper identifies "underthinking" in large language models like OpenAI’s GPT-4, where models frequently switch reasoning paths without fully exploring promising solutions, leading to errors on complex tasks such as challenging mathematical problems. Through experiments on multiple test sets and models, the authors demonstrate that frequent thought switching is linked to incorrect responses and introduce a metric to measure this underthinking based on token efficiency. To address the issue, they propose a thought switching penalty (TIP) decoding strategy that encourages deeper exploration of each reasoning path, resulting in improved accuracy without requiring model fine-tuning.

Arxiv paper - MetaMorph: Multimodal Understanding and Generation via Instruction Tuning

In this episode, we discuss MetaMorph: Multimodal Understanding and Generation via Instruction Tuning by Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, Zhuang Liu. The paper introduces Visual-Predictive Instruction Tuning (VPiT), which enhances pretrained large language models to generate both text and visual tokens by training on mixed image and text data. The study finds that visual generation naturally arises from improved visual understanding and that understanding data is more effective than generation data for enhancing both capabilities. Using VPiT, the authors develop the MetaMorph model, which achieves strong performance in visual understanding and generation by leveraging the inherent vision capabilities of language models through simple instruction tuning.

Arxiv paper - Improving Video Generation with Human Feedback

In this episode, we discuss Improving Video Generation with Human Feedback by Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, Xintao Wang, Xiaohong Liu, Fei Yang, Pengfei Wan, Di Zhang, Kun Gai, Yujiu Yang, Wanli Ouyang. The paper introduces a pipeline that utilizes human feedback to enhance video generation, addressing issues like unsmooth motion and prompt-video misalignment. It presents **VideoReward**, a multi-dimensional reward model trained on a large-scale human preference dataset, and develops three alignment algorithms—Flow-DPO, Flow-RWR, and Flow-NRG—to optimize flow-based video models. Experimental results show that VideoReward outperforms existing models, Flow-DPO achieves superior performance over other methods, and Flow-NRG allows for personalized video quality adjustments during inference.