AI Breakdown

Arxiv paper - The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models

In this episode, we discuss The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models by Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo. The paper introduces BIGGEN BENCH, a comprehensive benchmark designed to evaluate nine distinct language model capabilities across 77 diverse tasks with instance-specific criteria that better reflect human judgment. It addresses limitations of existing benchmarks, such as abstract evaluation metrics and coverage bias. The authors apply BIGGEN BENCH to assess 103 advanced language models using five evaluator models, making all resources publicly accessible.

Arxiv paper - DanceGRPO: Unleashing GRPO on Visual Generation

In this episode, we discuss DanceGRPO: Unleashing GRPO on Visual Generation by Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, Ping Luo. The paper presents DanceGRPO, a unified reinforcement learning framework that adapts Group Relative Policy Optimization to various generative paradigms, including diffusion models and rectified flows, across multiple visual generation tasks. It effectively addresses challenges in stability, compatibility with ODE-based sampling, and video generation, demonstrating significant performance improvements over existing methods. DanceGRPO enables scalable and versatile RL-based alignment of model outputs with human preferences in visual content creation.

Arxiv paper - Visual Planning: Let’s Think Only with Images

In this episode, we discuss Visual Planning: Let's Think Only with Images by Yi Xu, Chengzu Li, Han Zhou, Xingchen Wan, Caiqi Zhang, Anna Korhonen, Ivan Vulić. This paper proposes Visual Planning, a new approach that uses purely visual sequences to perform reasoning and planning without relying on text. They introduce a reinforcement learning framework, VPRL, which enhances large vision models for improved performance on visual navigation tasks like FROZENLAKE and MAZE. Their results show that visual planning surpasses traditional text-based methods, offering a more intuitive way to handle spatial and geometric reasoning.

Arxiv paper - A Preliminary Study for GPT-4o on Image Restoration

In this episode, we discuss A Preliminary Study for GPT-4o on Image Restoration by Hao Yang, Yan Yang, Ruikun Zhang, Liyuan Pan. This paper presents the first comprehensive evaluation of OpenAI’s GPT-4o model on various image restoration tasks, revealing that while its outputs are visually appealing, they often lack pixel-level structural accuracy. The authors demonstrate that GPT-4o can effectively serve as a visual prior to improve existing restoration networks in tasks like dehazing, deraining, and low-light enhancement. They also provide practical guidelines and release a dataset of GPT-4o-restored images to support future research in image restoration.

Arxiv paper - DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion

In this episode, we discuss DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion by Qitao Zhao, Amy Lin, Jeff Tan, Jason Y. Zhang, Deva Ramanan, Shubham Tulsiani. The paper introduces DiffusionSfM, a novel data-driven framework that directly infers 3D scene geometry and camera poses from multi-view images using a transformer-based denoising diffusion model. It represents scene geometry and camera parameters as pixel-wise ray origins and endpoints in a global coordinate frame and incorporates specialized mechanisms to handle challenges like missing data and unbounded coordinates. Experiments on synthetic and real datasets show that DiffusionSfM outperforms existing classical and learning-based SfM methods while effectively modeling uncertainty.

Arxiv paper - RayZer: A Self-supervised Large View Synthesis Model

In this episode, we discuss RayZer: A Self-supervised Large View Synthesis Model by Hanwen Jiang, Hao Tan, Peng Wang, Haian Jin, Yue Zhao, Sai Bi, Kai Zhang, Fujun Luan, Kalyan Sunkavalli, Qixing Huang, Georgios Pavlakos. RayZer is a self-supervised multi-view 3D vision model that learns 3D scene understanding without any 3D supervision, including camera poses or scene geometry. It predicts camera parameters and reconstructs scenes from unposed, uncalibrated images using only 2D image supervision, enabled by a framework that disentangles camera and scene representations and a transformer leveraging ray-based 3D priors. RayZer achieves novel view synthesis performance on par with or better than methods relying on ground-truth pose annotations.

Arxiv paper - Reinforcement Learning for Reasoning in Large Language Models with One Training Example

In this episode, we discuss Reinforcement Learning for Reasoning in Large Language Models with One Training Example by Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen. The paper demonstrates that reinforcement learning with verifiable reward using only one or two training examples (1-shot RLVR) substantially improves mathematical reasoning in large language models, nearly doubling performance on benchmarks like MATH500. This method generalizes across different models, algorithms, and examples, showing unique phenomena such as post-saturation generalization and the importance of policy gradient loss and exploration encouragement. The authors provide open-source code and data, highlighting the potential for more data-efficient RLVR approaches in improving LLM capabilities.

Arxiv paper - MINERVA: Evaluating Complex Video Reasoning

In this episode, we discuss MINERVA: Evaluating Complex Video Reasoning by Arsha Nagrani, Sachit Menon, Ahmet Iscen, Shyamal Buch, Ramin Mehran, Nilpa Jha, Anja Hauth, Yukun Zhu, Carl Vondrick, Mikhail Sirotenko, Cordelia Schmid, Tobias Weyand. The paper introduces MINERVA, a new video reasoning dataset featuring complex multi-step questions with detailed reasoning traces to evaluate multimodal models beyond final answers. It benchmarks state-of-the-art models, revealing challenges mainly in temporal localization and visual perception rather than logical reasoning. The dataset and evaluation tools are publicly released to advance research in interpretable video understanding.

Arxiv paper - The Leaderboard Illusion

In this episode, we discuss The Leaderboard Illusion by Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D'Souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah Smith, Beyza Ermis, Marzieh Fadaee, Sara Hooker. The paper reveals that Chatbot Arena's leaderboard rankings are biased due to undisclosed private testing, allowing some providers to selectively disclose only their best-performing AI variants. It highlights significant data access inequalities favoring proprietary models, leading to overfitting on Arena-specific metrics rather than general model quality. The authors propose actionable reforms to improve transparency and fairness in AI benchmarking within the Arena.

Arxiv paper - Towards Understanding Camera Motions in Any Video

In this episode, we discuss Towards Understanding Camera Motions in Any Video by Zhiqiu Lin, Siyuan Cen, Daniel Jiang, Jay Karhade, Hewei Wang, Chancharik Mitra, Tiffany Ling, Yuhan Huang, Sifan Liu, Mingyu Chen, Rushikesh Zawar, Xue Bai, Yilun Du, Chuang Gan, Deva Ramanan. The paper presents CameraBench, a large-scale, expertly annotated video dataset and benchmark for analyzing camera motion using a novel taxonomy developed with cinematographers. It reveals that existing models struggle with either semantic or geometric aspects of camera motion, but fine-tuning generative video-language models on CameraBench improves performance across tasks. The work aims to advance automatic understanding of camera motions, supported by human studies, tutorials, and diverse video applications.