AI Breakdown

Arxiv paper - MINERVA: Evaluating Complex Video Reasoning

In this episode, we discuss MINERVA: Evaluating Complex Video Reasoning by Arsha Nagrani, Sachit Menon, Ahmet Iscen, Shyamal Buch, Ramin Mehran, Nilpa Jha, Anja Hauth, Yukun Zhu, Carl Vondrick, Mikhail Sirotenko, Cordelia Schmid, Tobias Weyand. The paper introduces MINERVA, a new video reasoning dataset featuring complex multi-step questions with detailed reasoning traces to evaluate multimodal models beyond final answers. It benchmarks state-of-the-art models, revealing challenges mainly in temporal localization and visual perception rather than logical reasoning. The dataset and evaluation tools are publicly released to advance research in interpretable video understanding.

Arxiv paper - The Leaderboard Illusion

In this episode, we discuss The Leaderboard Illusion by Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D'Souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah Smith, Beyza Ermis, Marzieh Fadaee, Sara Hooker. The paper reveals that Chatbot Arena's leaderboard rankings are biased due to undisclosed private testing, allowing some providers to selectively disclose only their best-performing AI variants. It highlights significant data access inequalities favoring proprietary models, leading to overfitting on Arena-specific metrics rather than general model quality. The authors propose actionable reforms to improve transparency and fairness in AI benchmarking within the Arena.

Arxiv paper - Towards Understanding Camera Motions in Any Video

In this episode, we discuss Towards Understanding Camera Motions in Any Video by Zhiqiu Lin, Siyuan Cen, Daniel Jiang, Jay Karhade, Hewei Wang, Chancharik Mitra, Tiffany Ling, Yuhan Huang, Sifan Liu, Mingyu Chen, Rushikesh Zawar, Xue Bai, Yilun Du, Chuang Gan, Deva Ramanan. The paper presents CameraBench, a large-scale, expertly annotated video dataset and benchmark for analyzing camera motion using a novel taxonomy developed with cinematographers. It reveals that existing models struggle with either semantic or geometric aspects of camera motion, but fine-tuning generative video-language models on CameraBench improves performance across tasks. The work aims to advance automatic understanding of camera motions, supported by human studies, tutorials, and diverse video applications.

Arxiv paper - Describe Anything: Detailed Localized Image and Video Captioning

In this episode, we discuss Describe Anything: Detailed Localized Image and Video Captioning by Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, Yin Cui. The paper presents the Describe Anything Model (DAM) for detailed localized captioning that integrates local detail and global context using a focal prompt and localized vision backbone. It introduces a semi-supervised data pipeline (DLC-SDP) to address limited training data by leveraging segmentation datasets and unlabeled images. Additionally, the authors propose DLC-Bench, a new benchmark for evaluating detailed localized captioning, where DAM achieves state-of-the-art results across multiple tasks.

Arxiv paper - MCNC: MANIFOLD-CONSTRAINED REPARAMETERIZATION FOR NEURAL COMPRESSION

In this episode, we discuss MCNC: MANIFOLD-CONSTRAINED REPARAMETERIZATION FOR NEURAL COMPRESSION by The authors of the paper are: - Chayne Thrash - Ali Abbasi - Reed Andreas - Parsa Nooralinejad - Soroush Abbasi Koohpayegani - Hamed Pirsiavash - Soheil Kolouri. The paper introduces Manifold-Constrained Neural Compression (MCNC), a novel model compression technique that confines parameters to low-dimensional, pre-defined nonlinear manifolds. This approach leverages the over-parameterization of deep networks to find high-quality solutions while achieving superior compression rates. Experiments across computer vision and NLP tasks show that MCNC outperforms existing methods in compression efficiency, accuracy, and reconstruction speed.

Arxiv paper - Self-Improving Robust Preference Optimization

In this episode, we discuss Self-Improving Robust Preference Optimization by Eugene Choi, Arash Ahmadian, Matthieu Geist, Oilvier Pietquin, Mohammad Gheshlaghi Azar. The paper introduces Self-Improving Robust Preference Optimization (SRPO), an offline RLHF framework that enables models to self-improve and generalize across tasks by jointly optimizing a self-improvement and generative policy through a min-max objective. SRPO reformulates this objective into a non-adversarial offline loss that can be efficiently optimized using supervised learning. Experiments show SRPO significantly outperforms existing methods like DPO and IPO on benchmarks such as XSum and Arena-Hard, achieving higher win rates against human and AI baselines.

Arxiv paper - LLM Post-Training: A Deep Dive into Reasoning Large Language Models

In this episode, we discuss LLM Post-Training: A Deep Dive into Reasoning Large Language Models by Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip H. S. Torr, Fahad Shahbaz Khan, Salman Khan. The paper surveys post-training techniques for Large Language Models (LLMs) that enhance performance beyond initial pretraining, focusing on fine-tuning, reinforcement learning, and test-time scaling. It addresses challenges like catastrophic forgetting and reward hacking while exploring model alignment and scalable adaptation. The survey also provides a public repository to track ongoing advancements in post-training methods.

Arxiv paper - Welcome to the Era of Experience

In this episode, we discuss Welcome to the Era of Experience by David Silver, Richard S. Sutton. The paper discusses the forthcoming era of artificial intelligence marked by agents with superhuman capabilities. These agents will primarily learn through experience. The note highlights the essential features that will characterize this new phase in AI development.

Arxiv paper - MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation

In this episode, we discuss MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation by Sihyun Yu, Meera Hahn, Dan Kondratyuk, Jinwoo Shin, Agrim Gupta, José Lezama, Irfan Essa, David Ross, Jonathan Huang. The paper introduces MALT Diffusion, a new diffusion model designed for generating long videos by dividing them into short segments and using recurrent attention to maintain a memory latent vector for long-term context. It presents training techniques to ensure consistent quality over extended frames and demonstrates superior performance on long video benchmarks, significantly improving FVD scores. Additionally, MALT shows strong results in text-to-video generation, capable of producing longer videos than existing methods.

Arxiv paper - InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

In this episode, we discuss InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models by The authors of the paper "InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models" are as follows: 1. **Jinguo Zhu** 2. **Weiyun Wang** 3. **Zhe Chen** 4. ... InternVL3 advances the InternVL series by jointly training on multimodal and text data in a unified pre-training stage, avoiding the complexities of adapting text-only models to handle visual inputs. It incorporates features like variable visual position encoding and advanced fine-tuning techniques, achieving state-of-the-art performance on benchmarks such as MMMU and competing with leading proprietary models. Committed to open science, the authors plan to publicly release both the training data and model weights to support further research in multimodal large language models.