AI Breakdown

arxiv - MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

In this episode, we discuss MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI by Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, Wenhu Chen. MMMU is a new benchmark for evaluating multimodal models using college-level questions from various disciplines to test advanced reasoning and subject knowledge. The benchmark contains 11.5K questions across six core disciplines and 30 subjects, featuring diverse visual content like graphs and music sheets. Initial testing on 14 models, including the sophisticated GPT-4V, showed a best accuracy of 56%, suggesting ample scope for improvement in artificial general intelligence.

arxiv preprint - MLP-Mixer: An all-MLP Architecture for Vision

In this episode we discuss MLP-Mixer: An all-MLP Architecture for Vision
by Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, Alexey Dosovitskiy. The paper presents MLP-Mixer, an architecture that relies solely on multi-layer perceptrons (MLPs) for image classification tasks, demonstrating that neither convolutions nor attention mechanisms are necessary for high performance. The MLP-Mixer operates with two types of layers: one that processes features within individual image patches, and another that blends features across different patches. The model achieves competitive results on benchmarks when trained on large datasets or with modern regularization techniques, suggesting a new direction for image recognition research beyond conventional CNNs and Transformers.

arxiv preprint - Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine

In this episode we discuss Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
by Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, Eric Horvitz. The paper discusses enhancing the performance of GPT-4, a generalist language model, in medical question-answering tasks without domain-specific training. By innovatively engineering prompts, the researchers created Medprompt, which significantly outperformed specialized models, achieving state-of-the-art results on the MultiMedQA benchmark suite with fewer model calls. Moreover, Medprompt was also successful in generalizing its capabilities to other fields, demonstrating its broad applicability across various competency exams beyond medicine.

arxiv preprint - Nash Learning from Human Feedback

In this episode we discuss Nash Learning from Human Feedback by Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, Bilal Piot from Google DeepMind. The paper introduces Nash Learning from Human Feedback (NLHF), a new approach for tuning large language models (LLMs) based on human preferences, different from the traditional reinforcement learning from human feedback (RLHF). The NLHF technique involves learning a preference model from paired comparisons and refining the LLM's policy towards a Nash equilibrium, where no alternative policy produces more preferred responses. They developed a Nash-MD algorithm and gradient descent approaches for implementing NLHF, and demonstrated its effectiveness on a text summarization task, suggesting NLHF as a promising direction for aligning LLMs with human preferences.

arxiv preprint - Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

In this episode we discuss Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation
by Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, Liefeng Bo. The paper presents a novel framework designed for character animation that synthesizes consistent and controllable videos from still images using diffusion models. It introduces a ReferenceNet that utilizes spatial attention to keep the character's appearance consistent and integrates a pose guider for movement controllability along with a technique to ensure smooth temporal transitions. The method exhibits superior performance on character animation, including fashion video and human dance synthesis benchmarks, outperforming other image-to-video methods.

arxiv preprint - Knowledge is a Region in Weight Space for Fine-tuned Language Models

In this episode we discuss Knowledge is a Region in Weight Space for Fine-tuned Language Models
by Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, Leshem Choshen. The paper investigates the relationships between different neural network models when trained on diverse datasets, focusing on their weight space and loss landscape. The study reveals that language models finetuned on the same task but different datasets form clusters in weight space, and it is possible to navigate between these clusters to create new models with strong or even improved performance on various tasks. By utilizing this understanding, the research introduces a method where initiating finetuning from the central point of a model cluster achieves better results than starting with a pretrained model, as evidenced by an average accuracy improvement of 3.06 across 11 out of 12 datasets.

arxiv preprint - MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training

In this episode we discuss MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
by Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel. The paper introduces MobileCLIP, a new efficient image-text model family optimized for mobile devices with a novel multi-modal reinforced training method that enhances accuracy without increasing on-device computational demands. MobileCLIP achieves better latency-accuracy trade-offs in zero-shot classification and retrieval tasks and outperforms existing models in speed and accuracy. The reinforced training method improves learning efficiency by factors of 10 to 1000 times, demonstrated by advancements in a CLIP model with a ViT-B/16 image backbone across 38 benchmarks.

arxiv preprint - Simplifying Transformer Blocks

In this episode we discuss Simplifying Transformer Blocks
by Bobby He, Thomas Hofmann. The paper studies the possibility of simplifying standard transformer blocks without reducing training speed by experimenting with the removal of certain components such as skip connections and normalization layers. Using signal propagation theory along with empirical research, the authors justify modifications that allow for these simplifications. Their findings indicate that the streamlined transformer models match the performance and training speed of traditional transformers while offering increased training throughput and reduced parameter count.

arxiv - Visual In-Context Prompting

In this episode, we discuss Visual In-Context Prompting by Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Chunyuan Li, Jianwei Yang, Lei Zhang, Jianfeng Gao. This paper introduces a new framework for improving zero-shot learning capabilities in vision tasks called universal visual in-context prompting, which works by allowing an encoding-decoding architecture to utilize various types of prompts like strokes, boxes, and points, as well as reference image segments as context. Unlike existing methods, which are limited to referring segmentation, the framework extends to a broader range of tasks including open-set segmentation and detection. The authors demonstrate notable performance enhancements, with the proposed method achieving competitive results on close-set in-domain datasets like COCO and promising outcomes on open-set datasets such as ADE20K, with planned code release on GitHub.

Arxiv Preprint - GAIA: a benchmark for General AI Assistants

In this episode we discuss GAIA: a benchmark for General AI Assistants
by Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom. The paper introduces GAIA, a benchmark designed to assess the capabilities of General AI Assistants in performing tasks that are simple for humans yet difficult for AIs, such as reasoning, multi-modal tasks, web browsing, and general tool-use. It highlights a significant performance discrepancy, with humans scoring a 92% success rate contrasting with a mere 15% for an advanced AI model (GPT-4 with plugins). The authors propose this benchmark as a measure to guide AI research towards achieving robustness in tasks where humans excel, challenging the prevailing focus on skills that are difficult for humans, and establishing a leaderboard for tracking AI progress.