AI Breakdown

arxiv preprint - InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes

In this episode, we discuss InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes by Mohamad Shahbazi, Liesbeth Claessens, Michael Niemeyer, Edo Collins, Alessio Tonioni, Luc Van Gool, Federico Tombari. InseRF is a new approach for inserting generated objects into 3D scene reconstructions using NeRF, based on textual descriptions and 2D reference images. This method overcomes the limitations of existing scene editing techniques, which struggle with the generation of new objects, by performing a 2D insertion in a reference view and extrapolating it to 3D with the help of single-view reconstruction and monocular depth estimation priors. Extensive evaluations show that InseRF achieves controllable and 3D-consistent object insertions, outperforming current methods, and it does so without needing explicit 3D models as input.

arxiv preprint - A Simple LLM Framework for Long-Range Video Question-Answering

In this episode, we discuss A Simple LLM Framework for Long-Range Video Question-Answering by Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, Gedas Bertasius. The LLoVi framework innovates in long-range video question-answering (LVQA) by combining visual captioners with Large Language Models (LLMs) such as GPT-3.5 or GPT-4, foregoing complex long-range video modeling structures. Short video clips from a long video are captioned and these captions are then synthesized by an LLM to answer questions over the entire video length, proving more effective at LVQA than previous methods. In benchmarks, LLoVi notably outperformed previous best-performing approaches on several datasets, such as EgoSchema, NeXT-QA, IntentQA, and NeXT-GQA, and the code for LLoVi will be made publicly available.

arxiv preprint - Mixtral of Experts

In this episode, we discuss Mixtral of Experts by Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed. Mixtral 8x7B is a Sparse Mixture of Experts (SMoE) language model, building on Mistral 7B's architecture with 8 experts per layer, among which two experts are selected per token for processing, allowing access to 47B parameters but using only 13B actively. It excels in benchmarks, surpassing Llama 2 70B and GPT-3.5, especially in areas like math, code generation, and multilingual tasks. A special instruction-following version called Mixtral 8x7B – Instruct also outperforms leading models, with both models being open-sourced under the Apache 2.0 license.

arxiv preprint - Weight subcloning: direct initialization of transformers using larger pretrained ones

In this episode we discuss Weight subcloning: direct initialization of transformers using larger pretrained ones
by Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, Mohammad Rastegari. The paper introduces a new method called weight subcloning to expedite the training of small transformer models by initializing them with weights from larger pretrained models. This method ranks neurons by importance to reduce dimensions and removes blocks to align with the smaller model's layer count, resulting in significantly faster training times. Weight subcloning allows the transfer of knowledge from larger to smaller models, improving speed and potentially accuracy without the need for a pretrained model of the exact desired size.

arxiv preprint - Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task

In this episode we discuss Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task
by Maya Okawa, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka. The paper investigates how conditional diffusion models generalize compositionally by studying their ability to generate novel data combinations within a controlled synthetic environment. Key discoveries include that compositional ability hinges on the data-generating process structure, and there's a sudden emergence of compositional performance linked to individual task proficiency. The findings also show that rarely seen concepts in training are tougher to compose for new outputs, shedding light on the generative models' capabilities from the perspective of data availability and structure.

arxiv preprint - LLM in a flash: Efficient Large Language Model Inference with Limited Memory

In this episode, we discuss LLM in a flash: Efficient Large Language Model Inference with Limited Memory by Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, Mehrdad Farajtabar. The paper introduces an approach to operate large language models (LLMs) efficiently on devices with limited DRAM by using flash memory to store and selectively load model parameters. It proposes an inference cost model specific to flash memory to optimize data transfers and introduces "windowing" and "row-column bundling" techniques to improve data read efficiency. By implementing these strategies, the paper demonstrates that LLMs up to twice the size of the DRAM can be run 4-5 times faster on CPU and 20-25 times faster on GPU compared to standard loading methods, while also incorporating sparsity and context-awareness for enhanced performance.

arxiv preprint - The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction

In this episode, we discuss The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction by Pratyusha Sharma, Jordan T. Ash, Dipendra Misra. The paper presents Layer-Selective Rank Reduction (LASER), an innovative method that enhances Transformer-based Large Language Models (LLMs) by reducing higher-order features in their weight matrices post-training, without adding parameters or data. Extensive experiments show that LASER significantly boosts the performance of various LLMs on multiple datasets. The authors also delve into the theoretical understanding of LASER, examining the conditions under which it is most beneficial and the principles of how it works.

arxiv preprint - DreaMoving: A Human Video Generation Framework based on Diffusion Models

In this episode we discuss DreaMoving: A Human Video Generation Framework based on Diffusion Models
by Mengyang Feng, Jinlin Liu, Kai Yu, Yuan Yao, Zheng Hui, Xiefan Guo, Xianhui Lin, Haolan Xue, Chen Shi, Xiaowen Li, Aojie Li, Xiaoyang Kang, Biwen Lei, Miaomiao Cui, Peiran Ren, Xuansong Xie. DreaMoving is a framework that uses diffusion models to create customized human dance videos, where a target person can be seen performing specific dance moves. It consists of two main components: the Video ControlNet, which oversees motion control, and the Content Guider, which ensures the target individual's identity is maintained throughout the video. The framework is designed to be user-friendly and flexible, allowing for a wide range of video styles and is further detailed on its project page.

arxiv preprint - Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution

In this episode we discuss Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
by Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, Avital Oliver, Piotr Padlewski, Alexey Gritsenko, Mario Lučić, Neil Houlsby. The paper introduces NaViT (Native Resolution Vision Transformer), which unlike traditional computer vision models does not require resizing images to a fixed resolution, instead handling arbitrary resolutions and aspect ratios through sequence packing. NaViT demonstrates better training efficiency and can be applied to various standard computer vision tasks, where it also achieves improved robustness and fairness results. This approach allows for flexible input handling at test time, optimizing performance-cost trade-offs, and represents a significant shift from conventional CNN-based computer vision pipelines.

arxiv preprint - UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces

In this episode, we discuss UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces by Jiannan Wu, Yi Jiang, Bin Yan, Huchuan Lu, Zehuan Yuan, Ping Luo. The paper introduces UniRef++, a unified architecture designed to address four reference-based object segmentation tasks: referring image segmentation (RIS), few-shot image segmentation (FSS), referring video object segmentation (RVOS), and video object segmentation (VOS). At the core of UniRef++ is the UniFusion module, which enables multiway fusion adjusted to task-specific references, along with a unified Transformer architecture for instance-level segmentation. UniRef++ demonstrates state-of-the-art performance on RIS and RVOS benchmarks, competitive results on FSS and VOS, and can be integrated with existing models, like SAM, for parameter-efficient finetuning.