AI Breakdown

arxiv preprint - Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation

In this episode we discuss Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation
by Shuai Yang, Yifan Zhou, Ziwei Liu, Chen Change Loy. The paper introduces a novel framework for adapting image models to videos through zero-shot text-guided video-to-video translation. The framework consists of two parts: key frame translation and full video translation. The key frame translation generates key frames with hierarchical cross-frame constraints to ensure coherence, which are then propagated to other frames using temporal-aware patch matching and frame blending. The framework achieves global style and local texture temporal consistency without the need for re-training or optimization and performs better than existing methods in producing high-quality and coherent videos.

arxiv preprint - RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models

In this episode we discuss RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models
by Xingchen Zhou, Ying He, F. Richard Yu, Jianqiang Li, You Li. The paper proposes a framework called RePaint-NeRF for editing the content in Neural Radiance Fields (NeRF). Traditional NeRF methods struggle with content editing, so the framework leverages diffusion models to guide changes in the designated 3D content. It effectively allows for editing appearance and shape changes of 3D objects in NeRF, improving editability, diversity, and application range.

arxiv preprint - ZipIt! Merging Models from Different Tasks without Training

In this episode we discuss ZipIt! Merging Models from Different Tasks without Training
by George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, Judy Hoffman. The paper introduces a method called "ZipIt!" that can merge two deep visual recognition models trained on separate tasks without additional training. The method incorporates a "zip" operation to handle non-shared features within each model and allows for partial merging up to a specified layer, creating a multi-head model. Experimental results demonstrate a substantial improvement of 20-60% compared to previous approaches, making it possible to merge models trained on different tasks.

CVPR 2023 - Integral Neural Networks

In this episode, we discuss, CVPR 2023 award candidate, Integral Neural Networks by Kirill Solodskikh, Azim Kurbanov, Ruslan Aydarkhanov, Irina Zhelavskaya, Yury Parfenov, Dehua Song, and Stamatios Lefkimmiatis.
The paper introduces a novel type of deep neural networks called Integral Neural Networks (INNs), which deviate from the traditional representation of network layers as N-dimensional weight tensors. Instead, INNs use a continuous layer representation along the filter and channel dimensions. The weights of INNs are represented as continuous functions defined on N-dimensional hypercubes, and the discrete transformations of inputs to the layers are replaced by continuous integration operations. During the inference stage, the continuous layers can be converted back to the traditional tensor representation using numerical integral quadratures. This representation allows for the arbitrary discretization of a network with various discretization intervals for the integral kernels. The paper demonstrates that INNs can be used for model pruning directly on edge devices, with only a small performance loss at high rates of structural pruning, without the need for fine-tuning. Experimental results on multiple tasks and neural network architectures show that INNs achieve similar performance to their discrete counterparts, while preserving approximately the same performance even with high rates of structural pruning (up to 30%) without fine-tuning. In comparison, conventional pruning methods under the same conditions result in a 65% accuracy loss. The code for implementing INNs is available at gitee.

arxiv preprint - Faith and Fate: Limits of Transformers on Compositionality

In this episode we discuss Faith and Fate: Limits of Transformers on Compositionality
by Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, Yejin Choi. The paper examines the limitations of large language models (LLMs) like Transformers on compositional tasks. It explores their performance on three specific tasks and finds that Transformers tend to solve these tasks by reducing them to linearized subgraph matching rather than demonstrating systematic problem-solving skills. The paper also discusses how Transformers' performance decreases as task complexity increases.

arxiv preprint - LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

In this episode we discuss LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
by Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, William Yang Wang. The paper introduces LayoutGPT, a method that uses Large Language Models (LLMs) to generate layouts from text instructions. LayoutGPT utilizes a style sheet language to generate plausible layouts in 2D images and 3D indoor scenes, and performs well in converting challenging language concepts into accurate layout arrangements. When combined with an image generation model, LayoutGPT outperforms text-to-image models and achieves performance comparable to human users in designing visually correct layouts. It also shows promise in 3D indoor scene synthesis, showcasing its potential in different visual domains.

CVPR2023 - 3D Human Pose Estimation via Intuitive Physics

In this episode we discuss 3D Human Pose Estimation via Intuitive Physics
by Shashank Tripathi, Lea Müller, Chun-Hao P. Huang, Omid Taheri, Michael J. Black, Dimitrios Tzionas. This paper introduces a method called IPMAN (Intuitive Physics-based Human Pose Estimation) that aims to estimate 3D human pose from images while producing physically plausible body configurations. The method leverages intuitive-physics terms to infer the pressure heatmap on the body, the center of pressure (CoP), and the body's center of mass (CoM) to encourage floor contact and overlapping CoP and CoM. The proposed method is evaluated on standard datasets and a new dataset with complex poses and body-floor contact, showing improved accuracy compared to state-of-the-art methods.

arxiv preprint - Textbooks Are All You Need

In this episode, we discuss Textbooks Are All You Need by Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, Yuanzhi Li. The paper introduces phi-1, a new language model for code that is smaller in size compared to other models. Despite its smaller scale, phi-1 performs well in accuracy tests and displays some surprising emergent properties. The study highlights the importance of high-quality data in improving the performance of large language models and reducing training requirements.

CVPR 2023 - Honorable mention award winner - DynIBaR: Neural Dynamic Image-Based Rendering

In this episode, we discuss DynIBaR: Neural Dynamic Image-Based Rendering by Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, Noah Snavely. The paper presents a new approach called "DynIBaR" that can generate novel views from a monocular video of a dynamic scene. Existing methods struggle with complex object motions and uncontrolled camera paths, resulting in blurry or inaccurate renderings. DynIBaR addresses these limitations by using a volumetric image-based rendering framework that combines features from nearby views in a motion-aware manner, enabling the synthesis of photo-realistic views from long videos with complex dynamics and varied camera movements. The approach outperforms existing methods on dynamic scene datasets and is also applied successfully to challenging real-world videos with difficult camera and object motion.

Preprint: Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

In this episode, we discuss Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale by Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz,Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, Wei-Ning Hsu from Meta AI. The paper "Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale" presents a breakthrough in generative modeling for speech, addressing the lack of scalability and task generalization in current speech generative models. The authors introduce Voicebox, a non-autoregressive flow-matching model trained on over 50K hours of speech that can perform mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. Similar to large-scale generative models for language and vision, Voicebox can solve tasks not explicitly trained on through in-context learning.