AI Breakdown

CVPR 2023 - MobileNeRF: Exploiting the Polygon Rasterization Pipeline

In this episode we discuss MobileNeRF: Exploiting the Polygon Rasterization Pipeline
by Authors:
- Zhiqin Chen
- Thomas Funkhouser
- Peter Hedman
- Andrea Tagliasacchi
Affiliations:
- Zhiqin Chen: 1,2
- Thomas Funkhouser: 1
- Peter Hedman: 1
- Andrea Tagliasacchi: 1,2,3
1: Google Research
2: Simon Fraser University
3: University of Toronto. The paper introduces a new representation for Neural Radiance Fields (NeRFs) based on textured polygons that can synthesize novel images efficiently with standard rendering pipelines. This approach enables NeRFs to be rendered with the traditional polygon rasterization pipeline, achieving interactive frame rates even on mobile devices. The NeRF is represented as a set of polygons with textures representing binary opacities and feature vectors interpreted by a small, view-dependent MLP running in a fragment shader to produce a final pixel color.

CVPR 2023 - TimeBalance: Temporally-Invariant and Temporally-Distinctive Video

In this episode we discuss TimeBalance: Temporally-Invariant and Temporally-Distinctive Video
by Authors: Ishan Rajendrakumar Dave, Mamshad Nayeem Rizve, Chen Chen, Mubarak Shah
Affiliation: Center for Research in Computer Vision, University of Central Florida, Orlando, USA
Contact Emails: {ishandave, nayeemrizve}@knights.ucf.edu, {chen.chen, shah}@crcv.ucf.edu. The paper proposes a semi-supervised learning framework called TimeBalance for video domain tasks that utilizes self-supervised representations. Unlike existing methods that rely on hard input inductive biases, TimeBalance utilizes a temporally-invariant and a temporally-distinctive teacher to distill knowledge from unlabeled videos based on a novel temporal similarity-based reweighting scheme. The method achieves state-of-the-art performance on three action recognition benchmarks: UCF101, HMDB51, and Kinetics400. Code for TimeBalance is available on GitHub.

CVPR 2023 - On Distillation of Guided Diffusion Models

In this episode we discuss On Distillation of Guided Diffusion Models
by Authors:
- Chenlin Meng
- Robin Rombach
- Ruiqi Gao
- Diederik Kingma
- Stefano Ermon
- Jonathan Ho
- Tim Salimans
Affiliations:
- Chenlin Meng: Stanford University
- Robin Rombach: Stability AI & LMU Munich
- Ruiqi Gao, Diederik Kingma, Jonathan Ho, Tim Salimans: Google Research, Brain Team
- Stefano Ermon: Stanford University. The paper proposes an approach to distill classifier-free guided diffusion models, a type of high-resolution image generation model, into faster models that require fewer sampling steps. This is achieved by first learning a single model to match the output of the combined conditional and unconditional models and then progressively distilling that model to a diffusion model. The approach is shown to generate visually comparable images to the original model while being up to 256 times faster to sample from on ImageNet 64x64 and CIFAR-10 datasets. It also achieves high-fidelity images using as few as 1 to 4 denoising steps on ImageNet 256x256 and LAION datasets for diffusion models trained on the latent-space, and is effective in text-guided image editing and inpainting.

CVPR 2023 - Vid2Seq: Large-Scale Pretraining of a Visual Language Model

In this episode we discuss Vid2Seq: Large-Scale Pretraining of a Visual Language Model
by Authors:
- Antoine Yang
- Arsha Nagrani
- Paul Hongsuck Seo
- Antoine Miech
- Jordi Pont-Tuset
- Ivan Laptev
- Josef Sivic
- Cordelia Schmid
Affiliations:
- Antoine Yang: †Inria Paris and D´epartement d’informatique de l’ENS, CNRS, PSL Research University
- Arsha Nagrani, Paul Hongsuck Seo, Jordi Pont-Tuset, and Cordelia Schmid: §Google Research
- Antoine Miech: ♯ DeepMind
- Ivan Laptev: †Inria Paris and D´epartement d’informatique de l’ENS, CNRS, PSL Research University
- Josef Sivic: ¶Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague. The paper presents Vid2Seq, a single-stage dense event captioning model that uses multi-modal input and is pretrained on narrated videos. The model augments a language model with time tokens to predict event boundaries and textual descriptions in the same output sequence. The authors leverage unlabeled narrated videos for this task by reformulating sentence boundaries as pseudo event boundaries and using transcribed speech sentences as event captions. Vid2Seq improves the state of the art on various benchmarks and generalizes well to video paragraph and clip captioning tasks in few-shot settings.

CVPR 2023 - Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos

In this episode we discuss Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos
by Authors:
- Kun Su*
- Kaizhi Qian
- Eli Shlizerman
- Antonio Torralba
- Chuang Gan
Affiliations:
- Kun Su: University of Washington
- Kaizhi Qian: MIT-IBM Watson AI Lab
- Eli Shlizerman: University of Washington
- Antonio Torralba: MIT
- Chuang Gan: MIT-IBM Watson AI Lab, UMass Amherst. The paper proposes a physics-driven diffusion model to synthesize high-fidelity impact sound for silent video clips. The model incorporates physics priors, including physics parameters estimated from noisy real-world impact sound examples and learned residual parameters interpreting the sound environment via neural networks. The diffusion model combines physics priors and visual information for impact sound synthesis. Experimental results show that the proposed model outperforms existing systems in generating realistic impact sounds while maintaining interpretability and transparency for sound editing.

CVPR 2023 - Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

In this episode we discuss Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
by Authors:
- Feng Liang
- Bichen Wu
- Xiaoliang Dai
- Kunpeng Li
- Yinan Zhao
- Hang Zhang
- Peizhao Zhang
- Peter Vajda
- Diana Marculescu
Affiliations:
- Feng Liang and Diana Marculescu are affiliated with The University of Texas at Austin.
- Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Peizhao Zhang, Peter Vajda are affiliated with Meta Reality Labs.
- Hang Zhang is affiliated with Cruise.. The paper proposes a method to improve the performance of open-vocabulary semantic segmentation, which involves segmenting an image into semantic regions according to text descriptions that may not have been seen during training. The current two-stage approach involves generating class-agnostic mask proposals and then using pre-trained vision-language models like CLIP to classify masked regions. However, the authors identify the bottleneck of this approach to be the pre-trained CLIP model, which doesn't perform well on masked images. To address this issue, they propose fine-tuning CLIP on a collection of masked image regions and their corresponding text descriptions, collected by mining an existing image-caption dataset. They also use a method called "mask prompt tuning" to utilize the "blank" areas in masked images. The authors demonstrate that their method achieves significant improvement over the previous state-of-the-art on the ADE20K-150 dataset.

CVPR 2023 - DrapeNet: Garment Generation and Self-Supervised Draping

In this episode we discuss DrapeNet: Garment Generation and Self-Supervised Draping
by Authors:
- Luca De Luigi
- Ren Li
- Benoît Guillard
- Mathieu Salzmann
- Pascal Fua
Affiliations:
- Luca De Luigi: University of Bologna, luca.deluigi4@unibo.it
- Ren Li, Benoît Guillard, Mathieu Salzmann, Pascal Fua: CVLab, EPFL, {name.surname}@epfl.ch. The paper presents a new approach to drape garments over human bodies using self-supervision to train a single network for multiple garments instead of one network per clothing item. The network predicts a 3D deformation field based on the latent codes of a generative network that models garments as unsigned distance fields. The approach enables the generation and draping of previously unseen garments with different topologies, which can be edited by manipulating their latent codes. The fully differentiable formulation also allows for accurate 3D modeling of garments from partial observations. The code is publicly available.

CVPR 2023 - Planning-oriented Autonomous Driving

In this episode we discuss Planning-oriented Autonomous Driving
by Authors:
- Yihan Hu
- Jiazhi Yang
- Li Chen
- Keyu Li
- Chonghao Sima
- Xizhou Zhu
- Siqi Chai
- Senyao Du
- Tianwei Lin
- Wenhai Wang
- Lewei Lu
- Xiaosong Jia
- Qiang Liu
- Jifeng Dai
- Yu Qiao
- Hongyang Li
Affiliations:
- Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, and Xiaosong Jia: OpenDriveLab and OpenGVLab, Shanghai AI Laboratory
- Siqi Chai, Senyao Du, Tianwei Lin, and Qiang Liu: Wuhan University
- Wenhai Wang and Hongyang Li: OpenDriveLab and OpenGVLab, Shanghai AI Laboratory (†Project lead)
- Lewei Lu: SenseTime Research. The paper discusses how current autonomous driving systems use standalone modules or a multi-task paradigm, which can lead to errors or poor task coordination. The authors propose a framework called Unified Autonomous Driving (UniAD) that prioritizes tasks based on their contribution to planning and incorporates full-stack driving tasks in one network. They tested UniAD on the nuScenes benchmark and showed it outperformed previous state-of-the-art methods in all aspects. The code and models are publicly available.

CVPR 2023 - Align and Attend: Multimodal Summarization with Dual Contrastive Losses

In this episode we discuss Align and Attend: Multimodal Summarization with Dual Contrastive Losses
by Authors:
- Bo He
- Jun Wang
- Jielin Qiu
- Trung Bui
- Abhinav Shrivastava
- Zhaowen Wang
Affiliations:
- Bo He, Jun Wang, and Abhinav Shrivastava: University of Maryland, College Park
- Jielin Qiu: Carnegie Mellon University
- Trung Bui and Zhaowen Wang: Adobe Research. The paper proposes a new approach called Align and Attend Multimodal Summarization (A2Summ) for extracting important information from multiple modalities to create reliable summaries. It introduces a unified transformer-based model that aligns and attends to the multimodal input, while also addressing the issue of ignoring temporal correspondence between different modalities and intrinsic correlation between different samples. The proposed model achieves state-of-the-art performance on standard video summarization and multimodal summarization datasets and the authors also introduce a new large-scale multimodal summarization dataset called BLiSS.

CVPR 2023 - MobileNeRF: Exploiting the Polygon Rasterization Pipeline

Paper titled MobileNeRF: Exploiting the Polygon Rasterization Pipeline. 

The paper was published in CVPR 2023 conference by Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. The paper introduces a new representation of Neural Radiance Fields, called MobileNeRF, that can render 3D scenes at interactive frame rates on a wide range of compute platforms, including mobile phones.