AI Breakdown

CVPR 2023 - Spatio-Temporal Pixel-Level Contrastive Learning-based Source-Free

In this episode we discuss Spatio-Temporal Pixel-Level Contrastive Learning-based Source-Free
by Authors:
- Shao-Yuan Lo
- Poojan Oza
- Sumanth Chennupati
- Alejandro Galindo
- Vishal M. Patel
Affiliations:
- Shao-Yuan Lo: Johns Hopkins University
- Poojan Oza: Amazon
- Sumanth Chennupati: Amazon
- Alejandro Galindo: Amazon
- Vishal M. Patel: Johns Hopkins University. The paper discusses unsupervised domain adaptation (UDA) of semantic segmentation, which transfers labeled source knowledge to an unlabeled target domain by accessing both the source and target data. However, access to the source data is often restricted or infeasible, making UDA less practical. To address this, recent works have explored Source-Free Domain Adaptation (SFDA), but current SFDA approaches use only image-level information, which is sub-optimal for video applications. The paper proposes a Spatio-Temporal Pixel-Level (STPL) contrastive learning method that uses spatio-temporal information to tackle the absence of source data better, achieving state-of-the-art performance on Video Semantic Segmentation (VSS) benchmarks compared to existing UDA and SFDA approaches.

CVPR 2023 - Ego-Body Pose Estimation via Ego-Head Pose Estimation

In this episode we discuss Ego-Body Pose Estimation via Ego-Head Pose Estimation
by Authors:
- Jiaman Li
- C. Karen Liu†
- Jiajun Wu†
Affiliation:
- Stanford University
Contact:
- {jiamanli,karenliu,jiajunwu}@cs.stanford.edu. The paper proposes a new method, EgoEgo, for estimating 3D human motion from egocentric video sequences that addresses two challenges. The first challenge is that the user's body is often unobserved by the front-facing camera, and the second challenge is that collecting large-scale, high-quality datasets with paired egocentric videos and 3D human motions requires accurate motion capture devices. EgoEgo decomposes the problem into two stages connected by the head motion as an intermediate representation. It first estimates accurate head motion using SLAM and a learning approach and then generates multiple plausible full-body motions using conditional diffusion. The approach eliminates the need for paired egocentric video and human motion datasets, enabling the leverage of large-scale datasets separately. The EgoEgo model performs significantly better than the current state-of-the-art methods on both a synthetic dataset developed by the authors and real data.

CVPR 2023 - MobileNeRF: Exploiting the Polygon Rasterization Pipeline

In this episode we discuss MobileNeRF: Exploiting the Polygon Rasterization Pipeline
by Authors:
- Zhiqin Chen
- Thomas Funkhouser
- Peter Hedman
- Andrea Tagliasacchi
Affiliations:
- Zhiqin Chen: 1,2
- Thomas Funkhouser: 1
- Peter Hedman: 1
- Andrea Tagliasacchi: 1,2,3
1: Google Research
2: Simon Fraser University
3: University of Toronto. The paper introduces a new representation for Neural Radiance Fields (NeRFs) based on textured polygons that can synthesize novel images efficiently with standard rendering pipelines. This approach enables NeRFs to be rendered with the traditional polygon rasterization pipeline, achieving interactive frame rates even on mobile devices. The NeRF is represented as a set of polygons with textures representing binary opacities and feature vectors interpreted by a small, view-dependent MLP running in a fragment shader to produce a final pixel color.

CVPR 2023 - TimeBalance: Temporally-Invariant and Temporally-Distinctive Video

In this episode we discuss TimeBalance: Temporally-Invariant and Temporally-Distinctive Video
by Authors: Ishan Rajendrakumar Dave, Mamshad Nayeem Rizve, Chen Chen, Mubarak Shah
Affiliation: Center for Research in Computer Vision, University of Central Florida, Orlando, USA
Contact Emails: {ishandave, nayeemrizve}@knights.ucf.edu, {chen.chen, shah}@crcv.ucf.edu. The paper proposes a semi-supervised learning framework called TimeBalance for video domain tasks that utilizes self-supervised representations. Unlike existing methods that rely on hard input inductive biases, TimeBalance utilizes a temporally-invariant and a temporally-distinctive teacher to distill knowledge from unlabeled videos based on a novel temporal similarity-based reweighting scheme. The method achieves state-of-the-art performance on three action recognition benchmarks: UCF101, HMDB51, and Kinetics400. Code for TimeBalance is available on GitHub.

CVPR 2023 - On Distillation of Guided Diffusion Models

In this episode we discuss On Distillation of Guided Diffusion Models
by Authors:
- Chenlin Meng
- Robin Rombach
- Ruiqi Gao
- Diederik Kingma
- Stefano Ermon
- Jonathan Ho
- Tim Salimans
Affiliations:
- Chenlin Meng: Stanford University
- Robin Rombach: Stability AI & LMU Munich
- Ruiqi Gao, Diederik Kingma, Jonathan Ho, Tim Salimans: Google Research, Brain Team
- Stefano Ermon: Stanford University. The paper proposes an approach to distill classifier-free guided diffusion models, a type of high-resolution image generation model, into faster models that require fewer sampling steps. This is achieved by first learning a single model to match the output of the combined conditional and unconditional models and then progressively distilling that model to a diffusion model. The approach is shown to generate visually comparable images to the original model while being up to 256 times faster to sample from on ImageNet 64x64 and CIFAR-10 datasets. It also achieves high-fidelity images using as few as 1 to 4 denoising steps on ImageNet 256x256 and LAION datasets for diffusion models trained on the latent-space, and is effective in text-guided image editing and inpainting.

CVPR 2023 - Vid2Seq: Large-Scale Pretraining of a Visual Language Model

In this episode we discuss Vid2Seq: Large-Scale Pretraining of a Visual Language Model
by Authors:
- Antoine Yang
- Arsha Nagrani
- Paul Hongsuck Seo
- Antoine Miech
- Jordi Pont-Tuset
- Ivan Laptev
- Josef Sivic
- Cordelia Schmid
Affiliations:
- Antoine Yang: †Inria Paris and D´epartement d’informatique de l’ENS, CNRS, PSL Research University
- Arsha Nagrani, Paul Hongsuck Seo, Jordi Pont-Tuset, and Cordelia Schmid: §Google Research
- Antoine Miech: ♯ DeepMind
- Ivan Laptev: †Inria Paris and D´epartement d’informatique de l’ENS, CNRS, PSL Research University
- Josef Sivic: ¶Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague. The paper presents Vid2Seq, a single-stage dense event captioning model that uses multi-modal input and is pretrained on narrated videos. The model augments a language model with time tokens to predict event boundaries and textual descriptions in the same output sequence. The authors leverage unlabeled narrated videos for this task by reformulating sentence boundaries as pseudo event boundaries and using transcribed speech sentences as event captions. Vid2Seq improves the state of the art on various benchmarks and generalizes well to video paragraph and clip captioning tasks in few-shot settings.

CVPR 2023 - Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos

In this episode we discuss Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos
by Authors:
- Kun Su*
- Kaizhi Qian
- Eli Shlizerman
- Antonio Torralba
- Chuang Gan
Affiliations:
- Kun Su: University of Washington
- Kaizhi Qian: MIT-IBM Watson AI Lab
- Eli Shlizerman: University of Washington
- Antonio Torralba: MIT
- Chuang Gan: MIT-IBM Watson AI Lab, UMass Amherst. The paper proposes a physics-driven diffusion model to synthesize high-fidelity impact sound for silent video clips. The model incorporates physics priors, including physics parameters estimated from noisy real-world impact sound examples and learned residual parameters interpreting the sound environment via neural networks. The diffusion model combines physics priors and visual information for impact sound synthesis. Experimental results show that the proposed model outperforms existing systems in generating realistic impact sounds while maintaining interpretability and transparency for sound editing.

CVPR 2023 - Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

In this episode we discuss Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
by Authors:
- Feng Liang
- Bichen Wu
- Xiaoliang Dai
- Kunpeng Li
- Yinan Zhao
- Hang Zhang
- Peizhao Zhang
- Peter Vajda
- Diana Marculescu
Affiliations:
- Feng Liang and Diana Marculescu are affiliated with The University of Texas at Austin.
- Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Peizhao Zhang, Peter Vajda are affiliated with Meta Reality Labs.
- Hang Zhang is affiliated with Cruise.. The paper proposes a method to improve the performance of open-vocabulary semantic segmentation, which involves segmenting an image into semantic regions according to text descriptions that may not have been seen during training. The current two-stage approach involves generating class-agnostic mask proposals and then using pre-trained vision-language models like CLIP to classify masked regions. However, the authors identify the bottleneck of this approach to be the pre-trained CLIP model, which doesn't perform well on masked images. To address this issue, they propose fine-tuning CLIP on a collection of masked image regions and their corresponding text descriptions, collected by mining an existing image-caption dataset. They also use a method called "mask prompt tuning" to utilize the "blank" areas in masked images. The authors demonstrate that their method achieves significant improvement over the previous state-of-the-art on the ADE20K-150 dataset.

CVPR 2023 - DrapeNet: Garment Generation and Self-Supervised Draping

In this episode we discuss DrapeNet: Garment Generation and Self-Supervised Draping
by Authors:
- Luca De Luigi
- Ren Li
- Benoît Guillard
- Mathieu Salzmann
- Pascal Fua
Affiliations:
- Luca De Luigi: University of Bologna, luca.deluigi4@unibo.it
- Ren Li, Benoît Guillard, Mathieu Salzmann, Pascal Fua: CVLab, EPFL, {name.surname}@epfl.ch. The paper presents a new approach to drape garments over human bodies using self-supervision to train a single network for multiple garments instead of one network per clothing item. The network predicts a 3D deformation field based on the latent codes of a generative network that models garments as unsigned distance fields. The approach enables the generation and draping of previously unseen garments with different topologies, which can be edited by manipulating their latent codes. The fully differentiable formulation also allows for accurate 3D modeling of garments from partial observations. The code is publicly available.

CVPR 2023 - Planning-oriented Autonomous Driving

In this episode we discuss Planning-oriented Autonomous Driving
by Authors:
- Yihan Hu
- Jiazhi Yang
- Li Chen
- Keyu Li
- Chonghao Sima
- Xizhou Zhu
- Siqi Chai
- Senyao Du
- Tianwei Lin
- Wenhai Wang
- Lewei Lu
- Xiaosong Jia
- Qiang Liu
- Jifeng Dai
- Yu Qiao
- Hongyang Li
Affiliations:
- Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, and Xiaosong Jia: OpenDriveLab and OpenGVLab, Shanghai AI Laboratory
- Siqi Chai, Senyao Du, Tianwei Lin, and Qiang Liu: Wuhan University
- Wenhai Wang and Hongyang Li: OpenDriveLab and OpenGVLab, Shanghai AI Laboratory (†Project lead)
- Lewei Lu: SenseTime Research. The paper discusses how current autonomous driving systems use standalone modules or a multi-task paradigm, which can lead to errors or poor task coordination. The authors propose a framework called Unified Autonomous Driving (UniAD) that prioritizes tasks based on their contribution to planning and incorporates full-stack driving tasks in one network. They tested UniAD on the nuScenes benchmark and showed it outperformed previous state-of-the-art methods in all aspects. The code and models are publicly available.