AI Breakdown

CVPR 2023 - Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

In this episode we discuss Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
by Authors:
- Feng Liang
- Bichen Wu
- Xiaoliang Dai
- Kunpeng Li
- Yinan Zhao
- Hang Zhang
- Peizhao Zhang
- Peter Vajda
- Diana Marculescu
Affiliations:
- Feng Liang and Diana Marculescu are affiliated with The University of Texas at Austin.
- Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Peizhao Zhang, Peter Vajda are affiliated with Meta Reality Labs.
- Hang Zhang is affiliated with Cruise.. The paper proposes a method to improve the performance of open-vocabulary semantic segmentation, which involves segmenting an image into semantic regions according to text descriptions that may not have been seen during training. The current two-stage approach involves generating class-agnostic mask proposals and then using pre-trained vision-language models like CLIP to classify masked regions. However, the authors identify the bottleneck of this approach to be the pre-trained CLIP model, which doesn't perform well on masked images. To address this issue, they propose fine-tuning CLIP on a collection of masked image regions and their corresponding text descriptions, collected by mining an existing image-caption dataset. They also use a method called "mask prompt tuning" to utilize the "blank" areas in masked images. The authors demonstrate that their method achieves significant improvement over the previous state-of-the-art on the ADE20K-150 dataset.

CVPR 2023 - DrapeNet: Garment Generation and Self-Supervised Draping

In this episode we discuss DrapeNet: Garment Generation and Self-Supervised Draping
by Authors:
- Luca De Luigi
- Ren Li
- Benoît Guillard
- Mathieu Salzmann
- Pascal Fua
Affiliations:
- Luca De Luigi: University of Bologna, luca.deluigi4@unibo.it
- Ren Li, Benoît Guillard, Mathieu Salzmann, Pascal Fua: CVLab, EPFL, {name.surname}@epfl.ch. The paper presents a new approach to drape garments over human bodies using self-supervision to train a single network for multiple garments instead of one network per clothing item. The network predicts a 3D deformation field based on the latent codes of a generative network that models garments as unsigned distance fields. The approach enables the generation and draping of previously unseen garments with different topologies, which can be edited by manipulating their latent codes. The fully differentiable formulation also allows for accurate 3D modeling of garments from partial observations. The code is publicly available.

CVPR 2023 - Planning-oriented Autonomous Driving

In this episode we discuss Planning-oriented Autonomous Driving
by Authors:
- Yihan Hu
- Jiazhi Yang
- Li Chen
- Keyu Li
- Chonghao Sima
- Xizhou Zhu
- Siqi Chai
- Senyao Du
- Tianwei Lin
- Wenhai Wang
- Lewei Lu
- Xiaosong Jia
- Qiang Liu
- Jifeng Dai
- Yu Qiao
- Hongyang Li
Affiliations:
- Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, and Xiaosong Jia: OpenDriveLab and OpenGVLab, Shanghai AI Laboratory
- Siqi Chai, Senyao Du, Tianwei Lin, and Qiang Liu: Wuhan University
- Wenhai Wang and Hongyang Li: OpenDriveLab and OpenGVLab, Shanghai AI Laboratory (†Project lead)
- Lewei Lu: SenseTime Research. The paper discusses how current autonomous driving systems use standalone modules or a multi-task paradigm, which can lead to errors or poor task coordination. The authors propose a framework called Unified Autonomous Driving (UniAD) that prioritizes tasks based on their contribution to planning and incorporates full-stack driving tasks in one network. They tested UniAD on the nuScenes benchmark and showed it outperformed previous state-of-the-art methods in all aspects. The code and models are publicly available.

CVPR 2023 - Align and Attend: Multimodal Summarization with Dual Contrastive Losses

In this episode we discuss Align and Attend: Multimodal Summarization with Dual Contrastive Losses
by Authors:
- Bo He
- Jun Wang
- Jielin Qiu
- Trung Bui
- Abhinav Shrivastava
- Zhaowen Wang
Affiliations:
- Bo He, Jun Wang, and Abhinav Shrivastava: University of Maryland, College Park
- Jielin Qiu: Carnegie Mellon University
- Trung Bui and Zhaowen Wang: Adobe Research. The paper proposes a new approach called Align and Attend Multimodal Summarization (A2Summ) for extracting important information from multiple modalities to create reliable summaries. It introduces a unified transformer-based model that aligns and attends to the multimodal input, while also addressing the issue of ignoring temporal correspondence between different modalities and intrinsic correlation between different samples. The proposed model achieves state-of-the-art performance on standard video summarization and multimodal summarization datasets and the authors also introduce a new large-scale multimodal summarization dataset called BLiSS.

CVPR 2023 - MobileNeRF: Exploiting the Polygon Rasterization Pipeline

Paper titled MobileNeRF: Exploiting the Polygon Rasterization Pipeline. 

The paper was published in CVPR 2023 conference by Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. The paper introduces a new representation of Neural Radiance Fields, called MobileNeRF, that can render 3D scenes at interactive frame rates on a wide range of compute platforms, including mobile phones.

CVPR 2023 - EXIF as Language: Learning Cross-Modal

This episode is about the paper titled “EXIF as Language: Learning Cross-Modal” at CVPR 2023, by Chenhao Zheng, Ayush Shrivastava, and Andrew Owens from University of Michigan. In this paper, the authors propose a new approach to learning visual representations, which captures information about the camera that recorded a given photo.