AI Breakdown

CVPR 2023 - MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving

In this episode we discuss MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving
by Jiale Li, Hang Dai, Hao Han, Yong Ding. This paper proposes a multi-modal 3D semantic segmentation model (MSeg3D) for autonomous driving, combining LiDAR and camera data. The authors address several challenges with multi-modal solutions, including modality heterogeneity, limited sensor field of view intersection, and multi-modal data augmentation. MSeg3D uses joint intra-modal feature extraction and inter-modal feature fusion, and achieves state-of-the-art results on several datasets. The authors also provide their code on GitHub for public use.

arxiv preprint - PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360$^{\circ}$

In this episode we discuss PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360$^{\circ}$
by Sizhe An, Hongyi Xu, Yichun Shi, Guoxian Song, Umit Ogras, Linjie Luo. The paper introduces PanoHead, a 3D-aware generative model that can synthesize high-quality, view-consistent images of full heads in 360 degrees. Existing 3D generative adversarial networks (GANs) struggle to preserve 3D consistency in large view angles, but PanoHead addresses this by using unstructured images for training and implementing a two-stage self-adaptive image alignment. The authors also propose a tri-grid neural volume representation that effectively handles front-face and back-head feature entanglement, resulting in high-quality 3D head synthesis with accurate geometry and diverse appearances.

CVPR 2023 - OmniMAE: Single Model Masked Pretraining on Images and Videos

In this episode we discuss OmniMAE: Single Model Masked Pretraining on Images and Videos
by Authors:
- Rohit Girdhar
- Alaaeldin El-Nouby
- Mannat Singh
- Kalyan Vasudev Alwala
- Armand Joulin
- Ishan Misra
Affiliation:
- FAIR, Meta AI. The paper discusses how a common architecture can be used to train a single unified model for multiple visual modalities, namely images and videos, using masked autoencoding. The proposed vision transformer model achieves comparable or better visual representations than single-modality representations on both image and video benchmarks, without requiring any labeled data. Additionally, the model can be trained efficiently by dropping a large proportion of image and video patches. The proposed model achieves new state-of-the-art performance on the ImageNet and Something Something-v2 video benchmarks.

CVPR 2023 - NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination

In this episode we discuss NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination
by Haoqian Wu, Zhipeng Hu, Lincheng Li, Yongqiang Zhang, Changjie Fan, Xin Yu. The paper proposes an end-to-end inverse rendering pipeline that decomposes materials and illumination from multi-view images, while considering near-field indirect illumination. They introduce Monte Carlo sampling based path tracing, cache the indirect illumination as neural radiance, and leverage Spherical Gaussians to represent smooth environment illuminations and apply importance sampling techniques to enhance efficiency. They also develop a novel radiance consistency constraint between implicit neural radiance and path tracing results of unobserved rays to significantly improve decomposition performance. Experimental results demonstrate that their method outperforms state-of-the-art methods on multiple synthetic and real datasets.

CVPR 2023 - PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos

In this episode we discuss PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos
by Zhiqiang Shen, Xiaoxiao Sheng, Longguang Wang, Yulan Guo, Qiong Liu, Xi Zhou. The paper proposed a self-supervised learning framework, called PointCMP, for point cloud videos, in which high labeling costs make unsupervised methods appealing. PointCMP uses a two-branch structure to simultaneously learn local and global spatio-temporal information. The framework includes a mutual similarity-based augmentation module to generate hard samples for better discrimination and generalization performance, resulting in state-of-the-art performance on benchmark datasets and outperforming fully-supervised methods. Transfer learning experiments also demonstrate the superior quality of representations learned with PointCMP.

CVPR 2023 - A Strong Baseline for Generalized Few-Shot Semantic Segmentation

In this episode we discuss A Strong Baseline for Generalized Few-Shot Semantic Segmentation
by Sina Hajimiri, Malik Boudiaf, Ismail Ben Ayed, Jose Dolz. The paper focuses on introducing a generalized few-shot segmentation framework with a simple and easy-to-optimize inference phase and training process. They propose a model based on the InfoMax principle, where the Mutual Information (MI) between the learned feature representations and their corresponding predictions is maximized. The proposed model improves the few-shot segmentation benchmarks, PASCAL-5i and COCO-20i, by 7% to 26% and 3% to 12%, respectively, for novel classes in 1-shot and 5-shot scenarios. The code used in the study is publicly available.

CVPR 2023 - MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision

In this episode we discuss MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision
by Antoine Guédon, Tom Monnier, Pascal Monasse, Vincent Lepetit. The paper introduces a method that can learn to explore and reconstruct large environments in 3D from color images only, without relying on depth sensors or 3D supervision. The method learns to predict a "volume occupancy field" from color images and uses it to identify the Next Best View (NBV) to improve scene coverage. As a result, the method performs well on new scenes and outperforms recent methods that require depth sensors, making it a more realistic option for outdoor scenes captured with a drone.

CVPR 2023 - Stare at What You See: Masked Image Modeling without Reconstruction

In this episode we discuss Stare at What You See: Masked Image Modeling without Reconstruction
by Hongwei Xue, Peng Gao, Hongyang Li, Yu Qiao, Hao Sun, Houqiang Li, Jiebo Luo. The paper proposes a new approach to Masked Image Modeling (MIM) called MaskAlign. The authors argue that the features extracted by powerful teacher models already contain rich semantic correlations across regions in an intact image, eliminating the need for reconstruction. MaskAlign learns the consistency of visible patch features extracted by the student model and intact image features extracted by the teacher model, and uses a Dynamic Alignment (DA) module to tackle input inconsistency between them. The proposed approach achieves state-of-the-art performance with higher efficiency and is available on GitHub.

CVPR 2023 - SimpleNet: A Simple Network for Image Anomaly Detection and Localization

In this episode we discuss SimpleNet: A Simple Network for Image Anomaly Detection and Localization
by Zhikang Liu, Yiming Zhou, Yuansheng Xu, Zilei Wang. The paper introduces a new deep learning network called SimpleNet for detecting and localizing anomalies. SimpleNet has four main components that include a pre-trained Feature Extractor, a shallow Feature Adapter, a simple Anomaly Feature Generator, and a binary Anomaly Discriminator. The authors base their approach on three intuitions which involve transforming pre-trained features to target-oriented features, generating synthetic anomalies in feature space, and using a simple discriminator. SimpleNet performs better than previous methods on the MVTec AD benchmark with an anomaly detection AUROC of 99.6% and a high frame rate of 77 FPS on a 3080ti GPU.

CVPR 2023 - Self-Supervised Video Forensics by Audio-Visual Anomaly Detection

In this episode we discuss Self-Supervised Video Forensics by Audio-Visual Anomaly Detection
by Chao Feng, Ziyang Chen, Andrew Owens. The paper proposes a method for detecting inconsistencies between the visual and audio signals in manipulated videos using anomaly detection. The method trains an autoregressive model on real, unlabeled data to generate audio-visual feature sequences capturing temporal synchronization. The model flags videos with a low probability of being genuine at test time and achieves strong performance in detecting manipulated speech videos, despite being trained solely on real videos.