AI Breakdown

CVPR 2023 - OmniMAE: Single Model Masked Pretraining on Images and Videos

In this episode we discuss OmniMAE: Single Model Masked Pretraining on Images and Videos
by Authors:
- Rohit Girdhar
- Alaaeldin El-Nouby
- Mannat Singh
- Kalyan Vasudev Alwala
- Armand Joulin
- Ishan Misra
Affiliation:
- FAIR, Meta AI. The paper discusses how a common architecture can be used to train a single unified model for multiple visual modalities, namely images and videos, using masked autoencoding. The proposed vision transformer model achieves comparable or better visual representations than single-modality representations on both image and video benchmarks, without requiring any labeled data. Additionally, the model can be trained efficiently by dropping a large proportion of image and video patches. The proposed model achieves new state-of-the-art performance on the ImageNet and Something Something-v2 video benchmarks.

CVPR 2023 - NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination

In this episode we discuss NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination
by Haoqian Wu, Zhipeng Hu, Lincheng Li, Yongqiang Zhang, Changjie Fan, Xin Yu. The paper proposes an end-to-end inverse rendering pipeline that decomposes materials and illumination from multi-view images, while considering near-field indirect illumination. They introduce Monte Carlo sampling based path tracing, cache the indirect illumination as neural radiance, and leverage Spherical Gaussians to represent smooth environment illuminations and apply importance sampling techniques to enhance efficiency. They also develop a novel radiance consistency constraint between implicit neural radiance and path tracing results of unobserved rays to significantly improve decomposition performance. Experimental results demonstrate that their method outperforms state-of-the-art methods on multiple synthetic and real datasets.

CVPR 2023 - PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos

In this episode we discuss PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos
by Zhiqiang Shen, Xiaoxiao Sheng, Longguang Wang, Yulan Guo, Qiong Liu, Xi Zhou. The paper proposed a self-supervised learning framework, called PointCMP, for point cloud videos, in which high labeling costs make unsupervised methods appealing. PointCMP uses a two-branch structure to simultaneously learn local and global spatio-temporal information. The framework includes a mutual similarity-based augmentation module to generate hard samples for better discrimination and generalization performance, resulting in state-of-the-art performance on benchmark datasets and outperforming fully-supervised methods. Transfer learning experiments also demonstrate the superior quality of representations learned with PointCMP.

CVPR 2023 - A Strong Baseline for Generalized Few-Shot Semantic Segmentation

In this episode we discuss A Strong Baseline for Generalized Few-Shot Semantic Segmentation
by Sina Hajimiri, Malik Boudiaf, Ismail Ben Ayed, Jose Dolz. The paper focuses on introducing a generalized few-shot segmentation framework with a simple and easy-to-optimize inference phase and training process. They propose a model based on the InfoMax principle, where the Mutual Information (MI) between the learned feature representations and their corresponding predictions is maximized. The proposed model improves the few-shot segmentation benchmarks, PASCAL-5i and COCO-20i, by 7% to 26% and 3% to 12%, respectively, for novel classes in 1-shot and 5-shot scenarios. The code used in the study is publicly available.

CVPR 2023 - MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision

In this episode we discuss MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision
by Antoine Guédon, Tom Monnier, Pascal Monasse, Vincent Lepetit. The paper introduces a method that can learn to explore and reconstruct large environments in 3D from color images only, without relying on depth sensors or 3D supervision. The method learns to predict a "volume occupancy field" from color images and uses it to identify the Next Best View (NBV) to improve scene coverage. As a result, the method performs well on new scenes and outperforms recent methods that require depth sensors, making it a more realistic option for outdoor scenes captured with a drone.

CVPR 2023 - Stare at What You See: Masked Image Modeling without Reconstruction

In this episode we discuss Stare at What You See: Masked Image Modeling without Reconstruction
by Hongwei Xue, Peng Gao, Hongyang Li, Yu Qiao, Hao Sun, Houqiang Li, Jiebo Luo. The paper proposes a new approach to Masked Image Modeling (MIM) called MaskAlign. The authors argue that the features extracted by powerful teacher models already contain rich semantic correlations across regions in an intact image, eliminating the need for reconstruction. MaskAlign learns the consistency of visible patch features extracted by the student model and intact image features extracted by the teacher model, and uses a Dynamic Alignment (DA) module to tackle input inconsistency between them. The proposed approach achieves state-of-the-art performance with higher efficiency and is available on GitHub.

CVPR 2023 - SimpleNet: A Simple Network for Image Anomaly Detection and Localization

In this episode we discuss SimpleNet: A Simple Network for Image Anomaly Detection and Localization
by Zhikang Liu, Yiming Zhou, Yuansheng Xu, Zilei Wang. The paper introduces a new deep learning network called SimpleNet for detecting and localizing anomalies. SimpleNet has four main components that include a pre-trained Feature Extractor, a shallow Feature Adapter, a simple Anomaly Feature Generator, and a binary Anomaly Discriminator. The authors base their approach on three intuitions which involve transforming pre-trained features to target-oriented features, generating synthetic anomalies in feature space, and using a simple discriminator. SimpleNet performs better than previous methods on the MVTec AD benchmark with an anomaly detection AUROC of 99.6% and a high frame rate of 77 FPS on a 3080ti GPU.

CVPR 2023 - Self-Supervised Video Forensics by Audio-Visual Anomaly Detection

In this episode we discuss Self-Supervised Video Forensics by Audio-Visual Anomaly Detection
by Chao Feng, Ziyang Chen, Andrew Owens. The paper proposes a method for detecting inconsistencies between the visual and audio signals in manipulated videos using anomaly detection. The method trains an autoregressive model on real, unlabeled data to generate audio-visual feature sequences capturing temporal synchronization. The model flags videos with a low probability of being genuine at test time and achieves strong performance in detecting manipulated speech videos, despite being trained solely on real videos.

CVPR 2023 - Foundation Model Drives Weakly Incremental Learning for Semantic Segmentation

In this episode we discuss Foundation Model Drives Weakly Incremental Learning for Semantic Segmentation
by Chaohui Yu, Qiang Zhou, Jingliang Li, Jianlong Yuan, Zhibin Wang, Fan Wang. The paper proposes a novel and data-efficient framework for weakly incremental learning for semantic segmentation (WILSS) called FMWISS. WILSS aims to learn to segment new classes from cheap and readily available image-level labels. The proposed framework uses pre-training based co-segmentation to generate dense pseudo labels and a teacher-student architecture to optimize noisy pseudo masks with a dense contrastive loss. Additionally, memory-based copy-paste augmentation is introduced to address the catastrophic forgetting problem of old classes. The framework achieves superior performance on Pascal VOC and COCO datasets compared to state-of-the-art methods.

CVPR 2023 - Delving into Discrete Normalizing Flows on SO(3) Manifold for Probabilistic Rotation Modeling

In this episode we discuss Delving into Discrete Normalizing Flows on SO(3) Manifold for Probabilistic Rotation Modeling
by Yulin Liu, Haoran Liu, Yingda Yin, Yang Wang, Baoquan Chen, He Wang. The paper proposes a new normalizing flow method for the SO(3) manifold, which is an important quantity in computer vision, graphics, and robotics but has unique non-Euclidean properties that make it difficult to adapt existing normalizing flows. The proposed method combines a Mobius transformation-based coupling layer and a quaternion affine transformation to effectively express arbitrary distributions on SO(3) and allows for conditional building of the target distribution given input observations. Extensive experiments show that the proposed rotation normalizing flows outperform baselines on both unconditional and conditional tasks.