AI Breakdown

CVPR 2023 - Context-Based Trit-Plane Coding for Progressive Image Compression

In this episode we discuss Context-Based Trit-Plane Coding for Progressive Image Compression
by Seungmin Jeon, Kwang Pyo Choi, Youngo Park, Chang-Su Kim. The paper proposes the context-based trit-plane coding (CTC) algorithm for progressive image compression. CTC enables compact encoding of trit-planes by developing a context-based rate reduction module to estimate trit probabilities accurately. The context-based distortion reduction module refines partial latent tensors from the trit-planes to improve image quality. The proposed CTC algorithm outperforms the baseline trit-plane codec significantly and increases time complexity marginally.

CVPR 2023 - Interactive Cartoonization with Controllable Perceptual Factors

In this episode we discuss Interactive Cartoonization with Controllable Perceptual Factors
by Namhyuk Ahn, Patrick Kwon, Jihye Back, Kibeom Hong, Seungkwon Kim. The paper proposes a new method for cartoonization, which involves rendering natural photos into cartoon styles with editing features of texture and color. The proposed method uses a model architecture with separate decoders for texture and color, and introduces a texture controller to generate diverse cartoon textures. Additionally, an HSV color augmentation is used to induce the networks to generate diverse and controllable color translation, resulting in profound quality improvement over baselines. This is the first deep approach that allows control of the cartoonization at inference.

CVPR 2023 - Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning

In this episode we discuss Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning
by Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Son Dinh Tran, Yi Xu, Belinda Zeng, Trishul Chilimbi. The paper discusses the use of contrastive loss in learning representations from multiple modalities. It argues that perfect modality alignment is sub-optimal for downstream prediction tasks and proposes three approaches to construct meaningful latent modality structures. The proposed approach achieves consistent improvements over existing methods on various multi-modal tasks and demonstrates its effectiveness and generalizability.

CVPR 2023 - MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving

In this episode we discuss MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving
by Jiale Li, Hang Dai, Hao Han, Yong Ding. This paper proposes a multi-modal 3D semantic segmentation model (MSeg3D) for autonomous driving, combining LiDAR and camera data. The authors address several challenges with multi-modal solutions, including modality heterogeneity, limited sensor field of view intersection, and multi-modal data augmentation. MSeg3D uses joint intra-modal feature extraction and inter-modal feature fusion, and achieves state-of-the-art results on several datasets. The authors also provide their code on GitHub for public use.

arxiv preprint - PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360$^{\circ}$

In this episode we discuss PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360$^{\circ}$
by Sizhe An, Hongyi Xu, Yichun Shi, Guoxian Song, Umit Ogras, Linjie Luo. The paper introduces PanoHead, a 3D-aware generative model that can synthesize high-quality, view-consistent images of full heads in 360 degrees. Existing 3D generative adversarial networks (GANs) struggle to preserve 3D consistency in large view angles, but PanoHead addresses this by using unstructured images for training and implementing a two-stage self-adaptive image alignment. The authors also propose a tri-grid neural volume representation that effectively handles front-face and back-head feature entanglement, resulting in high-quality 3D head synthesis with accurate geometry and diverse appearances.

CVPR 2023 - OmniMAE: Single Model Masked Pretraining on Images and Videos

In this episode we discuss OmniMAE: Single Model Masked Pretraining on Images and Videos
by Authors:
- Rohit Girdhar
- Alaaeldin El-Nouby
- Mannat Singh
- Kalyan Vasudev Alwala
- Armand Joulin
- Ishan Misra
Affiliation:
- FAIR, Meta AI. The paper discusses how a common architecture can be used to train a single unified model for multiple visual modalities, namely images and videos, using masked autoencoding. The proposed vision transformer model achieves comparable or better visual representations than single-modality representations on both image and video benchmarks, without requiring any labeled data. Additionally, the model can be trained efficiently by dropping a large proportion of image and video patches. The proposed model achieves new state-of-the-art performance on the ImageNet and Something Something-v2 video benchmarks.

CVPR 2023 - NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination

In this episode we discuss NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination
by Haoqian Wu, Zhipeng Hu, Lincheng Li, Yongqiang Zhang, Changjie Fan, Xin Yu. The paper proposes an end-to-end inverse rendering pipeline that decomposes materials and illumination from multi-view images, while considering near-field indirect illumination. They introduce Monte Carlo sampling based path tracing, cache the indirect illumination as neural radiance, and leverage Spherical Gaussians to represent smooth environment illuminations and apply importance sampling techniques to enhance efficiency. They also develop a novel radiance consistency constraint between implicit neural radiance and path tracing results of unobserved rays to significantly improve decomposition performance. Experimental results demonstrate that their method outperforms state-of-the-art methods on multiple synthetic and real datasets.

CVPR 2023 - PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos

In this episode we discuss PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos
by Zhiqiang Shen, Xiaoxiao Sheng, Longguang Wang, Yulan Guo, Qiong Liu, Xi Zhou. The paper proposed a self-supervised learning framework, called PointCMP, for point cloud videos, in which high labeling costs make unsupervised methods appealing. PointCMP uses a two-branch structure to simultaneously learn local and global spatio-temporal information. The framework includes a mutual similarity-based augmentation module to generate hard samples for better discrimination and generalization performance, resulting in state-of-the-art performance on benchmark datasets and outperforming fully-supervised methods. Transfer learning experiments also demonstrate the superior quality of representations learned with PointCMP.

CVPR 2023 - A Strong Baseline for Generalized Few-Shot Semantic Segmentation

In this episode we discuss A Strong Baseline for Generalized Few-Shot Semantic Segmentation
by Sina Hajimiri, Malik Boudiaf, Ismail Ben Ayed, Jose Dolz. The paper focuses on introducing a generalized few-shot segmentation framework with a simple and easy-to-optimize inference phase and training process. They propose a model based on the InfoMax principle, where the Mutual Information (MI) between the learned feature representations and their corresponding predictions is maximized. The proposed model improves the few-shot segmentation benchmarks, PASCAL-5i and COCO-20i, by 7% to 26% and 3% to 12%, respectively, for novel classes in 1-shot and 5-shot scenarios. The code used in the study is publicly available.

CVPR 2023 - MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision

In this episode we discuss MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision
by Antoine Guédon, Tom Monnier, Pascal Monasse, Vincent Lepetit. The paper introduces a method that can learn to explore and reconstruct large environments in 3D from color images only, without relying on depth sensors or 3D supervision. The method learns to predict a "volume occupancy field" from color images and uses it to identify the Next Best View (NBV) to improve scene coverage. As a result, the method performs well on new scenes and outperforms recent methods that require depth sensors, making it a more realistic option for outdoor scenes captured with a drone.