AI Breakdown

CVPR 2023 - Iterative Next Boundary Detection for Instance Segmentation of Tree Rings in Microscopy Images of Shrub Cross Sections

In this episode we discuss Iterative Next Boundary Detection for Instance Segmentation of Tree Rings in Microscopy Images of Shrub Cross Sections
by Alexander Gillert, Giulia Resente, Alba Anadon-Rosell, Martin Wilmking, Uwe Freiherr von Lukas. The paper proposes a new iterative method called Iterative Next Boundary Detection (INBD) for detecting tree rings in microscopy images of shrub cross sections. This is a difficult task due to the concentric circular ring shape of the objects and the high precision requirements. INBD models the natural growth direction, starting from the center of the shrub cross section and detecting the next ring boundary in each iteration step, and outperforms existing methods in experiments. The dataset and source code are also made available.

CVPR 2023 - Towards Unified Scene Text Spotting based on Sequence Generation

In this episode we discuss Towards Unified Scene Text Spotting based on Sequence Generation
by Taeho Kil, Seonghyeon Kim, Sukmin Seo, Yoonsik Kim, Daehee Kim. The proposed paper presents a UNIfied scene Text Spotter, called UNITS, to overcome the limitations of auto-regressive models used for end-to-end text spotting. UNITS unifies various detection formats, allowing it to detect text in arbitrary shapes, and applies starting-point prompting to extract more texts beyond the number of instances it was trained on. Experimental results show that UNITS achieves competitive performance compared to state-of-the-art methods and can extract a larger number of texts than it was trained on. Code for the method is provided on GitHub.

CVPR 2023 - Learning Neural Duplex Radiance Fields for Real-Time View Synthesis

In this episode we discuss Learning Neural Duplex Radiance Fields for Real-Time View Synthesis
by Ziyu Wan, Christian Richardt, Aljaž Božič, Chao Li, Vijay Rengarajan, Seonghyeon Nam, Xiaoyu Xiang, Tuotuo Li, Bo Zhu, Rakesh Ranjan, Jing Liao. The paper proposes a novel approach to rendering photorealistic images using Neural Radiance Fields (NeRFs) in a more efficient manner. NeRFs require hundreds of deep MLP evaluations for each pixel, which is prohibitively expensive for real-time rendering. The proposed approach overcomes this by distilling and baking NeRFs into highly efficient mesh-based neural representations that are compatible with the massively parallel graphics rendering pipeline. The approach uses screen-space convolutions instead of MLPs to exploit local geometric relationships between nearby pixels and is further boosted by a multi-view distillation optimization strategy. Extensive experiments demonstrate the effectiveness and superiority of the approach on a range of standard datasets.

CVPR 2023 - Context-Based Trit-Plane Coding for Progressive Image Compression

In this episode we discuss Context-Based Trit-Plane Coding for Progressive Image Compression
by Seungmin Jeon, Kwang Pyo Choi, Youngo Park, Chang-Su Kim. The paper proposes the context-based trit-plane coding (CTC) algorithm for progressive image compression. CTC enables compact encoding of trit-planes by developing a context-based rate reduction module to estimate trit probabilities accurately. The context-based distortion reduction module refines partial latent tensors from the trit-planes to improve image quality. The proposed CTC algorithm outperforms the baseline trit-plane codec significantly and increases time complexity marginally.

CVPR 2023 - Interactive Cartoonization with Controllable Perceptual Factors

In this episode we discuss Interactive Cartoonization with Controllable Perceptual Factors
by Namhyuk Ahn, Patrick Kwon, Jihye Back, Kibeom Hong, Seungkwon Kim. The paper proposes a new method for cartoonization, which involves rendering natural photos into cartoon styles with editing features of texture and color. The proposed method uses a model architecture with separate decoders for texture and color, and introduces a texture controller to generate diverse cartoon textures. Additionally, an HSV color augmentation is used to induce the networks to generate diverse and controllable color translation, resulting in profound quality improvement over baselines. This is the first deep approach that allows control of the cartoonization at inference.

CVPR 2023 - Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning

In this episode we discuss Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning
by Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Son Dinh Tran, Yi Xu, Belinda Zeng, Trishul Chilimbi. The paper discusses the use of contrastive loss in learning representations from multiple modalities. It argues that perfect modality alignment is sub-optimal for downstream prediction tasks and proposes three approaches to construct meaningful latent modality structures. The proposed approach achieves consistent improvements over existing methods on various multi-modal tasks and demonstrates its effectiveness and generalizability.

CVPR 2023 - MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving

In this episode we discuss MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving
by Jiale Li, Hang Dai, Hao Han, Yong Ding. This paper proposes a multi-modal 3D semantic segmentation model (MSeg3D) for autonomous driving, combining LiDAR and camera data. The authors address several challenges with multi-modal solutions, including modality heterogeneity, limited sensor field of view intersection, and multi-modal data augmentation. MSeg3D uses joint intra-modal feature extraction and inter-modal feature fusion, and achieves state-of-the-art results on several datasets. The authors also provide their code on GitHub for public use.

arxiv preprint - PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360$^{\circ}$

In this episode we discuss PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360$^{\circ}$
by Sizhe An, Hongyi Xu, Yichun Shi, Guoxian Song, Umit Ogras, Linjie Luo. The paper introduces PanoHead, a 3D-aware generative model that can synthesize high-quality, view-consistent images of full heads in 360 degrees. Existing 3D generative adversarial networks (GANs) struggle to preserve 3D consistency in large view angles, but PanoHead addresses this by using unstructured images for training and implementing a two-stage self-adaptive image alignment. The authors also propose a tri-grid neural volume representation that effectively handles front-face and back-head feature entanglement, resulting in high-quality 3D head synthesis with accurate geometry and diverse appearances.

CVPR 2023 - OmniMAE: Single Model Masked Pretraining on Images and Videos

In this episode we discuss OmniMAE: Single Model Masked Pretraining on Images and Videos
by Authors:
- Rohit Girdhar
- Alaaeldin El-Nouby
- Mannat Singh
- Kalyan Vasudev Alwala
- Armand Joulin
- Ishan Misra
Affiliation:
- FAIR, Meta AI. The paper discusses how a common architecture can be used to train a single unified model for multiple visual modalities, namely images and videos, using masked autoencoding. The proposed vision transformer model achieves comparable or better visual representations than single-modality representations on both image and video benchmarks, without requiring any labeled data. Additionally, the model can be trained efficiently by dropping a large proportion of image and video patches. The proposed model achieves new state-of-the-art performance on the ImageNet and Something Something-v2 video benchmarks.

CVPR 2023 - NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination

In this episode we discuss NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination
by Haoqian Wu, Zhipeng Hu, Lincheng Li, Yongqiang Zhang, Changjie Fan, Xin Yu. The paper proposes an end-to-end inverse rendering pipeline that decomposes materials and illumination from multi-view images, while considering near-field indirect illumination. They introduce Monte Carlo sampling based path tracing, cache the indirect illumination as neural radiance, and leverage Spherical Gaussians to represent smooth environment illuminations and apply importance sampling techniques to enhance efficiency. They also develop a novel radiance consistency constraint between implicit neural radiance and path tracing results of unobserved rays to significantly improve decomposition performance. Experimental results demonstrate that their method outperforms state-of-the-art methods on multiple synthetic and real datasets.