AI Breakdown

CVPR 2023, highlight paper - DiffRF: Rendering-Guided 3D Radiance Field Diffusion

In this episode we discuss DiffRF: Rendering-Guided 3D Radiance Field Diffusion
by Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Matthias Nießner. The paper introduces a novel approach for 3D radiance field synthesis called DiffRF, which is based on denoising diffusion probabilistic models. Unlike existing diffusion-based methods that operate on images, latent codes, or point cloud data, DiffRF directly generates volumetric radiance fields. The model addresses the challenge of obtaining ground truth radiance field samples by pairing the denoising formulation with a rendering loss. DiffRF learns multi-view consistent priors, enabling free-view synthesis and accurate shape generation, and naturally enables conditional generation such as masked completion or single-view 3D synthesis at inference time.

CVPR 2023, highlight paper - SPARF: Neural Radiance Fields from Sparse and Noisy Poses

In this episode we discuss SPARF: Neural Radiance Fields from Sparse and Noisy Poses
by Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, Federico Tombari. This paper introduces Sparse Pose Adjusting Radiance Field (SPARF), a method for synthesizing photorealistic novel views with only a few input images and noisy camera poses. SPARF uses multi-view geometry constraints to jointly learn the Neural Radiance Field (NeRF) and refine the camera poses. The approach sets a new state-of-the-art in the sparse-view regime on multiple challenging datasets by enforcing a global and geometrically accurate solution through a multi-view correspondence objective and depth consistency loss.

CVPR 2023, highlight paper - F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories

In this episode we discuss F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories by Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, Wenping Wang. The paper presents a new grid-based NeRF called F2-NeRF which allows arbitrary input camera trajectories and is faster to train. Existing fast grid-based NeRF training frameworks are designed for bounded scenes and rely on space warping but cannot process arbitrary trajectories. The paper proposes a new space-warping method called perspective warping to handle unbounded scenes and demonstrates its effectiveness through experiments on standard and newly collected datasets.

CVPR 2023, highlight paper - Learning Human-to-Robot Handovers from Point Clouds

In this episode we discuss Learning Human-to-Robot Handovers from Point Clouds
by Sammy Christen, Wei Yang, Claudia Pérez-D'Arpino, Otmar Hilliges, Dieter Fox, Yu-Wei Chao. The paper proposes the first framework to teach robots how to perform vision-based human-to-robot handovers, a crucial task for human-robot interaction. The authors leverage recent advances in realistic simulations for handovers and introduce a method trained with a two-stage teacher-student framework, motion and grasp planning, reinforcement learning, and self-supervision. They report significant performance gains over baselines, both in simulation and in real-world transfer.

CVPR 2023, highlight paper - Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion

In this episode we discuss Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion
by Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, Baining Guo. The paper proposes a 3D generative model called "Rodin" that uses diffusion models to create 3D digital avatars represented as neural radiance fields in an efficient way. The model utilizes 3D-aware convolution to attend to projected features in the 2D feature plane, preserving the integrity of diffusion in 3D while achieving computational efficiency. The paper also includes latent conditioning for global coherence and hierarchical synthesis to enhance avatar details. The results show that Rodin can generate highly detailed avatars with realistic hairstyles and facial hair.

CVPR 2023, highlight paper - HairStep: Transfer Synthetic to Real Using Strand and Depth Maps for Single-View 3D Hair Modeling

In this episode we discuss HairStep: Transfer Synthetic to Real Using Strand and Depth Maps for Single-View 3D Hair Modeling
by Yujian Zheng, Zirong Jin, Moran Li, Haibin Huang, Chongyang Ma, Shuguang Cui, Xiaoguang Han. The paper discusses the problem of learning-based single-view 3D hair modelling, which has difficulties in collecting real image and 3D hair data. Using synthetic data as prior knowledge for real domain introduces a challenge of domain gap, which can be bridged using orientation maps instead of hair images as input. However, existing methods using orientation maps are sensitive to noise and far from a competent representation. Therefore, the paper proposes a novel intermediate representation called HairStep, which consists of a strand map and a depth map, providing sufficient information for accurate 3D hair modelling and feasible to be inferred from real images. The proposed approach achieves state-of-the-art performance on single-view 3D hair reconstruction.

CVPR 2023, highlight paper - LaserMix for Semi-Supervised LiDAR Semantic Segmentation

In this episode we discuss LaserMix for Semi-Supervised LiDAR Semantic Segmentation
by Lingdong Kong, Jiawei Ren, Liang Pan, Ziwei Liu. The paper proposes a semi-supervised learning framework, called LaserMix, for LiDAR semantic segmentation, leveraging the strong spatial cues of LiDAR point clouds to better exploit unlabeled data. The framework mixes laser beams from different LiDAR scans and then encourages the model to make consistent and confident predictions before and after mixing. The framework is demonstrated to be effective with comprehensive experimental analysis on popular LiDAR segmentation datasets, achieving competitive results over fully-supervised counterparts with fewer labels and improving the supervised-only baseline significantly by 10.8%.

CVPR 2023, highlight paper - Neural Volumetric Memory for Visual Locomotion Control

In this episode we discuss Neural Volumetric Memory for Visual Locomotion Control
by Ruihan Yang, Ge Yang, Xiaolong Wang. The paper discusses the use of legged robots for autonomous locomotion on challenging terrains using a forward-facing depth camera. Due to the partial observability of the terrain, the robot has to rely on past observations to infer the terrain currently beneath it. The authors propose a new memory architecture called Neural Volumetric Memory (NVM), which explicitly models the 3D geometry of the scene and aggregates feature volumes from multiple camera views. The approach was tested on a physical robot and showed superior performance compared to other methods, with representations stored in the neural volumetric memory capturing sufficient geometric information to reconstruct the scene.

CVPR 2023, highlight paper - Normal-guided Garment UV Prediction for Human Re-texturing

In this episode we discuss Normal-guided Garment UV Prediction for Human Re-texturing
by Yasamin Jafarian, Tuanfeng Y. Wang, Duygu Ceylan, Jimei Yang, Nathan Carr, Yi Zhou, Hyun Soo Park. The paper presents a method to edit dressed human images and videos without the need for 3D reconstruction of dynamic clothing. The approach estimates a geometry aware texture map between the garment region in an image and the texture space, using 3D surface normals predicted from the image. The method captures the underlying geometry of the garment in a self-supervised way and outperforms state-of-the-art human UV map estimation approaches on both real and synthetic data.

CVPR 2023, highlight paper - HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling

In this episode we discuss HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
by Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, Changil Kim. The paper discusses the challenges of creating a memory-efficient and high-quality 6-DoF (Six-Degrees-of-Freedom) video representation for dynamic scenes. Existing methods fail to achieve real-time rendering, high-quality rendering, and small memory footprint for challenging real-world scenes. To address these issues, the authors present HyperReel - a novel 6-DoF video representation that relies on a ray-conditioned sample prediction network and a memory-efficient dynamic volume representation. The system achieves state-of-the-art quality, real-time rendering, and small memory requirements, even for scenes with view-dependent appearance.