AI Breakdown

CVPR 2023, highlight paper - Learning Human-to-Robot Handovers from Point Clouds

In this episode we discuss Learning Human-to-Robot Handovers from Point Clouds
by Sammy Christen, Wei Yang, Claudia Pérez-D'Arpino, Otmar Hilliges, Dieter Fox, Yu-Wei Chao. The paper proposes the first framework to teach robots how to perform vision-based human-to-robot handovers, a crucial task for human-robot interaction. The authors leverage recent advances in realistic simulations for handovers and introduce a method trained with a two-stage teacher-student framework, motion and grasp planning, reinforcement learning, and self-supervision. They report significant performance gains over baselines, both in simulation and in real-world transfer.

CVPR 2023, highlight paper - Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion

In this episode we discuss Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion
by Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, Baining Guo. The paper proposes a 3D generative model called "Rodin" that uses diffusion models to create 3D digital avatars represented as neural radiance fields in an efficient way. The model utilizes 3D-aware convolution to attend to projected features in the 2D feature plane, preserving the integrity of diffusion in 3D while achieving computational efficiency. The paper also includes latent conditioning for global coherence and hierarchical synthesis to enhance avatar details. The results show that Rodin can generate highly detailed avatars with realistic hairstyles and facial hair.

CVPR 2023, highlight paper - HairStep: Transfer Synthetic to Real Using Strand and Depth Maps for Single-View 3D Hair Modeling

In this episode we discuss HairStep: Transfer Synthetic to Real Using Strand and Depth Maps for Single-View 3D Hair Modeling
by Yujian Zheng, Zirong Jin, Moran Li, Haibin Huang, Chongyang Ma, Shuguang Cui, Xiaoguang Han. The paper discusses the problem of learning-based single-view 3D hair modelling, which has difficulties in collecting real image and 3D hair data. Using synthetic data as prior knowledge for real domain introduces a challenge of domain gap, which can be bridged using orientation maps instead of hair images as input. However, existing methods using orientation maps are sensitive to noise and far from a competent representation. Therefore, the paper proposes a novel intermediate representation called HairStep, which consists of a strand map and a depth map, providing sufficient information for accurate 3D hair modelling and feasible to be inferred from real images. The proposed approach achieves state-of-the-art performance on single-view 3D hair reconstruction.

CVPR 2023, highlight paper - LaserMix for Semi-Supervised LiDAR Semantic Segmentation

In this episode we discuss LaserMix for Semi-Supervised LiDAR Semantic Segmentation
by Lingdong Kong, Jiawei Ren, Liang Pan, Ziwei Liu. The paper proposes a semi-supervised learning framework, called LaserMix, for LiDAR semantic segmentation, leveraging the strong spatial cues of LiDAR point clouds to better exploit unlabeled data. The framework mixes laser beams from different LiDAR scans and then encourages the model to make consistent and confident predictions before and after mixing. The framework is demonstrated to be effective with comprehensive experimental analysis on popular LiDAR segmentation datasets, achieving competitive results over fully-supervised counterparts with fewer labels and improving the supervised-only baseline significantly by 10.8%.

CVPR 2023, highlight paper - Neural Volumetric Memory for Visual Locomotion Control

In this episode we discuss Neural Volumetric Memory for Visual Locomotion Control
by Ruihan Yang, Ge Yang, Xiaolong Wang. The paper discusses the use of legged robots for autonomous locomotion on challenging terrains using a forward-facing depth camera. Due to the partial observability of the terrain, the robot has to rely on past observations to infer the terrain currently beneath it. The authors propose a new memory architecture called Neural Volumetric Memory (NVM), which explicitly models the 3D geometry of the scene and aggregates feature volumes from multiple camera views. The approach was tested on a physical robot and showed superior performance compared to other methods, with representations stored in the neural volumetric memory capturing sufficient geometric information to reconstruct the scene.

CVPR 2023, highlight paper - Normal-guided Garment UV Prediction for Human Re-texturing

In this episode we discuss Normal-guided Garment UV Prediction for Human Re-texturing
by Yasamin Jafarian, Tuanfeng Y. Wang, Duygu Ceylan, Jimei Yang, Nathan Carr, Yi Zhou, Hyun Soo Park. The paper presents a method to edit dressed human images and videos without the need for 3D reconstruction of dynamic clothing. The approach estimates a geometry aware texture map between the garment region in an image and the texture space, using 3D surface normals predicted from the image. The method captures the underlying geometry of the garment in a self-supervised way and outperforms state-of-the-art human UV map estimation approaches on both real and synthetic data.

CVPR 2023, highlight paper - HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling

In this episode we discuss HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
by Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, Changil Kim. The paper discusses the challenges of creating a memory-efficient and high-quality 6-DoF (Six-Degrees-of-Freedom) video representation for dynamic scenes. Existing methods fail to achieve real-time rendering, high-quality rendering, and small memory footprint for challenging real-world scenes. To address these issues, the authors present HyperReel - a novel 6-DoF video representation that relies on a ray-conditioned sample prediction network and a memory-efficient dynamic volume representation. The system achieves state-of-the-art quality, real-time rendering, and small memory requirements, even for scenes with view-dependent appearance.

CVPR 2023, highlight paper - Learning Customized Visual Models with Retrieval-Augmented Knowledge

In this episode we discuss Learning Customized Visual Models with Retrieval-Augmented Knowledge
by Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, Chunyuan Li. The paper proposes a framework called REACT (REtrieval-Augmented CusTomization) to build customized visual models for specific domains. Instead of using expensive pre-training, REACT retrieves relevant image-text pairs from a web-scale database as external knowledge and only trains new modularized blocks while freezing original weights. The framework is shown to be effective in various tasks, including zero-shot classification, with up to 5.4% improvement on ImageNet and 3.7% on the ELEVATER benchmark compared to CLIP.

CVPR 2023, highlight paper - SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries

In this episode we discuss SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries
by Ahmed Imtiaz Humayun, Randall Balestriero, Guha Balakrishnan, Richard Baraniuk. This paper presents a new method called SplineCam that enables exact computation of the geometry of a deep network's (DN) mapping, including its decision boundary, without resorting to approximations such as sampling or architecture simplification. SplineCam works for any DN architecture based on Continuous Piece-Wise Linear (CPWL) nonlinearities and can be used for regression DNs as well. This method facilitates comparison of architectures, generalizability measurement, and sampling from the decision boundary on or off the manifold.

CVPR 2023, highlight paper - Nighttime Smartphone Reflective Flare Removal Using Optical Center Symmetry Prior

In this episode we discuss Nighttime Smartphone Reflective Flare Removal Using Optical Center Symmetry Prior
by Yuekun Dai, Yihang Luo, Shangchen Zhou, Chongyi Li, Chen Change Loy. The paper proposes a method to address the problem of reflective flare in photos caused by light reflecting inside lenses and creating bright spots or a "ghosting effect". Existing methods for detecting these bright spots often fail to identify reflective flares created by various types of light and may even mistakenly remove light sources in scenarios with multiple light sources. The proposed method uses an optical center symmetry prior, which suggests that the reflective flare and light source are always symmetrical around the lens's optical center. The authors also create a reflective flare removal dataset called BracketFlare using continuous bracketing to capture the reflective flare pattern. The method demonstrated effective results on both synthetic and real-world datasets.