AI Breakdown

CVPR 2023 - Self-positioning Point-based Transformer for Point Cloud Understanding

In this episode we discuss Self-positioning Point-based Transformer for Point Cloud Understanding by Authors: - Jinyoung Park - Sanghyeok Lee - Sihyeon Kim - Yunyang Xiong - Hyunwoo J. Kim Affiliations: - Jinyoung Park, Sanghyeok Lee, Sihyeon Kim, and Hyunwoo J. Kim: Korea University - Yunyang Xiong: Meta Reality Labs. The paper presents a new architecture called Self-Positioning point-based Transformer (SPoTr) designed to capture local and global shape contexts in point clouds with reduced complexity. It consists of local self-attention and self-positioning point-based global cross-attention. The self-positioning points, located adaptively based on the input shape, consider both spatial and semantic information to improve expressive power, while the global cross-attention allows the attention module to compute attention weights with only a small set of self-positioning points, improving scalability. SPoTr achieves improved accuracy on three point cloud tasks and offers interpretability through the analysis of self-positioning points. Code is available on Github.

CVPR 2023 - Decoupled Multimodal Distilling for Emotion Recognition

In this episode we discuss Decoupled Multimodal Distilling for Emotion Recognition by Authors: Yong Li, Yuanzhi Wang, Zhen Cui Affiliation: PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China. The paper proposes a decoupled multimodal distillation approach for human multimodal emotion recognition (MER). The proposed approach mitigates the issue of multimodal heterogeneities by enhancing the discriminative features of each modality through crossmodal knowledge distillation. A graph distillation unit (GD-Unit) is used for each decoupled part, and the GD paradigm provides a flexible knowledge transfer manner where the distillation weights can be automatically learned, enabling diverse crossmodal knowledge transfer patterns. The experimental results show that the proposed approach consistently outperforms state-of-the-art MER methods, and the visualization results exhibit meaningful distributional patterns w.r.t. the modality-irrelevant/-exclusive feature spaces.

CVPR 2023 - DistractFlow: Improving Optical Flow Estimation via

In this episode we discuss DistractFlow: Improving Optical Flow Estimation via  Realistic Distractions and Pseudo Labeling by Authors: Jisoo Jeong, Hong Cai, Risheek Garrepalli, Fatih Porikli Affiliation: Qualcomm AI Research†. The paper proposes a novel data augmentation technique, called DistractFlow, for training optical flow estimation models. This approach introduces distractions to the input frames, using a mixing ratio to combine one of the frames in the pair with a distractor image depicting a similar domain. The distracted pairs allow the model to learn related variations and become robust against challenging deviations. The approach can be applied to training any optical flow estimation models and improves existing models, outperforming the latest state of the art on multiple benchmarks.

CVPR 2023 - Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style

In this episode we discuss Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style by Authors: 1. Fengyin Lin 2. Mingkang Li 3. Da Li 4. Timothy Hospedales 5. Yi-Zhe Song Affiliations: 1. Beijing University of Posts and Telecommunications 2. Samsung AI Centre, Cambridge 3. University of Edinburgh 4. SketchX, CVSSP, University of Surrey. The paper presents a novel approach to zero-shot sketch-based image retrieval (ZS-SBIR) that tackles all variants of the problem using just one network. The authors aim to make the matching process more explainable, and achieve this through a transformer-based cross-modal network that compares groups of key local patches. The network includes three novel components: a self-attention module, a cross-attention module, and a kernel-based relation network. Experiment results show superior performance across all ZS-SBIR settings, and the explainable goal is achieved through visualizing cross-modal token correspondences and sketch to photo synthesis. Code and models are available for reproducibility.

CVPR 2023 - Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields

In this episode we discuss Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields by Author 1: Yue Chen Affiliation: Xi’an Jiaotong University Author 2: Xingyu Chen Affiliation: Xi’an Jiaotong University Author 3: Xuan Wang Affiliation: Ant Group Author 4: Qi Zhang Affiliation: Tencent AI Lab Author 5: Yu Guo Affiliation: Xi’an Jiaotong University Author 6: Ying Shan Affiliation: Tencent AI Lab Author 7: Fei Wang Affiliation: Xi’an Jiaotong University. The paper proposes a method called L2G-NeRF for bundle-adjusting Neural Radiance Fields (NeRF). NeRF has achieved realistic synthesis of novel views but is limited by the requirement of accurate camera poses. L2G-NeRF performs pixel-wise flexible alignment followed by frame-wise constrained parametric alignment to improve high-fidelity reconstruction and resolve large camera pose misalignment. The method outperforms current state-of-the-art and is an easy-to-use plugin that can be applied to NeRF variants and other neural field applications.

CVPR 2023 - A New Benchmark: On the Utility of Synthetic Data with Blender for

In this episode we discuss A New Benchmark: On the Utility of Synthetic Data with Blender for by Authors: Hui Tang and Kui Jia. The paper discusses the limitations of deep learning in computer vision due to the need for large-scale labeled training data and the impracticality of exhaustive data annotation. To address this, the authors propose generating synthetic data via 3D rendering with domain randomization. Through their research, they systematically verify important learning insights and discover new laws of various data regimes and network architectures in generalization. They also investigate the effect of image formation factors on generalization and use simulation-to-reality adaptation as a downstream task for comparing the transferability between synthetic and real data for pre-training. Finally, they develop a new benchmark for image classification, S2RDA, to provide more significant challenges for transfer from simulation to reality.

CVPR 2023 - NeRFLiX: High-Quality Neural View Synthesis

In this episode we discuss NeRFLiX: High-Quality Neural View Synthesis by Authors: - Kun Zhou - Wenbo Li - Yi Wang - Tao Hu - Nianjuan Jiang - Xiaoguang Han - Jiangbo Lu. The paper proposes NeRFLiX, a degradation-driven inter-viewpoint mixer which is a general NeRF-agnostic restorer paradigm for improving the synthesis quality of NeRF-based approaches. NeRFs are successful in novel view synthesis but suffer from rendering artifacts such as noise and blur, and imperfect calibration information. NeRFLiX removes these artifacts and improves performance by fusing highly related, high-quality training images using an inter-viewpoint aggregation framework. Large-scale training data and a degradation modeling approach are utilized to achieve these improvements.

CVPR 2023 - RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors

In this episode we discuss RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors by Authors: 1. Rui-Qi Wu 2. Zheng-Peng Duan 3. Chun-Le Guo 4. Zhi Chai 5. Chongyi Li Affiliations: 1. VCIP, CS, Nankai University 2. Hisilicon Technologies Co. Ltd. 3. S-Lab, Nanyang Technological University. The paper discusses a new approach to real image dehazing, which addresses the challenges of existing methods that struggle to process real-world hazy images due to the lack of paired real data and robust priors. The proposed method synthesizes more realistic hazy data and introduces more robust priors into the network. The approach includes a phenomenological pipeline that considers diverse degradation types and a Real Image Dehazing network via high-quality Codebook Priors (RIDCP) that utilizes a VQGAN pre-trained on a large-scale high-quality dataset to obtain the discrete codebook encapsulating high-quality priors. Extensive experiments confirm the effectiveness of the proposed approach.

CVPR 2023 - Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

In this episode we discuss Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models by Authors: - Andreas Blattmann - Robin Rombach - Huan Ling - Tim Dockhorn - Seung Wook Kim - Sanja Fidler - Karsten Kreis Affiliations: - Andreas Blattmann and Robin Rombach: LMU Munich - Huan Ling, Seung Wook Kim, Sanja Fidler, and Karsten Kreis: NVIDIA, Vector Institute, and University of Toronto - Tim Dockhorn: University of Waterloo. The paper discusses the use of Latent Diffusion Models (LDMs) to generate high-quality videos without excessive computational demands. The authors pre-train an LDM on images before introducing a temporal dimension to create a video generator, and fine-tune the model on encoded image sequences to achieve state-of-the-art performance on real driving videos of resolution 512 x 1024. They also demonstrate the use of LDMs for text-to-video modeling and personalized content creation. The authors highlight the efficiency and expressiveness of their approach, which can easily leverage pre-trained image LDMs and generalize across different fine-tuned LDMs.

CVPR 2023 - Tracking through Containers and Occluders in the Wild

In this episode we discuss Tracking through Containers and Occluders in the Wild
by Authors:
- Basile Van Hoorick
- Pavel Tokmakov
- Simon Stent
- Jie Li
- Carl Vondrick
Affiliations:
- Basile Van Hoorick and Carl Vondrick: Columbia University
- Pavel Tokmakov and Jie Li: Toyota Research Institute
- Simon Stent: Woven Planet. The paper introduces a benchmark and model called TCOW for visual tracking in cluttered and dynamic environments, which is a difficult challenge for computer vision. The goal is to segment both the projected extent of the target object and the surrounding container or occluder in a given video sequence. The authors create a mixture of synthetic and annotated real datasets to support both supervised learning and structured evaluation of model performance and evaluate two transformer-based video models, finding a considerable performance gap in achieving object permanence.