AI Breakdown

CVPR 2023 - Feature Shrinkage Pyramid for Camouflaged Object Detection

In this episode we discuss Feature Shrinkage Pyramid for Camouflaged Object Detection by Authors: - Zhou Huang - Hang Dai - Tian-Zhu Xiang - Shuo Wang - Huai-Xin Chen - Jie Qin - Huan Xiong Affiliations: - Zhou Huang: Sichuan Changhong Electric Co., Ltd., China; UESTC, China - Hang Dai: University of Glasgow, UK - Tian-Zhu Xiang: G42, UAE - Shuo Wang: ETH Zurich, Switzerland - Huai-Xin Chen: 2UESTC, China - Jie Qin: 6CCST, NUAA, China - Huan Xiong: MBZUAI, UAE. The paper proposes a novel transformer-based Feature Shrinkage Pyramid Network (FSPNet) to improve camouflaged object detection. Current vision transformers have limitations in locality modeling and feature aggregation, resulting in less effective detection of subtle cues from indistinguishable backgrounds. FSPNet addresses these issues with a non-local token enhancement module and a feature shrinkage decoder with adjacent interaction modules. The proposed model outperforms existing competitors on three challenging datasets, demonstrating its effectiveness in camouflaged object detection.

CVPR 2023 - A Bag-of-Prototypes Representation for Dataset-Level Applications

In this episode we discuss A Bag-of-Prototypes Representation for Dataset-Level Applications by Authors: 1. Weijie Tu 2. Weijian Deng 3. Tom Gedeon 4. Liang Zheng Affiliations: 1. Australian National University 2. Curtin University. The paper proposes a bag-of-prototypes (BoP) dataset representation for measuring the relationship between datasets for two dataset-level tasks: assessing training set suitability and test set difficulty. The BoP representation consists of a codebook of K prototypes clustered from a reference dataset and is used to obtain a K-dimensional histogram for each dataset to be encoded. Without assuming access to dataset labels, the BoP representation provides a detailed characterization of the dataset's semantic distribution and cooperates well with Jensen-Shannon divergence for measuring dataset-to-dataset similarity. The authors demonstrate the superiority of the BoP representation over existing representations on multiple benchmarks.

CVPR 2023 - Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries

In this episode we discuss Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries by Authors: Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Affiliations: 1. Photogrammetry and Remote Sensing, ETH Zurich 2. ETH AI Center, ETH Zurich.. The paper addresses the problem of reconstructing 2D floorplans from 3D scans. Unlike existing approaches that use multi-stage pipelines, the authors propose a single-stage structured prediction task using a novel Transformer architecture that generates polygons for multiple rooms in a holistic manner without intermediate stages. The method achieves state-of-the-art results on two datasets and allows for easy extension to predict semantic room types and architectural elements. The code and models are available online.

CVPR 2023 - Self-positioning Point-based Transformer for Point Cloud Understanding

In this episode we discuss Self-positioning Point-based Transformer for Point Cloud Understanding by Authors: - Jinyoung Park - Sanghyeok Lee - Sihyeon Kim - Yunyang Xiong - Hyunwoo J. Kim Affiliations: - Jinyoung Park, Sanghyeok Lee, Sihyeon Kim, and Hyunwoo J. Kim: Korea University - Yunyang Xiong: Meta Reality Labs. The paper presents a new architecture called Self-Positioning point-based Transformer (SPoTr) designed to capture local and global shape contexts in point clouds with reduced complexity. It consists of local self-attention and self-positioning point-based global cross-attention. The self-positioning points, located adaptively based on the input shape, consider both spatial and semantic information to improve expressive power, while the global cross-attention allows the attention module to compute attention weights with only a small set of self-positioning points, improving scalability. SPoTr achieves improved accuracy on three point cloud tasks and offers interpretability through the analysis of self-positioning points. Code is available on Github.

CVPR 2023 - Decoupled Multimodal Distilling for Emotion Recognition

In this episode we discuss Decoupled Multimodal Distilling for Emotion Recognition by Authors: Yong Li, Yuanzhi Wang, Zhen Cui Affiliation: PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China. The paper proposes a decoupled multimodal distillation approach for human multimodal emotion recognition (MER). The proposed approach mitigates the issue of multimodal heterogeneities by enhancing the discriminative features of each modality through crossmodal knowledge distillation. A graph distillation unit (GD-Unit) is used for each decoupled part, and the GD paradigm provides a flexible knowledge transfer manner where the distillation weights can be automatically learned, enabling diverse crossmodal knowledge transfer patterns. The experimental results show that the proposed approach consistently outperforms state-of-the-art MER methods, and the visualization results exhibit meaningful distributional patterns w.r.t. the modality-irrelevant/-exclusive feature spaces.

CVPR 2023 - DistractFlow: Improving Optical Flow Estimation via

In this episode we discuss DistractFlow: Improving Optical Flow Estimation via  Realistic Distractions and Pseudo Labeling by Authors: Jisoo Jeong, Hong Cai, Risheek Garrepalli, Fatih Porikli Affiliation: Qualcomm AI Research†. The paper proposes a novel data augmentation technique, called DistractFlow, for training optical flow estimation models. This approach introduces distractions to the input frames, using a mixing ratio to combine one of the frames in the pair with a distractor image depicting a similar domain. The distracted pairs allow the model to learn related variations and become robust against challenging deviations. The approach can be applied to training any optical flow estimation models and improves existing models, outperforming the latest state of the art on multiple benchmarks.

CVPR 2023 - Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style

In this episode we discuss Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style by Authors: 1. Fengyin Lin 2. Mingkang Li 3. Da Li 4. Timothy Hospedales 5. Yi-Zhe Song Affiliations: 1. Beijing University of Posts and Telecommunications 2. Samsung AI Centre, Cambridge 3. University of Edinburgh 4. SketchX, CVSSP, University of Surrey. The paper presents a novel approach to zero-shot sketch-based image retrieval (ZS-SBIR) that tackles all variants of the problem using just one network. The authors aim to make the matching process more explainable, and achieve this through a transformer-based cross-modal network that compares groups of key local patches. The network includes three novel components: a self-attention module, a cross-attention module, and a kernel-based relation network. Experiment results show superior performance across all ZS-SBIR settings, and the explainable goal is achieved through visualizing cross-modal token correspondences and sketch to photo synthesis. Code and models are available for reproducibility.

CVPR 2023 - Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields

In this episode we discuss Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields by Author 1: Yue Chen Affiliation: Xi’an Jiaotong University Author 2: Xingyu Chen Affiliation: Xi’an Jiaotong University Author 3: Xuan Wang Affiliation: Ant Group Author 4: Qi Zhang Affiliation: Tencent AI Lab Author 5: Yu Guo Affiliation: Xi’an Jiaotong University Author 6: Ying Shan Affiliation: Tencent AI Lab Author 7: Fei Wang Affiliation: Xi’an Jiaotong University. The paper proposes a method called L2G-NeRF for bundle-adjusting Neural Radiance Fields (NeRF). NeRF has achieved realistic synthesis of novel views but is limited by the requirement of accurate camera poses. L2G-NeRF performs pixel-wise flexible alignment followed by frame-wise constrained parametric alignment to improve high-fidelity reconstruction and resolve large camera pose misalignment. The method outperforms current state-of-the-art and is an easy-to-use plugin that can be applied to NeRF variants and other neural field applications.

CVPR 2023 - A New Benchmark: On the Utility of Synthetic Data with Blender for

In this episode we discuss A New Benchmark: On the Utility of Synthetic Data with Blender for by Authors: Hui Tang and Kui Jia. The paper discusses the limitations of deep learning in computer vision due to the need for large-scale labeled training data and the impracticality of exhaustive data annotation. To address this, the authors propose generating synthetic data via 3D rendering with domain randomization. Through their research, they systematically verify important learning insights and discover new laws of various data regimes and network architectures in generalization. They also investigate the effect of image formation factors on generalization and use simulation-to-reality adaptation as a downstream task for comparing the transferability between synthetic and real data for pre-training. Finally, they develop a new benchmark for image classification, S2RDA, to provide more significant challenges for transfer from simulation to reality.

CVPR 2023 - NeRFLiX: High-Quality Neural View Synthesis

In this episode we discuss NeRFLiX: High-Quality Neural View Synthesis by Authors: - Kun Zhou - Wenbo Li - Yi Wang - Tao Hu - Nianjuan Jiang - Xiaoguang Han - Jiangbo Lu. The paper proposes NeRFLiX, a degradation-driven inter-viewpoint mixer which is a general NeRF-agnostic restorer paradigm for improving the synthesis quality of NeRF-based approaches. NeRFs are successful in novel view synthesis but suffer from rendering artifacts such as noise and blur, and imperfect calibration information. NeRFLiX removes these artifacts and improves performance by fusing highly related, high-quality training images using an inter-viewpoint aggregation framework. Large-scale training data and a degradation modeling approach are utilized to achieve these improvements.