AI Breakdown

CVPR 2023 - SpaText: Spatio-Textual Representation for Controllable Image Generation

In this episode we discuss SpaText: Spatio-Textual Representation for Controllable Image Generation
by Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, Xi Yin. The paper presents SpaText, a new method for text-to-image generation that allows for open-vocabulary scene control. By providing a global text prompt and annotated segmentation map with free-form natural language descriptions, SpaText enables fine-grained control over the shapes and layout of different regions and objects in the generated images. The method leverages CLIP-based spatio-textual representation and extends the classifier-free guidance method in diffusion models to the multi-conditional case, achieving state-of-the-art results in image generation with free-form textual scene control.

CVPR 2023 - Neural Part Priors: Learning to Optimize Part-Based Object Completion in

In this episode we discuss Neural Part Priors: Learning to Optimize Part-Based Object Completion in
by Alexey Bokhovkin, Angela Dai. The paper proposes learning Neural Part Priors (NPPs) to improve 3D scene understanding. NPPs are parametric spaces of objects and their parts that allow for optimization to fit new input 3D scans while maintaining global scene consistency. The use of coordinate field MLPs facilitates optimization at test time, resulting in more accurate reconstructions and outperforming the state-of-the-art in part decomposition and object completion on the ScanNet dataset. The proposed method improves both object understanding and global scene consistency.

CVPR 2023 - Active Finetuning: Exploiting Annotation Budget

In this episode we discuss Active Finetuning: Exploiting Annotation Budget
by Yichen Xie, Han Lu, Junchi Yan, Xiaokang Yang, Masayoshi Tomizuka, Wei Zhan. The paper proposes a new paradigm called "active finetuning" for computer vision tasks, which focuses on selecting samples for annotation in pretraining-finetuning. The proposed method, called ActiveFT, selects a subset of data that is similar in distribution to the entire unlabeled pool and maintains diversity by optimizing a parametric model in the continuous space. The experiments show that ActiveFT outperforms baselines on image classification and semantic segmentation. The code is available on GitHub.

CVPR 2023 - Towards Bridging the Performance Gaps of Joint Energy-based Models

In this episode we discuss Towards Bridging the Performance Gaps of Joint Energy-based Models
by Xiulong Yang, Qing Su, Shihao Ji. The paper introduces a variety of training techniques to improve the performance of the Joint Energy-based Model (JEM), which combines a discriminative and a generative model in a single network. The proposed techniques aim to bridge the accuracy gap in classification and the generation quality gap compared to state-of-the-art generative models. The authors incorporate a sharpness-aware minimization framework and exclude data augmentation from the maximum likelihood estimate pipeline to achieve state-of-the-art performance in image classification, generation, calibration, out-of-distribution detection, and adversarial robustness on multiple datasets.

CVPR 2023 - ZBS: Zero-shot Background Subtraction via Instance-level Background

In this episode we discuss ZBS: Zero-shot Background Subtraction via Instance-level Background
by Yongqi An, Xu Zhao, Tao Yu, Haiyun Guo, Chaoyang Zhao, Ming Tang, Jinqiao Wang. The paper presents an unsupervised background subtraction (BGS) algorithm based on zero-shot object detection called Zero-shot Background Subtraction (ZBS). The proposed method uses zero-shot object detection to build an open-vocabulary instance-level background model, which can effectively extract foreground objects by comparing detection results with the background model. ZBS performs well in sophisticated scenarios and can detect objects outside predefined categories. The experimental results show that ZBS outperforms state-of-the-art unsupervised BGS methods by 4.70% F-Measure on the CDnet 2014 dataset. The code is available at https://github.com/CASIA-IVA-Lab/ZBS.

CVPR 2023 - HyperCUT: Video Sequence from a Single Blurry Image

In this episode we discuss HyperCUT: Video Sequence from a Single Blurry Image
by Bang-Dang Pham, Phong Tran, Anh Tran, Cuong Pham, Rang Nguyen, Minh Hoai. The paper proposes an effective self-supervised ordering scheme for training image-to-video deblurring models. The challenge of this task is the ambiguity of frame ordering. The proposed method maps each video sequence to a vector in a latent high-dimensional space and assigns an explicit order for each sequence to avoid order-ambiguity issues. The authors also propose a real-image dataset for the image-to-video deblurring problem that covers popular domains such as face, hand, and street. Experimental results confirm the effectiveness of the method.

CVPR 2023 - Feature Shrinkage Pyramid for Camouflaged Object Detection

In this episode we discuss Feature Shrinkage Pyramid for Camouflaged Object Detection by Authors: - Zhou Huang - Hang Dai - Tian-Zhu Xiang - Shuo Wang - Huai-Xin Chen - Jie Qin - Huan Xiong Affiliations: - Zhou Huang: Sichuan Changhong Electric Co., Ltd., China; UESTC, China - Hang Dai: University of Glasgow, UK - Tian-Zhu Xiang: G42, UAE - Shuo Wang: ETH Zurich, Switzerland - Huai-Xin Chen: 2UESTC, China - Jie Qin: 6CCST, NUAA, China - Huan Xiong: MBZUAI, UAE. The paper proposes a novel transformer-based Feature Shrinkage Pyramid Network (FSPNet) to improve camouflaged object detection. Current vision transformers have limitations in locality modeling and feature aggregation, resulting in less effective detection of subtle cues from indistinguishable backgrounds. FSPNet addresses these issues with a non-local token enhancement module and a feature shrinkage decoder with adjacent interaction modules. The proposed model outperforms existing competitors on three challenging datasets, demonstrating its effectiveness in camouflaged object detection.

CVPR 2023 - A Bag-of-Prototypes Representation for Dataset-Level Applications

In this episode we discuss A Bag-of-Prototypes Representation for Dataset-Level Applications by Authors: 1. Weijie Tu 2. Weijian Deng 3. Tom Gedeon 4. Liang Zheng Affiliations: 1. Australian National University 2. Curtin University. The paper proposes a bag-of-prototypes (BoP) dataset representation for measuring the relationship between datasets for two dataset-level tasks: assessing training set suitability and test set difficulty. The BoP representation consists of a codebook of K prototypes clustered from a reference dataset and is used to obtain a K-dimensional histogram for each dataset to be encoded. Without assuming access to dataset labels, the BoP representation provides a detailed characterization of the dataset's semantic distribution and cooperates well with Jensen-Shannon divergence for measuring dataset-to-dataset similarity. The authors demonstrate the superiority of the BoP representation over existing representations on multiple benchmarks.

CVPR 2023 - Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries

In this episode we discuss Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries by Authors: Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Affiliations: 1. Photogrammetry and Remote Sensing, ETH Zurich 2. ETH AI Center, ETH Zurich.. The paper addresses the problem of reconstructing 2D floorplans from 3D scans. Unlike existing approaches that use multi-stage pipelines, the authors propose a single-stage structured prediction task using a novel Transformer architecture that generates polygons for multiple rooms in a holistic manner without intermediate stages. The method achieves state-of-the-art results on two datasets and allows for easy extension to predict semantic room types and architectural elements. The code and models are available online.

CVPR 2023 - Self-positioning Point-based Transformer for Point Cloud Understanding

In this episode we discuss Self-positioning Point-based Transformer for Point Cloud Understanding by Authors: - Jinyoung Park - Sanghyeok Lee - Sihyeon Kim - Yunyang Xiong - Hyunwoo J. Kim Affiliations: - Jinyoung Park, Sanghyeok Lee, Sihyeon Kim, and Hyunwoo J. Kim: Korea University - Yunyang Xiong: Meta Reality Labs. The paper presents a new architecture called Self-Positioning point-based Transformer (SPoTr) designed to capture local and global shape contexts in point clouds with reduced complexity. It consists of local self-attention and self-positioning point-based global cross-attention. The self-positioning points, located adaptively based on the input shape, consider both spatial and semantic information to improve expressive power, while the global cross-attention allows the attention module to compute attention weights with only a small set of self-positioning points, improving scalability. SPoTr achieves improved accuracy on three point cloud tasks and offers interpretability through the analysis of self-positioning points. Code is available on Github.