AI Breakdown

CVPR 2023 - Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations

In this episode we discuss Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations by Yiwu Zhong, Licheng Yu, Yang Bai, Shangwen Li, Xueting Yan, Yin Li. The paper proposes a method to learn a video representation that encodes both action steps and their temporal ordering from a large-scale dataset of web instructional videos without human annotations. The method involves jointly learning a video representation for individual step concepts and a deep probabilistic model to capture temporal dependencies and individual variations in the step ordering. The model achieves significant improvements in step classification and forecasting as well as promising results in zero-shot inference and predicting diverse and plausible steps for incomplete procedures. The code is available on GitHub.

CVPR 2023 - Focused and Collaborative Feedback Integration

In this episode we discuss Focused and Collaborative Feedback Integration
by Qiaoqiao Wei, Hui Zhang, Jun-Hai Yong. The paper proposes Focused and Collaborative Feedback Integration (FCFI), an approach for click-based interactive image segmentation. FCFI fully exploits feedback by focusing on a local area around the new click and correcting the feedback based on high-level feature similarities. It updates the feedback and deep features collaboratively, achieving state-of-the-art performance with less computational overhead than previous methods on four benchmarks. The source code is available on GitHub.

CVPR 2023 - Seeing What You Said: Talking Face Generation Guided

In this episode we discuss Seeing What You Said: Talking Face Generation Guided
by Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T. Tan, Haizhou Li. The paper discusses the generation of talking faces, also known as speech-to-lip generation, which reconstructs facial motions concerning lips based on speech input. The authors propose using a lip-reading expert to improve the intelligibility of the generated lip regions by penalizing incorrect generation results. They also introduce contrastive learning and a transformer in their approach to enhance lip-speech synchronization and audio-video encoding. The proposal achieved superior performance in reading intelligibility and lip-speech synchronization compared to other state-of-the-art methods.

CVPR 2023 - Instance-Aware Domain Generalization for Face Anti-Spoofing

In this episode we discuss Instance-Aware Domain Generalization for Face Anti-Spoofing
by Qianyu Zhou, Ke-Yue Zhang, Taiping Yao, Xuequan Lu, Ran Yi, Shouhong Ding, Lizhuang Ma. The paper discusses the development of a Face Anti-Spoofing (FAS) system based on Domain Generalization (DG) which aligns features on the instance level without relying on domain labels. This is in contrast to previous methods that focused on domain-level alignment and used artificial domain labels that did not accurately reflect real domain distributions. The proposed Instance-Aware Domain Generalization framework utilizes Asymmetric Instance Adaptive Whitening, Dynamic Kernel Generator, and Categorical Style Assembly to improve generalization and eliminate style-sensitive feature correlation. The paper concludes that their method outperforms state-of-the-art competitors.

CVPR 2023 - SpaText: Spatio-Textual Representation for Controllable Image Generation

In this episode we discuss SpaText: Spatio-Textual Representation for Controllable Image Generation
by Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, Xi Yin. The paper presents SpaText, a new method for text-to-image generation that allows for open-vocabulary scene control. By providing a global text prompt and annotated segmentation map with free-form natural language descriptions, SpaText enables fine-grained control over the shapes and layout of different regions and objects in the generated images. The method leverages CLIP-based spatio-textual representation and extends the classifier-free guidance method in diffusion models to the multi-conditional case, achieving state-of-the-art results in image generation with free-form textual scene control.

CVPR 2023 - Neural Part Priors: Learning to Optimize Part-Based Object Completion in

In this episode we discuss Neural Part Priors: Learning to Optimize Part-Based Object Completion in
by Alexey Bokhovkin, Angela Dai. The paper proposes learning Neural Part Priors (NPPs) to improve 3D scene understanding. NPPs are parametric spaces of objects and their parts that allow for optimization to fit new input 3D scans while maintaining global scene consistency. The use of coordinate field MLPs facilitates optimization at test time, resulting in more accurate reconstructions and outperforming the state-of-the-art in part decomposition and object completion on the ScanNet dataset. The proposed method improves both object understanding and global scene consistency.

CVPR 2023 - Active Finetuning: Exploiting Annotation Budget

In this episode we discuss Active Finetuning: Exploiting Annotation Budget
by Yichen Xie, Han Lu, Junchi Yan, Xiaokang Yang, Masayoshi Tomizuka, Wei Zhan. The paper proposes a new paradigm called "active finetuning" for computer vision tasks, which focuses on selecting samples for annotation in pretraining-finetuning. The proposed method, called ActiveFT, selects a subset of data that is similar in distribution to the entire unlabeled pool and maintains diversity by optimizing a parametric model in the continuous space. The experiments show that ActiveFT outperforms baselines on image classification and semantic segmentation. The code is available on GitHub.

CVPR 2023 - Towards Bridging the Performance Gaps of Joint Energy-based Models

In this episode we discuss Towards Bridging the Performance Gaps of Joint Energy-based Models
by Xiulong Yang, Qing Su, Shihao Ji. The paper introduces a variety of training techniques to improve the performance of the Joint Energy-based Model (JEM), which combines a discriminative and a generative model in a single network. The proposed techniques aim to bridge the accuracy gap in classification and the generation quality gap compared to state-of-the-art generative models. The authors incorporate a sharpness-aware minimization framework and exclude data augmentation from the maximum likelihood estimate pipeline to achieve state-of-the-art performance in image classification, generation, calibration, out-of-distribution detection, and adversarial robustness on multiple datasets.

CVPR 2023 - ZBS: Zero-shot Background Subtraction via Instance-level Background

In this episode we discuss ZBS: Zero-shot Background Subtraction via Instance-level Background
by Yongqi An, Xu Zhao, Tao Yu, Haiyun Guo, Chaoyang Zhao, Ming Tang, Jinqiao Wang. The paper presents an unsupervised background subtraction (BGS) algorithm based on zero-shot object detection called Zero-shot Background Subtraction (ZBS). The proposed method uses zero-shot object detection to build an open-vocabulary instance-level background model, which can effectively extract foreground objects by comparing detection results with the background model. ZBS performs well in sophisticated scenarios and can detect objects outside predefined categories. The experimental results show that ZBS outperforms state-of-the-art unsupervised BGS methods by 4.70% F-Measure on the CDnet 2014 dataset. The code is available at https://github.com/CASIA-IVA-Lab/ZBS.

CVPR 2023 - HyperCUT: Video Sequence from a Single Blurry Image

In this episode we discuss HyperCUT: Video Sequence from a Single Blurry Image
by Bang-Dang Pham, Phong Tran, Anh Tran, Cuong Pham, Rang Nguyen, Minh Hoai. The paper proposes an effective self-supervised ordering scheme for training image-to-video deblurring models. The challenge of this task is the ambiguity of frame ordering. The proposed method maps each video sequence to a vector in a latent high-dimensional space and assigns an explicit order for each sequence to avoid order-ambiguity issues. The authors also propose a real-image dataset for the image-to-video deblurring problem that covers popular domains such as face, hand, and street. Experimental results confirm the effectiveness of the method.