AI Breakdown

CVPR 2023 - Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel

In this episode we discuss Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel
by Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, Shih-Fu Chang. The paper discusses the challenge of multi-channel video-language retrieval, which requires models to understand information from different sources such as video and text. The authors investigate different options for representing videos and fusing video and text information using a principled model design space. The evaluation of four combinations on five video-language datasets reveals that discrete text tokens with a pretrained contrastive text model perform the best, even outperforming state-of-the-art models on some datasets. The authors attribute this to the ability of text tokens to capture key visual information and align naturally with strong text retrieval models.

CVPR 2023 - Interventional Bag Multi-Instance Learning On Whole-Slide Pathological Images

In this episode we discuss Interventional Bag Multi-Instance Learning On Whole-Slide Pathological Images
by Tiancheng Lin, Zhimiao Yu, Hongyu Hu, Yi Xu, Chang Wen Chen. The paper proposes a new scheme called Interventional Bag Multi-Instance Learning (IBMIL) to improve the classification of whole slide pathological images. Existing methods focus on improving feature extraction and aggregation but may capture spurious correlations between bags and labels. IBMIL uses backdoor adjustment for interventional training to suppress bias caused by contextual priors and achieves consistent performance boosts, making it a state-of-the-art method. Code for IBMIL is available on GitHub.

CVPR 2023 - Devil is in the Queries: Advancing Mask Transformers for Real-world Medical

In this episode we discuss Devil is in the Queries: Advancing Mask Transformers for Real-world Medical
by Mingze Yuan, Yingda Xia, Hexin Dong, Zifan Chen, Jiawen Yao, Mingyan Qiu, Ke Yan, Xiaoli Yin, Yu Shi, Xin Chen, Zaiyi Liu, Bin Dong, Jingren Zhou, Le Lu, Ling Zhang, Li Zhang. The paper proposes a method for medical image segmentation that is capable of accurately identifying rare and clinically significant conditions, known as tail conditions. The method utilizes object queries in Mask Transformers to assign soft clusters during training and detect out-of-distribution (OOD) regions during inference, which is referred to as MaxQuery. The authors also introduce a query-distribution (QD) loss to improve segmentation of inliers and OOD indication. The proposed framework outperforms previous state-of-the-art algorithms on pancreatic and liver tumor segmentation tasks.

CVPR 2023 - Inverting the Imaging Process by Learning an Implicit Camera Model

In this episode we discuss Inverting the Imaging Process by Learning an Implicit Camera Model
by Xin Huang, Qi Zhang, Ying Feng, Hongdong Li, Qing Wang. The paper introduces a new approach for modeling the physical imaging process of a camera as an implicit neural network, which is able to learn and control camera parameters. This approach is tested on two challenging inverse imaging tasks: all-in-focus and HDR imaging. The results show that the new implicit neural camera model is able to produce visually appealing and accurate images, making it a promising tool for a wide range of inverse imaging tasks.

CVPR 2023 - Label-Free Liver Tumor Segmentation

In this episode we discuss Label-Free Liver Tumor Segmentation
by Qixin Hu, Yixiong Chen, Junfei Xiao, Shuwen Sun, Jieneng Chen, Alan Yuille, Zongwei Zhou. The paper discusses the use of synthetic tumors in CT scans to train AI models to accurately segment liver tumors without the need for manual annotation. These synthetic tumors are realistic in shape and texture and have proven effective in training the AI models, which demonstrated similar performance to models trained on real tumors. This highlights the potential for significantly reducing manual efforts for tumor annotation and the ability to improve the success rate of detecting small liver tumors, while also allowing for rigorous assessment of AI robustness.

CVPR 2023 - Regularized Vector Quantization for Tokenized Image Synthesis

In this episode we discuss Regularized Vector Quantization for Tokenized Image Synthesis
by Jiahui Zhang, Fangneng Zhan, Christian Theobalt, Shijian Lu. The paper proposes a regularized vector quantization framework for quantizing images into discrete representations, which has been a fundamental problem in generative modeling. Existing approaches either learn the discrete representation deterministically or stochastically, but suffer from various drawbacks, such as severe codebook collapse, low codebook utilization, and perturbed reconstruction objective. The proposed framework mitigates these issues effectively by applying regularization from two perspectives and introducing a probabilistic contrastive loss. Experiments show that the framework consistently outperforms prevailing vector quantization methods across various generative models.

CVPR 2023 - Reliability in Semantic Segmentation: Are We on the Right Track?

In this episode we discuss Reliability in Semantic Segmentation: Are We on the Right Track?
by Pau de Jorge, Riccardo Volpi, Philip Torr, Gregory Rogez. The paper discusses a study on the reliability of modern semantic segmentation models in terms of robustness and uncertainty estimation. The authors analyze a variety of models and compare their performance on four metrics: robustness, calibration, misclassification detection, and out-of-distribution detection. They find that recent models are more robust but not more reliable in terms of uncertainty estimation, and suggest improving calibration as a way to improve other uncertainty metrics. This is the first study of its kind on modern segmentation models and is intended to assist practitioners and researchers in this fundamental vision task.

CVPR 2023 - ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection

In this episode we discuss ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection
by Jeeseung Park, Jin-Woo Park, Jong-Seok Lee. The paper proposes a new method for improving the performance of human-object interaction (HOI) detectors, which are used in scene understanding. The proposed method, called Vision Transformer based Pose-Conditioned Self-Loop Graph (ViPLO), combines a novel feature extraction method with a graph structure that updates human node encoding with local features of human joints. This approach achieves state-of-the-art results on two public benchmarks, with a significant performance gain on the HICO-DET dataset. The source code is also available for public use.

CVPR 2023 - FREDOM: Fairness Domain Adaptation Approach to Semantic Scene Understanding

In this episode we discuss FREDOM: Fairness Domain Adaptation Approach to Semantic Scene Understanding
by Thanh-Dat Truong, Ngan Le, Bhiksha Raj, Jackson Cothren, Khoa Luu. The paper proposes a new approach called Fairness Domain Adaptation (FREDOM) for semantic scene segmentation that addresses fairness concerns in domain adaptation. The proposed adaptation framework is based on the fair treatment of class distributions, and a new conditional structural constraint is introduced to ensure consistency of predicted segmentation. The proposed method, which includes a Conditional Structure Network with a self-attention mechanism, outperformed existing methods in two standard benchmarks and promoted fairness in model predictions.

CVPR 2023 - Self-Supervised Learning for Multimodal Non-Rigid 3D Shape Matching

In this episode we discuss Self-Supervised Learning for Multimodal Non-Rigid 3D Shape Matching
by Dongliang Cao, Florian Bernard. The paper proposes a self-supervised multimodal learning strategy to bridge the gap between mesh-based and point cloud-based shape matching methods. Meshes provide rich topological information but require curation, while point clouds are commonly used for real-world data but lack the same matching quality. The proposed approach combines mesh-based functional map regularization with a contrastive loss that links mesh and point cloud data. Results show that the method achieves state-of-the-art performance on benchmark datasets and exhibits cross-dataset generalization ability. Code is available for use.