AI Breakdown

CVPR 2023 - Foundation Model Drives Weakly Incremental Learning for Semantic Segmentation

In this episode we discuss Foundation Model Drives Weakly Incremental Learning for Semantic Segmentation
by Chaohui Yu, Qiang Zhou, Jingliang Li, Jianlong Yuan, Zhibin Wang, Fan Wang. The paper proposes a novel and data-efficient framework for weakly incremental learning for semantic segmentation (WILSS) called FMWISS. WILSS aims to learn to segment new classes from cheap and readily available image-level labels. The proposed framework uses pre-training based co-segmentation to generate dense pseudo labels and a teacher-student architecture to optimize noisy pseudo masks with a dense contrastive loss. Additionally, memory-based copy-paste augmentation is introduced to address the catastrophic forgetting problem of old classes. The framework achieves superior performance on Pascal VOC and COCO datasets compared to state-of-the-art methods.

CVPR 2023 - Delving into Discrete Normalizing Flows on SO(3) Manifold for Probabilistic Rotation Modeling

In this episode we discuss Delving into Discrete Normalizing Flows on SO(3) Manifold for Probabilistic Rotation Modeling
by Yulin Liu, Haoran Liu, Yingda Yin, Yang Wang, Baoquan Chen, He Wang. The paper proposes a new normalizing flow method for the SO(3) manifold, which is an important quantity in computer vision, graphics, and robotics but has unique non-Euclidean properties that make it difficult to adapt existing normalizing flows. The proposed method combines a Mobius transformation-based coupling layer and a quaternion affine transformation to effectively express arbitrary distributions on SO(3) and allows for conditional building of the target distribution given input observations. Extensive experiments show that the proposed rotation normalizing flows outperform baselines on both unconditional and conditional tasks.

CVPR 2023 - StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos

In this episode we discuss StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos
by Nikita Dvornik, Isma Hadji, Ran Zhang, Konstantinos G. Derpanis, Animesh Garg, Richard P. Wildes, Allan D. Jepson. The paper introduces StepFormer, a self-supervised model that locates key-steps in instructional videos with no human supervision. Traditional methods require video-level human annotations, which do not scale to large datasets. StepFormer uses automatically-generated subtitles as the only source of supervision and a sequence of text narrations using an order-aware loss function that filters out irrelevant phrases. The model outperforms all previous unsupervised and weakly-supervised approaches on step detection and localization and demonstrates an emergent property to solve zero-shot multi-step localization.

CVPR 2023 - SketchXAI: A First Look at Explainability for Human Sketches

In this episode we discuss SketchXAI: A First Look at Explainability for Human Sketches
by Zhiyu Qu, Yulia Gryaditskaya, Ke Li, Kaiyue Pang, Tao Xiang, Yi-Zhe Song. The paper introduces human sketches to the landscape of Explainable Artificial Intelligence (XAI). Sketch is argued to be a "human-centered" data form that represents a natural interface to study explainability. The authors design a simple explainability-friendly sketch encoder that accommodates the intrinsic properties of strokes and define the first ever XAI task for sketch, stroke location inversion (SLI). The authors offer qualitative results and snapshots of the SLI process, as well as providing code available at https://sketchxai.github.io.

CVPR 2023 - Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training

In this episode we discuss Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
by Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, Dhruv Mahajan. The paper discusses improvements to the contrastive pre-training pipeline for vision-language models used in zero-shot recognition problems. The authors propose a filtering strategy called CAT to reduce dataset size, an approach called Concept Distillation to leverage strong unimodal representations and modify the traditional contrastive alignment objective with an importance-sampling approach to up-sample the importance of hard-negatives without adding complexity. Their Distilled and Hard-negative Training (DiHT) approach improves performance on 20 tasks in a zero-shot benchmark of 29 tasks and bridges the gap between zero-shot and few-shot performance in linear probing. Demo code is available on GitHub.

CVPR 2023 - Progressive Random Convolutions for Single Domain Generalization

In this episode we discuss Progressive Random Convolutions for Single Domain Generalization
by Seokeon Choi, Debasmit Das, Sungha Choi, Seunghan Yang, Hyunsin Park, Sungrack Yun. The paper proposes a method called Progressive Random Convolution (Pro-RandConv) for single domain generalization, which aims to train a model with only one source domain to perform well on arbitrary unseen target domains. The proposed method recursively stacks random convolution layers with a small kernel size instead of increasing the kernel size, which can mitigate semantic distortions and create more effective virtual domains. They also develop a random convolution block to support texture and contrast diversification. Without complex generators or adversarial learning, the proposed method outperforms state-of-the-art methods on single domain generalization benchmarks.

CVPR 2023 - ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction

In this episode we discuss ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction
by Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P. Breckon, Jue Wang. The paper presents ACR, a new method for reconstructing two hands from monocular RGB images in arbitrary scenarios, addressing the challenges posed by occlusions and mutual confusion. Unlike existing methods, ACR leverages center and part-based attention for feature extraction to explicitly mitigate interdependencies between hands and their parts, and to learn a cross-hand prior that better handles interacting hands. The method outperforms the best interacting-hand approaches on the InterHand2.6M dataset and shows comparable performance with state-of-the-art single-hand methods on the FreiHand dataset. Qualitative results on various datasets further demonstrate the effectiveness of the approach for arbitrary hand reconstruction.

CVPR 2023 - MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors

In this episode we discuss MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors
by Yuang Zhang, Tiancai Wang, Xiangyu Zhang. The paper proposes a new pipeline, called MOTRv2, that improves end-to-end multi-object tracking by incorporating an extra object detector. The pipeline first adopts an anchor formulation of queries and then uses the detector to generate proposals as anchors, providing detection prior to MOTR. This improves detection performance and eases the conflict between joint learning detection and association tasks in MOTR. MOTRv2 achieved state-of-the-art performance on the BDD100K dataset and ranked 1st in the 1st Multiple People Tracking in Group Dance Challenge. Code is available on GitHub.

CVPR 2023 - Divide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning

In this episode we discuss Divide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning
by Shi Chen, Qi Zhao. The paper proposes a new framework for visual reasoning inspired by human reasoning, which addresses the limitations of current methods. Existing methods rely on statistical priors and struggle with novel objects or biased question-answer distributions, whereas humans are able to decompose difficult problems and correlate different concepts based on their semantic relationships. The proposed framework uses a principled object factorization method and a neural module network to decompose objects and derive prototypes, which are used to measure similarity on a common semantic space and make decisions with compositional reasoning. The framework is capable of answering questions with diverse objects and provides an interpretable interface for understanding the decision-making process of models.

CVPR 2023 - 3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification

In this episode we discuss 3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification
by Jiazhao Zhang, Liu Dai, Fanpeng Meng, Qingnan Fan, Xuelin Chen, Kai Xu, He Wang. The paper proposes a framework for object goal navigation in 3D environments using two sub-policies - corner-guided exploration policy and category-aware identification policy. Unlike other approaches that use 2D maps, scene graphs, or image sequences, this framework leverages fine-grained spatial information to improve ObjectNav capability. Through extensive experiments, the proposed framework outperforms other modular-based methods on Matterport3D and Gibson datasets while requiring significantly less computational cost for training. The code for the framework will be released to the community.