AI Breakdown

CVPR 2023 - StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos

In this episode we discuss StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos
by Nikita Dvornik, Isma Hadji, Ran Zhang, Konstantinos G. Derpanis, Animesh Garg, Richard P. Wildes, Allan D. Jepson. The paper introduces StepFormer, a self-supervised model that locates key-steps in instructional videos with no human supervision. Traditional methods require video-level human annotations, which do not scale to large datasets. StepFormer uses automatically-generated subtitles as the only source of supervision and a sequence of text narrations using an order-aware loss function that filters out irrelevant phrases. The model outperforms all previous unsupervised and weakly-supervised approaches on step detection and localization and demonstrates an emergent property to solve zero-shot multi-step localization.

CVPR 2023 - SketchXAI: A First Look at Explainability for Human Sketches

In this episode we discuss SketchXAI: A First Look at Explainability for Human Sketches
by Zhiyu Qu, Yulia Gryaditskaya, Ke Li, Kaiyue Pang, Tao Xiang, Yi-Zhe Song. The paper introduces human sketches to the landscape of Explainable Artificial Intelligence (XAI). Sketch is argued to be a "human-centered" data form that represents a natural interface to study explainability. The authors design a simple explainability-friendly sketch encoder that accommodates the intrinsic properties of strokes and define the first ever XAI task for sketch, stroke location inversion (SLI). The authors offer qualitative results and snapshots of the SLI process, as well as providing code available at https://sketchxai.github.io.

CVPR 2023 - Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training

In this episode we discuss Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
by Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, Dhruv Mahajan. The paper discusses improvements to the contrastive pre-training pipeline for vision-language models used in zero-shot recognition problems. The authors propose a filtering strategy called CAT to reduce dataset size, an approach called Concept Distillation to leverage strong unimodal representations and modify the traditional contrastive alignment objective with an importance-sampling approach to up-sample the importance of hard-negatives without adding complexity. Their Distilled and Hard-negative Training (DiHT) approach improves performance on 20 tasks in a zero-shot benchmark of 29 tasks and bridges the gap between zero-shot and few-shot performance in linear probing. Demo code is available on GitHub.

CVPR 2023 - Progressive Random Convolutions for Single Domain Generalization

In this episode we discuss Progressive Random Convolutions for Single Domain Generalization
by Seokeon Choi, Debasmit Das, Sungha Choi, Seunghan Yang, Hyunsin Park, Sungrack Yun. The paper proposes a method called Progressive Random Convolution (Pro-RandConv) for single domain generalization, which aims to train a model with only one source domain to perform well on arbitrary unseen target domains. The proposed method recursively stacks random convolution layers with a small kernel size instead of increasing the kernel size, which can mitigate semantic distortions and create more effective virtual domains. They also develop a random convolution block to support texture and contrast diversification. Without complex generators or adversarial learning, the proposed method outperforms state-of-the-art methods on single domain generalization benchmarks.

CVPR 2023 - ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction

In this episode we discuss ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction
by Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P. Breckon, Jue Wang. The paper presents ACR, a new method for reconstructing two hands from monocular RGB images in arbitrary scenarios, addressing the challenges posed by occlusions and mutual confusion. Unlike existing methods, ACR leverages center and part-based attention for feature extraction to explicitly mitigate interdependencies between hands and their parts, and to learn a cross-hand prior that better handles interacting hands. The method outperforms the best interacting-hand approaches on the InterHand2.6M dataset and shows comparable performance with state-of-the-art single-hand methods on the FreiHand dataset. Qualitative results on various datasets further demonstrate the effectiveness of the approach for arbitrary hand reconstruction.

CVPR 2023 - MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors

In this episode we discuss MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors
by Yuang Zhang, Tiancai Wang, Xiangyu Zhang. The paper proposes a new pipeline, called MOTRv2, that improves end-to-end multi-object tracking by incorporating an extra object detector. The pipeline first adopts an anchor formulation of queries and then uses the detector to generate proposals as anchors, providing detection prior to MOTR. This improves detection performance and eases the conflict between joint learning detection and association tasks in MOTR. MOTRv2 achieved state-of-the-art performance on the BDD100K dataset and ranked 1st in the 1st Multiple People Tracking in Group Dance Challenge. Code is available on GitHub.

CVPR 2023 - Divide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning

In this episode we discuss Divide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning
by Shi Chen, Qi Zhao. The paper proposes a new framework for visual reasoning inspired by human reasoning, which addresses the limitations of current methods. Existing methods rely on statistical priors and struggle with novel objects or biased question-answer distributions, whereas humans are able to decompose difficult problems and correlate different concepts based on their semantic relationships. The proposed framework uses a principled object factorization method and a neural module network to decompose objects and derive prototypes, which are used to measure similarity on a common semantic space and make decisions with compositional reasoning. The framework is capable of answering questions with diverse objects and provides an interpretable interface for understanding the decision-making process of models.

CVPR 2023 - 3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification

In this episode we discuss 3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification
by Jiazhao Zhang, Liu Dai, Fanpeng Meng, Qingnan Fan, Xuelin Chen, Kai Xu, He Wang. The paper proposes a framework for object goal navigation in 3D environments using two sub-policies - corner-guided exploration policy and category-aware identification policy. Unlike other approaches that use 2D maps, scene graphs, or image sequences, this framework leverages fine-grained spatial information to improve ObjectNav capability. Through extensive experiments, the proposed framework outperforms other modular-based methods on Matterport3D and Gibson datasets while requiring significantly less computational cost for training. The code for the framework will be released to the community.

CVPR 2023 - GP-VTON: Towards General Purpose Virtual Try-on via Collaborative Local-Flow Global-Parsing Learning

In this episode we discuss GP-VTON: Towards General Purpose Virtual Try-on via Collaborative Local-Flow Global-Parsing Learning
by Zhenyu Xie, Zaiyu Huang, Xin Dong, Fuwei Zhao, Haoye Dong, Xijin Zhang, Feida Zhu, Xiaodan Liang. The paper proposes a General-Purpose Virtual Try-ON framework, named GP-VTON, for transferring a garment onto a specific person. The proposed framework addresses the limitations of existing methods which fail to preserve semantic information of the garment parts, result in texture distortion and limit the scalability of the system. It introduces a Local-Flow Global-Parsing (LFGP) warping module and a Dynamic Gradient Truncation (DGT) training strategy, resulting in better warping of different garment parts and avoiding texture squeezing. The proposed framework outperforms existing state-of-the-art methods on two high-resolution benchmarks.

CVPR 2023 - StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-Shot Learning

In this episode we discuss StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-Shot Learning
by Yuqian Fu, Yu Xie, Yanwei Fu, Yu-Gang Jiang. The paper proposes a novel model-agnostic meta Style Adversarial training (StyleAdv) method for Cross-Domain Few-Shot Learning (CD-FSL), a task that aims to transfer prior knowledge learned on a source dataset to novel target datasets. This is achieved by using a style adversarial attack method that synthesizes "virtual" and "hard" adversarial styles for model training, gradually making the model robust to visual styles and boosting its generalization ability. The proposed method achieves state-of-the-art results on eight various target datasets, whether built upon ResNet or ViT. Code is available on GitHub.