AI Breakdown

CVPR 2023 - ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videos

In this episode we discuss ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videos
by Zhou Yu, Lixiang Zheng, Zhou Zhao, Fei Wu, Jianping Fan, Kui Ren, Jun Yu. The paper discusses the challenge of building benchmarks for video question answering (VideoQA) models that can systematically analyze their capabilities. Existing benchmarks have limitations such as non-compositional simple questions and language biases. The authors present ANetQA, a large-scale benchmark that supports fine-grained compositional reasoning on untrimmed videos from ActivityNet, with spatio-temporal scene graphs and diverse questions generated from fine-grained templates. The benchmark attains 1.4 billion unbalanced and 13.4 million balanced QA pairs, and comprehensive experiments are performed for state-of-the-art methods, with the best model achieving 44.5% accuracy and human performance topping out at 84.5%.

CVPR 2023 - Neuralizer: General Neuroimage Analysis without Re-Training

In this episode we discuss Neuralizer: General Neuroimage Analysis without Re-Training
by Steffen Czolbe, Adrian V. Dalca. The paper discusses the challenges in using deep learning for neuroimage processing tasks such as segmentation and registration. The authors introduce a new model called Neuralizer that can generalize to previously unseen tasks and modalities without the need for re-training or fine-tuning. The model can solve processing tasks across multiple image modalities and datasets, and outperforms task-specific baselines even when few annotated subjects are available. The goal is to provide a tool that can be adopted by neuroscientists and clinical researchers who may lack the resources or expertise to train deep learning models.

CVPR 2023 - Shape-Erased Feature Learning for Visible-Infrared Person Re-Identification

In this episode we discuss Shape-Erased Feature Learning for Visible-Infrared Person Re-Identification
by Jiawei Feng, Ancong Wu, Wei-Shi Zheng. The paper proposes a new approach to address the challenging problem of visible-infrared person re-identification (VI-ReID) by learning diverse modality-shared semantic concepts. The proposed method aims to force the ReID model to extract more and different modality-shared features for identification by erasing body-shape-related semantic concepts in the learned features. This is achieved through a shape-erased feature learning paradigm that decorrelates modality-shared features in two orthogonal subspaces. The experimental results on three datasets demonstrate the effectiveness of the proposed method.

CVPR 2023 - STMixer: A One-Stage Sparse Action Detector

In this episode we discuss STMixer: A One-Stage Sparse Action Detector
by Tao Wu, Mengqi Cao, Ziteng Gao, Gangshan Wu, Limin Wang. The paper proposes a new one-stage sparse action detector called STMixer which is based on two core designs. The first design is a query-based adaptive feature sampling module that allows STMixer to mine discriminative features from the entire spatiotemporal domain. The second design is a dual-branch feature mixing module that permits STMixer to dynamically attend and mix video features along the spatial and temporal dimension respectively for better feature decoding. The proposed STMixer achieves state-of-the-art results on the AVA, UCF101-24, and JHMDB datasets.

CVPR 2023 - Balanced Spherical Grid for Egocentric View Synthesis

In this episode we discuss Balanced Spherical Grid for Egocentric View Synthesis
by Changwoon Choi, Sang Min Kim, Young Min Kim. The paper presents EgoNeRF, an efficient solution for reconstructing large-scale environments from a few seconds of 360 videos for virtual reality (VR) assets. The authors adopted a spherical coordinate parameterization instead of Cartesian coordinate grids, which tend to be inefficient for unbounded scenes. This approach aligns better with egocentric images' rays and also enables factorization for performance enhancement. Additionally, the authors use resampling techniques and a combination of balanced grids to avoid singularities and represent unbounded scenes respectively. They extensively evaluate their approach with synthetic and real-world egocentric 360 video datasets and report state-of-the-art performance consistently.

CVPR 2023 - Train-Once-for-All Personalization

In this episode we discuss Train-Once-for-All Personalization by Authors: - Hong-You Chen - Yandong Li - Yin Cui - Mingda Zhang - Wei-Lun Chao - Li Zhang Affiliations: - Hong-You Chen and Wei-Lun Chao are affiliated with The Ohio State University. - Yandong Li, Yin Cui, Mingda Zhang, and Li Zhang are affiliated with Google Research. Contact information: - Hong-You Chen and Wei-Lun Chao: Yandong Li, Yin Cui, Mingda Zhang, and Li Zhang:  The paper proposes a framework called Train-once-for-All PERsonalization (TAPER) for training a "personalization-friendly" model that can be customized for different end-users based on their task descriptions. The framework learns a set of "basis" models and a mixer predictor, which can combine the weights of the basis models on-the-fly to create a personalized model for a given end-user. TAPER consistently outperforms baseline methods and can synthesize smaller models for deployment on resource-limited devices, and can even be specialized without task descriptions based on past predictions.

CVPR 2023 - MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

In this episode we discuss MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
by Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, Baining Guo. The paper proposes a joint audio-video generation framework called Multi-Modal Diffusion (MM-Diffusion) that generates high-quality realistic videos with aligned audio. The model consists of two-coupled denoising autoencoders and a sequential multi-modal U-Net. A random-shift based attention block is used to ensure semantic consistency across modalities, enabling efficient cross-modal alignment. The model achieves superior results in unconditional audio-video generation and zero-shot conditional tasks, and Turing tests indicate dominant preferences for the model. Code and pre-trained models are available for download.

CVPR 2023 - Robust Test-Time Adaptation in Dynamic Scenarios

In this episode we discuss Robust Test-Time Adaptation in Dynamic Scenarios
by Longhui Yuan, Binhui Xie, Shuang Li. The paper discusses the limitations of test-time adaptation (TTA) methods in dynamic scenarios where the test data is sampled gradually over time, and proposes a new method called Robust Test-Time Adaptation (RoTTA) to address these limitations. RoTTA includes a robust batch normalization scheme, a memory bank for category-balanced data sampling, and a time-aware reweighting strategy with a teacher-student model to stabilize the training procedure. The paper presents extensive experiments to prove the effectiveness of RoTTA in continual test-time adaptation on correlatively sampled data streams, making it an easy-to-implement choice for rapid deployment.

CVPR 2023 - Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes

In this episode we discuss Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes
by Zhen Li, Lingli Wang, Mofang Cheng, Cihui Pan, Jiaqi Yang. The paper proposes an efficient method for inverse rendering of large-scale real-world indoor scenes, which reconstructs global illumination and physically-plausible SVBRDFs. They introduce a new compact representation called Texture-based Lighting (TBL), which models direct and infinite-bounce indirect lighting of the entire scene using 3D mesh and HDR textures. The proposed method outperforms existing methods and enables physically-plausible mixed-reality applications such as material editing, editable novel view synthesis, and relighting.

CVPR 2023 - Jedi: Entropy-based Localization and Removal of Adversarial Patches

In this episode we discuss Jedi: Entropy-based Localization and Removal of Adversarial Patches
by Bilel Tarchoun, Anouar Ben Khalifa, Mohamed Ali Mahjoub, Nael Abu-Ghazaleh, Ihsen Alouani. The paper proposes a new defense against adversarial patches that is resilient to realistic patch attacks, called Jedi. Jedi tackles the patch localization problem from an information theory perspective using two new ideas: using entropy analysis to improve the identification of potential patch regions, and an autoencoder to improve the localization of adversarial patches. Jedi achieves high-precision adversarial patch localization, and can be applied on pre-trained off-the-shelf models without changes to their training or inference. It detects on average 90% of adversarial patches and recovers up to 94% of successful patch attacks.