AI Breakdown

CVPR 2023 - Robust Test-Time Adaptation in Dynamic Scenarios

In this episode we discuss Robust Test-Time Adaptation in Dynamic Scenarios
by Longhui Yuan, Binhui Xie, Shuang Li. The paper discusses the limitations of test-time adaptation (TTA) methods in dynamic scenarios where the test data is sampled gradually over time, and proposes a new method called Robust Test-Time Adaptation (RoTTA) to address these limitations. RoTTA includes a robust batch normalization scheme, a memory bank for category-balanced data sampling, and a time-aware reweighting strategy with a teacher-student model to stabilize the training procedure. The paper presents extensive experiments to prove the effectiveness of RoTTA in continual test-time adaptation on correlatively sampled data streams, making it an easy-to-implement choice for rapid deployment.

CVPR 2023 - Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes

In this episode we discuss Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes
by Zhen Li, Lingli Wang, Mofang Cheng, Cihui Pan, Jiaqi Yang. The paper proposes an efficient method for inverse rendering of large-scale real-world indoor scenes, which reconstructs global illumination and physically-plausible SVBRDFs. They introduce a new compact representation called Texture-based Lighting (TBL), which models direct and infinite-bounce indirect lighting of the entire scene using 3D mesh and HDR textures. The proposed method outperforms existing methods and enables physically-plausible mixed-reality applications such as material editing, editable novel view synthesis, and relighting.

CVPR 2023 - Jedi: Entropy-based Localization and Removal of Adversarial Patches

In this episode we discuss Jedi: Entropy-based Localization and Removal of Adversarial Patches
by Bilel Tarchoun, Anouar Ben Khalifa, Mohamed Ali Mahjoub, Nael Abu-Ghazaleh, Ihsen Alouani. The paper proposes a new defense against adversarial patches that is resilient to realistic patch attacks, called Jedi. Jedi tackles the patch localization problem from an information theory perspective using two new ideas: using entropy analysis to improve the identification of potential patch regions, and an autoencoder to improve the localization of adversarial patches. Jedi achieves high-precision adversarial patch localization, and can be applied on pre-trained off-the-shelf models without changes to their training or inference. It detects on average 90% of adversarial patches and recovers up to 94% of successful patch attacks.

CVPR 2023 - Improving Generalization with Domain Convex Game

In this episode we discuss Improving Generalization with Domain Convex Game
by Fangrui Lv, Jian Liang, Shuang Li, Jinming Zhang, Di Liu. The paper explores the effectiveness of domain augmentation in domain generalization. The authors propose a new perspective on DG as a convex game between domains and design a regularization term based on supermodularity to enhance model generalization for each diversified domain. They also construct a sample filter to eliminate low-quality samples to avoid potentially harmful information. The framework presented in the paper provides a new avenue for the formal analysis of DG, which is supported by heuristic analysis and extensive experiments.

CVPR 2023 - Masked Motion Encoding for Self-Supervised Video Representation Learning

In this episode we discuss Masked Motion Encoding for Self-Supervised Video Representation Learning
by Xinyu Sun, Peihao Chen, Liangwei Chen, Changhao Li, Thomas H. Li, Mingkui Tan, Chuang Gan. The paper proposes a new pre-training paradigm called Masked Motion Encoding (MME) for learning discriminative video representation from unlabeled videos. The authors address the limitations of previous approaches that only focused on predicting appearance contents in masked regions. MME reconstructs both appearance and motion information to explore temporal clues and focuses on representing long-term motion and obtaining fine-grained temporal clues from sparsely sampled videos. The model is pre-trained with MME and able to anticipate long-term and fine-grained motion details. Code is available on GitHub.

CVPR 2023 - Efficient Scale-Invariant Generator with Column-Row Entangled Pixel Synthesis

In this episode we discuss Efficient Scale-Invariant Generator with Column-Row Entangled Pixel Synthesis
by Thuan Hoang Nguyen, Thanh Van Le, Anh Tran. The paper proposes a new generative model called Column-Row Entangled Pixel Synthesis (CREPS) that can efficiently and scalably synthesize photo-realistic images of any arbitrary resolution. Existing GAN-based solutions suffer from inconsistency and texture sticking issues when scaling output resolution, while INR-based generators have a huge memory footprint and slow inference, making them unsuitable for large-scale or real-time systems. CREPS avoids these problems by using a novel bi-line representation that decomposes layer-wise feature maps into separate "thick" column and row encodings, enabling it to synthesize scale-consistent and alias-free images at any resolution with proper training and inference speed.

CVPR 2023 - IMP: Iterative Matching and Pose Estimation with Adaptive Pooling

In this episode we discuss IMP: Iterative Matching and Pose Estimation with Adaptive Pooling
by Fei Xue, Ignas Budvytis, Roberto Cipolla. The paper proposes an iterative matching and pose estimation framework (IMP) that leverages the geometric connections between the two tasks. They introduce a geometry-aware recurrent attention-based module which jointly outputs sparse matches and camera poses. They also introduce an efficient version of IMP called EIMP, that dynamically discards keypoints without potential matches, reducing the quadratic time complexity of attention computation. The proposed method outperforms previous approaches in terms of accuracy and efficiency on YFCC100m, Scannet, and Aachen Day-Night datasets.

CVPR 2023 - Discriminative Co-Saliency and Background Mining Transformer for Co-Salient Object Detection

In this episode we discuss Discriminative Co-Saliency and Background Mining Transformer for Co-Salient Object Detection
by Long Li, Junwei Han, Ni Zhang, Nian Liu, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Fahad Shahbaz Khan. The paper proposes a Discriminative co-saliency and background Mining Transformer (DMT) framework for co-salient object detection. The framework includes several economical multi-grained correlation modules that explicitly mine both co-saliency and background information to effectively model their discrimination. These modules include a region-to-region correlation module, contrast-induced pixel-to-token correlation, and co-saliency token-to-token correlation modules. The proposed framework is experimentally validated on three benchmark datasets and the source code is available on GitHub.

CVPR 2023 - ABLE-NeRF: Attention-Based Rendering with Learnable Embeddings for Neural Radiance Field

In this episode we discuss ABLE-NeRF: Attention-Based Rendering with Learnable Embeddings for Neural Radiance Field
by Zhe Jun Tang, Tat-Jen Cham, Haiyu Zhao. The paper presents an alternative approach to the Neural Radiance Field (NeRF) method for representing 3D scenes that addresses view-dependent effects such as murky glossy and translucent surfaces. The proposed method, called ABLE-NeRF, uses a self-attention-based framework on volumes along a ray and incorporates Learnable Embeddings to capture view-dependent effects. The results show that ABLE-NeRF significantly reduces blurry glossy surfaces and produces realistic translucent surfaces, surpassing Ref-NeRF in all three image quality metrics.

CVPR 2023 - A Dynamic Multi-Scale Voxel Flow Network for Video Prediction

In this episode we discuss A Dynamic Multi-Scale Voxel Flow Network for Video Prediction
by Xiaotao Hu, Zhewei Huang, Ailin Huang, Jun Xu, Shuchang Zhou. The paper proposes a Dynamic Multi-scale Voxel Flow Network (DMVFN) for video prediction using only RGB images. The proposed network is efficient and achieves better performance than previous methods that require extra inputs for promising performance. The core of DMVFN is a differentiable routing module that effectively perceives the motion scales of video frames and selects adaptive sub-networks for different inputs at the inference stage. DMVFN outperforms state-of-the-art iterative-based OPT on generated image quality and is an order of magnitude faster than Deep Voxel Flow.