AI Breakdown

CVPR 2023 - Train-Once-for-All Personalization

In this episode we discuss Train-Once-for-All Personalization by Authors: - Hong-You Chen - Yandong Li - Yin Cui - Mingda Zhang - Wei-Lun Chao - Li Zhang Affiliations: - Hong-You Chen and Wei-Lun Chao are affiliated with The Ohio State University. - Yandong Li, Yin Cui, Mingda Zhang, and Li Zhang are affiliated with Google Research. Contact information: - Hong-You Chen and Wei-Lun Chao: Yandong Li, Yin Cui, Mingda Zhang, and Li Zhang:  The paper proposes a framework called Train-once-for-All PERsonalization (TAPER) for training a "personalization-friendly" model that can be customized for different end-users based on their task descriptions. The framework learns a set of "basis" models and a mixer predictor, which can combine the weights of the basis models on-the-fly to create a personalized model for a given end-user. TAPER consistently outperforms baseline methods and can synthesize smaller models for deployment on resource-limited devices, and can even be specialized without task descriptions based on past predictions.

CVPR 2023 - MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

In this episode we discuss MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
by Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, Baining Guo. The paper proposes a joint audio-video generation framework called Multi-Modal Diffusion (MM-Diffusion) that generates high-quality realistic videos with aligned audio. The model consists of two-coupled denoising autoencoders and a sequential multi-modal U-Net. A random-shift based attention block is used to ensure semantic consistency across modalities, enabling efficient cross-modal alignment. The model achieves superior results in unconditional audio-video generation and zero-shot conditional tasks, and Turing tests indicate dominant preferences for the model. Code and pre-trained models are available for download.

CVPR 2023 - Robust Test-Time Adaptation in Dynamic Scenarios

In this episode we discuss Robust Test-Time Adaptation in Dynamic Scenarios
by Longhui Yuan, Binhui Xie, Shuang Li. The paper discusses the limitations of test-time adaptation (TTA) methods in dynamic scenarios where the test data is sampled gradually over time, and proposes a new method called Robust Test-Time Adaptation (RoTTA) to address these limitations. RoTTA includes a robust batch normalization scheme, a memory bank for category-balanced data sampling, and a time-aware reweighting strategy with a teacher-student model to stabilize the training procedure. The paper presents extensive experiments to prove the effectiveness of RoTTA in continual test-time adaptation on correlatively sampled data streams, making it an easy-to-implement choice for rapid deployment.

CVPR 2023 - Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes

In this episode we discuss Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes
by Zhen Li, Lingli Wang, Mofang Cheng, Cihui Pan, Jiaqi Yang. The paper proposes an efficient method for inverse rendering of large-scale real-world indoor scenes, which reconstructs global illumination and physically-plausible SVBRDFs. They introduce a new compact representation called Texture-based Lighting (TBL), which models direct and infinite-bounce indirect lighting of the entire scene using 3D mesh and HDR textures. The proposed method outperforms existing methods and enables physically-plausible mixed-reality applications such as material editing, editable novel view synthesis, and relighting.

CVPR 2023 - Jedi: Entropy-based Localization and Removal of Adversarial Patches

In this episode we discuss Jedi: Entropy-based Localization and Removal of Adversarial Patches
by Bilel Tarchoun, Anouar Ben Khalifa, Mohamed Ali Mahjoub, Nael Abu-Ghazaleh, Ihsen Alouani. The paper proposes a new defense against adversarial patches that is resilient to realistic patch attacks, called Jedi. Jedi tackles the patch localization problem from an information theory perspective using two new ideas: using entropy analysis to improve the identification of potential patch regions, and an autoencoder to improve the localization of adversarial patches. Jedi achieves high-precision adversarial patch localization, and can be applied on pre-trained off-the-shelf models without changes to their training or inference. It detects on average 90% of adversarial patches and recovers up to 94% of successful patch attacks.

CVPR 2023 - Improving Generalization with Domain Convex Game

In this episode we discuss Improving Generalization with Domain Convex Game
by Fangrui Lv, Jian Liang, Shuang Li, Jinming Zhang, Di Liu. The paper explores the effectiveness of domain augmentation in domain generalization. The authors propose a new perspective on DG as a convex game between domains and design a regularization term based on supermodularity to enhance model generalization for each diversified domain. They also construct a sample filter to eliminate low-quality samples to avoid potentially harmful information. The framework presented in the paper provides a new avenue for the formal analysis of DG, which is supported by heuristic analysis and extensive experiments.

CVPR 2023 - Masked Motion Encoding for Self-Supervised Video Representation Learning

In this episode we discuss Masked Motion Encoding for Self-Supervised Video Representation Learning
by Xinyu Sun, Peihao Chen, Liangwei Chen, Changhao Li, Thomas H. Li, Mingkui Tan, Chuang Gan. The paper proposes a new pre-training paradigm called Masked Motion Encoding (MME) for learning discriminative video representation from unlabeled videos. The authors address the limitations of previous approaches that only focused on predicting appearance contents in masked regions. MME reconstructs both appearance and motion information to explore temporal clues and focuses on representing long-term motion and obtaining fine-grained temporal clues from sparsely sampled videos. The model is pre-trained with MME and able to anticipate long-term and fine-grained motion details. Code is available on GitHub.

CVPR 2023 - Efficient Scale-Invariant Generator with Column-Row Entangled Pixel Synthesis

In this episode we discuss Efficient Scale-Invariant Generator with Column-Row Entangled Pixel Synthesis
by Thuan Hoang Nguyen, Thanh Van Le, Anh Tran. The paper proposes a new generative model called Column-Row Entangled Pixel Synthesis (CREPS) that can efficiently and scalably synthesize photo-realistic images of any arbitrary resolution. Existing GAN-based solutions suffer from inconsistency and texture sticking issues when scaling output resolution, while INR-based generators have a huge memory footprint and slow inference, making them unsuitable for large-scale or real-time systems. CREPS avoids these problems by using a novel bi-line representation that decomposes layer-wise feature maps into separate "thick" column and row encodings, enabling it to synthesize scale-consistent and alias-free images at any resolution with proper training and inference speed.

CVPR 2023 - IMP: Iterative Matching and Pose Estimation with Adaptive Pooling

In this episode we discuss IMP: Iterative Matching and Pose Estimation with Adaptive Pooling
by Fei Xue, Ignas Budvytis, Roberto Cipolla. The paper proposes an iterative matching and pose estimation framework (IMP) that leverages the geometric connections between the two tasks. They introduce a geometry-aware recurrent attention-based module which jointly outputs sparse matches and camera poses. They also introduce an efficient version of IMP called EIMP, that dynamically discards keypoints without potential matches, reducing the quadratic time complexity of attention computation. The proposed method outperforms previous approaches in terms of accuracy and efficiency on YFCC100m, Scannet, and Aachen Day-Night datasets.

CVPR 2023 - Discriminative Co-Saliency and Background Mining Transformer for Co-Salient Object Detection

In this episode we discuss Discriminative Co-Saliency and Background Mining Transformer for Co-Salient Object Detection
by Long Li, Junwei Han, Ni Zhang, Nian Liu, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Fahad Shahbaz Khan. The paper proposes a Discriminative co-saliency and background Mining Transformer (DMT) framework for co-salient object detection. The framework includes several economical multi-grained correlation modules that explicitly mine both co-saliency and background information to effectively model their discrimination. These modules include a region-to-region correlation module, contrast-induced pixel-to-token correlation, and co-saliency token-to-token correlation modules. The proposed framework is experimentally validated on three benchmark datasets and the source code is available on GitHub.