AI Breakdown

CVPR 2023 - SAP-DETR: Bridging the Gap Between Salient Points and Queries-Based Transformer Detector for Fast Model Convergency

In this episode we discuss SAP-DETR: Bridging the Gap Between Salient Points and Queries-Based Transformer Detector for Fast Model Convergency
by Yang Liu, Yao Zhang, Yixin Wang, Yang Zhang, Jiang Tian, Zhongchao Shi, Jianping Fan, Zhiqiang He. The paper proposes SAlient Point-based DETR (SAP-DETR), a new approach to object detection that treats it as a transformation from salient points to instance objects. SAP-DETR addresses the issue of centralizing reference points that can deteriorate queries' saliency and confuse detectors. By explicitly initializing a query-specific reference point for each object query and gradually aggregating them into an instance object, SAP-DETR can effectively bridge the gap between salient points and query-based Transformer detector with a significant convergency speed. The method achieves competitive performance and stably promotes state-of-the-art approaches.

CVPR 2023 - FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs

In this episode we discuss FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs
by Luke Rowe, Martin Ethier, Eli-Henry Dykhne, Krzysztof Czarnecki. The paper proposes a framework called FJMP for generating a set of joint future trajectory predictions in multi-agent driving scenarios. FJMP models the future scene interaction dynamics using a sparse directed interaction graph and decomposes the joint prediction task into a sequence of marginal and conditional predictions according to the partial ordering of the graph. The results show that FJMP outperforms non-factorized approaches and ranks 1st on the multi-agent test leaderboard of the INTERACTION dataset.

CVPR 2023 - Unsupervised Continual Semantic Adaptation through Neural Rendering

In this episode we discuss Unsupervised Continual Semantic Adaptation through Neural Rendering
by Zhizheng Liu, Francesco Milano, Jonas Frey, Roland Siegwart, Hermann Blum, Cesar Cadena. The paper proposes a method for continual multi-scene adaptation for semantic segmentation tasks, in which no ground-truth labels are available during deployment and performance on previous scenes must be maintained. The method involves training a Semantic-NeRF network for each scene by fusing the predictions of a segmentation model and using the view-consistent rendered semantic labels as pseudo-labels to adapt the model. The Semantic-NeRF model enables 2D-3D knowledge transfer and can be stored in long-term memory to reduce forgetting. The proposed approach outperforms both a voxel-based baseline and a state-of-the-art unsupervised domain adaptation method on the ScanNet dataset.

CVPR 2023 - Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars

In this episode we discuss Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars
by Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, Yebin Liu. The paper proposes a novel 3D GAN framework for unsupervised learning of generative, high-quality, and 3D-consistent facial avatars from unstructured 2D images. The proposed framework combines both fine-grained expression control of mesh-guided explicit deformation and the flexibility of implicit volumetric representation. The proposed representation, called Generative Texture-Rasterized Tri-planes, achieves both deformation accuracy and topological flexibility and demonstrates state-of-the-art 3D-aware synthesis quality and animation ability through extensive experiments.

CVPR 2023 - Catch Missing Details: Image Reconstruction with Frequency Augmented

In this episode we discuss Catch Missing Details: Image Reconstruction with Frequency Augmented
by Xinmiao Lin, Yikang Li, Jenhao Hsiao, Chiuman Ho, Yu Kong. The paper proposes a new architecture called Frequency Augmented VAE (FA-VAE) to address the issue of rapid quality degradation in image reconstruction with popular VQ-VAE models as the compression rate increases. The proposed architecture incorporates a Frequency Complement Module (FCM) to capture missing frequency information and a Dynamic Spectrum Loss (DSL) to balance between frequencies for optimal reconstruction. The paper also introduces a Cross-Attention Autoregressive Transformer (CAT) to improve the generation quality and image-text semantic alignment in text-to-image synthesis. Experiments conducted on benchmark datasets show that FA-VAE and CAT outperform state-of-the-art methods in their respective tasks.

CVPR 2023 - Better “CMOS” Produces Clearer Images:

In this episode we discuss Better “CMOS” Produces Clearer Images:
by Xuhai Chen, Jiangning Zhang, Chao Xu, Yabiao Wang, Chengjie Wang, Yong Liu. The paper discusses the problem of space-variant blur in blind image super-resolution methods, which severely affects their performance. To tackle this issue, the authors introduce two new datasets and design a Cross-MOdal fuSion network (CMOS) that estimates both blur and semantics simultaneously. The CMOS incorporates a feature Grouping Interac-tive Attention (GIA) module to make the two modalities interact effectively and avoid inconsistency. The experiments demonstrate the superiority of their method in terms of quantitative metrics like PSNR/SSIM.

CVPR 2023 - Detecting and Grounding Multi-Modal Media Manipulation

In this episode we discuss Detecting and Grounding Multi-Modal Media Manipulation
by Rui Shao, Tianxing Wu, Ziwei Liu. This paper discusses a new research problem for detecting and grounding multi-modal media manipulation, which requires deeper reasoning across different modalities. The authors propose a new dataset and a novel model called HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities. Dedicated manipulation detection and grounding heads are integrated from shallow to deep levels based on the interacted multi-modal information. The authors conduct comprehensive experiments and set up rigorous evaluation metrics, demonstrating the superiority of their model and revealing valuable observations to facilitate future research in multi-modal media manipulation.

CVPR 2023 - Improving GAN Training via Feature Space Shrinkage

In this episode we discuss Improving GAN Training via Feature Space Shrinkage
by Haozhe Liu, Wentian Zhang, Bing Li, Haoqian Wu, Nanjun He, Yawen Huang, Yuexiang Li, Bernard Ghanem, Yefeng Zheng. The paper proposes a new method, called AdaptiveMix, for training Generative Adversarial Networks (GANs) from a robust image classification perspective. The proposed method shrinks data regions in the image representation space of the discriminator, making it easier to train the GANs. Hard samples are constructed by mixing a pair of training images to narrow down the feature distance between hard and easy samples. The proposed approach is evaluated on several datasets and shown to facilitate GAN training and improve generated image quality. The code is publicly available for use.

CVPR 2023 - CRAFT: Concept Recursive Activation FacTorization for Explainability

In this episode we discuss CRAFT: Concept Recursive Activation FacTorization for Explainability
by Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, Thomas Serre. The paper introduces a new approach called CRAFT to identify "what" and "where" a model looks at in an image. The approach generates concept-based explanations and introduces new ingredients to the automatic concept extraction literature. The proposed concept importance estimation technique is more faithful to the model than previous methods and the usefulness of the approach is demonstrated in both human and computer vision experiments. The code for CRAFT is freely available on GitHub.

CVPR 2023 - gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction

In this episode we discuss gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction
by Zerui Chen, Shizhe Chen, Cordelia Schmid, Ivan Laptev. The paper presents a method for reconstructing 3D shapes of hands and manipulated objects from monocular RGB images using signed distance functions (SDFs) as a framework. The authors exploit the hand structure to guide the SDF-based shape reconstruction by estimating poses of hands and objects and aligning SDFs with highly-articulated hand poses. They also use temporal information to enhance the method's robustness to occlusion and motion blurs. Extensive experiments on challenging benchmarks demonstrate significant improvements over the state-of-the-art in 3D shape reconstruction.