AI Breakdown

CVPR 2023 - Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation

In this episode we discuss Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation
by Mingjie Li, Bingqian Lin, Zicong Chen, Haokun Lin, Xiaodan Liang, Xiaojun Chang. The paper proposes a knowledge graph with dynamic structure and nodes to enhance automatic radiology reporting. Existing models that use medical knowledge graphs have limited effectiveness because they have fixed structures that don't update during training. The proposed model, named DCL, allows for the addition of specific knowledge extracted from retrieved reports in a bottom-up manner, integrating each image feature with an updated graph. The model also introduces image-report contrastive and image-report matching losses to better represent visual features and textual information. Evaluation on two datasets shows that DCL outperforms previous state-of-the-art models.

CVPR 2023 - NeuDA: Neural Deformable Anchor for High-Fidelity Implicit Surface Reconstruction

In this episode we discuss NeuDA: Neural Deformable Anchor for High-Fidelity Implicit Surface Reconstruction
by Bowen Cai, Jinchi Huang, Rongfei Jia, Chengfei Lv, Huan Fu. The paper proposes a new approach called Neural Deformable Anchor (NeuDA) for implicit surface reconstruction using differentiable ray casting. Unlike previous methods, NeuDA leverages hierarchical voxel grids to capture sharp local topologies and maintain anchor grids where each vertex stores a 3D position instead of direct embedding. The paper also introduces a hierarchical positional encoding method for the anchor structure to exploit the properties of high-frequency and low-frequency geometry and appearance. Experiments on two datasets demonstrate NeuDA's ability to produce promising mesh surfaces.

CVPR 2023 - NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images

In this episode we discuss NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images
by Mingwu Zheng, Haiyu Zhang, Hongyu Yang, Di Huang. The paper presents a new 3D face rendering model, called Neu-Face, that uses neural rendering techniques to learn accurate and physically-meaningful underlying 3D representations. It incorporates the neural BRDFs (bidirectional reflectance distribution function) into physically based rendering to capture sophisticated facial geometry and appearance clues. The model uses approximated BRDF integration and a new low-rank prior to effectively lower ambiguities and boost performance. The experiments show the superiority of Neu-Face in human face rendering, with decent generalization ability to common objects. The code is available at NeuFace.

CVPR 2023 - SAP-DETR: Bridging the Gap Between Salient Points and Queries-Based Transformer Detector for Fast Model Convergency

In this episode we discuss SAP-DETR: Bridging the Gap Between Salient Points and Queries-Based Transformer Detector for Fast Model Convergency
by Yang Liu, Yao Zhang, Yixin Wang, Yang Zhang, Jiang Tian, Zhongchao Shi, Jianping Fan, Zhiqiang He. The paper proposes SAlient Point-based DETR (SAP-DETR), a new approach to object detection that treats it as a transformation from salient points to instance objects. SAP-DETR addresses the issue of centralizing reference points that can deteriorate queries' saliency and confuse detectors. By explicitly initializing a query-specific reference point for each object query and gradually aggregating them into an instance object, SAP-DETR can effectively bridge the gap between salient points and query-based Transformer detector with a significant convergency speed. The method achieves competitive performance and stably promotes state-of-the-art approaches.

CVPR 2023 - FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs

In this episode we discuss FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs
by Luke Rowe, Martin Ethier, Eli-Henry Dykhne, Krzysztof Czarnecki. The paper proposes a framework called FJMP for generating a set of joint future trajectory predictions in multi-agent driving scenarios. FJMP models the future scene interaction dynamics using a sparse directed interaction graph and decomposes the joint prediction task into a sequence of marginal and conditional predictions according to the partial ordering of the graph. The results show that FJMP outperforms non-factorized approaches and ranks 1st on the multi-agent test leaderboard of the INTERACTION dataset.

CVPR 2023 - Unsupervised Continual Semantic Adaptation through Neural Rendering

In this episode we discuss Unsupervised Continual Semantic Adaptation through Neural Rendering
by Zhizheng Liu, Francesco Milano, Jonas Frey, Roland Siegwart, Hermann Blum, Cesar Cadena. The paper proposes a method for continual multi-scene adaptation for semantic segmentation tasks, in which no ground-truth labels are available during deployment and performance on previous scenes must be maintained. The method involves training a Semantic-NeRF network for each scene by fusing the predictions of a segmentation model and using the view-consistent rendered semantic labels as pseudo-labels to adapt the model. The Semantic-NeRF model enables 2D-3D knowledge transfer and can be stored in long-term memory to reduce forgetting. The proposed approach outperforms both a voxel-based baseline and a state-of-the-art unsupervised domain adaptation method on the ScanNet dataset.

CVPR 2023 - Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars

In this episode we discuss Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars
by Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, Yebin Liu. The paper proposes a novel 3D GAN framework for unsupervised learning of generative, high-quality, and 3D-consistent facial avatars from unstructured 2D images. The proposed framework combines both fine-grained expression control of mesh-guided explicit deformation and the flexibility of implicit volumetric representation. The proposed representation, called Generative Texture-Rasterized Tri-planes, achieves both deformation accuracy and topological flexibility and demonstrates state-of-the-art 3D-aware synthesis quality and animation ability through extensive experiments.

CVPR 2023 - Catch Missing Details: Image Reconstruction with Frequency Augmented

In this episode we discuss Catch Missing Details: Image Reconstruction with Frequency Augmented
by Xinmiao Lin, Yikang Li, Jenhao Hsiao, Chiuman Ho, Yu Kong. The paper proposes a new architecture called Frequency Augmented VAE (FA-VAE) to address the issue of rapid quality degradation in image reconstruction with popular VQ-VAE models as the compression rate increases. The proposed architecture incorporates a Frequency Complement Module (FCM) to capture missing frequency information and a Dynamic Spectrum Loss (DSL) to balance between frequencies for optimal reconstruction. The paper also introduces a Cross-Attention Autoregressive Transformer (CAT) to improve the generation quality and image-text semantic alignment in text-to-image synthesis. Experiments conducted on benchmark datasets show that FA-VAE and CAT outperform state-of-the-art methods in their respective tasks.

CVPR 2023 - Better “CMOS” Produces Clearer Images:

In this episode we discuss Better “CMOS” Produces Clearer Images:
by Xuhai Chen, Jiangning Zhang, Chao Xu, Yabiao Wang, Chengjie Wang, Yong Liu. The paper discusses the problem of space-variant blur in blind image super-resolution methods, which severely affects their performance. To tackle this issue, the authors introduce two new datasets and design a Cross-MOdal fuSion network (CMOS) that estimates both blur and semantics simultaneously. The CMOS incorporates a feature Grouping Interac-tive Attention (GIA) module to make the two modalities interact effectively and avoid inconsistency. The experiments demonstrate the superiority of their method in terms of quantitative metrics like PSNR/SSIM.

CVPR 2023 - Detecting and Grounding Multi-Modal Media Manipulation

In this episode we discuss Detecting and Grounding Multi-Modal Media Manipulation
by Rui Shao, Tianxing Wu, Ziwei Liu. This paper discusses a new research problem for detecting and grounding multi-modal media manipulation, which requires deeper reasoning across different modalities. The authors propose a new dataset and a novel model called HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities. Dedicated manipulation detection and grounding heads are integrated from shallow to deep levels based on the interacted multi-modal information. The authors conduct comprehensive experiments and set up rigorous evaluation metrics, demonstrating the superiority of their model and revealing valuable observations to facilitate future research in multi-modal media manipulation.