AI Breakdown

CVPR 2023 - Model-Agnostic Gender Debiased Image Captioning

In this episode we discuss Model-Agnostic Gender Debiased Image Captioning
by Yusuke Hirota, Yuta Nakashima, Noa Garcia. The paper discusses the issue of gender bias in image captioning models and proposes a framework named LIBRA to mitigate such bias. Prior attempts to address this problem by focusing on people created gender-stereotypical words, and it affected gender prediction. The researchers hypothesize that there are two types of bias - exploiting context to predict gender and the probability of generating stereotypical words. The proposed framework learns from synthetic data to decrease both types of bias, correct gender misclassification, and change gender-stereotypical words to more neutral ones.

CVPR 2023 - Magic3D: High-Resolution Text-to-3D Content Creation

In this episode we discuss Magic3D: High-Resolution Text-to-3D Content Creation
by Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin. The paper introduces a two-stage optimization framework called Magic3D to address the slow optimization and low-resolution image space supervision limitations of the pre-trained text-to-image diffusion model called DreamFusion. The first stage involves obtaining a coarse model using a low-resolution diffusion prior and accelerating it with a sparse 3D hash grid structure. In the second stage, a textured 3D mesh model is optimized using an efficient differentiable renderer interacting with a high-resolution latent diffusion model. Magic3D can create high-quality 3D mesh models in 40 minutes, 2x faster than DreamFusion, while achieving higher resolution. User studies show that 61.7% of raters prefer Magic3D over DreamFusion.

CVPR 2023 - Self-Correctable and Adaptable Inference for Generalizable Human Pose Estimation

In this episode we discuss Self-Correctable and Adaptable Inference for Generalizable Human Pose Estimation
by Zhehan Kan, Shuoshuo Chen, Ce Zhang, Yushun Tang, Zhihai He. The paper introduces a self-correctable and adaptable inference (SCAI) method to address the generalization challenge of network prediction. Utilizing human pose estimation as an example, they learn a correction network to correct the prediction result conditioned by a fitness feedback error. This feedback error is generated by a learned fitness feedback network that maps the prediction result to the original input domain and compares it against the original input, which can be used as feedback to guide the correction process and as a loss function to optimize the correction network during the inference process. Experimental results demonstrate that the proposed SCAI method significantly improves the generalization capability and performance of human pose estimation.

CVPR 2023 - Adaptive Spot-Guided Transformer for Consistent Local Feature Matching

In this episode we discuss Adaptive Spot-Guided Transformer for Consistent Local Feature Matching
by Jiahuan Yu, Jiahao Chang, Jianfeng He, Tianzhu Zhang, Feng Wu. The paper proposes Adaptive Spot-Guided Transformer (ASTR), a new approach for local feature matching that models both local consistency and scale variations in a coarse-to-fine architecture. ASTR uses a spot-guided aggregation module to avoid interfering with irrelevant areas during feature aggregation and an adaptive scaling module to adjust the size of grids according to depth information. The method outperforms state-of-the-art approaches on five standard benchmarks. Code for ASTR will be released on https://astr2023.github.io.

CVPR 2023 - Hierarchical Dense Correlation Distillation for Few-Shot Segmentation

In this episode we discuss Hierarchical Dense Correlation Distillation for Few-Shot Segmentation
by Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chenyao Wang, Shu Liu, Jingyong Su, Jiaya Jia. The paper proposes a Hierarchically Decoupled Matching Network (HDMNet) for few-shot semantic segmentation (FSS), where a class-agnostic model segments unseen classes with only a few annotations. The method focuses on mining pixel-level support correlation based on the transformer architecture, and uses self-attention modules for establishing hierarchical dense features for cascade matching between query and support features. The proposed matching module reduces train-set overfitting and introduces correlation distillation leveraging semantic correspondence from coarse to fine resolution, resulting in decent performance on COCO-20i dataset, achieving 50% mIoU on one-shot and 56% on five-shot segmentation. Code is available on the project website.

CVPR 2023 - Generating Features with Increased Crop-related Diversity for Few-Shot Object Detection

In this episode we discuss Generating Features with Increased Crop-related Diversity for Few-Shot Object Detection
by Jingyi Xu, Hieu Le, Dimitris Samaras. The paper proposes a novel data generation model, based on a variational autoencoder (VAE), for training robust object detectors in few-shot settings. The model is designed to generate crops with increased crop-related diversity to account for the variability in object proposals generated by two-stage detectors. By transforming the latent space, the model produces features with diverse difficulty levels by varying the latent norm, which is rescaled based on the intersection-over-union (IoU) score of the input crop with respect to the ground-truth box. The experiments show that the generated features consistently improve state-of-the-art few-shot object detection methods on PASCAL VOC and MS COCO datasets.

CVPR 2023 - JAWS: Just A Wild Shot for Cinematic Transfer in Neural Radiance Fields

In this episode we discuss JAWS: Just A Wild Shot for Cinematic Transfer in Neural Radiance Fields
by Xi Wang, Robin Courant, Jinglei Shi, Eric Marchand, Marc Christie. The paper introduces JAWS, an optimization-driven approach to transfer visual cinematic features from a reference video clip to a newly generated clip, using implicit neural representations. The method computes cinematic features in an INR and optimizes extrinsic and intrinsic camera parameters and timing to replicate the reference clip. The approach leverages the differentiability of neural representations to backpropagate cinematic losses through a NeRF network and includes enhancements like guidance maps for quality improvement. Results demonstrate successful replication of well-known cinematic sequences.

CVPR 2023 - SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text

In this episode we discuss SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text
by Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Aneeshan Sain, Subhadeep Koley, Tao Xiang, Yi-Zhe Song. The paper discusses an extension of scene understanding that includes human sketch as a modality, resulting in a complete trilogy of scene representation from three diverse modalities - sketch, photo, and text. The focus is on learning a joint embedding that supports the flexibility of using any combination of modalities as a query for downstream tasks like retrieval and simultaneously utilizing the embedding for either discriminative or generative tasks. The proposed embedding is capable of accommodating a variety of scene-related tasks without any task-specific modifications.

CVPR 2023 - Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos

In this episode we discuss Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos
by Liao Wang, Qiang Hu, Qihan He, Ziyu Wang, Jingyi Yu, Tinne Tuytelaars, Lan Xu, Minye Wu. The paper introduces a new technique called Residual Radiance Field (ReRF), a compact neural representation for achieving real-time free-view rendering on long-duration dynamic scenes. ReRF explicitly models residual information between adjacent timestamps in the spatial-temporal feature space using a global coordinate-based tiny MLP as the feature decoder. The paper also presents a special free-view video (FVV) codec based on ReRF that achieves three orders of magnitude compression rate and provides a companion ReRF player to support online streaming of long-duration FVVs of dynamic scenes. Extensive experiments demonstrate the effectiveness of ReRF for compactly representing dynamic radiance fields, enabling an unprecedented free-viewpoint viewing experience in speed and quality.

CVPR 2023 - DANI-Net: Uncalibrated Photometric Stereo by Differentiable Shadow Handling, Anisotropic Reflectance Modeling, and Neural Inverse Rendering

In this episode we discuss DANI-Net: Uncalibrated Photometric Stereo by Differentiable Shadow Handling, Anisotropic Reflectance Modeling, and Neural Inverse Rendering
by Zongrui Li, Qian Zheng, Boxin Shi, Gang Pan, Xudong Jiang. The paper proposes a deep learning approach, called DANI-Net, to solve the challenging problem of uncalibrated photometric stereo (UPS) which is complicated by unknown lighting. UPS is particularly difficult for non-Lambertian objects with complex shapes and irregular shadows, and for general materials with complex reflectance such as anisotropic reflectance. Unlike previous methods that use non-differentiable shadow maps and assume isotropic material, DANI-Net benefits from cues of shadow and anisotropic reflectance through two differentiable paths, resulting in superior and robust performance on multiple real-world datasets.