AI Breakdown

CVPR 2023 - Self-Correctable and Adaptable Inference for Generalizable Human Pose Estimation

In this episode we discuss Self-Correctable and Adaptable Inference for Generalizable Human Pose Estimation
by Zhehan Kan, Shuoshuo Chen, Ce Zhang, Yushun Tang, Zhihai He. The paper introduces a self-correctable and adaptable inference (SCAI) method to address the generalization challenge of network prediction. Utilizing human pose estimation as an example, they learn a correction network to correct the prediction result conditioned by a fitness feedback error. This feedback error is generated by a learned fitness feedback network that maps the prediction result to the original input domain and compares it against the original input, which can be used as feedback to guide the correction process and as a loss function to optimize the correction network during the inference process. Experimental results demonstrate that the proposed SCAI method significantly improves the generalization capability and performance of human pose estimation.

CVPR 2023 - Adaptive Spot-Guided Transformer for Consistent Local Feature Matching

In this episode we discuss Adaptive Spot-Guided Transformer for Consistent Local Feature Matching
by Jiahuan Yu, Jiahao Chang, Jianfeng He, Tianzhu Zhang, Feng Wu. The paper proposes Adaptive Spot-Guided Transformer (ASTR), a new approach for local feature matching that models both local consistency and scale variations in a coarse-to-fine architecture. ASTR uses a spot-guided aggregation module to avoid interfering with irrelevant areas during feature aggregation and an adaptive scaling module to adjust the size of grids according to depth information. The method outperforms state-of-the-art approaches on five standard benchmarks. Code for ASTR will be released on https://astr2023.github.io.

CVPR 2023 - Hierarchical Dense Correlation Distillation for Few-Shot Segmentation

In this episode we discuss Hierarchical Dense Correlation Distillation for Few-Shot Segmentation
by Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chenyao Wang, Shu Liu, Jingyong Su, Jiaya Jia. The paper proposes a Hierarchically Decoupled Matching Network (HDMNet) for few-shot semantic segmentation (FSS), where a class-agnostic model segments unseen classes with only a few annotations. The method focuses on mining pixel-level support correlation based on the transformer architecture, and uses self-attention modules for establishing hierarchical dense features for cascade matching between query and support features. The proposed matching module reduces train-set overfitting and introduces correlation distillation leveraging semantic correspondence from coarse to fine resolution, resulting in decent performance on COCO-20i dataset, achieving 50% mIoU on one-shot and 56% on five-shot segmentation. Code is available on the project website.

CVPR 2023 - Generating Features with Increased Crop-related Diversity for Few-Shot Object Detection

In this episode we discuss Generating Features with Increased Crop-related Diversity for Few-Shot Object Detection
by Jingyi Xu, Hieu Le, Dimitris Samaras. The paper proposes a novel data generation model, based on a variational autoencoder (VAE), for training robust object detectors in few-shot settings. The model is designed to generate crops with increased crop-related diversity to account for the variability in object proposals generated by two-stage detectors. By transforming the latent space, the model produces features with diverse difficulty levels by varying the latent norm, which is rescaled based on the intersection-over-union (IoU) score of the input crop with respect to the ground-truth box. The experiments show that the generated features consistently improve state-of-the-art few-shot object detection methods on PASCAL VOC and MS COCO datasets.

CVPR 2023 - JAWS: Just A Wild Shot for Cinematic Transfer in Neural Radiance Fields

In this episode we discuss JAWS: Just A Wild Shot for Cinematic Transfer in Neural Radiance Fields
by Xi Wang, Robin Courant, Jinglei Shi, Eric Marchand, Marc Christie. The paper introduces JAWS, an optimization-driven approach to transfer visual cinematic features from a reference video clip to a newly generated clip, using implicit neural representations. The method computes cinematic features in an INR and optimizes extrinsic and intrinsic camera parameters and timing to replicate the reference clip. The approach leverages the differentiability of neural representations to backpropagate cinematic losses through a NeRF network and includes enhancements like guidance maps for quality improvement. Results demonstrate successful replication of well-known cinematic sequences.

CVPR 2023 - SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text

In this episode we discuss SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text
by Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Aneeshan Sain, Subhadeep Koley, Tao Xiang, Yi-Zhe Song. The paper discusses an extension of scene understanding that includes human sketch as a modality, resulting in a complete trilogy of scene representation from three diverse modalities - sketch, photo, and text. The focus is on learning a joint embedding that supports the flexibility of using any combination of modalities as a query for downstream tasks like retrieval and simultaneously utilizing the embedding for either discriminative or generative tasks. The proposed embedding is capable of accommodating a variety of scene-related tasks without any task-specific modifications.

CVPR 2023 - Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos

In this episode we discuss Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos
by Liao Wang, Qiang Hu, Qihan He, Ziyu Wang, Jingyi Yu, Tinne Tuytelaars, Lan Xu, Minye Wu. The paper introduces a new technique called Residual Radiance Field (ReRF), a compact neural representation for achieving real-time free-view rendering on long-duration dynamic scenes. ReRF explicitly models residual information between adjacent timestamps in the spatial-temporal feature space using a global coordinate-based tiny MLP as the feature decoder. The paper also presents a special free-view video (FVV) codec based on ReRF that achieves three orders of magnitude compression rate and provides a companion ReRF player to support online streaming of long-duration FVVs of dynamic scenes. Extensive experiments demonstrate the effectiveness of ReRF for compactly representing dynamic radiance fields, enabling an unprecedented free-viewpoint viewing experience in speed and quality.

CVPR 2023 - DANI-Net: Uncalibrated Photometric Stereo by Differentiable Shadow Handling, Anisotropic Reflectance Modeling, and Neural Inverse Rendering

In this episode we discuss DANI-Net: Uncalibrated Photometric Stereo by Differentiable Shadow Handling, Anisotropic Reflectance Modeling, and Neural Inverse Rendering
by Zongrui Li, Qian Zheng, Boxin Shi, Gang Pan, Xudong Jiang. The paper proposes a deep learning approach, called DANI-Net, to solve the challenging problem of uncalibrated photometric stereo (UPS) which is complicated by unknown lighting. UPS is particularly difficult for non-Lambertian objects with complex shapes and irregular shadows, and for general materials with complex reflectance such as anisotropic reflectance. Unlike previous methods that use non-differentiable shadow maps and assume isotropic material, DANI-Net benefits from cues of shadow and anisotropic reflectance through two differentiable paths, resulting in superior and robust performance on multiple real-world datasets.

CVPR 2023 - Hubs and Hyperspheres: Reducing Hubness and Improving Transductive Few-shot Learning with Hyperspherical Embeddings

In this episode we discuss Hubs and Hyperspheres: Reducing Hubness and Improving Transductive Few-shot Learning with Hyperspherical Embeddings
by Daniel J. Trosten, Rwiddhi Chakraborty, Sigurd Løkse, Kristoffer Knutsen Wickstrøm, Robert Jenssen, Michael C. Kampffmeyer. This paper proposes two approaches to address the hubness problem in distance-based classification in transductive few-shot learning. The authors prove that uniform distribution of representations on the hypersphere can eliminate hubness and the proposed approaches optimize a tradeoff between uniformity and local similarity preservation, reducing hubness while retaining class structure. Experiment results show that the proposed methods significantly improve transductive few-shot learning accuracy for a variety of classifiers.

CVPR 2023 - TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation

In this episode we discuss TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation
by Taeyeop Lee, Jonathan Tremblay, Valts Blukis, Bowen Wen, Byeong-Uk Lee, Inkyu Shin, Stan Birchfield, In So Kweon, Kuk-Jin Yoon. In this paper, the authors propose a method called "Test-Time Adaptation for Category-level Object Pose Estimation" or TTA-COPE, for addressing source-to-target domain gaps. They design a pose ensemble approach using pose-aware confidence and a self-training loss. Unlike previous methods, TTA-COPE processes test data in a sequential, online manner and does not require access to the source domain at runtime. Experimental results show improved category-level object pose performance under semi-supervised and unsupervised settings. The project page for TTA-COPE is available at https://taeyeop.com/ttacope.