AI Breakdown

CVPR 2023 - Post-Processing Temporal Action Detection

In this episode we discuss Post-Processing Temporal Action Detection
by Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, Tao Xiang. The paper proposes a novel model-agnostic post-processing method, called Gaussian Approximated Post-processing (GAP), to improve the performance of Temporal Action Detection (TAD) methods without requiring model redesign and retraining. The existing TAD methods usually have a pre-processing step that temporally downsamples the video, leading to a reduction in inference resolution and negative impact on the TAD performance. GAP models the start and end points of action instances with a Gaussian distribution and enables temporal boundary inference at a sub-snippet level. Experimental results demonstrate that GAP consistently improves a wide variety of pre-trained off-the-shelf TAD models on the ActivityNet and THUMOS benchmarks.

CVPR 2023 - Temporal Interpolation Is All You Need for Dynamic Neural Radiance Fields

In this episode we discuss Temporal Interpolation Is All You Need for Dynamic Neural Radiance Fields
by Sungheon Park, Minjung Son, Seokhwan Jang, Young Chun Ahn, Ji-Yeon Kim, Nahyup Kang. The paper presents a novel technique for training spatiotemporal neural radiance fields for dynamic scenes based on temporal interpolation of feature vectors. The proposed method includes two feature interpolation approaches, one using neural networks and another using grids. The multi-level feature interpolation network captures short-term and long-term time ranges, while the grid representation reduces training time by over 100 times compared to previous neural-net-based methods without sacrificing rendering quality. The addition of a smoothness term and concatenating static and dynamic features further improves the performance of the proposed models. The method achieves state-of-the-art performance in both rendering quality and training speed.

CVPR 2023 - DATE: Domain Adaptive Product Seeker for E-commerce

In this episode we discuss DATE: Domain Adaptive Product Seeker for E-commerce
by Haoyuan Li, Hao Jiang, Tao Jin, Mengyan Li, Yan Chen, Zhijie Lin, Yang Zhao, Zhou Zhao. The paper presents a framework for Product Retrieval (PR) and Grounding (PG) that can seek image and object-level products respectively according to a textual query to aid in better shopping experience. The authors collected two benchmark datasets from Taobao Mall and Live domains with about 474k and 101k image-query pairs for PR, and manually annotated the object bounding boxes in each image for PG. They propose a Domain Adaptive Product Seeker (DATE) framework that can perform un-supervised Domain Adaptation (PG-DA) by transferring knowledge from annotated to unannotated domains. The DATE achieved satisfactory performance in fully-supervised PR, PG and un-supervised PG-DA.

CVPR 2023 - Visibility Aware Human-Object Interaction Tracking from Single RGB Camera

In this episode we discuss Visibility Aware Human-Object Interaction Tracking from Single RGB Camera
by Xianghui Xie, Bharat Lal Bhatnagar, Gerard Pons-Moll. The paper proposes a method to track the 3D human and object, their contacts, and their relative translation across frames from a single RGB camera while being robust to heavy occlusions. The authors improved on the previous methods that assumed a fixed depth and suffered from significant drops in performance when the object was occluded. The proposed method uses a neural field reconstruction conditioned on per-frame SMPL model estimates and a transformer-based neural network that leverages neighboring frames to make predictions for the occluded frames, achieving significantly better performance than state-of-the-art methods.

CVPR 2023 - DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

In this episode we discuss DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
by Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, Kfir Aberman. The paper discusses a new approach to personalize text-to-image diffusion models by fine-tuning the pre-trained model with a few images of a particular subject, allowing the model to learn a unique identifier associated with that subject. The unique identifier enables the synthesis of novel photorealistic images of the subject in different scenes. Through a new autogenous class-specific prior preservation loss, the technique facilitates subject synthesis in diverse poses, lighting conditions, and views, providing impressive results for various applications, including subject recontextualization, text-guided view synthesis, and artistic rendering.

CVPR 2023 - PolyFormer: Referring Image Segmentation as Sequential Polygon Generation

In this episode we discuss PolyFormer: Referring Image Segmentation as Sequential Polygon Generation
by Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, R. Manmatha. The paper presents a new approach to referring image segmentation that uses sequential polygon generation instead of directly predicting pixel-level masks. The method, called Polygon Transformer (PolyFormer), takes a sequence of image patches and text query tokens as input and outputs a sequence of polygon vertices. A regression-based decoder is also proposed for more accurate geometric localization. In experiments, PolyFormer outperforms prior methods on challenging datasets and shows strong generalization ability on referring video segmentation without fine-tuning.

CVPR 2023 - Noisy Correspondence Learning with Meta Similarity Correction

In this episode we discuss Noisy Correspondence Learning with Meta Similarity Correction
by Haochen Han, Kaiyao Miao, Qinghua Zheng, Minnan Luo. The paper proposes a Meta Similarity Correction Network (MSCN) to address the problem of noisy correspondence datasets, which causes performance degradation in cross-modal retrieval methods. MSCN provides reliable similarity scores by viewing a binary classification task as the meta-process that encourages discrimination from positive and negative meta-data. Additionally, the paper presents an effective data purification strategy that uses meta-data as prior knowledge to remove noisy samples. The proposed method is evaluated in both synthetic and real-world noise datasets, demonstrating its effectiveness in improving cross-modal retrieval performance.

CVPR 2023 - Query-Dependent Video Representation for Moment Retrieval and Highlight Detection

In this episode we discuss Query-Dependent Video Representation for Moment Retrieval and Highlight Detection
by WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, Jae-Pil Heo. The paper introduces Query-Dependent DETR (QD-DETR), a detection transformer model designed for video moment retrieval and highlight detection (MR/HD). The previous transformer-based models did not exploit the information of a given query, neglecting the relevance between the text query and video contents. QD-DETR addresses this issue by introducing cross-attention layers to inject the context of the text query into video representation and manipulating video-query pairs to produce irrelevant pairs. Additionally, the paper presents an input-adaptive saliency predictor that adaptively defines the criterion of saliency scores for given video-query pairs. The performance of QD-DETR outperforms state-of-the-art methods on QVHighlights, TVSum, and Charades-STA datasets.

CVPR 2023 - Exploring Data Geometry for Continual Learning

In this episode we discuss Exploring Data Geometry for Continual Learning
by Zhi Gao, Chen Xu, Feng Li, Yunde Jia, Mehrtash Harandi, Yuwei Wu. The paper explores the concept of continual learning, which involves effectively learning from a constantly changing stream of data without forgetting the knowledge gained from the old data. The study analyzes how data geometry can be used to achieve this goal, especially for non-Euclidean data structures that cannot be captured using Euclidean space. The proposed method involves dynamically expanding the geometry of the underlying space to account for new data and preserving old data's geometric structures. The approach achieves better performance than traditional Euclidean-based methods.

CVPR 2023 - The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection

In this episode we discuss The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection
by Geoffroi Côté, Fahim Mannan, Simon Thibault, Jean-François Lalonde, Felix Heide. This paper proposes a novel approach to joint optimization of camera lens systems with other image processing components, particularly neural networks. The authors introduce a differentiable spherical lens simulation model and an optimization strategy that includes quantized continuous glass variables and manufacturability constraints to facilitate the optimization and selection of glass materials. The proposed method achieves improved detection performance in automotive object detection, even when lens designs are simplified to two- or three-element lenses, despite degrading image quality.