AI Breakdown

CVPR 2023, highlight paper - Learning Customized Visual Models with Retrieval-Augmented Knowledge

In this episode we discuss Learning Customized Visual Models with Retrieval-Augmented Knowledge
by Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, Chunyuan Li. The paper proposes a framework called REACT (REtrieval-Augmented CusTomization) to build customized visual models for specific domains. Instead of using expensive pre-training, REACT retrieves relevant image-text pairs from a web-scale database as external knowledge and only trains new modularized blocks while freezing original weights. The framework is shown to be effective in various tasks, including zero-shot classification, with up to 5.4% improvement on ImageNet and 3.7% on the ELEVATER benchmark compared to CLIP.

CVPR 2023, highlight paper - SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries

In this episode we discuss SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries
by Ahmed Imtiaz Humayun, Randall Balestriero, Guha Balakrishnan, Richard Baraniuk. This paper presents a new method called SplineCam that enables exact computation of the geometry of a deep network's (DN) mapping, including its decision boundary, without resorting to approximations such as sampling or architecture simplification. SplineCam works for any DN architecture based on Continuous Piece-Wise Linear (CPWL) nonlinearities and can be used for regression DNs as well. This method facilitates comparison of architectures, generalizability measurement, and sampling from the decision boundary on or off the manifold.

CVPR 2023, highlight paper - Nighttime Smartphone Reflective Flare Removal Using Optical Center Symmetry Prior

In this episode we discuss Nighttime Smartphone Reflective Flare Removal Using Optical Center Symmetry Prior
by Yuekun Dai, Yihang Luo, Shangchen Zhou, Chongyi Li, Chen Change Loy. The paper proposes a method to address the problem of reflective flare in photos caused by light reflecting inside lenses and creating bright spots or a "ghosting effect". Existing methods for detecting these bright spots often fail to identify reflective flares created by various types of light and may even mistakenly remove light sources in scenarios with multiple light sources. The proposed method uses an optical center symmetry prior, which suggests that the reflective flare and light source are always symmetrical around the lens's optical center. The authors also create a reflective flare removal dataset called BracketFlare using continuous bracketing to capture the reflective flare pattern. The method demonstrated effective results on both synthetic and real-world datasets.

CVPR 2023, highlight paper - Beyond mAP: Towards better evaluation of instance segmentation

In this episode we discuss Beyond mAP: Towards better evaluation of instance segmentation
by Rohit Jena, Lukas Zhornyak, Nehal Doiphode, Pratik Chaudhari, Vivek Buch, James Gee, Jianbo Shi. The paper proposes new measures to account for duplicate predictions in instance segmentation, which the commonly used Average Precision metric does not penalize. The authors suggest a Semantic Sorting and NMS module to remove duplicates based on a pixel occupancy matching scheme. They argue that this approach can mitigate hedged predictions and preserve AP, allowing for a better trade-off between false positives and high recall. The experiments show that modern segmentation networks have a considerable amount of duplicates, which can be reduced with the proposed method.

CVPR 2023, highlight paper - Real-Time Evaluation in Online Continual Learning: A New Hope

In this episode we discuss Real-Time Evaluation in Online Continual Learning: A New Hope
by Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip H. S. Torr, Bernard Ghanem. The paper proposes a practical real-time evaluation of Continual Learning (CL) methods, which takes into account computational costs and does not assume unlimited training time. The evaluation was conducted on a large-scale dataset called CLOC, containing 39 million time-stamped images with geolocation labels. The results show that a simple baseline outperforms state-of-the-art CL methods, suggesting that existing methods may not be applicable in realistic settings. The authors recommend considering computational cost in the development of online continual learning methods.

CVPR 2023, highlight paper - Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagation

In this episode we discuss Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagation
by Heng Yang, Marco Pavone. The paper proposes a two-stage object pose estimation method that uses conformal keypoint detection and geometric uncertainty propagation to endow an estimation with provable and computable worst-case error bounds. Conformal keypoint detection converts heuristic detections into circular or elliptical prediction sets that cover the groundtruth keypoints with a user-specified marginal probability. Geometric uncertainty propagation propagates geometric constraints to the 6D object pose, leading to a Pose UnceRtainty SEt (PURSE) that guarantees coverage of the groundtruth pose with the same probability. The proposed method achieves better or similar accuracy as existing methods on the LineMOD Occlusion dataset while providing correct uncertainty quantification.

CVPR 2023 - Normalizing Flow based Feature Synthesis for Outlier-Aware Object Detection

In this episode we discuss Normalizing Flow based Feature Synthesis for Outlier-Aware Object Detection
by Nishant Kumar, Siniša Šegvić, Abouzar Eslami, Stefan Gumhold. The paper proposes a novel outlier-aware object detection framework that improves on existing approaches by learning the joint data distribution of all inlier classes with an invertible normalizing flow. This ensures that the synthesized outliers have a lower likelihood than inliers from all object classes, resulting in a better decision boundary between inlier and outlier objects. The approach shows significant improvement over the state-of-the-art for outlier-aware object detection on both image and video datasets, making it a promising solution for real-world deployment of reliable object detectors.

CVPR 2023 - Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

In this episode we discuss Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
by Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, Shalini De Mello. The paper presents ODISE, a model that unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation. The approach leverages the frozen internal representations of both models to outperform the previous state of the art by significant margins on both open-vocabulary panoptic and semantic segmentation tasks. They achieve 23.4 PQ and 30.0 mIoU on the ADE20K dataset with 8.3 PQ and 7.9 mIoU absolute improvement over the previous state of the art. The code and models are open-sourced.

CVPR 2023 - Egocentric Video Task Translation

In this episode we discuss Egocentric Video Task Translation
by Zihui Xue, Yale Song, Kristen Grauman, Lorenzo Torresani. The paper proposes a more unified approach to video understanding tasks, specifically in the context of wearable cameras. The authors argue that the egocentric perspective of a person presents an interconnected web of tasks, such as object manipulation and navigation, which should be addressed in conjunction rather than in isolation. The proposed EgoTask Translation (EgoT2) model takes multiple task-specific models and learns to translate their outputs for improved performance on all tasks simultaneously. The model demonstrated superior results compared to existing transfer paradigms on four benchmark challenges.

CVPR 2023 - Learning to Name Classes for Vision and Language Models

In this episode we discuss Learning to Name Classes for Vision and Language Models
by Sarah Parisot, Yongxin Yang, Steven McDonagh. The paper proposes a solution to two challenges faced by large-scale vision and language models in achieving impressive zero-shot recognition performances. These challenges include sensitivity to handcrafted class names defining queries and difficulty in adapting to new, smaller datasets. The proposed solution suggests learning optimal word embeddings for each class as a function of visual content to retain zero-shot capabilities for new classes, adapt models to new datasets, and adjust potentially erroneous or ambiguous class names. The solution is shown to yield significant performance gains in multiple scenarios and provides insights into model biases and labeling errors.