AI Breakdown

CVPR 2023, highlight paper - Beyond mAP: Towards better evaluation of instance segmentation

In this episode we discuss Beyond mAP: Towards better evaluation of instance segmentation
by Rohit Jena, Lukas Zhornyak, Nehal Doiphode, Pratik Chaudhari, Vivek Buch, James Gee, Jianbo Shi. The paper proposes new measures to account for duplicate predictions in instance segmentation, which the commonly used Average Precision metric does not penalize. The authors suggest a Semantic Sorting and NMS module to remove duplicates based on a pixel occupancy matching scheme. They argue that this approach can mitigate hedged predictions and preserve AP, allowing for a better trade-off between false positives and high recall. The experiments show that modern segmentation networks have a considerable amount of duplicates, which can be reduced with the proposed method.

CVPR 2023, highlight paper - Real-Time Evaluation in Online Continual Learning: A New Hope

In this episode we discuss Real-Time Evaluation in Online Continual Learning: A New Hope
by Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip H. S. Torr, Bernard Ghanem. The paper proposes a practical real-time evaluation of Continual Learning (CL) methods, which takes into account computational costs and does not assume unlimited training time. The evaluation was conducted on a large-scale dataset called CLOC, containing 39 million time-stamped images with geolocation labels. The results show that a simple baseline outperforms state-of-the-art CL methods, suggesting that existing methods may not be applicable in realistic settings. The authors recommend considering computational cost in the development of online continual learning methods.

CVPR 2023, highlight paper - Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagation

In this episode we discuss Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagation
by Heng Yang, Marco Pavone. The paper proposes a two-stage object pose estimation method that uses conformal keypoint detection and geometric uncertainty propagation to endow an estimation with provable and computable worst-case error bounds. Conformal keypoint detection converts heuristic detections into circular or elliptical prediction sets that cover the groundtruth keypoints with a user-specified marginal probability. Geometric uncertainty propagation propagates geometric constraints to the 6D object pose, leading to a Pose UnceRtainty SEt (PURSE) that guarantees coverage of the groundtruth pose with the same probability. The proposed method achieves better or similar accuracy as existing methods on the LineMOD Occlusion dataset while providing correct uncertainty quantification.

CVPR 2023 - Normalizing Flow based Feature Synthesis for Outlier-Aware Object Detection

In this episode we discuss Normalizing Flow based Feature Synthesis for Outlier-Aware Object Detection
by Nishant Kumar, Siniša Šegvić, Abouzar Eslami, Stefan Gumhold. The paper proposes a novel outlier-aware object detection framework that improves on existing approaches by learning the joint data distribution of all inlier classes with an invertible normalizing flow. This ensures that the synthesized outliers have a lower likelihood than inliers from all object classes, resulting in a better decision boundary between inlier and outlier objects. The approach shows significant improvement over the state-of-the-art for outlier-aware object detection on both image and video datasets, making it a promising solution for real-world deployment of reliable object detectors.

CVPR 2023 - Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

In this episode we discuss Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
by Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, Shalini De Mello. The paper presents ODISE, a model that unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation. The approach leverages the frozen internal representations of both models to outperform the previous state of the art by significant margins on both open-vocabulary panoptic and semantic segmentation tasks. They achieve 23.4 PQ and 30.0 mIoU on the ADE20K dataset with 8.3 PQ and 7.9 mIoU absolute improvement over the previous state of the art. The code and models are open-sourced.

CVPR 2023 - Egocentric Video Task Translation

In this episode we discuss Egocentric Video Task Translation
by Zihui Xue, Yale Song, Kristen Grauman, Lorenzo Torresani. The paper proposes a more unified approach to video understanding tasks, specifically in the context of wearable cameras. The authors argue that the egocentric perspective of a person presents an interconnected web of tasks, such as object manipulation and navigation, which should be addressed in conjunction rather than in isolation. The proposed EgoTask Translation (EgoT2) model takes multiple task-specific models and learns to translate their outputs for improved performance on all tasks simultaneously. The model demonstrated superior results compared to existing transfer paradigms on four benchmark challenges.

CVPR 2023 - Learning to Name Classes for Vision and Language Models

In this episode we discuss Learning to Name Classes for Vision and Language Models
by Sarah Parisot, Yongxin Yang, Steven McDonagh. The paper proposes a solution to two challenges faced by large-scale vision and language models in achieving impressive zero-shot recognition performances. These challenges include sensitivity to handcrafted class names defining queries and difficulty in adapting to new, smaller datasets. The proposed solution suggests learning optimal word embeddings for each class as a function of visual content to retain zero-shot capabilities for new classes, adapt models to new datasets, and adjust potentially erroneous or ambiguous class names. The solution is shown to yield significant performance gains in multiple scenarios and provides insights into model biases and labeling errors.

CVPR 2023 - Model-Agnostic Gender Debiased Image Captioning

In this episode we discuss Model-Agnostic Gender Debiased Image Captioning
by Yusuke Hirota, Yuta Nakashima, Noa Garcia. The paper discusses the issue of gender bias in image captioning models and proposes a framework named LIBRA to mitigate such bias. Prior attempts to address this problem by focusing on people created gender-stereotypical words, and it affected gender prediction. The researchers hypothesize that there are two types of bias - exploiting context to predict gender and the probability of generating stereotypical words. The proposed framework learns from synthetic data to decrease both types of bias, correct gender misclassification, and change gender-stereotypical words to more neutral ones.

CVPR 2023 - Magic3D: High-Resolution Text-to-3D Content Creation

In this episode we discuss Magic3D: High-Resolution Text-to-3D Content Creation
by Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin. The paper introduces a two-stage optimization framework called Magic3D to address the slow optimization and low-resolution image space supervision limitations of the pre-trained text-to-image diffusion model called DreamFusion. The first stage involves obtaining a coarse model using a low-resolution diffusion prior and accelerating it with a sparse 3D hash grid structure. In the second stage, a textured 3D mesh model is optimized using an efficient differentiable renderer interacting with a high-resolution latent diffusion model. Magic3D can create high-quality 3D mesh models in 40 minutes, 2x faster than DreamFusion, while achieving higher resolution. User studies show that 61.7% of raters prefer Magic3D over DreamFusion.

CVPR 2023 - Self-Correctable and Adaptable Inference for Generalizable Human Pose Estimation

In this episode we discuss Self-Correctable and Adaptable Inference for Generalizable Human Pose Estimation
by Zhehan Kan, Shuoshuo Chen, Ce Zhang, Yushun Tang, Zhihai He. The paper introduces a self-correctable and adaptable inference (SCAI) method to address the generalization challenge of network prediction. Utilizing human pose estimation as an example, they learn a correction network to correct the prediction result conditioned by a fitness feedback error. This feedback error is generated by a learned fitness feedback network that maps the prediction result to the original input domain and compares it against the original input, which can be used as feedback to guide the correction process and as a loss function to optimize the correction network during the inference process. Experimental results demonstrate that the proposed SCAI method significantly improves the generalization capability and performance of human pose estimation.