AI Breakdown

CVPR 2023 - Improving GAN Training via Feature Space Shrinkage

In this episode we discuss Improving GAN Training via Feature Space Shrinkage
by Haozhe Liu, Wentian Zhang, Bing Li, Haoqian Wu, Nanjun He, Yawen Huang, Yuexiang Li, Bernard Ghanem, Yefeng Zheng. The paper proposes a new method, called AdaptiveMix, for training Generative Adversarial Networks (GANs) from a robust image classification perspective. The proposed method shrinks data regions in the image representation space of the discriminator, making it easier to train the GANs. Hard samples are constructed by mixing a pair of training images to narrow down the feature distance between hard and easy samples. The proposed approach is evaluated on several datasets and shown to facilitate GAN training and improve generated image quality. The code is publicly available for use.

CVPR 2023 - CRAFT: Concept Recursive Activation FacTorization for Explainability

In this episode we discuss CRAFT: Concept Recursive Activation FacTorization for Explainability
by Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, Thomas Serre. The paper introduces a new approach called CRAFT to identify "what" and "where" a model looks at in an image. The approach generates concept-based explanations and introduces new ingredients to the automatic concept extraction literature. The proposed concept importance estimation technique is more faithful to the model than previous methods and the usefulness of the approach is demonstrated in both human and computer vision experiments. The code for CRAFT is freely available on GitHub.

CVPR 2023 - gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction

In this episode we discuss gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction
by Zerui Chen, Shizhe Chen, Cordelia Schmid, Ivan Laptev. The paper presents a method for reconstructing 3D shapes of hands and manipulated objects from monocular RGB images using signed distance functions (SDFs) as a framework. The authors exploit the hand structure to guide the SDF-based shape reconstruction by estimating poses of hands and objects and aligning SDFs with highly-articulated hand poses. They also use temporal information to enhance the method's robustness to occlusion and motion blurs. Extensive experiments on challenging benchmarks demonstrate significant improvements over the state-of-the-art in 3D shape reconstruction.

CVPR 2023 - The Resource Problem of Using Linear Layer Leakage Attack in Federated Learning

In this episode we discuss The Resource Problem of Using Linear Layer Leakage Attack in Federated Learning
by Joshua C. Zhao, Ahmed Roushdy Elkordy, Atul Sharma, Yahya H. Ezzeldin, Salman Avestimehr, Saurabh Bagchi. The paper discusses the usage of secure aggregation in federated learning, which promises to maintain privacy by only allowing the server access to a decrypted aggregate update. The paper focuses on linear layer leakage methods, which are the only data reconstruction attacks that can scale regardless of the number of clients or batch sizes. However, the method of injecting a fully-connected layer to increase the leakage rate results in a large resource overhead. The authors propose using sparsity to decrease the model size overhead and computation time while maintaining an equivalent total leakage rate of 77%.

CVPR 2023 - Spatiotemporal Self-supervised Learning for Point Clouds in the Wild

In this episode we discuss Spatiotemporal Self-supervised Learning for Point Clouds in the Wild
by Yanhao Wu, Tong Zhang, Wei Ke, Sabine Süsstrunk, Mathieu Salzmann. The paper discusses a new self-supervised learning strategy for semantic segmentation of point clouds that leverages positive pairs in both the spatial and temporal domain. The authors designed a point-to-cluster learning strategy to distinguish objects and a cluster-to-cluster learning strategy based on unsupervised object tracking that exploits temporal correspondences. The approach was demonstrated through extensive experiments showing improved performance over state-of-the-art point cloud SSL methods on two large-scale LiDAR datasets and transferring models to other point cloud segmentation benchmarks.

CVPR 2023 - Masked Image Modeling with Local Multi-Scale Reconstruction

In this episode we discuss Masked Image Modeling with Local Multi-Scale Reconstruction
by Haoqing Wang, Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhi-Hong Deng, Kai Han. The paper proposes a new self-supervised representation learning approach called Masked Image Modeling (MIM) that achieves outstanding success, but with a huge computational burden and slow learning process. To address this, the paper proposes a design that applies MIM to multiple local layers, including lower and upper layers, to explicitly guide them. The approach also facilitates multi-scale semantic understanding by reconstructing fine and coarse-scale supervision signals. This approach achieves comparable or better performance on classification, detection, and segmentation tasks than existing MIM models, with significantly less pre-training burden. Code is available with both MindSpore and PyTorch.

CVPR 2023 - CCuantuMM: Cycle-Consistent Quantum-Hybrid Matching of Multiple Shapes

In this episode we discuss CCuantuMM: Cycle-Consistent Quantum-Hybrid Matching of Multiple Shapes
by Harshil Bhatia, Edith Tretschk, Zorah Lähner, Marcel Seelbach Benkner, Michael Moeller, Christian Theobalt, Vladislav Golyanik. This paper proposes a quantum-hybrid approach for the challenging problem of jointly matching multiple, non-rigidly deformed 3D shapes, which is NP-hard. The approach is cycle-consistent and iterative, making it suitable for modern adiabatic quantum hardware and scaling linearly with the total number of input shapes. The N-shape case is reduced to a sequence of three-shape matchings, and high-quality solutions with low energy are retrieved using quantum annealing. The proposed approach significantly outperforms previous quantum-hybrid and classical multi-matching methods on benchmark datasets.

CVPR 2023 - Contrastive Mean Teacher for Domain Adaptive Object Detectors

In this episode we discuss Contrastive Mean Teacher for Domain Adaptive Object Detectors
by Shengcao Cao, Dhiraj Joshi, Liang-Yan Gui, Yu-Xiong Wang. The paper proposes a unified framework called Contrastive Mean Teacher (CMT) that integrates mean-teacher self-training and contrastive learning to overcome the domain gap in object detection. CMT extracts object-level features using low-quality pseudo-labels and optimizes them via contrastive learning without requiring labels in the target domain. The proposed framework achieves a new state-of-the-art target-domain performance of 51.9% mAP on Foggy Cityscapes, outperforming the best previous method by 2.1% mAP.

ICLR 2023 - F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models

In this episode we discuss F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
by Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, Anelia Angelova. I'm sorry, there is no abstract provided for me to summarize. Could you please provide the abstract or more information about the paper you would like me to summarize?

CVPR 2023 - ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

In this episode we discuss ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
by Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, Silvio Savarese. The paper introduces ULIP, a framework that learns a unified representation of images, texts, and 3D point clouds to overcome the limited recognition capabilities of current 3D models due to datasets with a small number of annotated data and a pre-defined set of categories. ULIP pre-trains with object triplets from the three modalities, using a pre-trained vision-language model to overcome the shortage of training triplets, and then learns a 3D representation space aligned with the common image-text space using synthesized triplets. Results show that ULIP improves the performance of multiple recent 3D backbones, achieving state-of-the-art performance in both standard and zero-shot 3D classification on several datasets.