AI Breakdown

CVPR 2023 - The Resource Problem of Using Linear Layer Leakage Attack in Federated Learning

In this episode we discuss The Resource Problem of Using Linear Layer Leakage Attack in Federated Learning
by Joshua C. Zhao, Ahmed Roushdy Elkordy, Atul Sharma, Yahya H. Ezzeldin, Salman Avestimehr, Saurabh Bagchi. The paper discusses the usage of secure aggregation in federated learning, which promises to maintain privacy by only allowing the server access to a decrypted aggregate update. The paper focuses on linear layer leakage methods, which are the only data reconstruction attacks that can scale regardless of the number of clients or batch sizes. However, the method of injecting a fully-connected layer to increase the leakage rate results in a large resource overhead. The authors propose using sparsity to decrease the model size overhead and computation time while maintaining an equivalent total leakage rate of 77%.

CVPR 2023 - Spatiotemporal Self-supervised Learning for Point Clouds in the Wild

In this episode we discuss Spatiotemporal Self-supervised Learning for Point Clouds in the Wild
by Yanhao Wu, Tong Zhang, Wei Ke, Sabine Süsstrunk, Mathieu Salzmann. The paper discusses a new self-supervised learning strategy for semantic segmentation of point clouds that leverages positive pairs in both the spatial and temporal domain. The authors designed a point-to-cluster learning strategy to distinguish objects and a cluster-to-cluster learning strategy based on unsupervised object tracking that exploits temporal correspondences. The approach was demonstrated through extensive experiments showing improved performance over state-of-the-art point cloud SSL methods on two large-scale LiDAR datasets and transferring models to other point cloud segmentation benchmarks.

CVPR 2023 - Masked Image Modeling with Local Multi-Scale Reconstruction

In this episode we discuss Masked Image Modeling with Local Multi-Scale Reconstruction
by Haoqing Wang, Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhi-Hong Deng, Kai Han. The paper proposes a new self-supervised representation learning approach called Masked Image Modeling (MIM) that achieves outstanding success, but with a huge computational burden and slow learning process. To address this, the paper proposes a design that applies MIM to multiple local layers, including lower and upper layers, to explicitly guide them. The approach also facilitates multi-scale semantic understanding by reconstructing fine and coarse-scale supervision signals. This approach achieves comparable or better performance on classification, detection, and segmentation tasks than existing MIM models, with significantly less pre-training burden. Code is available with both MindSpore and PyTorch.

CVPR 2023 - CCuantuMM: Cycle-Consistent Quantum-Hybrid Matching of Multiple Shapes

In this episode we discuss CCuantuMM: Cycle-Consistent Quantum-Hybrid Matching of Multiple Shapes
by Harshil Bhatia, Edith Tretschk, Zorah Lähner, Marcel Seelbach Benkner, Michael Moeller, Christian Theobalt, Vladislav Golyanik. This paper proposes a quantum-hybrid approach for the challenging problem of jointly matching multiple, non-rigidly deformed 3D shapes, which is NP-hard. The approach is cycle-consistent and iterative, making it suitable for modern adiabatic quantum hardware and scaling linearly with the total number of input shapes. The N-shape case is reduced to a sequence of three-shape matchings, and high-quality solutions with low energy are retrieved using quantum annealing. The proposed approach significantly outperforms previous quantum-hybrid and classical multi-matching methods on benchmark datasets.

CVPR 2023 - Contrastive Mean Teacher for Domain Adaptive Object Detectors

In this episode we discuss Contrastive Mean Teacher for Domain Adaptive Object Detectors
by Shengcao Cao, Dhiraj Joshi, Liang-Yan Gui, Yu-Xiong Wang. The paper proposes a unified framework called Contrastive Mean Teacher (CMT) that integrates mean-teacher self-training and contrastive learning to overcome the domain gap in object detection. CMT extracts object-level features using low-quality pseudo-labels and optimizes them via contrastive learning without requiring labels in the target domain. The proposed framework achieves a new state-of-the-art target-domain performance of 51.9% mAP on Foggy Cityscapes, outperforming the best previous method by 2.1% mAP.

ICLR 2023 - F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models

In this episode we discuss F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
by Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, Anelia Angelova. I'm sorry, there is no abstract provided for me to summarize. Could you please provide the abstract or more information about the paper you would like me to summarize?

CVPR 2023 - ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

In this episode we discuss ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
by Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, Silvio Savarese. The paper introduces ULIP, a framework that learns a unified representation of images, texts, and 3D point clouds to overcome the limited recognition capabilities of current 3D models due to datasets with a small number of annotated data and a pre-defined set of categories. ULIP pre-trains with object triplets from the three modalities, using a pre-trained vision-language model to overcome the shortage of training triplets, and then learns a 3D representation space aligned with the common image-text space using synthesized triplets. Results show that ULIP improves the performance of multiple recent 3D backbones, achieving state-of-the-art performance in both standard and zero-shot 3D classification on several datasets.

CVPR 2023 - Masked Autoencoding Does Not Help Natural Language Supervision at Scale

In this episode we discuss Masked Autoencoding Does Not Help Natural Language Supervision at Scale
by Floris Weers, Vaishaal Shankar, Angelos Katharopoulos, Yinfei Yang, Tom Gunter. The paper explores the effectiveness of combining self-supervision and natural language supervision for training general purpose image encoders. While recent works have shown promising results with small pre-training datasets, the paper investigates whether the same approach is effective with larger datasets (>100M samples). The authors find that combining masked auto-encoders and contrastive language image pre-training provides little to no benefit over CLIP when trained on a large corpus of 1.4B images, providing clarity on the effectiveness of self supervision for large-scale image-text training.

CVPR 2023 - Weakly Supervised Video Representation Learning with Unaligned Text for Sequential Videos

In this episode we discuss Weakly Supervised Video Representation Learning with Unaligned Text for Sequential Videos
by Sixun Dong, Huazhang Hu, Dongze Lian, Weixin Luo, Yicheng Qian, Shenghua Gao. The paper proposes a weakly supervised approach for sequential video understanding, where time-stamp level text-video alignment is not provided. The proposed method uses a transformer to aggregate frame-level features for video representation and a pre-trained text encoder to encode texts corresponding to each action and the whole video. The proposed multiple granularity loss includes a video-paragraph contrastive loss and a frame-sentence contrastive loss, where pseudo frame-sentence correspondence is generated to supervise the network training. Experimental results demonstrate the effectiveness of the proposed approach, outperforming baselines by a large margin.

CVPR 2023 - Attribute-preserving Face Dataset Anonymization via Latent Code Optimization

In this episode we discuss Attribute-preserving Face Dataset Anonymization via Latent Code Optimization
by Simone Barattin, Christos Tzelepis, Ioannis Patras, Nicu Sebe. The paper presents a task-agnostic approach for anonymizing the identities of faces in a dataset of images while retaining the facial attributes necessary for downstream tasks. The proposed method optimizes the latent representation of images in the latent space of a pre-trained GAN, ensuring the desired distance between the original image and its anonymized version, with an identity obfuscation loss. A novel feature-matching loss is used to preserve facial attributes, and experiments show that the method better preserves these attributes compared to existing approaches. Code and pre-trained models are publicly available.