AI Breakdown

CVPR 2023 - Unsupervised Contour Tracking of Live Cells by Mechanical and Cycle Consistency Losses

In this episode we discuss Unsupervised Contour Tracking of Live Cells by Mechanical and Cycle Consistency Losses
by Junbong Jang, Kwonmoo Lee, Tae-Kyun Kim. The paper proposes a deep learning-based method for tracking the dynamic changes of cellular morphology in live cell videos. The proposed method includes point correspondence and considering local shapes and textures on the contour, which previous methods did not. Unsupervised learning is used, consisting of mechanical and cyclical consistency losses, to train the contour tracker. The proposed method outperforms existing methods and is publicly available.

CVPR 2023 - Probabilistic Prompt Learning for Dense Prediction

In this episode we discuss Probabilistic Prompt Learning for Dense Prediction
by Hyeongjun Kwon, Taeyong Song, Somi Jeong, Jin Kim, Jinhyun Jang, Kwanghoon Sohn. This paper proposes a new approach called "probabilistic prompt learning" to improve the performance of dense prediction tasks. The authors introduce learnable class-agnostic attribute prompts to describe universal attributes across object classes, which are combined with class information and visual-context knowledge to create a class-specific textual distribution. Text representations are then sampled and used to guide the dense prediction task using a probabilistic pixel-text matching loss, resulting in improved stability and generalization capabilities. The effectiveness of the proposed method is demonstrated through extensive experiments and ablation studies.

CVPR 2023 - SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage

In this episode we discuss SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage
by Yifan Wang, Aleksander Holynski, Xiuming Zhang, Xuaner Zhang. The paper presents SunStage, a lightweight alternative to a light stage that captures facial appearance and relighting data using only a smartphone camera and the sun. The method requires the user to capture a selfie video outdoors and uses the varying angles between the sun and face for joint reconstruction of facial geometry, reflectance, camera pose, and lighting parameters. The approach is able to reconstruct detailed facial appearance and geometry, enabling compelling effects such as relighting, novel view synthesis, and reflectance editing.

CVPR 2023 - Feature Separation and Recalibration for Adversarial Robustness

In this episode we discuss Feature Separation and Recalibration for Adversarial Robustness
by Woo Jae Kim, Yoonki Cho, Junsik Jung, Sung-Eui Yoon. The paper proposes a novel approach called Feature Separation and Recalibration (FSR) to improve the robustness of deep neural networks against adversarial attacks. The FSR method recalibrates the non-robust feature activations, which are responsible for model mispredictions under adversarial attacks, by disentangling them from the robust feature activations and adjusting them to restore potentially useful cues for correct model predictions. The results of extensive experiments show that FSR outperforms traditional deactivation techniques and improves the robustness of existing adversarial training methods by up to 8.57% with minimal computational overhead.

CVPR 2023 - DETR with Additional Global Aggregation for Cross-domain Weakly Supervised Object Detection

In this episode we discuss DETR with Additional Global Aggregation for Cross-domain Weakly Supervised Object Detection
by Zongheng Tang, Yifan Sun, Si Liu, Yi Yang. The paper proposes a method for cross-domain weakly supervised object detection (CDWSOD) by adapting the detector from source to target domain through weak supervision using DETR (transformers-based object detection model). The proposed method, DETR-GA, simultaneously makes "instance-level + image-level" predictions and utilizes "strong + weak" supervisions. The method uses query-based aggregation that helps in locating corresponding positions, excluding distractions from non-relevant regions, and making strong and weak supervision mutually benefit each other for domain alignment. Extensive experiments show that DETR-GA significantly improves cross-domain detection accuracy and advances the state-of-the-art.

CVPR 2023 - Diffusion-SDF: Text-to-Shape via Voxelized Diffusion

In this episode we discuss Diffusion-SDF: Text-to-Shape via Voxelized Diffusion
by Muheng Li, Yueqi Duan, Jie Zhou, Jiwen Lu. The paper presents a new generative 3D modeling framework called Diffusion-SDF for synthesizing 3D shapes from text. The proposed framework uses a SDF autoencoder and Voxelized Diffusion model to generate representations for voxelized signed distance fields (SDFs) of 3D shapes. The researchers developed a novel UinU-Net architecture that improves the reconstruction of patch-independent SDF representations, enabling better text-to-shape synthesis. The results show that the Diffusion-SDF approach generates higher quality and diversified 3D shapes that conform well to given text descriptions, outperforming previous approaches.

CVPR 2023 - Aligning Step-by-Step Instructional Diagrams to Video Demonstrations

In this episode we discuss Aligning Step-by-Step Instructional Diagrams to Video Demonstrations
by Jiahao Zhang, Anoop Cherian, Yanbin Liu, Yizhak Ben-Shabat, Cristian Rodriguez, Stephen Gould. The paper presents a novel approach to align instruction steps depicted as assembly diagrams with segments from in-the-wild videos that depict the actions. The authors propose a supervised contrastive learning method that is guided by a set of novel losses to align videos with the subtle details of assembly diagrams. They introduce a new dataset, IAW, consisting of 183 hours of videos and nearly 8,300 illustrations with ground truth alignments to evaluate the effectiveness of their method. The experimental results demonstrate superior performance compared to alternatives on two defined tasks of nearest neighbor retrieval and alignment of instruction steps and video segments.

CVPR 2023 - AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR

In this episode we discuss AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR
by Paul Hongsuck Seo, Arsha Nagrani, Cordelia Schmid. The paper proposes a method called AVFormer for augmenting audio-only models with visual information for audiovisual automatic speech recognition (AV-ASR). The method involves injecting visual embeddings into a frozen ASR model using lightweight trainable adaptors, which can be trained on a small amount of weakly labelled video data with minimal additional training time and parameters. A simple curriculum scheme is also introduced during training, which is shown to be crucial for the model to jointly process audio and visual information effectively. The proposed model achieves state-of-the-art zero-shot results on three AV-ASR benchmarks while preserving decent performance on traditional audio-only speech recognition benchmarks.

CVPR 2023 - Hard Patches Mining for Masked Image Modeling

In this episode we discuss Hard Patches Mining for Masked Image Modeling
by Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, Zhaoxiang Zhang. The paper proposes a new framework called Hard Patches Mining (HPM) for pre-training in masked image modeling (MIM). The authors argue that MIM models should not only focus on predicting specific contents of masked patches but also on producing challenging problems by themselves. HPM uses an auxiliary loss predictor that predicts patch-wise losses and decides where to mask next, using a relative relationship learning strategy to prevent overfitting. Experiments demonstrate the effectiveness of HPM in constructing masked images and the efficacy of the ability to be aware of where it is hard to reconstruct.

CVPR 2023 - Revisiting Rolling Shutter Bundle Adjustment: Toward Accurate and Fast Solution

In this episode we discuss Revisiting Rolling Shutter Bundle Adjustment: Toward Accurate and Fast Solution
by Bangyan Liao, Delin Qu, Yifei Xue, Huiqing Zhang, Yizhen Lao. The paper proposes a solution for accurate and fast bundle adjustment (BA) to estimate the 6-DoF pose using a rolling shutter camera. The proposed method addresses the challenges in existing works, such as relying on high frame rate video, restrictive assumptions on camera motion, and poor efficiency. The authors demonstrate the positive influence of image point normalization and propose a visual residual covariance model to improve accuracy. Additionally, they propose a combination of normalization and covariance standardization weighting to avoid planar degeneracy and propose an acceleration strategy based on the sparsity of the Jacobian matrix and Schur complement. Experimental results show the effectiveness and efficiency of the proposed solution over existing works.