AI Breakdown

CVPR 2023 - Diffusion-SDF: Text-to-Shape via Voxelized Diffusion

In this episode we discuss Diffusion-SDF: Text-to-Shape via Voxelized Diffusion
by Muheng Li, Yueqi Duan, Jie Zhou, Jiwen Lu. The paper presents a new generative 3D modeling framework called Diffusion-SDF for synthesizing 3D shapes from text. The proposed framework uses a SDF autoencoder and Voxelized Diffusion model to generate representations for voxelized signed distance fields (SDFs) of 3D shapes. The researchers developed a novel UinU-Net architecture that improves the reconstruction of patch-independent SDF representations, enabling better text-to-shape synthesis. The results show that the Diffusion-SDF approach generates higher quality and diversified 3D shapes that conform well to given text descriptions, outperforming previous approaches.

CVPR 2023 - Aligning Step-by-Step Instructional Diagrams to Video Demonstrations

In this episode we discuss Aligning Step-by-Step Instructional Diagrams to Video Demonstrations
by Jiahao Zhang, Anoop Cherian, Yanbin Liu, Yizhak Ben-Shabat, Cristian Rodriguez, Stephen Gould. The paper presents a novel approach to align instruction steps depicted as assembly diagrams with segments from in-the-wild videos that depict the actions. The authors propose a supervised contrastive learning method that is guided by a set of novel losses to align videos with the subtle details of assembly diagrams. They introduce a new dataset, IAW, consisting of 183 hours of videos and nearly 8,300 illustrations with ground truth alignments to evaluate the effectiveness of their method. The experimental results demonstrate superior performance compared to alternatives on two defined tasks of nearest neighbor retrieval and alignment of instruction steps and video segments.

CVPR 2023 - AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR

In this episode we discuss AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR
by Paul Hongsuck Seo, Arsha Nagrani, Cordelia Schmid. The paper proposes a method called AVFormer for augmenting audio-only models with visual information for audiovisual automatic speech recognition (AV-ASR). The method involves injecting visual embeddings into a frozen ASR model using lightweight trainable adaptors, which can be trained on a small amount of weakly labelled video data with minimal additional training time and parameters. A simple curriculum scheme is also introduced during training, which is shown to be crucial for the model to jointly process audio and visual information effectively. The proposed model achieves state-of-the-art zero-shot results on three AV-ASR benchmarks while preserving decent performance on traditional audio-only speech recognition benchmarks.

CVPR 2023 - Hard Patches Mining for Masked Image Modeling

In this episode we discuss Hard Patches Mining for Masked Image Modeling
by Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, Zhaoxiang Zhang. The paper proposes a new framework called Hard Patches Mining (HPM) for pre-training in masked image modeling (MIM). The authors argue that MIM models should not only focus on predicting specific contents of masked patches but also on producing challenging problems by themselves. HPM uses an auxiliary loss predictor that predicts patch-wise losses and decides where to mask next, using a relative relationship learning strategy to prevent overfitting. Experiments demonstrate the effectiveness of HPM in constructing masked images and the efficacy of the ability to be aware of where it is hard to reconstruct.

CVPR 2023 - Revisiting Rolling Shutter Bundle Adjustment: Toward Accurate and Fast Solution

In this episode we discuss Revisiting Rolling Shutter Bundle Adjustment: Toward Accurate and Fast Solution
by Bangyan Liao, Delin Qu, Yifei Xue, Huiqing Zhang, Yizhen Lao. The paper proposes a solution for accurate and fast bundle adjustment (BA) to estimate the 6-DoF pose using a rolling shutter camera. The proposed method addresses the challenges in existing works, such as relying on high frame rate video, restrictive assumptions on camera motion, and poor efficiency. The authors demonstrate the positive influence of image point normalization and propose a visual residual covariance model to improve accuracy. Additionally, they propose a combination of normalization and covariance standardization weighting to avoid planar degeneracy and propose an acceleration strategy based on the sparsity of the Jacobian matrix and Schur complement. Experimental results show the effectiveness and efficiency of the proposed solution over existing works.

CVPR 2023, highlight paper - Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning

In this episode we discuss Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning
by Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, Alan Yuille. The paper introduces a virtual benchmark called Super-CLEVR to isolate different factors of variation that affect the performance of Visual Question Answering (VQA) models on out-of-distribution data and domain generalization. The benchmark considers four factors, including visual complexity, question redundancy, concept distribution, and concept compositionality, to enable testing of VQA methods in situations where test data differs from training data along each of these axes. The authors study four existing methods and propose a new probabilistic NSVQA (P-NSVQA) method, which outperforms others on three of the four domain shift factors, indicating that disentangling reasoning and perception, combined with probabilistic uncertainty, forms a strong VQA model that is more robust to domain shifts.

CVPR 2023, highlight paper - Quantum Multi-Model Fitting

In this episode we discuss Quantum Multi-Model Fitting
by Matteo Farina, Luca Magri, Willi Menapace, Elisa Ricci, Vladislav Golyanik, Federica Arrigoni. This paper introduces the first quantum approach to multi-model fitting (MMF), a fundamental computer vision problem. The authors propose a formulation that can be efficiently sampled on modern adiabatic quantum computers, without the relaxation of the objective function. They also propose an iterative and decomposed version of their method, which supports real-world-sized problems and show promising experimental results on various datasets. The source code is available on GitHub.

CVPR 2023, highlight paper - DiffRF: Rendering-Guided 3D Radiance Field Diffusion

In this episode we discuss DiffRF: Rendering-Guided 3D Radiance Field Diffusion
by Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Matthias Nießner. The paper introduces a novel approach for 3D radiance field synthesis called DiffRF, which is based on denoising diffusion probabilistic models. Unlike existing diffusion-based methods that operate on images, latent codes, or point cloud data, DiffRF directly generates volumetric radiance fields. The model addresses the challenge of obtaining ground truth radiance field samples by pairing the denoising formulation with a rendering loss. DiffRF learns multi-view consistent priors, enabling free-view synthesis and accurate shape generation, and naturally enables conditional generation such as masked completion or single-view 3D synthesis at inference time.

CVPR 2023, highlight paper - SPARF: Neural Radiance Fields from Sparse and Noisy Poses

In this episode we discuss SPARF: Neural Radiance Fields from Sparse and Noisy Poses
by Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, Federico Tombari. This paper introduces Sparse Pose Adjusting Radiance Field (SPARF), a method for synthesizing photorealistic novel views with only a few input images and noisy camera poses. SPARF uses multi-view geometry constraints to jointly learn the Neural Radiance Field (NeRF) and refine the camera poses. The approach sets a new state-of-the-art in the sparse-view regime on multiple challenging datasets by enforcing a global and geometrically accurate solution through a multi-view correspondence objective and depth consistency loss.

CVPR 2023, highlight paper - F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories

In this episode we discuss F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories by Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, Wenping Wang. The paper presents a new grid-based NeRF called F2-NeRF which allows arbitrary input camera trajectories and is faster to train. Existing fast grid-based NeRF training frameworks are designed for bounded scenes and rely on space warping but cannot process arbitrary trajectories. The paper proposes a new space-warping method called perspective warping to handle unbounded scenes and demonstrates its effectiveness through experiments on standard and newly collected datasets.