AI Breakdown

CVPR 2023 - RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors

In this episode we discuss RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors by Authors: 1. Rui-Qi Wu 2. Zheng-Peng Duan 3. Chun-Le Guo 4. Zhi Chai 5. Chongyi Li Affiliations: 1. VCIP, CS, Nankai University 2. Hisilicon Technologies Co. Ltd. 3. S-Lab, Nanyang Technological University. The paper discusses a new approach to real image dehazing, which addresses the challenges of existing methods that struggle to process real-world hazy images due to the lack of paired real data and robust priors. The proposed method synthesizes more realistic hazy data and introduces more robust priors into the network. The approach includes a phenomenological pipeline that considers diverse degradation types and a Real Image Dehazing network via high-quality Codebook Priors (RIDCP) that utilizes a VQGAN pre-trained on a large-scale high-quality dataset to obtain the discrete codebook encapsulating high-quality priors. Extensive experiments confirm the effectiveness of the proposed approach.

CVPR 2023 - Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

In this episode we discuss Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models by Authors: - Andreas Blattmann - Robin Rombach - Huan Ling - Tim Dockhorn - Seung Wook Kim - Sanja Fidler - Karsten Kreis Affiliations: - Andreas Blattmann and Robin Rombach: LMU Munich - Huan Ling, Seung Wook Kim, Sanja Fidler, and Karsten Kreis: NVIDIA, Vector Institute, and University of Toronto - Tim Dockhorn: University of Waterloo. The paper discusses the use of Latent Diffusion Models (LDMs) to generate high-quality videos without excessive computational demands. The authors pre-train an LDM on images before introducing a temporal dimension to create a video generator, and fine-tune the model on encoded image sequences to achieve state-of-the-art performance on real driving videos of resolution 512 x 1024. They also demonstrate the use of LDMs for text-to-video modeling and personalized content creation. The authors highlight the efficiency and expressiveness of their approach, which can easily leverage pre-trained image LDMs and generalize across different fine-tuned LDMs.

CVPR 2023 - Tracking through Containers and Occluders in the Wild

In this episode we discuss Tracking through Containers and Occluders in the Wild
by Authors:
- Basile Van Hoorick
- Pavel Tokmakov
- Simon Stent
- Jie Li
- Carl Vondrick
Affiliations:
- Basile Van Hoorick and Carl Vondrick: Columbia University
- Pavel Tokmakov and Jie Li: Toyota Research Institute
- Simon Stent: Woven Planet. The paper introduces a benchmark and model called TCOW for visual tracking in cluttered and dynamic environments, which is a difficult challenge for computer vision. The goal is to segment both the projected extent of the target object and the surrounding container or occluder in a given video sequence. The authors create a mixture of synthetic and annotated real datasets to support both supervised learning and structured evaluation of model performance and evaluate two transformer-based video models, finding a considerable performance gap in achieving object permanence.

CVPR 2023 - NeuralEditor: Editing Neural Radiance Fields via Manipulating Point Clouds

In this episode we discuss NeuralEditor: Editing Neural Radiance Fields via Manipulating Point Clouds by Authors: - Jun-Kun Chen - Jipeng Lyu - Yu-Xiong Wang Affiliations: - Jun-Kun Chen and Yu-Xiong Wang: University of Illinois at Urbana-Champaign - Jipeng Lyu: Peking University. The paper introduces NeuralEditor, a new system that allows for easy shape editing of neural radiance fields (NeRFs), which are typically difficult to edit. The system uses the point cloud representation of the scene to build NeRFs and introduces a rendering scheme based on deterministic integration within density-adaptive voxels. The system enables precise point cloud reconstruction and achieves state-of-the-art performance in shape deformation and scene morphing tasks. Code, benchmark, and demo video are available.

CVPR 2023 - Structured Kernel Estimation for Photon-Limited Deconvolution

In this episode we discuss Structured Kernel Estimation for Photon-Limited Deconvolution
by Authors: Yash Sanghvi, Zhiyuan Mao, Stanley H. Chan
Affiliation: School of Electrical and Computer Engineering, Purdue University. The paper proposes a new method for estimating blur in low light conditions with strong photon shot noise, where existing image restoration networks perform poorly. The authors use a gradient-based backpropagation method to estimate the blur kernel and model it using a low-dimensional representation with key points on the motion trajectory, reducing the search space and improving regularity of estimation. Results show improved performance compared to end-to-end trained neural networks when applied to deconvolution in an iterative framework. The code and pretrained models are available online.

CVPR 2023 - Shakes on a Plane: Unsupervised Depth Estimation

In this episode we discuss Shakes on a Plane: Unsupervised Depth Estimation
by Authors:
- Ilya Chugunov
- Yuxuan Zhang
- Felix Heide
Affiliation:
- Princeton University. The paper discusses a new method for recovering high-quality scene depth from long-burst sequences captured by mobile burst photography pipelines. The researchers investigate using natural hand tremor to obtain enough parallax information to recover scene depth. They introduce a test-time optimization approach that simultaneously estimates scene depth and camera motion by fitting a neural RGB-D representation to long-burst data. The method uses a plane plus depth model, which is trained end-to-end and performs coarse-to-fine refinement by controlling which multi-resolution volume features the network has access to at what time during training. The results demonstrate geometrically accurate depth reconstructions with no additional hardware or separate data pre-processing and pose-estimation steps.

CVPR 2023 - Visual Programming: Compositional visual reasoning without training

In this episode we discuss Visual Programming: Compositional visual reasoning without training by Authors: Tanmay Gupta and Aniruddha Kembhavi Affiliation: - PRIOR @ Allen Institute for AI. The paper introduces VISPROG, a neuro-symbolic approach to solving complex visual tasks based on natural language instructions. The system generates python-like modular programs that are executed to produce the solution and a comprehensive rationale. The approach avoids the need for task-specific training and instead uses the in-context learning ability of large language models. The paper demonstrates the flexibility of VISPROG on four diverse tasks, including image editing and factual knowledge object tagging, and shows its potential to expand AI systems to perform complex tasks.

CVPR 2023 - OmniObject3D: Large-Vocabulary 3D Object Dataset for

In this episode we discuss OmniObject3D: Large-Vocabulary 3D Object Dataset for by Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, Dahua Lin, Ziwei Liu. The paper proposes OmniObject3D, a large vocabulary 3D object dataset containing 6,000 high-quality real-scanned objects in 190 daily categories with rich annotations. The dataset aims to facilitate the development of 3D perception, reconstruction, and generation in the real world and is evaluated on four benchmarks: robust 3D perception, novel-view synthesis, neural surface reconstruction, and 3D object generation. The extensive studies on these benchmarks reveal new observations, challenges, and opportunities for future research in realistic 3D vision.

CVPR 2023 - What Can Human Sketches Do for Object Detection?

In this episode we discuss What Can Human Sketches Do for Object Detection? by Authors: - Pinaki Nath Chowdhury - Ayan Kumar Bhunia - Aneeshan Sain - Subhadeep Koley - Tao Xiang - Yi-Zhe Song Affiliation: SketchX, CVSSP, University of Surrey, United Kingdom. The paper proposes a new object detection framework that utilizes sketches to detect objects. It is the first attempt to cultivate the expressiveness of sketches for the task of object detection, with instance-aware and part-aware detection capabilities. The model is designed to work without knowing the category of objects beforehand and without requiring bounding boxes or class labels. The framework combines an existing sketch-based image retrieval (SBIR) model with the generalization ability of CLIP to build highly generalizable sketch and photo encoders that can be adapted for object detection. The proposed framework outperforms both supervised and weakly-supervised object detectors on zero-shot setups in standard object detection datasets like PASCAL-VOC and MS-COCO.

CVPR 2023 - Efficient Multimodal Fusion via Interactive Prompting

In this episode we discuss Efficient Multimodal Fusion via Interactive Prompting
by Authors:
- Yaowei Li
- Ruijie Quan
- Linchao Zhu
- Yi Yang
Affiliations:
- Yaowei Li: ReLER, AAII, University of Technology Sydney
- Ruijie Quan, Linchao Zhu, Yi Yang: CCAI, Zhejiang University
Contact information:
- Yaowei Li: yaowei.li@uts.edu.au
- Ruijie Quan, Linchao Zhu, Yi Yang: {quanruijie, zhulinchao, yangyics}@zju.edu.cn. The paper proposes an efficient and flexible multimodal fusion method, called PMF, for fusing unimodally pre-trained transformers. The proposed method disentangles vanilla prompts into three types to learn different optimizing objectives for multimodal learning. The method adds prompt vectors only on the deep layers of the unimodal transformers, significantly reducing the training memory usage. Experimental results show that the proposed method achieves comparable performance to several other multimodal finetuning methods with less than 3% trainable parameters and up to 66% saving of training memory usage.