AI Breakdown

CVPR 2023 - NeuralEditor: Editing Neural Radiance Fields via Manipulating Point Clouds

In this episode we discuss NeuralEditor: Editing Neural Radiance Fields via Manipulating Point Clouds by Authors: - Jun-Kun Chen - Jipeng Lyu - Yu-Xiong Wang Affiliations: - Jun-Kun Chen and Yu-Xiong Wang: University of Illinois at Urbana-Champaign - Jipeng Lyu: Peking University. The paper introduces NeuralEditor, a new system that allows for easy shape editing of neural radiance fields (NeRFs), which are typically difficult to edit. The system uses the point cloud representation of the scene to build NeRFs and introduces a rendering scheme based on deterministic integration within density-adaptive voxels. The system enables precise point cloud reconstruction and achieves state-of-the-art performance in shape deformation and scene morphing tasks. Code, benchmark, and demo video are available.

CVPR 2023 - Structured Kernel Estimation for Photon-Limited Deconvolution

In this episode we discuss Structured Kernel Estimation for Photon-Limited Deconvolution
by Authors: Yash Sanghvi, Zhiyuan Mao, Stanley H. Chan
Affiliation: School of Electrical and Computer Engineering, Purdue University. The paper proposes a new method for estimating blur in low light conditions with strong photon shot noise, where existing image restoration networks perform poorly. The authors use a gradient-based backpropagation method to estimate the blur kernel and model it using a low-dimensional representation with key points on the motion trajectory, reducing the search space and improving regularity of estimation. Results show improved performance compared to end-to-end trained neural networks when applied to deconvolution in an iterative framework. The code and pretrained models are available online.

CVPR 2023 - Shakes on a Plane: Unsupervised Depth Estimation

In this episode we discuss Shakes on a Plane: Unsupervised Depth Estimation
by Authors:
- Ilya Chugunov
- Yuxuan Zhang
- Felix Heide
Affiliation:
- Princeton University. The paper discusses a new method for recovering high-quality scene depth from long-burst sequences captured by mobile burst photography pipelines. The researchers investigate using natural hand tremor to obtain enough parallax information to recover scene depth. They introduce a test-time optimization approach that simultaneously estimates scene depth and camera motion by fitting a neural RGB-D representation to long-burst data. The method uses a plane plus depth model, which is trained end-to-end and performs coarse-to-fine refinement by controlling which multi-resolution volume features the network has access to at what time during training. The results demonstrate geometrically accurate depth reconstructions with no additional hardware or separate data pre-processing and pose-estimation steps.

CVPR 2023 - Visual Programming: Compositional visual reasoning without training

In this episode we discuss Visual Programming: Compositional visual reasoning without training by Authors: Tanmay Gupta and Aniruddha Kembhavi Affiliation: - PRIOR @ Allen Institute for AI. The paper introduces VISPROG, a neuro-symbolic approach to solving complex visual tasks based on natural language instructions. The system generates python-like modular programs that are executed to produce the solution and a comprehensive rationale. The approach avoids the need for task-specific training and instead uses the in-context learning ability of large language models. The paper demonstrates the flexibility of VISPROG on four diverse tasks, including image editing and factual knowledge object tagging, and shows its potential to expand AI systems to perform complex tasks.

CVPR 2023 - OmniObject3D: Large-Vocabulary 3D Object Dataset for

In this episode we discuss OmniObject3D: Large-Vocabulary 3D Object Dataset for by Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, Dahua Lin, Ziwei Liu. The paper proposes OmniObject3D, a large vocabulary 3D object dataset containing 6,000 high-quality real-scanned objects in 190 daily categories with rich annotations. The dataset aims to facilitate the development of 3D perception, reconstruction, and generation in the real world and is evaluated on four benchmarks: robust 3D perception, novel-view synthesis, neural surface reconstruction, and 3D object generation. The extensive studies on these benchmarks reveal new observations, challenges, and opportunities for future research in realistic 3D vision.

CVPR 2023 - What Can Human Sketches Do for Object Detection?

In this episode we discuss What Can Human Sketches Do for Object Detection? by Authors: - Pinaki Nath Chowdhury - Ayan Kumar Bhunia - Aneeshan Sain - Subhadeep Koley - Tao Xiang - Yi-Zhe Song Affiliation: SketchX, CVSSP, University of Surrey, United Kingdom. The paper proposes a new object detection framework that utilizes sketches to detect objects. It is the first attempt to cultivate the expressiveness of sketches for the task of object detection, with instance-aware and part-aware detection capabilities. The model is designed to work without knowing the category of objects beforehand and without requiring bounding boxes or class labels. The framework combines an existing sketch-based image retrieval (SBIR) model with the generalization ability of CLIP to build highly generalizable sketch and photo encoders that can be adapted for object detection. The proposed framework outperforms both supervised and weakly-supervised object detectors on zero-shot setups in standard object detection datasets like PASCAL-VOC and MS-COCO.

CVPR 2023 - Efficient Multimodal Fusion via Interactive Prompting

In this episode we discuss Efficient Multimodal Fusion via Interactive Prompting
by Authors:
- Yaowei Li
- Ruijie Quan
- Linchao Zhu
- Yi Yang
Affiliations:
- Yaowei Li: ReLER, AAII, University of Technology Sydney
- Ruijie Quan, Linchao Zhu, Yi Yang: CCAI, Zhejiang University
Contact information:
- Yaowei Li: yaowei.li@uts.edu.au
- Ruijie Quan, Linchao Zhu, Yi Yang: {quanruijie, zhulinchao, yangyics}@zju.edu.cn. The paper proposes an efficient and flexible multimodal fusion method, called PMF, for fusing unimodally pre-trained transformers. The proposed method disentangles vanilla prompts into three types to learn different optimizing objectives for multimodal learning. The method adds prompt vectors only on the deep layers of the unimodal transformers, significantly reducing the training memory usage. Experimental results show that the proposed method achieves comparable performance to several other multimodal finetuning methods with less than 3% trainable parameters and up to 66% saving of training memory usage.

CVPR 2023 - Query-Dependent Video Representation

In this episode we discuss Query-Dependent Video Representation
by Authors: WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo.
Affiliation:
- WonJun Moon, Sangeek Hyun, and Jae-Pil Heo: Sungkyunkwan University.
- SangUk Park and Dongchan Park: Pyler.. The paper presents Query-Dependent DETR (QD-DETR), a detection transformer that is tailored for video moment retrieval and highlight detection (MR/HD). The authors identify a key issue with existing transformer-based models, which is their failure to fully exploit the information of a given query. To address this issue, QD-DETR introduces cross-attention layers to explicitly inject query context into video representation and trains the model on negative video-query pairs to encourage precise accordance between query-video pairs. QD-DETR outperforms state-of-the-art methods on several datasets.

CVPR 2023 - Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings

In this episode we discuss Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings
by Authors:
- Ayan Kumar Bhunia
- Subhadeep Koley
- Amandeep Kumar
- Aneeshan Sain
- Pinaki Nath Chowdhury
- Tao Xiang
- Yi-Zhe Song
Affiliations:
- SketchX, CVSSP, University of Surrey, United Kingdom
- iFlyTek-Surrey Joint Research Centre on Artificial Intelligence. The paper discusses the saliency of human sketches and proposes a method for using sketches as weak labels to detect salient objects in images. The method uses a photo-to-sketch generation model with a 2D attention mechanism to generate sequential sketch coordinates corresponding to a given visual photo. Attention maps accumulated across the time steps give rise to salient regions, and experiments show that the sketch-based saliency detection model performs competitively compared to the state-of-the-art.

CVPR 2023 - Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning

In this episode we discuss Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning
by Authors:
- Ajinkya Tejankar
- Maziar Sanjabi
- Qifan Wang
- Sinong Wang
- Hamed Firooz
- Hamed Pirsiavash
- Liang Tan
Affiliations:
- University of California, Davis (Ajinkya Tejankar, Hamed Pirsiavash)
- Meta AI (Maziar Sanjabi, Qifan Wang, Sinong Wang, Hamed Firooz, Liang Tan). The paper discusses a vulnerability of self-supervised learning to backdoor attacks through patch-based data poisoning. To defend against such attacks, the paper proposes a three-step defense pipeline involving training a model on the poisoned data, using a defense algorithm called PatchSearch to remove poisoned samples from the training set, and finally training a model on the cleaned-up training set. The results show that PatchSearch is an effective defense, outperforming baselines and state-of-the-art defense approaches. The code is available online.