AI Breakdown

arxiv - AVIS: Autonomous Visual Information Seeking

In this episode we discuss AVIS: Autonomous Visual Information Seeking
by The author's name cannot be determined from the snippet provided as it only includes the title of the paper.. The paper introduces AVIS, an autonomous visual question-answering framework that utilizes a Large Language Model to strategically utilize external tools and provide answers to visual questions that require external knowledge. The framework includes a planner, reasoner, and working memory component that work together to analyze and extract key information from external tools. The collected user behavior serves as a guide for the system to enhance its decision-making capacity. AVIS achieves state-of-the-art results on knowledge-intensive visual question answering benchmarks.

CVPR 2023, award candidate - Data-driven Feature Tracking for Event Cameras

In this episode we discuss Data-driven Feature Tracking for Event Cameras
by Nico Messikommer, Carter Fang, Mathias Gehrig, Davide Scaramuzza. The paper details a data-driven feature tracking method for event cameras that improves upon existing techniques that require parameter tuning and struggle with noise and generalization. The proposed method utilizes a frame attention module to share information across feature tracks, resulting in improved performance with a 120% increase in relative feature age and lower latency compared to existing approaches. Multimedia materials and code are available to supplement the paper.

CVPR 2023 - SIEDOB: Semantic Image Editing by Disentangling Object and Background

In this episode we discuss SIEDOB: Semantic Image Editing by Disentangling Object and Background
by Wuyang Luo, Su Yang, Xinjian Zhang, Weishan Zhang. The paper presents a new method for semantic image editing called Semantic Image Editing by Disentangling Object and Background (SIEDOB). This method separates objects and backgrounds into separate subnetworks for more efficient processing by first decomposing the input into background regions and instance-level objects, which are then fed into dedicated generators. The paper also introduces innovative designs to produce high-quality edited images and outperforms existing methods in synthesizing realistic and diverse objects and texture-consistent backgrounds.

CVPR 2023 - GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts

In this episode we discuss GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts
by Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, He Wang. The paper proposes a method called Generalizable and Actionable Parts (GAParts) for learning cross-category domain-generalizable object perception and manipulation. This involves defining 9 GAPart classes to construct a part-centric interactive dataset named GAPartNet with rich part-level annotations for over 8,000 part instances on 1,166 objects. The authors investigate three cross-category tasks and propose a robust 3D segmentation method that integrates adversarial learning techniques to address domain gaps between seen and unseen object categories and manipulation heuristics that generalize well to unseen object categories in both the simulator and the real world.

CVPR 2023 - Improving Image Recognition by Retrieving from Web-Scale Image-Text Data

In this episode we discuss Improving Image Recognition by Retrieving from Web-Scale Image-Text Data
by Ahmet Iscen
Alireza Fathi
Cordelia Schmid. The paper proposes a new attention-based memory module for retrieval augmented models that enhances recognition capabilities by retrieving similar examples for visual input from an external memory set. The method removes irrelevant retrieved examples and retains useful ones. The study demonstrates the benefits of using a massive-scale memory dataset of 1B image-text pairs and achieves state-of-the-art accuracies in three classification tasks. The paper also discusses challenges associated with scaling large transformer models and suggests using world knowledge to create a massive-scale index/memory for use with a small model for the given inference task.

CVPR 2023 - Modality-invariant Visual Odometry for Embodied Vision

In this episode we discuss Modality-invariant Visual Odometry for Embodied Vision
by Marius Memmel1*, Roman Bachmann2, and Amir Zamir2 are the authors of the paper titled "Modality-invariant Visual Odometry for Embodied Vision".. This paper proposes a modality-invariant approach to visual odometry (VO) for embodied vision, which is important for effective localization in noisy environments. The proposed Transformer-based approach can handle diverse or changing sensor suites of navigation agents and outperforms previous methods. It can also be extended to learn from multiple arbitrary input modalities, such as surface normals, point clouds, or internal measurements for flexible and learned VO models.

arXiv preprint - T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for

In this episode we discuss T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for
by Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. The paper proposes a method to enhance the control and editing abilities of large-scale text-to-image (T2I) models. These models can generate complex structures and meaningful semantics, but current methods rely heavily on text prompts and lack flexible user control capability. The proposed method, called T2I-Adapter, learns to align internal knowledge in T2I models with external control signals, achieving rich control and editing effects in generation results. The T2I-Adapter is lightweight, flexible, composable, and generalizable, providing more accurate controllable guidance to existing T2I models without affecting their original generation ability.

ArXiv Preprint - Birth of a Transformer: A Memory Viewpoint

In this episode we discuss Birth of a Transformer: A Memory Viewpoint
by The authors of the paper are Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Hervé Jegou and Léon Bottou.. The paper titled "Birth of a Transformer: A Memory Viewpoint" delves into the internal workings of large language models based on transformers. The authors introduce a synthetic dataset to study how transformers balance global knowledge and context-specific knowledge. The study finds that two-layer transformers use an induction head mechanism to predict context-specific bigrams, and the authors introduce a natural model for individual weight matrices as associative memories. Through their empirical study, the authors provide theoretical insights on how gradients enable the learning of weight matrices during training and analyze the role of data-distributional properties.

CVPR 2023 - PivoTAL: Prior-Driven Supervision for Weakly-Supervised Temporal Action Localization

In this episode, we discuss "PivoTAL: Prior-Driven Supervision for Weakly-Supervised Temporal Action Localization" by Mamshad Nayeem Rizve, Gaurav Mittal, Ye Yu, Matthew Hall, Sandra Sajeev, Mubarak Shah, Mei Chen from Microsoft and University of Central Florida. It introduces a novel approach to address the problem of localizing actions in untrimmed videos with only video-level supervision. Existing methods rely on classifying individual frames and post-processing to aggregate predictions, but this often leads to incomplete localization. PivoTAL takes a different approach by directly learning to localize action snippets, leveraging spatio-temporal regularities in videos through action-specific scene prior, action snippet generation prior, and a learnable Gaussian prior. The proposed method, evaluated on benchmark datasets, demonstrates a significant improvement (at least 3% avg mAP) compared to existing methods. The results highlight the effectiveness of the prior-driven supervision approach in weakly-supervised temporal action localization.

CVPR 2023 - Polynomial Implicit Neural Representations For Large Diverse Datasets

In this episode we discuss Polynomial Implicit Neural Representations For Large Diverse Datasets
by Rajhans Singh, Ankita Shukla, Pavan Turaga. The paper proposes a new approach to implicit neural representations (INR) which are popularly used for signal and image representation in various tasks. The current INR architectures rely on sinusoidal positional encoding, limiting their representational power. The proposed Poly-INR model eliminates the need for positional encodings by representing an image with a polynomial function and using element-wise multiplications between features and affine-transformed coordinate locations. The model performs comparably to state-of-the-art generative models without convolution, normalization, or self-attention layers and with fewer trainable parameters.