AI Breakdown

CVPR 2023 - Modality-invariant Visual Odometry for Embodied Vision

In this episode we discuss Modality-invariant Visual Odometry for Embodied Vision
by Marius Memmel1*, Roman Bachmann2, and Amir Zamir2 are the authors of the paper titled "Modality-invariant Visual Odometry for Embodied Vision".. This paper proposes a modality-invariant approach to visual odometry (VO) for embodied vision, which is important for effective localization in noisy environments. The proposed Transformer-based approach can handle diverse or changing sensor suites of navigation agents and outperforms previous methods. It can also be extended to learn from multiple arbitrary input modalities, such as surface normals, point clouds, or internal measurements for flexible and learned VO models.

arXiv preprint - T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for

In this episode we discuss T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for
by Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. The paper proposes a method to enhance the control and editing abilities of large-scale text-to-image (T2I) models. These models can generate complex structures and meaningful semantics, but current methods rely heavily on text prompts and lack flexible user control capability. The proposed method, called T2I-Adapter, learns to align internal knowledge in T2I models with external control signals, achieving rich control and editing effects in generation results. The T2I-Adapter is lightweight, flexible, composable, and generalizable, providing more accurate controllable guidance to existing T2I models without affecting their original generation ability.

ArXiv Preprint - Birth of a Transformer: A Memory Viewpoint

In this episode we discuss Birth of a Transformer: A Memory Viewpoint
by The authors of the paper are Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Hervé Jegou and Léon Bottou.. The paper titled "Birth of a Transformer: A Memory Viewpoint" delves into the internal workings of large language models based on transformers. The authors introduce a synthetic dataset to study how transformers balance global knowledge and context-specific knowledge. The study finds that two-layer transformers use an induction head mechanism to predict context-specific bigrams, and the authors introduce a natural model for individual weight matrices as associative memories. Through their empirical study, the authors provide theoretical insights on how gradients enable the learning of weight matrices during training and analyze the role of data-distributional properties.

CVPR 2023 - PivoTAL: Prior-Driven Supervision for Weakly-Supervised Temporal Action Localization

In this episode, we discuss "PivoTAL: Prior-Driven Supervision for Weakly-Supervised Temporal Action Localization" by Mamshad Nayeem Rizve, Gaurav Mittal, Ye Yu, Matthew Hall, Sandra Sajeev, Mubarak Shah, Mei Chen from Microsoft and University of Central Florida. It introduces a novel approach to address the problem of localizing actions in untrimmed videos with only video-level supervision. Existing methods rely on classifying individual frames and post-processing to aggregate predictions, but this often leads to incomplete localization. PivoTAL takes a different approach by directly learning to localize action snippets, leveraging spatio-temporal regularities in videos through action-specific scene prior, action snippet generation prior, and a learnable Gaussian prior. The proposed method, evaluated on benchmark datasets, demonstrates a significant improvement (at least 3% avg mAP) compared to existing methods. The results highlight the effectiveness of the prior-driven supervision approach in weakly-supervised temporal action localization.

CVPR 2023 - Polynomial Implicit Neural Representations For Large Diverse Datasets

In this episode we discuss Polynomial Implicit Neural Representations For Large Diverse Datasets
by Rajhans Singh, Ankita Shukla, Pavan Turaga. The paper proposes a new approach to implicit neural representations (INR) which are popularly used for signal and image representation in various tasks. The current INR architectures rely on sinusoidal positional encoding, limiting their representational power. The proposed Poly-INR model eliminates the need for positional encodings by representing an image with a polynomial function and using element-wise multiplications between features and affine-transformed coordinate locations. The model performs comparably to state-of-the-art generative models without convolution, normalization, or self-attention layers and with fewer trainable parameters.

CVPR 2023 - Iterative Next Boundary Detection for Instance Segmentation of Tree Rings in Microscopy Images of Shrub Cross Sections

In this episode we discuss Iterative Next Boundary Detection for Instance Segmentation of Tree Rings in Microscopy Images of Shrub Cross Sections
by Alexander Gillert, Giulia Resente, Alba Anadon-Rosell, Martin Wilmking, Uwe Freiherr von Lukas. The paper proposes a new iterative method called Iterative Next Boundary Detection (INBD) for detecting tree rings in microscopy images of shrub cross sections. This is a difficult task due to the concentric circular ring shape of the objects and the high precision requirements. INBD models the natural growth direction, starting from the center of the shrub cross section and detecting the next ring boundary in each iteration step, and outperforms existing methods in experiments. The dataset and source code are also made available.

CVPR 2023 - Towards Unified Scene Text Spotting based on Sequence Generation

In this episode we discuss Towards Unified Scene Text Spotting based on Sequence Generation
by Taeho Kil, Seonghyeon Kim, Sukmin Seo, Yoonsik Kim, Daehee Kim. The proposed paper presents a UNIfied scene Text Spotter, called UNITS, to overcome the limitations of auto-regressive models used for end-to-end text spotting. UNITS unifies various detection formats, allowing it to detect text in arbitrary shapes, and applies starting-point prompting to extract more texts beyond the number of instances it was trained on. Experimental results show that UNITS achieves competitive performance compared to state-of-the-art methods and can extract a larger number of texts than it was trained on. Code for the method is provided on GitHub.

CVPR 2023 - Learning Neural Duplex Radiance Fields for Real-Time View Synthesis

In this episode we discuss Learning Neural Duplex Radiance Fields for Real-Time View Synthesis
by Ziyu Wan, Christian Richardt, Aljaž Božič, Chao Li, Vijay Rengarajan, Seonghyeon Nam, Xiaoyu Xiang, Tuotuo Li, Bo Zhu, Rakesh Ranjan, Jing Liao. The paper proposes a novel approach to rendering photorealistic images using Neural Radiance Fields (NeRFs) in a more efficient manner. NeRFs require hundreds of deep MLP evaluations for each pixel, which is prohibitively expensive for real-time rendering. The proposed approach overcomes this by distilling and baking NeRFs into highly efficient mesh-based neural representations that are compatible with the massively parallel graphics rendering pipeline. The approach uses screen-space convolutions instead of MLPs to exploit local geometric relationships between nearby pixels and is further boosted by a multi-view distillation optimization strategy. Extensive experiments demonstrate the effectiveness and superiority of the approach on a range of standard datasets.

CVPR 2023 - Context-Based Trit-Plane Coding for Progressive Image Compression

In this episode we discuss Context-Based Trit-Plane Coding for Progressive Image Compression
by Seungmin Jeon, Kwang Pyo Choi, Youngo Park, Chang-Su Kim. The paper proposes the context-based trit-plane coding (CTC) algorithm for progressive image compression. CTC enables compact encoding of trit-planes by developing a context-based rate reduction module to estimate trit probabilities accurately. The context-based distortion reduction module refines partial latent tensors from the trit-planes to improve image quality. The proposed CTC algorithm outperforms the baseline trit-plane codec significantly and increases time complexity marginally.

CVPR 2023 - Interactive Cartoonization with Controllable Perceptual Factors

In this episode we discuss Interactive Cartoonization with Controllable Perceptual Factors
by Namhyuk Ahn, Patrick Kwon, Jihye Back, Kibeom Hong, Seungkwon Kim. The paper proposes a new method for cartoonization, which involves rendering natural photos into cartoon styles with editing features of texture and color. The proposed method uses a model architecture with separate decoders for texture and color, and introduces a texture controller to generate diverse cartoon textures. Additionally, an HSV color augmentation is used to induce the networks to generate diverse and controllable color translation, resulting in profound quality improvement over baselines. This is the first deep approach that allows control of the cartoonization at inference.