AI Breakdown

arxiv preprint - Segment Anything Meets Point Tracking

In this episode we discuss Segment Anything Meets Point Tracking
by Frano Rajič, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, Fisher Yu. The paper introduces a method called SAM-PT for tracking and segmenting objects in dynamic videos. SAM-PT uses point selection and propagation techniques to create masks for video object segmentation. The method demonstrates its effectiveness across various benchmarks and includes enhancements such as point initialization, tracking positive and negative points, multiple mask decoding passes, and a point re-initialization strategy for improved accuracy.

ICASSP 2023 - A Speech Representation Anonymization Framework via Selective Noise Perturbation

In this episode we discuss A Speech Representation Anonymization Framework via Selective Noise Perturbation
by Minh Tran, Mohammad Soleymani. The paper presents a framework for anonymizing speech data by adding selective noise perturbation to speech representations. It includes a Privacy-risk Saliency Estimator (PSE) that predicts the importance of different representation positions. The approach achieves competitive utility and privacy without re-training any component.

arxiv preprint - One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

In this episode we discuss One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
by Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, Zhiqiang Shen. The paper introduces GLoRA, an approach for fine-tuning tasks that is parameter-efficient. GLoRA utilizes a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, resulting in improved flexibility and capability across different tasks and datasets. Experimental results demonstrate that GLoRA outperforms previous methods in various benchmarks, achieving superior accuracy with fewer parameters and computations.

arxiv preprint - Tracking Everything Everywhere All at Once

In this episode we discuss Tracking Everything Everywhere All at Once
by Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, Noah Snavely. The paper introduces a new method called OmniMotion for estimating dense and long-range motion from a video sequence. Traditional approaches like sparse feature tracking and dense optical flow are insufficient in capturing the complete motion. OmniMotion uses a quasi-3D canonical volume and bijections between local and canonical space to accurately estimate motion for every pixel in a video. The method outperforms previous state-of-the-art techniques in terms of accuracy and consistency, as demonstrated through evaluations on benchmark datasets and real-world footage.

arxiv preprint - ViNT: A Foundation Model for Visual Navigation

In this episode we discuss ViNT: A Foundation Model for Visual Navigation
by Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, Sergey Levine. The paper presents ViNT, a pre-trained foundation model for visual navigation in robotics. It utilizes a Transformer-based architecture and is trained with a goal-reaching objective. ViNT demonstrates positive transfer on different navigation datasets from various robotic platforms and can handle kilometer-scale navigation problems. It can also be adapted to new task specifications using prompt-tuning and is proposed as a promising solution for mobile robotics.

arxiv preprint - Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation

In this episode we discuss Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation
by Shuai Yang, Yifan Zhou, Ziwei Liu, Chen Change Loy. The paper introduces a novel framework for adapting image models to videos through zero-shot text-guided video-to-video translation. The framework consists of two parts: key frame translation and full video translation. The key frame translation generates key frames with hierarchical cross-frame constraints to ensure coherence, which are then propagated to other frames using temporal-aware patch matching and frame blending. The framework achieves global style and local texture temporal consistency without the need for re-training or optimization and performs better than existing methods in producing high-quality and coherent videos.

arxiv preprint - RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models

In this episode we discuss RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models
by Xingchen Zhou, Ying He, F. Richard Yu, Jianqiang Li, You Li. The paper proposes a framework called RePaint-NeRF for editing the content in Neural Radiance Fields (NeRF). Traditional NeRF methods struggle with content editing, so the framework leverages diffusion models to guide changes in the designated 3D content. It effectively allows for editing appearance and shape changes of 3D objects in NeRF, improving editability, diversity, and application range.

arxiv preprint - ZipIt! Merging Models from Different Tasks without Training

In this episode we discuss ZipIt! Merging Models from Different Tasks without Training
by George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, Judy Hoffman. The paper introduces a method called "ZipIt!" that can merge two deep visual recognition models trained on separate tasks without additional training. The method incorporates a "zip" operation to handle non-shared features within each model and allows for partial merging up to a specified layer, creating a multi-head model. Experimental results demonstrate a substantial improvement of 20-60% compared to previous approaches, making it possible to merge models trained on different tasks.

CVPR 2023 - Integral Neural Networks

In this episode, we discuss, CVPR 2023 award candidate, Integral Neural Networks by Kirill Solodskikh, Azim Kurbanov, Ruslan Aydarkhanov, Irina Zhelavskaya, Yury Parfenov, Dehua Song, and Stamatios Lefkimmiatis.
The paper introduces a novel type of deep neural networks called Integral Neural Networks (INNs), which deviate from the traditional representation of network layers as N-dimensional weight tensors. Instead, INNs use a continuous layer representation along the filter and channel dimensions. The weights of INNs are represented as continuous functions defined on N-dimensional hypercubes, and the discrete transformations of inputs to the layers are replaced by continuous integration operations. During the inference stage, the continuous layers can be converted back to the traditional tensor representation using numerical integral quadratures. This representation allows for the arbitrary discretization of a network with various discretization intervals for the integral kernels. The paper demonstrates that INNs can be used for model pruning directly on edge devices, with only a small performance loss at high rates of structural pruning, without the need for fine-tuning. Experimental results on multiple tasks and neural network architectures show that INNs achieve similar performance to their discrete counterparts, while preserving approximately the same performance even with high rates of structural pruning (up to 30%) without fine-tuning. In comparison, conventional pruning methods under the same conditions result in a 65% accuracy loss. The code for implementing INNs is available at gitee.

arxiv preprint - Faith and Fate: Limits of Transformers on Compositionality

In this episode we discuss Faith and Fate: Limits of Transformers on Compositionality
by Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, Yejin Choi. The paper examines the limitations of large language models (LLMs) like Transformers on compositional tasks. It explores their performance on three specific tasks and finds that Transformers tend to solve these tasks by reducing them to linearized subgraph matching rather than demonstrating systematic problem-solving skills. The paper also discusses how Transformers' performance decreases as task complexity increases.