AI Breakdown

arxiv preprint - Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

In this episode we discuss Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting
by Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, Michael Bendersky. The paper introduces Pairwise Ranking Prompting (PRP) as a technique to improve document ranking using Large Language Models (LLMs). Despite their impressive performance on natural language tasks, LLMs have struggled to outperform fine-tuned rankers on benchmark datasets. However, PRP achieves state-of-the-art ranking results with moderate-sized LLMs and offers benefits such as supporting both generation and scoring LLM APIs and being insensitive to input ordering.

arxiv preprint - LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

In this episode we discuss LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding by Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, Tong Sun. The paper introduces LLaVAR, an enhanced visual instruction tuning method for text-rich image understanding. The method addresses the limitation of existing pipelines in comprehending textual details within images by incorporating text-rich images and OCR tools. Experimental results show that LLaVAR improves the performance of the LLaVA model on text-based visual question answering datasets, achieving up to 20% accuracy improvement. The model also exhibits promising interaction skills with humans based on real-world online content that combines text and images.

arxiv preprint - Generate Anything Anywhere in Any Scene

In this episode we discuss Generate Anything Anywhere in Any Scene by Yuheng Li, Haotian Liu, Yangming Wen, Yong Jae Lee. The paper proposes a data augmentation training strategy for personalized object generation in text-to-image diffusion models. They also introduce a plug-and-play adapter layers approach to control the location and size of the generated personalized objects. Additionally, a regionally-guided sampling technique is introduced to maintain image quality during inference. The proposed model shows promising results in terms of fidelity for personalized objects, making it suitable for applications in art, entertainment, and advertising design.

arxiv preprint - BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion

In this episode we discuss BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion by Michael J. Black, Priyanka Patel, Joachim Tesch, Jinlong Yang. This paper presents BEDLAM, a large-scale synthetic dataset for 3D human pose and shape estimation. Unlike previous datasets, BEDLAM is realistic and diverse, featuring monocular RGB videos with ground-truth 3D bodies, including varied body shapes, motions, skin tones, hair, and realistic clothing. The dataset also includes realistic scenes with varying lighting and camera motions. Trained regressors using BEDLAM achieve state-of-the-art accuracy on real-image benchmarks, demonstrating the importance of good synthetic training data for accurate estimation. The BEDLAM dataset and resources are provided for research purposes, along with detailed information about the data generation process.

ICASSP 2023 - Aura: Privacy-preserving Augmentation to Improve Test Set Diversity in Speech Enhancement

In this episode we discuss Aura: Privacy-preserving Augmentation to Improve Test Set Diversity in Speech Enhancement
by Xavier Gitiaux, Aditya Khant, Ebrahim Beyrami, Chandan Reddy, Jayant Gupchup, Ross Cutler. The paper presents Aura, a privacy-preserving method to enhance test set diversity in speech enhancement models. Usually, these models are trained on public data, which leads to performance issues when applied to customer data due to privacy constraints. Aura addresses this by creating diverse test sets using pre-trained feature extractors and clustering techniques, resulting in improved model rankings and increased test set diversity. The paper introduces the "ears-off" methodology, a generic approach to measure test set diversity, and demonstrates Aura's effectiveness in speech enhancement tasks.

arxiv preprint - Segment Anything Meets Point Tracking

In this episode we discuss Segment Anything Meets Point Tracking
by Frano Rajič, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, Fisher Yu. The paper introduces a method called SAM-PT for tracking and segmenting objects in dynamic videos. SAM-PT uses point selection and propagation techniques to create masks for video object segmentation. The method demonstrates its effectiveness across various benchmarks and includes enhancements such as point initialization, tracking positive and negative points, multiple mask decoding passes, and a point re-initialization strategy for improved accuracy.

ICASSP 2023 - A Speech Representation Anonymization Framework via Selective Noise Perturbation

In this episode we discuss A Speech Representation Anonymization Framework via Selective Noise Perturbation
by Minh Tran, Mohammad Soleymani. The paper presents a framework for anonymizing speech data by adding selective noise perturbation to speech representations. It includes a Privacy-risk Saliency Estimator (PSE) that predicts the importance of different representation positions. The approach achieves competitive utility and privacy without re-training any component.

arxiv preprint - One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

In this episode we discuss One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
by Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, Zhiqiang Shen. The paper introduces GLoRA, an approach for fine-tuning tasks that is parameter-efficient. GLoRA utilizes a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, resulting in improved flexibility and capability across different tasks and datasets. Experimental results demonstrate that GLoRA outperforms previous methods in various benchmarks, achieving superior accuracy with fewer parameters and computations.

arxiv preprint - Tracking Everything Everywhere All at Once

In this episode we discuss Tracking Everything Everywhere All at Once
by Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, Noah Snavely. The paper introduces a new method called OmniMotion for estimating dense and long-range motion from a video sequence. Traditional approaches like sparse feature tracking and dense optical flow are insufficient in capturing the complete motion. OmniMotion uses a quasi-3D canonical volume and bijections between local and canonical space to accurately estimate motion for every pixel in a video. The method outperforms previous state-of-the-art techniques in terms of accuracy and consistency, as demonstrated through evaluations on benchmark datasets and real-world footage.

arxiv preprint - ViNT: A Foundation Model for Visual Navigation

In this episode we discuss ViNT: A Foundation Model for Visual Navigation
by Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, Sergey Levine. The paper presents ViNT, a pre-trained foundation model for visual navigation in robotics. It utilizes a Transformer-based architecture and is trained with a goal-reaching objective. ViNT demonstrates positive transfer on different navigation datasets from various robotic platforms and can handle kilometer-scale navigation problems. It can also be adapted to new task specifications using prompt-tuning and is proposed as a promising solution for mobile robotics.