AI Breakdown

arxiv Preprint - Nougat: Neural Optical Understanding for Academic Documents

In this episode we discuss Nougat: Neural Optical Understanding for Academic Documents
by Lukas Blecher, Guillem Cucurull, Thomas Scialom, Robert Stojnic. The paper introduces Nougat, a neural optical understanding model for academic documents. Nougat utilizes a Visual Transformer model and Optical Character Recognition (OCR) to convert scientific documents into a markup language, bridging the gap between human-readable and machine-readable text. The method is versatile, capable of processing scanned papers and books, and includes a pre-trained model and code on GitHub, as well as a pipeline for creating datasets.

arxiv Preprint - Graph of Thoughts: Solving Elaborate Problems with Large Language Models

In this episode we discuss Graph of Thoughts: Solving Elaborate Problems with Large Language Models
by Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, Torsten Hoefler. The paper introduces a framework called Graph of Thoughts (GoT) that enhances the prompting capabilities of large language models (LLMs). GoT models the information generated by an LLM as an arbitrary graph, where LLM thoughts are vertices and edges represent dependencies between these thoughts. The paper demonstrates that GoT outperforms state-of-the-art methods on different tasks and can be used to spearhead new prompting schemes.

arxiv Preprint - Large Language Models as Zero-Shot Conversational Recommenders

In this episode we discuss Large Language Models as Zero-Shot Conversational Recommenders
by Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Prasad Majumder, Nathan Kallus, Julian McAuley. This paper presents empirical studies on conversational recommendation tasks using large language models (LLMs) in a zero-shot setting, without fine-tuning. The authors introduce a new dataset of recommendation-related conversations, the largest public real-world conversational recommendation dataset to date, and find that LLMs outperform existing fine-tuned conversational recommendation models on this dataset and two others. The authors also propose probing tasks to investigate the mechanisms behind LLM performance and analyze both the models' behaviors and the characteristics of the datasets.

arxiv Preprint - A Survey on Large Language Model based Autonomous Agents

In this episode we discuss A Survey on Large Language Model based Autonomous Agents
by Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, Ji-Rong Wen. The authors of this paper conducted a comprehensive survey on the topic of autonomous agents based on large language models (LLMs). They propose a unified framework for constructing LLM-based agents and provide a systematic review of previous work in this area. Additionally, they discuss the applications, evaluation strategies, and future directions for LLM-based AI agents.

arxiv Preprint - EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding

In this episode we discuss EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding
by Karttikeya Mangalam, Raiymbek Akshulakov, Jitendra Malik. The paper presents EgoSchema, a benchmark dataset and evaluation metric for assessing the long-form video language understanding capabilities of vision and language systems. The dataset consists of over 5000 multiple choice question-answer pairs based on 250 hours of real video data, and the questions require selecting the correct answer from five options based on a three-minute video clip. The authors highlight that existing video understanding datasets lack long temporal structures, and they show that state-of-the-art video and language models have limitations in long-term video understanding.

ICCV 2023 - UnLoc: A Unified Framework for Video Localization Tasks

In this episode we discuss UnLoc: A Unified Framework for Video Localization Tasks
by Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang, Weina Ge, David Ross, Cordelia Schmid. The paper introduces UnLoc, a unified framework for video localization using large-scale image-text pretrained models. UnLoc eliminates the need for action proposals, motion-based features, and representation masking by combining moment retrieval, temporal localization, and action segmentation in a single stage model. Experimental results show that UnLoc outperforms previous methods and achieves state-of-the-art results in all three localization tasks.

arxiv Preprint - Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies

In this episode we discuss Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies
by Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, William Yang Wang. The paper provides a comprehensive review of self-correction strategies for large language models (LLMs). It examines recent work on self-correction techniques, categorizing them into training-time, generation-time, and post-hoc correction methods. The authors also discuss the applications of self-correction and highlight future directions and challenges in this area.

ICLR 2023 - Rethinking the Expressive Power of GNNs via Graph Biconnectivity

In this episode we discuss Rethinking the Expressive Power of GNNs via Graph Biconnectivity
by Bohang Zhang, Shengjie Luo, Liwei Wang, Di He. This paper introduces a new approach called Generalized Distance Weisfeiler-Lehman (GD-WL) to study the expressive power of Graph Neural Networks (GNNs). The authors show that most existing GNN architectures are not expressive for certain metrics related to graph biconnectivity, except for the ESAN framework. They demonstrate that GD-WL is provably expressive for all biconnectivity metrics and outperforms previous GNN architectures in practical experiments.

ICLR 2023 - Conditional Antibody Design as 3D Equivariant Graph Translation

In this episode we discuss Conditional Antibody Design as 3D Equivariant Graph Translation
by Xiangzhe Kong, Wenbing Huang, Yang Liu. The paper introduces a method called Multi-channel Equivariant Attention Network (MEAN) for antibody design. MEAN addresses challenges faced by existing deep-learning-based methods by formulating antibody design as a conditional graph translation problem and incorporating additional components. The MEAN model utilizes a proposed attention mechanism and generates both 1D CDR sequences and 3D structures, outperforming state-of-the-art models in sequence and structure modeling, antigen-binding CDR design, and binding affinity optimization.

arxiv Preprint - ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation

In this episode we discuss ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation
by Xuefeng Hu, Ke Zhang, Lu Xia, Albert Chen, Jiajia Luo, Yuyin Sun, Ken Wang, Nan Qiao, Xiao Zeng, Min Sun, Cheng-Hao Kuo, Ram Nevatia. The paper discusses ReCLIP, a source-free domain adaptation method for large-scale pre-training vision-language models like CLIP. ReCLIP addresses the challenges of domain gaps and misalignment by learning a projection space and utilizing cross-modality self-training with pseudo labels. Experimental results show that ReCLIP reduces the average error rate of CLIP across 22 image classification benchmarks.