Arxiv paper - Describe Anything: Detailed Localized Image and Video Captioning

In this episode, we discuss Describe Anything: Detailed Localized Image and Video Captioning by Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, Yin Cui. The paper presents the Describe Anything Model (DAM) for detailed localized captioning that integrates local detail and global context using a focal prompt and localized vision backbone. It introduces a semi-supervised data pipeline (DLC-SDP) to address limited training data by leveraging segmentation datasets and unlabeled images. Additionally, the authors propose DLC-Bench, a new benchmark for evaluating detailed localized captioning, where DAM achieves state-of-the-art results across multiple tasks.