CVPR 2023 - Aligning Step-by-Step Instructional Diagrams to Video Demonstrations

In this episode we discuss Aligning Step-by-Step Instructional Diagrams to Video Demonstrations by Jiahao Zhang, Anoop Cherian, Yanbin Liu, Yizhak Ben-Shabat, Cristian Rodriguez, Stephen Gould. The paper presents a novel approach to align instruction steps depicted as assembly diagrams with segments from in-the-wild videos that depict the actions. The authors propose a supervised contrastive learning method that is guided by a set of novel losses to align videos with the subtle details of assembly diagrams. They introduce a new dataset, IAW, consisting of 183 hours of videos and nearly 8,300 illustrations with ground truth alignments to evaluate the effectiveness of their method. The experimental results demonstrate superior performance compared to alternatives on two defined tasks of nearest neighbor retrieval and alignment of instruction steps and video segments.