CVPR 2023, highlight paper - Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning

In this episode we discuss Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning by Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, Alan Yuille. The paper introduces a virtual benchmark called Super-CLEVR to isolate different factors of variation that affect the performance of Visual Question Answering (VQA) models on out-of-distribution data and domain generalization. The benchmark considers four factors, including visual complexity, question redundancy, concept distribution, and concept compositionality, to enable testing of VQA methods in situations where test data differs from training data along each of these axes. The authors study four existing methods and propose a new probabilistic NSVQA (P-NSVQA) method, which outperforms others on three of the four domain shift factors, indicating that disentangling reasoning and perception, combined with probabilistic uncertainty, forms a strong VQA model that is more robust to domain shifts.