CVPR 2023 - Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos

In this episode we discuss Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos by Liao Wang, Qiang Hu, Qihan He, Ziyu Wang, Jingyi Yu, Tinne Tuytelaars, Lan Xu, Minye Wu. The paper introduces a new technique called Residual Radiance Field (ReRF), a compact neural representation for achieving real-time free-view rendering on long-duration dynamic scenes. ReRF explicitly models residual information between adjacent timestamps in the spatial-temporal feature space using a global coordinate-based tiny MLP as the feature decoder. The paper also presents a special free-view video (FVV) codec based on ReRF that achieves three orders of magnitude compression rate and provides a companion ReRF player to support online streaming of long-duration FVVs of dynamic scenes. Extensive experiments demonstrate the effectiveness of ReRF for compactly representing dynamic radiance fields, enabling an unprecedented free-viewpoint viewing experience in speed and quality.