CVPR 2023, highlight paper - Neural Volumetric Memory for Visual Locomotion Control

In this episode we discuss Neural Volumetric Memory for Visual Locomotion Control by Ruihan Yang, Ge Yang, Xiaolong Wang. The paper discusses the use of legged robots for autonomous locomotion on challenging terrains using a forward-facing depth camera. Due to the partial observability of the terrain, the robot has to rely on past observations to infer the terrain currently beneath it. The authors propose a new memory architecture called Neural Volumetric Memory (NVM), which explicitly models the 3D geometry of the scene and aggregates feature volumes from multiple camera views. The approach was tested on a physical robot and showed superior performance compared to other methods, with representations stored in the neural volumetric memory capturing sufficient geometric information to reconstruct the scene.