Fusion4D: Real-time Performance Capture of Challenging Scenes

  • Mingsong Dou ,
  • Sameh Khamis ,
  • Yury Degtyarev ,
  • Philip Davidson ,
  • Sean Ryan Fanello ,
  • Adarsh Kowdle ,
  • Sergio Orts Escolano ,
  • Christoph Rhemann ,
  • David Kim ,
  • Jonathan Taylor ,
  • Pushmeet Kohli ,
  • Vladimir Tankovich ,
  • Shahram Izadi

ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH 2016 | , Vol 35

Publication

We contribute a new pipeline for live multi-view performance capture, generating temporally coherent high-quality reconstructions in real-time. Our algorithm supports both incremental reconstruction, improving the surface estimation over time, as well as parameterizing the nonrigid scene motion. Our approach is highly robust to both large frame-to-frame motion and topology changes, allowing us to reconstruct extremely challenging scenes. We demonstrate advantages over related real-time techniques that either deform an online generated template or continually fuse depth data nonrigidly into a single reference model. Finally, we show geometric reconstruction results on par with offline methods which require orders of magnitude more processing time and many more RGBD cameras.

Fusion4D: Real-time Performance Capture of Challenging Scenes

We contribute a new pipeline for live multi-view performance capture, generating temporally coherent high-quality reconstructions in real-time. Our algorithm supports both incremental reconstruction, improving the surface estimation over time, as well as parameterizing the nonrigid scene motion. Our approach is highly robust to both large frame-to-frame motion and topology changes, allowing us to reconstruct extremely challenging scenes. We demonstrate advantages over related real-time techniques that either deform an online generated template or continually fuse depth data non-rigidly into a single reference model. Finally, we show geometric reconstruction results on par with offline methods which require orders of magnitude more processing time and many more RGBD cameras.