Repurposing a Relighting Network for Realistic Compositions of Captured Scenes

Multi-view stereo can be used to rapidly create realistic virtual content, such as textured meshes or a geometric proxy for free-viewpoint Image-Based Rendering (IBR). These solutions greatly simplify the content creation process compared to traditional methods, but it is difficult to modify the content of the scene. We propose a novel approach to create scenes by composing (parts of) multiple captured scenes. The main difficulty of such compositions is that lighting conditions in each captured scene are different; to obtain a realistic composition we need to make lighting coherent. We propose a two-pass solution, by adapting a multi-view relighting network. We first match the lighting conditions of each scene separately and then synthesize shadows between scenes in a subsequent pass. We also improve the realism of the composition by estimating the change in ambient occlusion in contact areas between parts and compensate for the color balance of the different cameras used for capture. We illustrate our method with results on multiple compositions of outdoor scenes and show its application to multi-view image composition, IBR and textured mesh creation.

@InProceedings{NPD20a,
  author       = "Nicolet, Baptiste and Philip, Julien and Drettakis, George",
  title        = "Repurposing a Relighting Network for Realistic Compositions of Captured Scenes",
  booktitle    = "Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games",
  month        = "May",
  year         = "2020",
  organization = "ACM",
  keywords     = "Image Manipulation Image-based rendering Image processing",
  url          = "http://www-sop.inria.fr/reves/Basilic/2020/NPD20a"
}