Exploiting Repetitions for Image-Based Rendering of Facades

Street-level imagery is now abundant but does not have sufficient capture density to be usable for Image-Based Rendering (IBR) of facades. We present a method that exploits repetitive elements in facades - such as windows - to perform data augmentation, in turn improving camera calibration, reconstructed geometry and overall rendering quality for IBR. The main intuition behind our approach is that a few views of several instances of an element provide similar information to many views of a single instance of that element. We first select similar instances of an element from 3-4 views of a facade and transform them into a common coordinate system, creating a "platonic" element. We use this common space to refine the camera calibration of each view of each instance and to reconstruct a 3D mesh of the element with multi-view stereo, that we regularize to obtain a piecewise-planar mesh aligned with dominant image contours. Observing the same element under multiple views also allows us to identify reflective areas - such as glass panels - which we use at rendering time to generate plausible reflections using an environment map. Our detailed 3D mesh, augmented set of views, and reflection mask enable image-based rendering of much higher quality than results obtained using the input images directly.

@Article{RBDD18,
  author       = "Rodriguez, Simon and Bousseau, Adrien and Durand, Fr\'edo and Drettakis, George",
  title        = "Exploiting Repetitions for Image-Based Rendering of Facades",
  journal      = "Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering)",
  number       = "4",
  volume       = "37",
  year         = "2018",
  keywords     = "Image-based rendering, reconstruction, repetitions",
  url          = "http://www-sop.inria.fr/reves/Basilic/2018/RBDD18"
}