GraphDeco

ModalNeRF: Neural Modal Analysis and Synthesis for Free-Viewpoint Navigation in Dynamically Vibrating Scenes
Presentation | Team members | Collaborations | Publications | Job offers | Contact

 

ModalNeRF: Neural Modal Analysis and Synthesis for Free-Viewpoint Navigation in Dynamically Vibrating Scenes

Automne Petitjean, Yohan Poirier-Ginter, Ayush Tewari, Guillaume Cordonnier, George Drettakis
Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering), Volume 42, Number 4 - July 2023
Download the publication : modal_nerf_submission.pdf [25.3Mo]  
Recent advances in Neural Radiance Fields enable the capture of scenes with motion. However, editing the motion is hard; no existing method allows editing beyond the space of motion existing in the original video, nor editing based on physics. We present the first approach that allows physically-based editing of motion in a scene captured with a single hand-held video camera, containing vibrating or periodic motion. We first introduce a Lagrangian representation, representing motion as the displacement of particles, which is learned while training a radiance field. We use these particles to create a continuous representation of motion over the sequence, which is then used to perform a modal analysis of the motion thanks to a Fourier transform on the particle displacement over time. The resulting extracted modes allow motion synthesis, and easy editing of the motion, while inheriting the ability for free-viewpoint synthesis in the captured 3D scene from the radiance field. We demonstrate our new method on synthetic and real captured scenes.

Images and movies

 

See also



Acknowledgements and Funding

This research was funded by the ERC Advanced Grant FUNGRAPH (No.788065, http://fungraph.inria.fr). The authors are grateful to Adobe for generous donations and the OPAL infrastructure from Université Côte d'Azur for providing resources and support. The authors would also like to thank the anonymous reviewers for their valuable feedback.

BibTex references

@Article{PPTCD23,
  author       = "Petitjean, Automne and Poirier-Ginter, Yohan and Tewari, Ayush and Cordonnier, Guillaume and Drettakis, George",
  title        = "ModalNeRF: Neural Modal Analysis and Synthesis for Free-Viewpoint Navigation in Dynamically Vibrating Scenes",
  journal      = "Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering)",
  number       = "4",
  volume       = "42",
  month        = "July",
  year         = "2023",
  url          = "http://www-sop.inria.fr/reves/Basilic/2023/PPTCD23"
}

Other publications in the database

» Yohan Poirier-Ginter
» Guillaume Cordonnier
» George Drettakis