Publications

International Journals

A Shape-Aware Retargeting Approach to Transfer Human Motion and Appearance in Monocular Videos
Thiago L. Gomes, Renato Martins, Joao Ferreira, Rafael Azevedo, Guilherme Torres, and Erickson R. Nascimento
International Journal of Computer Vision (IJCV), 2021
arXiv / project webpage / bibtex / github code

In this paper, we propose a unifying formulation for transferring appearance and retargeting human motion from monocular videos. Our method is composed of four main components and synthesizes new videos of people in a different context where they were initially recorded. Differently from recent human neural rendering methods, our approach takes into account jointly body shape, appearance and motion constraints in the transfer.

Extending Maps with Semantic and Contextual Object Information for Robot Navigation : a Learning-Based Framework using Visual and Depth Cues
Renato Martins, Dhiego Bersan, Mario Campos, Erickson R. Nascimento
Springer Journal of Intelligent and Robotic Systems (JINT), 2020
arXiv / project webpage / bibtex / github code

This work addresses the problem of building augmented metric representations of scenes with semantic information from RGB-D images. We propose a complete framework to create an enhanced map representation of the environment with object-level information to be used in several applications such as human-robot interaction, assistive robotics, visual navigation, or in manipulation tasks. Our formulation leverages a CNN-based object detector (Yolo) with a 3D model-based segmentation technique to perform instance semantic segmentation, and to localize, identify, and track different classes of objects in the scene.

Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio
Joao P. Ferreira, Thiago M. Coutinho, Thiago L. Gomes, Jose F. Neto, Rafael Azevedo, Renato Martins, Erickson R. Nascimento
Elsevier Journal of Computers and Graphics (Computers & Graphics), 2020
arXiv / project webpage / bibtex / github code

In this project, we design a novel human motion generation method, based on graph convolutional networks (GCN), to tackle the problem of automatic dance generation from audio information. Our method proposes an adversarial learning scheme conditioned by music audios to generate natural dance motions, preserving the key movements of different music styles. The automatic generated motions for different dance styles (such as "ballet" and "salsa") are used to animate virtual human avatars.

International Conferences

Do As I Do: Transferring Human Motion and Appearance between Monocular Videos with Spatial and Temporal Constraints
Thiago L. Gomes, Renato Martins, Joao Ferreira, Erickson R. Nascimento
IEEE Winter Conference on Applications of Computer Vision (WACV), 2020
arXiv / project webpage / bibtex / github code

In this paper, we propose a unifying formulation for transferring appearance and retargeting human motion from monocular videos that regards all these aspects. Our method is composed of four main components and synthesizes new videos of people in a different context where they were initially recorded. Differently from recent appearance transferring methods, our approach takes into account body shape, appearance and motion constraints.

GEOBIT: A Geodesic-Based Binary Descriptor Invariant to Non-Rigid Deformations for RGB-D Images
Erickson R. Nascimento, Guilherme Potje, Renato Martins, Felipe Chamone, Mario F. M. Campos and Ruzena Bajcsy
IEEE International Conference on Computer Vision (ICCV), 2019
arXiv / project webpage / bibtex / github code

We introduce a novel binary RGB-D descriptor invariant to isometric deformations. Our method uses geodesic isocurves on smooth textured manifolds. It combines appearance and geometric information from RGB-D images to tackle non-rigid transformations. We used our descriptor to track multiple textured depth maps and demonstrate that it produces reliable feature descriptors even in the presence of strong non-rigid deformations and depth noise.

Semantic Map Augmentation for Robot Navigation: A Learning Approach based on Visual and Depth Data
Dhiego Bersan, Renato Martins, Mario Campos, Erickson R. Nascimento
IEEE Latin American Robotics Symposium (LARS), 2018
talk slides / bibtex / project webpage / github code

In this work, we propose an open framework for building hybrid maps, i.e., combining both environment structure (metric map) and environment semantics (objects classes) to support autonomous robot perception and navigation tasks. We detect and model objects in the scene from RGB-D images, using convolutional neural networks to extract a semantic layer of the different objects in the scene. Our final environment representation is a metric map augmented with the semantic information of the detected objects.

An Efficient Rotation and Translation Decoupled Initialization from Large Field of View Depth Images
Renato Martins, Eduardo Fernandez Moral and Patrick Rives
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017
talk slides / bibtex / github code

This paper describes a registration technique using the normal vectors of depth images. The technique is computed in a decoupled and non-iterative way, and with a large convergence domain. This formulation can be used with an initialization framework to improve the convergence of direct registration methods.


A New Metric for Evaluating Semantic Segmentation: Leveraging Global and Contour Accuracy
Eduardo Fernandez-Moral, Renato Martins, Denis Wolf and Patrick Rives
IEEE Intelligent Vehicles Symposium (IV), 2018
IEEE/RSJ International Conference on Intelligent Robots and Systems Workshop on Planning, Perception and Navigation for Intelligent Vehicles (IROS PPNIV), 2017
talk slides / bibtex

In this paper, we propose a new metric to evaluate semantic segmentation. This new metric accounts for both global and contour accuracy in a simple formulation to overcome the weaknesses of the most commonly used metrics.

Adaptive Direct RGB-D Registration and Mapping for Large Motions
Renato Martins, Eduardo Fernandez Moral and Patrick Rives
Asian Conference on Computer Vision (ACCV), 2016
poster / bibtex

This work addresses the challenging cases of large motions in direct image registration. We explore the complementary aspects of a classical direct VO and direct point-to-plane strategies, in terms of convergence, by using a modified cost function, where the geometric term prevails in the first coarse iterations, while the intensity data term dominates in the finer increments.


Increasing the Convergence Domain of RGB-D Direct Registration Methods for Vision-based Localization in Large Scale Environments
Renato Martins, Patrick Rives
IEEE Intelligent Transportation Systems Conference Workshop on Planning, Perception and Navigation for Intelligent Vehicles (ITSC PPNIV), 2016
talk slides / bibtex

In this paper, we show the outcome of a more stable and robust direct registration task in the density/sparsity of the representation (the number of keyframes) in outdoor scene mapping. This allows storing a sparser local representation whilst maintaining a topological structure at large-scale that is accurate enough to ensure the convergence of a task in the neighbourhood of the scene model.

Dense Accurate Urban Mapping from Spherical RGB-D Images
Renato Martins, Eduardo Fernandez Moral, Patrick Rives
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015
talk slides / bibtex

This work is about exploiting jointly intensity and depth information to generate more precise keyframes for visual odometry or image rendering. The main core of the paper is a depth regularization that considers both geometric and photometric image constraints (planar and superpixel segmentation).

A Compact Spherical RGBD Keyframe-based Representation
Tawsif Gokhool, Renato Martins, Patrick Rives, Noela Despre
IEEE International Conference on Robotics and Automation (ICRA), 2015
highlight talk / poster / bibtex

We proposed in this paper an ego-centric spherical representation to efficiently store a full RGB-D model of a 3D environment. For that, we used the notions of "keyframe" to select the most informative frames, along with the propation/correction of the depth image by representing the uncertainties of the geometry and the pose.

Dissertations

“Direct visual odometry and dense large-scale environment mapping from panoramic RGB-D images”
PhD Thesis prepared at INRIA, Ecole des Mines de Paris, Université Paris Sciences et Lettres, 2017
Renato Martins
Thesis committee:
  Philippe Martinet, École Centrale de Nantes (Chair)
  Cédric Demonceaux, Université Bourgogne Franche-Comté (External Examiner)
  Josechu Guerrero, Universidad de Zaragoza (External Examiner)
  Alessandro Correa Victorino, Universidade Federal de Minas Gerais (Committee Member)
  Florent Lafarge, INRIA (Committee Member)
  El Mustapha Mouaddib, Université de Picardie Jules Verne (Committee Member)
  Patrick Rives, INRIA (Supervisor)
Dissertation: PDF

“Exploiting redundancies and constraints between odometries and absolute sensors for ground robotics localization” (in Portuguese)
MSc Thesis in Electrical Engineering, Universidade Estadual de Campinas, 2013
Renato Martins
Master thesis committee:
  José Raul Azinheira, Instituto Superior Técnico (Committee Member)
  Wagner Caradori do Amaral, Universidade Estadual de Campinas (Committee Member)
  Ely Carneiro de Paiva, Universidade Estadual de Campinas (Co-Supervisor)
  Paulo A. Valente Ferreira, Universidade Estadual de Campinas (Supervisor)
  Samuel Siqueira Bueno, CTI Renato Archer (Co-Supervisor)
Dissertation: PDF

Other Publications and National Brazilian Conferences

“Autonomous navigation strategy between rows of crops based on LiDAR” (in Portuguese)
Randerson A. de Lemos, Gabriel Sobral, Luiz G. B. Mirisola, Renato Martins, Mauro F. Koyama, Ely C de Paiva, and Samuel S. Bueno
Brazilian Symposium on Intelligent Automation (SBAI), Porto Alegre , 2017
paper

This article presents an autonomous navigation strategy in cultivars (crops) planted in lines. With a low-cost sensory solution, the strategy is based on a single 2D sweeping LiDAR sensor.

“Kinematics and localization for land robotics applications using multiple encoders” (in Portuguese)
Renato Martins, Samuel S. Bueno, Luiz G. B. Mirisola, Ely C. de Paiva and P. Ferreira
Brazilian Symposium on Intelligent Automation (SBAI), São João del Rey , 2011
paper

This paper proposes a new 2D localization methodology that optimizes, in a least squares sense, the information gathered from multiple encoders (from four wheels and steering) mounted in an outdoor robotic vehicle. The optimization strategy is designed to reduce the drift from differential odometry. The model is evaluated with data from simulation and real data acquired with an eletric outdoor robotic vehicle of the project VERO.

“Fusion of GPS and odometry for unmanned ground vehicle localization” (in Portuguese)
Renato Martins, Samuel S. Bueno, Luiz G. B. Mirisola, Ely C. de Paiva and P. Ferreira
Brazilian Symposium on Intelligent Automation (SBAI), São João del Rey , 2011
paper

This paper proposes a localization methodology based on GPS and odometry fusion. An important aspect is a new odometry formulation, wich results from a least squares optimization of the information gathered from multiple encoders (four wheels and steering) of an outdoor robotic vehicle. The sensor fusion is evaluated using both EKF and UKF filters on real experimental data acquired with the eletric robotic vehicle of the project VERO.

“Implementation and Usage of a Sound-Tactile Model for Sightless People” (in Portuguese)
J. Vilhete Viegas d'Abreu, and Renato Martins
Avances en Sistemas e Informática , 2008
paper / project page

This paper describes the designing and construction of sound-­tactile mock-up models to support blind people exploration. The sound-­tactile mock-up models are from indoor and outdoor human-made spaces in the main campus of the University of Campinas.