ESM Visual Tracking

Demonstrations

Visual Tracking

The ESM visual tracking has been tested on several sequences under disparate illumination conditions and it works with any camera with a single viewpoint. The algorithm is intrinsically robust to partial occlusion but it has been improved by using robust optimization techniques. The videos provided below shows several examples of the ESM visual tracking algorithm.

• Visual tracking of planar objects with single viewpoint cameras

In the first video, a planar object is tracked using a perspective camera despite several illumination changes [1] [3]. Strong camera displacements can be handled in real-time by the ESM visual tracking. In the second video, a chip board is tracked at very high speed (100 Hz). The specular reflections do not affect the tracking and the area of interest can be partially out of the camera field of view. The chip board is not exactly planar and undergo severe projective distortion in the image. In the third video, the back of a car is tracked. The initial area of interest is only 40x40 pixel while the final one is 5 times bigger. Despite the very high change of scale the tracking is accurately performed. The ESM visual tracking can be used with any single viewpoint camera [2]. For example, in the last two videos, a planar object is tracked in omnidirectional images acquired with a parabolic mirror. In the fifth video, the image on the top-left corner is the reprojection of the tracked area of interested in a reference frame.



1. S. Benhimane and E. Malis:
   "Homography-based 2D Visual Tracking and Servoing"
   International Journal of Robotic Research (Special Issue on Vision and Robotics joint with the International Journal of Computer Vision), 2007.


2. C. Mei, S. Benhimane, E. Malis, P. Rives:
   "Homography-based Tracking for Central Catadioptric Cameras"
   IEEE/RSJ International Conference on Intelligent Robots Systems, Beijing, China, October 2006.


3. S. Benhimane and E. Malis:
   "Real-time image-based tracking of planes using efficient second-order minimization"
   IEEE/RSJ International Conference on Intelligent Robots Systems, Sendai, Japan, October 2004.

• Visual tracking of rigid objects with unknow shape

In each video, an area of interest of a rigid object is tracked using a perspective camera. A reference template is selected manually in the first image of the sequence. The shape of the rigid object is unknown and it is thus recovered online during the visual tracking. In the videos, a red grid is superposed to the current area of interest in order to show the 3D structure of the object.



1. E. Malis:
   "An efficient unified approach to direct visual tracking of rigid and deformable surfaces"
   accepted to IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, USA, October 2007.

• Visual tracking of deformable surfaces

In each video, a part of a deformable object is tracked using a perspective camera. A reference template is selected manually in the first image of the sequence. In the top left corner we show the undeformed image (i.e. the reprojection of the current image in the reference frame). The undeformed image is almost unchanged during the entire sequence proving that the ESM visual tracking is able to estimate the deformation of each pixel of the reference template.



1. E. Malis:
   "An efficient unified approach to direct visual tracking of rigid and deformable surfaces"
   accepted to IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, USA, October 2007.

• Visual tracking robust to arbitray illumination changes

We propose a new approach to the direct image alignment of either Lambertian or non-Lambertian objects under shadows, inter-reflections, glints as well as ambient, diffuse and specular reflections which may vary in power, type, number and space. The method is based on a proposed model of illumination changes together with an appropriate geometric model of image motion. The parameters related to these models are simultaneously obtained through the ESM optimization technique which minimizes directly the intensity discrepancies. Comparison results with existing direct methods show significant improvements in the tracking performance.



1. G. Silveira and E. Malis:
   "Real-time Visual Tracking under Arbitrary Illumination Changes"
   IEEE Computer Vision and Pattern Recognition, Minneapolis, USA, June 2007.

• Outliers rejection for robust visual tracking

When the models used in the visual tracking are not accurate enough the optimization may fail. For example, if the target is partially occluded the overall motion of the area of interest will not be coherent. In the general case, outlier measures can be discarded by using robust cost functions [1]. We have tested the use of M-estimators in the ESM visual tracking algorithm. The video below show an example of the robustness of the algorithm when tracking a planar object with severe illumination changes and specular reflections. In this case, illumination changes and specular reflections are not explicitly modeled (see section above) but treated as outliers. Although M-estimators allows to handle the case of partial occlusions that can be hardly modeled, the price to pay is a higher computation time and a lower convergence rate.



1. E. Malis and E. Marchand:
   "Experiments with Robust Estimation Techniques in Real-time Robot Vision"
   IEEE/RSJ International Conference on Intelligent Robots Systems, Beijing, China, October 2006.

• Stereo visual tracking

The ESM technique has been successfully applied to the visual tracking using a stereo pair. The video below show an example of the visual tracking of a sphere. The user selects a region of interest in the left image (the blue rectangle in this case). After the corresponding region in the right image is found, the visual tracking starts.

Visual Servoing

• Positioning a 6 d.o.f. robot manipulator with respect to any rigid object using the direct visual servoing

We propose a new approach (named direct visual servoing) to vision-based control that does not require or estimate any 3D metric information, while also not depending on the object's shape or the camera motion. Thus, we do not rely on prior knowledge (leading to system flexibility), as well as we achieve robustness to camera calibration parameters. The sole requirement about the object is to be rigid. In addition, the visual tracking exploits all possible image information (pixel intensities are directly used without any feature extraction process) allowing us to attain high levels of accuracy while being computationally efficient. This latter is also important since real-time performance is always a major concern in robotic systems. Finally, the proposed control law possesses a very large domain of convergence due to a straightforward path planning step. This is highly desirable so that tasks can be performed despite large initial displacements.



1. G. Silveira and E. Malis :
   "Direct Visual Servoing with respect to Rigid Objects"
   accepted to IEEE/RSJ International Conference on Intelligent Robot and Systems, USA, October 2007.

• Positioning a 6 d.o.f. robot manipulator with with respect to planar objects using the homography-based 2D visual servoing

The homography-based 2D visual servoing [1] [3] can be used to position a robot manipulator with respect to a planar object without having any measure (off-line or on-line) of the normal to the plane. Only the homography between the current and the reference image is needed for computing an error which is isomorph to the camera pose and to compute a stable control law. Thus, the same control law can be used with any single viewpoint camera [2]. In the first video, the object is static and the robot is correctly positioned despite the initial displacement is very big. In the second video, the target is moving at 0.1 m/s and the robot is able to correctly track the target without any additional filter.



1. S. Benhimane and E. Malis:
   "Homography-based 2D Visual Tracking and Servoing"
   International Journal of Robotic Research (Special Issue on Vision and Robotics joint with the International Journal of Computer Vision), 2007.


2. S. Benhimane and E. Malis :
   "Homography-based 2D Visual Servoing"
   IEEE International Conference on Robotics and Automation, Orlando, USA, May 2006.


3. S. Benhimane and E. Malis :
   "A new approach to vision-based robot control with omni-directional cameras"
   IEEE International Conference on Robotics and Automation, Orlando, USA, May 2006.

• Positioning with the 2 1/2 D visual servoing

The ESM algorithm can be used to integrate visual tracking and a large class of vision-based control laws [1] into a unified approach [2]. The videos below show a positioning task using the 2 1/2 D visual servoing . The ESM algorithm is also used for matching the current image to the reference template with a coarse approximation of the homography matrix. An approximation of the normal to the plane is needed to have an unique solution for the homography decomposition.



1. E. Malis and F. Chaumette:
   "Theoretical improvements in the stability analysis of a new class of model-free visual servoing methods"
   IEEE Transaction on Robotics and Automation, 18(2):176-186, April 2002.


2. E. Malis and S. Benhimane:
   "A unified approach to visual tracking and servoing"
   Robotics and Autonomous Systems, Volume 52, Issue 1, Pages 39-52, 31 July 2005.