This paper concerns the stability analysis of image-based visual servoing methods with respect to uncertainties on the depths of the observed object. In the recent past, research on image-based visual servoing has been concentrated on potential problems of stability and on robustness with respect to camera calibration errors. Only little attention, if any, has been devoted to the robustness of image-based visual servoing to depth estimation errors. It is generally believed that a rough approximation of the depth distribution is sufficient to ensure the stability of the control law. In this paper, we prove that the robustness domain is not so wide and that an extreme care must be taken when approximating the depth distribution.