In this paper, we present a generic and flexible system for vision-based robot control. The system integrates visual tracking and visual servoing approaches in a unifying framework. In this framework, the generality is obtained using a template matching algorithm based on an efficient second-order minimization. Contrarily to feature-based visual servoing schemes, we avoid the design of feature-dependent visual tracking algorithms. By integrating the visual tracking process with the visual servoing techniques, we can easily deal with constrained tasks. This reduces the computation cost and improves the precision of the system. The experimental results prove the efficiency of the unified system in real conditions.