We use sliding mode control theory to design a 3D vision based controller that is robust to bounded parametric estimation errors. First, a model of an eye-in-hand robotic system is derived and sources of uncertainties are listed. Additionally, bounds on the different uncertainties are discussed and their influence on the overall gain of the system is derived. Due to an appropriate selection of the sliding surface, based on the quaternion representation for rotations, a switching controller is proposed. Six degrees of freedom vision based tracking experiments under weak calibration conditions emphasize the practical efficiency of the algorithm.