R&D
CENTER
KOENJA

Hand-Eye Calibration: Sending the Robot Where the Camera Sees

The camera says 'it is here' in its own coordinates, and the robot moves only in its own coordinates. This lesson explains the concept, procedure and accuracy tips of Hand-Eye calibration, which finds the transform between them.

Advanced4 min0 views
Key points
  1. Tell four coordinate systems apart: robot base, robot hand (flange/TCP), camera and object.
  2. A camera mounted on the robot is eye-in-hand. A camera fixed outside is eye-to-hand. They solve for different transforms.
  3. Image a reference object from several poses and solve AX = XB to get the transform X. There are several solvers, such as Tsai and Park.
  4. Accuracy depends on giving each pose a very different rotation and checking the result with the residual (RMS).

Vision finds a mushroom and reports "it is at (12, −40, 380) mm from the camera." For the robot to go there, you must convert this value to robot base coordinates. Finding this transform is Hand-Eye calibration. If the value is off by 1 mm, every grasp is off by 1 mm.

Four Coordinate Systems and Their Transforms

BaseRobot base FlangeRobot hand CameraCamera ObjectObject T_base←flangeGiven by robot X = T_flange←camFound by calibration T_cam←objMeasured by vision T_base←obj = T_base←flange · X · T_cam←obj
The eye-in-hand setup, with the camera on the robot hand. If you know X alone, which links the robot pose to the vision measurement, you get the object in robot coordinates.

A transform is written as a 4×4 matrix that holds position (3 values) and rotation (3 values) together. You already know two of the three transforms.

  • Base → hand: The robot controller computes it from the joint angles and reports it in real time.
  • Camera → object: Vision measures it.
  • Hand → camera (X): It depends on how the camera was mounted, so nobody knows it. Finding this is the calibration.

eye-in-hand and eye-to-hand

  • eye-in-hand — The camera is mounted on the robot hand and moves with it. It can image from close range at many positions, which suits wide areas and detailed inspection. The value to find is hand → camera.
  • eye-to-hand — The camera is fixed above the workbench. It can keep imaging while the robot moves. The value to find is base → camera.

RoboVision supports both setups.

AX = XB: Why Several Poses Are Needed

Fix a calibration plate (such as a chessboard) in place, move the robot through several poses, and image the plate. Between two poses, call the motion of the robot hand A and the motion of the plate seen by the camera B. They are the same rigid-body motion viewed in different coordinates, so AX = XB holds. The unknown X has 6 degrees of freedom, so one pair of poses cannot solve it. You need several poses with different rotation axes.

There are several ways to solve this equation. RoboVision provides all five OpenCV solvers (Tsai, Park, Horaud, Andreff, Daniilidis), so you can compare results on the same data. Methods that solve rotation and translation separately (Tsai, Park) and methods that solve them together (such as Daniilidis, using dual quaternions) differ slightly in how they respond to noise.

Calibrating Without a Plate

In the field, placing a plate is sometimes hard. fineAi also uses other methods.

  • Gripper-based calibration — Image the gripper itself instead of a plate. RoboVision achieved an RMS of 0.50 mm with it.
  • TCP point matching — In eye-to-hand, move the robot hand to several positions and image each position with a 3D camera. Then use the Kabsch algorithm to find the rotation and translation that best overlay the two point sets. Attaching a sphere to the tip and fitting the sphere center is more precise. Digital Twin Pick&Place uses this method.

Tips for Better Accuracy

  1. Large, varied rotations — The rotation axes between poses must differ a lot. With translation only, the rotation part of X is poorly determined.
  2. Enough poses — Take more than the minimum (10 to 20) to average out noise.
  3. Use the whole field of view — If the plate is imaged only in the center, errors at the edges, where lens distortion is large, stay hidden.
  4. Calibrate the camera intrinsics first — If lens distortion and focal length (intrinsic parameters) are wrong, Hand-Eye is wrong too.
  5. Verify with the residual — Use the result X to recompute the plate position in each pose and look at the residual (RMS). If only one pose stands out, remove it and solve again.
  6. Redo it when the mechanism changes — Recalibrate if the camera bracket was touched or a collision occurred. In operation, self-correction also helps: it collects the difference between predicted and measured positions and updates a small correction value.