R&D
CENTER
KOENJA

Digital Twin and Pick & Place: Verify Virtually Before the Real Robot Moves

If you copy a robot cell into virtual space, you can check paths, collisions and cycle time before the real robot moves. This lesson follows the whole Pick & Place process, from 3D recognition to grasp planning and live mirroring.

Core4 min0 views
Key points
  1. A digital twin is a virtual model of the robot, gripper and workbench geometry and kinematics. It uses the same coordinate system as the real cell.
  2. Pick & Place runs in this order: 3D recognition, grasp candidate generation and ranking, inverse-kinematics path, force-guided approach, place planning.
  3. Live mirroring receives the real joint angles and makes the twin follow them. It stops if the motion differs from what is expected.
  4. Collecting operating data (predicted vs measured) and updating the correction values keeps accuracy over time.

Testing a new motion for the first time on a real robot is risky and slow. A path may hit a jig, a joint may reach its limit, or the cycle may run longer than expected. A digital twin finds these problems first, in virtual space.

What a Twin Is Built From

  • URDF — A standard format that describes the link (arm segment) geometry and the joint positions and limits of a robot. fineAi Digital Twin builds the RB5-850E and RB10-1300e robots, the DG-3F-M gripper, the staging rack and the basket from URDF.
  • Kinematics — Forward kinematics (FK) gets the fingertip position from the joint angles. Inverse kinematics (IK) gets the joint angles from a desired fingertip position. The twin computes paths quickly with analytic IK.
  • Same coordinate system — The twin's coordinates must match the real cell for you to trust the virtual results. For this, it uses the Hand-Eye Calibration result as is.

The Whole Pick & Place Process

① 3D recognitionCloud · registration ② Grasp candidatesGenerate·rank·verify ③ PathIK · check in twin ④ Approach·graspForce-guided advance ⑤ PlaceGrid · low drop Live mirroring: the twin follows the real joint angles and stops on mismatch
Processing flow of Digital Twin Pick&Place v1.1. Every step appears on the twin screen.

① 3D recognition — Clean up the point cloud from the 3D camera (EpicEye Pixel Pro) and separate the objects. Then match them to the registered template to get position and pose. Use YOLO detection alongside if needed. The lesson 3D Vision and Point Clouds covers the processing in detail.

② Grasp candidates — For each object, generate several poses the gripper can hold. Rank them by collision with nearby objects, stability of the surface the fingers touch, and whether the robot can reach them. Verify from the best candidate down and pick the first one that passes.

③ Path — Compute the joint path to the chosen pose with IK. Replay it in the twin to confirm it moves without collision.

④ Approach and grasp — In the last few cm, the 3D measurement error shows up directly. So the robot does not go to a fixed position in one move. It advances 6 mm at a time and uses the force sensor to check whether the gripper pads have made contact. For a soft target such as a mushroom, after gripping it twists the wrist ±15° three times to detach it (twist pluck).

⑤ Place — Plan an empty spot in a basket or on a grid, and place there. For targets that bruise easily, place from a low height (low drop).

Live Mirroring: The Twin Follows the Real Robot

While the real robot moves, the twin receives the joint angles that arrive at the controller data port and moves the virtual robot the same way. You can see that the robot on screen and the real robot are in the same pose. If the planned path and the real motion differ by more than a set amount, it stops immediately.

Self-Correction During Operation

If the camera bracket sags slightly or the mechanism expands with temperature, the original calibration values drift little by little. During operation, the system collects the difference between the "predicted position" and the "position actually touched". It then uses least squares to recompute the small correction (ops offset) that best explains that residual. Accuracy holds without a person recalibrating.

What the Twin Gives You

  • Safety — Find collisions and joint limits before the real run.
  • Time — Prepare a new motion without stopping the line (offline teaching).
  • Explanation — Record the session and replay the moment a problem occurred to find the cause. RoboVision also offers a virtual camera to check the field of view and capture coverage in advance.