R&D
CENTER
KOENJA

Physical AI: AI That Sees, Decides and Moves

Physical AI goes beyond showing inspection results on a screen. The robot uses those results to pick, move and finish objects. This lesson explains the components and the operating loop with a shiitake mushroom harvesting case.

Intro4 min0 views
Key points
  1. Physical AI repeats a loop in the real world: perception (vision), decision (AI), action (robot), confirmation (sensors).
  2. Calibration, which converts vision coordinates to robot coordinates, and the digital twin, which verifies motion before the real run, are the key links.
  3. Force/torque sensors and a finite state machine (FSM) let the robot stop safely or retry when things differ from expectations.
  4. Language models (LLM, VLM) join the loop to understand task instructions and explain results.

The result of a vision inspection is usually an OK/NG signal. Separate equipment takes that signal and removes the defective part. Physical AI goes one step further: the robot acts directly on what it saw and decided. It decides case by case where to pick, with what force to grip, and which way to twist.

Perceive2D·3D vision DecideDetect · plan grasp ActRobot · gripper ConfirmForce · recapture
The Physical AI operating loop. It does not end with one decision. It checks the result of each action and corrects the next one.

Components

Element Role In fineAi
Vision Perceive position, pose and state of the target Basler 2D, Zivid, Photoneo and EpicEye 3D cameras
AI model Detect targets, judge defects YOLO detection and segmentation, PatchCore/PaDiM anomaly detection
Robot Movement and work Rainbow Robotics RB5-850 and RB10-1300e collaborative robots
End tool Picking, welding, etc. OnRobot 2FG7, Tesollo DG-3F-M 3-finger (12-DOF) gripper
Sensor Confirm contact and force 6-axis force/torque sensor
Calibration Camera coordinates → robot coordinates Hand-Eye Calibration
Virtual verification Check path, collision and cycle in advance Digital Twin

Case: Automated Shiitake Harvesting

The CUBE FARM growing system is linked with RoboVision, and a robot picks fully grown shiitake mushrooms and puts them in a basket.

  1. Perceive — Image the growing rack with a 3D camera, find the mushrooms and fix their 3D positions.
  2. Decide — For each mushroom, generate candidate gripper poses and rank them by collision risk and stability. Choose a fitting one from the preset grasps defined in advance (18 types).
  3. Act — The robot approaches, grips with the 3-finger gripper, and twists the wrist left and right (twist pluck) to pick.
  4. Confirm — Check contact and grip state with the force/torque sensor. If it differs from what is expected, the finite state machine (FSM) retries or moves to the next mushroom.
  5. Record — The harvest result goes back to CUBE FARM and is managed with the growth data.

Case: Weld Inspection

A camera and lights are mounted at the end of the robot arm, and it captures while moving along the weld line. The images are used to find defects, and the 3D bead cross-section is measured to assign an ISO 5817 level. Results marked as escapes (missed defects) or false rejects (overkill) accumulate in the dataset and are used for retraining.

What Language Models Do

The finetheAi Physical AI Platform (PaaS) uses a language model that runs locally (Ollama).

  • Task instructions — It breaks a natural-language instruction such as "when basket 3 is full, switch to basket 4" into task steps. An orchestrator built on LangGraph hands work to several agents.
  • Result interpretation — A vision-language model (VLM) looks at an anomaly detection heatmap and explains in a sentence what kind of anomaly it is.
  • Operation reports — It summarizes the day's results into a daily report.

The language model does not move the robot directly. A control layer that follows verified paths and safety rules performs the actual motion, and the language model handles planning and explanation on top of it.

Safety First

Even a collaborative robot that works next to people needs safety devices as a baseline. RoboVision supports emergency stop (E-STOP) and pause, and has a built-in simulation mode to check motion without a real robot. Check a new motion in simulation and the digital twin first, then move it to the real robot.