Vision-Guided Pruning in Isaac Sim
UR5e live RGB-D approach, gated spur release, and home return in a two-tree Blender orchard.
Problem
A pruning arm has to approach a thin branch from wrist RGB-D and dual ToF, then release only when tracking and geometry still agree. Jaw occlusion can wipe out image features at the worst moment. A finished recording is not a finished cut, so failed tracking and failed closure stay in the public record.
Approach
The original UR5e and mock pruner run in Isaac Sim 6.0 / Isaac Lab 3 among both source Blender trees, exported to USD with bark maps and collisions. Seeded pyramidal Lucas–Kanade tracking back-projects RTX optical-Z depth into bounded Cartesian commands. Dual 8×8 ray-cast ToF, contact, and geometry gates must pass before a visual-jaw surrogate releases one rigid spur. An independent grader checks the saved capture rather than the renderer’s success label. Branch identity, axis, and radius come from mesh metadata, not a learned detector.
What I built
I built the Isaac capture loop, live RGB-D controller, release gates, independent sequence grader, and the published success and failure recordings. The robot, orchard meshes, and sensor frames come from pinned upstream sources. This repository composes them, measures one known target, and keeps failed runs visible.
Result
The selected success records 200 frames / 20 s, 68 vision commands, and 17/17 sequence checks. One selected spur releases at 7.8 s, falls 809.49 mm, and the arm returns within 0.001 mm of home. A separate failed episode stops at 7.7 s (confidence 0.14165 < 0.15). Discrete rigid-piece release, not wood fracture.
Keep the failure next to the success
Tracking confidence fell below 0.15 at 7.7 s in the failed episode. Closure stopped at 2/3; no release. The public recording is the failed episode, not a cleaned retake.
What drives the arm
Live control uses seeded LK features and RTX depth. Farneback flow is an offline diagnostic. Depth is simulator ground truth. The jaw is a visual proxy; there is no blade CAD or wood fracture model.